NASA: Earned Value Management Implementation across Major Spaceflight Projects Is Uneven
Published: 2012-11-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NASA historically has experienced cost growth and schedule slippage in its portfolio of major projects and has taken actions to improve in this area, including adopting the use of EVM. EVM is a tool developed to help project managers monitor risks. GAO was asked to examine (1) the extent to which NASA is using EVM to manage its major space flight acquisitions, (2) the challenges that NASA has faced in implementing an effective EVM system, and (3) NASA's efforts to improve its use of EVM. To address these questions, GAO obtained contractor and project EVM data and used established formulas and tools to analyze the data and assess NASA's implementation of EVM on 10 major spaceflight projects; interviewed relevant NASA headquarters, center and mission directorate officials on their views on EVM; and reviewed prior reports on EVM and organizational transformations. GAO compared NASA policies and guidance on EVM to best practices contained in GAO's cost estimating best practices guide.
section_title: What GAO Found
paragraphs: The National Aeronautics and Space Administration's (NASA) 10 major spaceflight projects discussed in this report have not yet fully implemented earned value management (EVM). As a result, NASA is not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. GAO assessed the 10 projects against three fundamental EVM practices that, according to GAO's best practices cost guide, are necessary for maintaining a reliable EVM system. GAO found shortfalls in two of three fundamental practices. Specifically, we found that
More than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard.
Only 4 of the 10 projects established formal surveillance reviews, which ensure that key data produced by the system was reliable. The remaining 6 projects provided evidence of monthly EVM data reviews; however, the rigor of both the formal and informal surveillance reviews is questionable given the numerous data anomalies GAO found.
GAO also found that 3 projects had reliable EVM data while 7 had only partially reliable data. For the EVM data to be considered reliable per best practices it must be complete and accurate with all data anomalies explained.
NASA EVM focal points, headquarters officials, project representatives, and program executives cited cultural and other challenges as impediments to the effective use of EVM at the agency. Traditionally, NASA's culture has focused on managing science and engineering challenges and not on monitoring cost and schedule data, like an effective EVM system produces. As a result, several representatives said this information traditionally has not been valued across the agency. This sentiment was also echoed in a NASA study of EVM implementation. Also cited as a challenge to the effective use of EVM was NASA's insufficient number of staff with the skills to analyze EVM data. Without a sufficient number of staff with such skills, NASA's ability to conduct a sound analysis of the EVM data is limited. However, NASA has not conducted an EVM skills gap analysis to determine the extent of its workforce needs.
NASA has undertaken several initiatives aimed at improving the agency's use of EVM. For example, NASA strengthened its spaceflight management policy to reflect the industry EVM standard and has developed the processes and tools for projects to meet these standards through its new EVM system. While these are positive steps, the revised policy contains only the minimum requirements for earned value management. For example, it lacks a requirement for rigorous surveillance of how projects are implementing EVM and also does not require use of the agency's newly developed EVM system to help meet the new requirements. NASA has attempted to address EVM shortcomings through policy changes over the years, but these efforts have failed to adequately address the cultural resistance to implementing EVM.
section_title: What GAO Recommends
paragraphs: GAO recommends that NASA establish a time frame for requiring new spaceflight projects to implement its new EVM system; conduct an EVM skills gap assessment; develop a change management plan for EVM; and strengthen its EVM requirements by requiring projects to implement formal EVM surveillance. NASA concurred with two recommendations and partially concurred with two others citing resource constraints. Despite NASA's plans to address some issues GAO identified, not addressing all key issues lessens the usefulness of EVM at NASA.

section_title: Letter
paragraphs: The National Aeronautics and Space Administration (NASA) has a  portfolio of 21 major spaceflight projects that are expected to cost $19.4  billion to develop. Historically, NASA has experienced problems with  persistent cost growth and schedule slippage in the majority of its major  projects. As a result, NASA acquisition management has remained on  GAO’s high risk list since it was first introduced in 1990. Our work has  shown that several factors contribute to NASA’s cost and schedule  performance, including poor cost estimating and underestimating risks  associated with the development of its major systems.
Over the years, NASA has undertaken a number of actions to improve its  acquisition management function and has shown some progress in  improving its performance. One of these actions has been to adopt  earned value management (EVM). EVM is a project management tool  developed by the Department of Defense in the 1960s to help project  managers monitor risks. EVM measures the value of work accomplished  in a given period and compares it with the planned value of work  scheduled for that period and the actual cost of work accomplished.  EVM’s intended purpose is to integrate a project’s cost, schedule, and  technical efforts for management and provide reliable data to decision  makers.
Starting in 1997, NASA began to require its projects to implement EVM  on all significant contracts. In 2005, NASA broadened its application of  EVM to encompass significant project efforts implemented by in-house  civil service and associated support contractor personnel. NASA has had  uneven success in effectively implementing EVM, according to GAO and  NASA Inspector General reports conducted since 1999. NASA has  recognized a need for improved EVM implementation, and has an  ongoing effort focused on doing so. In light of these issues, the House  Science, Space, and Technology Committee and the Senate Committee  on Commerce, Science, and Transportation asked GAO to examine the  use of EVM at NASA. Specifically, we assessed (1) the extent to which  NASA is using EVM to manage its major space flight acquisitions, (2) the  challenges that NASA faces in implementing an effective EVM system,  and (3) NASA’s efforts to improve its use of EVM.
To determine the extent to which NASA’s major spaceflight projects are  using EVM to manage the acquisition, we reviewed 10 major spaceflight  projects, each with a life cycle cost estimate of more than $250 million  that had been approved to proceed into the implementation phase of  development. Collectively, the projects we reviewed will cost over $14  billion to develop (See table 2 for a list of the projects). We collected all  available EVM data for the 10 projects for the period of August 2010 to  August 2011 and used established earned value formulas and tools to  identify cost and schedule variances and trends. We also assessed the  projects’ implementation of three fundamental EVM practices that,  according to GAO’s Cost Estimating and Assessment Guide, are  necessary for maintaining a reliable EVM system—using a certified  system, conducting integrated baseline reviews, and performing  surveillance. We analyzed monthly project management review briefings  to support our analysis. To assess the reliability of the cost data, we  electronically tested the data for anomalies, and reviewed relevant project  documentation and interviewed agency and project officials about the  data. To determine the challenges the agency faces in its efforts to  implement an effective EVM system, we developed standard sets of  questions and interviewed NASA headquarters officials, program  executives, and EVM focal points at each center and the Human  Exploration and Operations and Science Mission Directorates to discuss  their roles as well as the extent to which EVM data is used to inform  decision making. We also reviewed prior GAO and relevant NASA  Inspector General reports on EVM. Also, we submitted written questions  to the project offices to obtain their views on implementing EVM on their  projects. To determine the steps that NASA is taking to improve its use of  EVM, we compared NASA policies and guidance intended to improve the  agency’s implementation of EVM to best practices for earned value  management as documented in GAO’s cost guide. We also interviewed  agency officials responsible for developing and implementing the NASA’s  new system to discuss ongoing initiatives. For additional details on our  scope and methodology, see appendix I.
We conducted this performance audit from July 2011 through November  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: EVM is a project management tool that, when properly used, can provide  accurate assessments of project progress, produce early warning signs of  impending schedule delays and cost overruns, and provide unbiased  estimates of anticipated costs at completion. Pulling together essential  cost, schedule, and technical information in a meaningful, coherent  fashion is a challenge for most projects. Without such information,  managers can have a distorted view of a project’s status and risks. EVM  also allows individuals outside the project to see a standardized metric  describing the cost and schedule performance of that particular project  and compare it consistently with other projects.
EVM measures the value of work accomplished in a given period and  compares it with the planned value of work scheduled for that period and  with the actual cost of work accomplished. Differences in these values are  measured in both cost and schedule variances. Positive variances  indicate that activities are costing less or are completed ahead of  schedule. Negative variances indicate activities are costing more or are  falling behind schedule. For example, if a contractor completed $5 million  worth of work and the work actually cost $6.7 million, there would be a  $1.7 million negative cost variance. Schedule variances are also  measured in dollars, but they compare the earned value of the work  completed with the value of work that was expected to be completed. For  example, if a contractor completed $5 million worth of work at the end of  the month but was budgeted to complete $10 million worth of work, there  would be a $5 million negative schedule variance.
Earned value provides information that is necessary for understanding the  health of a project and an objective view of project status. Cost and  schedule variances can also be used in estimating the cost and time  needed to complete the project. While some data that an EVM system  produces are retrospective and indicate performance to date, EVM data  can also be used to predict future performance. For example, estimates  at completion for a project can be calculated by using efficiency indices  which are based on a project’s past cost and schedule performance. See  appendix II for additional information on the importance of EVM.
A project’s EVM data comes from multiple sources. For example, each  contractor that supports a project will produce and deliver EVM reports to  the project for the work they and their subcontractors perform, if the  contract so requires. In addition, the project will collect EVM information  for the work that it performs in-house at a NASA center. All of this lower  level EVM data can then be consolidated at the project level to produce a  project level EVM report. Pulling together EVM data from multiple levels  into a project level report gives the project a comprehensive outlook of its  cost and schedule, and provides the project manager with early warning  of potential cost and schedule overruns.
EVM has evolved from an industrial engineering tool to a government and  industry best practice, providing improved information to conduct  oversight of acquisition programs. As such, it is guided by industry best  practices and standard, and is required by regulations and requirements  at the federal government and the agency level at NASA. These  requirements and standards are summarized in table 1 below.
Almost two decades of NASA’s past efforts to improve its use of earned  value management have had uneven success. An EVM Focal Point  Council was created in 1996 and focal points were designated at each  NASA center and the Office of Procurement and the Office of the Chief  Financial Officer to provide an open forum to share experiences and  develop a network of support within the NASA EVM community. In 1997,  the agency issued NASA Policy Directive 9501.3 “Earned Value  Management Performance” which established the basis for applying EVM  to NASA contracts. Prior to this policy, centers used their own individual  policies on performance measurement systems. However, in 1999, the  NASA Inspector General reported that EVM policy was not consolidated  as an overall program management responsibility and that the Agency  Program Management Council did not receive comprehensive EVM  information.policy from the Office of the Chief Financial Officer to the Office of the  Chief Engineer to emphasize EVM as a project management tool, rather  than a financial management tool.
As a result, in 2003, NASA shifted responsibility for the EVM  In 2004, GAO reported that only 2 of 10 NASA projects reviewed used  EVM consistently and appropriately. Several barriers to EVM implementation were identified, such as lack of reliable financial data,  trained EVM staff, data analysis tools, and incentives. Among other  things, GAO recommended that NASA take action to ensure that a true  EVM system is used as an organizational management tool to bring cost  to the forefront in NASA’s decision-making process and that acquisition  and EVM management policies and procedures be enforced. In response  to our recommendations, NASA stated that it was updating NASA  Procedural Requirements 7120.5, its program and project management  processes and requirements policy, to improve its cost estimating and  ensure that its cost estimate and earned value analyses were effectively  used, and the updated policy was issued in 2005. Similar to other  agencies in our 2009 report on governmentwide use of EVM, we reported  on weaknesses in NASA’s EVM policies and practices and recommended  that the agency modify its policies governing EVM to ensure that they are  consistent with best practices. In particular, we found problems with the  EVM training requirements for personnel responsible for investment  oversight and management responsibilities. In addition, NASA’s policy for  revising project cost and schedule baselines did not have adequately  defined criteria on the acceptable reasons for permitting a rebaselining.  We also found weaknesses in how the NASA projects we reviewed  implemented EVM and managed their negative performance trends.  NASA acknowledged the identified weaknesses and stated that it was  revising its NASA procedural requirements for programs and projects to  include expanded and strengthened policies governing EVM application  and processes, and revised policies for rebaselining projects.
In January 2010, the Agency Program Management Council approved the  funding for the EVM capability project to develop an EVM system that  complies with the guidelines in ANSI/EIA-748. Another goal of the project  was to determine whether it was feasible to implement a single EVM  system that would integrate the scope, schedule, and budget of EVM data  for NASA’s in-house managed efforts and contractor data across the  agency. According to agency officials, NASA invested about $2 million  into the capability project to pilot the EVM system through two projects:  the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) and the  Constellation Extra-Vehicular Activity project. Through the pilots, the  capability project demonstrated that an agency-wide EVM system was  feasible. As a result, a finalized set of processes, tools, guidance, and  training products that compose NASA’s new EVM system was developed.  This new system was peer reviewed and approved by a panel of EVM  experts.
section_title: NASA Projects Have Not Consistently Implemented Key EVM Practices and Most Did Not Have Access to Reliable EVM Data
paragraphs: Our assessment of 10 major spaceflight projects showed that NASA has  not yet fully implemented EVM and thus is not taking full advantage of an  important tool that could help reduce acquisition risk. GAO found that the  projects had shortfalls in two of the three fundamental practices that we  assessed. Specifically, we found that half of the projects did not use an  EVM system that was certified as compliant with the ANSI/EIA-748  standard. Most of the projects conducted an integrated baseline review  (IBR), a practice that ensures the performance measurement baseline  reflects all requirements and that resources are adequate to complete the  work. Specifically, 9 projects conducted an IBR for their overall efforts;  however, the Stratospheric Observatory for Infrared Astronomy (SOFIA)  project office only conducted an IBR of one of its major contractors. In  addition, we found that only 4 of the 10 projects had established formal  independent surveillance reviews to ensure that key elements of the EVM  process were maintained over time so that the data produced by the  system provided timely indications of actual or potential problems. For the  6 projects that did not have formal independent surveillance in place,  each provided evidence that they instituted monthly EVM data reviews,  which according to project officials, helps them to continually monitor cost  and schedule performance. However, the rigor of both the formal and  informal surveillance reviews is questionable given the numerous EVM  data anomalies we found in the monthly EVM reports. Specifically, we  found many unexplained anomalies, such as the presence of negative  numbers or missing data, which caused us to question the reliability of the  data. Out of the 10 projects we reviewed, we found that just 3 projects  had reliable EVM data while the remaining 7 had only partially reliable  data.
Overall, the projects are using EVM, but NASA has not consistently  implemented EVM across these projects. For example, we found that  several projects were not implementing EVM at the project level, which is  considered a best practice. Table 2 summarizes the performance of each  of the 10 projects in meeting the three fundamental EVM practices and  the reliability of the data.
Of the 10 projects we reviewed, 4 projects had a certified EVM system, 3  did not, and 3 had a mixture in which some contractors and  subcontractors had certified systems and some did not. When an EVM  system is certified, the agency has assurance that the implemented  system was validated for compliance with the ANSI/EIA-748 standard by  independent and qualified staff and therefore can be considered to  provide reliable and valid data from which to manage a project. The  Global Precipitation Measurement (GPM), Tracking and Data Relay  Satellite System (TDRS), Landsat Data Continuity Mission (LDCM), and  James Webb Space Telescope (JWST) were the only projects that  provided evidence that the contract performance reports provided came  from EVM systems that were certified as compliant with the ANSI/EIA-748  standard. The Lunar Atmosphere and Dust Environment Explorer  (LADEE), Magnetospheric Multiscale (MMS) and Radiation Belt Storm  Probes (RBSP) projects did not have EVM systems that were certified to  be compliant with the ASNI/EIA-748 standard. Finally, the Jet Propulsion  Laboratory, a federally funded research and development center that the  California Institute of Technology manages under a contract with NASA,  was the only NASA Center with a certified EVM system. The Jet  Propulsion Laboratory is responsible for managing the Orbiting Carbon  Observatory 2 (OCO-2) project. The Mars Atmosphere and Volatile  Evolution Mission (MAVEN) and SOFIA prime contractors also had  certified systems; however, their project offices did not. NASA does not  require a certified EVM system for their in-house work.
Using the project’s integrated master schedule and contract performance  reports, we assessed the EVM data provided by the projects against  selected fundamental ANSI/EIA guidelines to determine the extent to  which each project’s EVM system, whether certified or not, was meeting  them. The guidelines we reviewed included whether the work breakdown  structure (WBS)—which provides the basis of the project schedule—was  consistent between the EVM report and the schedule, whether the  schedule identified significant task interdependencies, and whether the  project had identified a time-phased budget baseline for tracking cost and  schedule variances. As shown in figure 1, a work breakdown structure  breaks down product-oriented elements into a hierarchical structure that  shows how elements relate to one another as well as to the overall end  product. By subdividing a project into smaller elements, management can  more easily plan and schedule the program’s activities and assign  responsibility for the work.
We found that even for the projects that had certified systems, there were  problems with consistency between the WBS and the EVM report and the  schedule. For example, we found discrepancies in the hierarchical  structure and numbering of WBS elements for JWST, an $8.8 billion  project. Specifically, the project’s WBS dictionary showed mission  assurance efforts numbered differently than contractor reports for two  contractors, each of which had mission assurance labeled with different  WBS numbers. NASA officials explained that neither the spacecraft nor  the near infrared camera contractor was required to follow the project- level WBS structure or numbering scheme. NASA officials explained that  while it is not a requirement for the project and contractor WBSs to be the  same, it is recommended that the prime contractor lower-level WBS  numbering scheme be consistent with the overall project WBS numbering  format. Doing so allows easier total project integration of cost and EVM  data for project reporting. Consistency of the WBS element between the  cost estimate and the schedule facilitates updating the cost estimate with  actual costs and ensures there is correlation between the cost estimate  and schedule.
Our review of the project schedules also revealed that about half of the  schedules were missing predecessor and/or successor dependencies  and had constraints that prevented the schedule from responding  properly to updates. Since the schedule is the foundation for the EVM  baseline, it must be properly sequenced. This means knowing how one  activity (the predecessor) affects another (the successor) and how each  affects the critical path. When the schedule is not sequenced correctly,  the reliability of the EVM data is called into question. Our review found  that the MMS project was missing dependencies for 31 percent of its  remaining activities for its instrument suite contract and 36 percent of the  remaining activities for the instrument suite were constrained. Due to  the major sequencing issues in the MMS instrument schedule, we  questioned the reliability of the overall network and the schedule’s ability  to correctly calculate float values and the critical path. MMS project  officials said they believed many of the constrained activities we found  were not valid because they reside in another schedule. In addition,  officials said some of the constraints found are in the Harness area, and if  removed, these constraints would have no effect on the overall schedule.  Furthermore, MMS officials said some of the sequencing issues may be  caused by manual integration because some instrument provider  schedules are in Microsoft Project and others are in Primavera, and  therefore it is not possible to ensure all tasks have been linked properly.  As part of their schedule health check process, the MMS project  scheduler tests the schedule for missing dependencies, logic errors, and  reasonable durations and the results are shared with the project office  and the contractor so that appropriate action can be taken. However,  when we removed the 15 level of effort type activities from the missing  dependencies count, the schedule still showed 28 percent of the  remaining activities missing dependencies. We also removed the 14 level  of effort type activities and 4 Harness from the constraint count, the  schedule still showed 33 percent of the remaining activities were  constrained. MMS program officials said that they have a process in place  to manage float and the critical path, however, the schedule we reviewed  still showed significant sequencing issues.
Finally, we found that 4 of the 14 the schedules we analyzed were not  resource loaded. This means that the schedule did not have the required  labor, materials, equipment, and other resources assigned to the  appropriate activities. When the schedule is not resource loaded, then  costs need to be spread over time using some other method that may not  be as straightforward as having the costs integrated directly within the  schedule. Having a resource-loaded schedule is a best practice for  developing the time phased budget baseline. The time-phased budget  baseline represents the plan that performance is measured against for  the life of a project. It takes into account that program activities occur in a  sequenced order, based on finite resources, with budgets representing  those resources spread over time. Deviations from the baseline identify  areas where management should focus their attention.
subsections: section_title: Majority of Projects Conducted an Integrated Baseline Review
paragraphs: In keeping with best practices, 9 of the 10 projects conducted integrated  baseline reviews. An IBR is an evaluation of the performance  measurement baseline—the foundation for an EVM system—to  determine whether all project requirements have been addressed, risks  have been identified, mitigation plans are in place, and available and  planned resources are sufficient to complete the work. Conducting an IBR  increases confidence that the performance measurement baseline  provides reliable cost and schedule data for managing the project and  that it projects accurate estimated costs at completion. OMB has  endorsed the IBR as a critical process for risk management on major  investments and requires agencies to conduct IBRs for all contracts that  require EVM. Since an IBR’s goal is to verify that the technical baseline’s  budget and schedule are adequate for performing the work, it offers many  key benefits such as laying a solid foundation for successfully executing  the project and enabling better understanding of the risks.
Officials for the SOFIA project did not conduct an integrated baseline  review at the project level; however, its prime contractor for the  engineering and modification of the airborne observatory platform did  conduct an integrated baseline review. According to project officials, the  lack of a project-level IBR is related to the EVM system being  implemented “on the fly” late in the development phase for SOFIA, as a  result of an audit recommendation in 2010. However, project officials  noted that the EVM baseline was established concurrently with an agency  approved re-plan and Joint Cost and Schedule Confidence Level analysis  in 2009 and 2010 and was reviewed by a Standing Review Board as part  of that process.
section_title: Majority of Projects Do Not Have a Comprehensive Surveillance System in Place
paragraphs: Four of the 10 projects we assessed had a comprehensive EVM  surveillance system in place. Of the remaining 6 projects, 1 had formal  surveillance at the project level but its contractor did not, and 2 projects  did not have a formal surveillance system at the project level, only their  prime contractors did. The remaining 3 projects did not have any formal  surveillance, but provided evidence that EVM data, such as cost and  schedule variances, were being reviewed during their monthly status  reviews. Beyond reviewing cost and schedule variances and variances at  completion, formal surveillance reviews ensure that the processes and  procedures continue to satisfy the ANSI/EIA EVM guidelines. A formal  surveillance plan involves establishing an independent surveillance  organization with members who have practical experience using EVM.  This organization then conducts periodic surveillance reviews to ensure  the integrity of the contractor’s EVM system and where necessary  discusses corrective actions to mitigate risks and manage cost and  schedule performance. Effective surveillance ensures that the key  elements of the EVM process are maintained over time and on  subsequent applications.
NASA delegates surveillance of contractor EVM systems to the Defense  Contract Management Agency (DCMA); however, NASA has no entity to  perform independent surveillance reviews to ensure that the ANSI/EIA- 748 standard is being met for EVM efforts performed in-house or by  nonprofit organizations. Without an independent surveillance function, an  organization’s ability to use EVM as intended may be hampered since  surveillance monitors problems with the performance measurement  baseline and EVM data. If the kinds of problems that formal surveillance  can identify go undetected, EVM data may be distorted and may not be  meaningful for decision making.
section_title: Unreliable EVM Data Limit NASA’s Ability to Measure Project Performance
paragraphs: Only 3 of the 10 projects we reviewed, MAVEN, RBSP, and OCO-2,  produced fully reliable data for managing the project and reporting status.  The other projects only partially met the criterion to have an EVM system  in place that produces reliable data. If done correctly, EVM data can  provide an objective means for measuring project status and forecasting  potential project cost overruns and schedule slippages so that timely  action can be taken to minimize their impact. To do so, however, the  underlying EVM data must be reliable, meaning that they are complete  and accurate and all data anomalies are explained. In our analysis, we  found multiple cases of data anomalies that caused us to question the  reliability of the EVM data. For example, we found several EVM reports  where a contractor reported that no work was planned or accomplished,  but actual costs were incurred without an explanation in the variance  analysis report to say why this happened. Additionally, we found cases  where a contractor reported that work was planned and actual costs were  incurred, but a negative amount of work was performed—work that was  previously reported as completed was now reported as not completed.  Further, we also found several instances where a project reported an  estimate at completion but no budget at completion. Finally, we found  instances of negative values in the EVM reports. When explanations were  provided in the variance analysis reports, the reasons for these anomalies  included use of estimated, rather than actual costs, or adjustments from  prior periods due to mistakes or errors with the underlying EVM systems.  For example, the SOFIA project said that many of the negative values in  its EVM reports were due to over-reporting of earlier progress,  mischarges by employees, delayed cost postings, inappropriate use of  charge codes. When there are data anomalies such as those we  identified for SOFIA, the EVM data can become skewed and can distort  true performance.
Variance thresholds try to quantify an acceptable range of deviation; those that do not  exceed a threshold are usually not a concern while those that do are worthy of further  inspection to determine the best course of action to minimize any negative impacts to the  cost and schedule objectives. should be minimized and the reason for each should be fully explained in  the monthly EVM variance analysis reports. To do less limits the  completeness and accuracy of the EVM data and makes the resulting  variance determinations unreliable. While an industry standard for what  constitutes an acceptable volume of anomalies does not exist, EVM  experts in the public and private sectors stated that the occurrence of  EVM data anomalies should be rare.
For four projects that provided subcontractor EVM data, we tried to map  EVM data at the project level to lower level EVM data at the subcontractor  level. However, we were only able to successfully map the data for one of  the projects. This mapping allows project managers to track cost and  schedule by defined deliverables to more precisely identify which  components are causing cost or schedule overruns and to more  effectively mitigate the root cause of the overruns. When the reports do  not allow for traceability, project managers are not able to effectively  measure progress, use the reports to monitor and control costs based on  the original baseline, and/or to track where and why there were  differences. For example, when we attempted to map the EVM data in the  lower level reports for MAVEN’s spacecraft, science operations center,  remote sensing instrument, and Langmuir Probes and Waves instrument  efforts to the overall MAVEN project EVM report, we were not able to see  how the costs tracked from one report to another and therefore could not  reconcile the costs between the reports. Such issues raise the question of  which reports contain the true costs for these efforts. However, MAVEN  officials walked us through the process they use to ensure that lower-level  reports map to the project level reports. Furthermore, MAVEN officials  said they do not mandate that their contractors follow a certain reporting  format, instead any adjustments necessary to ensure that the lower-level  reports map to the project-level reports are made by the project office.  Though the MAVEN project does not prescribe to a standard reporting  format, attempting to manually resolve incompatible pieces of data can  become time-consuming, expensive and can lead to data reliability  issues. We also had similar difficulty mapping various levels of EVM data  for the MMS project. The MMS project was able to demonstrate how the  Southwest Research Institute (SwRI) budget at completion in the lower- level report mapped to the SwRI budget at completion in the MMS project  report, but because of the way the contractor submits their data, project  officials said that the two reports will never match. Although the project  was able to explain how the data tracked, again, attempting to manually  resolve incompatible pieces of data can become time-consuming and can  lead to data reliability issues. Since our review, MMS officials said the  project is working to capture data at lower WBS levels which will allow for  a closer tie between the cost and schedule data.
section_title: Projects Have Not Consistently Applied EVM
paragraphs: All the projects we reviewed were using some EVM to manage their work.  However, the extent to which EVM is implemented across NASA’s in- house projects and their contractors varies by project and center. For  example, 3 of the 10 projects we reviewed did not report project level  EVM data. Implementing EVM at the project level rather than just for the  contract is considered a best practice. In addition, OMB policy requires  the use of an EVM system for both in-house and contractor work and  when there is both government and contractor work, the data from the  two EVM systems must be consolidated at the reporting level for total  program management and visibility. Integrating government and  contractor cost, schedule, and performance status at the project level  should result in better project execution through more effective  management. In addition, some of the in-house projects we analyzed  were only required to meet EVM principles and gather “EVM-like” data.  Further, contractors such as the nonprofit organization managing the  MMS instrument suite, do not report its EVM in the standard contract  performance report format because they are only required to meet the  intent of the standard. Other contractors for the JWST and MAVEN  projects are required to provide EVM reports that show cost and schedule  data by WBS elements.
section_title: Cultural, Technical, and Other Challenges Seen as Impediments to EVM Implementation
paragraphs: NASA EVM focal points, headquarters officials, project representatives,  and program executives cited cultural and technical challenges, as well  as other challenges, as impediments to the effective use of EVM at the  agency. NASA’s culture traditionally has focused on solving science and  engineering challenges and not on monitoring cost and schedule data  such as data produced by an effective EVM system. Technical challenges  were also cited as an impediment to effective EVM use, but opinions  differed within NASA on the extent of their impact. The technical  challenges cited involved difficulty in gathering sufficiently detailed data  for timely inclusion and analysis in an EVM system. In addition, though  NASA has not conducted an EVM skills gap analysis, NASA  representatives said it is a challenge for the agency to implement EVM  effectively due to a lack of sufficient staff with the skills and experience to  analyze EVM data.
subsections: section_title: NASA Culture Seen as Not Valuing EVM
paragraphs: Almost half of the more than 30 NASA representatives we interviewed,  including a large number of those charged with implementing EVM  agencywide—known as EVM focal points—and officials from the Chief  Engineer’s office, cited NASA’s culture as a challenge to the effective use  of EVM at the agency. Specifically, several NASA representatives said  that historically, NASA’s culture has not focused on or valued the kind of  information that EVM can highlight. For example, a NASA EVM analyst  said the culture has been focused on science and engineering and that  accomplishment of that work has been the first priority for managers.  Discussion of the cost of the work has been a secondary concern. A  NASA EVM focal point described overcoming a culture that is based on  applied and basic research rather than discrete tasks with discrete  deliverables, which are among the requirements for effective EVM  implementation. Further, a senior official at NASA headquarters told us  that in project reviews, discussion of cost and schedule information, like  EVM data and analysis, tends to be pushed to the very end of the review  meeting and generally is not discussed in detail. Because EVM data is  not universally valued within NASA, in some cases, the data generated to  satisfy a project’s EVM requirement may be of limited use. For example,  one NASA EVM focal point said some managers are just “checking the  box” with respect to using EVM. The task is performed but the  requirement to collect EVM data was viewed as a nuisance that ultimately  did not provide worthwhile information. Nonetheless, several of those we  interviewed said persistent inquiries about EVM data from senior  management at headquarters, especially over the last couple of years,  are having a positive impact on the culture and forcing projects to pay  more attention to the data.
section_title: Technical and Other Challenges Cited as Making EVM Use More Difficult
paragraphs: The NASA representatives we interviewed also cited technical challenges  as having an impact on the effective use of EVM, although their views  varied on the extent of these challenges. For example, about half of the  focal points we interviewed reported that a challenge to using an EVM  system at NASA was aligning it with the agency’s accounting system, the  SAP Core Financial System. One of the problems cited was that EVM  data collection may require more detailed data than a project has  collected for the agency’s accounting system and this may require the  use of estimated costs instead of actual costs. Estimating these costs can  create additional work for the project, delay the production of EVM data,  and limit the reliability of the EVM data that is produced. Nonetheless, a  NASA manager said, projects could more effectively plan their work to  better accommodate the accounting system. For example, NASA’s  accounting system is set up to measure and report on labor in terms of  full-time equivalents. A project, however, may have set up its earned  value management system with a different measure for labor, such as  productive hours. As a result, the accounting system cannot fill in the  proper numbers for an earned value analysis, potentially causing more  work for the project. Further, the EVM data could be less accurate due to  the use of estimates rather the actual figures. However, if the project had  planned from the outset to have the same measure of labor as the  accounting system, there would not be a problem having this data fit the  EVM system.
An Office of the Chief Financial Officer representative did not believe that  the projects have consistently demonstrated that the accounting system is  a problem, but nonetheless agreed that potential work-arounds and slight  changes to processes are potential solutions for these issues. The Office  of the Chief Financial Officer has started an initiative to address both the  level of detail of the data and improve the monitoring of contractor cost  performance at levels that may be lower than levels at which obligations  are made and costs are reported in the financial system. A report on a  NASA EVM pilot project noted that the greatest impediment for  implementing EVM is cultural resistance, not technical challenges.  Specifically, it noted that “it’s not the EVM Process. It’s not the EVM  Tools. It’s not the SAP Accounting System. It is the NASA culture.”
NASA’s use of contracts with one entity to provide goods or services to  several different NASA projects was also cited as a challenge to use of  EVM. For example, NASA centers may have a contract with one firm to  provide engineering support services. Multiple projects may seek services  using a single task order on this contract. Because of the way the NASA  accounting system is configured, this approach can create artificial  variances when looking at EVM data on a month-to-month basis.  According to a NASA representative, contractual requirements can  correct these issues and allow for a closer accounting of the funds for  EVM purposes and in fact, one NASA center has already instituted such  requirements in a new contract for services. NASA plans to address this  issue as current service contracts expire, but it will take time for the new  data requirements that would provide the desired data to be implemented.
Other challenges cited include the difficulty of gathering sound EVM data  from nonprofit subcontractors, such as universities. One project, for  example, reported that EVM data from nonprofit subcontractors were  immature or non-existent. The nonprofits may be doing a significant  amount of work for a center, but are not equipped to collect earned value  data at the level of detail needed, Office of the Chief Financial Officer and  center representatives reported. EVM focal points said the problem of  collecting EVM data from nonprofits to feed into a larger project-level  EVM system could be mitigated through contract language that clearly  specifies what data are required from contractors. According to these  officials, this kind of language has been included in Jet Propulsion  Laboratory contracts and has been successfully demonstrated as a result.  Furthermore, NASA is concerned that if nonprofits and small businesses  are required to have a fully compliant or certified EVM system, they may  not be able to bid on the work. However, the Federal Acquisition  Regulation is clear that no offeror can be eliminated from consideration  for a contract award because the entity does not have a compliant or  certified EVM system.
section_title: Understanding of EVM Varies Widely Across NASA
paragraphs: NASA representatives we interviewed said there was a need for improved  abilities across the agency to analyze EVM data and implement EVM  systems. Specifically, several focal points said the challenge for NASA is  not as much in obtaining EVM data because most of the information  comes from private contractors responsible for much of NASA’s work, but  in having a staff that can analyze the data and integrate it at the project  level. A senior NASA official also noted that the career civil servants, who  typically are the first level of review for EVM data, do not have  background or training in EVM and cannot conduct a sound analysis of  the data. A project representative echoed this comment and noted a  general awareness of EVM within projects but a shortage of in-depth  knowledge to understand EVM fundamentals and how to interpret the  data it produces. For example, some projects seek to reset EVM  baselines to match funding allocations, which thwarts efforts to examine  cost and schedule trends. One EVM focal point told us it has been difficult  to convince project managers that EVM can predict what will happen in  their projects given the highly technical nature of their work. For example,  a senior manager of a program that experienced significant schedule  delays and cost overruns stated that he is an “EVM skeptic” and that he  does not see EVM data as helpful in helping him track the performance of  a project.
Additionally, the employee skill sets available to analyze and implement  EVM vary widely from center to center, headquarters officials said. In  recent years, NASA has provided EVM training to a large number of  employees; however, the agency has not conducted a skills gaps  analysis, which could help to determine the number and extent of the  staff’s EVM expertise. NASA centers may have staff skill levels reflective  of the level of EVM use at the center. Some centers have many projects  producing EVM data while others may only rarely work on a project that  uses EVM. Without a sufficient number of trained staff to analyze  contractor data and implement in-house EVM efforts, NASA will likely  continue to struggle to effectively use EVM as a valuable project  management tool.
section_title: NASA Policy Is in Line with Best Practices but Implementation Remains the Challenge
paragraphs: NASA has undertaken several initiatives aimed at improving the agency’s  use of EVM. For example, NASA strengthened its spaceflight  management policy to require projects to comply with the 32 ANSI/EIA- 748 guidelines and has developed the processes and tools for projects to  meet this requirement through its new EVM system. While these are  positive steps, the policy continues to lack a requirement for rigorous  oversight or surveillance of how projects are implementing EVM and  NASA does not require projects to use the new EVM system to implement  the EVM requirement of the revised policy. In addition, the issues that  have impeded NASA’s ability to effectively implement EVM, such as its  culture, are longstanding and, as a result, NASA has not had much  success implementing EVM in the past.
The agency’s recent revision of NASA Procedural Requirements  7120.5—the policy that governs NASA’s spaceflight projects and contains  project EVM requirements—strengthened the EVM requirements over  prior versions of the policy. For example, the revised policy requires all  spaceflight projects to demonstrate compliance with each of the 32 EVM  guidelines as set forth in ANSI/EIA-748, whereas the prior policy only  required projects to comply with seven high-level EVM principles. The  new requirements took effect through the release of an interim directive  on September 28, 2011 and have since been made final in NASA’s most  recent update to 7120.5. As a result, projects meeting EVM reporting  thresholds that enter implementation after that date are required to  comply with the new requirement. According to an agency official, the  Office of the Chief Engineer and the mission directorates will determine  which projects that began development under the prior policy must  comply with the new EVM requirements. At major milestones, Office of  the Chief Engineer representatives will review whether the projects have  implemented the 32 EVM guidelines.
However, the new policy still only contains the minimum requirements for  earned value management, such as the thresholds for implementing EVM  and the requirement to comply with ANSI/EIA-748 guidelines. The policy  does not require projects to implement formal independent surveillance of  their EVM systems. Without effective surveillance, agencies cannot  ensure they are meeting the ANSI/EIA-748 guidelines because internal  management systems are not being reviewed to determine if they are  providing reliable cost, schedule and technical performance data. In  addition, effective surveillance helps pinpoint problems, and is useful for  verifying the effectiveness of corrective action plans used to mitigate EVM  system deficiencies. While projects are not required to implement formal  independent surveillance, NASA does plan to conduct periodic  surveillance of project EVM systems. For example, NASA plans to  conduct EVM assessments at two key decision point life cycle reviews  and through the Office of the Chief Engineer compliance surveys.these methods will increase the agency’s surveillance efforts, best  practices call for project level surveillance to be an ongoing, continuous  process conducted by an independent surveillance function.
The policy also does not require projects to use NASA’s new EVM  system, although the system was designed to help projects meet the  ANSI/EIA-748 guidelines. We found that the system meets the intent of  the ANSI/EIA-748 guidelines. Examples of how NASA’s EVM system will  satisfy three key ANSI/EIA-748 guidelines are summarized in table 3  below.
For the projects required to comply with the new policy, use of the  agency-developed EVM system would meet the ANSI/EIA guidelines;  however, some projects will be permitted to continue using their individual  EVM systems as long as the 32 guidelines are met. According to agency  officials, while future revisions to the policy may require use of the  standardized agency-developed EVM system by all projects, at this time,  the agency does not plan to require projects to use the agency-developed  system in order to meet the guidelines. Instead, senior managers will  determine on a case by case basis whether a project will use the  agency’s new EVM system. Currently, only the Space Launch System  and ICESat-2 projects have been selected to implement the new EVM  system. According to an agency senior official, the Agency Program  Management Council approved a phased rollout of the new system  because NASA does not have the resources to implement it agency-wide.  For example, there are not enough NASA subject matter experts to  provide the support needed by the projects when applying the new EVM  system and to build the institutional capability at the centers. Their  approach aims to incrementally build the capacity to do EVM, and seek  increased acceptance of EVM as the requirement for its use is expanded.
subsections: section_title: Strong Leadership Needed to Fully Implement EVM
paragraphs: Over the years, NASA has attempted to address its EVM shortcomings  through a series of policy changes, but these efforts have failed to  adequately address the cultural resistance to implementing EVM  highlighted by many of the NASA officials we interviewed. NASA has  made uneven progress since we reported in 2004 that the agency needed  to improve its use of EVM as a project management tool. Furthermore, a  2008 NASA internal study noted that projects needed to be educated on  the value and approaches for using EVM and to provide support for  setting up EVM within the projects early. Also, an internal agency briefing  on EVM stated that a change management initiative would be necessary  in order to successfully implement EVM at NASA centers. Our work has  also shown that implementing a large-scale initiative, such as EVM,  requires more than just policy changes. To see real change and, in  effect, a cultural shift at NASA, top leadership must provide to employees  a succinct and compelling reason that effective implementation of EVM is  important. Articulating a compelling reason for implementing EVM  enables employees and other stakeholders to understand the expected  outcome of the management initiative and engenders not only their  cooperation, but also their ownership of the outcome, which our work has  shown can take at least 5 to 7 years to fully implement. NASA, by having  a policy that is not comprehensive, allowing projects to opt out of using  the new EVM system, and not committing resources to adequately train  staff, continues to limit progress in the cultural change needed to  implement EVM. Without breaking through the cultural resistance to EVM  and committing to efforts intended to strengthen the use of EVM, NASA is  missing an opportunity to make full use of a key tool that could help it to  manage its projects more effectively.
section_title: Conclusions
paragraphs: Implementing an effective earned value management system and using it  across a large federal agency with well-established processes is without  doubt a challenging task. However, NASA has had uneven progress to  date. NASA acknowledges that EVM can be a valuable tool for monitoring  project development and has initiated an effort to implement an  agencywide system. Currently, only a few of the 10 major spaceflight  projects we reviewed were able to produce reliable EVM data, raising  concern that they cannot produce reliable estimates of cost at completion.  Moreover, until the data are sufficiently reliable, NASA, as well as  external stakeholders, lose valuable insights into project performance that  EVM provides. A sound EVM system is not merely an accounting tool; it  can alert managers to developing problems so that they can be proactive  in reducing the project’s cost and schedule overruns. However, NASA is  not making full use of a key tool that could help it address the cost and  schedule issues that have kept NASA acquisition management on GAO’s  high risk list for more than 20 years.
Though NASA’s recent efforts to improve its EVM capability and  strengthen its policy are steps in the right direction, implementation—fully  integrating EVM into management processes—has been the biggest  challenge and remains so today. NASA faces cultural and technical  challenges that it must overcome to successfully implement an earned  value system and to use this data on a regular basis to inform decision- making. Managing change will be key if NASA’s latest effort to overcome  these challenges and implement an agencywide EVM plan is to succeed.  To accomplish effective earned value management, strong leadership is  required to set an expectation that reliable and credible data are  necessary to manage a successful project. This should be buttressed with  a sound EVM policy and system that are required, and a commitment of  resources to enable staff. Without sustained momentum and commitment,  its current efforts could suffer the same consequence as those in the  past.
section_title: Recommendations for Executive Action
paragraphs: To improve NASA management and oversight of its spaceflight projects,  we recommend that the NASA Administrator direct the appropriate offices  to take the following four actions:
Establish a time frame by which all new spaceflight projects will be  required to implement NASA’s newly developed EVM system, unless  the project is proposing to use a certified system, to ensure that in- house efforts are compliant with ANSI/EIA-748. The time frame  selected should take in to account the need to increase NASA’s  institutional capability for conducting EVM and analyzing and reporting  the data.
Conduct an EVM skills gap analysis to identify areas requiring  augmented capability across the agency. Based on the results of the  assessment, develop a workforce training plan to address any  deficiencies.
Develop an EVM change management plan to assist managers and  employees throughout the agency with accepting and embracing  earned value techniques while reducing the operational impact on the  agency. The plan should include a strategy for having the agency’s  senior leadership communicate their commitment to implementation of  EVM.
To improve the reliability of project EVM data, NASA Procedural  Requirements (NPR) 7120.5 should be modified to require projects to  implement a formal surveillance program that:
Ensures anomalies in contractor-delivered and in-house monthly  earned value management reports are identified and explained,  and report periodically to the center and mission directorate’s  leadership on relevant trends in the number of unexplained  anomalies.
Ensures consistent use of WBSs for both the EVM report and the  schedule.
Ensures that lower level EVM data reconcile to project level EVM  data using the same WBS structure.  Improves underlying schedules so that they are properly  sequenced using predecessor and successor dependencies and  are free of constraints to the extent practicable so that the EVM  baseline is reliable.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to NASA for comment. In its written  comments, reproduced in appendix IV, NASA’s Chief Engineer stated that  the agency concurred with two recommendations and partially concurred  with two other recommendations. In particular, the agency agreed with  our recommendation to perform an EVM skills gap analysis and develop a  workforce training plan to address any deficiencies identified. To that end,  NASA plans to conduct a skills gap assessment and to augment its EVM  training program to address the gaps identified. In addition, the agency  also concurred with our recommendation to develop an EVM change  management plan and is planning to develop a strategy targeted at all  levels of the workforce from project team members to the agency’s  leadership.
The agency partially concurred with our recommendation that NASA  establish a time frame by which all new spaceflight projects will be  required to implement NASA's newly developed EVM system, stating that  they already require projects to perform EVM with an ANSI/EIA 748  compliant system. NASA stated that its phased rollout approach for  implementing the agency’s EVM system is based on available resources,  budgetary constraints, and institutional and project needs.  However,  NASA's approach does not include a timeframe for when projects will be  required to use the new system. We recommended that NASA establish a  timeframe for rolling out the system to all projects because a large  number of projects are not in compliance with NASA’s requirement, and  very few are implementing the new EVM system. Using the newly  developed EVM system could help projects better ensure NASA's  projects are using a system that is compliant with the ANSI/EIA standard.  The agency also noted its disagreement with the notion that all projects,  in particular those that have a skilled EVM workforce and a compliant  system in place, should be forced to use the agency's new system.  Accordingly, we acknowledged in our report that there may be situations  where a project should not be required to use the agency’s EVM system,  such as when a project already uses a certified system or for current,  ongoing projects. Furthermore, we reported that NASA lacks the  appropriate level of surveillance of its projects’ EVM systems to monitor  project adherence to the EVM standard; in addition, the extent to which  EVM has been effectively implemented across NASA's projects varies. If  NASA chooses not to require projects to use its new system it should take  steps to ensure that it monitors their compliance with the standard.  Finally, while we appreciate that NASA must balance its resources with its  needs, the benefits that an effective EVM system can provide, such as  allowing project managers to identify cost growth and take actions to stem  further growth, warrants prioritization of resources to ensure earlier  widespread implementation of NASA's EVM system.
The agency also partially concurred with our recommendation that NPR  7120.5 be modified to require projects to implement a formal EVM  surveillance program. Citing resource constraints, NASA commented that  it does not plan to implement a formal surveillance program, but agreed  that the reliability and utility of the EVM data needed to be improved. As a  result, the agency plans to establish a surveillance process, expand the  workforce’s EVM skills, and provide analytical tools including developing  an EVM System Acceptance and Surveillance Guide. Furthermore, NASA  said that it was not appropriate to incorporate the surveillance  requirement in NPR 7120.5 because of the level of detail associated with  requirements in that policy. The most important part of our  recommendation is that EVM surveillance should be required to ensure  better quality data. We reported that only 4 of the 10 projects we  assessed had a comprehensive EVM surveillance system in place and  the others had limited or no surveillance being performed and only 3 of  the 10 projects had fully reliable data. Without an effective surveillance  program, NASA cannot ensure its projects are meeting the ANSI/EIA-748  standard because internal management systems are not being reviewed  to determine if they are providing reliable cost, schedule and technical  performance data.
In its response, NASA also noted that the project data we used in our  report is over a year old and does not take in to account progress made  over the past year. We disagree and note that in the report we discuss  progress the agency has made over the past year, such as strengthening  the EVM requirements in its policy and developing its new EVM system.  Furthermore, we did not solely rely on project EVM data to develop our  findings. For example, interviews with project officials and additional  documentation they provided further validated our findings. Finally, it is  important to note that NASA Acquisition Management has been on GAO’s  High Risk list for many years due to the agency’s cost and schedule  performance on its major projects. EVM is an important project  management tool that can assist project managers in managing and  assessing performance. Not addressing key issues that impact the  availability of accurate and reliable data could lessen the usefulness of  this key project management tool.
NASA also provided technical comments, which have been addressed in  the report, as appropriate.
We are sending copies of this report to interested congressional  committees, NASA’s Administrator, and other interested parties. In  addition, this report will also be available at no charge on the GAO  website at http://www.gao.gov.
Should you or your staff have any questions concerning this report,  please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine to what extent NASA’s major spaceflight projects are using  earned value management (EVM) to manage acquisitions, we reviewed  all NASA major spaceflight projects with a life cycle cost of over $250  million that were in the implementation phase and thus required to report  EVM. There were 13 projects that met these criteria. Of these, 2 projects  had recently launched and the launch of a 3rd was imminent. These 3  were excluded from our assessment because the work on these projects  was nearly complete. Collectively, the 10 projects we reviewed will cost  over $14 billion to develop.
Our review looked at EVM data for the period of August 2010 to August  2011. While the majority of the 10 projects we reviewed had at least 6  months of EVM data, a few did not because the project had only recently  entered the implementation phase. Additionally, some projects were  undergoing a re-plan and, therefore, were not required to provide EVM  data for certain periods of time. In particular, the Tracking and Data Relay  Satellite Sustainment contract had 5 months of data, the Mars  Atmosphere Volatile Evolution Mission project had only 3 months of data,  and the Orbiting Carbon Observatory 2 Orbital contract had just 4 months  of data. The James Webb Space Telescope contract had EVM data for  the whole period; however, the contractor underwent a major replan in  which all EVM data except for the reporting of actual costs were  suspended from January 2011 to April 2011. Although the Global  Participation Measurement (GPM) project provided EVM reports for the  entire project, we did not conduct an analysis of the project EVM data,  because the performance reports did not contain the detailed data we  needed for our analysis. However, we were able to assess the  performance data for GPM’s Microwave Imager Instrument.
To determine cost and schedule performance for the selected major  projects based on an evaluation of the earned value data, we analyzed  project and contractor data and documentation including contract  performance reports; project work breakdown structures; project  schedules; integrated baseline review briefings; the extent to which  surveillance of the EVM system was occurring; and monthly management  briefings for the 10 major spaceflight projects. Specifically, we compared  project documentation with EVM and scheduling best practices as  identified in GAO’s Cost Estimating and Assessment Guide and Schedule  Assessment Guide. To the extent practicable, we assessed how each of  the 10 projects’ EVM data adhered to 3 of the American National  Standard Institute’s (ANSI) and Electronic Industries Alliance (EIA) 32  guidelines. In addition, we assessed the projects against 3 fundamental  EVM practices that we believe are necessary for maintaining a reliable  EVM system, as identified in our cost guide. We also analyzed the  contract performance reports for each project to determine the level of  data reliability. Specifically, we identified instances of the following: (1)  negative planned value, earned value, or actual cost; (2) planned value  and earned value without actual cost; (3) earned value and actual cost  without planned value; (4) actual cost without planned value or earned  value; (5) earned value without planned value and actual cost; (6)  inconsistencies between the estimated cost at completion and the  planned cost at completion; (7) actual cost exceeding estimated cost at  completion; and (8) planned or earned values exceeding planned cost at  completion.
For the contracts that had more than 6 months of data, we used contract  performance report data in order to generate our estimated overrun or  underrun of the contract cost at completion by using formulas accepted  by the EVM community and printed in the GAO Cost Estimating and  Assessment Guide. To perform this analysis, we examined contractor  performance reports over the period for which we had data to show  trends in cost and schedule performances. We generated multiple  formulas for the projected contract cost at completion that were based on  how much of the contract had been completed up to August 2011 or  earlier for some projects. The ranges in the estimates at completion are  driven by using different efficiency indices based on the project’s past  cost and schedule performance to forecast the cost of the remaining work  and adding that cost to the actual costs to date. The efficiency indices  capture how the project has performed in the past and can be useful in  predicting how it will perform in the future.
We also analyzed monthly project management review briefings to  support our analysis. Finally, we analyzed the earned value data  contained in EVM performance reports obtained from the projects. To  perform this analysis, we compared the cost of work completed with  budgeted costs for scheduled work to show trends in cost and schedule  performances.
To assess the reliability of the cost data, in addition to electronically  testing the data for anomalies, we also reviewed relevant project  documentation and interviewed agency and project officials about the  data. We then followed up on these anomalies with the project offices that  manage each of the spaceflight projects by sharing our preliminary  analysis for each of the 10 projects. When warranted, we updated our  analyses based on the agency’s response and additional documentation  provided to us. The data that we used were sufficiently reliable for how  we portrayed them in our report and we are making recommendations to  the agency to improve NASA’s data reliability based on the findings  discussed in our report. We did not test the adequacy of the agency or  contractor accounting systems.
To support and clarify information in our documentation reviews, we  interviewed agency officials at NASA headquarters and EVM Focal Point  Working Group members—the agency officials that are responsible for  developing an integrated, consistent approach for implementing EVM  throughout NASA, as well as addressing EVM review and surveillance  issues and activities—at each center and the Human Exploration and  Operations and Science mission directorates to discuss their roles as well  as the extent to which EVM data is used to inform decision making. We  interviewed officials at NASA headquarters in Washington, D.C.; and  officials from Ames Research Center in Moffett Field, California; Dryden  Flight Research Center in Edwards, California; Glenn Research Center in  Cleveland, Ohio; Goddard Space Flight Center in Greenbelt, Maryland;  Johnson Space Center in Houston, Texas; the Jet Propulsion Laboratory  in Pasadena, California; Kennedy Space Center in Florida; Langley  Research Center in Hampton, Virginia; Marshall Space Flight Center in  Huntsville, Alabama; and Stennis Space Center in Mississippi.  Additionally, we received responses to questions regarding the EVM data  from each of the 10 projects we selected for review. These questions  addressed how EVM practices are implemented at the project level and  how the project utilizes EVM data.
To determine the challenges that NASA has faced in implementing an  effective EVM system, we interviewed NASA headquarters personnel to  discuss the status and plans for implementing the agency-wide EVM  system. We developed a standard set of questions and interviewed EVM  Focal Point Working Group members at each center and the Human  Exploration and Operations and Science mission directorates to assess  the challenges of implementing EVM at individual centers and across the  agency. We also interviewed a selection of senior officials and program  executives at NASA headquarters that represent projects from each  mission directorate and NASA center included in our engagement to  obtain their perspective on the challenges of implementing and using  EVM on their projects. We also reviewed prior GAO and NASA Inspector  General reports that discuss the agency’s prior efforts to implement EVM.  We examined GAO and NASA Inspector General reports that discuss the  importance of effective organizational change. Additionally, we received  written responses to a standard set of questions regarding the challenges  associated with implementing EVM from each of the 10 projects we  selected for review.
To determine the steps that NASA is taking to improve its use of earned  value management, we examined the results of NASA’s EVM capability  pilot projects and draft policies and guidance and compared these with  best practices in EVM as discussed in GAO’s Cost Estimating and  Assessment Guide, the ANSI/EIA-748 standard, and OMB Circular A-11,  Preparation, Submission, and Execution of the Budget and the Capital  Programming Guide. In addition, we interviewed NASA headquarters  personnel and EVM Focal Point Working Group members at each center  and the Human Exploration and Operations and Science mission  directorates to discuss and obtain information on ongoing initiatives the  agency has undertaken.
We conducted this performance audit from June 2011 to November 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Importance of Earned Value Management
paragraphs: Pulling together essential cost, schedule, and technical information in a  meaningful, coherent fashion is always a challenge for any project.  Without this information, management of the project will be fragmented,  presenting a distorted view of project status. For several decades, the  Department of Defense (DOD) has utilized a tool called earned value  management (EVM) to compare the value of work performed to the  work’s actual cost. Earned value management goes beyond the two- dimensional approach of comparing budgeted costs to actual costs. It  attempts to compare the value of work accomplished during a given  period with the work scheduled for that period. By using the value of  completed work as a basis for estimating the cost and time needed to  complete the project, the earned value concept should alert project and  senior managers to the potential problems early in the project.
In 1996, DOD adopted 32 criteria for evaluating the quality of earned  value management systems. These 32 criteria are organized into 5 basic  categories: organization, planning and budgeting, accounting  considerations, analysis and management reports, and revisions and data  maintenance. In general terms, the criteria require contractors to define the contractual scope of work using a work breakdown measure the progress of work based on objective indicators; structure;  identify organizational responsibility for the work;  integrate internal management subsystems;  schedule and budget authorized work;  collect the cost of labor and materials associated with the work  performed; analyze any variances from planned cost and schedules; forecast costs at contract completion; and  control changes.
The criteria have evolved to become an American National Standards  Institute (ANSI) and Electronic Industries Alliance (EIA) standard for EVM,  which has been adopted by major U.S. government agencies, industry,  and the governments of Canada and Australia. The full application of  EVM system criteria is appropriate for large-cost reimbursable contracts  where the government bears the cost risk. For such contracts, the  management discipline described by the criteria is essential. In addition,  data from an EVM system have been proven to provide objective reports  of contract status, allowing numerous indices and performance measures  to be calculated. These can then be used to develop accurate estimates  of anticipated costs at completion, providing early warning of impending  schedule delays and cost overruns.
The standard format for tracking earned value is through a Contract  Performance Report (CPR). The CPR is a monthly compilation of cost,  schedule and technical data which displays the performance  measurement baseline, any cost and schedule variances from that  baseline, the amount of management reserve used to date, the portion of  the contract that is authorized unpriced work, and the contractor’s latest  revised estimate to complete the project.
As a result, the CPR can be used as an effective management tool  because it provides the project manager with early warning of potential  cost and schedule overruns. Using data from the CPR, a project manager  can assess trends in cost and schedule performance. This information is  useful because trends can be difficult to reverse. Studies have shown that  once projects are 15 percent complete, the performance indicators are  indicative of the final outcome. For example, a CPR showing a negative  trend for schedule status would indicate that the project is behind  schedule. By analyzing the CPR, one could determine the cause of the  schedule problem such as delayed flight tests, changes in requirements,  or test problems because the CPR contains a section that describes the  reasons for the negative status. A negative schedule condition is a cause  for concern, because it can be a predictor of later cost problems since  additional spending is often necessary to resolve problems. For instance,  if a project finishes 6 months later than planned, additional costs will be  expended to cover the salaries of personnel and their overhead beyond  what was originally expected. CPR data provides the basis for  independent assessments of a project’s cost and schedule status and can  be used to project final costs at completion in addition to determining  when a project should be completed.
Examining a project’s management reserve is another way that a project  can use a CPR to determine potential issues early on. Management  reserves, which are funds that may be used as needed, provide flexibility  to cope with problems or unexpected events. EVM experts agree that  transfers of management reserve should be tracked and reported  because they are often problem indicators. An alarming situation arises if  the CPR shows that the management reserve is being used at a faster  pace than the project is progressing toward completion. For example, a  problem would be indicated if a project has used 80 percent of its  management reserve but only completed 40 percent of its work. A  project’s management reserve should contain at least 10 percent of the  cost to complete a project so that funds will always be available to cover  future unexpected problems that are more likely to surface as the project  moves into the testing and evaluation phase.
An Integrated Baseline Review (IBR) is conducted to ensure the reliability  of the EVM data and that the performance measurement baseline  accurately captures all the work to be accomplished. Data from the CPR  can then be used to assess project status—typically, monthly. Cost and  schedule variances are examined and various estimates at completion  are developed and compared to available funding. The results are shared  with management for evaluating contractor performance. Finally, because  EVM requires detailed planning for near-term work, as time progresses,  planning packages are converted into detailed work packages. This cycle  continues until all work has been planned and the project is complete.
An IBR is an evaluation of the performance measurement baseline to  determine whether all project requirements have been addressed, risks  identified, and mitigation plans put in place and all available and planned  resources are sufficient to complete the work. Too often, projects overrun  because estimates fail to account for the full technical definition,  unexpected changes, and risks. Using poor estimates to develop the  performance measurement baseline will result in an unrealistic baseline  for performance measurement.
After the CPRs start being delivered to the government, it is important to  oversee the project by conducting surveillance of the EVM system.  Surveillance is reviewing a contractor’s EVM system as it is applied to  one or more projects. Its purpose is to focus on how well a contractor is  using its EVM system to manage cost, schedule, and technical  performance. For instance, surveillance checks whether the contractor’s  EVM system summarizes timely and reliable cost, schedule, and  technical performance information directly from its internal management  system; complies with the contractor’s implementation of ANSI/EIA-748  guidelines; provides timely indications of actual or potential problems by  performing spot checks, sample data traces, and random interviews;  maintains baseline integrity; gives information that depicts actual  conditions and trends; and provides comprehensive variance analyses at  the appropriate levels, including corrections for cost, schedule, technical,  and other problem areas.
Effective surveillance ensures that the key elements of the EVM process  are maintained over time and on subsequent applications. EVM system  surveillance ensures that the contractor is following its own corporate  processes and procedures and confirms that the contractor’s processes  and procedures continue to satisfy the ANSI guidelines.
The surveillance team designated to perform project reviews should  consist of a few experienced staff who fully understand the contractor’s  EVM system and the processes being reviewed. The surveillance  organization should appoint the team leader and ensure that all  surveillance team members are independent. This means that they  should not be responsible for any part of the projects they assess. Key  activities on the surveillance team’s agenda include reviewing documents,  addressing government project office concerns, and discussing prior  surveillance findings and any open issues. Sufficient time should be  allocated to all these activities to complete them. The documents for  review should give the team an overview of the project’s implementation  of the EVM process.
Successful surveillance is predicated on access to objective information  that verifies that the project team is using EVM effectively to manage the  contract and complies with company EVM procedures. Objective  information includes project documentation created in the normal conduct  of business. Besides collecting documentation, the surveillance team  should interview control account managers and other project staff to see if  they can describe how they comply with EVM policies, procedures, or  processes. During interviews, the surveillance team should ask them to  verify their responses with objective project documentation such as work  authorizations, cost and schedule status data, variance analysis reports,  and back-up data for any estimates at completion.
section_title: Appendix III: Case Studies of Selected Projects’ Implementation of Earned Value Management
paragraphs: We conducted case studies of 10 major spaceflight system acquisition  projects. This appendix provides a brief description of each project,  including an analysis of the project’s earned value data and trends. As  part of our analysis, we assessed the projects’ implementation of three  fundamental earned value management (EVM) practices that we believe  are necessary for maintaining a reliable EVM system—using a certified  American National Standards Institute (ANSI) and Electronic Industries  Alliance (EIA) compliant system, performing surveillance, and conducting  integrated baseline reviews.
We also assessed the projects’ EVM data against three ANSI and EIA  guidelines. These guidelines state that the authorized work elements for  the project should be defined typically using a work breakdown structure  (WBS) that has been tailored to the project and that the WBS is the same  for the cost estimate, schedule, and EVM. The ANSI/EIA guidelines also  state that projects should have a schedule that describes the sequence of  work by listing activities in the order in which they are to be carried out  and identifying significant task interdependencies required to meet project  requirements. Finally, the ANSI/EIA guidelines state that a project should  establish and maintain a time-phased budget baseline to track cost and  schedule variances in an EVM system.
As mentioned above, this appendix includes an analysis of each project’s  earned value trends from August 2010 to August 2011. These data and  trends are often described in terms of cost and schedule variances. Cost  variances compare the earned value of the completed work with the  actual cost of the work performed. Schedule variances are also measured  in dollars, but they compare the earned value of the completed work with  the value of the work that was expected to be completed. Positive  variances are good—they indicate that activities are costing less than  expected or are completed ahead of schedule. Negative variances are  bad—they indicate activities are costing more than expected or are falling  behind schedule. Variances are merely measures that indicate that work  is not being performed according to plan and that it must be assessed  further to understand why. Although our EVM cost projections may show  that a project is experiencing negative cost variances and schedule  slippages, this does not mean that a project has exceeded its agency  baseline commitment and will require additional funds to complete the  project. These estimates use a project’s EVM baseline, which represents  only a portion of the agency baseline commitment for a project. The EVM  baseline contains the cost and schedule contained in a project’s  management agreement minus unallocated future expenses and  schedule margin held by the project and others above the project.
As of August 2011, the budget at completion for the 10 projects was  estimated to be $6.4 billion. To estimate the project variance at  completion, we examined the trends for the earned value data for the  entire project, if data was collected at that level, or elements of the  project. Table 4 provides a summary of the projects’ implementation of each EVM best practice we assessed and projected costs.
In October 2011, the Global Precipitation Measurement mission was approved for a replan.  The contract for the Science Operations Center, Remote sensing Package, Langmuir Probe and  Waves Instrument does not exceed $50 million. Therefore, the supplier is not required to have a  certified system for this contract.  Orbiting Carbon Observatory 2 is in the process of being rebaselined due to a change in the launch  vehicle.
With timely and effective action taken by project and executive  management, it is possible to reverse negative performance trends so  that the projected negative cost variances at completion may be reduced.
To get such results, management needs to obtain reliable EVM data from  EVM systems that adhere to the ANSI/EIA-748 standard for informed  decision making. Until project offices undertake a rigorous validation of  their EVM data, NASA faces an increased risk that managers may not be  receiving the information they need to effectively manage their projects.
The following information describes the key that we used in tables 5  through 14 to convey the results of our assessment of the 10 case study  projects’ implementation of EVM practices.
subsections: section_title: Global Precipitation Measurement
paragraphs: The Global Precipitation Measurement (GPM) mission, a joint NASA and  Japan Aerospace Exploration Agency (JAXA) project, seeks to improve  the scientific understanding of the global water cycle and the accuracy of  precipitation forecasts. The GPM is composed of a core spacecraft  carrying two main instruments: a Dual-frequency Precipitation Radar and  a GPM Microwave Imager (GMI). GPM builds on the work of the Tropical  Rainfall Measuring Mission, and will provide an opportunity to calibrate  measurements of global precipitation when it launches in 2014.
This analysis focuses only on the GMI-1 effort. Ball Aerospace and  Technology Company is the prime contractor for GMI. GMI’s current  contract value is $217 million, which represents approximately 23 percent  of the total GPM project budget of $932.8 million. The GMI instrument  was delivered to Goddard Space Flight Center in February 2012 for  integration into NASA’s upcoming Earth science spacecraft. All remaining  effort for GMI-1 is post delivery support, which is all level of effort. The  GPM project provided EVM reports for the entire project, but we did not  conduct an analysis of the project EVM data because the performance  reports did not contain the detailed data we needed for our analysis.
section_title: GPM Microwave Imager Contractor Uses a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: The GMI-1 contractor met the three fundamental ANSI/EIA-748 practices  necessary for a reliable EVM system. The Defense Contract Management  Agency (DCMA) certified that the GMI-1 contractor’s EVM system  complied with the ANSI/EIA standard in April 2008. Though the contractor  has a certified system, the implementation of that system is questionable  based on our findings below. We assessed GMI-1’s EVM data against  three ANSI/EIA guidelines. These guidelines state that the authorized  work elements for the project should be defined typically using a WBS  that has been tailored to the project and the WBS is the same for the cost  estimate, schedule, and EVM. Our review found that the WBS in the GMI- 1 schedule did not match the WBS used for the EVM data. Project  officials said that the WBS for the project schedule did not match the  WBS used in GMI contractor’s EVM reports because GMI-1 is only one  element of the total project and the project level schedule has a simplified  summary of the GMI-1 schedule that was used for completeness. The  ANSI/EIA guidelines also state that projects should have a schedule that  describes the sequence of work by listing activities in the order in which  they are to be carried out and identifying significant task  interdependencies required to meet project requirements. Our review of  the GMI-1 schedule found some sequencing issues. For example, about  5 percent of the remaining activities were missing predecessor and  successor links, which are necessary for properly sequencing work so  that the schedule will update in response to changes. We also found that  19 percent of the remaining activities had date constraints, which also  hinder the schedule’s ability to respond dynamically to status updates  resulting in an artificial or unrealistic view of the project plan. These  sequencing issues and constraint dates within the schedule affect the  reliability of the overall network and the schedule’s ability to correctly  calculate float values and the critical path. Project officials agreed with our  findings but said that these issues are corrected as they are discovered,  so there is no impact to the project. Finally, though resource loading the  schedule is not required to meet the ANSI/EIA guidelines, it is a best  practice and therefore resources should be accounted for in the schedule  in order to develop this baseline, according to the GAO cost guide. We  found that the GPM schedule was not resource loaded, it is GAO’s  assessment that the project did not show evidence that it had established  and maintained a time-phased budget baseline.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The project conducted an integrated baseline review (IBR) in January  2006. From that review, officials believed the results were less than  satisfactory due to the contractor’s inability to demonstrate the integration  of contract schedule and cost in accordance with their EVM system  description. In particular, 64 areas of concern were identified during the  IBR, including major concerns with data continuity, cost/schedule risk,  and EVM processes. As a result of these issues and changes to the  contract another IBR was performed in January 2011.
section_title: EVM Surveillance Is Being Performed
paragraphs: Joint surveillance reviews of the EVM data are being performed by DCMA  and the contractor. According to DCMA, although they have found some  deficiencies in the contractor’s EVM data, the contractor has responded  with acceptable corrective action plans.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to July  2011. Our review of the GMI-1 EVM data found various data anomalies  that call into question the reliability of the data. For example, we found  negative values for EVM data without any explanation in three monthly  reports. Project officials responded that the negative values all fell within  the contract threshold and therefore the GMI-1 contractor did not need to  provide an explanation in the variance analysis report. In addition, there  were many instances of costs and performances being recorded when no  work had been scheduled. In response to this finding, project officials  explained that this work had been accounted for in previous months,  which explain the missing values. Anomalous EVM data prevents the  project from being able to gain meaningful and proactive insight into  potential cost and schedule performance shortfalls, and take corrective  action to avoid shortfalls in the future. Figure 2 below illustrates that as of  July 2011, the project was reporting a negative cumulative cost variance  of $13 million and a negative cumulative schedule variance of   $3 million.
GPM project officials said the cumulative negative cost and schedule  variances were due to slips caused by suppliers not delivering flight  hardware as planned, which pushed uncompleted work into the future.  The negative variances were also caused by tasks being worked that  were not included in the baseline. Officials also said the project  experienced unfavorable variances in labor costs across all integrated  project teams, which were further affected by the unfavorable 2010 year  end indirect rate adjustment.officials said the contractor increased staffing and added extra shifts,   To address the negative schedule variance,  which increased labor costs thereby increasing the contract value.  Officials noted, however, that the increase in contract value did not  translate into an increase in the baseline, just an increase in the project  funding. Due to both the negative cost and schedule variances, we are  forecasting a negative variance ranging from $14 million to $22 million at  contract completion. According to NASA, the project is not overrunning  its approved baseline commitment, because the EVM baseline does not  include unallocated future expenses held at the project and headquarters  level.
section_title: James Webb Space Telescope
paragraphs: The James Webb Space Telescope (JWST) is a large, infrared-optimized  space telescope that is designed to find the first galaxies that formed in  the early universe. Its focus will include searching for first light, assembly  of galaxies, origins of stars and planetary systems, and origins of the  elements necessary for life. Scheduled to launch in October 2018,  JWST’s instruments will be designed to work primarily in the infrared  range of the electromagnetic spectrum, with some capability in the visible  range. JWST will have a large primary mirror composed of 18 smaller  mirrors, measuring 6.5 meters (21.3 feet) in diameter, and a sunshield  that is the size of a tennis court. A successor to the Hubble Space  Telescope and the Spitzer Space Telescope, JWST will reside in an orbit  about 1 million miles from the Earth.
NASA rebaselined JWST in September 2011 and made changes in the  project’s management in response to cost and schedule performance  issues and the recommendations of the Independent Comprehensive  Review Panel report. As part of the rebaseline, NASA took the lead role  for systems engineering from the prime contractor. The telescope, along  with a segmented primary mirror, will deliver infrared light to the Fine  Guidance Sensor and fine pointing updates to the Observatory and four  scientific instruments including the Near-Infrared Camera (NIRCam), the  Near-Infrared Spectrograph, the Mid-Infrared Instrument and the Fine  Guidance Sensor/Near InfraRed Imager and Slitless Spectrograph. For  work being performed by its international partners, such as the Near- Infrared Spectrograph, EVM data is not collected.
At the time of our review, there was no EVM data for the overall JWST  project or for the work done in-house on the Integrated Science  Instrument Module. According to project officials, the Integrated Science  Instrument Module effort began over a decade ago, and significant parts  of the project, particularly those undertaken in-house at GSFC were not  structured to enable EVM to be implemented easily. However, the JWST  project office is collecting EVM data from Northrop Grumman Aerospace  Systems and obtains copies of the Lockheed Martin Space Systems- Advanced Technology EVM data, which is the University of Arizona’s  prime contractor. Northrop Grumman Aerospace Systems is responsible  for developing and launching the JWST Observatory, which comprises  the spacecraft, sunshield, and the optical telescope element, systems  integration and test observatory verification, observatory commissioning,  and ground and launch support equipment. The University of Arizona is  responsible for developing the Near-Infrared Camera science instrument.
section_title: Observatory Contractor Has a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: The observatory contractor met all three fundamental ANSI/EIA-748  practices necessary for a reliable EVM system. For the observatory  portion of the JWST project, Northrop Grumman Aerospace Systems has  a certified EVM System, which it uses to fulfill the earned value reporting  requirement. Though the contractor has a certified system, the  implementation of that system is questionable based on our findings  below. We assessed the observatory contractor’s EVM data against three  fundamental ANSI/EIA guidelines. These guidelines state that the  authorized work elements for the project should be defined typically using  a WBS that has been tailored to the project and the WBS is the same for  the cost estimate, schedule, and EVM. We found that while consistent  naming conventions existed between the WBS and contract performance  reports, there were discrepancies in the hierarchical structure and  numbering of the WBS elements. For example, the WBS dictionary shows  the Mission Assurance listed as 3.0, while the contract performance  report provided by the observatory contractor has Mission Assurance  listed as 2.0. NASA officials responded that the observatory contractor is  not required to follow the project level WBS hierarchical structure nor the  WBS numbering scheme. They further stated that the project and the  observatory WBS structures are not identical because procurement of the  observatory is only one element of the overall JWST project. NASA did  not provide us with a schedule in a format that would allow us to  determine if the schedule had the proper sequencing in place. As a result,  we were unable to determine if significant task interdependencies such as  predecessor and successor links were in place to ensure that the  schedule would update in response to changes. In addition, because we  could not view the schedule in its native software, we were unable to  determine if the schedule was resource loaded, which is a best practice  for establishing and maintaining a time-phased budget baseline.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: An Integrated Baseline Review was conducted in February 2010;  however, because of the rebaseline in September 2011, according to the  project an additional IBR was held for the observatory in October 2012.
section_title: EVM Surveillance Is Being Performed
paragraphs: Surveillance is being performed by DCMA. In addition, EVM data is  reviewed monthly by multiple individuals on the project office as well as at  higher levels of NASA headquarters and the Goddard Space Flight  Center.
section_title: Data Resulting from the EVM System Was Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to July  2011. Our review of the Observatory EVM data found various data  anomalies that call into question the reliability of the data. For example,  we found actual costs recorded without any work being performed,  inconsistencies between the reported estimate at completion and budget  at completion, large month to month performance swings, and  unexplained variances. NASA officials explained that during this time  period the observatory contractor was engaged in re-planning efforts so  NASA did not want them to expend resources reporting performance  management data to an outdated performance measurement baseline  that did not reflect the new rebaseline assumptions. Further, while  variances analyses were provided in the variance analysis reports for  WBS elements that exceeded contractual thresholds, there was no  explanation for the anomalies we found. A variance analysis report  provides a detailed, narrative report explaining significant cost and  schedule variances and other contract problems and topics. Without this  information, management cannot understand the reasons for the  variances and the contractor’s plan for fixing them. When information is  missing in a variance analysis report, the EVM data will not be meaningful  or useful as a management tool.
As of July 2011, the observatory portion of the JWST Project was 51  percent complete with a positive cumulative cost variance of $2.4 million.  For the same period the project was also experiencing a positive  cumulative schedule variance of $0.9 million as seen in figure 3 below. In  January 2011, the observatory contractor began replanning the remaining  effort in order to meet the October 2018 launch readiness date. As a  result, NASA suspended performance measurement reporting during the  period of January 2011 through April 2011. However, the observatory  contractor was still required to submit contract performance reports  depicting actual cost and estimate at completion data.
Since there were only 3 months of EVM data after the rebaseline, we  were not able to forecast a variance at completion. Since our  assessment, JWST project officials said they have made some significant  improvements in implementation and use of EVM that includes an EVM  approach for the in-house work that will provide EVM metrics to measure  progress. In addition, the officials said the project is also doing  managerial analysis on its contracts and project components and  producing independent estimate of completion each month based on the  EVM data.
section_title: Near-Infrared Camera Contractor Has a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: The Near-Infrared Camera (NIRCam) contractor, Lockheed Martin, met  one of the three fundamental ANSI/EIA-748 practices necessary for a  reliable EVM system. The contractor has a certified EVM System that it is  using to report EVM data for NIRCam. Our review found similar problems  with the WBS hierarchical structure and numbering of the elements. For  example, the WBS dictionary shows Mission Assurance listed as 3.0  while the NIRCam contract performance report shows this effort under  element 5.5. NASA officials explained that the NIRCam contractor is not  required to follow the project level WBS hierarchical structure or  numbering of elements. As stated above, we did not receive a schedule in  its native software, so we were unable to confirm whether the schedule  was sequenced using predecessor and successor links or if it was  resource loaded, necessary for establishing the time-phased budget  baseline.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: As stated above, while the JWST project conducted an IBR in February  2010, the rebaseline necessitated a new IBR, which occurred in March  2012.
section_title: EVM Surveillance Is not Being performed
paragraphs: While formal surveillance is not occurring for the Lockheed Martin EVM  data, monthly reviews of the EVM data are performed by both the project  and program offices and by independent groups. As part of these  reviews, trending metrics are prepared and presented to management as  part of the internal project reviews and monthly status reviews.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 through  July 2011 and found various data anomalies that call into question the  reliability of the data. For example, we found EVM data with negative  values, no work scheduled but work performed, and actual costs being  incurred without any work being performed. NASA officials stated that  variances that exceeded contractual thresholds should be reflected in the  variance analysis reports, however many of these anomalies did not  breach a variance threshold so the report provided no explanation. In  addition, NASA officials explained that many of these anomalies occurred  during the project replan, which was formally approved in September  2011, when the project was rebaselined. As a result of the replan, NASA  suspended EVM data reporting, which resulted in many of the anomalies  we found.
As of July 2011, the NIRCam portion of the JWST project was 98 percent  complete. In July 2011, there was a negative cumulative cost variance of  $33 million and a negative cumulative schedule variance of $4.4 million  as seen in figure 4 below. The reasons for the downward trend reflected  in the graph below were not explained because variance analysis  reporting was suspended during the replan period.
Based on the downward trend, we are forecasting a negative variance at  completion ranging anywhere from $34 million to $48 million. This  analysis is based on information prior to the project’s 2011 replan. The  variances projected do not take in to account the establishment of a new  EVM baseline as a result of the replan.
section_title: Landsat Data Continuity Mission
paragraphs: The Landsat Data Continuity Mission (LDCM) is a joint mission between  NASA and the United States Geological Survey (USGS) that seeks to  extend the ability to detect and quantitatively characterize changes on the  global land surface at a scale where natural and man-made causes of  change can be detected and differentiated. It is the successor mission to  Landsat 7. The Landsat data series, begun in 1972, has provided the  longest continuous record of changes in the Earth’s surface as seen from  space.
Landsat data is a resource for people who work in agriculture, geology,  forestry, regional planning, education, mapping, and global change  research. The LDCM provides remotely sensed, highly calibrated,  moderate resolution, multispectral imagery affording systematic global  coverage of the Earth’s land surfaces on a seasonal basis and makes the  data readily available for large-scale and long-term Earth System Science  and Land use/land cover change research and management. The project  plans to launch early in February 2013.
LDCM consists of an Operational Land Imager (OLI) and a Thermal  Infrared Sensor (TIRS) science instrument, a spacecraft, and a mission  operations element. LDCM does not collect EVM data at the project level.  The decision not to perform EVM at the project level was reviewed  extensively prior to proceeding into the design and development phase,  according to project officials. Also, there is no EVM data for the  spacecraft effort because this work is being done under a firm fixed price  contract and NASA regulations do not require EVM for firm fixed price  contracts. The TIRS instrument, built in-house at NASA’s Goddard Space  Flight Center, was added late in the formulation phase with an aggressive  delivery schedule and delivered in February 2012. The Ground System is  being built and delivered by the USGS. The developmental part of the OLI  contract with Ball Aerospace and Technology Corporation was completed  with delivery of the instrument in early October 2011. Following on-orbit  checkout, the contract will transfer to USGS for management. The OLI  instrument is the only part of the project that performed EVM.
section_title: Operational Land Imager Contractor Has a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: The OLI contractor met two of the three fundamental ANSI/EIA-748  practices necessary for a reliable EVM system. In 2007, the Defense  Contract Management Agency (DCMA) certified Ball Aerospace and  Technology Corporation’s EVM system. Though the contractor has a  certified system, the implementation of that system is questionable based  on our findings below. We assessed the contractor EVM data against  three ANSI/EIA guidelines. These guidelines state that the authorized  work elements for the project should be defined typically using a WBS  that has been tailored to the project and the WBS is the same for the cost  estimate, schedule, and EVM. Our review found that the WBS in the OLI  schedule was consistent with the WBS used for the EVM data. The  ANSI/EIA guidelines also state that projects should have a schedule that  describes the sequence of work by listing activities in the order in which  they are to be carried out and identifying significant task  interdependencies required to meet project requirements. Our analysis  found 11 percent of the remaining activities were missing dependencies,  13 percent had lags, and 24 percent had constraints, among other things.  When schedules are not sequenced properly, float values and the  calculated critical path will not be valid. Project officials said the schedule  sequencing is driven by external forces such as facilities availability,  spacecraft and ground system interfaces, DCMA inspections, and so  forth. The effects of these forces on schedule sequencing and critical path  are reviewed extensively and the validity of the critical path is not typically  an issue for project management. Finally, the ANSI/EIA guidelines state  that a project should establish and maintain a time-phased budget  baseline to track cost and schedule variances in an EVM system. Though  resource loading the schedule is not required to meet the ANSI/EIA  guideline, it is a best practice and therefore resources should be  accounted for in the schedule in order to develop this baseline, according  to the GAO cost guide. Our analysis found that the OLI schedule was  resource loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The project conducted an IBR in 2007. The IBR identified 80 areas of  concern and as a result the LDCM Project did not accept the  Performance Measurement Baseline at the IBR. However since 2007,  Ball Aerospace and Technology Corporation has addressed the areas of  concern.
section_title: EVM Surveillance Is Being performed
paragraphs: Joint surveillance reviews are being conducted on Ball Aerospace and  Technology Corporation’s EVM system by DCMA and the Defense  Contract Audit Agency.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to August  2011. While the EVM data reflected several data anomalies, Ball  Aerospace provided detailed explanations for each of them. For example,  negative values were attributed to year end rate savings, labor  corrections, or material transfers that were greater than the current month  actual costs which resulted in a negative number.
The EVM data assessed below reflects all work associated with the  LDCM OLI instrument. Figure 5 below illustrates that as of August 2011,  the project was a reporting a negative cumulative cost variance of  approximately $46 million and a negative cumulative schedule variance of  $1.3 million.
The negative cumulative cost and schedule variances were due to  various technical challenges experienced during instrument development,  including detector fabrication issues, coatings issues that necessitated  the build of a second calibration subassembly, and instrument integration  onto the baseplate taking two weeks longer than expected to complete  fabrication. Due to the negative variances, we are forecasting a negative  variance at completion ranging from $49 million to $52 million. According  to a project official, the project is not overrunning its agency baseline  commitment, because the EVM baseline does not include unallocated  future expenses held by the project or NASA headquarters. The  developmental part of the OLI instrument contract with Ball Aerospace  and Technology Corporation was completed with delivery of the  instrument in early October 2011.
section_title: Lunar Atmosphere and Dust Environment Explorer
paragraphs: The Lunar Atmosphere and Dust Environment Explorer (LADEE) will  determine the global density, composition, and time variability of the lunar  atmosphere. LADEE’s measurements will determine the size, charge, and  spatial distribution of electrostatically transported dust grains. Additionally,  it will carry an optical laser communications demonstrator that will test  high-bandwidth communication from lunar orbit. Finally, it will broaden  the scientific understanding of other planetary bodies regarding  exospheres or very thin atmospheres, like the moon.
section_title: Project Office Does not Have a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: LADEE met only one of three key fundamental ANSI/EIA-748 practices  for reliable EVM system. NASA’s EVM guidance says that projects must  start reporting EVM data once the project enters the project  implementation phase, if the project’s life-cycle cost is at or greater than  $20 million. While LADEE may not have a contract that exceeds $20  million, the overall LADEE project cost is about $262.9 million.  Nevertheless, project officials said LADEE is responsible only for  gathering “EVM-like” data at the project level. The “EVM-like” data is  collected using the “EVM Lite” process, which attempts to meet the  ANSI/EIA-748 standard where applicable. Project officials said that when  LADEE was initiated in February 2008, NPR 7120.5D was still in effect  which required application of EVM principles. This is, in effect, “EVM-Lite”  or “EVM-Like.” Officials further stated that prior to August 2011, the  LADEE project evaluated candidate EVM techniques using sample data  from January to March 2011. Based on that evaluation period, LADEE  decided to use the “EVM Lite” technique to collect the necessary data to  manage the project. From April to June 2011, additional evaluations of  this technique continued. Therefore, standard reporting of the LADEE  “EVM-like” project level data did not begin until August 1, 2011. We  reviewed all available EVM data from September 2010 to June 2011,  which was before the standard EVM reporting period began.
Though LADEE does not have a certified system, we assessed how well  LADEE project was meeting three ANSI/EIA guidelines. These guidelines  state that the authorized work elements for the project should be defined  typically using a WBS tailored to the project and the WBS should be the  same for the schedule, cost estimate, and EVM. Our analysis found that  the WBS in the LADEE schedule matched the WBS used for EVM data.  The ANSI/EIA guidelines also states that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifies significant task  interdependencies required to meet project requirements. Our review of  the LADEE schedule found 3 percent of the remaining activities were  missing predecessor or successor links, which cause the schedule to not  properly update in response to changes. We also found that about 6  percent of the remaining activities had date constraints and/or lags, which  also hinder the schedule from responding dynamically to changes and  can portray an artificial or unrealistic view of the project plan. While these  issues may be relatively small, any missing dependencies, constraints,  and lags may disrupt the reliability of the overall network. Finally, the  ANSI/EIA guidelines state that a project should establish and maintain a  time-phased budget baseline to track cost and schedule variances in an  EVM system. Though resource loading the schedule is not required to  meet the ANSI guidelines, it is a best practice and therefore resources  should be accounted for in the schedule in order to develop this baseline,  according to the GAO cost guide. We found that the LADEE schedule  was not resource-loaded, therefore, it is GAO’s assessment that the  project did not show evidence that it had established and maintained a  time-phased budget baseline.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: LADEE conducted an IBR in December 2010. From that review, a list of  concerns about the EVM data and schedule were identified. Specifically,  the project was not using a consistent approach to collect EVM data,  calling into question the credibility of the data. In addition, it was unclear if  an objective assessment of cost and schedule performance could be  made using data generated by the “EVM Lite” approach. Since that  review, officials said all issues and actions have been addressed and  were formally closed by the IBR panel in July 2011.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While formal surveillance is not occurring, EVM data assurance reviews  are being performed during monthly management reviews with the Lunar  Quest Program office.
section_title: Data Resulting From the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from September 2010 to June  2011. Our review of LADEE’s EVM data found various data anomalies  that call into question the reliability of the data. For example, from  October 2010 to December 2010 there were some instances of negative  values in the EVM reports that were unexplained. Since a variance  analysis report provides a detailed narrative explaining significant cost  and schedule variances, when this information is missing, management  cannot understand the reasons for variances and the plan for fixing them.  Also, the EVM data provided by the project office was not presented in a  standard EVM format. This could be attributed to the fact that LADEE is  required to provide only “EVM-like” data.
Figure 6 below illustrates that as of June 2011, the LADEE project was  reporting a positive cumulative cost variance of $3 million and a negative  cumulative schedule variance of $10 million.
LADEE project officials provided no information regarding positive cost  and negative schedule variance drivers. As such, we have no insight into  what could be causing deviations from the plan. Based on the positive  cost variance and negative schedule variance thus far, we are forecasting  a negative variance at completion ranging from $0.1 million to $13 million  dollars. According to NASA, the project is not overrunning its commitment  because the EVM baseline does not include unallocated future expenses  held at the project and NASA headquarters level.
section_title: Magnetospheric Multiscale
paragraphs: The Magnetospheric Multiscale (MMS) is made up of four identically  instrumented spacecraft. The mission will use the Earth’s magnetosphere  as a laboratory to study the microphysics of magnetic reconnection,  energetic particle acceleration, and turbulence. Magnetic reconnection is  the primary process by which energy is transferred from solar wind to  Earth’s magnetosphere and is the physical process determining the size  of a space weather storm. The four spacecraft will fly in a pyramid  formation, adjustable over a range of 10 to 400 kilometers. The data from  MMS will be used as a basis for predictive models of space weather in  support of exploration.
The MMS spacecraft is being designed, developed, and tested in-house  at Goddard Spaceflight Center (GSFC) while instrument development  activities are under contract with Southwest Research Institute (SwRI).  The Mission Operations Center and the Flight Dynamics Operations Area  will be developed and operated at GSFC. The Science Operations Center  for the instruments will be developed and operated at the Laboratory for  Atmospheric and Space Physics at the University of Colorado and is  under contract to SwRI. The MMS project office is collecting EVM data  both at the project level as well as from SwRI, which is responsible for the  entire instrument suite. Therefore, the SwRI Instrument Suite effort is a  subset of the overall MMS project level EVM report.
section_title: Project Office Does Not Have a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: At the project level, MMS met one of three fundamental ANSI/EIA-748  practices for a reliable EVM system. MMS does not have a certified EVM  system that complies with the ANSI/EIA-748 standard. NASA project  officials said in-house projects are required only to be ANSI/EIA  compliant, and are not required to have a certified system. Although the  MMS project does not have a certified system, we assessed how well the  MMS project was meeting three ANSI/EIA guidelines. These guidelines  state that the authorized work elements for the project should be defined  typically using a WBS that has been tailored to the project and the WBS  is the same for the cost estimate, schedule, and EVM. Our review found  that the WBS in the MMS schedule did not match the WBS used for the  EVM data, which is not in line with best practices. According to project  officials, the MMS project was started before the requirements for earned  value management were developed. As a result, the schedule and WBS  were created without significant consideration of a one-to-one correlation  between the two. A project official stated that MMS has retrofitted its EVM  system to provide as close a correlation as possible without having to  rebuild the WBS.
The ANSI/EIA guidelines also state that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifying significant task  interdependencies required to meet project requirements. Our review of  the MMS schedule found some sequencing issues within the schedule.  For example, 9 percent of predecessor and successor tasks were not  linked to one another, which is necessary for properly sequencing work  so that the schedule will update properly once changes are made. This  number accounts for removing all external tasks and level of effort (LOE)  type activities. We also found that 21 percent of the remaining activities  were constrained. This number also does not include LOE type activities.  In fact, the majority of these constraints were hard constraints. Hard  constraints can sometimes be impossible to meet, given the network  characteristics, and can thereby result in schedules that are logically  impossible to carry out. The presence of constraints also impacts the  schedule’s ability to respond dynamically to changes and may portray an  unrealistic view of the project plan. As a result, these sequencing issues  and date constraints within the schedule affect the reliability of the overall  network and the schedule’s ability to correctly calculate float values and  the critical path. Finally, the ANSI/EIA guidelines state that a project  should establish and maintain a time-phased budget baseline to track  cost and schedule variances in an EVM system. Though resource loading  the schedule is not required to meet the ANSI/EIA guideline, it is a best  practice and therefore resources should be accounted for in the schedule  in order to develop this baseline, according to the GAO cost guide. We  found that that the MMS schedule was resource loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The MMS project office conducted an IBR in June 2010. This resulted in  positive comments, general observations, and constructive  recommendations by the review team.
section_title: EVM Surveillance Is Not being Performed
paragraphs: While formal surveillance is not occurring at the project level, EVM data  assurance reviews are being performed by the Explorers and  Heliophysics Program Office, GSFC Flight Projects Directorate  managers, GSFC Chief Financial Officer’s Office, Standing Review  Board, and NASA headquarters.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to July  2011. Our review of the MMS project level EVM data found some minor  data reliability issues. The 12 months of project level EVM data reflect all  of the performed in-house effort at GSFC for the project and the  instrument suite contractor’s summary level effort. Also, the data provided  was not reported in the standard contract performance report format.  Project officials said a defined contract performance report was not  dictated to the MMS project, but that all reporting has the same  information as a “standard” contract performance report even though the  formatting may be different. However, beginning in October 2011, MMS  began reporting with the standard format 1 contract performance report,  which provides cost and schedule data for each element in the project’s  product-oriented WBS.
We tried to map the EVM data in the lower level report for the instrument  suite to the MMS project level report and we were not able to see how the  costs tracked from one report to another. For example, the project level  EVM data showed that the instrument suite contractor’s July 2011 budget  at completion was $296 million, whereas the lower level EVM data in the  instrument suite contractor’s report showed the budget at completion to  be $217 million. The MMS project was able to demonstrate how the SwRI  budget at completion in the lower-level report mapped to the SwRI budget  at completion in the MMS project report. However, officials said because  of the way the contractor submits their data the two reports will never  match. Though we acknowledge that the project was able to explain how  the data tracked, attempting to manually resolve incompatible pieces of  data can become time-consuming, expensive and can lead to data  reliability issues. Project officials said that MMS is working to capture data  at lower WBS levels, which will allow for a closer tie between the cost and  schedule data. In addition, MMS said that in addition to receiving the  SwRI EVM reports, the project now internally calculates earned value  metrics on the contractor provided instrument suite EVM data, which  gives MMS completely internally derived earned value performance  reports, based on the project team’s assessment without bias from  contractor data.
Figure 7 below illustrates that as of July 2011, the project was reporting a  negative cumulative cost variance of $18 million and a negative  cumulative schedule variance of $25 million.
In October 2010, the project was in the midst of a replan and not all data  was available at the time of report submission to generate detailed  variance explanations. The replan was conducted so that the earned  value baseline was the same as the cost plan required by the agency for  monthly plan versus actual reporting. The goal of the replan was to  prevent the project from having to report variances against two different  plans. Since the replan, however, the project has experienced a  downward trend in both cost and schedule performance. As a result, we  are forecasting a negative variance at completion ranging from $47  million to $80 million dollars. NASA stated that the project is not  overrunning its approved baseline commitment, because the EVM  baseline does not include unallocated future expenses held at the project  and headquarters level.
section_title: Instrument Suite Contractor Does not Have a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: SwRI, the contractor responsible for the entire instrument suite for MMS,  met one of the three fundamental ANSI/EIA-748 practices for a reliable  EVM system. SwRI does not have a certified EVM system that complies  with the ANSI/EIA-748 standard. According to a project official, the SwRI  contract does not require SwRI to have a certified system but only to be  compliant with the ANSI/EIA-748. NASA convened an independent team  to review the contractor’s readiness for EVM system certification and  concluded that while the contractor has qualified people to support  implementation of EVM, a single point of failure exists without a  documented process. Not documenting the process is a problem because  if the people who know the process leave, new staff will not know what to  do. In addition, the team found that even though the right software tools  are in place to support EVM, more integration is needed to reduce  manual inputs. Finally, the team reported that compliance with the  ANSI/EIA-748 standard would not be achievable without management  support and resources. Despite these findings, the project office believed  that contractor’s EVM data is useful in examining trends and overall  performance of the instrument suite effort.
Though SwRI does not have a certified system, we assessed how well  the contractor was meeting three ANSI/EIA guidelines. These guidelines  state that the authorized work elements for the project should be defined  typically using a WBS that has been tailored to the project and the WBS  is the same for the cost estimate, schedule, and EVM. Our review found  that the WBS used in the schedule was not consistent with the WBS used  for the EVM data. The ANSI/EIA guidelines also state that projects should  have a schedule that describes the sequence of work by listing activities  in the order in which they are to be carried out and identifying significant  task interdependencies required to meet project requirements. Our review  of the schedule found some sequencing issues. For example, the  Primavera schedule provided showed 31 percent of the remaining  activities were missing dependencies and 36 percent were constrained.  Officials said majority of the constrained tasks were due to external  dependencies. Due to the major sequencing issues in the MMS SwRI  schedule, we question the reliability of the overall network and the  schedule’s ability to correctly calculate float values and the critical path.  MMS project officials said they believed many of the constrained activities  are not valid because they reside in another schedule. Also, some  constraints, in the Harness area for example, if removed, have no effect  on the overall schedule. In addition, officials also said some of the  sequencing issues may be caused by the manual integration because  since some instrument provider schedules are in Microsoft Project and  others are in Primavera, and therefore it is not possible to ensure all tasks  have been linked properly. However, the MMS project scheduler tests the  schedule for missing dependencies, logic errors, and reasonable  durations and the results are shared with the project office and the  contractor so that appropriate action can be taken. Lastly, officials said  several of the activities identified in our analysis are not really schedule  items but level of effort type activities. When we removed the 15 LOE  type activities from the missing dependencies count, the schedule still  showed 28 percent of the remaining activities were missing  dependencies. When we removed the 14 LOE type activities and the 4  Harness activities from the constraint count, the schedule still showed 33  percent of remaining activities were constrained. Because the schedule is  the foundation for the EVM baseline, we question the reliability of the  Instrument Suite EVM data. Finally, the ANSI/EIA guidelines state that a  project should establish and maintain a time-phased budget baseline to  track cost and schedule variances in an EVM system. Though resource  loading the schedule is not required to meet the ANSI/EIA guidelines, it is  a best practice and therefore resources should be accounted for in the  schedule in order to develop this baseline, according to the GAO cost  guide. We found that the MMS schedule was resource-loaded.
subsections: section_title: Instrument Suite Contractor Conducted an Integrated Baseline Review
paragraphs: The MMS project conducted an IBR of the MMS instrument suite effort in  January 2010 and all action items have been closed out.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While the instrument suite contractor does not have a formal surveillance  program, the MMS project office has an EVM analyst, schedule team,  resources team, instrument management team, and project management  team who all review the instrument suite EVM data on a monthly basis. In  addition, the Solar Terrestrial Probes project office, Science Mission  Directorate management, and The Aerospace Corporation review the  instrument suite EVM data monthly.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to August  2011. Our review of the instrument suite EVM data found various data  anomalies that call into question the reliability of the data. For example,  there were negative numbers reported for EVM data in four of the months  that we reviewed. For some of the negative numbers, there was no  explanation for the cause. For others, the negative values were due to  correcting several months of translation errors brought on by a known  issue with importing data from the schedule into the EVM system  software. There were also errors such as inflated EVM data that once  corrected, resulted in negative values. Figure 8 below illustrates that as  August 2011, the project was reporting a negative cumulative cost  variance of   $4 million and negative cumulative schedule variance of $6 million.
The cost variance in August 2011 dramatically improved from the  downward trend during the previous months due to the project applying  almost $13 million from its management reserve to the instrument suite  contract. However, due to the negative cumulative cost and schedule  variances, we are forecasting a negative variance at completion ranging  from $10 million to $24 million.
section_title: Mars Atmosphere and Volatile Environment
paragraphs: The Mars Atmosphere and Volatile EvolutioN (MAVEN) mission is part of  NASA’s Mars Scout program, a robotic orbiter mission that will provide a  comprehensive picture of the Mars upper atmosphere, ionosphere, solar  energetic drivers, and atmospheric losses. Set to launch in 2013, MAVEN  will deliver comprehensive answers to long-standing questions regarding  the loss of Mars’ atmosphere, climate history, liquid water, and  habitability. MAVEN will provide the first direct measurements ever taken  to address key scientific questions about Mars’ evolution.
Lockheed Martin is building the MAVEN spacecraft and will carry out  mission operations for MAVEN. NASA’s Jet Propulsion Laboratory will  navigate the spacecraft. The Laboratory for Atmospheric and Space  Physics (LASP) at the University of Colorado at Boulder will coordinate  the science team and science operations and lead the education and  public outreach activities. NASA’s Goddard Spaceflight Center will  provide management and technical oversight for the mission and will also  provide mission systems engineering, mission design, and safety and  mission assurance.
The MAVEN project office is using EVM at the project level as well as  collecting EVM data from Lockheed Martin, the spacecraft contractor, and  LASP, which is responsible for the Science Operations Center, Remote  Sensing, and Langmuir Probe and Waves instrument efforts. Both the  Lockheed Martin and LASP contracts are subsets of the overall MAVEN  Project EVM report.
section_title: Project Does Not Have a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: MAVEN fully met one of the three key practices for implementing EVM at  the project level. Specifically, the project did not have a certified EVM  system and is not required to have a certified system. Nevertheless, we  assessed how well the MAVEN Project was meeting three ANSI/EIA  guidelines. As part of our analysis, we assessed MAVEN’s EVM data  against three ANSI/EIA guidelines. These guidelines state that the  authorized work elements for the project should be defined typically using  a WBS that has been tailored to the project and that the WBS should be  the same for the cost estimate, schedule, and EVM. We found that the  project’s WBS was consistent between the schedule and EVM data.
The ANSI/EIA guidelines also state that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifying significant task  interdependencies required to meet project requirements. Our review  found some sequencing issues in the schedule. For example, 5 percent of  the activities were missing dependencies, 6 percent had open ended logic  links, and 20 percent had constraints, among other things. When  schedules are not sequenced properly, float values and the calculated  critical path will not be valid. In addition, the project conducted an  integrated baseline review in July 2011. The MAVEN project office  provided a May 2012 Schedule Health Check Report that showed the  number of missing dependencies and constraints had been reduced  considerably. Though we cannot validate the improvement in the  schedule without performing our own assessment, we believe that  MAVEN is working towards producing a more reliable schedule. This  review found that the project schedule and technical design were in good  shape, but noted concerns that more resources were needed to  implement and maintain EVM, there were cost and schedule integration  issues that caused the budgets for some work packages to not be in sync  with the schedule, and reliable critical path analysis was at risk because  of missing schedule links and constraints. Finally, a project should  establish and maintain a time-phased budget baseline at the control  account level, against which performance can be measured. Though  resource loading the schedule is not required to meet the ANSI/EIA  guidelines, it is a best practice and therefore resources should be  accounted for in the schedule in order to develop this baseline, according  to the GAO cost guide. We found that the schedule was resource-loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: Project conducted an IBR in July 2011 and all nine areas of concern were  addressed and closed.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While formal surveillance is not occurring at the project level, EVM data  assurance reviews are being performed by the Mars Program Office  representatives, MAVEN standing Review Board representatives, and  Aerospace Corporation representatives at both the project level and for  LASP efforts.
section_title: Data Resulting from the EVM System Is Reliable
paragraphs: We reviewed contract performance reports from June 2011 to August  2011. EVM data prior to spring 2011 was not available because MAVEN  had not been confirmed into the implementation phase. Our review of the  MAVEN project EVM data found that there was a mistake causing the  costs to be overstated by twice their actual amount. Other than this  mistake, which affected most of the data in the report, no other errors  were found. The project EVM data reflects all work on this project  including effort related to the project office, two in-house instruments, and  the Space Sciences Laboratory, as well as major efforts from Lockheed  Martin and LASP. Consolidated reporting of all components at a summary  level began with the May 2011 data, with the first full summary report  delivered on July 15, 2011. Therefore, we had only 3 months of data to  review. Figure 9 below illustrates that as of August 2011, the project was  reporting a positive cumulative cost variance of $5 million while also  experiencing a negative cumulative schedule variance of $5 million.
The positive cumulative cost variance was due to a decrease in labor  charges, delayed material costs, a reduction in re-work, and the  leveraging of common engineering products from other projects. The  negative schedule variance was being driven by the Neutral Gas and Ion  Mass Spectrometer instrument, which experienced technical issues such  as vendor machines not being manufactured to specifications. Because  we had only 3 months of data, we did not forecast an estimate at  completion.
In addition, we tried to map the EVM data in the lower level reports for  spacecraft, Science Operations Center, Remote Sensing, and Langmuir  Probe and Waves efforts to the overall MAVEN project EVM report and in  some cases we were not able to see how the costs tracked from one  report to another. For example, while we could easily trace the costs for  the Science Operations Center effort from the lower level EVM report to  the overall MAVEN project report, we could not clearly map the costs for  the spacecraft, Remote Sensing or Langmuir Probe and Waves efforts. In  particular, for Remote Sensing and Langmuir Probe and Waves efforts,  the lower level EVM report cost elements did not have their costs  burdened at the WBS level, which could account for some of the  differences between the lower level report costs and the overall MAVEN  project costs for those elements. MAVEN project officials walked us  through their process of how they ensure lower-level reports map to the  project level reports. In addition, MAVEN project provided supporting  documentation that validated this assertion. Though MAVEN project  officials helped explain the mapping, officials said they do not mandate  that their contractors follow a certain reporting format, instead any  adjustments necessary to ensure that the lower-level reports map to the  project-level reports are made manually by the project office. Though  MAVEN project does not prescribe to a standard reporting format,  attempting to manually resolve incompatible pieces of data can become  time-consuming, expensive and can lead to data reliability issues.  Although the agency provided explanations for the mapping issues, the  ability to reconcile the costs between the reports should be evident,  without additional explanations.
section_title: Lockheed Martin Has a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: Lockheed Martin, the spacecraft contractor, met the three fundamental  ANSI/EIA practices necessary for a reliable EVM system. In August 2008,  the Defense Contract Management Agency (DCMA) certified that  Lockheed Martin’s EVM system is compliant with the ANSI/EIA standard.  However, the implementation of that EVM system is questionable based  on our findings. We assessed the contractor’s data against the three  ANSI/EIA guidelines. These guidelines state that the authorized work  elements for the project should be defined typically using a WBS that has  been tailored to the project and the WBS is the same for the cost  estimate, schedule, and EVM. We found that the project’s WBS in the  schedule was consistent with the WBS used in the EVM data. However,  we found issues with the schedule. The ANSI/EIA guidelines also state  that projects should have a schedule that describes the sequence of work  by listing activities in the order in which they are to be carried out and  identifying significant task interdependencies required to meet project  requirements. Our review found some sequencing issues in the schedule.  For example, 2 percent of the activities remaining were missing  predecessor and successor links, which are necessary for properly  sequencing work so that the schedule will update in response to changes.  We also found that 37 percent of the remaining activities had constraints,  which also hinder the schedule’s ability to respond dynamically to status  updates resulting in an artificial or unrealistic view of the project plan.  These issues with the schedule affected the reliability of the overall  network and the schedule’s ability to correctly calculate float values and  the critical path. Project officials further explained that some of the “start  no earlier than” constraints were due to resource availability and “finish no  later than” constraints were used intentionally to plan task activities to  occur as late as possible. Finally, the ANSI/EIA guidelines state that a  project should establish and maintain a time-phased budget baseline to  track cost and schedule variances in an EVM system. Though resource  loading the schedule is not required to meet the ANSI/EIA guidelines, it is  a best practice and therefore resources should be accounted for in the  schedule in order to develop this baseline, according to the GAO cost  guide. We found that the contractor schedule was resource loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The project conducted the spacecraft’s IBR in April 2011. During the  review, several areas of concern regarding the schedule were identified.  The project office said that during the integrated baseline review, the  review team identified many of the same observations with the schedule  as our findings. As a result, the project office directed the contractor to  eliminate the constraints, lags, and missing logic links in their integrated  master schedule. Since the November 2011 schedule submittal, the  contractor has decreased the number of sequencing issues in the  schedule, according to project officials. The project office also said that  the contractor continues to conduct schedule health checks to uncover  any sequencing issues and provide the project office with schedule  variance reports and critical path analysis, which are discussed during  monthly management meetings.
section_title: EVM Surveillance Is Being Performed
paragraphs: Joint surveillance reviews of the EVM data are being performed by  DCMA, MAVEN project officials, and the contractor.
section_title: Data Resulting from the EVM System Is Reliable
paragraphs: We reviewed contract management reports from January 2011 to August  2011. Our review of the EVM data found no major issues with data  reliability. However, from January to May 2011, there were no variance  analysis reports produced to explain significant cost and schedule  variances and other contract problems and topics because they did not  meet reporting thresholds. Without this information, however,  management cannot understand the reasons for variances and the  contractor’s plan for fixing them.
Figure 10 below illustrates that as of August 2011, the project was  reporting a positive cumulative cost variance of $6 million and a negative  cumulative schedule variance of $3 million.
One reason for the positive cost variance was due to the ability to  leverage a lower subcontractor rate than planned, which resulted in a cost  savings. The negative schedule variance was also being driven by the  mechanism subsystem falling behind schedule due to the shop being  overloaded with work and the mechanism designers supporting other  NASA efforts, among other things. Due to the positive cost variance, we  are forecasting a positive variance at completion ranging from $1 million  to $14 million.
section_title: Laboratory for Atmospheric and Space Physics Does Not Have a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: LASP, the contractor responsible for the Science Operations Center,  Remote Sensing package, and Langmuir Probe and Waves instrument  efforts met one of three fundamental ANSI/EIA-748 practices necessary  for a reliable EVM system. To date, LASP does not have a certified EVM  system. Though the contractor is not required to have a certified system,  we assessed how well the contractor was meeting three ANSI/EIA  guidelines. These guidelines state that the authorized work elements for  the project should be defined typically using a WBS that has been tailored  for effective internal management control of the project and the WBS is  the same for the cost estimate, schedule, and EVM. We found some  slight inconsistencies in the WBS numbering between the EVM report and  the schedule for the Remote Sensing package, which calls into question  the reliability of their overall schedule network. Moreover, the Langmuir  Probe and Waves Instrument had issues with consistency between the  WBS and the schedule. In particular, there was varying levels of  information between the two WBSs, making it hard to use the WBS as a  common thread between the EVM data and the schedule. Since the WBS  is a critical component of EVM, it should be the same for developing the  EVM performance measurement baseline and the schedule. Without a  common link between these two features, project managers cannot fully  understand project cost and schedule variances.
The ANSI/EIA guidelines also state that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifying significant task  interdependencies required to meet project requirements. We found some  sequencing issues with the Langmuir Probe and Waves Instrument  schedule. For example, 47 percent of the remaining activities had  constraints, which defeated the purpose of a using a dynamic schedule.  The quality of the schedule was also hampered by the presence of  schedule lags on 18 percent of the remaining activities. Schedule lags  must be justified because they cannot be easily monitored or included in  risk assessments. Finally, the ANSI/EIA guidelines state that a project  should establish and maintain a time-phased budget baseline to track  cost and schedule variances in an EVM system. Though resource loading  the schedule is not required to meet the ANSI/EIA guidelines, it is a best  practice and therefore resources should be accounted for in the schedule  in order to develop this baseline, according to the GAO cost guide. We  found that all three components showed evidence of a time-phased  budget baseline.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The project conducted an IBR in March 2011 and several areas of  concern were noted. MAVEN project officials stated that many of the  schedule issues we found were also discovered during the IBR and have  now been corrected.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While formal surveillance is not occurring, EVM data assurance reviews  are being performed by the, project office, Mars Program Office, and  MAVEN Standing Review Board representatives. Also, The Aerospace  Corporation is working as an advisor to Science Mission Directorate’s  Planetary Systems Division.
section_title: Data Resulting from the EVM System Is Reliable
paragraphs: Our review of MAVEN’s Science Operations Center, Remote Sensing  package, and Langmuir Probe and Waves instrument EVM data found no  major issues with data reliability. However, there was a lack of variance  analysis reports for these efforts. For example, Science Operations  Center had no variance analysis reports with the explanations for any of  the months reported. In addition, the Remote Sensing package, and  Langmuir Probe and Waves instrument variance analysis reports did not  provide any explanation for major performance swings from one month to  another.
Figure 11 below illustrates that as of August 2011, the Science  Operations Center portion of the MAVEN project was reporting a positive  cumulative cost variance of $0.01 million and a slightly negative  cumulative schedule variance of $0.01 million.
However, since the variance analysis reports provided no information  regarding what is driving the positive cost and slightly negative schedule  variances, we have no insight into the causes for deviations from the  plan. Based on the positive cost variance thus far, we are forecasting a  positive variance at completion of less than $0.5 million at contract  completion.
Figure 12 below illustrates that as of August 2011 the Remote Sensing  package portion of the MAVEN project was reporting a positive  cumulative cost variance of $0.8 million and a slightly negative cumulative  schedule variance of $0.6 million.
Factors behind the positive cost variance include an accounting lag on  the invoicing and payment process associated with the procurements,  which results in the appearance of cost efficiency. This issue has been  occurring for several months and is causing the EVM metrics to be  skewed to reflect false positive cost variances. The variance analysis  reports do not give any explanation for why there is a negative schedule  variance situation as of August 2011. As a result of the positive cost  variance, we are forecasting a positive variance at completion ranging  from $1 million to $7 million.
Figure 13 below illustrates that as of August 2011 the Langmuir Probe  and Waves portion of the MAVEN project was reporting a negative  cumulative cost variance of $0.6 million and negative cumulative  schedule variance of $0.3 million.
Reasons for the negative cost and schedule variances are due to costs  for outside services and materials being more than planned as well as  additional work required to troubleshoot problems and mitigate risks. Due  to these problems, we are forecasting a negative variance at completion  ranging from $2 million and $3 million. According to NASA, the project is  not overrunning its commitment because the EVM baseline does not  include unallocated future expenses held at the project and headquarters  level.
section_title: Orbiting Carbon Observatory 2
paragraphs: NASA’s Orbiting Carbon Observatory 2 (OCO-2) is designed to enable  more reliable predictions of climate change and is based on the original  OCO mission that failed to reach orbit in 2009. It will make precise, time- dependent global measurements of atmospheric carbon dioxide. These  measurements will be combined with data from a ground-based network  to provide scientists with the information needed to better understand the  processes that regulate atmospheric carbon dioxide and its role in the  carbon cycle. NASA expects enhanced understanding of the carbon cycle  will improve predictions of future atmospheric carbon dioxide increases  and the potential impact on the climate.
The OCO-2 mission consists of a dedicated spacecraft with a single  instrument, flying in a near-polar, sun-synchronous orbit. The Jet  Propulsion Laboratory (JPL) has overall responsibility for project  management. The OCO-2 spacecraft is being built by Orbital Sciences  Corporation while the instrument is being built in-house at JPL. Orbital  Sciences Corporation submits spacecraft effort EVM data monthly and  the project incorporates that data into the overall project EVM report. The  project is facing a launch delay because the Taurus XL launch vehicle  failed on the Glory Mission, and the contract was terminated. The project  will be rebaselined as a result of NASA having to select a new launch  vehicle. The current $477.2 million total project cost is a preliminary  amount pending the outcome of the rebaseline process.
section_title: Project Using a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: The OCO-2 project met all three fundamental ANSI/EIA-748 practices  necessary for a reliable EVM system. JPL has a certified EVM system  that complies with the ANSI/EIA standard. However, the implementation  of its EVM system is questionable based on our findings below. We  assessed the JPL EVM data against three ANSI/EIA guidelines. These  guidelines state that the authorized work elements for the project should  be defined typically using a WBS that has been tailored to the project and  the WBS is the same for the cost estimate, schedule, and EVM. We found  that the project had a work breakdown structure that was consistent with  the WBS used for the EVM data. The ANSI/EIA guidelines also state that  projects should have a schedule that describes the sequence of work by  listing activities in the order in which they are to be carried out and  identifying significant task interdependencies required to meet project  requirements. Our review found some sequencing issues in the schedule.  In particular, 14 percent of predecessor and successor tasks were not  linked to one another, which is necessary for properly sequencing work  so that it will update in response to changes. We also found 15 percent of  the remaining activities were constrained, which also hinders the  schedule from responding dynamically to changes and can portray an  artificial or unrealistic view of the project plan. These sequencing issues  and constraint dates within the schedule affect the reliability of the overall  network and the schedule’s ability to correctly calculate float values and  the critical path. Project officials said the missing dependencies are  mainly handoffs produced by level of effort (LOE) activities. Officials said  these activities are not necessary for valid schedule network logic, and  under no circumstances do these activities drive the critical path. They  also said the constrained activities are largely composed of mandated  delivery dates, which JPL uses as control points to manage subsystem  schedule performance prior to assembly, test, and launch operations  delivery, and coordinate major meeting logistics. Finally, the ANSI/EIA  guideline states that a project should establish and maintain a time- phased budget baseline to track cost and schedule variances in an EVM  system. Though resource loading the schedule is not required to meet the  ANSI guidelines, it is a best practice and therefore resources should be  accounted for in the schedule in order to develop this baseline, according  to the GAO cost guide. To develop the baseline, resources must be  accounted for in the schedule and our review found that the schedule was  resource loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: The project conducted an integrated baseline review in March 2011.
section_title: EVM Surveillance Is Being Performed
paragraphs: JPL also has a formal surveillance plan in place for monitoring the EVM  data. In particular, project officials said each month detailed earned value  data analysis is performed on each cost account and each work package  so that conclusive understanding of the performance status can be  reached and communicated within the project team. In addition, during  the project monthly management reviews, both cost and schedule  variances and reasons causing them are presented.
section_title: Data Resulting from the EVM System Are Reliable
paragraphs: We reviewed contractor management reports from December 2010 to  July 2011. Though we found no major data reliability issues when we tried  to map the EVM data in the lower level spacecraft report to the JPL  project level report, we could not understand how the costs tracked from  one report to another. In a subsequent interview, officials explained, with  supporting documentation, how the lower level Orbital Sciences budget at  completion mapped to the budget at completion found in the JPL Project  level EVM report. Though we appreciate the explanations provided by  project officials regarding the differences between the two reports, the  issue remains that without additional documentation and explanations by  project officials, GAO or another independent party could not have  reconciled the data.
Figure 14 below illustrates that as of July 2011, the project was reporting  a positive cumulative cost variance of $0.6 million and a negative  schedule variance of $8 million as seen in the graph below. According to  project officials, the schedule variance was being caused by spectrometer  slit instability and an incompatible memory chip in the Remote Electronics  Module.
Due to the positive cost variance and negative schedule variance we are  forecasting a negative variance at completion ranging from $1 million to  $26 million. Project officials said the project is not overrunning its  commitment because the EVM baseline provided to GAO does not  include unallocated future expenses held at the project and headquarters  level.
section_title: Spacecraft Contractor Has a Certified EVM System Compliant with ANSI/EIA Standard
paragraphs: The spacecraft contractor, Orbital Sciences Corporation, met two of the  three fundamental ANSI/EIA-748 practices necessary for a reliable EVM  System. At the time of our review, the Orbital Sciences did not have a  certified EVM system that complied with the ANSI/EIA-748 standard. In  January 2012, DCMA certified the Orbital Sciences EVM System. Though  the contractor now has a certified system, the implementation of that  system is questionable based on our findings below. We assessed how  well the contractor’s EVM system was meeting three ANSI/EIA  guidelines. These guidelines state that the authorized work elements for  the project should be defined typically using a WBS that has been tailored  to the project and the WBS is the same for the cost estimate, schedule,  and EVM. We found that the spacecraft’s WBS in the schedule was not  consistent with the WBS used for the EVM data. The ANSI/EIA guidelines  also state that projects should have a schedule that describes the  sequence of work by listing activities in the order in which they are to be  carried out and identifying significant task interdependencies required to  meet project requirements. We found some sequencing issues in the  spacecraft contractor schedule. We found that 27 percent of the  remaining activities in the spacecraft’s schedule were missing  predecessor and successor links, which are necessary for properly  sequencing work so that the schedule will update in response to changes.  In addition, 24 percent of the remaining activities had date constraints,  which also hinder the schedule’s ability to respond dynamically to status  updates resulting in an artificial or unrealistic view of the project plan.  These sequencing issues and constraint dates within the schedule affect  the reliability of the overall network and the schedule’s ability to correctly  calculate float values and the critical path.
subsections: section_title: Spacecraft Contractor Conducted an Integrated Baseline Review
paragraphs: Orbital Sciences conducted an integrated baseline review of the  spacecraft effort in March 2011.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While formal surveillance is not occurring, project officials stated that  EVM performance data is reviewed during the monthly status reviews.
section_title: Data Resulting from the EVM System Are Reliable
paragraphs: We reviewed contract performance reports from April 2011 to July 2011.  Because we had only 4 months of data to review, we were unable to  forecast estimate variance at completion. As of July 2011, the project was  reporting a positive cumulative cost variance of $0.4 million and a  negative cumulative schedule variance of $2 million as seen in the graph  below. The positive cost variance was being driven by lower than  expected burden rates and less staff supporting project management,  flight assurance, and systems engineering efforts.
The positive cost variance was being driven by lower than expected  burden rates and less staff supporting project management, flight  assurance, and systems engineering efforts.
section_title: Radiation Belt Storm Probes
paragraphs: The Radiation Belt Storm Probes (RBSP) mission will explore the Sun’s  influence on the Earth and near-Earth space by studying the planet’s  radiation belts at various scales of space and time. This insight into the  physical dynamics of the Earth’s radiation belts will provide scientists with  data to make predictions of changes in this little understood region of  space. Understanding the radiation belt environment has practical  applications in the areas of spacecraft system design, mission planning,  spacecraft operations, and astronaut safety. The RBSP project built two  spacecraft that will be used to measure the particles, magnetic and  electric fields, and waves that reside in the Van Allen radiation belts.  RBSP launched on August 30, 2012 on a two-year prime mission.
The RBSP spacecrafts and ground system are being designed,  developed, and tested by the John Hopkins University’s Applied Physics  Laboratory.level.
section_title: Project Does Not Have a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: At the project level, the Applied Physics Laboratory only fully met one and  partially met another of the three fundamental practices necessary for a  reliable EVM system. RBSP is the first full NASA mission to use EVM at  the Applied Physics Laboratory. According to the RBSP project manager,  the RBSP project implemented a limited earned value management  system in Phase B as a risk mitigation activity for Phase C/D. This early  implementation was a risk mitigation activity, which allowed the project’s  control account manager, instrument provider, and project office to better  understand the reporting process and the use of the EVM system. The  use of EVM during this phase was also intended to allow for timely,  accurate, and useful EVM reporting during the formal reporting in later  phases of the project. Since then Applied Physics Laboratory has made  good progress and is in the process of meeting the intent of being  compliant with the 32 ANSI/EIA-748 guidelines.
We assessed how well the Applied Physics Laboratory was meeting the  three ANSI/EIA guidelines. These guidelines state that the authorized  work elements for the project should be defined typically using a WBS  that has been tailored to the project and the WBS is the same for the cost  estimate, schedule, and EVM. Our review, we found that that the WBS  used in the schedule was not consistent with the WBS used for the EVM  data. Project officials said that the project utilizes the Applied Physics  Laboratory WBS for all earned value management activity, which is then  mapped to the NASA WBS for reporting to the sponsor. Also, the Applied  Physics Laboratory WBS is uniformly utilized and consistent across all  control accounts in the EVM system. This internal WBS ties into both the  contract performance report and the integrated master schedule utilized  on the project.
The ANSI/EIA guidelines also state that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifying significant task  interdependencies required to meet project requirements. Our review  found some sequencing issues in the schedule. For example, 23 percent  of the remaining activities were missing predecessor and successor links,  which are necessary for properly sequencing work so that the schedule  will update in response to changes. We also found that 29 percent of the  remaining activities had date constraints, which also hinder the  schedule’s ability to respond dynamically to status updates resulting in an  artificial or unrealistic view of the project plan. When schedules are not  sequenced properly, float values and the calculated critical path will not  be valid. Project officials explained that sequencing issues were a result  of the project consciously including constrained instrument deliveries and  deliverables, level of effort activities and material and subcontractor  expenditures in the integrated master schedule, and though the RBSP  integrated master schedule had these issues it was able to monitor the  critical and near critical paths of all spacecraft systems and subsystems.  Finally, the ANSI/EIA guidelines state that a project should establish and  maintain a time-phased budget baseline to track cost and schedule  variances in an EVM system. Though resource loading the schedule is  not required to meet the ANSI/EIA guidelines, it is a best practice and  therefore resources should be accounted for in the schedule in order to  develop this baseline, according to the GAO cost guide. Our review found  that the schedule was resource loaded.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: NASA conducted an integrated baseline review in August 2009. Of the  seven IBR objectives identified, two were partially met and five were met.  In December 2010, the IBR deputy chief notified the Applied Physics  Laboratory that all areas of concern had been closed. As a result, the  overall consensus of the government review team was that the integrated  baseline review was successful.
section_title: EVM Surveillance Is Not Being Performed
paragraphs: While formal surveillance is not being performed, the project office  reviews the EVM data on a monthly basis. Although they do not perform  formal surveillance, an Applied Physics Laboratory official said that they  performed additional monthly independent reviews of the RBSP EVM  system throughout Phase C/D.
section_title: Data Resulting from the EVM System Are Reliable
paragraphs: We reviewed contract performance reports from August 2010 to April  2011 and July 2011 to August 2011. We were not provided reports for  May and June 2011. This was due to a re-plan determined necessary by  the Living with a Star Program Office and NASA headquarters in May  2011 due to changes in the launch manifest. Figure 16 below illustrates  that as of August 2011, the project was reporting a negative cumulative  cost variance of approximately $32 million and a negative cumulative  schedule variance of $3 million.
The negative cumulative cost variance was caused by sustained effort on  the radio frequency communications, as well as by work on the avionics  equipment, and ground system software launch and post launch  components being behind schedule. The schedule variance is minimal  since the project is nearing completion. Due to the negative cost variance  we are forecasting a negative variance at completion from $40 million to  $41 million. Project officials noted the forecasted variance at completion  is below the revised contract value of $351.1 million, although above the  project’s estimated budget at completion. In addition, officials said this is  a project-level variance, and does not account for the application of  unallocated future expenses to fund the movement of the launch date and  to keep the project on track. When the project launched in August 2012,  its estimate at completion was below the $351.1 million budget at  completion.
section_title: Stratospheric Observatory for Infrared Astronomy
paragraphs: The Stratospheric Observatory for Infrared Astronomy (SOFIA) is a joint  project between NASA and Deutsche Zentrum für Luft-und Raumfahrt  (DLR), the German space agency, to install a 2.5 meter telescope, as well  as other scientific instruments capable of infrared and sub-millimeter  observations, in a specially modified Boeing 747SP aircraft. This airborne  observatory is designed to provide routine access to the visual, infrared,  far-infrared, and sub-millimeter parts of the electromagnetic spectrum. Its  mission objectives include studying many different kinds of astronomical  objects and phenomena, including star birth and death; the formation of  new solar systems; planets, comets, and asteroids in our solar system;  and black holes at the center of galaxies. Currently, five U.S. and two  German funded interchangeable instruments for the observatory are  being developed to allow a range of scientific measurement to be taken  by SOFIA.
The SOFIA project office is using EVM at the project level as well as  collecting EVM data from the L-3 Communications Integrated Systems  L.P. (L-3), which is responsible for the airborne system platform effort.  The EVM data provided to the project for the airborne observatory  platform effort is a subset of the overall SOFIA project level EVM report.  The German component of the SOFIA project does not generate earned  value data and are not part of the project’s budget baseline. The  University Space Research Association (USRA) has a support contract to  help the Ames Research Center manage SOFIA’s science and mission  operations in cooperation with the Deutsches SOFIA Institut. The USRA  contract was established before NASA began requiring earned value  management compliance. As a result, they are not required to generate  earned value data. However, all of these components are subsets of the  overall SOFIA project.
section_title: Project does not have a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: SOFIA project did not meet any of the three fundamental ANSI/EIA-748  practices necessary for a reliable EVM system. Project officials said in- house projects are not required to have certified EVM systems. Though  the SOFIA project does not have a certified system, we assessed how  well the project was meeting three ANSI/EIA guidelines. These guidelines  state that the authorized work elements for the project should be defined  typically using a WBS that has been tailored to the project and the WBS  is the same for the cost estimate, schedule, and EVM. We found that the  WBS used in the SOFIA schedule was consistent with the WBS used for  the EVM data. The ANSI/EIA guidelines also state that projects should  have a schedule that describes the sequence of work by listing activities  in the order in which they are to be carried out and identifying significant  task interdependencies required to meet project requirements. Our review  found some sequencing issues in the schedule. For example, 16 percent  of the remaining activities had open ended logic links, and 1 percent had  date constraints, among other things. SOFIA project officials explained  that the majority of the missing links were due to using hammock  activities to resource load the schedule, which resulted in missing  successors. Out of all of the activities missing successors, only 11 were  not hammocked activities, which accounted for a very small amount. In  addition, SOFIA project officials said that constraints in the schedule were  justified since they represented external deliveries or fiscal year funding  availability. They added that the one hard constraint, Must Start On, was  used to represent a fixed date for an international visit. While the majority  of these explanations seem reasonable, when schedules are not  sequenced properly, the slack values and the calculated critical path will  not be valid.
subsections: section_title: Project Did Not Conducted an Integrated Baseline Review
paragraphs: An IBR was not conducted at the project level. Project officials said  SOFIA, an in-house project, did not begin collecting EVM data until very  late in the development phase, and because of this an IBR was not  conducted. Officials added although SOFIA did not conduct a project- level IBR, the EVM baseline was established concurrently with an Agency  approved re-plan and joint confidence level analysis in the 2009/2010  time frame, and was reviewed by a Standing Review Board. Though the  Standing Review Board review satisfied some of the objectives of an IBR,  including confirmation of the schedule and budget baselines (e.g.,  schedule review, risk review, key milestones identified), it did not address  the full IBR checklist (e.g. work authorizations, control account plans,  EVM system description).
section_title: EVM Surveillance Is Not being Performed
paragraphs: While formal surveillance is not occurring at the project level, EVM data  assurance reviews are being performed monthly and quarterly by the  SOFIA project office representatives.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to August  2011. We found various issues that bring into question the reliability of the  SOFIA Project EVM data. For example, we found negative values due to  an “over-reporting” of progress in previous months that was caused by a  problem with translating percent complete progress from the University  Space Research Association schedule to the SOFIA integrated master  schedule as well as corrections/modifications in costs posted for support  service contracts. In addition, we found other anomalies in the data that  SOFIA project officials explained were most likely due to mischarges by  employees, delayed cost postings, or employees continuing to use  charge codes inappropriately. While the cost impact of these problems  was not that large for any one WBS element, each of these issues causes  us to question the reliability of the data. According to project officials, the  variances that caused these anomalies did not meet the reporting  threshold. A variance analysis report provides a detailed, narrative report  explaining significant cost and schedule variances and other contract  problems and topics. Without this information, management cannot  understand the reasons for variances and the contractor’s plan for fixing  them. When variance analysis reports are not produced, the EVM data  will not be meaningful or useful as a management tool.
Figure 17 below illustrates that as of August 2011, the SOFIA project was  reporting a positive cumulative cost variance of $3.5 million and a  negative cumulative schedule variance of $6.6 million.
Due to the cost and schedule variances we are forecasting a negative  variance at completion ranging from $1.4 million to $76 million.
section_title: Airborne System Platform Contractor Has a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: The contractor met all three fundamental ANSI/EIA-748 practices  necessary for a reliable EVM system. In 2002, DCMA certified that the  contractor has an EVM system compliant with the ANSI/EIA standard.
subsections: section_title: Project Conducted an Integrated Baseline Review of Airborne System Platform Effort
paragraphs: The project conducted an IBR in November 2007.
section_title: EVM Surveillance Is Being Performed
paragraphs: It is unclear if EVM surveillance is being performed on the L-3 EVM data.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We reviewed contract performance reports from August 2010 to July  2011. We found various data anomalies that call into question the  reliability of the contractor’s EVM data for the contract performance  reports we reviewed. For example, we found negative values, actual cost  of work performed being reported without work being scheduled and/or  performed. Project officials stated that these anomalies were caused by  performance being taken in later months and also because these issues  did not trip the threshold reporting.
Figure 18 below illustrates that as of July 2011, the airborne system  platform contractor was reporting a positive cumulative cost variance of  $3.2 million and a negative schedule variance of $0.6 million.
The favorable cost variance is due to efficiencies in project oversight and  engineering. The unfavorable cumulative schedule variance is being  driven by a lack of government furnished equipment supposed to be  received from NASA that is causing a backlog of tasks. Due to the cost  and schedule variances we are forecasting a positive variance at  completion ranging from $3 million to $4 million.
section_title: Tracking and Data Relay Satellite Replenishment
paragraphs: The Tracking and Data Relay Satellite (TDRS) System consists of in-orbit  communication satellites stationed at geosynchronous altitude coupled  with two ground stations located in New Mexico and Guam. The satellite  network and ground stations provide mission services for near-Earth user  satellites and orbiting vehicles. TDRS-K and L are the 11th and 12th  satellites, respectively, to be built for the TDRS system. They will  contribute to the existing network by providing continuous high bandwidth  digital voice, video, and mission payload data, as well as health and  safety data relay services to Earth-orbiting spacecraft such as the  International Space Station and the Hubble Space Telescope. NASA is  planning to launch TDRS-K in December 2012 followed by the TDRS-L  launch in February 2014.
NASA is collecting EVM data from both the spacecraft and sustainment  efforts. In December 2007, NASA awarded a fixed price incentive contract  to design, develop, fabricate, integrate, test, ship, provide launch support,  conduct on-orbit checkout operations and provide sustaining engineering  support for two spacecraft, TDRS-K and TDRS-L, to Boeing Satellite  Systems, Inc (Boeing).
section_title: Spacecraft Contractor Has a Certified EVM System Compliant with the ANSI/EIA Standard
paragraphs: The spacecraft contractor, Boeing, met the three fundamental ANSI/EIA- 748 practices necessary for a reliable EVM system. Boeing has a certified  EVM system that complies with the ANSI/EIA EVM standard. Though the  contractor has a certified system, the implementation of that system is  questionable based on our findings below. We also assessed whether the  spacecraft contractor’s EVM data against three ANSI/EIA guidelines.  These guidelines state that the authorized work elements for the project  should be defined typically using a WBS that has been tailored to the  project and the WBS is the same for the cost estimate, schedule, and  EVM. We found that the WBS used in the spacecraft integrated master  schedule was consistent with the WBS used for the EVM data.
The ANSI/EIA guidelines also state that projects should have a schedule  that describes the sequence of work by listing activities in the order in  which they are to be carried out and identifying significant task  interdependencies required to meet project requirements. We found some  sequencing issues in the contractor’s schedule. For example, 13 percent  of the remaining activities were constrained. When schedules are not  sequenced properly, float values and the calculated critical path will not  be valid. Project officials acknowledged the constraints and said they are  a result of having to adjust support activities due to spacecraft integration  and test delays as well as alignments to the current/actual manifest dates,  which differ from the Boeing contractual and launch readiness dates. In  addition, critical path metrics are generated and analyzed monthly to track  the Boeing performance against the critical path activities. Finally, the  ANSI/EIA guidelines state that a project should establish and maintain a  time-phased budget baseline to track cost and schedule variances in an  EVM system. Though resource loading the schedule is not required to  meet the ANSI/EIA guidelines, it is a best practice and therefore  resources should be accounted for in the schedule in order to develop  this baseline, according to the GAO cost guide. We found that the  schedule was not resource loaded. Project officials said the integrated  master schedule is resource loaded but not inside the Microsoft Project  schedule because it does not directly interface with the Boeing  financial/EVM system. They stated that the integrated master schedule is  produced using Microsoft Project and imported into a planning software  tool. After the resource loading effort is performed, the planning data is  transferred into the EVM system. The tool integrates the Microsoft Project  schedule with the Boeing financial system and the resource allocations  for each task. When adjustments are required Boeing again utilizes the  planning tool.
subsections: section_title: Project Conducted an Integrated Baseline Review
paragraphs: An integrated baseline review was conducted in 2008 where 211 issues  were raised. All of these issues have since been resolved and closed.
section_title: EVM Surveillance Is Being Performed
paragraphs: DCMA prepares a monthly EVM analysis report and performs continuous  surveillance of Boeing’s EVM implementation by sampling various control  account managers for interviews about the process.
section_title: Data Resulting from the EVM System Are Somewhat Reliable
paragraphs: We examined contract performance reports from August 2010 through  August 2011. Figure 19 illustrates that as of August 2011, the project was  reporting a negative cumulative cost variance of approximately $131  million and a negative cumulative schedule variance of $7 million.
The negative cumulative cost variance was being driven by higher staffing  levels to support integration, the Preliminary Design Review and Critical  Design Review, as well as an incorrect assessment of project  requirements and the inability to use heritage specifications. Labor costs  were also higher than expected due to part failures and the late  completion of component qualifications. Extended test activities also  contributed to the cumulative negative cost variance. Finally, more than  expected resources were needed to complete board and slice designs,  generate drawings, and assemble and test components because of the  complexity of the design. Due to the negative cost and schedule variance  we are forecasting a variance at completion ranging from $152 million to  $185 million. Project officials said GAO’s independent variance at  completion gives the impression that NASA may request additional  funding to complete the TDRS K and L. Because this is a fixed price,  incentive fee contract, NASA officials said the agency is obligated only to  pay up to the price ceiling of the contract.
section_title: TDRS Sustainment Effort EVM Summary
paragraphs: Boeing is the contractor for both the spacecraft and sustainment efforts.  As noted above, Boeing met all three fundamental practices for a reliable  EVM system.
In addition, the 2008 integrated baseline review was conducted for both  the spacecraft and sustainment effort. As well, the formal surveillance  performed by Boeing and the Defense Contract Management Agency  applies to the sustainment effort  We reviewed contract performance reports from April 2011 to August  2011. Because we had only 5 months of data, we were not able to  forecast an independent estimate at completion. Figure 20 below  illustrates that as of August 2011, the project was reporting a positive  cumulative cost variance of $0.08 million. Because there were no  variance analysis reports accompanying the sustaining effort, we were  unable to determine what was causing the positive cost variance. Project  officials stated that the Performance Measurement Baseline for this effort  was almost entirely level of effort, so minimal variances would be  occurring. They added that during the August 2011 time period, the  contract reflected an April 2012 launch date even though the launch was  being delayed. Consequently, Boeing’s reports were reflecting work  scheduled to occur in support of the earlier launch date when in fact very  little effort was being done. As a result, since minimal costs were  incurred, this resulted in a positive cumulative cost variance.
section_title: Appendix IV: Comments from the National Aeronautics and Space Administration
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelby S. Oakley and Karen  Richey (Assistant Directors), Greg Campbell; Jennifer K. Echard; Tisha D.  Derricotte; Laura Greifner; Kristine R. Hassinger; Ben Jaskiewicz; William  Laing; Richard Lee; Eric Lofgren; Kenneth E. Patton; Jose A. Ramos;  Carrie W. Rogers; Stacey L. Steele; Roxanna T. Sun; and Umesh  Thakkar made key contributions to this report.