Information Technology: Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System Acquisitions
Published: 2009-10-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2009, the federal government planned to spend about $71 billion on information technology (IT) investments. To more effectively manage such investments, in 2005 the Office of Management and Budget (OMB) directed agencies to implement earned value management (EVM). EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of anticipated costs at completion. GAO was asked to assess selected agencies' EVM policies, determine whether they are adequately using earned value techniques to manage key system acquisitions, and eval- uate selected investments' earned value data to determine their cost and schedule performances. To do so, GAO compared agency policies with best practices, performed case studies, and reviewed documenta- tion from eight agencies and 16 major investments with the highest levels of IT development-related spending in fiscal year 2009.
section_title: What GAO Found
paragraphs: While all eight agencies have established policies requiring the use of EVM on major IT investments, these policies are not fully consistent with best practices. In particular, most lack training requirements for all relevant personnel responsible for investment oversight. Most policies also do not have adequately defined criteria for revising program cost and schedule baselines. Until agencies expand and enforce their EVM policies, it will be difficult for them to gain the full benefits of EVM. GAO's analysis of 16 investments shows that agencies are using EVM to manage their system acquisitions; however, the extent of implementation varies. Specifically, for 13 of the 16 investments, key practices necessary for sound EVM execution had not been implemented. For example, the project schedules for these investments contained issues--such as the improper sequencing of key activities--that undermine the quality of their performance baselines. This inconsistent application of EVM exists in part because of the weaknesses contained in agencies' policies, combined with a lack of enforcement of policies already in place. Until key EVM practices are fully implemented, these investments face an increased risk that managers cannot effectively optimize EVM as a management tool. Furthermore, earned value data trends of these investments indicate that most are currently experiencing shortfalls against cost and schedule targets. The total life-cycle costs of these programs have increased by about $2 billion. Based on GAO's analysis of current performance trends, 11 programs will likely incur cost overruns that will total about $1 billion at contract completion--in particular, 2 of these programs account for about 80 percent of this projection. As such, GAO estimates the total cost overrun to be about $3 billion at program completion (see figure). However, with timely and effective management action, it is possible to reverse negative trends so that the projected cost overruns may be reduced.

section_title: Letter
paragraphs: In fiscal year 2009, the federal government planned to spend over $70  billion on information technology (IT) investments, many of which involve  systems and technologies to modernize legacy systems, increase  communication and networking capabilities, and transition to new  systems designed to significantly improve the government’s ability to carry  out critical mission functions into the 21st century. To more effectively  manage such investments, the Office of Management and Budget (OMB)  has a number of key initiatives under way—one of which was established  in 2005 and directs agencies to implement earned value management  (EVM). EVM is a project management approach that, if implemented  appropriately, provides objective reports of project status, produces early  warning signs of impending schedule slippages and cost overruns, and  provides unbiased estimates of anticipated costs at completion.
This report responds to your request that we review the federal  government’s use of EVM. Specifically, our objectives were to (1) assess  whether key departments and agencies have appropriately established  EVM policies, (2) determine whether these agencies are adequately using  earned value techniques to manage key system acquisitions, and   (3) evaluate the earned value data of these selected investments to  determine their cost and schedule performances.
To address our objectives, we reviewed agency EVM policies and  individual programs’ EVM-related documentation, including cost  performance reports and project schedules, from eight agencies and 16  major investments from those agencies, respectively. The eight agencies  account for about 75 percent of the planned IT spending for fiscal year  2009. The 16 programs selected for case study represent investments with  about $3.5 billion in total planned spending for system development work  in fiscal year 2009. We compared the agencies’ policies and practices with  federal standards and best practices of leading organizations to determine  the effectiveness of their use of earned value data in managing IT  investments. We also analyzed the earned value data from the programs to  determine whether they are projected to finish within planned cost and  schedule targets. In addition, we interviewed relevant agency officials,  including key personnel on programs that we selected for case study and  officials responsible for implementing EVM.
We conducted this performance audit from February to October 2009, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objective. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objective. Appendix I contains further details about our  objectives, scope, and methodology. See also the page of related products  at the end of this report for previous work that we have done on certain  programs in our case studies.
subsections: section_title: Background
paragraphs: Each year, OMB and federal agencies work together to determine how  much the government plans to spend on IT projects and how these funds  are to be allocated. Planned federal IT spending in fiscal year 2009 totaled  about $71 billion—of which $22 billion was planned for IT system  development work, and the remainder was planned for operations and  maintenance of existing systems. OMB plays a key role in overseeing  federal agencies’ IT investments and how they are managed, stemming  from its functions of assisting the President in overseeing the preparation  of the federal budget and supervising budget preparation in executive  branch agencies. In helping to formulate the President’s spending plans,  OMB is responsible for evaluating the effectiveness of agency programs,  policies, and procedures; assessing competing funding demands among  agencies; and setting funding priorities. To carry out these responsibilities,  OMB depends on agencies to collect and report accurate and complete  information; these activities depend, in turn, on agencies having effective  IT management practices.
To drive improvement in the implementation and management of IT  projects, Congress enacted the Clinger-Cohen Act in 1996, expanding the  responsibilities delegated to OMB and agencies under the Paperwork  Reduction Act. The Clinger-Cohen Act requires agencies to engage in  performance- and results-based management, and to implement and  enforce IT management policies and guidelines. The act also requires OMB  to establish processes to analyze, track, and evaluate the risks and results  of major capital investments in information systems made by executive  agencies.
Over the past several years, we have reported and testified on OMB’s  initiatives to highlight troubled projects, justify IT investments, and use  project management tools. We have made multiple recommendations to  OMB and federal agencies to improve these initiatives to further enhance  the oversight and transparency of federal IT projects. As a result, OMB  recently used this body of work to develop and implement improved  processes to oversee and increase transparency of IT investments.  Specifically, in June 2009, OMB publicly deployed a Web site that displays  dashboards of all major federal IT investments to provide OMB and others  with the ability to track the progress of these investments over time.
subsections: section_title: EVM Provides Insight on Program Cost and Schedule
paragraphs: Given the size and significance of the government’s investment in IT, it is  important that projects be managed effectively to ensure that public  resources are wisely invested. Effectively managing projects entails,  among other things, pulling together essential cost, schedule, and  technical information in a meaningful, coherent fashion so that managers  have an accurate view of the program’s development status. Without  meaningful and coherent cost and schedule information, program  managers can have a distorted view of a program’s status and risks. To  address this issue, in the 1960s, the Department of Defense (DOD)  developed the EVM technique, which goes beyond simply comparing  budgeted costs with actual costs. This technique measures the value of  work accomplished in a given period and compares it with the planned  value of work scheduled for that period and with the actual cost of work  accomplished.
Differences in these values are measured in both cost and schedule  variances. Cost variances compare the value of the completed work (i.e.,  the earned value) with the actual cost of the work performed. For  example, if a contractor completed $5 million worth of work and the work  actually cost $6.7 million, there would be a negative $1.7 million cost  variance. Schedule variances are also measured in dollars, but they  compare the earned value of the completed work with the value of the  work that was expected to be completed. For example, if a contractor  completed $5 million worth of work at the end of the month but was  budgeted to complete $10 million worth of work, there would be a  negative $5 million schedule variance. Positive variances indicate that  activities are costing less or are completed ahead of schedule. Negative  variances indicate activities are costing more or are falling behind  schedule. These cost and schedule variances can then be used in  estimating the cost and time needed to complete the program.
Without knowing the planned cost of completed work and work in  progress (i.e., the earned value), it is difficult to determine a program’s  true status. Earned value allows for this key information, which provides  an objective view of program status and is necessary for understanding the  health of a program. As a result, EVM can alert program managers to  potential problems sooner than using expenditures alone, thereby  reducing the chance and magnitude of cost overruns and schedule  slippages. Moreover, EVM directly supports the institutionalization of key  processes for acquiring and developing systems and the ability to  effectively manage investments—areas that are often found to be  inadequate on the basis of our assessments of major IT investments.
section_title: Federal Guidance Calls for Using EVM to Improve IT Management
paragraphs: In August 2005, OMB issued guidance outlining steps that agencies must  take for all major and high-risk development projects to better ensure  improved execution and performance and to promote more effective  oversight through the implementation of EVM. Specifically, this guidance  directs agencies to (1) develop comprehensive policies to ensure that their  major IT investments are using EVM to plan and manage development;   (2) include a provision and clause in major acquisition contracts or agency  in-house project charters directing the use of an EVM system that is  compliant with the American National Standards Institute (ANSI)  standard; (3) provide documentation demonstrating that the contractor’s  or agency’s in-house EVM system complies with the national standard;   (4) conduct periodic surveillance reviews; and (5) conduct integrated  baseline reviews on individual programs to finalize their cost, schedule,  and performance goals.
Building on OMB’s requirements, in March 2009, we issued a guide on best  practices for estimating and managing program costs. This guide  highlights the policies and practices adopted by leading organizations to  implement an effective EVM program. Specifically, in the guide, we  identify the need for organizational policies that establish clear criteria for  which programs are required to use EVM, specify compliance with the  ANSI standard, require a standard product-oriented structure for defining  work products, require integrated baseline reviews, provide for specialized  training, establish criteria and conditions for rebaselining programs, and  require an ongoing surveillance function. In addition, we identify key  practices that individual programs can use to ensure that they establish a  sound EVM system, that the earned value data are reliable, and that the  data are used to support decision making.
section_title: Prior Reviews on Agency Use of EVM to Acquire and Manage IT Systems Have Identified Weaknesses
paragraphs: We have previously reported on the weaknesses associated with the  implementation of sound EVM programs at various agencies, as well as on  the lack of aggressive management action to correct poor cost and  schedule performance trends based on earned value data for major system  acquisition programs:  In July 2008, we reported that the Federal Aviation Administration’s EVM  policy was not fully consistent with best practices. For example, the  agency required its program managers to obtain EVM training, but did not  enforce completion of this training or require other relevant personnel to  obtain this training. In addition, although the agency was using EVM to  manage IT acquisitions, not all programs were ensuring that their earned  value data were reliable. Specifically, of the three programs collecting  EVM data, only one program adequately ensured that its earned value data  were reliable. As a result, the agency faced an increased risk that  managers were not getting the information they needed to effectively  manage the programs. In response to our findings and recommendations,  the Federal Aviation Administration reported that it had initiatives under  way to improve its EVM oversight processes.
In September 2008, we reported that the Department of the Treasury’s  EVM policy was not fully consistent with best practices. For example,  while the department’s policy addressed some practices, such as  establishing clear criteria for which programs are to use EVM, it did not  address others, such as requiring and enforcing EVM training. In addition,  six programs at Treasury and its bureaus were not consistently  implementing practices needed for establishing a comprehensive EVM  system. For example, when executing work plans and recording actual  costs, a key practice for ensuring that the data resulting from the EVM  system are reliable, only two of the six investments that we reviewed  incorporated government costs with contractor costs. As a result, we  reported that Treasury may not be able to effectively manage its critical  programs. In response to our findings and recommendations, Treasury  reported that it would release a revised EVM policy and further noted that  initiatives to improve EVM-related training were under way.
In a series of reports and testimonies from September 2004 to June 2009, we  reported that the National Oceanic and Atmospheric Administration’s  National Polar-orbiting Operational Environmental Satellite System program  was likely to overrun its contract at completion on the basis of our analysis of  contractor EVM data. Specifically, the program had delayed key milestones  and experienced technical issues in the development of key sensors, which  we stated would affect cost and schedule estimates. As predicted, in June  2006 the program was restructured, decreasing its complexity, delaying the  availability of the first satellite by 3 to 5 years, and increasing its cost estimate  from $6.9 billion to $12.5 billion. However, the program has continued to face  significant technical and management issues. As of June 2009, launch of the  first satellite was delayed by 14 months, and our current projected total cost  estimate is approximately $15 billion. We made multiple recommendations to  improve this program, including establishing a realistic time frame for  revising the cost and schedule baselines, developing plans to mitigate the risk  of gaps in satellite continuity, and tracking the program executive  committee’s action items from inception to closure.
section_title: Agencies’ EVM Policies Are Not Comprehensive
paragraphs: While the eight agencies we reviewed have established policies requiring  the use of EVM on their major IT investments, none of these policies are  fully consistent with best practices, such as standardizing the way work  products are defined. We recently reported that leading organizations  establish EVM policies that    establish clear criteria for which programs are to use EVM;  require programs to comply with the ANSI standard; require programs to use a product-oriented structure for defining work  products; require programs to conduct detailed reviews of expected costs,  schedules, and deliverables (called an integrated baseline review);  require and enforce EVM training;   define when programs may revise cost and schedule baselines (called  require system surveillance—that is, routine validation checks to ensure  that major acquisitions are continuing to comply with agency policies and  standards.
Table 1 describes the key components of an effective EVM policy.
The eight agencies we reviewed do not have comprehensive EVM policies.  Specifically, none of the agencies’ policies are fully consistent with all  seven key components of an effective EVM policy. Table 2 provides a  detailed assessment, by agency, and a discussion of the agencies’ policies  follows the table.
Criteria for implementing EVM on all IT major investments: Seven of  the eight agencies fully defined criteria for implementing EVM on major IT  investments. The agencies with sound policies typically defined “major”  investments as those exceeding a certain cost threshold, and, in some  cases, agencies defined lower tiers of investments requiring reduced levels  of EVM compliance. Veterans Affairs only partially met this key practice  because its policy did not clearly state whether programs or major  subcomponents of programs (projects and subprojects) had to comply  with EVM requirements. According to agency officials, this lack of clarity  may cause EVM to be inconsistently applied across the investments.  Without an established policy that clearly defines the conditions under  which new or ongoing acquisition programs are required to implement  EVM, these agencies cannot ensure that EVM is being appropriately  applied on their major investments.
Compliance with the ANSI standard: Seven of the eight agencies required  that all work activities performed on major investments be managed by an  EVM system that complies with industry standards. One agency,  Transportation, partially met this key practice because its policy contained  inconsistent criteria for when investments must comply with standards.  Specifically, in one section, the policy requires a certain class of  investments to adhere to a subset of the ANSI standard; however, in  another section, the policy merely states that the investments must comply  with general EVM principles. This latter section is vague and could be  interpreted in multiple ways, either more broadly or narrowly than the  specified subset of the ANSI standard. Without consistent criteria on  investment compliance, Transportation may be unable to ensure that the  work activities for some of its major investments are establishing sound  EVM systems that produce reliable earned value data and provide the  basis for informed decision making.
Standard structure for defining the work products: DOD was the only  agency to fully meet this key practice by developing and requiring the use  of standard product-oriented work breakdown structures. Four agencies  did not meet this key practice, while the other three only partially  complied. Of those agencies that partially complied, National Aeronautics  and Space Administration (NASA) policy requires mission (or space flight)  projects to use a standardized product-oriented work breakdown  structure; however, IT projects do not have such a requirement. NASA  officials reported that they are working to develop a standard structure for  their IT projects; however, they were unable to provide a time frame for  completion. Homeland Security and Justice have yet to standardize their  product structures.
Among the agencies that did not implement this key practice, reasons  included, among other things, the difficulty in establishing a standard  structure for component agencies that conduct different types of work  with varying complexity. While this presents a challenge, agencies could  adopt an approach similar to DOD’s and develop various standard work  structures based on the kinds of work being performed by the various  component agencies (e.g., automated information system, IT  infrastructure, and IT services). Without fully implementing a standard  product-oriented structure (or structures), agencies will be unable to  collect and share data among programs and may not have the information  they need to make decisions on specific program components.
Integrated baseline review: All eight agencies required major IT  investments to conduct an integrated baseline review to ensure that  program baselines fully reflect the scope of work to be performed, key  risks, and available resources. For example, DOD required that these  reviews occur within 6 months of contract award and after major  modifications have taken place, among other things.
Training requirements: Commerce was the only agency to fully meet this  key practice by requiring and enforcing EVM training for all personnel  with investment oversight and program management responsibilities.  Several of the partially compliant agencies required EVM training for  project managers—but did not extend this requirement to other program  management personnel or executives with investment oversight  responsibilities. Many agencies told us that it would be a significant  challenge to require and enforce EVM training for all relevant personnel,  especially at the executive level. Instead, most agencies have made  voluntary EVM training courses available agencywide. However, without  comprehensive EVM training requirements and enforcement, agencies  cannot effectively ensure that programs have the appropriate skills to  validate and interpret EVM data, and that their executives will be able to  make fully informed decisions based on the EVM analysis.
Rebaselining criteria: Three of the eight agencies fully met this key  practice. For example, the Justice policy outlines acceptable reasons for  rebaselining, such as when the baseline no longer reflects the current  scope of work being performed, and requires investments to explain why  their current plans are no longer feasible and to develop realistic cost and  schedule estimates for remaining work. Among the five partially compliant  agencies, Agriculture and Veterans Affairs provided policies, but in draft  form; NASA was in the process of updating its policy to include more  detailed criteria for rebaselining; and Homeland Security did not define  acceptable reasons but did require an explanation of the root causes for  cost and schedule variances and the development of new cost and  schedule estimates. In several cases, agencies were unaware of the  detailed rebaselining criteria to be included in their EVM policies. Until  their policies fully meet this key practice, agencies face an increased risk  that their executive managers will make decisions about programs with  incomplete information, and that these programs will continue to overrun  costs and schedules because their underlying problems have not been  identified or addressed.
System surveillance: All eight agencies required ongoing EVM system  surveillance of all programs (and contracts with EVM requirements) to  ensure their continued compliance with industry standards. For example,  Agriculture required its surveillance teams to submit reports—to the  programs and the Chief Information Officer—with documented findings  and recommendations regarding compliance. Furthermore, the agency  also established a schedule to show when EVM surveillance is expected to  take place on each of its programs.
section_title: Agencies’ Key Acquisition Programs Are Using EVM, but Are Not Consistently Implementing Key Practices
paragraphs: Our studies of 16 major system acquisition programs showed that all  agencies are using EVM; however, the extent of that implementation varies  among the programs. Our work on best practices in EVM identified 11 key  practices that are implemented on acquisition programs of leading  organizations. These practices can be organized into three management  areas: establishing a sound EVM system, ensuring reliable data, and using  earned value data to make decisions. Table 3 lists these 11 key EVM  practices by management area.
Of the 16 case study programs, 3 demonstrated a full level of maturity in  all three management areas; 3 had full maturity in two areas; and 4 had  reached full maturity in one area. The remaining 6 programs did not  demonstrate full levels of maturity in any of the management areas;  however, in all but 1 case, they were able to demonstrate partial  capabilities in each of the three areas. Table 4 identifies the 16 case study  programs and summarizes our results for these programs. Following the  table is a summary of the programs’ implementation of each key area of  EVM program management responsibility. Additional details on the 16  case studies are provided in appendix II.
subsections: section_title: Most Programs Did Not Fully Establish Comprehensive EVM Systems
paragraphs: Most programs did not fully implement the key practices needed to  establish comprehensive EVM systems. Of the 16 programs, 3 fully  implemented the practices in this program management area, and 13  partially implemented the practices. The Decennial Response Integration  System, Next Generation Identification, and Surveillance and Broadcast  System programs demonstrated that they had fully implemented the six  practices in this area. For example, our analysis of the Decennial  Response Integration System program schedule showed that activities  were properly sequenced, realistic durations were established, and labor  and material resources were assigned. The Surveillance and Broadcast  System program conducted a detailed integrated baseline review to  validate its performance baseline. It was also the only program to fully  institutionalize EVM at the program level—meaning that it collects  performance data on the contractor and government work efforts—in  order to get a complete view into program status.
Thirteen programs demonstrated that they partially implemented the six  key practices in this area. In most cases, programs had work breakdown  structures that defined work products to an appropriate level of detail and  had identified the personnel responsible for delivering these work  products. However, for all 13 programs, the project schedules contained  issues that undermined the quality of their performance baselines.
Weaknesses in these schedules included the improper sequencing of  activities, such as incomplete or missing linkages between tasks; a lack of  resources assigned to all activities; invalid critical paths (the sequence of  activities that, if delayed, will impact the planned completion date of the  project); and the excessive or unjustified use of constraints, which impairs  the program’s ability to forecast the impact of ongoing delays on future  planned work activities. These weaknesses are of concern because the  schedule serves as the performance baseline against which earned value is  measured. As such, poor schedules undermine the overall quality of a  program’s EVM system. Other key weaknesses included the following  examples:    Nine programs did not adequately determine an objective measure of  earned value and develop the performance baseline—that is, key practices  most appropriately addressed through a comprehensive integrated  baseline review, which none of them fully performed. For example, the Air  and Space Operations Center—Weapon System program conducted an  integrated baseline review in May 2007 to validate one segment of work  contained in the baseline; however, the program had not conducted  subsequent reviews for the remaining work because doing so would  preclude staff from completing their normal work activities. Other reasons  cited by the programs for not performing these reviews included the lack  of a fully defined scope of work or management’s decision to use ongoing  EVM surveillance to satisfy these practices. Without having performed a  comprehensive integrated baseline review, programs have not sufficiently  evaluated the validity of their baseline plan to determine whether all  significant risks contained in the plan have been identified and mitigated,  and that the metrics used to measure the progress made on planned work  elements are appropriate.
Four programs did not define the scope of effort using a work breakdown  structure. For example, the Veterans Health Information Systems and  Technology Architecture—Foundations Modernization program provided  a list of its subprograms; however, it did not define the scope of the  detailed work elements that comprise each subprogram. Without a work  breakdown structure, programs lack a basis for planning the performance  baseline and assigning responsibility for that work, both of which are  necessary to accomplish a program’s objectives.
section_title: Many Programs Did Not Fully Implement Practices to Ensure Data Reliability
paragraphs: Many programs did not fully ensure that their EVM data were reliable. Of  the 16 programs, 7 fully implemented the practices for ensuring the  reliability of the prime contractor and government performance data, and  9 partially implemented the practices. All 7 programs that demonstrated  full implementation conduct monthly reviews of earned value data with  technical engineering staff and other key personnel to ensure that the data  are consistent with actual performance; perform detailed performance  trend analyses to track program progress, cost, and schedule drivers; and  make estimates of cost at completion. Four programs that we had  previously identified as having schedule weaknesses (Farm Program  Modernization; Joint Tactical Radio System—Handheld, Manpack, Small  Form Fit; Juno; and Warfighter Information Network—Tactical) were  aware of these issues and had sufficient controls in place to mitigate them  in order to ensure that the earned value data are reliable.
Nine programs partially implemented the three practices for ensuring that  earned value data are reliable. In all cases, the program had processes in  place to review earned value data (from monthly contractor EVM reports  in all but one case), identify and record cost and schedule variances, and  forecast estimates at completion. However, 5 of these programs did not  adequately analyze EVM performance data and properly record variances  from the performance baseline. For example, 2 programs did not  adequately document justifications for cost and schedule variances,  including root causes, potential impacts, and corrective actions. Other  weaknesses in this area include anomalies in monthly performance  reports, such as negative dollars being spent for work performed, which  impacts the validity of performance data. In addition, 7 of these programs  did not demonstrate that they could adequately execute the work plan and  record costs because, among other things, they were unaware of the  schedule weaknesses we identified and did not have sufficient internal  controls in place to deal with these issues to improve the reliability of the  earned value data. Lastly, 2 of these programs could not adequately  forecast estimates at completion due, in part, to anomalies in the prime  contractor’s EVM reports, in combination with the weaknesses contained  in the project schedule.
section_title: Most Programs Used Earned Value Data for Decision-making Purposes
paragraphs: Programs were uneven in their use of earned value data to make decisions.  Of the 16 programs, 9 fully implemented the practices for using earned  value data for decision making, 6 partially implemented them, and 1 did  not implement them. Among the 9 fully implemented programs, both the  Automated Commercial Environment and Juno programs integrated their  EVM and risk management processes to support the program manager in  making better decisions. The Automated Commercial Environment  program actively recorded risks associated with major variances from the  EVM reports in the program’s risk register. Juno further used the earned  value data to analyze threats against remaining management reserve and  to estimate the cost impact of these threats.
Six programs demonstrated limited capabilities in using earned value data  for making decisions. In most cases, these programs included earned value  performance trend data in monthly program management review briefings.  However, the majority had processes for taking management action to  address the cost and schedule drivers causing poor trends that were ad  hoc and separate from the programs’ risk management processes—and, in  most cases, the risks and issues found in the EVM reports did not  correspond to the risks contained in the program risk registers. In  addition, 4 of these programs were not able to adequately update the  performance baseline as changes occurred because, in many cases, the  original baseline was not appropriately validated. For example, the Mars  Science Laboratory program just recently updated its performance  baseline as part of a recent replan effort. However, without validating the  original and current baselines with a project-level integrated baseline  review, it is unclear whether the changes to the baseline were reasonable,  and whether the risks assumed in the baseline have been identified and  appropriately mitigated.
One program (Veterans Health Information Systems and Technology  Architecture—Foundations Modernization) was not using earned value  data for decision making. Specifically, the program did not actively  manage earned value performance trends, nor were these data  incorporated into programwide management reviews.
section_title: Inconsistent Implementation Is Due in Part to Weaknesses in Policy and Lack of Enforcement
paragraphs: The inconsistent application of EVM across the investments exists in part  because of the weaknesses we previously identified in the eight agencies’  policies, as well as a lack of enforcement of the EVM policy components  already in place. For example, deficiencies in all three management areas  can be attributed, in part, to a lack of comprehensive EVM training  requirements—which was a policy component that most agencies did not  fully address. The only 3 programs that had fully implemented all key EVM  practices either had comprehensive training requirements in their agency  EVM policy or enforced rigorous training requirements beyond that for  which the policy called. Most of the remaining programs met the minimum  requirements of their agencies’ policies. However, all programs that had  attained full maturity in two management areas had also implemented  more stringent training requirements, although none could match the  efforts made on the other 3 programs. Without making this training a  comprehensive requirement, these agencies are at risk that their major  system acquisition programs will continue to have management and  technical staff who lack the skills to fully implement key EVM practices.
Our case study analysis also highlighted multiple areas in which programs  were not in compliance with their agencies’ established EVM policies. This  is an indication that agencies are not adequately enforcing program  compliance. These policy areas include requiring EVM compliance at the  start of the program, validating the baseline with an integrated baseline  review, and conducting ongoing EVM surveillance.
Until key EVM practices are fully implemented, selected programs face an  increased risk that program managers cannot effectively optimize EVM as  a management tool to mitigate and reverse poor cost and schedule  performance trends.
section_title: Earned Value Data Show Trends of Cost Overruns and Schedule Slippages on Most Programs
paragraphs: Earned value data trends of the 16 case study programs indicate that most  are currently experiencing cost overruns and schedule slippages, and,  based on our analysis, it is likely that when these programs are completed,  the total cost overrun will be about $3 billion. To date, these programs,  collectively, have already overrun their original life-cycle cost estimates by  almost $2 billion (see table 5).
Taking the current earned value performance into account, our analysis  of the 16 case study programs indicated that most are experiencing  shortfalls against their currently planned cost and schedule targets.  Specifically, earned value performance data over a 12-month period  showed that the 16 programs combined have exceeded their cost targets  by $275 million. During that period, they also experienced schedule  variances and were unable to accomplish almost $93 million worth of  planned work. In most cases, the negative cost and schedule performance  trends were attributed to ongoing technical issues in the development or  testing of system components.
Furthermore, our projections of future estimated costs at completion  based on our analysis of current contractor performance trends indicate  that these programs will most likely continue to experience cost overruns  to completion, totaling almost $1 billion. In contrast, the programs’  contractors estimate the cost overruns at completion will be  approximately $469.7 million. These estimates are based on the  contractors’ assumption that their efficiency in completing the remaining  work will significantly improve over what has been done to date.  Furthermore, it should be noted that in 4 cases, the contractor-estimated  overrun is smaller than the cost variances they have already  accumulated—which is an indication that these estimates are aggressively  optimistic.
With the inclusion of the overruns already incurred to date, the total  increase in life-cycle costs will be about $3 billion. Our analysis is  presented in table 6. Additional details on the 16 case studies are provided  in appendix II.
Eleven programs are expected to incur a cost overrun at contract  completion. In particular, two programs (i.e., the James Webb Space  Telescope and Veterans Health Information Systems and Technology  Architecture—Foundations Modernization programs) will likely  experience a combined overrun of $798.7 million, which accounts for  about 80 percent of our total projection.
With timely and effective action taken by program and executive  management, it is possible to reverse negative performance trends so that  the projected cost overruns at completion may be reduced. To get such  results, management at all levels could be strengthened, including  contractor management, program office management, and executive-level  management. For example, programs could strengthen program office  controls and contractor oversight by obtaining earned value data weekly  (instead of monthly) so that they can make decisions with immediate and  greater impact. Additionally, key risks could be elevated to the program  level and, if necessary, to the executive level to ensure that appropriate  mitigation plans are in place and that they are tracked to closure.
section_title: Conclusions
paragraphs: Key agencies have taken a number of important steps to improve the  management of major acquisitions through the implementation of EVM.  Specifically, the agencies have established EVM policies and require their  major system acquisition programs to use EVM. However, none of the  eight agencies that we reviewed have comprehensive EVM policies. Most  of these policies omit or lack sufficient guidance on the type of work  structure needed to effectively use EVM data and on the training  requirements for all relevant personnel. Without comprehensive policies, it  will be difficult for the agencies to gain the full benefits of EVM.
Few of our 16 case study programs had fully implemented EVM  capabilities, raising concerns that programs cannot efficiently produce  reliable estimates of cost at completion. Many of these weaknesses found  on these programs can be traced back to inadequate agency EVM policies  and raise questions concerning the agencies’ enforcement of the policies  already established, including the completion of the integrated baseline  reviews and system surveillance. Until agencies expand and enforce their  EVM policies, it will be difficult for them to optimize the effectiveness of  this management tool, and they will face an increased risk that managers  are not getting the information they need to effectively manage the  programs.
In addition to concerns about their implementation of EVM, the programs’  earned value data show trends toward cost overruns that are likely to  collectively total about $3 billion. Without timely and aggressive  management action, this projected overrun will be realized, resulting in  the expenditure of over $1 billion more than currently planned.
section_title: Recommendations for Executive Action
paragraphs: To address the weaknesses identified in agencies’ policies and practices in  using EVM, we are making recommendations to the eight major agencies  included in this review. Specifically, we recommend that the following  three actions be taken by the Secretaries of the Departments of  Agriculture, Commerce, Defense, Homeland Security, Justice,  Transportation, and Veterans Affairs and the Administrator of the National  Aeronautics and Space Administration:    modify policies governing EVM to ensure that they address the  weaknesses that we identified, taking into consideration the criteria used  in this report;   direct key system acquisition programs to implement the EVM practices  that address the detailed weaknesses that we identified in appendix II,  taking into consideration the criteria used in this report; and   direct key system acquisition programs to take action to reverse current  negative performance trends, as shown in the earned value data, to  mitigate the potential cost and schedule overruns.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided the selected eight agencies with a draft of our report for  review and comment. The Department of Homeland Security responded  that it had no comments. The remaining seven agencies generally agreed  with our results and recommendations. Agencies also provided technical  comments, which we incorporated in the report as appropriate.
The comments of the agencies are summarized in the following text:  In e-mail comments on a draft of the report, officials from the U.S.  Department of Agriculture’s Office of the Chief Information Officer stated  that the department has begun to address the weaknesses in its EVM  policy identified in the report.
In written comments on a draft of the report, the Secretary of Commerce  stated that, regarding the second and third recommendations, the  Department of Commerce was pleased that the Decennial Response  Integration System was found to have fully implemented all 11 key EVM  practices, and that the Field Data Collection Automation program fully  implemented six key practices. The department added that its recent  actions on the Field Data Collection Automation program should move  this program to full compliance with the key EVM practices. Furthermore,  regarding the first recommendation, the Secretary stated that while the  department understands and appreciates the value of standardized work  breakdown structures, it maintained that the development of these work  structures should take place at the department’s operating units (e.g.,  Census Bureau), given the wide diversity of missions and project  complexity among these units. As noted in our report, we agree that  agencies could develop standard work structures based on the kinds of  work being performed by the various component agencies. Therefore, we  support these efforts described by the department because they are  generally consistent with the intent of our recommendation. Commerce’s  comments are printed in appendix III.
In written comments on a draft of the report, the Department of Defense’s  Director of Defense Procurement and Acquisition Policy stated that the  department concurred with our recommendations. Among other things,  DOD stated that it is essential to maintain the appropriate oversight of  acquisition programs, including the use of EVM data to understand  program status and anticipate potential problems. DOD’s comments are  printed in appendix IV.
In written comments on a draft of the report, the Department of Justice’s  Assistant Attorney General for Administration stated that, after discussion  with our office, it was agreed that the second recommendation, related to  implementing EVM practices that address identified weakness, was  inadvertently directed to the department, and that no response was  necessary. We agreed because the case study program reviewed fully met  all key EVM practices. The department concurred with the two remaining  recommendations related to modifying EVM policies and reversing  negative performance trends. Furthermore, the Assistant Attorney General  noted that Justice had begun to take steps to improve its use of EVM, such  as modifying its policy to require EVM training for all personnel with  investment oversight and program management responsibilities. Justice’s  comments are printed in appendix V.
In written comments on a draft of the report, the National Aeronautics and  Space Administration’s Deputy Administrator stated that the agency  concurred with two recommendations and partially concurred with one  recommendation. In particular, the Deputy Administrator agreed that  opportunities exist for improving the implementation of EVM, but stated  that NASA classifies the projects included in the scope of the audit as space  flight projects (not as IT-specific projects), which affects the applicability of  the agency’s EVM policies and guidance that were reviewed. We recognize  that different classifications of IT exist; however, consistent with other  programs included in the audit, the selected NASA projects integrate and  rely on various elements of IT. As such, we reviewed both the agency’s  space flight and IT-specific guidance. Furthermore, the agency partially  concurred with one recommendation because it stated that efforts were  either under way or planned that will address the weaknesses we identified.  We support the efforts that NASA described in its comments because they  are generally consistent with the intent of our recommendation. NASA’s  comments are printed in appendix VI.
In e-mail comments on a draft of the report, the Department of  Transportation’s Director of Audit Relations stated that the department is  taking immediate steps to modify its policies governing EVM, taking into  consideration the criteria used in the draft report.
In written comments on a draft of the report, the Secretary of Veterans  Affairs stated that the Department of Veterans Affairs generally agreed  with our conclusions and concurred with our recommendations.  Furthermore, the Secretary stated that Veterans Affairs has initiatives  under way to address the weaknesses identified in the report. Veterans  Affairs’ comments are printed in appendix VII.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees; the Secretaries of the Departments of  Agriculture, Commerce, Defense, Homeland Security, Justice,  Transportation, and Veterans Affairs; the Administrator of the National  Aeronautics and Space Administration; and other interested parties. In  addition, the report will be available at no charge on our Web site at  http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix VIII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess whether key departments and agencies  have appropriately established earned value management (EVM) policies,  (2) determine whether these agencies are adequately using earned value  techniques to manage key system acquisitions, and (3) evaluate the earned  value data of these selected investments to determine their cost and  schedule performances.
For this governmentwide review, we assessed eight agencies and 16  investments. We initially identified the 10 agencies with the highest  amount of spending for information technology (IT) development,  modernization, and enhancement work as reported in the Office of  Management and Budget’s (OMB) Fiscal Year 2009 Exhibit 53. These  agencies were the Departments of Agriculture, Commerce, Defense,  Health and Human Services, Homeland Security, Justice, Transportation,  the Treasury, and Veterans Affairs and the National Aeronautics and Space  Administration. We excluded Treasury from our selection because we  recently performed an extensive review of EVM at that agency. We also  subsequently removed Health and Human Services from our selection  because the agency did not have investments in system acquisition that  met our dollar threshold (as defined in the following text). The resulting  eight agencies also made up about 75 percent of the government’s planned  IT spending for fiscal year 2009.
To ensure that we examined significant investments, we chose from  investments (related to system acquisition) that were expected to receive  development, modernization, and enhancement funding in fiscal year 2009  in excess of $90 million. We limited the number of selected investments to  a maximum of 3 per agency. For agencies with more than 3 investments  that met our threshold, we selected the top 3 investments with the highest  planned spending. For agencies with 3 or fewer such investments, we  chose all of the investments meeting our dollar threshold. Lastly, we  excluded investments with related EVM work already under way at GAO.
To assess whether key agencies have appropriately established EVM  policies, we analyzed agency policies and guidance for EVM. Specifically,  we compared these policies and guidance documents with both OMB’s  requirements and key best practices recognized within the federal  government and industry for the implementation of EVM. These best  practices are contained in the GAO cost guide. We also interviewed key  agency officials to obtain information on their ongoing and future EVM  plans.
To determine whether these agencies are adequately using earned value  techniques to manage key system acquisitions, we analyzed program  documentation, including project work breakdown structures, project  schedules, integrated baseline review briefings, risk registers, and monthly  management briefings for the 16 selected investments. Specifically, we  compared program documentation with EVM and scheduling best  practices as identified in the cost guide. We determined whether the  program implemented, partially implemented, or did not implement each  of the 11 practices. We also interviewed program officials (and observed  key program status review meetings) to obtain clarification on how EVM  practices are implemented and how the data are used for decision-making  purposes.
To evaluate the earned value data of the selected investments to determine  their cost and schedule performances, we analyzed the earned value data  contained in contractor EVM performance reports obtained from the  programs. To perform this analysis, we compared the cost of work  completed with budgeted costs for scheduled work for a 12-month period  to show trends in cost and schedule performances. We also used data from  these reports to estimate the likely costs at completion through  established earned value formulas. This resulted in three different values,  with the middle value being the most likely. To assess the reliability of the  cost data, we compared it with other available supporting documents  (including OMB and agency financial reports); electronically tested the  data to identify obvious problems with completeness or accuracy; and  interviewed agency and program officials about the data. For the purposes  of this report, we determined that the cost data were sufficiently reliable.  We did not test the adequacy of the agency or contractor cost-accounting  systems. Our evaluation of these cost data was based on what we were  told by the agency and the information they could provide.
We conducted this performance audit from February to October 2009 at  the agencies’ offices in the Washington, D.C., metropolitan area; Fort  Monmouth, New Jersey; Jet Propulsion Lab, Pasadena, California;  Hanscom Air Force Base, Massachusetts; and Naval Base San Diego,  California. Our work was done in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Case Studies of Selected Programs’ Implementation of Earned Value Management
paragraphs: We conducted case studies of 16 major system acquisition programs (see  table 7). For each of these programs, the remaining sections of this  appendix provide the following: a brief description of the program,  including a graphic illustration of the investment’s life cycle; an  assessment of the program’s implementation of the 11 key EVM practices;  and an analysis of the program’s recent earned value (EV) data and trends.  These data and trends are often described in terms of cost and schedule  variances. Cost variances compare the earned value of the completed  work with the actual cost of the work performed. Schedule variances are  also measured in dollars, but they compare the earned value of the  completed work with the value of the work that was expected to be  completed. Positive variances are good—they indicate that activities are  costing less than expected or are completed ahead of schedule. Negative  variances are bad—they indicate activities are costing more than expected  or are falling behind schedule.
The following information describes the key that we used in tables 8  through 23 to convey the results of our assessment of the 16 case study  programs’ implementation of the 11 EVM practices.
The program fully implemented all EVM  practices in this program management area.
The program partially implemented the EVM  practices in this program management area.
The program did not implement the EVM  practices in this program management area.
subsections: section_title: Farm Program Modernization
paragraphs: The Farm Program Modernization (MIDAS) program is intended to  address the long-term needs in delivering farm benefit programs via  business process reengineering and implementation of a commercial off- the-shelf enterprise resource planning solution. MIDAS is an initiative of  the Farm Service Agency, which is responsible for administering 35 farm  benefit programs. To support these programs, the agency uses two  primary systems—a distributed network of legacy computers and a  centralized Web farm (to store customer data and host Web-based  applications)—both of which have shortcomings. While MIDAS is to  replace these computers, it is also intended to provide new applications  and redesigned business processes. The Web farm is expected to remain in  operation in a supporting role for the program. Currently, MIDAS is in the  initiation phase of its life cycle and plans to award the system integration  contract in the first quarter of fiscal year 2010.
MIDAS fully met 6 of the 11 key practices for implementing EVM and  partially met 5 practices. Specifically, a key weakness in the EVM system  is the lack of a comprehensive integrated baseline review. Instead, MIDAS  focused solely on evaluating the program’s compliance with industry  standards and chose not to validate the quality of the baseline. Program  officials stated that they plan to conduct a full review to address the risks  and realism of the baseline after the prime contract has been awarded.  Furthermore, while the MIDAS schedule is generally sound, resources  were not assigned to all activities, and the critical path (the longest  duration path through the sequenced list of key activities) could not be  identified because the current schedule ends in September 2009. Finally,  MIDAS met all key practices associated with data reliability, such as  executing the work plan and recording costs, as well as all key practices  for decision making.
section_title: Decennial Response Integration System
paragraphs: The Decennial Response Integration System (DRIS) is to be used during  the 2010 Census for collecting and integrating census responses from all  sources, including forms and telephone interviews. The system is to  improve accuracy and timeliness by standardizing the response data and  providing the data to other Census Bureau systems for analysis and  processing. Among other things, DRIS is expected to process census data  provided by respondents via census forms, telephone agents, and  enumerators; assist the public via telephone; and monitor the quality and  status of data capture operations. The DRIS program’s estimated life-cycle  costs have increased by $372 million, which is mostly due to increases in  both paper and telephone workloads. For example, the paper workload  increased due to an April 2008 redesign of the 2010 Census that reverted  planned automated operations to paper-based processes and requires  DRIS to process an additional estimated 40 million paper forms.
DRIS fully implemented all 11 of the key EVM practices necessary to  manage its system acquisition program. Specifically, the program  implemented all practices for establishing a comprehensive EVM system,  such as defining the scope of work and scheduling the work. The  program’s schedule appropriately captured and sequenced key activities  and assigned realistic resources to all key activities. Furthermore, the  DRIS team ensured that the resulting EVM data were appropriately  verified and validated for reliability by analyzing performance data to  identify the magnitude and effect of problems causing key variances,  tracking related risks in the program’s risks register, and performing  quality checks of the schedule and critical path. Lastly, the DRIS program  management team conducted rigorous reviews of EV performance on a  monthly basis and took the appropriate management actions to mitigate  risks.
section_title: Field Data Collection Automation
paragraphs: The Field Data Collection Automation (FDCA) program is intended to  provide automation support for the 2010 Census field data collection  operations. The program includes the development of handheld computers  for identifying and correcting addresses for all known living quarters in the  United States (known as address canvassing) and the systems, equipment,  and infrastructure that field staff will use to collect data. FDCA handheld  computers were originally to be used for other census field operations,  such as following up with nonrespondents through personal interviews.  However, in April 2008, due to problems identified during testing and cost  overruns and schedule slippages in the FDCA program, the Secretary of  Commerce announced a redesign of the 2010 Census, and rebaselined  FDCA in October 2008. As a result, FDCA’s life-cycle costs have increased  from an estimated $596 million to $801 million, a $205 million increase.  Furthermore, the responsibility for the design, development, and testing of  IT systems for other key field operations was moved from the FDCA  contractor to the Census Bureau.
FDCA fully met 6 of the 11 key practices for implementing EVM and  partially met 5 others. Specifically, the program fully met most practices  for establishing a comprehensive EVM system, such as defining the scope  of the work effort; however, it only partially met the practice for  scheduling the work. Specifically, the program schedule contained  weaknesses, including key milestones with fixed completion dates—which  hampers the program’s ability to see the impact of delays experienced on  open tasks on successor tasks. As such, the FDCA program cannot use the  schedule as an active management tool. Furthermore, anomalies in the  prime contractor’s EVM reports, combined with weaknesses in the master  schedule, affect FDCA’s ability to execute the work plan, analyze  variances, and make reliable estimates of cost at completion. Lastly, cost  and schedule drivers identified in EVM reports were not fully consistent  with the program’s risk register, which prevents the program from taking  the appropriate management action to mitigate risks and effectively using  EV data for decisions.
section_title: Air and Space Operations Center— Weapon System
paragraphs: The Air and Space Operations Center—Weapon System (AOC) is the air  and space operations planning, execution, and assessment system for the  Joint Force Air Component Commander. According to the agency, there  are currently 11 AOCs located around the world, each aligned to the  Combatant Commands of the Unified Command Plan, with additional  support units for training, help desk, testing, and contingency manpower  augmentation. Each AOC is designed to enable commanders to exercise  command and control of air, space, information operations, and combat  support forces to achieve the objectives of the joint force commander and  combatant commander in joint and coalition military operations. As such,  the AOC system is intended as the planning and execution engine of any  air campaign.
AOC fully met 7 of the 11 key practices and partially met 4 others. AOC  applied EVM at the contract level and has a capable government team that  has made it an integral part of project management. AOC performed  detailed analyses of the EV data and reviews the data with engineering  staff to ensure that the appropriate metrics have been applied for accurate  reporting. AOC has also integrated EVM with its risk management  processes to ensure that resources are applied to watch or mitigate risks  associated with the cost and schedule drivers reported in the EVM reports.  Weaknesses found in AOC’s EVM processes relate to the development and  validation of the contractor baseline. In particular, AOC has not performed  an integrated baseline review for all work that is currently on contract.  The master schedule also contained issues, such as a high number of  converging tasks and out-of-sequence tasks, that hamper AOC’s ability to  determine the start dates of future tasks. Taken together, these issues  undermine the reliability of the schedule as a baseline to measure EV  performance.
section_title: Joint Tactical Radio System—Handheld, Manpack, Small Form Fit
paragraphs: The Joint Tactical Radio System (JTRS) program is developing software- defined radios that are expected to interoperate with existing radios and  increase communications and networking capabilities. The JTRS- Handheld, Manpack, Small Form Fit (HMS) product office, within the  JTRS Ground Domain program office, is developing handheld, manpack,  and small form fit radios. In 2006, the program was restructured to include  two concurrent phases of development. Phase I includes select small form  fit radios, while Phase II includes small form fit radios with enhanced  security as well as handheld and manpack variants. Subsequent to the  program’s restructure, the department updated its migration strategy for  replacing legacy radios with new tactical radios. As such, the total planned  quantity of JTRS-HMS radios was reduced from an original baseline of  328,514—established in May 2004—to 95,551. As a result, the total life- cycle cost of the JTRS-HMS program was reduced from an estimated $19.2  billion to $11.6 billion, a $7.6 billion decrease.
JTRS-HMS fully met 10 of the 11 key practices and partially met 1 practice.  Specifically, JTRS-HMS implemented most practices for establishing a  comprehensive EVM system, such as performing rigorous reviews to  validate the baseline; however, the current schedule contained some  weaknesses, such as out-of-sequence logic and activities without  resources assigned. Program officials were aware of these issues and  attributed them to weaknesses in subcontractor schedules that are  integrated on a monthly basis. The JTRS-HMS program fully met practices  for ensuring that the resulting EV data were appropriately verified and  validated for reliability and demonstrated that the program management  team was using these data for decision-making purposes.
section_title: Warfighter Information Network—Tactical
paragraphs: The Warfighter Information Network—Tactical (WIN-T) program is  designed to be the Army’s high-speed and high-capacity backbone  communications network. The program connects Department of the Army  units with higher levels of command and provides the Army’s tactical  portion of the Global Information Grid—a Department of Defense  initiative aimed at building a secure network and set of information  capabilities modeled after the Internet. WIN-T was restructured in June  2007 following a unit cost increase above the critical cost growth  threshold (known as a Nunn-McCurdy breach). As a result of the  restructuring, it was determined that WIN-T would be fielded in four  increments. The third increment is expected to provide the Army with a  full networking on-the-move capability and fully support the Army’s  Future Combat Systems. In May 2009, the Increment 3 program baseline  was approved, and the life-cycle cost for the program was estimated at  $38.2 billion. Our assessment of EVM practices and EV data was  performed on WIN-T Increment 3.
WIN-T fully met 7 of the 11 key practices for implementing EVM, partially  met 1 practice, and did not meet 3 practices. Specifically, WIN-T only  partially met the practices for establishing a comprehensive EVM system.  The schedule contained weaknesses, including fixed completion dates— which prevented the schedule from showing the impact of delays  experienced on open or successor tasks or the expected completion dates  of key activities. Furthermore, WIN-T has not conducted an integrated  baseline review on the current scope of work since rebaselining the prime  contract in December 2007. According to program officials, this review has  not been conducted because they have not yet finalized the contract.  However, as of August 2009, it has been 20 months since work began,  which increases the risk that the program has not been measuring  progress against a reasonable baseline. Without conducting this review to  validate the performance baseline, the baseline cannot be adequately  updated as changes occur, and EV data cannot be used effectively for  decision-making purposes.
section_title: Automated Commercial Environment
paragraphs: The Automated Commercial Environment (ACE) program is the  commercial trade processing system being developed by the U.S. Customs  and Border Protection to facilitate trade while strengthening border  security. The program is to provide trade compliance and border security  staff with the right information at the right time, while minimizing  administrative burden. Deployed in phases, ACE is expected to be  expanded to provide cargo processing capabilities across all modes of  transportation and intended to replace existing systems with a single,  multimodal manifest system for land, air, rail, and sea cargo. Ultimately,  ACE is expected to become the central data collection system for the  federal agencies that, by law, require international trade data, and should  deliver these capabilities in a secure, paper-free, Web-enabled  environment. As a result of poorly managed requirements, the total life- cycle development cost of the ACE program increased from an estimated  $1.5 billion to $2.2 billion—a $700 million increase.
ACE fully met 9 of the 11 key practices for implementing EVM and  partially met the remaining 2 practices. Specifically, ACE fully met 5 of 6  practices for establishing a comprehensive EVM system, such as defining  the scope of the work effort and developing the performance baseline, but  partially met the practice for scheduling the work, in part, because  resources were not assigned to all activities in the master schedule. ACE  fully met 2 practices for ensuring that the data resulting from the EVM  system were reliable, such as adequately analyzing EV performance data,  but could not fully execute the work plan because of the weaknesses  found in the schedule. Lastly, ACE demonstrated that the program  management team was basing decisions on EVM data.
It should be noted that the ACE program is being defined incrementally— whereby the performance baseline is continuously updated as task orders  for new work are issued. As such, the use of EVM to determine the true  progress made and to project reliable final costs at completion is limited.
section_title: Integrated Deepwater System—Common Operational Picture
paragraphs: The Integrated Deepwater System is a 25-year, $24 billion major  acquisition program to recapitalize the U.S. Coast Guard’s aging fleet of  boats, airplanes, and helicopters, ensuring that all work together through a  modern, capable communications system. This initiative is designed to  enhance maritime domain awareness and enable the Coast Guard to meet  its post-September 11 mission requirements. The program is composed of  15 major acquisition projects, including the Common Operational Picture  (COP) program.
Deepwater COP is to provide relevant, real-time operational intelligence  and surveillance data to human capital managers, allowing them to direct  and monitor all assigned forces and first responders. This is expected to  allow commanders to distribute critical information to federal, state, and  local agencies quickly; reduce duplication; enable earlier alerting; and  enhance maritime awareness.
Deepwater COP fully met 7 of the 11 key practices and partially met 4  others. Specifically, COP fully met 5 of the 6 practices for establishing a  comprehensive EVM system, such as adequately defining all major  elements of the work breakdown structure and developing the  performance baseline. However, the program’s master schedule contained  weaknesses, such as a large number of concurrent tasks and activities  without resources assigned. Officials were aware of some, but not all, of  the weaknesses in the schedule and had controls in place to mitigate the  weakness they were aware of in order to improve the reliability of the  resulting EV data. Lastly, COP was unable to fully meet 1 of the practices  for using EV data for management decisions because it could not  demonstrate that cost and schedule drivers impacting EV performance  were linked to its risk management processes.
section_title: Western Hemisphere Travel Initiative
paragraphs: The Western Hemisphere Travel Initiative (WHTI) program made  modifications to vehicle processing lanes at ports of entry on the nation’s  northern and southern borders. WHTI is designed to allow U.S. Customs  and Border Protection to effectively address new requirements imposed  by the Intelligence Reform and Terrorism Prevention Act of 2004  (completing these requirements by June 1, 2009). WHTI development was  completed and its implementation addressed the 39 highest volume ports  of entry, which support 95 percent of land border traffic. The initiative  requires travelers to present a passport or other authorized travel  document that denotes identity and citizenship when entering the United  States.
WHTI fully met 6 of the 11 key practices for implementing EVM and  partially met the remaining 5 practices. Specifically, weaknesses identified  in validating the performance baseline and scheduling the work limited the  program’s ability to establish a comprehensive EVM system. Although the  program held an integrated baseline review to validate the baseline in  March 2008, the review did not cover many key aspects, such as  identifying corrective actions needed to mitigate program risks.  Furthermore, the master schedule contained deficiencies, such as  activities that were out of sequence or lacking dependencies. While  program officials described their use of processes for ensuring the  reliability of the EVM system’s data, such as capturing significant cost and  schedule drivers in the risk register, the provided documentation did not  corroborate what we were told. When combined, these weaknesses  preclude the program from effectively making decisions about the  program based on EV data.
section_title: Next Generation Identification
paragraphs: The Next Generation Identification (NGI) program is designed to support  the Federal Bureau of Investigation’s mission to reduce terrorist and  criminal activities by providing timely, relevant criminal justice  information to the law enforcement community. Today, the bureau  operates and maintains one of the largest repositories of biometric- supported criminal history records in the world. The electronic  identification and criminal history services support more than 82,000  criminal justice agencies, authorized civil agencies, and international  organizations. NGI is intended to ensure that the bureau’s biometric  systems are able to seamlessly share data that are complete, accurate,  current, and timely. To accomplish this, the current system will be  replaced or upgraded with new functionalities and state-of-the-art  equipment. NGI is expected to be scaleable to accommodate five times the  current workload volume with no increase in support manpower and will  be flexible to respond to changing requirements.
NGI fully implemented all 11 key EVM practices. Specifically, the program  implemented all practices for establishing a comprehensive EVM system,  such as defining the scope of work and scheduling the work. For example,  the schedule properly captured key activities, established reasonable  durations, and established a sound critical path, all of which contribute to  establishing a reliable baseline that performance can be measured against.  Furthermore, the NGI team ensured that the resulting EV data were  appropriately verified and validated for reliability by, for example,  integrating the analysis of cost and schedule variances with the program’s  risk register to mitigate emerging and existing risks associated with key  drivers causing major variances. In addition, the program’s risk register  includes cost and schedule impacts for every risk and links to the  management reserve process. Lastly, NGI demonstrated that it is using EV  data to make decisions by performing continuous quality checks of the  schedule, reviewing open risks and opportunities, and reviewing EV data  in weekly management reports.
section_title: James Webb Space Telescope
paragraphs: The James Webb Space Telescope (JWST) is designed to be the scientific  successor to the Hubble Space Telescope and expected to be the premier  observatory of the next decade. It is intended to seek to study and answer  fundamental astrophysical questions, ranging from the formation and  structure of the Universe to the origin of planetary systems and the origins  of life. The telescope is an international collaboration of the National  Aeronautics and Space Administration (NASA), the Canadian Space  Agency, and the European Space Agency. JWST required the development  of several new technologies, including a folding segmented primary mirror  that will unfold after launch and a cryocooler for cooling midinfrared  detectors to 7 degrees Kelvin.
JWST fully met 4 of the 11 key practices and partially met 7 practices. The  project only partially met practices for establishing a comprehensive EVM  system because of weaknesses in the work breakdown structure, in which  the prime contractor has not fully defined the scope of each work element.  In addition, the project only partially met the practice for scheduling work  because of weaknesses resulting from manual integration of  approximately 30 schedules, although officials did explain some  mitigations for this risk. We also found deficiencies in the lower-level  schedules, such as missing linkages between tasks, resources not being  assigned, and excessively high durations. Furthermore, JWST only  partially implemented practices to ensure that the data resulting from the  EVM system are reliable, due, in part, to variance analysis reports being  done quarterly (instead of monthly), which limits the project’s ability to  analyze and respond to cost and schedule variances in a timely manner.  When combined, these weaknesses preclude the program from effectively  making decisions about the program based on EV data.
section_title: Juno
paragraphs: Juno is part of the New Frontiers Program. The overarching scientific goal  of the Juno mission is to improve our understanding of the origin and  evolution of Jupiter. As the archetype of giant planets, Jupiter may provide  knowledge that will improve our understanding of both the origin of our  solar system and the planetary systems being discovered around other  stars. The Juno project is expected to use a solar-powered spacecraft to  make global maps of the gravity, magnetic fields, and atmospheric  composition of Jupiter. The spacecraft is to make 33 orbits of Jupiter to  sample the planet’s full range of latitudes and longitudes.
Juno fully met 8 of the 11 key practices for implementing EVM and  partially met 3 practices. Specifically, the project fully met 3 practices for  establishing a comprehensive EVM system, but only partially met the  practices for scheduling the work, determining the objective measure of  earned value, and establishing the performance baseline. Juno was unable  to fully meet these practices because the project’s master schedule  contained issues with the sequencing of work activities and lacked a  comprehensive integrated baseline review. Although an integrated  baseline review was conducted for a major contract in February 2009, the  program did not validate the baseline, scope of work to be performed, or  key risks and mitigation plans for the Juno project as a whole, which  increases the risk that the project is measuring performance against an  unreasonable baseline. Juno fully implemented all 3 practices associated  with data reliability and the 2 practices associated with using EV data for  decision-making purposes.
section_title: Mars Science Laboratory
paragraphs: The Mars Science Laboratory (MSL) is part of the Mars Exploration  Program. The program seeks to understand whether Mars was, is, or can  be a habitable world. To answer this question, the MSL project is expected  to investigate how geologic, climatic, and other processes have worked to  shape Mars and its environment over time, as well as how they interact  today. To accomplish this, the MSL project plans to place a mobile science  laboratory on the surface of Mars to quantitatively assess a local site as a  potential habitat for life, past or present. The project is considered one of  NASA’s flagship projects and designed to be the most advanced rover ever  sent to explore the surface of Mars. Due to technical issues identified  during the development of key components, the MSL launch date has  recently slipped 2 years—from September 2009 to October 2011, and the  project’s life-cycle cost estimate has increased from about $1.63 billion to  $2.29 billion, a $652 million increase.
MSL fully met 5 of the 11 key practices and partially met 6 others.  Specifically, MSL fully met 3 practices for establishing a comprehensive  EVM system, but only partially met 3 others because of weaknesses in the  sequencing of all activities in the schedule and the lack of an integrated  baseline review to validate the baseline and assess the achievability of the  plan. While the project has taken steps to mitigate the latter weakness by  requiring work agreements that document, among other things, the  objective value of work and related risks for planned work packages, this  is not a comprehensive review of the project’s baseline. Furthermore, MSL  only partially implemented practices associated with data reliability  because its analysis of cost and schedule variances did not include the  root causes for variances and corrective actions, which prevents the  project from tracking and mitigating related risks. Lastly, without an initial  validation of the performance baseline, the baseline cannot be  appropriately updated to reflect program changes, thereby limiting the use  of EV data for management decisions.
section_title: En Route Automation Modernization
paragraphs: The En Route Automation Modernization (ERAM) program is to replace  existing software and hardware in the air traffic control automation  computer system and its backup system, the Direct Radar Channel, and  other associated interfaces, communications, and support infrastructure at  en route centers across the country. This is a critical effort because ERAM  is expected to upgrade hardware and software for facilities that control  high-altitude air traffic. ERAM consists of two major components. One  component has been fully deployed and is currently in operation at  facilities across the country. The other component is scheduled for  deployment through fiscal year 2011.
ERAM fully met 7 of the 11 key practices and partially met 4 others. ERAM  applies EVM at the contract level and incorporates EV data into its overall  management of the program. However, ERAM did not perform a  comprehensive review of the baseline when the contract was finalized, or  take similar actions to validate the baseline and ensure that the appropriate  EV metrics had been applied. While ERAM does perform limited checks of  the contractor schedule, our analysis showed some issues with the  sequencing of activities and the use of constraints that may undermine the  reliability of the schedule as a baseline to measure performance.
However, it should be noted that the EV data are not a reflection of the  total ERAM program. The government is also responsible for acquisition  work—to which EVM is not being applied. Our analysis of the master  schedule showed that ERAM would be unable to meet four major  upcoming initial operating capability milestones due to issues associated  with government work activities. Program officials noted that these  milestones have since been pushed out. Since EVM is not applied at the  program level, it is unclear whether these delays will impact overall cost.
section_title: Surveillance and Broadcast System
paragraphs: The Surveillance and Broadcast System (SBS) is to provide new  surveillance solutions that employ technology using avionics and ground  stations for improved accuracy and update rates and to provide shared  situational awareness (including visual updates of traffic, weather, and  flight notices) between pilots and air traffic control. These technologies  are considered critical to achieving the Federal Aviation Administration’s  strategic goals of decreasing the rate of accidents and incursions,  improving the efficiency of air traffic, and reducing congestion.
SBS fully implemented all 11 key EVM practices. Specifically, SBS has  institutionalized EVM at the program level—meaning that it collects and  manages performance data on the contractor and government work  efforts—in order to get a comprehensive view into program status. As part  of this initiative, SBS performed detailed validation reviews of the  contractor and program baselines; issued various process rules on  resource planning, EV metrics, and data analysis; and collected  government timecard data in order to ensure consistent EV application. In  addition, the program management team conducted rigorous reviews of  EV performance with the SBS program manger and the program’s internal  management review board on a monthly basis. Our analysis of the SBS  master schedule showed that it was developed in accordance with  scheduling best practices. For example, the schedule was properly  sequenced, and the resources were assigned. Furthermore, SBS briefed  the program manager monthly on the quality of the schedule to identify,  for example, tasks without predecessors.
section_title: Veterans Health Information Systems and Technology Architecture— Foundations Modernization
paragraphs: The Veterans Health Information Systems and Technology Architecture— Foundations Modernization (VistA-FM) program addresses the need to  transition the Veterans Affairs electronic medical record system to a new  architecture. According to the department, the current system is costly  and difficult to maintain and does not integrate well with newer software  packages. VistA-FM is designed to provide a new architectural framework  as well as additional standardization and common services components.  This is intended to eliminate redundancies in coding and support  interoperability among applications. Ultimately, the new architecture will  lay the foundation for a new generation of computer systems in support of  caring for America’s veterans. During the course of our review, the  department’s Chief Information Officer suspended multiple components of  the VistA-FM program until a new development plan can be put in place.  This action was taken as part of a new departmentwide initiative to  identify troubled IT projects and improve their execution.
VistA-FM partially met 4 key practices and did not meet 7 others, despite  reporting compliance with the American National Standards Institute  (ANSI) standard in its 2010 business case submission. Specifically, the  program is still working to establish a comprehensive EVM system to meet  ANSI compliance, among other things. For example, the work breakdown  structure is organized around key program milestones instead of product  deliverables, and does not fully describe the scope of work to be  performed. Although the program’s subprojects maintain their own  schedules, VistA-FM does not currently have an integrated master  schedule at the program level. This is of concern because it is not possible  to establish the program’s critical path and the time-phased budget  baseline, a key component of EVM. The reliability of the data is also a  potential issue because the program’s EVM reports do not offer adequate  detail to provide insight into data reliability issues. Additionally, the  performance baseline has not been appropriately updated; program  officials stated this update is in progress, but they did not have a  completion date.
section_title: Appendix III: Comments from the Department of Commerce
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: Comments from the Department of Justice
section_title: Appendix VI: Comments from the National Aeronautics and Space Administration
section_title: Appendix VII: Comments from the Department of Veterans Affairs
section_title: Appendix VIII: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Carol Cha (Assistant Director), Neil Doherty, Kaelin  Kuhn, Jason Lee, Lee McCracken, Colleen Phillips, Karen Richey, Teresa  Smith, Matthew Snyder, Jonathan Ticehurst, Kevin Walsh, and China  Williams.
section_title: Related GAO Products
paragraphs: Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-09-326SP. Washington, D.C.: March 30, 2009.  Discusses the Department of Defense’s Joint Tactical Radio System— Handheld, Manpack, Small Form Fit and Warfighter Information  Network—Tactical programs.
Information Technology: Census Bureau Testing of 2010 Decennial  Systems Can Be Strengthened. GAO-09-262. Washington, D.C.: March 5,  2009.   Discusses the Department of Commerce’s Decennial Response Integration  System and Field Data Collection Automation programs.
NASA: Assessments of Selected Large-Scale Projects. GAO-09-306SP.  Washington, D.C.: March 2, 2009.  Discusses the National Aeronautics and Space Administration’s James  Webb Space Telescope and Mars Science Laboratory programs.
Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage  Information Technology Acquisitions, but Needs to Clarify Policy and  Strengthen Oversight. GAO-08-756. Washington, D.C.: July 18, 2008.  Discusses the Department of Transportation’s En Route Automation  Modernization and Surveillance and Broadcast System programs.
Information Technology: Agriculture Needs to Strengthen Management  Practices for Stabilizing and Modernizing Its Farm Program Delivery  Systems. GAO-08-657. Washington, D.C.: May 16, 2008.  Discusses the U.S. Department of Agriculture’s Farm Program  Modernization program.
Information Technology: Improvements for Acquisition of Customs  Trade Processing System Continue, but Further Efforts Needed to Avoid  More Cost and Schedule Shortfalls. GAO-08-46. Washington, D.C.: October  25, 2007.  Discusses the Department of Homeland Security’s Automated Commercial  Environment program.
Defense Acquisitions: The Global Information Grid and Challenges  Facing Its Implementation. GAO-04-858. Washington, D.C.: July 28, 2004.  Discusses the Department of Defense’s Warfighter Information Network— Tactical program.