Information Technology: Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System Acquisitions
Released: 2009-11-06T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT projects and how these funds are to be allocated. Planned federal IT spending in fiscal year 2009 totaled about $71 billion—of which $22 billion was planned for IT system development work, and the remainder was planned for operations and maintenance of existing systems. OMB plays a key role in overseeing federal agencies’ IT investments and how they are managed, stemming from its functions of assisting the President in overseeing the preparation of the federal budget and supervising budget preparation in executive branch agencies. In helping to formulate the President’s spending plans, OMB is responsible for evaluating the effectiveness of agency programs, policies, and procedures; assessing competing funding demands among agencies; and setting funding priorities. To carry out these responsibilities, OMB depends on agencies to collect and report accurate and complete information; these activities depend, in turn, on agencies having effective IT management practices.

To drive improvement in the implementation and management of IT projects, Congress enacted the Clinger-Cohen Act in 1996, expanding the responsibilities delegated to OMB and agencies under the Paperwork Reduction Act. The Clinger-Cohen Act requires agencies to engage in performance- and results-based management, and to implement and enforce IT management policies and guidelines. The act also requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies.

Over the past several years, we have reported and testified on OMB’s initiatives to highlight troubled projects, justify IT investments, and use project management tools. We have made multiple recommendations to OMB and federal agencies to improve these initiatives to further enhance the oversight and transparency of federal IT projects. As a result, OMB recently used this body of work to develop and implement improved processes to oversee and increase transparency of IT investments. Specifically, in June 2009, OMB publicly deployed a Web site that displays dashboards of all major federal IT investments to provide OMB and others with the ability to track the progress of these investments over time.

EVM Provides Insight on Program Cost and Schedule

Given the size and significance of the government’s investment in IT, it is important that projects be managed effectively to ensure that public resources are wisely invested. Effectively managing projects entails, among other things, pulling together essential cost, schedule, and technical information in a meaningful, coherent fashion so that managers have an accurate view of the program’s development status. Without meaningful and coherent cost and schedule information, program managers can have a distorted view of a program’s status and risks. To address this issue, in the 1960s, the Department of Defense (DOD) developed the EVM technique, which goes beyond simply comparing budgeted costs with actual costs. This technique measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for that period and with the actual cost of work accomplished.

Differences in these values are measured in both cost and schedule variances. Cost variances compare the value of the completed work (i.e., the earned value) with the actual cost of the work performed. For example, if a contractor completed $5 million worth of work and the work actually cost $6.7 million, there would be a negative $1.7 million cost variance. Schedule variances are also measured in dollars, but they compare the earned value of the completed work with the value of the work that was expected to be completed. For example, if a contractor completed $5 million worth of work at the end of the month but was budgeted to complete $10 million worth of work, there would be a negative $5 million schedule variance. Positive variances indicate that activities are costing less or are completed ahead of schedule. Negative variances indicate activities are costing more or are falling behind schedule. These cost and schedule variances can then be used in estimating the cost and time needed to complete the program.

Without knowing the planned cost of completed work and work in progress (i.e., the earned value), it is difficult to determine a program’s true status. Earned value allows for this key information, which provides an objective view of program status and is necessary for understanding the health of a program. As a result, EVM can alert program managers to potential problems sooner than using expenditures alone, thereby reducing the chance and magnitude of cost overruns and schedule slippages. Moreover, EVM directly supports the institutionalization of key processes for acquiring and developing systems and the ability to effectively manage investments—areas that are often found to be inadequate on the basis of our assessments of major IT investments.

Federal Guidance Calls for Using EVM to Improve IT Management

In August 2005, OMB issued guidance outlining steps that agencies must take for all major and high-risk development projects to better ensure improved execution and performance and to promote more effective oversight through the implementation of EVM. Specifically, this guidance directs agencies to (1) develop comprehensive policies to ensure that their major IT investments are using EVM to plan and manage development; (2) include a provision and clause in major acquisition contracts or agency in-house project charters directing the use of an EVM system that is compliant with the American National Standards Institute (ANSI) standard; (3) provide documentation demonstrating that the contractor’s or agency’s in-house EVM system complies with the national standard; (4) conduct periodic surveillance reviews; and (5) conduct integrated baseline reviews on individual programs to finalize their cost, schedule, and performance goals.

Building on OMB’s requirements, in March 2009, we issued a guide on best practices for estimating and managing program costs. This guide highlights the policies and practices adopted by leading organizations to implement an effective EVM program. Specifically, in the guide, we identify the need for organizational policies that establish clear criteria for which programs are required to use EVM, specify compliance with the ANSI standard, require a standard product-oriented structure for defining work products, require integrated baseline reviews, provide for specialized training, establish criteria and conditions for rebaselining programs, and require an ongoing surveillance function. In addition, we identify key practices that individual programs can use to ensure that they establish a sound EVM system, that the earned value data are reliable, and that the data are used to support decision making.

Prior Reviews on Agency Use of EVM to Acquire and Manage IT Systems Have Identified Weaknesses

We have previously reported on the weaknesses associated with the implementation of sound EVM programs at various agencies, as well as on the lack of aggressive management action to correct poor cost and schedule performance trends based on earned value data for major system acquisition programs: In July 2008, we reported that the Federal Aviation Administration’s EVM policy was not fully consistent with best practices. For example, the agency required its program managers to obtain EVM training, but did not enforce completion of this training or require other relevant personnel to obtain this training. In addition, although the agency was using EVM to manage IT acquisitions, not all programs were ensuring that their earned value data were reliable. Specifically, of the three programs collecting EVM data, only one program adequately ensured that its earned value data were reliable. As a result, the agency faced an increased risk that managers were not getting the information they needed to effectively manage the programs. In response to our findings and recommendations, the Federal Aviation Administration reported that it had initiatives under way to improve its EVM oversight processes.

In September 2008, we reported that the Department of the Treasury’s EVM policy was not fully consistent with best practices. For example, while the department’s policy addressed some practices, such as establishing clear criteria for which programs are to use EVM, it did not address others, such as requiring and enforcing EVM training. In addition, six programs at Treasury and its bureaus were not consistently implementing practices needed for establishing a comprehensive EVM system. For example, when executing work plans and recording actual costs, a key practice for ensuring that the data resulting from the EVM system are reliable, only two of the six investments that we reviewed incorporated government costs with contractor costs. As a result, we reported that Treasury may not be able to effectively manage its critical programs. In response to our findings and recommendations, Treasury reported that it would release a revised EVM policy and further noted that initiatives to improve EVM-related training were under way.

In a series of reports and testimonies from September 2004 to June 2009, we reported that the National Oceanic and Atmospheric Administration’s National Polar-orbiting Operational Environmental Satellite System program was likely to overrun its contract at completion on the basis of our analysis of contractor EVM data. Specifically, the program had delayed key milestones and experienced technical issues in the development of key sensors, which we stated would affect cost and schedule estimates. As predicted, in June 2006 the program was restructured, decreasing its complexity, delaying the availability of the first satellite by 3 to 5 years, and increasing its cost estimate from $6.9 billion to $12.5 billion. However, the program has continued to face significant technical and management issues. As of June 2009, launch of the first satellite was delayed by 14 months, and our current projected total cost estimate is approximately $15 billion. We made multiple recommendations to improve this program, including establishing a realistic time frame for revising the cost and schedule baselines, developing plans to mitigate the risk of gaps in satellite continuity, and tracking the program executive committee’s action items from inception to closure.

Agencies’ EVM Policies Are Not Comprehensive

While the eight agencies we reviewed have established policies requiring the use of EVM on their major IT investments, none of these policies are fully consistent with best practices, such as standardizing the way work products are defined. We recently reported that leading organizations establish EVM policies that establish clear criteria for which programs are to use EVM; require programs to comply with the ANSI standard; require programs to use a product-oriented structure for defining work products; require programs to conduct detailed reviews of expected costs, schedules, and deliverables (called an integrated baseline review); require and enforce EVM training; define when programs may revise cost and schedule baselines (called require system surveillance—that is, routine validation checks to ensure that major acquisitions are continuing to comply with agency policies and standards.

Table 1 describes the key components of an effective EVM policy.

The eight agencies we reviewed do not have comprehensive EVM policies. Specifically, none of the agencies’ policies are fully consistent with all seven key components of an effective EVM policy. Table 2 provides a detailed assessment, by agency, and a discussion of the agencies’ policies follows the table.

Criteria for implementing EVM on all IT major investments: Seven of the eight agencies fully defined criteria for implementing EVM on major IT investments. The agencies with sound policies typically defined “major” investments as those exceeding a certain cost threshold, and, in some cases, agencies defined lower tiers of investments requiring reduced levels of EVM compliance. Veterans Affairs only partially met this key practice because its policy did not clearly state whether programs or major subcomponents of programs (projects and subprojects) had to comply with EVM requirements. According to agency officials, this lack of clarity may cause EVM to be inconsistently applied across the investments. Without an established policy that clearly defines the conditions under which new or ongoing acquisition programs are required to implement EVM, these agencies cannot ensure that EVM is being appropriately applied on their major investments.

Compliance with the ANSI standard: Seven of the eight agencies required that all work activities performed on major investments be managed by an EVM system that complies with industry standards. One agency, Transportation, partially met this key practice because its policy contained inconsistent criteria for when investments must comply with standards. Specifically, in one section, the policy requires a certain class of investments to adhere to a subset of the ANSI standard; however, in another section, the policy merely states that the investments must comply with general EVM principles. This latter section is vague and could be interpreted in multiple ways, either more broadly or narrowly than the specified subset of the ANSI standard. Without consistent criteria on investment compliance, Transportation may be unable to ensure that the work activities for some of its major investments are establishing sound EVM systems that produce reliable earned value data and provide the basis for informed decision making.

Standard structure for defining the work products: DOD was the only agency to fully meet this key practice by developing and requiring the use of standard product-oriented work breakdown structures. Four agencies did not meet this key practice, while the other three only partially complied. Of those agencies that partially complied, National Aeronautics and Space Administration (NASA) policy requires mission (or space flight) projects to use a standardized product-oriented work breakdown structure; however, IT projects do not have such a requirement. NASA officials reported that they are working to develop a standard structure for their IT projects; however, they were unable to provide a time frame for completion. Homeland Security and Justice have yet to standardize their product structures.

Among the agencies that did not implement this key practice, reasons included, among other things, the difficulty in establishing a standard structure for component agencies that conduct different types of work with varying complexity. While this presents a challenge, agencies could adopt an approach similar to DOD’s and develop various standard work structures based on the kinds of work being performed by the various component agencies (e.g., automated information system, IT infrastructure, and IT services). Without fully implementing a standard product-oriented structure (or structures), agencies will be unable to collect and share data among programs and may not have the information they need to make decisions on specific program components.

Integrated baseline review: All eight agencies required major IT investments to conduct an integrated baseline review to ensure that program baselines fully reflect the scope of work to be performed, key risks, and available resources. For example, DOD required that these reviews occur within 6 months of contract award and after major modifications have taken place, among other things.

Training requirements: Commerce was the only agency to fully meet this key practice by requiring and enforcing EVM training for all personnel with investment oversight and program management responsibilities. Several of the partially compliant agencies required EVM training for project managers—but did not extend this requirement to other program management personnel or executives with investment oversight responsibilities. Many agencies told us that it would be a significant challenge to require and enforce EVM training for all relevant personnel, especially at the executive level. Instead, most agencies have made voluntary EVM training courses available agencywide. However, without comprehensive EVM training requirements and enforcement, agencies cannot effectively ensure that programs have the appropriate skills to validate and interpret EVM data, and that their executives will be able to make fully informed decisions based on the EVM analysis.

Rebaselining criteria: Three of the eight agencies fully met this key practice. For example, the Justice policy outlines acceptable reasons for rebaselining, such as when the baseline no longer reflects the current scope of work being performed, and requires investments to explain why their current plans are no longer feasible and to develop realistic cost and schedule estimates for remaining work. Among the five partially compliant agencies, Agriculture and Veterans Affairs provided policies, but in draft form; NASA was in the process of updating its policy to include more detailed criteria for rebaselining; and Homeland Security did not define acceptable reasons but did require an explanation of the root causes for cost and schedule variances and the development of new cost and schedule estimates. In several cases, agencies were unaware of the detailed rebaselining criteria to be included in their EVM policies. Until their policies fully meet this key practice, agencies face an increased risk that their executive managers will make decisions about programs with incomplete information, and that these programs will continue to overrun costs and schedules because their underlying problems have not been identified or addressed.

System surveillance: All eight agencies required ongoing EVM system surveillance of all programs (and contracts with EVM requirements) to ensure their continued compliance with industry standards. For example, Agriculture required its surveillance teams to submit reports—to the programs and the Chief Information Officer—with documented findings and recommendations regarding compliance. Furthermore, the agency also established a schedule to show when EVM surveillance is expected to take place on each of its programs.

Agencies’ Key Acquisition Programs Are Using EVM, but Are Not Consistently Implementing Key Practices

Our studies of 16 major system acquisition programs showed that all agencies are using EVM; however, the extent of that implementation varies among the programs. Our work on best practices in EVM identified 11 key practices that are implemented on acquisition programs of leading organizations. These practices can be organized into three management areas: establishing a sound EVM system, ensuring reliable data, and using earned value data to make decisions. Table 3 lists these 11 key EVM practices by management area.

Of the 16 case study programs, 3 demonstrated a full level of maturity in all three management areas; 3 had full maturity in two areas; and 4 had reached full maturity in one area. The remaining 6 programs did not demonstrate full levels of maturity in any of the management areas; however, in all but 1 case, they were able to demonstrate partial capabilities in each of the three areas. Table 4 identifies the 16 case study programs and summarizes our results for these programs. Following the table is a summary of the programs’ implementation of each key area of EVM program management responsibility. Additional details on the 16 case studies are provided in appendix II.

Most Programs Did Not Fully Establish Comprehensive EVM Systems

Most programs did not fully implement the key practices needed to establish comprehensive EVM systems. Of the 16 programs, 3 fully implemented the practices in this program management area, and 13 partially implemented the practices. The Decennial Response Integration System, Next Generation Identification, and Surveillance and Broadcast System programs demonstrated that they had fully implemented the six practices in this area. For example, our analysis of the Decennial Response Integration System program schedule showed that activities were properly sequenced, realistic durations were established, and labor and material resources were assigned. The Surveillance and Broadcast System program conducted a detailed integrated baseline review to validate its performance baseline. It was also the only program to fully institutionalize EVM at the program level—meaning that it collects performance data on the contractor and government work efforts—in order to get a complete view into program status.

Thirteen programs demonstrated that they partially implemented the six key practices in this area. In most cases, programs had work breakdown structures that defined work products to an appropriate level of detail and had identified the personnel responsible for delivering these work products. However, for all 13 programs, the project schedules contained issues that undermined the quality of their performance baselines.

Weaknesses in these schedules included the improper sequencing of activities, such as incomplete or missing linkages between tasks; a lack of resources assigned to all activities; invalid critical paths (the sequence of activities that, if delayed, will impact the planned completion date of the project); and the excessive or unjustified use of constraints, which impairs the program’s ability to forecast the impact of ongoing delays on future planned work activities. These weaknesses are of concern because the schedule serves as the performance baseline against which earned value is measured. As such, poor schedules undermine the overall quality of a program’s EVM system. Other key weaknesses included the following examples: Nine programs did not adequately determine an objective measure of earned value and develop the performance baseline—that is, key practices most appropriately addressed through a comprehensive integrated baseline review, which none of them fully performed. For example, the Air and Space Operations Center—Weapon System program conducted an integrated baseline review in May 2007 to validate one segment of work contained in the baseline; however, the program had not conducted subsequent reviews for the remaining work because doing so would preclude staff from completing their normal work activities. Other reasons cited by the programs for not performing these reviews included the lack of a fully defined scope of work or management’s decision to use ongoing EVM surveillance to satisfy these practices. Without having performed a comprehensive integrated baseline review, programs have not sufficiently evaluated the validity of their baseline plan to determine whether all significant risks contained in the plan have been identified and mitigated, and that the metrics used to measure the progress made on planned work elements are appropriate.

Four programs did not define the scope of effort using a work breakdown structure. For example, the Veterans Health Information Systems and Technology Architecture—Foundations Modernization program provided a list of its subprograms; however, it did not define the scope of the detailed work elements that comprise each subprogram. Without a work breakdown structure, programs lack a basis for planning the performance baseline and assigning responsibility for that work, both of which are necessary to accomplish a program’s objectives.

Many Programs Did Not Fully Implement Practices to Ensure Data Reliability

Many programs did not fully ensure that their EVM data were reliable. Of the 16 programs, 7 fully implemented the practices for ensuring the reliability of the prime contractor and government performance data, and 9 partially implemented the practices. All 7 programs that demonstrated full implementation conduct monthly reviews of earned value data with technical engineering staff and other key personnel to ensure that the data are consistent with actual performance; perform detailed performance trend analyses to track program progress, cost, and schedule drivers; and make estimates of cost at completion. Four programs that we had previously identified as having schedule weaknesses (Farm Program Modernization; Joint Tactical Radio System—Handheld, Manpack, Small Form Fit; Juno; and Warfighter Information Network—Tactical) were aware of these issues and had sufficient controls in place to mitigate them in order to ensure that the earned value data are reliable.

Nine programs partially implemented the three practices for ensuring that earned value data are reliable. In all cases, the program had processes in place to review earned value data (from monthly contractor EVM reports in all but one case), identify and record cost and schedule variances, and forecast estimates at completion. However, 5 of these programs did not adequately analyze EVM performance data and properly record variances from the performance baseline. For example, 2 programs did not adequately document justifications for cost and schedule variances, including root causes, potential impacts, and corrective actions. Other weaknesses in this area include anomalies in monthly performance reports, such as negative dollars being spent for work performed, which impacts the validity of performance data. In addition, 7 of these programs did not demonstrate that they could adequately execute the work plan and record costs because, among other things, they were unaware of the schedule weaknesses we identified and did not have sufficient internal controls in place to deal with these issues to improve the reliability of the earned value data. Lastly, 2 of these programs could not adequately forecast estimates at completion due, in part, to anomalies in the prime contractor’s EVM reports, in combination with the weaknesses contained in the project schedule.

Most Programs Used Earned Value Data for Decision-making Purposes

Programs were uneven in their use of earned value data to make decisions. Of the 16 programs, 9 fully implemented the practices for using earned value data for decision making, 6 partially implemented them, and 1 did not implement them. Among the 9 fully implemented programs, both the Automated Commercial Environment and Juno programs integrated their EVM and risk management processes to support the program manager in making better decisions. The Automated Commercial Environment program actively recorded risks associated with major variances from the EVM reports in the program’s risk register. Juno further used the earned value data to analyze threats against remaining management reserve and to estimate the cost impact of these threats.

Six programs demonstrated limited capabilities in using earned value data for making decisions. In most cases, these programs included earned value performance trend data in monthly program management review briefings. However, the majority had processes for taking management action to address the cost and schedule drivers causing poor trends that were ad hoc and separate from the programs’ risk management processes—and, in most cases, the risks and issues found in the EVM reports did not correspond to the risks contained in the program risk registers. In addition, 4 of these programs were not able to adequately update the performance baseline as changes occurred because, in many cases, the original baseline was not appropriately validated. For example, the Mars Science Laboratory program just recently updated its performance baseline as part of a recent replan effort. However, without validating the original and current baselines with a project-level integrated baseline review, it is unclear whether the changes to the baseline were reasonable, and whether the risks assumed in the baseline have been identified and appropriately mitigated.

One program (Veterans Health Information Systems and Technology Architecture—Foundations Modernization) was not using earned value data for decision making. Specifically, the program did not actively manage earned value performance trends, nor were these data incorporated into programwide management reviews.

Inconsistent Implementation Is Due in Part to Weaknesses in Policy and Lack of Enforcement

The inconsistent application of EVM across the investments exists in part because of the weaknesses we previously identified in the eight agencies’ policies, as well as a lack of enforcement of the EVM policy components already in place. For example, deficiencies in all three management areas can be attributed, in part, to a lack of comprehensive EVM training requirements—which was a policy component that most agencies did not fully address. The only 3 programs that had fully implemented all key EVM practices either had comprehensive training requirements in their agency EVM policy or enforced rigorous training requirements beyond that for which the policy called. Most of the remaining programs met the minimum requirements of their agencies’ policies. However, all programs that had attained full maturity in two management areas had also implemented more stringent training requirements, although none could match the efforts made on the other 3 programs. Without making this training a comprehensive requirement, these agencies are at risk that their major system acquisition programs will continue to have management and technical staff who lack the skills to fully implement key EVM practices.

Our case study analysis also highlighted multiple areas in which programs were not in compliance with their agencies’ established EVM policies. This is an indication that agencies are not adequately enforcing program compliance. These policy areas include requiring EVM compliance at the start of the program, validating the baseline with an integrated baseline review, and conducting ongoing EVM surveillance.

Until key EVM practices are fully implemented, selected programs face an increased risk that program managers cannot effectively optimize EVM as a management tool to mitigate and reverse poor cost and schedule performance trends.

Earned Value Data Show Trends of Cost Overruns and Schedule Slippages on Most Programs

Earned value data trends of the 16 case study programs indicate that most are currently experiencing cost overruns and schedule slippages, and, based on our analysis, it is likely that when these programs are completed, the total cost overrun will be about $3 billion. To date, these programs, collectively, have already overrun their original life-cycle cost estimates by almost $2 billion (see table 5).

Taking the current earned value performance into account, our analysis of the 16 case study programs indicated that most are experiencing shortfalls against their currently planned cost and schedule targets. Specifically, earned value performance data over a 12-month period showed that the 16 programs combined have exceeded their cost targets by $275 million. During that period, they also experienced schedule variances and were unable to accomplish almost $93 million worth of planned work. In most cases, the negative cost and schedule performance trends were attributed to ongoing technical issues in the development or testing of system components.

Furthermore, our projections of future estimated costs at completion based on our analysis of current contractor performance trends indicate that these programs will most likely continue to experience cost overruns to completion, totaling almost $1 billion. In contrast, the programs’ contractors estimate the cost overruns at completion will be approximately $469.7 million. These estimates are based on the contractors’ assumption that their efficiency in completing the remaining work will significantly improve over what has been done to date. Furthermore, it should be noted that in 4 cases, the contractor-estimated overrun is smaller than the cost variances they have already accumulated—which is an indication that these estimates are aggressively optimistic.

With the inclusion of the overruns already incurred to date, the total increase in life-cycle costs will be about $3 billion. Our analysis is presented in table 6. Additional details on the 16 case studies are provided in appendix II.

Eleven programs are expected to incur a cost overrun at contract completion. In particular, two programs (i.e., the James Webb Space Telescope and Veterans Health Information Systems and Technology Architecture—Foundations Modernization programs) will likely experience a combined overrun of $798.7 million, which accounts for about 80 percent of our total projection.

With timely and effective action taken by program and executive management, it is possible to reverse negative performance trends so that the projected cost overruns at completion may be reduced. To get such results, management at all levels could be strengthened, including contractor management, program office management, and executive-level management. For example, programs could strengthen program office controls and contractor oversight by obtaining earned value data weekly (instead of monthly) so that they can make decisions with immediate and greater impact. Additionally, key risks could be elevated to the program level and, if necessary, to the executive level to ensure that appropriate mitigation plans are in place and that they are tracked to closure.

Conclusions

Key agencies have taken a number of important steps to improve the management of major acquisitions through the implementation of EVM. Specifically, the agencies have established EVM policies and require their major system acquisition programs to use EVM. However, none of the eight agencies that we reviewed have comprehensive EVM policies. Most of these policies omit or lack sufficient guidance on the type of work structure needed to effectively use EVM data and on the training requirements for all relevant personnel. Without comprehensive policies, it will be difficult for the agencies to gain the full benefits of EVM.

Few of our 16 case study programs had fully implemented EVM capabilities, raising concerns that programs cannot efficiently produce reliable estimates of cost at completion. Many of these weaknesses found on these programs can be traced back to inadequate agency EVM policies and raise questions concerning the agencies’ enforcement of the policies already established, including the completion of the integrated baseline reviews and system surveillance. Until agencies expand and enforce their EVM policies, it will be difficult for them to optimize the effectiveness of this management tool, and they will face an increased risk that managers are not getting the information they need to effectively manage the programs.

In addition to concerns about their implementation of EVM, the programs’ earned value data show trends toward cost overruns that are likely to collectively total about $3 billion. Without timely and aggressive management action, this projected overrun will be realized, resulting in the expenditure of over $1 billion more than currently planned.

Recommendations for Executive Action

To address the weaknesses identified in agencies’ policies and practices in using EVM, we are making recommendations to the eight major agencies included in this review. Specifically, we recommend that the following three actions be taken by the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration: modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report; direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report; and direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Comments and Our Evaluation

We provided the selected eight agencies with a draft of our report for review and comment. The Department of Homeland Security responded that it had no comments. The remaining seven agencies generally agreed with our results and recommendations. Agencies also provided technical comments, which we incorporated in the report as appropriate.

The comments of the agencies are summarized in the following text: In e-mail comments on a draft of the report, officials from the U.S. Department of Agriculture’s Office of the Chief Information Officer stated that the department has begun to address the weaknesses in its EVM policy identified in the report.

In written comments on a draft of the report, the Secretary of Commerce stated that, regarding the second and third recommendations, the Department of Commerce was pleased that the Decennial Response Integration System was found to have fully implemented all 11 key EVM practices, and that the Field Data Collection Automation program fully implemented six key practices. The department added that its recent actions on the Field Data Collection Automation program should move this program to full compliance with the key EVM practices. Furthermore, regarding the first recommendation, the Secretary stated that while the department understands and appreciates the value of standardized work breakdown structures, it maintained that the development of these work structures should take place at the department’s operating units (e.g., Census Bureau), given the wide diversity of missions and project complexity among these units. As noted in our report, we agree that agencies could develop standard work structures based on the kinds of work being performed by the various component agencies. Therefore, we support these efforts described by the department because they are generally consistent with the intent of our recommendation. Commerce’s comments are printed in appendix III.

In written comments on a draft of the report, the Department of Defense’s Director of Defense Procurement and Acquisition Policy stated that the department concurred with our recommendations. Among other things, DOD stated that it is essential to maintain the appropriate oversight of acquisition programs, including the use of EVM data to understand program status and anticipate potential problems. DOD’s comments are printed in appendix IV.

In written comments on a draft of the report, the Department of Justice’s Assistant Attorney General for Administration stated that, after discussion with our office, it was agreed that the second recommendation, related to implementing EVM practices that address identified weakness, was inadvertently directed to the department, and that no response was necessary. We agreed because the case study program reviewed fully met all key EVM practices. The department concurred with the two remaining recommendations related to modifying EVM policies and reversing negative performance trends. Furthermore, the Assistant Attorney General noted that Justice had begun to take steps to improve its use of EVM, such as modifying its policy to require EVM training for all personnel with investment oversight and program management responsibilities. Justice’s comments are printed in appendix V.

In written comments on a draft of the report, the National Aeronautics and Space Administration’s Deputy Administrator stated that the agency concurred with two recommendations and partially concurred with one recommendation. In particular, the Deputy Administrator agreed that opportunities exist for improving the implementation of EVM, but stated that NASA classifies the projects included in the scope of the audit as space flight projects (not as IT-specific projects), which affects the applicability of the agency’s EVM policies and guidance that were reviewed. We recognize that different classifications of IT exist; however, consistent with other programs included in the audit, the selected NASA projects integrate and rely on various elements of IT. As such, we reviewed both the agency’s space flight and IT-specific guidance. Furthermore, the agency partially concurred with one recommendation because it stated that efforts were either under way or planned that will address the weaknesses we identified. We support the efforts that NASA described in its comments because they are generally consistent with the intent of our recommendation. NASA’s comments are printed in appendix VI.

In e-mail comments on a draft of the report, the Department of Transportation’s Director of Audit Relations stated that the department is taking immediate steps to modify its policies governing EVM, taking into consideration the criteria used in the draft report.

In written comments on a draft of the report, the Secretary of Veterans Affairs stated that the Department of Veterans Affairs generally agreed with our conclusions and concurred with our recommendations. Furthermore, the Secretary stated that Veterans Affairs has initiatives under way to address the weaknesses identified in the report. Veterans Affairs’ comments are printed in appendix VII.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees; the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs; the Administrator of the National Aeronautics and Space Administration; and other interested parties. In addition, the report will be available at no charge on our Web site at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VIII.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess whether key departments and agencies have appropriately established earned value management (EVM) policies, (2) determine whether these agencies are adequately using earned value techniques to manage key system acquisitions, and (3) evaluate the earned value data of these selected investments to determine their cost and schedule performances.

For this governmentwide review, we assessed eight agencies and 16 investments. We initially identified the 10 agencies with the highest amount of spending for information technology (IT) development, modernization, and enhancement work as reported in the Office of Management and Budget’s (OMB) Fiscal Year 2009 Exhibit 53. These agencies were the Departments of Agriculture, Commerce, Defense, Health and Human Services, Homeland Security, Justice, Transportation, the Treasury, and Veterans Affairs and the National Aeronautics and Space Administration. We excluded Treasury from our selection because we recently performed an extensive review of EVM at that agency. We also subsequently removed Health and Human Services from our selection because the agency did not have investments in system acquisition that met our dollar threshold (as defined in the following text). The resulting eight agencies also made up about 75 percent of the government’s planned IT spending for fiscal year 2009.

To ensure that we examined significant investments, we chose from investments (related to system acquisition) that were expected to receive development, modernization, and enhancement funding in fiscal year 2009 in excess of $90 million. We limited the number of selected investments to a maximum of 3 per agency. For agencies with more than 3 investments that met our threshold, we selected the top 3 investments with the highest planned spending. For agencies with 3 or fewer such investments, we chose all of the investments meeting our dollar threshold. Lastly, we excluded investments with related EVM work already under way at GAO.

To assess whether key agencies have appropriately established EVM policies, we analyzed agency policies and guidance for EVM. Specifically, we compared these policies and guidance documents with both OMB’s requirements and key best practices recognized within the federal government and industry for the implementation of EVM. These best practices are contained in the GAO cost guide. We also interviewed key agency officials to obtain information on their ongoing and future EVM plans.

To determine whether these agencies are adequately using earned value techniques to manage key system acquisitions, we analyzed program documentation, including project work breakdown structures, project schedules, integrated baseline review briefings, risk registers, and monthly management briefings for the 16 selected investments. Specifically, we compared program documentation with EVM and scheduling best practices as identified in the cost guide. We determined whether the program implemented, partially implemented, or did not implement each of the 11 practices. We also interviewed program officials (and observed key program status review meetings) to obtain clarification on how EVM practices are implemented and how the data are used for decision-making purposes.

To evaluate the earned value data of the selected investments to determine their cost and schedule performances, we analyzed the earned value data contained in contractor EVM performance reports obtained from the programs. To perform this analysis, we compared the cost of work completed with budgeted costs for scheduled work for a 12-month period to show trends in cost and schedule performances. We also used data from these reports to estimate the likely costs at completion through established earned value formulas. This resulted in three different values, with the middle value being the most likely. To assess the reliability of the cost data, we compared it with other available supporting documents (including OMB and agency financial reports); electronically tested the data to identify obvious problems with completeness or accuracy; and interviewed agency and program officials about the data. For the purposes of this report, we determined that the cost data were sufficiently reliable. We did not test the adequacy of the agency or contractor cost-accounting systems. Our evaluation of these cost data was based on what we were told by the agency and the information they could provide.

We conducted this performance audit from February to October 2009 at the agencies’ offices in the Washington, D.C., metropolitan area; Fort Monmouth, New Jersey; Jet Propulsion Lab, Pasadena, California; Hanscom Air Force Base, Massachusetts; and Naval Base San Diego, California. Our work was done in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Case Studies of Selected Programs’ Implementation of Earned Value Management

We conducted case studies of 16 major system acquisition programs (see table 7). For each of these programs, the remaining sections of this appendix provide the following: a brief description of the program, including a graphic illustration of the investment’s life cycle; an assessment of the program’s implementation of the 11 key EVM practices; and an analysis of the program’s recent earned value (EV) data and trends. These data and trends are often described in terms of cost and schedule variances. Cost variances compare the earned value of the completed work with the actual cost of the work performed. Schedule variances are also measured in dollars, but they compare the earned value of the completed work with the value of the work that was expected to be completed. Positive variances are good—they indicate that activities are costing less than expected or are completed ahead of schedule. Negative variances are bad—they indicate activities are costing more than expected or are falling behind schedule.

The following information describes the key that we used in tables 8 through 23 to convey the results of our assessment of the 16 case study programs’ implementation of the 11 EVM practices.

The program fully implemented all EVM practices in this program management area.

The program partially implemented the EVM practices in this program management area.

The program did not implement the EVM practices in this program management area.

Farm Program Modernization

The Farm Program Modernization (MIDAS) program is intended to address the long-term needs in delivering farm benefit programs via business process reengineering and implementation of a commercial off- the-shelf enterprise resource planning solution. MIDAS is an initiative of the Farm Service Agency, which is responsible for administering 35 farm benefit programs. To support these programs, the agency uses two primary systems—a distributed network of legacy computers and a centralized Web farm (to store customer data and host Web-based applications)—both of which have shortcomings. While MIDAS is to replace these computers, it is also intended to provide new applications and redesigned business processes. The Web farm is expected to remain in operation in a supporting role for the program. Currently, MIDAS is in the initiation phase of its life cycle and plans to award the system integration contract in the first quarter of fiscal year 2010.

MIDAS fully met 6 of the 11 key practices for implementing EVM and partially met 5 practices. Specifically, a key weakness in the EVM system is the lack of a comprehensive integrated baseline review. Instead, MIDAS focused solely on evaluating the program’s compliance with industry standards and chose not to validate the quality of the baseline. Program officials stated that they plan to conduct a full review to address the risks and realism of the baseline after the prime contract has been awarded. Furthermore, while the MIDAS schedule is generally sound, resources were not assigned to all activities, and the critical path (the longest duration path through the sequenced list of key activities) could not be identified because the current schedule ends in September 2009. Finally, MIDAS met all key practices associated with data reliability, such as executing the work plan and recording costs, as well as all key practices for decision making.

Decennial Response Integration System

The Decennial Response Integration System (DRIS) is to be used during the 2010 Census for collecting and integrating census responses from all sources, including forms and telephone interviews. The system is to improve accuracy and timeliness by standardizing the response data and providing the data to other Census Bureau systems for analysis and processing. Among other things, DRIS is expected to process census data provided by respondents via census forms, telephone agents, and enumerators; assist the public via telephone; and monitor the quality and status of data capture operations. The DRIS program’s estimated life-cycle costs have increased by $372 million, which is mostly due to increases in both paper and telephone workloads. For example, the paper workload increased due to an April 2008 redesign of the 2010 Census that reverted planned automated operations to paper-based processes and requires DRIS to process an additional estimated 40 million paper forms.

DRIS fully implemented all 11 of the key EVM practices necessary to manage its system acquisition program. Specifically, the program implemented all practices for establishing a comprehensive EVM system, such as defining the scope of work and scheduling the work. The program’s schedule appropriately captured and sequenced key activities and assigned realistic resources to all key activities. Furthermore, the DRIS team ensured that the resulting EVM data were appropriately verified and validated for reliability by analyzing performance data to identify the magnitude and effect of problems causing key variances, tracking related risks in the program’s risks register, and performing quality checks of the schedule and critical path. Lastly, the DRIS program management team conducted rigorous reviews of EV performance on a monthly basis and took the appropriate management actions to mitigate risks.

Field Data Collection Automation

The Field Data Collection Automation (FDCA) program is intended to provide automation support for the 2010 Census field data collection operations. The program includes the development of handheld computers for identifying and correcting addresses for all known living quarters in the United States (known as address canvassing) and the systems, equipment, and infrastructure that field staff will use to collect data. FDCA handheld computers were originally to be used for other census field operations, such as following up with nonrespondents through personal interviews. However, in April 2008, due to problems identified during testing and cost overruns and schedule slippages in the FDCA program, the Secretary of Commerce announced a redesign of the 2010 Census, and rebaselined FDCA in October 2008. As a result, FDCA’s life-cycle costs have increased from an estimated $596 million to $801 million, a $205 million increase. Furthermore, the responsibility for the design, development, and testing of IT systems for other key field operations was moved from the FDCA contractor to the Census Bureau.

FDCA fully met 6 of the 11 key practices for implementing EVM and partially met 5 others. Specifically, the program fully met most practices for establishing a comprehensive EVM system, such as defining the scope of the work effort; however, it only partially met the practice for scheduling the work. Specifically, the program schedule contained weaknesses, including key milestones with fixed completion dates—which hampers the program’s ability to see the impact of delays experienced on open tasks on successor tasks. As such, the FDCA program cannot use the schedule as an active management tool. Furthermore, anomalies in the prime contractor’s EVM reports, combined with weaknesses in the master schedule, affect FDCA’s ability to execute the work plan, analyze variances, and make reliable estimates of cost at completion. Lastly, cost and schedule drivers identified in EVM reports were not fully consistent with the program’s risk register, which prevents the program from taking the appropriate management action to mitigate risks and effectively using EV data for decisions.

Air and Space Operations Center— Weapon System

The Air and Space Operations Center—Weapon System (AOC) is the air and space operations planning, execution, and assessment system for the Joint Force Air Component Commander. According to the agency, there are currently 11 AOCs located around the world, each aligned to the Combatant Commands of the Unified Command Plan, with additional support units for training, help desk, testing, and contingency manpower augmentation. Each AOC is designed to enable commanders to exercise command and control of air, space, information operations, and combat support forces to achieve the objectives of the joint force commander and combatant commander in joint and coalition military operations. As such, the AOC system is intended as the planning and execution engine of any air campaign.

AOC fully met 7 of the 11 key practices and partially met 4 others. AOC applied EVM at the contract level and has a capable government team that has made it an integral part of project management. AOC performed detailed analyses of the EV data and reviews the data with engineering staff to ensure that the appropriate metrics have been applied for accurate reporting. AOC has also integrated EVM with its risk management processes to ensure that resources are applied to watch or mitigate risks associated with the cost and schedule drivers reported in the EVM reports. Weaknesses found in AOC’s EVM processes relate to the development and validation of the contractor baseline. In particular, AOC has not performed an integrated baseline review for all work that is currently on contract. The master schedule also contained issues, such as a high number of converging tasks and out-of-sequence tasks, that hamper AOC’s ability to determine the start dates of future tasks. Taken together, these issues undermine the reliability of the schedule as a baseline to measure EV performance.

Joint Tactical Radio System—Handheld, Manpack, Small Form Fit

The Joint Tactical Radio System (JTRS) program is developing software- defined radios that are expected to interoperate with existing radios and increase communications and networking capabilities. The JTRS- Handheld, Manpack, Small Form Fit (HMS) product office, within the JTRS Ground Domain program office, is developing handheld, manpack, and small form fit radios. In 2006, the program was restructured to include two concurrent phases of development. Phase I includes select small form fit radios, while Phase II includes small form fit radios with enhanced security as well as handheld and manpack variants. Subsequent to the program’s restructure, the department updated its migration strategy for replacing legacy radios with new tactical radios. As such, the total planned quantity of JTRS-HMS radios was reduced from an original baseline of 328,514—established in May 2004—to 95,551. As a result, the total life- cycle cost of the JTRS-HMS program was reduced from an estimated $19.2 billion to $11.6 billion, a $7.6 billion decrease.

JTRS-HMS fully met 10 of the 11 key practices and partially met 1 practice. Specifically, JTRS-HMS implemented most practices for establishing a comprehensive EVM system, such as performing rigorous reviews to validate the baseline; however, the current schedule contained some weaknesses, such as out-of-sequence logic and activities without resources assigned. Program officials were aware of these issues and attributed them to weaknesses in subcontractor schedules that are integrated on a monthly basis. The JTRS-HMS program fully met practices for ensuring that the resulting EV data were appropriately verified and validated for reliability and demonstrated that the program management team was using these data for decision-making purposes.

Warfighter Information Network—Tactical

The Warfighter Information Network—Tactical (WIN-T) program is designed to be the Army’s high-speed and high-capacity backbone communications network. The program connects Department of the Army units with higher levels of command and provides the Army’s tactical portion of the Global Information Grid—a Department of Defense initiative aimed at building a secure network and set of information capabilities modeled after the Internet. WIN-T was restructured in June 2007 following a unit cost increase above the critical cost growth threshold (known as a Nunn-McCurdy breach). As a result of the restructuring, it was determined that WIN-T would be fielded in four increments. The third increment is expected to provide the Army with a full networking on-the-move capability and fully support the Army’s Future Combat Systems. In May 2009, the Increment 3 program baseline was approved, and the life-cycle cost for the program was estimated at $38.2 billion. Our assessment of EVM practices and EV data was performed on WIN-T Increment 3.

WIN-T fully met 7 of the 11 key practices for implementing EVM, partially met 1 practice, and did not meet 3 practices. Specifically, WIN-T only partially met the practices for establishing a comprehensive EVM system. The schedule contained weaknesses, including fixed completion dates— which prevented the schedule from showing the impact of delays experienced on open or successor tasks or the expected completion dates of key activities. Furthermore, WIN-T has not conducted an integrated baseline review on the current scope of work since rebaselining the prime contract in December 2007. According to program officials, this review has not been conducted because they have not yet finalized the contract. However, as of August 2009, it has been 20 months since work began, which increases the risk that the program has not been measuring progress against a reasonable baseline. Without conducting this review to validate the performance baseline, the baseline cannot be adequately updated as changes occur, and EV data cannot be used effectively for decision-making purposes.

Automated Commercial Environment

The Automated Commercial Environment (ACE) program is the commercial trade processing system being developed by the U.S. Customs and Border Protection to facilitate trade while strengthening border security. The program is to provide trade compliance and border security staff with the right information at the right time, while minimizing administrative burden. Deployed in phases, ACE is expected to be expanded to provide cargo processing capabilities across all modes of transportation and intended to replace existing systems with a single, multimodal manifest system for land, air, rail, and sea cargo. Ultimately, ACE is expected to become the central data collection system for the federal agencies that, by law, require international trade data, and should deliver these capabilities in a secure, paper-free, Web-enabled environment. As a result of poorly managed requirements, the total life- cycle development cost of the ACE program increased from an estimated $1.5 billion to $2.2 billion—a $700 million increase.

ACE fully met 9 of the 11 key practices for implementing EVM and partially met the remaining 2 practices. Specifically, ACE fully met 5 of 6 practices for establishing a comprehensive EVM system, such as defining the scope of the work effort and developing the performance baseline, but partially met the practice for scheduling the work, in part, because resources were not assigned to all activities in the master schedule. ACE fully met 2 practices for ensuring that the data resulting from the EVM system were reliable, such as adequately analyzing EV performance data, but could not fully execute the work plan because of the weaknesses found in the schedule. Lastly, ACE demonstrated that the program management team was basing decisions on EVM data.

It should be noted that the ACE program is being defined incrementally— whereby the performance baseline is continuously updated as task orders for new work are issued. As such, the use of EVM to determine the true progress made and to project reliable final costs at completion is limited.

Integrated Deepwater System—Common Operational Picture

The Integrated Deepwater System is a 25-year, $24 billion major acquisition program to recapitalize the U.S. Coast Guard’s aging fleet of boats, airplanes, and helicopters, ensuring that all work together through a modern, capable communications system. This initiative is designed to enhance maritime domain awareness and enable the Coast Guard to meet its post-September 11 mission requirements. The program is composed of 15 major acquisition projects, including the Common Operational Picture (COP) program.

Deepwater COP is to provide relevant, real-time operational intelligence and surveillance data to human capital managers, allowing them to direct and monitor all assigned forces and first responders. This is expected to allow commanders to distribute critical information to federal, state, and local agencies quickly; reduce duplication; enable earlier alerting; and enhance maritime awareness.

Deepwater COP fully met 7 of the 11 key practices and partially met 4 others. Specifically, COP fully met 5 of the 6 practices for establishing a comprehensive EVM system, such as adequately defining all major elements of the work breakdown structure and developing the performance baseline. However, the program’s master schedule contained weaknesses, such as a large number of concurrent tasks and activities without resources assigned. Officials were aware of some, but not all, of the weaknesses in the schedule and had controls in place to mitigate the weakness they were aware of in order to improve the reliability of the resulting EV data. Lastly, COP was unable to fully meet 1 of the practices for using EV data for management decisions because it could not demonstrate that cost and schedule drivers impacting EV performance were linked to its risk management processes.

Western Hemisphere Travel Initiative

The Western Hemisphere Travel Initiative (WHTI) program made modifications to vehicle processing lanes at ports of entry on the nation’s northern and southern borders. WHTI is designed to allow U.S. Customs and Border Protection to effectively address new requirements imposed by the Intelligence Reform and Terrorism Prevention Act of 2004 (completing these requirements by June 1, 2009). WHTI development was completed and its implementation addressed the 39 highest volume ports of entry, which support 95 percent of land border traffic. The initiative requires travelers to present a passport or other authorized travel document that denotes identity and citizenship when entering the United States.

WHTI fully met 6 of the 11 key practices for implementing EVM and partially met the remaining 5 practices. Specifically, weaknesses identified in validating the performance baseline and scheduling the work limited the program’s ability to establish a comprehensive EVM system. Although the program held an integrated baseline review to validate the baseline in March 2008, the review did not cover many key aspects, such as identifying corrective actions needed to mitigate program risks. Furthermore, the master schedule contained deficiencies, such as activities that were out of sequence or lacking dependencies. While program officials described their use of processes for ensuring the reliability of the EVM system’s data, such as capturing significant cost and schedule drivers in the risk register, the provided documentation did not corroborate what we were told. When combined, these weaknesses preclude the program from effectively making decisions about the program based on EV data.

Next Generation Identification

The Next Generation Identification (NGI) program is designed to support the Federal Bureau of Investigation’s mission to reduce terrorist and criminal activities by providing timely, relevant criminal justice information to the law enforcement community. Today, the bureau operates and maintains one of the largest repositories of biometric- supported criminal history records in the world. The electronic identification and criminal history services support more than 82,000 criminal justice agencies, authorized civil agencies, and international organizations. NGI is intended to ensure that the bureau’s biometric systems are able to seamlessly share data that are complete, accurate, current, and timely. To accomplish this, the current system will be replaced or upgraded with new functionalities and state-of-the-art equipment. NGI is expected to be scaleable to accommodate five times the current workload volume with no increase in support manpower and will be flexible to respond to changing requirements.

NGI fully implemented all 11 key EVM practices. Specifically, the program implemented all practices for establishing a comprehensive EVM system, such as defining the scope of work and scheduling the work. For example, the schedule properly captured key activities, established reasonable durations, and established a sound critical path, all of which contribute to establishing a reliable baseline that performance can be measured against. Furthermore, the NGI team ensured that the resulting EV data were appropriately verified and validated for reliability by, for example, integrating the analysis of cost and schedule variances with the program’s risk register to mitigate emerging and existing risks associated with key drivers causing major variances. In addition, the program’s risk register includes cost and schedule impacts for every risk and links to the management reserve process. Lastly, NGI demonstrated that it is using EV data to make decisions by performing continuous quality checks of the schedule, reviewing open risks and opportunities, and reviewing EV data in weekly management reports.

James Webb Space Telescope

The James Webb Space Telescope (JWST) is designed to be the scientific successor to the Hubble Space Telescope and expected to be the premier observatory of the next decade. It is intended to seek to study and answer fundamental astrophysical questions, ranging from the formation and structure of the Universe to the origin of planetary systems and the origins of life. The telescope is an international collaboration of the National Aeronautics and Space Administration (NASA), the Canadian Space Agency, and the European Space Agency. JWST required the development of several new technologies, including a folding segmented primary mirror that will unfold after launch and a cryocooler for cooling midinfrared detectors to 7 degrees Kelvin.

JWST fully met 4 of the 11 key practices and partially met 7 practices. The project only partially met practices for establishing a comprehensive EVM system because of weaknesses in the work breakdown structure, in which the prime contractor has not fully defined the scope of each work element. In addition, the project only partially met the practice for scheduling work because of weaknesses resulting from manual integration of approximately 30 schedules, although officials did explain some mitigations for this risk. We also found deficiencies in the lower-level schedules, such as missing linkages between tasks, resources not being assigned, and excessively high durations. Furthermore, JWST only partially implemented practices to ensure that the data resulting from the EVM system are reliable, due, in part, to variance analysis reports being done quarterly (instead of monthly), which limits the project’s ability to analyze and respond to cost and schedule variances in a timely manner. When combined, these weaknesses preclude the program from effectively making decisions about the program based on EV data.

Juno

Juno is part of the New Frontiers Program. The overarching scientific goal of the Juno mission is to improve our understanding of the origin and evolution of Jupiter. As the archetype of giant planets, Jupiter may provide knowledge that will improve our understanding of both the origin of our solar system and the planetary systems being discovered around other stars. The Juno project is expected to use a solar-powered spacecraft to make global maps of the gravity, magnetic fields, and atmospheric composition of Jupiter. The spacecraft is to make 33 orbits of Jupiter to sample the planet’s full range of latitudes and longitudes.

Juno fully met 8 of the 11 key practices for implementing EVM and partially met 3 practices. Specifically, the project fully met 3 practices for establishing a comprehensive EVM system, but only partially met the practices for scheduling the work, determining the objective measure of earned value, and establishing the performance baseline. Juno was unable to fully meet these practices because the project’s master schedule contained issues with the sequencing of work activities and lacked a comprehensive integrated baseline review. Although an integrated baseline review was conducted for a major contract in February 2009, the program did not validate the baseline, scope of work to be performed, or key risks and mitigation plans for the Juno project as a whole, which increases the risk that the project is measuring performance against an unreasonable baseline. Juno fully implemented all 3 practices associated with data reliability and the 2 practices associated with using EV data for decision-making purposes.

Mars Science Laboratory

The Mars Science Laboratory (MSL) is part of the Mars Exploration Program. The program seeks to understand whether Mars was, is, or can be a habitable world. To answer this question, the MSL project is expected to investigate how geologic, climatic, and other processes have worked to shape Mars and its environment over time, as well as how they interact today. To accomplish this, the MSL project plans to place a mobile science laboratory on the surface of Mars to quantitatively assess a local site as a potential habitat for life, past or present. The project is considered one of NASA’s flagship projects and designed to be the most advanced rover ever sent to explore the surface of Mars. Due to technical issues identified during the development of key components, the MSL launch date has recently slipped 2 years—from September 2009 to October 2011, and the project’s life-cycle cost estimate has increased from about $1.63 billion to $2.29 billion, a $652 million increase.

MSL fully met 5 of the 11 key practices and partially met 6 others. Specifically, MSL fully met 3 practices for establishing a comprehensive EVM system, but only partially met 3 others because of weaknesses in the sequencing of all activities in the schedule and the lack of an integrated baseline review to validate the baseline and assess the achievability of the plan. While the project has taken steps to mitigate the latter weakness by requiring work agreements that document, among other things, the objective value of work and related risks for planned work packages, this is not a comprehensive review of the project’s baseline. Furthermore, MSL only partially implemented practices associated with data reliability because its analysis of cost and schedule variances did not include the root causes for variances and corrective actions, which prevents the project from tracking and mitigating related risks. Lastly, without an initial validation of the performance baseline, the baseline cannot be appropriately updated to reflect program changes, thereby limiting the use of EV data for management decisions.

En Route Automation Modernization

The En Route Automation Modernization (ERAM) program is to replace existing software and hardware in the air traffic control automation computer system and its backup system, the Direct Radar Channel, and other associated interfaces, communications, and support infrastructure at en route centers across the country. This is a critical effort because ERAM is expected to upgrade hardware and software for facilities that control high-altitude air traffic. ERAM consists of two major components. One component has been fully deployed and is currently in operation at facilities across the country. The other component is scheduled for deployment through fiscal year 2011.

ERAM fully met 7 of the 11 key practices and partially met 4 others. ERAM applies EVM at the contract level and incorporates EV data into its overall management of the program. However, ERAM did not perform a comprehensive review of the baseline when the contract was finalized, or take similar actions to validate the baseline and ensure that the appropriate EV metrics had been applied. While ERAM does perform limited checks of the contractor schedule, our analysis showed some issues with the sequencing of activities and the use of constraints that may undermine the reliability of the schedule as a baseline to measure performance.

However, it should be noted that the EV data are not a reflection of the total ERAM program. The government is also responsible for acquisition work—to which EVM is not being applied. Our analysis of the master schedule showed that ERAM would be unable to meet four major upcoming initial operating capability milestones due to issues associated with government work activities. Program officials noted that these milestones have since been pushed out. Since EVM is not applied at the program level, it is unclear whether these delays will impact overall cost.

Surveillance and Broadcast System

The Surveillance and Broadcast System (SBS) is to provide new surveillance solutions that employ technology using avionics and ground stations for improved accuracy and update rates and to provide shared situational awareness (including visual updates of traffic, weather, and flight notices) between pilots and air traffic control. These technologies are considered critical to achieving the Federal Aviation Administration’s strategic goals of decreasing the rate of accidents and incursions, improving the efficiency of air traffic, and reducing congestion.

SBS fully implemented all 11 key EVM practices. Specifically, SBS has institutionalized EVM at the program level—meaning that it collects and manages performance data on the contractor and government work efforts—in order to get a comprehensive view into program status. As part of this initiative, SBS performed detailed validation reviews of the contractor and program baselines; issued various process rules on resource planning, EV metrics, and data analysis; and collected government timecard data in order to ensure consistent EV application. In addition, the program management team conducted rigorous reviews of EV performance with the SBS program manger and the program’s internal management review board on a monthly basis. Our analysis of the SBS master schedule showed that it was developed in accordance with scheduling best practices. For example, the schedule was properly sequenced, and the resources were assigned. Furthermore, SBS briefed the program manager monthly on the quality of the schedule to identify, for example, tasks without predecessors.

Veterans Health Information Systems and Technology Architecture— Foundations Modernization

The Veterans Health Information Systems and Technology Architecture— Foundations Modernization (VistA-FM) program addresses the need to transition the Veterans Affairs electronic medical record system to a new architecture. According to the department, the current system is costly and difficult to maintain and does not integrate well with newer software packages. VistA-FM is designed to provide a new architectural framework as well as additional standardization and common services components. This is intended to eliminate redundancies in coding and support interoperability among applications. Ultimately, the new architecture will lay the foundation for a new generation of computer systems in support of caring for America’s veterans. During the course of our review, the department’s Chief Information Officer suspended multiple components of the VistA-FM program until a new development plan can be put in place. This action was taken as part of a new departmentwide initiative to identify troubled IT projects and improve their execution.

VistA-FM partially met 4 key practices and did not meet 7 others, despite reporting compliance with the American National Standards Institute (ANSI) standard in its 2010 business case submission. Specifically, the program is still working to establish a comprehensive EVM system to meet ANSI compliance, among other things. For example, the work breakdown structure is organized around key program milestones instead of product deliverables, and does not fully describe the scope of work to be performed. Although the program’s subprojects maintain their own schedules, VistA-FM does not currently have an integrated master schedule at the program level. This is of concern because it is not possible to establish the program’s critical path and the time-phased budget baseline, a key component of EVM. The reliability of the data is also a potential issue because the program’s EVM reports do not offer adequate detail to provide insight into data reliability issues. Additionally, the performance baseline has not been appropriately updated; program officials stated this update is in progress, but they did not have a completion date.

Appendix III: Comments from the Department of Commerce

Appendix IV: Comments from the Department of Defense

Appendix V: Comments from the Department of Justice

Appendix VI: Comments from the National Aeronautics and Space Administration

Appendix VII: Comments from the Department of Veterans Affairs

Appendix VIII: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact name above, individuals making contributions to this report included Carol Cha (Assistant Director), Neil Doherty, Kaelin Kuhn, Jason Lee, Lee McCracken, Colleen Phillips, Karen Richey, Teresa Smith, Matthew Snyder, Jonathan Ticehurst, Kevin Walsh, and China Williams.

Related GAO Products

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-09-326SP. Washington, D.C.: March 30, 2009. Discusses the Department of Defense’s Joint Tactical Radio System— Handheld, Manpack, Small Form Fit and Warfighter Information Network—Tactical programs.

Information Technology: Census Bureau Testing of 2010 Decennial Systems Can Be Strengthened. GAO-09-262. Washington, D.C.: March 5, 2009. Discusses the Department of Commerce’s Decennial Response Integration System and Field Data Collection Automation programs.

NASA: Assessments of Selected Large-Scale Projects. GAO-09-306SP. Washington, D.C.: March 2, 2009. Discusses the National Aeronautics and Space Administration’s James Webb Space Telescope and Mars Science Laboratory programs.

Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage Information Technology Acquisitions, but Needs to Clarify Policy and Strengthen Oversight. GAO-08-756. Washington, D.C.: July 18, 2008. Discusses the Department of Transportation’s En Route Automation Modernization and Surveillance and Broadcast System programs.

Information Technology: Agriculture Needs to Strengthen Management Practices for Stabilizing and Modernizing Its Farm Program Delivery Systems. GAO-08-657. Washington, D.C.: May 16, 2008. Discusses the U.S. Department of Agriculture’s Farm Program Modernization program.

Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls. GAO-08-46. Washington, D.C.: October 25, 2007. Discusses the Department of Homeland Security’s Automated Commercial Environment program.

Defense Acquisitions: The Global Information Grid and Challenges Facing Its Implementation. GAO-04-858. Washington, D.C.: July 28, 2004. Discusses the Department of Defense’s Warfighter Information Network— Tactical program.