Earned Value Management (EVM) as an Oversight Tool for Major Capital Investments
Released: 2007-11-21T00:00:00

Oversight of Major Capital Investments

For the federal government, capital investments include development and acquisition of major capital assets, such as equipment (e.g., hardware for weapons systems, satellites, and information technology (IT)); structures (e.g., the Capitol Visitor Center); intellectual property (e.g., software); and combinations thereof. A capital investment can play a critical role, therefore, in the accomplishment of agency missions and the success of public policies. Congress has often taken a strong interest in the performance, cost, and schedule of major capital investments. In addition, Congress has taken a strong interest in establishing and maintaining accountability mechanisms to ensure that the President, Office of Management and Budget (OMB), and agencies fulfill statutory duties, accomplish prescribed public policy goals, report transparently and fully, and act as careful stewards of public resources.

Earned Value Management (EVM), which is the subject of this report, is a management technique that provides metrics to help inform assessments of whether a capital investment is "on track" from three perspectives: planned cost, time schedule, and functionality (the latter concept often referred to as "performance," "technical performance," "capability," "scope of work," or "planned value"). Using these three perspectives, EVM compares the planned cost, schedule, and functionality against an (ideally) accurate measurement of what is actually happening. Frequently, EVM metrics serve to raise potential issues that require further investigation or study, rather than provide the full picture of a project's progress and status. EVM may be of increasing salience in executive branch practices and, implicitly or explicitly, in presentations to Congress. EVM might also be used as an oversight tool for Congress.

This report provides an overview of EVM terminology and concepts by using a hypothetical example and illustrating a few potential oversight issues. To place EVM in context, however, it is necessary to discuss first the risks associated with major capital investments, how capital investments are often evaluated and monitored throughout their life cycles, and how EVM appears to be increasingly adopted by executive branch agencies. After providing the hypothetical EVM example and some related caveats, the report concludes with potential oversight questions for Congress regarding EVM.

Managing Risks and Evaluating Projects

Managing Risks of Adverse Events and Forgone Opportunities

The planning and implementation of a major capital investment are not simple tasks. Consequently, major capital projects in both the public and private sectors are considered to be inherently risky. For purposes of capital asset investment, risk might be described as the probability that adverse events will occur or favorable opportunities will not be exploited. Unless a project is carefully managed, risk might increase substantially in both of these senses. Adverse events that might occur include schedule delays, cost overruns, and performance shortfalls or failures. Favorable opportunities that might not be pursued (or noticed) could include chances to complete a project more quickly, at less cost, and with better than planned performance in achieving the mission or one or more goals. In response, agencies must make substantial efforts to plan effectively and manage risk in order to achieve successful outcomes. Given the complexity and inherent risk of major capital investments, however, it should be noted that problems cannot always be avoided, even in well-planned and implemented projects.

Many stakeholders—including Congress, the President, agency employees, and the public—often take a strong interest in the performance, cost, and schedule of major capital investments. The interest stems directly from the mission-critical nature of many investments. Furthermore, interest in specific capital investments can become heightened when something has gone wrong. Indeed, the consequences of a poorly managed capital investment project can be significant, and when investments in major capital assets fail, they can fail spectacularly.

Journalists, inspectors general (IGs), and the Government Accountability Office (GAO) are sometimes the first actors to disclose publicly that a major capital investment by the U.S. government has gone awry. Nonetheless, such disclosures are often lagging indicators of problems, rather than leading or coinciding indicators. Furthermore, capital investment problems themselves can be lagging indicators of more fundamental problems with management capacity, transparency, and accountability.

Monitoring and Evaluating Capital Investments

In tracking the planned versus actual progress regarding the planned cost, schedule, and functionality of a major capital asset, EVM attempts to address several significant monitoring and evaluation issues. EVM does not, however, address all important monitoring and evaluation issues that could contribute to an assessment of an investment's status and progress. Therefore, EVM's use as a management and oversight tool for major capital investments arguably should be considered within a larger context.

A variety of techniques can be used to monitor and evaluate major capital investments prospectively, concurrently, and retrospectively. Furthermore, these activities can occur throughout the life cycle of a capital asset—that is, during initial planning, the acquisition or development process, and operational use, as well as after disposal. Because EVM is intended to inform assessments of whether a capital investment is on track with respect to planned cost, schedule, and functionality, EVM takes place primarily during the acquisition process for a new asset (or modification of an existing asset), but can also occur during planning. Figure 1 provides illustrative lists of capital planning and programming activities that agencies often undertake during each of the four phases of a capital investment's life cycle. The figure also shows corresponding lists of the monitoring and evaluation activities, including EVM, that are often used to support the agency activities.

Increasing Executive Branch Adoption of EVM for Budgeting, Management, and Reporting

Related Statutory Directives and Requirements

Concerns about project risks are not new, nor are efforts to mitigate risks of major capital investments and prevent failures. For example, the first edition of OMB's Capital Programming Guide for agencies was published in 1997. Before then, Congress had enacted statutory provisions that were related to major capital investments, including, among others, a provision amending the Federal Property and Administrative Services Act of 1949 (41 U.S.C. § 263(a)), which now declares that "[i]t is the policy of Congress that the head of each executive agency should achieve, on average, 90 percent of the cost, performance, and schedule goals established for major acquisition programs of the agency"; provisions to require the Secretary of Defense and heads of all executive agencies to establish cost, performance, and schedule goals for major acquisition programs; and the Clinger-Cohen Act of 1996 ( P.L. 104-106 , later codified and amended), which gave OMB and agencies explicit duties for monitoring the progress of information system investments. The duties included a requirement for agencies to

provide the means for senior management personnel of the executive agency to obtain timely information regarding the progress of an investment in an information system, including a system of milestones for measuring progress, on an independently verifiable basis, in terms of cost, capability of the system to meet specified requirements, timeliness, and quality[.]

Budget Formulation, Management, and Contract Management

OMB has issued a number of requirements and directions to executive agencies that specifically concern EVM. For example, OMB included in its Circular A-11 —an annual document that communicates budget formulation and execution requirements to agencies—detailed directions for agencies to use EVM for major capital assets (Part 7). Among other things, the circular directed agencies to submit an "exhibit 300" ("Capital Asset Plan and Business Case Summary") for each "major investment" as part of their annual budget submissions to OMB, for the White House's use when formulating the President's annual budget proposal for congressional consideration. The circular also said that an exhibit 300 is "designed to coordinate OMB's collection of agency information for its reports to the Congress required by the Federal Acquisition Streamlining Act of 1994 (FASA Title V) and the Clinger-Cohen Act of 1996." The circular defined EVM as follows, citing an industry standard to which agencies would be expected to adhere:

Earned value management (EVM) is a project (investment) management tool that effectively integrates the investment scope of work with schedule and cost elements for optimum investment planning and control. The qualities and operating characteristics of earned value management systems are described in American National Standards Institute (ANSI)/Electronic Industries Alliance (EIA) Standard—748—1998, Earned Value Management Systems, approved May 19, 1998. It was reaffirmed on August 28, 2002. A copy of Standard 748 is available from Global Engineering Documents (1-800-854-7179). Information on earned value management systems is available at http://www.acq.osd.mil/pm .

Citing "room for improvement in the execution of our IT projects," OMB has also provided directions concerning major IT capital investments to agency Chief Information Officers in an August 2005 memorandum. The memorandum stated that, at the time, agencies were "already required to meet four principal criteria" for IT capital investments. The criteria included, along with two others, (1) the establishment and validation of a performance measurement baseline with clear cost, schedule, and performance goals; and (2) the management and measurement of projects to within 10% of baseline goals through the use of an earned value management system compliant with the ANSI standard cited in Circular A-11 . The memorandum also added new expectations for agencies, including directions to (1) ensure that cost, schedule, and performance goals for new major IT projects are "independently validated for reasonableness" before beginning development; and (2) provide, for all ongoing major IT projects with development efforts, for independent validations of then-current cost, schedule, and performance baselines by March 31, 2006, and submission of proposed changes of baselines for OMB approval.

EVM requirements have also been added to executive branch acquisition regulations. On July 5, 2006, three agencies promulgated a final rule on behalf of the Civilian Agency Acquisition Council and the Defense Acquisition Regulations Council to amend the Federal Acquisition Regulation (FAR). The stated purpose of the change was, among other things, to "implement earned value management system (EVMS) policy in accordance with OMB Circular A-11, Part 7 and the supplement to Part 7, the Capital Planning [sic] Guide" and "help standardize the use of EVMS across the Government" for "those parts of [an] acquisition where developmental effort is required." The FAR changes required agencies to use an EVMS for certain major acquisitions for development, in accordance with Circular A-11 , and potentially also for other acquisitions, in accordance with individual agency procedures. The new regulations also specified that "[t]he qualities and operating characteristics of an earned value management system are described" in the ANSI/EIA Standard 748 that was cited by Circular A-11 (see the extended Circular A-11 quotation, above). Under the regulations, agency contracting officers were directed to, "at a minimum," "require contractors to submit EVMS monthly reports for those contracts for which an EVMS applies."

In addition, OMB says it has used EVM data to determine agency scores under the George W. Bush Administration's initiative concerning the management of agencies in the executive branch—the President's Management Agenda (PMA). Although the PMA's standards are publicly available, PMA evaluation practices have not been fully transparent outside of OMB and the executive agencies. Detailed rationales and worksheets behind PMA scores are created, but not publicly available or independently validated.

Illustrative and Hypothetical Example of EVM Measurements and Issues

Purpose of Example

Based on published requirements, it appears that the Bush Administration and most executive branch agencies are widely adopting EVM for major capital assets for purposes of budgeting, management, and reporting. Specifically, as noted above, EVM has been formally adopted in requirements for preparation of the President's annual budget proposal, agency contract management, and formal scoring criteria under the Administration's PMA initiative. It appears, therefore, that EVM metrics will be presented either implicitly or explicitly to Congress in a variety of venues, sometimes packaged along with the President's views and representations about budgetary and management priorities. Due to the substantially public nature of the budget process, the metrics will similarly be presented either implicitly or explicitly to the public at large, some of whom play a significant role in the budget process through provision of information and opinions to elected officials.

It appears, therefore, that a potential ongoing issue for Congress will be judging whether underlying EVM metrics are fully, forthrightly, timely, and transparently reported. In addition, it appears that a corresponding issue for Congress might be understanding, interpreting, and scrutinizing EVM metrics, as potential inputs to congressional oversight, appropriations of funds, and authorization (or reauthorization) of agency and presidential activities. In each of these situations, EVM information might be relevant to informing Congress's assessments of whether acquisitions of major capital assets are on track with respect to planned cost, schedule, and functionality and, in turn, inform subsequent decision making.

With these considerations in mind, the following tables and corresponding text present a hypothetical example of EVM metrics being applied to the development and acquisition of a major capital asset. The purpose of explaining how EVM works through an example, here, is to give concrete illustrations of the types of basic analyses that can be performed with EVM, as well as to illustrate how EVM can be used to identify potential topics for follow-up investigation and study (e.g., perhaps with congressional oversight or complementary evaluations included in Figure 1 ). The type of asset described here could be any of those described at the beginning of this report, including a weapons system, satellite, structure, IT system, etc. The example presented here is adapted from an example that has been published in a variety of forms for a number of years in federal government websites and publications, but also draws on other resources, as noted below. Some EVM-related terms are italicized and explained as they are introduced.

Hypothetical Example: EVM for "Project B"

Explanation of Planned Value

For this hypothetical example, Congress has been presented with EVM data about a major capital investment called "Project B." The project has six work units , represented by the letters "A" through "F," which are interrelated, but distinct subcomponents of the project. For a simple example relating to a structure, work units might include concrete, framing, roofing, electrical, plumbing, and interior. Collectively, work units A through F are also referred to as the project's work breakdown structure (WBS). In this example, Project B has been underway for a period of time already, and all the work units were planned to be successfully completed by the time of this EVM report. The original plan for Project B's costs, schedule for completion, and functionality is referred to as the project's baseline , identified here as B 1 (see Table 1 ). Later in the life of the project, it is possible that the project could be "re-baselined" (baselined again with new cost, schedule, and functionality plans, in order to, for example, conform to a management decision or reflect new circumstances).

To summarize the contents of Table 1 , the baseline plan B 1 for Project B shows that six work units (A-F) are planned to be completed at an overall cost of $100 for the time period covered by this report. Each work unit is listed with its own planned budget—that is, the amount of money that is planned to be spent to successfully build and complete the functionality planned for the work unit. Work unit D, therefore, is planned to be successfully completed within the time period covered by this EVM report at a cost of $25, and work unit E is similarly planned to be completed at a cost of $20. These numbers have a special name, planned value , that this report will also generically call a metric . (Other metrics will be introduced as this section progresses.) In a sense, when the agency that is paying for Project B and the agency's contractor agree to plan that $25 will be spent to successfully complete work unit D, it could be said that the functionality that corresponds to work unit D has taken a monetary value of $25. Assigning work unit D's functionality a quantitative value (denominated in dollars according to the planned budget for that functionality), in turn, will allow a quantitative representation of how much of work unit D may have been completed at any given point in time, as explained below.

Explanation of Earned Value and Schedule Variance

Table 2 adds two rows of EVM data, with two new metrics, to the previous table

The first row of data in Table 2 shows the same planned value that was portrayed in Table 1 for Project B and its constituent work units, according to the project's planned baseline B 1 . Note that the planned value for Project B's various work units continues to sum to $100, the overall cost of the contract to the agency. The second row is new. This row is intended to show how much functionality has actually been successfully developed by the date this EVM report was produced. Work unit A, for example, is shown with an "earned" value (denominated in dollars) of $10. This means that the functionality that was planned for work unit A has been successfully completed or earned, and becomes earned value , not merely value that was planned. As work is successfully performed, therefore, it is earned on the same basis as it was planned. Stated differently, work unit A has earned the full functionality that was planned, and the work unit and overall project are, therefore, credited with $10 of value that has been earned for purposes of work unit A.

Looking across the table, one sees similar situations with work units B, C, and E. In each case, the functionality that was planned for these work units has been successfully and fully developed, or earned. Work units D and F, however, are another story. Table 2 shows that work unit D was partially, but not fully, completed, and also that work unit F was never started. Only $10 worth of functionality, out of work unit D's planned value of $25, has been accomplished. In other words, only 40% of the work associated with work unit D has been successfully completed, and 60% of the work has not been successfully completed. As a result, work unit D is said to have a schedule variance of -$15 ($10 earned value minus $25 planned value equals -$15 schedule variance). With EVM, the term variance , therefore, implicitly means difference from the relevant plan (or baseline). The variance is denominated in dollar terms, because dollars (i.e., dollars of planned value) represent the functionality that is planned to be developed and, eventually, either delivered or not delivered. With only $10 of earned value, work unit D is $15 behind schedule, compared with the baseline B 1 planned value of $25. Or, equivalently expressed in percentage terms, work unit D is 60% behind schedule. Because work unit F was never started, work unit F has a schedule variance of -$20, and is 100% behind schedule. When work units A through F are summed, out of the planned value of $100 for Project B, only $65 of the planned functionality has been successfully developed, thanks to work units D and F falling behind schedule. Overall, Project B is therefore said to have an earned value of $65, $35 lower than the project's planned value, and is said to have a schedule variance of -$35, or in percentage terms, a 35% schedule variance (i.e., 35% behind schedule compared to what was planned to be accomplished). The 35% schedule variance might also be understood as a shortfall in the functionality that had been planned to be delivered within the time period of the EVM report.

An explanation of why work units D and F were not successfully completed, however, is not captured with EVM data. To construct such an explanation, other evaluation and monitoring methods from Figure 1 might need to be explored. In addition, it would probably be possible to use qualitative and quantitative information from individuals involved in the project, and potentially also from stakeholders, to construct interpretations of what might have caused the slip in schedule. As GAO has cautioned, "[i]t is important to understand that variances are neither good nor bad. They are merely measures that indicate that work is not being performed according to plan and that it must be assessed further to understand why."

The schedule variance metric discussed above (i.e., the difference between planned value and earned value) only captures part of the overall story of Project B: whether the project is on schedule or behind schedule. One has yet to look at whether the project is on track with regard to its actual cost, in comparison to the baseline B 1 . The question of whether Project B is on track with its planned cost is the subject of Table 3 , below.

Explanation of Actual Cost and Cost Variance

Table 3 presents two additional EVM metrics that are necessary to assess whether a project is experiencing cost overruns or coming in under budget compared to the baseline. For each work unit, the table compares the value of the functionality that was successfully delivered (earned value, which was discussed above) with the amount of money that was actually spent to achieve that functionality ( actual cost , the first of the two new metrics). By subtracting the actual cost from the earned value, one calculates the second new metric: cost variance .

Some illustrations provide intuition for understanding cost variance for individual work units and also the overall project. In Table 3 , the first row of EVM data shows that work unit A was successfully delivered, creating earned value of $10. The second row of EVM data shows that the actual cost of work unit A was only $9, or $1 under the earned value. Subtracting the $9 actual cost from the $10 earned value results in a $1 cost variance. A positive number for cost variance, therefore, means something was delivered under budget. Work unit C similarly came in under budget, as shown by its positive cost variance.

Work units B, D, and E, by contrast, show that cost overruns have occurred. A negative cost variance means that a work unit (or portion of a work unit) was delivered over budget: more money was spent for the work successfully accomplished than was planned. To achieve the full functionality that was delivered for work unit B (represented by the work unit's earned value), the actual cost was $22, considerably higher than the original budget (planned value). The cost variance for work unit B is -$7, representing a cost overrun. In percentage terms, the cost overrun can be calculated by dividing the cost variance (i.e., -$7, the excess amount spent over the budgeted amount) by the earned value (i.e., $15, the value of the functionality that was delivered). The cost overrun was, therefore, nearly -47%.

For work unit D, the picture looks even worse. In developing the functionality represented by $10 of earned value (only a portion of the hoped-for functionality represented by $25 in planned value), the actual cost was $30. That is, only 40% of work unit D's planned functionality was delivered successfully, but at a cost that exceeded the budget for the entire work unit. Work unit D's cost variance is -$20 ($10 earned value minus $30 actual cost), or -200%. The intuition behind the -200% figure is as follows. Only a portion of work unit D was successfully delivered (i.e., $10 out of work unit D's overall planned value of $25 was successfully delivered, becoming $10 in earned value). Looking at work unit D, overall, the portion of the work unit's functionality that was delivered successfully ended up costing three times more than what was planned ($30 actual cost instead of $10), which is a 200% overrun for that portion of work unit D that was successfully completed.

Finally, one might assess work unit F. Because work unit F was not started, no money was spent on the work unit, resulting in a cost variance of zero.

Looking at Project B overall, in terms of its B 1 baseline, the earned value for the major capital investment is $65 (nearly two-thirds of the planned functionality). In achieving $65 in earned value, however, $91 was spent. Stated differently, the portion of the overall project that was performed successfully (i.e, the $65 earned value) was, by definition, originally planned to cost $65 (i.e., the portion had a planned value of $65), but in the end, was completed at an actual, overall cost of $91. These metrics result in an overall cost variance for the project of -$26 (earned value of $65 minus an actual cost of $91), or exactly -40% (-$26 cost variance divided by the earned value of $65). Project B, therefore, could be said to have experienced a 40% cost overrun on the capability that has been successfully delivered. The capability that was successfully delivered in the time period of the EVM report, in turn, was substantially less than the capability that was planned to be delivered in the time period (-35%, the schedule variance, which shows a schedule slip and corresponding functionality shortfall).

EVM as More Useful Tool than Simple Spend Comparisons

EVM tutorials often compare EVM metrics with an oversight tool that is often considered not useful: the "spend comparison." The typical spend comparison approach, whereby contractors report actual expenditures against planned expenditures, is unrelated to the functionality that was successfully delivered (see Table 4 ).

Table 4 shows a simple comparison of planned spending and actual spending, which is unrelated to work performed and, therefore, potentially misleading and not a very useful comparison. The fact that the overall amount spent was $9 less than planned for this period is not useful for most purposes, because the metrics lack comparisons of spending with the functionality that was successfully delivered.

In Sum: EVM Metrics as Oversight Tools

Considered together, Tables 1 , 2 , and 3 cover central aspects of EVM: comparing the planned cost, time schedule, and functionality for a major capital investment against the actual cost, time schedule, and functionality of what was successfully delivered. Many observers emphasize that EVM might help facilitate the correction of problems during the investment process. For example, as one prominent federal agency website asserts, from a practitioner perspective, "[t]he benefits to project management of the earned value approach come from the disciplined planning conducted and the availability of metrics which show real variances from plan in order to generate necessary corrective actions."

Another potential benefit of EVM, from an oversight perspective, is to provide a picture of the status of a major capital investment at a "snapshot" in time. EVM calculations might enable project managers, contractors, program managers, overseers, and outside stakeholders to see quickly and simply whether or not a project is "on track" with its baseline plan. A negative schedule variance or cost variance indicates a schedule slip or cost overrun, whereas positive variances indicate a project is ahead of schedule or under budget. EVM metrics might, thereby, suggest the need for additional analysis in targeted ways to explain the root causes of problems and what could be done, if anything, to get a project back on track, or even provide evidence of more fundamental management, resource, and oversight problems. EVM metrics do not, however, necessarily tell the whole story, as discussed below.

Caveat: EVM Metrics Might Tell Only Part of the Story

The story that EVM tells can be a significant one. Once the basic jargon and concepts of EVM are understood, EVM provides a visually simple picture, on a periodic basis, of whether or not a project is on track with plans. For oversight purposes, EVM can thereby "flag" outliers and signal the potential need to ask further questions. The prospect of this scrutiny might, in turn, provide incentives to OMB and agencies to, among other things (1) closely monitor the progress of major capital investments; and, more fundamentally, (2) build the capacity (e.g., resources, staff, skills, management attention and discipline, and processes) for OMB and agencies to properly manage investment-related activities and evaluations.

Nonetheless, without further evaluation of a project, EVM metrics might tell only a partial story. For example, even when EVM metrics are accurately captured and portrayed, the metrics typically will not reveal why a project might be experiencing schedule or cost variances. A process or implementation evaluation might be necessary to address such questions (e.g., to verify and complement explanations that are provided by contractors). In addition, other evaluation methods (see Figure 1 ) are typically used to assess other significant and, sometimes, more fundamental questions, including (1) whether a project is worth undertaking (or continuing); and (2) after the investment process is completed, what impact an investment might be having on achievement of an agency's mission or the success of a public policy, compared to what would have happened without the investment.

In addition, a project's baseline plan and corresponding EVM data might not accurately represent the cost and schedule that are most likely necessary for a project to achieve a particular functionality. Cost, schedule, and functionality plans are, of course, made in the context of complexity and uncertainty, which oftentimes can guarantee that a capital investment project will not proceed according to plan. Alternatively, estimates of a project's costs, schedule, functionality, and benefits might be based on insufficient research and analysis. Furthermore, the use of metrics can create powerful incentives, including perverse incentives. One project management reference presents a humorous and skeptical, albeit seriously intended, scenario to caution against too quickly accepting EVM metrics at face value as accurate representations of the full picture. The scenario is worth excerpting at length—not as a portrayal of what is typical—but rather as an illustration of an overseer interpreting EVM metrics; thinking about incentives; and confronting the issue of what a "reasonable" plan should be for a project's cost, schedule, and functionality. Specifically, the author posits a scenario showing a

project is [quite substantially] ahead of schedule and underspent.... At first glance, this looks wonderful. But ask yourself how the variance happened. There are three possible explanations for how this project manager achieved the results shown:

1. Actual labor [costs] were considerably lower than expected and/or the people were more efficient than anticipated.

2. The project team had a "lucky break." The team had expected to have to work really hard to solve a problem, but it turned out to be very easy.

3. The project manager "sandbagged" his estimates. He padded everything, playing it safe.

If you believe situation 1, you will believe anything. It is unlikely that both variances would happen at the same time. Situation 2 happens occasionally. When all the planets are aligned—about once in a zillion years, you say. You bet!

Situation 3 is the most likely explanation. The project manager was playing it safe. And he would tell you that there is no problem. After all, if he continues along this course, the project will come in ahead of schedule and underspent, which means a manager will give money back to the company. No problem.

But is that true? First of all, I can almost guarantee that the project manager won't give any money back. He will find a way to spend it—by adding bells and whistles to the project, by buying unplanned equipment and supplies, or by throwing one huge party! No sane project manager wants to give the money back, because he knows that his next project will be cut.

However, suppose the manager did give back the money. Would that be okay? No. The reason is that the organization would have lost the opportunity to use the money to fund some other project.... But the money is available. It is tied up in the under-budget project. If that project were rescheduled and rebudgeted, the money would be available to keep the other project going, assuming that the [return on investment] is still justified.

The question is, naturally, "What is reasonable?" We certainly cannot expect to have zero variances in a project. And this is true. It all depends on the nature of our [commercial sector] business. Well-defined construction projects can be held to very small tolerances—as small as plus-or-minus 3 to 5 percent. Research and development projects are likely to run higher tolerances, perhaps in the range of 15 to 25 percent. Each organization has to develop acceptable tolerances from experience.

Whether the author's scenario and rules of thumb are fully applicable to federal government projects generally, much less in specific policy and technology domains, is debatable. Nevertheless, his scenario illustrates some of the psychological and implementation phenomena that might be at play in the context of major capital investments. In a federal government context, phenomena like these might be at play with any participating actor (e.g., advocate, critic, project manager, funder, contractor, overseer) at any point in the process of researching, proposing, funding, executing, operating, and evaluating major capital investments.

For example, who says the budget and planned functionality corresponding to a proposed project are accurate? What analyses are the judgments based upon? Are cost and benefit estimates reasonable? Who might have the capacity and independence to make such an assessment? Do agencies, inspectors general, or OMB have the staff resources to perform adequately the necessary analytical and monitoring tasks for major capital investments? Are the President, political appointees, and career civil service executives making decisions based on, or in spite of, reasonable and objective analysis and monitoring? What are the real reasons a project is under or over budget, or ahead of or behind schedule? How much scrutiny is too much? What perverse incentives and consequences might be created when project managers make good faith attempts to estimate costs, schedules, and functionality, but variances inevitably occur, perhaps due to uncertainty or bad luck, bringing high levels of distracting scrutiny to a project and its sponsoring agency or agencies?

Questions such as these are difficult to answer, but suggest that answers and corresponding lessons might be learned for specific policy areas and types of investments over time and with experience. This observation raises two further questions. For each type of actor involved in planning, funding, implementing, evaluating, and overseeing major capital investments (e.g., project managers, program managers, political appointees, career executives, congressional staff, Members of Congress, and the President), what are the necessary prerequisites to learning these lessons from experience? That is, what capacities and processes need to be in place in order to learn from experience? How many of these prerequisite capacities and processes are currently in place?

Potential Oversight Questions for Congress

Major capital investments give rise to many actual or potential issues for Congress. Issues that are potentially related to EVM are a significant subset of this larger picture. With the increasing emergence of EVM as a tool for gauging the status and progress of major capital investments, potential congressional oversight issues appear likely to continue emerging, as well. This report culminates with a sampling of potential oversight questions for Congress that draw on the report and other resources, as noted.

In light of what appears to be more widespread adoption of EVM practices for major capital investments by executive agencies and their contractors, what might be the advantages, disadvantages, costs, and benefits of legislatively requiring more formal, periodic, and publicly accessible agency reporting of EVM metrics for some, most, or all major capital investments? To what extent might recent congressional efforts to increase budget and financial transparency—for example, the establishment of a searchable website for federal contracts and grants required by the Federal Funding Accountability and Transparency Act of 2006 (FFATA; P.L. 109-282 ; 120 Stat. 1186)—be viewed as an illustration of what could be done with EVM information? What would be the advantages and disadvantages? The usefulness of EVM metrics depends upon the quality and reliability of the cost, schedule, and functionality data that underlie both a project's baseline plan and its reports on work performed. Given concerns that have been raised about the quality of agency capital investment business cases that are used to justify budget requests, as well as concerns that have been raised about the quality of agency project management, what might be options for Congress to cause OMB and agencies to improve any deficiencies in the underlying analyses and data that support capital investment decision making and reporting? What might be the advantages and disadvantages of requiring independent, publicly accessible assessments—including verification and validation, as appropriate—of the underlying quality of EVM data, including cost estimates and assessments of the functionality that is delivered in comparison with what was planned? What might be the advantages and disadvantages of requiring independent, publicly accessible assessments of the capacity of agencies to manage the capital planning and investment control (CPIC) process (e.g., agency capacity to research, formulate, plan, propose, develop, report on, operate, and evaluate major capital investments), which is arguably more fundamental as a driver of EVM data quality? Would assessments of key capabilities be amenable to systematic oversight through the use of "scorecards," similar to those used under the Bush Administration's President's Management Agenda, albeit with transparency outside the executive branch and independent validation? To the extent that OMB currently identifies gaps in agency capabilities, to what extent do annual budget requests (submitted to Congress by the President) and performance plans (produced by agencies under the Government Performance and Results Act of 1993, or GPRA; P.L. 103-62 ) address the gaps? Processes that support the capital investment process—including development of capital investment business cases, project management, EVM, contract oversight, management auditing, and evaluation—depend for success, in large part, upon adequate staff and skill resources in agencies. Have agencies allocated sufficient staff and financial resources to successfully undertake major capital investments? How might such judgments be made? Also, do agencies have sufficient ability to recruit, retain, and train personnel in disciplines including project management, risk management, contract management and oversight, information technology, and program evaluation? If so, what are their track records in recruiting, retaining, training, and allocating staff in sufficient number and competency in the management and oversight of major capital investments? If not, what could be done? What might be the advantages, disadvantages, costs, and benefits of applying EVM-related requirements to major capital investments, as appropriate, pursued by agencies in the legislative and judicial branches?