Architect of the Capitol: Incorporating All Leading Practices Could Improve Accuracy and Credibility of Projects' Cost Estimates
Released: 2014-03-25T00:00:00

Letter

Background

Cost-Estimating Leading Practices

The GAO Cost Estimating and Assessment Guide explains how four characteristics of a high-quality, reliable cost estimate can be understood in relation to 12 leading practices. The extent to which an agency meets the leading practices underlying each characteristic determines its For example, we consider the performance for that characteristic.comprehensive characteristic to be “substantially met” if the organization substantially meets the underlying leading practices of (1) developing an estimating plan and (2) determining an estimating structure. Because the leading practices are separate and discrete, an agency’s performance in each of the characteristics can vary. For example, an organization’s cost estimating methodology could be found to be comprehensive and well documented, but not accurate or credible, resulting in the organization producing cost estimates of limited reliability. Table 1 illustrates the relationship of the 12 leading practices to the four characteristics of a high-quality, reliable estimate.

AOC’s Project Development

AOC project development consists of three stages: planning, design, and construction. According to AOC’s guidance, AOC refines requirements and updates cost estimates as projects progress through these stages. AOC has guidance and requirements for cost estimates at each stage of a project’s development. In general, as projects develop over time and requirements are refined, the accuracy of cost estimates is expected to increase.

Given the current fiscal environment and existing building conditions, AOC must prioritize projects in its capital program and decide to either request funding for a project or defer it while mitigating potential facility issues. For example, in fiscal year 2014, AOC requested almost $155 million for 17 projects that AOC deemed urgent, while deferring 46 projects estimated to cost about $172 million.most recognizable projects that it considers urgent are the Cannon Building renewal and the Capitol Dome restoration.

Cannon Building’s Renewal

The Cannon Building, completed in 1908, is the oldest congressional office building and is occupied by members of the House of Representatives and their staffs. (See fig. 1.) The building houses 142 office suites, 5 conference rooms, 4 hearing rooms, and the Caucus Room, which can accommodate large meetings. The building also provides space for a library, food service, and a health unit.

According to AOC planning studies, the building is plagued with serious safety, health, environmental, and operational issues that are worsening. For example, without action, essential systems for heating and cooling that are located behind walls and in mechanical rooms will continue to deteriorate, potentially negatively affecting Members of Congress, staff, and constituents. AOC has been developing the scope of the Cannon Building’s renewal project since approximately 2004 when AOC’s consultant conducted a facility condition assessment (FCA) that identified the building’s deficiencies. In 2009, we reviewed AOC’s progress in developing the Cannon Building project and found that while it had followed a reasonable process to plan the building’s renewal such as through updating the FCA, it was important that AOC continue, as planned, to refine the project’s scope and cost estimate.review, AOC has proceeded with the design phase of the project, based Since our 2009 upon its budget of $753 million for the planning, design, and construction phases. The project is currently expected to be completed by 2025.

According to AOC officials and current design documents, AOC plans to correct most of the Cannon Building’s identified deficiencies and to address requirements such as energy conservation, physical security, hazardous materials abatement, and historic preservation. The project is to involve substantial reconfiguration of interior and exterior spaces to include reconstructing the building’s top floor, which now partially consists of storage space, and landscaping the courtyard. The project is also expected to provide refurbished windows and a new roof. Additional work is intended to preserve and repair the building’s stone exterior. The project is also expected to allow for complete replacement of all plumbing, heating and cooling, fire protection, electrical, and alarm systems; refurbish restrooms and make them more accessible to people with disabilities; and provide new wall and floor finishes in some areas. In addition, the project includes removing asbestos that may be contained in plaster ceilings and walls. AOC plans to conduct the work in phases corresponding to the four sections of the building and including an initial phase for utility work as shown in figure 2. Tenants displaced during construction of each section are to move to temporary offices while other occupants will remain in building sections not affected by construction.

Capitol Dome Restoration

The U.S. Capitol Dome, an important symbol of American democracy and an architectural icon, was constructed of cast iron more than 150 years ago. According to AOC, the dome has not undergone a complete restoration since 1960, and due to age and weather is now plagued by more than 1,000 cracks and deficiencies that are causing it to deteriorate. Figure 3 shows cracks at the exterior column base and deteriorating interior paint.

The project is intended to stop deterioration in the dome’s cast iron structure as well as to ensure the protection of the interior of the dome and rotunda. In the 1960 restoration, the dome was stripped of its paint so the ironwork could be repaired, primed with a rust inhibitor, and then repainted. As part of the current project, AOC is undertaking similar restorative work to include removing old paint, repairing the cast iron, and repainting.

The project has proceeded in phases as shown in table 2, with phase IIA restoration work currently in progress. In earlier phases, AOC completed interim painting, revalidated the project’s design, and restored the base— or skirt—of the dome. To complete the project, AOC is seeking appropriations of about $20 million for construction and other costs for phase IIC work in fiscal year 2015. As of February 2014, AOC estimated the total cost of the Capitol Dome restoration project at about $125 million.

AOC’s Policies and Guidance Conform to Leading Practices for Two Characteristics of Reliable Cost Estimates, but Fall Short in Two Others

As previously discussed, the GAO Cost Estimating and Assessment Guide defines 12 leading practices related to four characteristics— comprehensive, well documented, accurate, and credible—that are important to developing high-quality, reliable estimates. Our analysis determined how well AOC met a characteristic based on our assessment of AOC’s conformance to the leading practices related to that characteristic. We discuss characteristics using five rating categories— does not meet, minimally meets, partially meets, substantially meets, or fully meets. As shown in table 3, our analysis found that AOC’s cost- estimating policies and guidance contribute to estimates that are comprehensive and well documented in that AOC fully meets most of the tasks that underlie the leading practices associated with these two characteristics. We found that AOC’s cost-estimating policies and guidance partially met the accuracy characteristic and minimally met the credible characteristic based on their conformance to the leading practices associated with those characteristics. Table 3 also contains key examples of our rationale for our assessment of each leading practice and characteristic.

Characteristic Leading Practice Comprehensive Develop the estimating plan AOC has a formal process that develops the estimating plan, Key examples of rationale for assessment including describing responsible parties and defining specific tasks. The estimating structure follows a work breakdown structure (WBS)

AOC uses different types of cost estimates and has defined when each applies and what each entails.

AOC has formally defined program characteristics.

AOC points to supporting documents that, when developed, should contain ground rules and assumptions specific to a given project.

AOC guidance is specific about sources of data. These include commercially available construction cost databases as well as market and industry knowledge, staff knowledge, and historical data.

AOC Conforms to the Leading Practices Associated with Comprehensive and Well Documented Cost Estimates

Comprehensive

In reference to the leading practices underlying the characteristics of a comprehensive cost estimate, we found that AOC has a formal process for developing estimating plans, and follows an estimating approach incorporating a work breakdown structure (WBS) that is widely used in the construction industry.

In developing estimating plans, we found that AOC’s policies and guidance that describe the project-planning process provide the framework for meeting this leading practice. In the planning stage, AOC project managers establish project scope and outline roles and responsibilities for planning, design, and construction stages. To facilitate these efforts, AOC project managers are to use particular documents. One document, the Project Development Form, provides a basis for preparing requirements studies that help to define scope and estimate costs. AOC then is to use this information in developing Project Management Plans that are to describe other key components of project delivery. These components include defining the strategy for providing design and construction services, identifying project team members, establishing a communications plan, and setting project controls such for managing project changes. AOC also considers cost reporting as a component of the project delivery process. For example, AOC typically requires its design contractors to provide cost reports at the same time incremental design submittals are made. Because AOC’s project development process accounts for cost reporting, we determined that it satisfies the intent of the leading practice encouraging use of a formal process for developing estimating plans.

Well Documented

With regard to leading practices related to producing well-documented estimates, we found that AOC requires formal definitions of programprogram’s purpose; characteristics, such as the develops ground rules and assumptions from supporting documents; requires the use of cost data sources that are specific to the construction industry (i.e., obtain the data); requires documenting the estimate and supporting documents; and requires management approval of the cost estimate.

AOC fully meets the requirements of most of these leading practices. For example, AOC’s policies and guidance for the project development process provide the structure to enable an adequate understanding of program characteristics—such as key design features, technical definitions, and the acquisition strategy—that will comprise the cost estimate. In addition, AOC guidance establishes requirements for use of industry-accepted cost-data sources and allows for application of staff knowledge and historical data in developing estimates. AOC guidance also provides for estimates to be documented to show important parameters, assumptions, descriptions, methods, and calculations used to derive the estimate. However, while AOC requires management approval of its cost estimates, it falls short of fully meeting the requirements of this leading practice. In particular, AOC’s briefings do not include the level of information defined by our leading practice, particularly information related to risks associated with the underlying data and methods. This is due, in part, to AOC’s policies and guidance not requiring a risk and uncertainty analysis in developing estimates. We discuss this leading practice in the following section of this report.

AOC Partially and Minimally Conforms to the Leading Practices Associated with Accurate and Credible Cost Estimates, Respectively

We found that AOC policies and guidance do not require enough detail to suggest that resulting estimates would be accurate and credible.

Accurate

In reference to accuracy, we found that AOC’s policies and guidance partially meet the respective underlying leading practices pertaining to this characteristic. According to AOC’s policies and guidance, AOC is to develop estimates as projects progress through planning and design phases that are based on sufficiently detailed documentation of construction requirements. AOC is to then use these estimates to support budget requests for construction funding. However, AOC’s guidance does not require that cost estimates be updated with actual costs during a project’s construction phase—a leading practice. According to the Cost Guide, updating estimates to reflect actual costs as the project progresses allows agencies to review variances between planned and actual costs and provides insight as to how the project changed over time.

Credible

In reference to credibility, we found that AOC’s policies and guidance minimally meet the underlying leading practices pertaining to this characteristic. For example, we found that AOC’s guidance does not require following all steps for conducting a “sensitivity analysis”determining the estimate’s reasonableness, and conducting a risk and uncertainty analysis.

While AOC guidance does discuss conducting some sensitivity analysis, it skips many associated tasks of leading practices we identified, such as identifying key cost drivers, ground rules and assumptions for sensitivity testing, and evaluating the results to determine which drivers most affect the cost estimate. Similarly, while AOC guidance provides that contingencies be added to estimates to account for risk and uncertainty, AOC’s guidance does not provide documented reasons explaining how the actual budgeted amounts for unforeseen costs were developed. AOC officials told us that while their policies and guidance do not require sensitivity or “quantitative-risk and uncertainty analyses,” they perform such assessments qualitatively to establish budget contingency. However, the leading practice to determine whether a program is realistically budgeted is to perform a quantitative-risk and uncertainty analysis, so that the probability associated with achieving its point estimate can be determined. The results of a quantitative-risk and uncertainty analysis—the range of costs around a point estimate—can be useful to decision makers because it conveys the confidence level in achieving the most likely cost and informs them about cost, schedule, and technical risks. Not having an understanding of an estimate’s confidence level limits AOC’s ability to determine the appropriate level of contingency that is needed to address risks and uncertainty for a particular project and could lead AOC to ineffectively allocate resources across competing projects if contingency levels are either overstated or understated. In addition, absent information on estimates’ confidence levels, Congress will not have critical information for making well-informed funding decisions.

In addition, pertaining to the leading practice of developing a point estimate and comparing it to an independent cost estimate (ICE)—which affects both accurate and credible characteristics—we found that the independent estimates defined by AOC’s estimating process have a limited degree of independence and focus solely on proposed contractor costs rather than on the entire cost estimate, including both government and contractor efforts. According to our leading practices, an ICE, conducted by an organization outside the program office, provides an objective and unbiased assessment of whether the agency’s program estimate can be achieved. However, because AOC’s project management and cost-estimating functions are performed within the same organizational group, the estimates could potentially be influenced by AOC’s project managers and therefore could be susceptible to bias.

Because of the weaknesses in AOC’s policies and guidance pertaining to two of the four characteristics that are important to developing high- quality, reliable estimates, the project cost estimates that AOC produces may not always be reliable. Without reliable cost estimates, AOC’s projects risk experiencing cost overruns or budget surpluses, missed deadlines, and performance shortfalls. Furthermore, potential limitations in the reliability of its estimates may impair Congress’s ability to make well-informed funding decisions and affect how AOC allocates resources across competing projects in its capital program.

Strengths and Weaknesses in AOC’s Policies and Guidance Are Reflected in the Cannon Building and Capitol Dome Projects

In comparing the Cannon Office Building’s renewal and Capitol Dome’s restoration cost estimates to GAO’s leading practices, we found strengths and weaknesses that generally correspond to our assessment of AOC’s overall policies and guidance for developing cost estimates. We initially determined that both estimates were comprehensive while lacking, to a varying extent, in their documentation, accuracy, and credibility. Following our initial determination, AOC provided further documentation that resulted in improvements to each estimate’s assessment. Our final assessment found that the Cannon Building renewal estimate to be substantially comprehensive, well documented, and accurate while lacking in elements affecting its credibility, and the Capitol Dome restoration estimate to be substantially comprehensive and well documented while lacking in areas pertaining to accuracy and credibility.

In aggregate, because of weaknesses pertaining to characteristics that are important to the development of high-quality, reliable estimates, AOC’s cost estimates for the Cannon Building’s renewal and Capitol Dome’s restoration may not be fully reliable. Appendix III provides additional details on our comparison of these two cost estimates to our leading practices.

Cannon Building’s Renewal Estimate

Appendix III provides additional information on our assessment of these estimates compared to the characteristics of high-quality, reliable estimates. and allows for tracking of cost and schedule performance by defined elements of work. In addition, AOC conducted a life-cycle-cost analysis of the heating and cooling system alternatives it considered to include analyzing their energy consumption for the purpose of comparing operational costs. However, the estimate did not fully meet the characteristic as it did not include full life-cycle costs, from inception through design, construction, operation, and maintenance. Without a life- cycle cost analysis that captures the total cost of the project, AOC cannot evaluate design alternatives on a total-cost basis.

In terms of being well documented, we found that the estimate substantially met associated leading practices. In general, the information provided by AOC describes how the estimate was built up from engineering drawings, specifications, and design documents. In addition, there was evidence of documented management approval. However, the documentation AOC provided did not contain source data, such as from contractor bids or cost estimating databases. While AOC officials said they could obtain this information for our review, to the extent that it was available, our leading practices indicate that an estimate’s documentation should be detailed enough so that the derivation of each cost element can be traced to all sources allowing for the estimate to be easily replicated and updated. Because AOC could not readily provide the actual source data for the estimate, we determined that it did not fully meet the requirements of this characteristic.

For the accuracy characteristic, we found the estimate substantially met leading practices. AOC’s acquisition approach to the building’s renewal involves its contracting with an architect, construction manager, and construction contractor, each of whom produced separate estimates. These estimates have a limited degree of independence because they were conducted for the same program office and focus only on the proposed contractor cost. However, this array of estimates has enabled AOC to make comparisons among them, determine similarities and differences, and develop a reasonably accurate assessment of estimated costs.

For the credibility characteristic, we found the estimate partially met leading practices. During the course of our review, AOC conducted a risk and uncertainty analysis in accordance with a key leading practice for this characteristic. However, we found several issues affecting the quality of the analysis that AOC provided to us. For example, AOC’s analysis concluded that the Cannon Building’s renewal estimate had a confidence level that exceeded 90 percent—meaning there is a greater than a 90 percent probability that actual costs will be equal to or less than AOC’s estimate—which may be unreasonably high. However, an AOC official said that the agency will be reconsidering the confidence level once the design progresses further. In addition, we found that the method AOC used to model the project’s risks it identified (1) resulted in an unusually narrow range of estimated costs across the confidence intervals and (2) provides managers limited ability to understand the effects of individual risks.opposed to modeling them separately, AOC cannot identify relationships between risk elements and determine which risks have the greatest influence on project costs. As a result, AOC is limited in its ability to manage the risks. In addition, modeling the aggregated risks likely contributes to the overly narrow range of estimated costs over the Because AOC aggregated risks for the purposes of its analysis as confidence levels, which implies that AOC’s analysis overstates the effect of the risks.

Capitol Dome’s Restoration Estimate

In reference to the comprehensive characteristic, we found that the estimate substantially met leading practices. In particular, the estimate uses an industry-standard WBS format and contains sufficient detail of the technical characteristics of the project. However, the estimate did not fully meet leading practices, in part because we did not find a consolidated list of ground rules and assumptions or descriptions of how these affected the estimate. Contract documents provide indications of ground rules and assumptions affecting the project. For example, contract specifications and drawings for the recently-awarded phase II work describe assumptions pertaining to availability of funding for contract options and accessibility of the site during the workday. However, we could not determine if these represented a comprehensive assessment of all ground rules and assumptions. According to the Cost Guide, because ground rules and assumptions can significantly affect cost and introduce risk, it is important that they are clearly documented in the estimate to enable areas of potential risk to be identified and resolved.

For the well documented characteristic, we determined the estimate as having substantially met leading practices. We found, for example that cost elements from the estimate can be compared to information in the drawings and specifications that define the contract for the phase II restoration. This linkage enables a good understanding of key characteristics of the estimate. In addition, AOC attested that the estimate had been briefed to and approved by management. However, because records describing briefings of the estimate to management are not well documented, it is difficult to trace management’s recommendations for changes, feedback, and key decisions affecting the project. While AOC officials told us that its project staff and management communicate routinely as part of their normal business functions, documenting management briefings is important because some key personnel have changed over the life of the Capitol Dome restoration project, which originated more than 15 years ago. By not having well documented records of management briefings over the life of the project, AOC risks loss of continuity in its oversight.

Considering the accuracy characteristic, we determined that the estimate partially met leading practices. We found, for example, that the estimating software and database used to construct the estimate included standard information, such as costs for labor, materials, and equipment. However, AOC did not consider the actual cost of previously completed restoration phases in updates to its estimate for the phase II work. While AOC used information on the price it paid for certain items in the phase I restoration work that are also part of the phase II project, it did not determine the contractor’s actual cost to perform the work. AOC officials said that having information on actual contractor costs for items may be of limited usefulness because market conditions and other factors determine what contractors bid and AOC ultimately pays. However, not having information on actual costs makes it difficult for AOC to assess the difference between the price it paid and its contractors’ costs to determine the estimate’s reasonableness.

For the credibility characteristic, we determined that the estimate minimally met leading practices. We found, in contrast to the approach it took in developing the Cannon Building renewal estimate, that AOC did not conduct a risk and uncertainty analysis of the Capitol Dome restoration estimate. While AOC officials told us they have not conducted a quantitative-risk and uncertainty analysis of the estimate, they said they have taken steps to qualitatively assess and mitigate project risks. For example, AOC structured the phase II contract solicitation to have contractors include a base bid (phase IIA) and pre-priced options for later stages (phases IIB and IIC) of the work. According to AOC officials, by structuring the contract this way, AOC is protected from price escalations affecting the future phases. In addition, AOC established unit prices and quantity allowances for some restoration tasks that are of indefinite quantity, such as repairing cracks. According to AOC officials, this will help to control project costs because, while their quantity estimates may be imprecise, knowing the unit prices associated with the work should mitigate some cost risk. While these actions are encouraging, they are not documented within the context of a quantitative-risk and uncertainty analysis. As a result, we do not know how these actions relate to other risk mitigation efforts that may have been considered and why AOC chose these actions over other efforts. Moreover, not having a quantitative-risk and uncertainty analysis precludes establishing a level of confidence associated with achieving the estimated cost and limits AOC management’s ability to determine an appropriate level of contingency reserves that may be necessary to address risks that AOC has identified.

Conclusions

AOC project’s cost-estimating guidance policy and two resulting project estimates that we reviewed may not be fully reliable because they incorporated some, but not all, leading practices in cost estimating. Because AOC’s project cost estimates inform Congress’s funding decisions and affect AOC’s ability to effectively allocate resources across competing projects in its capital program, there is a risk that funding decisions and resource allocations could be made based on information that is not reliable. We recognize that incorporating GAO’s cost- estimating best practices into AOC’s cost-estimating policy and guidance may involve additional costs—such as for conducting a risk and uncertainty analysis for projects and conveying the confidence level of the estimate to Congress and AOC managers. However, without investing in these practices, Congress risks making funding decisions and AOC management risks making resource allocation decisions without the benefit that a robust analysis of levels of risk, uncertainty, and confidence provide decision makers.

Recommendations

To improve the Architect of the Capitol’s project-cost-estimating process, enhance the transparency of its related process, and allow for more informed decision making related to projects’ costs, we recommend that the Architect of the Capitol take the following two actions, to the extent that the benefits exceed the costs: incorporate leading practices we identified as lacking for cost estimating in the AOC’s cost-estimating guidance and policies, and for ongoing and future projects, submit the confidence level derived from risk and uncertainty analyses along with budget documentation to appropriate congressional decision makers, so that Congress is aware of the range of likely costs and AOC’s associated confidence levels.

Agency Comments and Our Evaluation

We provided a draft of this report to the Architect of the Capitol for review and comment. AOC agreed with our recommendations and provided us with additional context and information on specific actions that the AOC has taken or intends to take to more fully address our recommendations. For example, AOC said that it planned to revise its policies and procedures to require quantitative-risk and uncertainty analysis be done, as specified in the Cost Guide, for high-dollar-value projects prior to requesting construction funding. In addition, AOC said that it would explore the most effective approach for communicating to congressional decision makers the confidence level derived from risk and uncertainty analyses along with budget documentation. We made no changes to our draft based upon AOC’s comments. AOC’s comments are reprinted in appendix IV, followed by our response to AOC’s detailed comments.

We are sending copies of this report to appropriate congressional committees and the Architect of the Capitol. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or stjamesl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Lorelei St. James Director, Physical Infrastructure Issues.

Appendix I: The Leading Practices for Developing High-Quality Cost Estimates and Associated Tasks

Step Leading practice 1

Associated task

Determine estimate’s purpose, required level of detail, and overall scope;

Determine who will receive the estimate.

Determine the cost-estimating team and develop its master schedule;

Determine who will do the independent cost estimate;

Outline the cost-estimating approach;

Develop the estimate’s timeline.

In a technical baseline-description document, identify the program’s purpose and its system and performance characteristics and all system configurations; Any technology implications; Its program acquisition schedule and acquisition strategy; Its relationship to other existing systems, including predecessor or similar legacy systems; Support (manpower, training, etc.) and security needs and risk items; System quantities for development, test, and production;

Deployment and maintenance plans.

Define a work breakdown structure (WBS) and describe each element in a WBS dictionary (a major automated-information system may have only a cost element structure);

Choose the best estimating method for each WBS element; Identify potential cross-checks for likely cost and schedule drivers;

Develop a cost-estimating checklist.

Clearly define what the estimate includes and excludes; Identify global and program-specific assumptions, such as the estimate’s base year, including time phasing and life cycle; Identify program schedule information by phase and program acquisition strategy; Identify any schedule or budget constraints, inflation assumptions, and travel costs; Specify equipment the government is to furnish as well as the use of existing facilities or new modification or development; Identify prime contractor and major subcontractors;

Determine technology refresh cycles, technology assumptions, and new technology to

Define commonality with legacy systems and assumed heritage savings;

Describe effects of new ways of doing business.

Step Leading practice 6 Obtain data

Associated task

Create a data collection plan with emphasis on collecting current and relevant technical, programmatic, cost, and risk data; Investigate possible data sources;

Collect data and normalize them for cost accounting, inflation, learning, and quantity adjustments; Analyze the data for cost drivers, trends, and outliers and compare results against rules of thumb and standard factors derived from historical data; Interview data sources and document all pertinent information, including an assessment of data reliability and accuracy; Store data for future estimates.

Develop point estimate and compare it to an independent cost estimate

Develop the cost model, estimating each WBS element, using the best methodology from the data collected, and including all estimating assumptions; Express costs in constant year dollars; Time-phase the results by spreading costs in the years they are expected to occur, based on the program schedule; Sum the WBS elements to develop the overall point estimate; Validate the estimate by looking for errors like double counting and omitted costs;

Compare estimate against the independent cost estimate and examine where and why there are differences; Perform cross-checks on cost drivers to see if results are similar;

Update the model as more data become available or as changes occur and compare results against previous estimates. Test the sensitivity of cost elements to changes in estimating input values and key assumptions; Identify effects on the overall estimate of changing the program schedule or quantities;

Determine which assumptions are key cost drivers and which cost elements are affected most by changes.

Determine and discuss with technical experts the level of cost, schedule, and technical risk associated with each WBS element; Analyze each risk for its severity and probability;

Develop minimum, most likely, and maximum ranges for each risk element;

Determine type of risk distributions and reason for their use; Ensure that risks are correlated;

Use an acceptable statistical analysis method (e.g., Monte Carlo simulation) to develop a confidence interval around the point estimate; Identify the confidence level of the point estimate; Identify the amount of contingency funding and add this to the point estimate to determine the risk-adjusted cost estimate;

Recommend that the project or program office develop a risk management plan to track and mitigate risks.

Step Leading practice 10

Associated task

Document all steps used to develop the estimate so that a cost analyst unfamiliar with the program can recreate it quickly and produce the same result;

Document the purpose of the estimate, the team that prepared it, and who approved the estimate and on what date;

Describe the program, its schedule, and the technical baseline used to create the estimate; Present the program’s time-phased life-cycle cost;

Discuss all ground rules and assumptions; Include auditable and traceable data sources for each cost element and document for all data sources how the data were normalized;

Describe in detail the estimating methodology and rationale used to derive each WBS element’s cost (prefer more detail over less);

Describe the results of the risk, uncertainty, and sensitivity analyses and whether any contingency funds were identified;

Document how the estimate compares to the funding profile; Track how this estimate compares to any previous estimates.

Present estimate to management for approval

Develop a briefing that presents the documented life-cycle cost estimate; Include an explanation of the technical and programmatic baseline and any uncertainties;

Compare the estimate to an independent cost estimate (ICE) and explain any

Compare the estimate (life-cycle cost estimate (LCCE)) or independent cost estimate to the budget with enough detail to easily defend it by showing how it is accurate, complete, and high in quality; Focus in a logical manner on the largest cost elements and cost drivers;

Make the content clear and complete so that those who are unfamiliar with it can easily comprehend the competence that underlies the estimate results;

Make backup slides available for more probing questions; Act on and document feedback from management;

Request acceptance of the estimate.

Update the estimate to reflect changes in technical or program assumptions or keep it current as the program passes through new phases or milestones;

Replace estimates with EVM, and independent estimate at completion (EAC) from the

Report progress on meeting cost and schedule estimates; Perform a post mortem and document lessons learned for elements whose actual costs or schedules differ from the estimate;

Document all changes to the program and how they affect the cost estimate.

Appendix II: Objectives, Scope, and Methodology

The House Appropriations Committee report accompanying the fiscal year 2014 Legislative Branch Appropriations Bill (H.R. 2792) mandated that we review the Architect of the Capitol’s (AOC) cost estimating methodology to ensure that AOC is accounting for all of the variables that should contribute to project cost estimates. This report addresses the extent to which AOC’s policies and guidance for developing cost estimates conform to leading practices identified in GAO’s Cost Estimating and Assessment Guide and provide a reliable basis to support funding and capital program decisions. This report also examines whether the estimates for the Capitol Dome and the Cannon Building projects reflect leading practices.

To determine the extent to which AOC’s policies and guidance comply with leading practices for cost estimating and provide a reliable basis to support funding and program decisions, we compared AOC’s documents to leading practices set forth in GAO’s Cost Estimating and Assessment Guide. The Cost Guide identifies 12 leading practices that represent work across the federal government and are the basis for a high-quality, reliable cost estimate. An estimate created using the leading practices exhibits four broad characteristics: it is accurate, well documented, credible, and comprehensive. That is, each characteristic is associated with a specific set of leading practices. In turn, each leading practice is made up of a number of specific tasks. (See app, I for a listing of the tasks that make up each of the 12 leading practices.) When the tasks associated with the leading practices that define a characteristic are mostly or completely satisfied, we consider the characteristic to be “substantially” or “fully” met. When all four characteristics are at least “substantially met”—we consider a cost estimate to be reliable.shared our analysis with AOC officials to review, comment, and provide additional information, and we adjusted our analysis where appropriate.

In reference to our examination of the estimates for the Capitol Dome and Cannon Building projects, we selected these projects based on their significance to the Capitol Complex, comparatively high costs, and public visibility. In addition, our review of the Cannon Building estimate is responsive to a mandate in the House Appropriations Committee report accompanying the fiscal year 2010 Legislative Branch Appropriations Bill (H.R. 2918) requiring us to monitor the progress of the project.selected versions of the estimates to review that had the most complete information for our assessments. We assessed the reliability of data used in developing the estimates and found them to be sufficiently reliable for the purposes of this report. For example, while some source data were unavailable to us, we were able to assess AOC’s process for building its estimates with the data and check for errors. We also interviewed AOC’s staff and its technical consultant about these projects and observed existing conditions at the Cannon Building and Capitol Dome. While our review of these projects’ cost estimates provides key insights and illustrates products of AOC’s cost-estimating policies and guidance, the results of our review should not be used to make generalizations about all AOC project-cost estimates. We shared our analysis with AOC officials to review, comment, and provide additional information, and we adjusted our analysis where appropriate.

We conducted our work from September 2013 to March 2014 in accordance with all sections of GAO’s Quality Assurance Framework that are relevant to our objectives. The framework requires that we plan and perform the engagement to obtain sufficient and appropriate evidence to meet our stated objectives and to discuss any limitations in our work. We believe that the information and data obtained, and the analysis conducted, provide a reasonable basis for any findings and conclusions in this product.

Appendix III: GAO’s Summary Assessments of AOC’s Project Cost Estimates for the Cannon Building’s Renewal and Capitol Dome’s Restoration

Appendix III: GAO’s Summary Assessments of AOC’s Project Cost Estimates for the Cannon Building’s Renewal and Capitol Dome’s Restoration Estimating leading practice characteristic and rationale for assessment Comprehensive The cost estimate should include both government and contractor costs of the program over its full life cycle, from inception of the program through design, development, deployment, operation and maintenance, to retirement of the program. It should also completely define the program, reflect the current schedule, and be technically reasonable. Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. Finally, where information is limited and judgments must be made, the cost estimate should document all cost-influencing ground rules and assumptions.

The Cannon Building cost estimate did not include all life cycle costs as AOC guidance did not require these life cycle costs to be identified. The estimate tracked well with the work required, as it had a WBS structure consistent with this type of effort, and clearly laid out and appeared to update ground rules and assumptions based on evolving understanding of the scope of work.

The Capitol Dome cost estimate covers the construction phase of the dome’s restoration. The cost estimate relies on specifications and drawings that exist in separate files. The cost estimate uses a standard WBS structure but does not include the underlying data. AOC submitted an extract of the data for our review. While the specifications include ground rules and assumptions, the cost estimate does not specify them.

Well documented A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture in writing such things as the source data used, the calculations performed and their results, and the estimating methodology used to derive each WBS element’s cost. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources so that the estimate can be easily replicated and updated. The documentation should also discuss the technical baseline description and how the data were normalized. Finally, the documentation should include evidence that the cost estimate was reviewed and accepted by management.

The Cannon Building cost estimate documentation did not provide source data. It did provide a reasonable explanation for how the cost estimates were prepared, what type of data were used, and the basic estimating methodologies indicating some step by step processes. The documentation provided consisted of tables of unit values, quantities, and extended values. While extended costs were identified, it was difficult to follow the roll-up of these costs. Without a copy of or access to the cost model, we were unable to verify the accuracy of escalation calculations and we could not trace all of the logic from the source data to the resulting estimate. The calculations and changes to the baseline were not well documented but subsequent data and explanations highlighted the process for the changes. Additionally, soft costs and contingencies, comprising a substantial portion of total program costs, were only superficially addressed. There was evidence of management approval of the estimate.

The Capitol Dome cost estimate captures all source documents used. The cost estimate relies on a WBS and shows how the calculations were performed. A cost analyst unfamiliar with the program could develop the cost estimate, although in places it is not clear. The cost estimate has a brief reference to the source documents used. Costs are presented in constant-year and then-year dollars.

Estimating leading practice characteristic and rationale for assessment Accurate The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic. An estimate is accurate when it is based on an assessment of most likely costs, adjusted properly for inflation, and contains few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status. During the update process, variances between planned and actual costs should be documented, explained, and reviewed. Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs.

The Cannon Building cost estimate included an uncertainty analysis indicating what confidence level the budget fell within, which was greater than 90 percent. Additionally, in the absence of having the cost model used to prepare the estimate, we were only able to validate the accuracy of a very small sample of WBS elements. Furthermore, while the estimate documentation identified inflation adjustments, because of missing calculation and conversion factors, we were unable to determine if the estimate has been adjusted properly for inflation. However, the documentation did provide a discussion of what and how the inflation adjustments were applied.

The Capitol Dome cost estimate did not include a risk and uncertainty analysis indicating what confidence level for the estimate the budget was set at. The estimate is shown in constant-year and then-year dollars using indices for inflation. The indices used to adjust for inflation appear to be out-of-date. The estimate has few errors. AOC updated its cost estimate prior to beginning the project, but once the project began, AOC did not update the estimate. AOC did not document variances between planned and actual costs. AOC relied on historical records and on an industry database to support its estimate. The estimate uses an engineering build-up throughout, which appears appropriate for this project. Credible The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate. Further, the estimate’s cost drivers should be crosschecked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results.

The Cannon Building cost estimate includes analyses of sensitivity, risk, and uncertainty. However, the sensitivity analysis was performed after the fact and was not used to inform the budget or provided for management consideration of risk prior to project start. AOC’s risk and uncertainty analysis indicates that the budget for the project is set above the 90% confidence level, which may be unreasonably high. AOC has received multiple estimates from different sources (architect, construction manager, pre-construction contractor) as a cross-check for reasonableness. While these estimates are not necessarily outside the AOC program office’s influence, they do appear to represent independent estimates based on each company’s interpretation of the project’s requirements and helped to validate the reliability of the estimates.

Appendix IV: Comments from the Architect of the Capitol

GAO Comments

1. We continue to believe that a quantitative risk and uncertainty analysis enables more effective oversight than can be obtained by taking a qualitative approach. A quantitative risk and uncertainty analysis conveys the confidence level in achieving the most likely cost and provides actionable information about cost, schedule, and technical risks that cannot be obtained qualitatively. 2. While AOC’s contractors act independently, AOC’s Project Management Division is responsible for the estimates prepared via contract, and AOC’s Cost Estimating Group is responsible for reviewing them. Because AOC’s Project Management Division and Cost Estimating Group staff are part of the same Planning and Project Management organization, the Cost Estimating Group staff could be biased because of organizational influences and unable to provide a fully objective review of costs to the Project Management Division. 3. While our leading practices do not specify target confidence levels, experts we consulted with in developing our leading practices agreed that program cost estimates should be budgeted to at least the 55 percent confidence level and potentially as high as 80 percent. While AOC has good reasons for its interest in maintaining the project’s budget at a high confidence level, we believe that the confidence level for the project’s budget—exceeding 90 percent—may be excessive. 4. AOC identified 60 risks to the project and modeled them in aggregate, as opposed to individually, in conducting its quantitative risk and uncertainty analysis. Modeling the aggregated risks precludes AOC from identifying relationships among risk elements and determining which risks have the greatest influence on project costs. We continue to believe that modeling the risks in aggregate likely contributes to the overly narrow range of estimated costs over the confidence levels, and can overstate the effect of the risks. 5. Our leading practices indicate that an estimate’s documentation should be detailed enough so that the derivation of each cost element can be traced to all sources allowing the estimate to be easily replicated and updated. Because some information was not readily available in documents maintained by AOC, we were unable to determine if the estimate has been adjusted properly for inflation. 6. AOC has taken positive steps in conducting a life-cycle cost analysis that considered fixed costs and ongoing maintenance costs to inform the AOC’s selection of mechanical systems for heating and cooling the Cannon Building. However, AOC did not conduct life-cycle analyses for other components of the project, such as the building’s roof and alarm system. This precludes AOC from capturing the total cost of the project and evaluating design alternatives on a total-cost basis. 7. AOC takes positive steps to qualitatively identify risks and uncertainty and assess sensitivity. While this analysis is useful in setting contingencies and managing risks during construction, a quantitative risk and uncertainty analysis enables more effective oversight because it conveys the confidence level in achieving the most likely cost and provides actionable information about cost, schedule, and technical risks that cannot be obtained qualitatively. 8. While AOC has used some information from completed phases of the Capitol Dome’s restoration project to inform development of the estimate for later phases, not having information on actual costs makes it difficult for AOC to assess the difference between the price it paid and its contractors’ costs to determine the estimate’s reasonableness. 9. Our leading practices indicate that an estimate’s documentation should be detailed enough so that the derivation of each cost element can be traced to all sources allowing the estimate to be easily replicated and updated. Because some information was out-of-date at the time of our review, we were unable to determine if the estimate had the proper escalation adjustments.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above Michael Armes, Assistant Director; Karen Richey, Assistant Director; George Depaoli, Analyst-in- Charge; Laura Erion; Emile Ettedgui; Colin Fallon; Geoffrey Hamilton; James Manzo; Faye Morrison; and Vanessa Welker made key contributions to this report.