IRS Management: Cost Estimate for New Information Reporting System Needs to be Made More Reliable
Released: 2012-01-31T00:00:00

Letter

Background

IRDM Program

IRS initiated the IRDM program, in part, to implement new information reporting requirements, but more generally to increase voluntary compliance with tax laws by expanding and maximizing IRS’s ability to match existing and future information returns with tax return data and establishing a new business information matching program. Previously, IRS had only matched information returns to individuals’ and sole proprietors’ tax returns. Under IRDM, IRS plans to build several new IT systems and enhance some existing systems as well as implement numerous organizational and process changes. IRS plans for IRDM to use information returns to identify individual and business tax returns that are likely sources of revenue, which the current individual tax return matching system is not designed to identify. IRDM implementation is led by IRS’s Small Business/Self Employed division and MITS, which is leading the IRDM IT system development.

IRS Cost Estimation Process and Guidance

Cost estimates are a vital factor for sound management decision making and they aid in the formation of a project’s budget. IRS uses cost estimates, in part, to justify budget requests and prioritize the selection of IT projects for possible funding. After an IT project is approved, the cost estimate is later used as a starting point for developing the performance measurement baseline for EVM, a project management approach that, if implemented appropriately, provides management important tools such as objective reports of project status and early warning signs of impending schedule delays and cost overruns. Data from a reliable performance management system, such as EVM, are necessary inputs for an updated cost estimate, among other things.

OMB issued guidance on managing IT projects, which discusses cost estimation and refers to our cost guide for how to meet cost estimating requirements.continuously updated based on the latest information available to ensure that they are current, accurate, and valid. According to our cost guide, effective program and cost control requires ongoing revisions to the cost estimate, budget, and projected estimates at completion. Specifically, our guide states that estimates should be continuously updated with actual costs incurred to that point so that significant cost, schedule, or performance variances can be examined. In addition, it says that cost estimates should be updated to reflect significant changes to a project’s OMB guidelines state that cost estimates should be scope or specifications and when certain projects approach key milestones.

Within MITS, project managers and EPO are involved in estimating program costs. EPO is an independent group of cost estimation experts that assists project teams by developing and updating cost estimates using a standard documented process. Project managers are responsible for maintaining a program’s cost estimate. EPO only becomes involved in updating a program’s cost estimate at the request of project managers, according to EPO officials.

IRS procedures for developing, using, and updating cost estimates and EVM are described in several guidance documents, specifically:

EPO’s Estimator’s Reference Guide, which is used by EPO staff, is the general resource on the processes and procedures for developing and delivering IT cost estimates. It discusses the technical aspects of updating cost estimates, such as what documents are used in cost modeling once a project has begun. IRS’s Information Technology Investment Planning and Management Guide (Investment Guide) outlines the framework for selecting, managing, and evaluating IRS IT projects. The Investment Guide includes discussions of how IT projects are selected using cost information, and how managers should use cost information to monitor a project. Project managers are responsible for managing cost, schedule, and performance for a project.

MITS’s Estimation Procedures document describes IRS’s organizational approach to cost estimation, applicable to all IRS projects. The document is directed at project managers and it includes discussions of the steps and staff roles necessary to develop an estimate, and the circumstances when EPO typically becomes involved with updating a cost estimate. The document states that project managers are responsible for monitoring project progress and suggests initiating assistance from EPO if a project meets certain thresholds.

The Department of the Treasury’s Earned Value Management Guide provides guidance for implementing EVM on major IRS projects. Program or project managers have the ultimate responsibility for implementing and monitoring the EVM system for their program or project.

IRDM Program Cost Estimates

IRS developed two cost estimates for IRDM early in the program. In 2007, IRS developed a preliminary cost estimate for budgetary purposes when very little program information was available (referred to in this report as the 2007 preliminary cost estimate). As shown in table 1, in 2007, IRDM system development was estimated to cost about $5 million in fiscal year 2009 and about $23 million per year thereafter. In 2009, EPO developed a solution concept-based estimate (SCBE, referred to in this report as the 2009 SCBE), which was more rigorous than the 2007 preliminary estimate. The 2009 SCBE was developed before program implementation began, when MITS had more information than it did in 2007, but system design plans were still under development. The 2009 SCBE was about $36 million less through the first 4 years of the project than the 2007 preliminary estimate.

IRS used the 2007 preliminary estimate to justify the initial IRDM budget. According to a MITS official, the 2009 SCBE was not used for budgetary purposes because program specifications were undergoing modification in fiscal year 2010, requiring the full $23-million per-year funding. For example, in 2010, IRS made several changes to the complexity of the IRDM program, which included dividing it into four projects. One of the projects required restructuring as a separate development effort, using different resources and technologies. Changes also necessitated using new software development methods not previously used within IRS, and additional contracting support. According to MITS officials, such changes increased IRDM funding needs above the amounts supported by the 2009 SCBE. As a result, IRS did not revise its initial funding request for future fiscal years using the SCBE and instead relied on the 2007 preliminary estimate.

From fiscal years 2009 through 2011, IRS received about $52 million in total funding for IRDM, of which the IRS had spent $46 million (when accounting for $2.6 million carried over from fiscal year 2009 funds and $5.8 million carried over from fiscal year 2010), through fiscal year 2011, as shown in table 2.

In our May 2011 report,the most rigorous IRDM cost estimate available at the time and the 2007 preliminary estimate lacked documentation for a complete review. We found that the 2009 SCBE did not fully follow best practices. We recommended that if IRS updated the cost estimate, it should follow best practices from our cost guide. In response to our report, IRS said it would update the 2009 SCBE. IRS subsequently decided not to revise the estimate because, according to officials, they already have a plan, schedule and funding, the program is not over-budget, and the risks associated with IRDM, and the program’s size, do not warrant an update. we assessed the 2009 SCBE because it was Over the summer of 2011, MITS provided us with additional cost information. Officials referred to these documents as IRDM’s new cost estimate, and they were used in supporting IRS’s fiscal year 2012 IRDM budget. Consequently, in this report, we refer to the materials provided to us as the 2011 cost estimate. This estimate was not a SCBE or developed by EPO. The 2011 cost estimate was based on several data sources, including IRDM’s Exhibit 300, EVM data, spend plans, and schedule with work breakdown structures (WBS). MITS projects IRDM to cost $115 million for fiscal years 2012 through 2016, or about $23 million per year.

IRDM’s 2011 Cost Estimate Does Not Meet Best Practices for Reliability and Does Not Fully Support the Program’s Budget

IRDM’s Fiscal Year 2012 and Projected Budget Requests Are Not Supported by a Reliable Cost Estimate

According to best practices established by our cost guide, a cost estimate We should be comprehensive, well documented, accurate, and credible.assessed the 2011 cost estimate against cost estimation best practices because IRS told us the estimate was used to support its budget requests for fiscal year 2012 and beyond. While the 2011 IRDM cost estimate shows some characteristics of a reliable cost estimate, it does not fully meet best practices.

The estimate partially meets best practices for a comprehensive cost estimate, as shown in figure 1. It reflects the current program schedule and contains information about the program’s technical characteristics. The estimate provided some details about costs for IRDM’s fiscal year 2012 budget request, but the cost estimate does not cover the program’s entire life-cycle. Without fully accounting for life-cycle costs, management may have difficulty successfully planning program resource requirements and making informed resource-planning decisions. IRS defined assumptions used to estimate some IRDM costs, but did not provide the assumptions used to estimate labor or program operations costs. Furthermore, IRS did not include ground rules used to develop the estimate. Unless ground rules and assumptions are clearly defined, the cost estimate will not have a basis to identify and mitigate areas of potential risk.

The estimate minimally meets best practices for a well documented cost estimate, as shown in figure 2. IRS provided supporting information for some staff resources, but detailed data for the staffing level requested for fiscal year 2012 was missing. The cost estimate documentation says that the labor cost justification was captured in the resource loaded project schedules, but we found that these schedules only justified about 6 out of the 86 requested full-time equivalent (FTE) staff for IRDM. Furthermore, although IRS officials cited the WBS as the basis for cost projections, we found no evidence linking the WBS to cost. Multiple documents linked software and hardware specifications to cost, but they did not provide consistent cost information. As a best practice, documentation should describe the source data used, the estimating methodology, and show step-by-step how the estimate was developed. Without a well documented cost estimate, the program’s credibility may suffer because the documentation cannot explain the rationale of the methodology or the calculations underlying the cost elements.

The estimate minimally meets best practices for an accurate cost estimate, as shown in figure 3. Calculations in the estimate are mathematically correct. However, documentation that IRS provided to support estimated costs for IRDM hardware and software’s estimated costs did not match estimates in IRDM’s spend plans. IRS officials said the discrepancies occurred because the spend plans were developed using more recent cost information for software purchases that were not included in the supporting documentation. Additionally, the estimate does not list any confidence levels or provide a range of possible costs. According to best practices, unless an estimate is based on an assessment of the most likely costs and reflects the degree of uncertainty given all of the risks considered, management will not be able to make informed decisions. IRDM uses EVM to identify variances between planned and actual costs, but as discussed below, we found IRDM’s EVM data to be unreliable, and there was no evidence that IRS uses actual cost data to evaluate whether cost projections are realistic.

The estimate does not meet best practices for a credible cost estimate, as shown in figure 4. For example, the estimate was not crosschecked or assessed for risk and uncertainty. According to best practices, an estimate without risk and uncertainty analysis can be unrealistic because it does not assess how the cost estimate would be affected if, for example, the schedule slipped, the mission changed, or a proposed solution did not meet users’ needs. In addition, IRS did not perform a sensitivity analysis. Further, there is no evidence that another office performed a separate cost estimate—referred to as an independent cost estimate—to validate the 2011 cost estimate. In previous work we found that because of limited resources, IRS generally only does an additional independent cost estimate for its largest programs, and according to officials, IRDM is not considered a large enough program in terms of its funding level. While some of MITS’s cost estimates are done by EPO— which is independent of program management offices—the 2011 cost estimate was done by the IRDM program office.

Because the 2011 cost estimate does not meet best practices, it does not provide reliable support for IRDM’s fiscal year 2012 budget request, or any of the projected budget requests. IRS officials said current IRS policy does not require projects to routinely re-estimate project cost. The IRDM program office—which does not use the same software or modeling techniques as EPO—relied on spend plans, EVM, and other documents to estimate costs. In July 2011, MITS officials said that it would take 90 days for IRDM and EPO staff to complete a new cost estimate for IRDM. When considering FTEs and time, a new cost estimate developed by IRDM and EPO staff would require a total of about eight staff months.

IRDM Could Benefit From EPO Assistance on Updating Cost Estimates

EPO has specialized cost estimation tools, such as software that incorporates many best practices from our cost guide, and expertise that project teams can leverage to update cost estimates. If used correctly, EPO estimation procedures could help IRDM management to maintain reliable cost information for use in budget requests. EPO officials said they did not work with the IRDM team to maintain an updated SCBE because the team did not seek their assistance. IRDM was not required to do so because MITS guidance, as of September 2011, does not require project teams to consult with EPO when updating a cost estimate. Without EPO involvement, IRS has less assurance that cost estimate updates will follow best practices. Our cost guide states that cost estimates should be (1) updated to reflect actual costs and changes (i.e., significant modifications to a project’s scope or specifications) in order to keep the estimate current as the program passes through new phases and milestones and (2) updated if there are significant cost, schedule or performance variances. The continual updating of the cost estimate as the program matures not only results in a higher-quality estimate, but also gives cost estimators the opportunity to collect data for use in future estimates as well as incorporate lessons learned.

Our cost guide also states that cost estimation work should be done by a central independent estimating organization, and estimators should monitor programs to determine whether preliminary information and assumptions remain relevant and accurate. EPO has the following characteristics—unlike the IRDM project team—that could help provide more reliable cost estimate updates:

EPO is able to use robust cost estimation techniques, including the SEER-SEM software cost estimation model.project histories and cost relationships to produce cost estimates and can estimate costs consistent with best practices—such as adjusting for risk and incorporating the results of a sensitivity analysis. When EPO estimators validate or update a project’s funding requirements, they tailor SEER-SEM to the project and use it to consider actual cost data from the project team, according to EPO’s Estimator’s Reference Guide. Estimators calibrate the model to include the schedule for remaining work and evaluate and revise key cost drivers, according to the guidance. As mentioned previously, the 2011 IRDM cost estimate did not use a cost estimation model.

EPO, as a whole, has more cost estimation experience than the IRDM project team. EPO’s six cost estimators have 43 years of combined cost estimation experience. They all have received training in SEER- SEM and other cost estimation models. In addition, EPO officials said that project teams generally would not have the technical skills to update a SCBE using cost estimation models. Although the IRDM project team has some cost estimation experience and relevant training, IRDM officials do not have SEER-SEM training or experience. Further, our analysis of IRDM’s 2011 cost estimate illustrates that estimate updates done by project teams may not result in reliable cost information. According to EPO officials, an updated estimate developed by EPO would also be independent, more holistic, and would include elements that project teams may miss.

EPO-produced updates can help build a historical record of IRS cost estimate data. According to our cost guide, historical data are crucial to developing high-quality cost estimates because estimators usually develop estimates for new programs by relying on data from programs that already exist and adjusting for any differences. EPO officials told us that they are working to build a historical database that compares estimated costs to actual costs. As IRS’s central estimating organization, EPO is uniquely qualified to use cost estimate updates as an opportunity to obtain data that are consistent with other estimates and to use the data to build a historical cost database, which can ensure that future cost estimates are credible.

According to MITS officials, it is up to project teams to seek EPO assistance. However, MITS’s Estimation Procedures document, which provides cost estimation guidance to project teams, suggests some cost and schedule variance and project size thresholds that, if exceeded, should cause project managers to contact EPO for an updated estimate. It recommends EPO involvement in updating estimates when: cost or schedule variance are 10 percent or greater for major projects; cost or schedule variance are 25 percent or greater for non-major projects; or a project with development/modernization/enhancement (DME) costs greater than $5 million reaches milestone 3.

IRDM meets the first threshold. Specifically, IRDM meets the IRS’s criteria for a “major” project because IRDM’s projected life-cycle costs are about $166 million, based on funding projections submitted to OMB. Also, according to IRDM EVM data, as of September 30, 2011, the program had a greater than 18 percent cost variance and an almost 13 percent schedule variance. IRDM officials said they did not work with EPO to update the 2009 SCBE because it was not required by current MITS guidance.

DME is a term used by OMB to describe the program cost for new investments, changes or modifications to existing systems to improve capability or performance, changes mandated by the Congress or agency leadership, personnel costs for investment management, and direct support. For major IT investments, this amount should equal the sum of amounts reported for planning and acquisition plus the associated FTE costs reported in the Exhibit 300. developing cost estimates in 2006. Most of its work has focused on developing estimates for proposed projects, rather than updating estimates for existing projects. The Estimation Procedures document was developed in September 2011, and as of October 2011, officials said EPO has six cost estimators on staff. As a result, EPO officials said that project teams generally update IT project cost estimates without EPO assistance and that, as of October 2011, EPO has been involved in few cost estimate updates. However, senior MITS and EPO officials said they would like for EPO to have a greater role in cost estimate updates. If IRS does not have reliable cost estimate updates, projects may face risks and their budget requests may not be adequately justified to inform decision making; these outcomes could be even more significant for projects with cost or schedule variances or high DME costs, such as IRDM.

MITS Guidance Has Inconsistencies and Could Better Link Cost Estimation Procedures with Budget Requests

MITS guidance documents used by project managers do not clearly discuss the appropriate uses of different types of cost estimates. According to current guidance used by EPO estimators, non-SCBE cost estimates are less rigorous and are not for use in budgets, but as stated above, neither the IRDM initial budget request nor the current and projected budgets were developed using information from the 2009 SCBE.

Three IRS guidance documents describe the relationship between cost estimates and budgets. However, the documents are directed at different audiences, do not present consistent information, and contain different levels of detail. Specifically:

EPO’s Estimator’s Reference Guide, used by EPO estimators, states that budgets for IT projects should be established using SCBEs. IRS’s SCBEs rely on cost estimation methods that incorporate best practices from our cost guide, including considerations of risk, and provide a level of confidence associated with the estimate. According to our cost guide, for management to make good decisions, the program estimate must reflect the degree of uncertainty, so that a level of confidence can be given about the estimate. The Estimator’s Reference Guide also discusses techniques that estimators can use to update SCBEs, and aligns that process with annual budget submissions. Although this guidance contains many best practices, it is directed at cost estimators; therefore, project managers do not typically have access to it or use it.

The MITS Investment Guide, directed at MITS project managers, discusses the role of EPO in developing initial rough estimates and budget-ready SCBEs and states that, if a project does not yet have an SCBE, a rough estimate may be used as a placeholder in a budget request. The guide requires that MITS staff should work to ensure that if an SCBE exceeds an initial rough estimate, the project’s scope and SCBE fit within the appropriated budget. However, the guide does not discuss how, if at all, a budget request should be adjusted if an SCBE provides an estimated cost that is lower than the budget, or how any future cost information should be incorporated into budgets.

The Estimation Procedures document, directed at MITS project managers, does not define types of cost estimates or discuss whether they are appropriate for budget decisions. EPO officials said they did not believe it is necessary to characterize the different types of cost estimates in the Estimation Procedures document because they are not necessary for defining the organizational approach to estimation, which is the intent of the document. The document states that if updated cost estimates indicate that a project’s budget needs to change, the changes must be approved. However, it does not specify who should approve the estimates.

Without consistent guidance about what types of cost estimates are appropriate for budget requests, project teams may not use the best information available. Our cost guide states that, as a best practice, an estimate intended to support budgetary decisions should cover the project’s entire life-cycle and should be supported by a description of the program’s technical characteristics, which would be found in an SCBE. Using a cost estimate that lacks sufficient rigor—such as a preliminary cost estimate, instead of an SCBE—could lead to budget requests that do not accurately reflect program funding needs. For example, the 2007 preliminary IRDM estimate lacks an uncertainty analysis, which would provide a basis for adjusting the estimate to reflect unknown facts and circumstances that could affect costs, and as a result, IRDM managers do not have assurance that the program’s funding level remains appropriate. Further, not providing project managers with guidance on how to incorporate new cost information—either from an SCBE that has replaced a preliminary estimate or from an updated cost estimate—into budget requests could result in requests that do not reflect current or accurate funding needs for a project.

Unreliable EVM Data Raise Additional Concerns about IRDM Cost Estimate

IRS provided EVM data in the 2011 IRDM cost estimate to justify its budget requests, but we found that the program’s EVM data are not reliable in any of the areas we reviewed. Reliable data on actual performance, obtained from an EVM system, are a necessary input if an updated cost estimate is to be considered accurate and credible. Because IRDM’s 2011 cost estimate is based on unreliable EVM data, it does not provide adequate support for IRDM’s budget requests. Until IRS addresses deficiencies in its EVM data, it cannot provide reliable cost estimate updates for IRDM. EVM data reliability deficiencies, such as those we observed for IRDM, are common in federal agencies, and we have also previously reported on inconsistencies in implementation of EVM for IT projects at Treasury bureaus.

See GAO-09-3SP and Office of Management and Budget, Capital Programming Guide: Supplement to Circular A-11, Planning, Budgeting, and Acquisition of Capital Assets (Executive Office of the President, Washington, D.C.: August 2011). determine whether all program requirements have been addressed, risks have been identified, mitigation plans are in place, and available and planned resources are sufficient to complete the work.

Using qualified staff, conduct surveillance on the EVM system.

Surveillance is reviewing a contractor’s EVM system to observe ANSI compliance and how well a contractor is using its EVM system to manage cost, schedule, and technical performance.

Treasury’s EVM guidance requires a project of IRDM’s size to follow an abbreviated set of 10 ANSI guidelines, and to conduct surveillance on the EVM system. Other departments also scale ANSI guidelines according to the size of projects, which could result in some agencies not fully following certain best practices. Further, projects like IRDM, according to Treasury guidance, only need to complete an independent baseline validation—which although not defined in the guidance, appears to be a less rigorous version of an integrated baseline review. Following ANSI guidelines and conducting an integrated baseline review and surveillance can help ensure that EVM data can indicate how well a program is performing in terms of cost, schedule, and technical matters. This performance information is necessary for proactive program management and risk mitigation, and to maintain a reliable cost estimate. Where applicable, we assessed IRDM’s EVM data against the standards cited in the Treasury guidance.

IRDM’s EVM System Is Not Compliant with Key Required Guidelines

We assessed IRDM on three ANSI guidelines, which are fundamental elements for an EVM system and are included in Treasury’s abbreviated 10 EVM guidelines. We found that IRDM’s EVM system is not compliant with these guidelines. For an overview of our findings on IRDM’s EVM data reliability, see table 4 in appendix III.

For each selected guideline we found:

WBS: This ANSI guideline states that authorized work elements for the program should be defined, which typically includes using a WBS tailored for effective internal control. Further, a project’s schedule, cost estimate, and EVM system should be based on the same WBS, according to our cost guide. The WBSs used in IRDM’s schedules do not match the WBS used for EVM. The WBSs in four of the five IRDM schedules reflect detailed project-level tasks, while the WBS used for the EVM data is only broken down by contractor and government efforts, and does not include any project-level data. The WBS used to inform the 2011 IRDM cost estimate was not broken down by contractor or government data, and it did not provide costs for detailed tasks. Without a WBS, from which to measure progress and to serve as a consistent framework for the schedules and EVM, there is no basis for reliable EVM data, according to our cost guide. An IRS official said that the WBS in the IRDM schedules is not the primary source of financial information for IRDM. Instead, officials use IRS’s financial tracking system, which is much less detailed than the schedules’ WBS, to obtain project level data for EVM. This technique for gathering EVM data at a project level is contrary to the purpose of EVM, which is to integrate cost, schedule, and technical data from detailed work packages that can be monitored for variances against the original plan. Since a resource-loaded schedule forms the foundation for the EVM baseline, both the schedule and the EVM data should be based on the same WBS, according to our cost guide.Because the financial tracking system only provides project level data, and financial information cannot be traced to the WBS elements in the schedules, the cost associated with the project tasks is unknown.

Sequencing: This ANSI guideline states that projects should have a schedule that describes the sequence of work—that is, a list of activities in the order in which they are to be carried out—and identifies significant task interdependencies required to meet project requirements. All of the IRDM schedules had significant problems with sequencing. For example, many predecessor and successor tasks were not linked to one another, which are necessary for properly sequencing work so that the schedule will update in response to changes. Because the schedule is missing so many of these “logic links,” it will not automatically recalculate forecasted start and finish dates of remaining activities. Thus, any activities that are late will not automatically recalculate the dates for affected successor activities. IRS officials said they are aware of some issues with missing links. As a result of these missing links in all of the IRDM schedules we reviewed, IRDM’s schedules are not reliable. Because the schedules form the basis for the performance measurement baseline used to track cost and schedule variances in an EVM system, the data from the IRDM EVM system are not reliable.

Time-phased budget baseline: This ANSI guideline states that a program should establish and maintain a time-phased budget baseline, at the control account level, against which performance can be measured. Resources must be accounted for in a schedule in order to develop this baseline, according to our cost guide. IRS was unable to show evidence that it has established and maintained a time-phased budget baseline for IRDM. Specifically, the program does not have one overall schedule, and none of the IRDM schedules we reviewed were completely resource-loaded.

IRS officials said although they do not have a program level schedule for IRDM, the individual project schedules are linked through interdependencies. However, we could not identify these links in our analysis. IRS officials said the resource-loaded schedules do not show all project resources because some resources, such as contractor personnel being used across projects, are used in more than one IRDM project. Without schedules that include the resources needed to complete tasks, IRS was unable to prove that it had established and maintained a time- phased budget baseline. This baseline is a critical EVM component for measuring IRDM’s performance.

IRS Did Not Validate the Baseline for IRDM’s EVM System

According to Treasury EVM guidance, an independent baseline validation should be conducted for a DME project like IRDM. As stated above, GAO and OMB consider an integrated baseline review to be a key element of EVM data reliability, while Treasury guidance allows projects like IRDM to complete a less-rigorous independent baseline validation. IRS did not conduct an integrated baseline review or an independent baseline validation for IRDM. IRS officials told us that many of the activities typically done during an integrated baseline validation were performed— such as developing a WBS and schedules. However, as previously discussed we identified problems with the WBSs and schedules and the baseline process. Without a comprehensive integrated baseline review or independent baseline validation, and resolving any issues, IRS has not sufficiently evaluated the validity of IRDM’s baseline. This calls into question the reliability of IRDM’s EVM data, and could affect the program’s ability to identify and mitigate risks.

IRS Is Not Conducting Surveillance on IRDM’s EVM System

OMB and Treasury require surveillance on EVM systems, and, according to our cost guide, surveillance should be conducted to check whether the EVM system summarizes timely and reliable cost, schedule, and performance information, among other things. IRS officials said IRS is not performing surveillance on the IRDM EVM system because they did not believe it was required and because, according to officials, IRS does not have staff with the necessary technical skills to conduct surveillance. IRS officials said Treasury is reviewing whether Department-level surveillance would be efficient. It is important for the agency to conduct surveillance of EVM systems to ensure that contractors are following their own processes and satisfying ANSI guidelines.

Conclusions

The 2011 IRDM cost estimate does not fully meet best practices for a reliable estimate. It is important for IRDM to have a cost estimate that meets best practices to inform budgetary decisions and ensure that the program is implemented as planned. This standard is particularly important in a budgetary environment with scarce resources. IRDM’s 2011 cost estimate could be improved by using EPO’s expertise to ensure the cost estimate follows best practices from our cost guide. Additionally, more consistent guidance on using cost estimates to develop budget requests could help program managers for IRDM, as well as other programs, make budget decisions that are supported by reliable cost information.

IRS could increase the credibility of IRDM’s 2011 cost estimate by ensuring that IRDM’s EVM data are reliable. Such reliability could also allow for IRS to update projected costs for the remainder of IRDM’s implementation. Using a WBS that is developed using best practices from our cost guide could provide a baseline from which to measure progress, a key component to an EVM system. Similarly, developing a single integrated schedule for IRDM, that contains all resources needed to implement the program, could provide more meaningful EVM data. Finally, providing oversight of the EVM system, through an independent baseline validation and EVM surveillance, could help identify potential program risks and any possible issues with contractor performance.

Recommendations for Executive Action

To improve the quality of cost and budget information for IRS IT projects, we recommend that the Commissioner of Internal Revenue take the following four actions: 1. Ensure that the IRDM life-cycle cost estimate is reliable and that budget requests are justified by a reliable cost estimate that follows best practices. 2. Require project managers to consult with EPO to determine if projects could benefit from EPO assistance in updating cost estimates for programs that exceed thresholds recommended by MITS’s Estimation Procedures document. For those projects where EPO does not update the cost estimate, IRS should require that the decision and rationale be documented. EPO should use the information from updated cost estimates to develop a historical repository of cost estimation data. 3. Review all guidance applicable to cost estimates and take steps to ensure that they are consistent. As a first step, IRS guidance should require the use of current and reliable project cost estimates to inform budget requests, in accordance with the Estimator’s Reference Guide. 4. Improve the reliability of IRDM’s EVM data, specifically: address WBS issues by developing an EVM baseline for IRDM that reflects the same WBS as the detailed schedule and IRDM cost estimate; address sequencing issues and enable the development of a time- phased budget baseline by creating a single integrated master schedule for IRDM that is properly sequenced and resource- loaded so that effective and meaningful EVM data can be obtained to better manage the program; conduct an independent baseline validation for the IRDM EVM baseline; and conduct independent surveillance of EVM systems to ensure that data are reliable.

Agency Comments and Our Evaluation

We provided a draft of this report to the Commissioner of Internal Revenue for his review and comment. We received written comments from the Deputy Commissioner for Operations Support, which are reprinted in appendix IV. We sought clarification on IRS’s written response in regards to whether it agreed with two of our recommendations, and on a reference to OMB guidance. IRS provided us with additional comments, which are summarized below. In addition, the agency provided technical comments, which we incorporated into the report as appropriate. IRS agreed with one of our four recommendations, partially agreed with another, and disagreed with two.

While IRS’s comment letter did not address the recommendation, the Director of Risk Management in MITS’s Strategy and Planning Office told us the agency agrees with the recommendation to require certain IT project managers to consult with EPO about updating cost estimates, documenting decisions not to update cost estimates, and placing data from updated cost estimates in a repository. Similarly, IRS’s comment letter did not address our recommendation to ensure that its cost estimation guidance is consistent. However, IRS officials said they partially agree with this recommendation and have taken steps to ensure that their estimation practices and procedures follow consistent, documented guidance. They noted that in all instances, however, IRS IT cost estimates will be based on the best information available at the time the estimate is requested or required as opposed to our recommendation that IRS require the use of current and reliable cost estimates to inform all budget requests. We note in our report that using an unreliable cost estimate could lead to budget requests that do not accurately reflect program funding needs. Once done in conformance with guidance and best practices, current and reliable cost estimates can be maintained through normal required monitoring and cost tracking procedures, unless significant changes in project circumstances warrant updating the cost estimate.

IRS disagreed with our recommendation to ensure that the IRDM life- cycle cost estimate is reliable and that budget requests are justified by a reliable cost estimate that follows best practices. In its comment letter, IRS wrote that implementing the recommendation would require it to spend resources that do not directly contribute to the successful implementation of the IRDM program, and that the IRDM program is within its budget and schedule. At the end of fiscal year 2011, IRDM was under budget by more than 18 percent and behind schedule by almost 13 percent, equal to over 4 months behind schedule, according to IRDM’s EVM data. Because the IRDM program does not have a reliable cost estimate, budget authorities do not have reliable information to determine an appropriate funding level. Such variances indicate that the current funding level may not be appropriate. As we reported, IRS estimated in July 2011 that it would take about 90 days, comprising 8 staff months (or a direct staff cost likely less than $200,000, according to our calculations using a top government salary rate and general benefits rate), to complete a new cost estimate for IRDM. While we agree that federal resources are tight, we believe that such an investment could produce benefits that not only improve the reliability of the IRDM cost estimate, but also better ensure that IRS requests the correct amount of resources to ensure IRDM fully achieves successful implementation. Further, benefits of a new estimate would also stretch beyond that program and provide an important foundation for improving IRS cost estimates in general. The OMB Capital Programming Guide directs agencies to develop sound cost estimates based on our cost guide and states that during the budget process, the credibility of the costs will be examined, and OMB and the Congress will hold agencies accountable for meeting the schedule and performance goals within the cost estimates. More reliable cost estimates enable Congress and other budget authorities to make more complete and informed decisions.

IRS also disagreed with our recommendation to improve the reliability of EVM data, and stated that OMB’s revisions to Circular A-11 remove EVM system requirements due to negative cost benefit. We disagree with IRS, as Circular A-11, Appendix J and the Capital Programming Guide still contain language that directs agencies to use EVM for major projects. Former Circular A-11, section 300.7, instructed agencies to use EVM system requirements to identify areas where problems are occurring when reporting on ongoing investments. While current guidance for Exhibit 300 no longer explicitly discusses EVM reporting on ongoing investments under section 300.7, other sections of the guidance still direct the use of EVM for managing IT capital assets and state that in general cost, schedule and performance goals are to be controlled and monitored by using an EVM system. Moreover, our report assessed the reliability of IRDM’s EVM data because the data were included in IRDM’s 2011 cost estimate and the data are used to track the program’s progress. Regardless of OMB requirements, any data used for cost estimation and program management, particularly when it helps to support a budget request, should be reliable.

We will send copies of this report to the Chairmen and Ranking Members of Senate and House committees and subcommittees that have appropriation, authorization, and oversight responsibilities for IRS. We will also send copies to the Commissioner of Internal Revenue, the Secretary of the Treasury, the Chairman of the IRS Oversight Board, and the Acting Director of the Office of Management and Budget. Copies are also available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions concerning this report, please contact me at (202) 512-9110 or brostekm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

This report builds on our May 2011 reportService’s (IRS) Information Reporting and Document Matching (IRDM) program and analysis performed during a budget justification review of the IRDM program, conducted from June 2011 to August 2011, which we provided to Congress as technical assistance. In the May 2011 report, we assessed IRDM’s 2009 solution-concept based cost estimate. For the budget justification work and this report, we assessed IRDM’s 2007 preliminary cost estimate. Also for this report, we assessed IRDM’s 2011 cost estimate.

See GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). documentation provided to us. Once we determined that the 2011 cost estimate did not fully meet best practices, we determined why this occurred. To this end, we interviewed officials in IRDM’s Program Management Office and MITS’s Estimation Program Office (EPO). We also compared IRS guidance that addresses cost estimation—the EPO’s Estimator’s Reference Guide, MITS’s Estimation Procedures document, and IRS’s Information Technology Investment Planning and Management Guide—to criteria in our cost guide.

To assess the extent to which IRS’s practices for capturing IRDM’s actual costs and comparing them to estimated costs, or EVM, generate reliable performance data, we compared the EVM data for IRDM and IRS’s process for maintaining the data to the high-level EVM data reliability tasks outlined in our cost guide. We assessed the extent to which IRDM’s EVM data adhered to three of the American National Standard Institute’s (ANSI) 32 guidelines; we selected the three guidelines to represent some of the fundamental steps for maintaining a reliable EVM system, as identified in our cost guide, and because these guidelines are also included in the Department of the Treasury’s Earned Value Management Guide, which applies to IRS (see appendix III for a list of the data reliability tasks and ANSI guidelines we reviewed). In situations where Treasury’s Earned Value Management Guide did not require certain OMB or GAO best practices, we assessed IRDM’s EVM practices against the Treasury guidance. To do this analysis, we compared the work breakdown structures used in IRDM’s EVM system, schedules, and cost estimates and identified differences in each. We also assessed each of the five IRDM schedules against scheduling best practices for ensuring that the activities are sequenced and related using network logic, as identified in our cost guide.

For both objectives, we interviewed IRS officials in the MITS division, specifically, officials from the IRDM Program Management Office and the Investment Planning and Management Office, which includes EPO. We spoke primarily with officials at IRS Headquarters in Washington, D.C. and IRS’s division office in New Carrollton, Maryland, where the officials responsible for IRDM are located. To assess the reliability of the cost estimate data that we used to support findings in this report, we reviewed relevant program documentation, such as cost estimation spreadsheets, as available, to substantiate evidence obtained from interviews with knowledgeable agency officials. We found the data we used to be sufficiently reliable for the purposes of our report. As appropriate, we attributed the sources of the data. We are making recommendations to IRS to improve data reliability in the future.

We conducted this performance audit from August 2011 through January 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Assessment of MITS’s Current IRDM Cost Estimate

The following table outlines our assessment of the extent to which the Internal Revenue Service’s (IRS) 2011 Information Reporting and Document Matching (IRDM) program cost estimate meets best practices, depicted in figures 1-4.

Appendix III: High Level Assessment of the Reliability of IRDM’s EVM System

Appendix IV: Comments from the Internal Revenue Service

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Libby Mixon, Assistant Director; Laurel Ball; Amy Bowser; Bill Cordrey; Jennifer Echard; Robert Gebhart; Paul Middleton; Donna Miller; Sabine Paul; Karen Richey; Stacy Steele; and Lindsay Swenson made key contributions to this report.