IRS Management: Cost Estimate for New Information Reporting System Needs to be Made More Reliable
Published: 2012-01-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Internal Revenue Service (IRS) began developing the Information Reporting and Document Matching (IRDM) program in fiscal year 2009 to enhance IRSs ability to automatically compare different sources of tax information and thus improve its capacity to identify and address taxpayer noncompliance. GAOs May 2011 report recommended that IRS follow best practices from the GAOs Cost Estimating and Assessment Guide if IRS updated the cost estimate for building IRDM systems. IRS provided a new cost estimate for IRDM in August 2011. In this report, GAO assessed the extent to which (1) the IRDM funding request is supported by a reliable cost estimate and, if not reliably supported, why not; and (2) IRSs practices for capturing data on IRDMs actual costs and comparing them to estimated costsknown as earned value management (EVM)generate reliable performance data. GAO compared IRSs 2011 IRDM cost estimate to criteria in GAOs cost guide and analyzed IRDMs earned value management data.
section_title: What GAO Found
paragraphs: The 2011 Information Reporting and Document Matching (IRDM) cost estimate, used to justify the programs projected budgets of $115 million for fiscal years 2012 through 2016, generally does not meet best practices for reliability. The cost estimate did not fully meet any of the four best practices for a reliable cost estimate.
For example, the cost estimate minimally meets best practices for a well documented estimate because the Internal Revenue Service (IRS) did not provide detailed support for staff resources, and the cost estimate documentation only justified about 6 out of the 86 requested full time equivalent staff for IRDM, among other things. If documentation does not provide source data or cannotexplain the calculations underlying the cost elements, the estimates credibility may suffer. Although IRS has an independent office of cost estimators that can develop and update cost estimates using cost modeling software that generally follows GAOs best practices, this office did not develop the 2011 IRDM cost estimate. IRS policy does not require project teams to work with the office to update cost estimates. Additionally, IRSs cost estimation guidance for project managers is inconsistent regarding how cost estimates should be related to a budget, an inconsistency that could lead to budget requests that do not accurately estimate program funding needs.
The IRDM programs earned value management (EVM) data did not meet data reliability criteria in the areas GAO reviewed. For example, the IRDM project schedule was not properly sequencedmeaning activities were not properly linked in the order in which they are to be carried out. In addition, surveillance was not conducted on IRDMs EVM system, as required by the Office of Management and Budget and the Department of the Treasury. Surveillance involves having qualified staff review an EVM system. Because IRDMs 2011 cost estimate is based on unreliable EVM data, it does not provide adequate support for IRDMs budget requests. Until IRS addresses deficiencies in the EVM data, it cannot provide a reliable cost estimate for IRDM.
section_title: What GAO Recommends
paragraphs: GAO recommends that IRS ensure that IRDM has a reliable cost estimate, require certain project teams to work with its Estimation Program Office, improve cost estimation guidance, and improve the reliability of IRDMs EVM data. IRS agreed with one, partially agreed with one, and disagreed with two of GAOs recommendations. GAO generally disagrees with IRSs concerns, and still believes the recommendations have merit.

section_title: Letter
paragraphs: The financing of the federal government depends largely on the Internal  Revenue Service’s (IRS) efforts to collect taxes. To help carry out this  work, IRS initiated the Information Reporting and Document Matching  (IRDM) program in fiscal year 2009. IRS plans for the IRDM program to  build information technology (IT) systems that automatically compare— that is, match—different sources of tax information to improve tax  compliance. In May 2011, we issued a report that assessed IRS’s 2009  cost estimate for building IRDM IT systems, and found that it did not fully  meet best practices. We recommended that for any future updates to the  IRDM cost estimate, IRS ensure that the revised estimate be developed  in a manner that reflects the four characteristics of a reliable cost estimate  described in our report.IRDM in August 2011.
Since then, IRS provided a new cost estimate for  Having a reliable cost estimate—a summation of individual cost elements  using established methods and valid data—is vital for making informed  budgetary decisions and ensuring that a project is implemented as  planned. In an environment where federal funds are scarce, it is  imperative that IT projects, such as IRDM, are implemented as planned,  not only because of their value to the government, every dollar spent on one program will mean one less to fund other  efforts. Our objectives were to determine the extent to which: (1) the  IRDM funding request is supported by a reliable cost estimate and, if not  reliably supported, why not; and (2) IRS’s practices for capturing IRDM’s  actual costs and comparing them to estimated costs—known as “earned  value management” (EVM)—generate reliable performance data. We are  making four recommendations that warrant management’s consideration.
The IRDM system will increase information that is available for compliance purposes and  potentially raise significant revenue. Two components of the IRDM program, cost basis  and transaction settlement reporting, are expected to generate more than $16 billion over  the next 10 years, according to the Joint Committee on Taxation in 2008.
GAO-11-557. project data to provide reports on project status. We also compared  IRDM’s EVM system with guidance in the Department of the Treasury’s  Earned Value Management Guide. As part of that analysis, we assessed  IRDM’s implementation of three guidelines from the American National  Standards Institute (ANSI), which Treasury’s guidance states that IRDM  should be following. We selected the three guidelines to represent some  of the fundamental steps for maintaining a reliable EVM system, as  identified in our cost guide, and because these guidelines are also  Treasury Department requirements. For both research objectives, we  interviewed officials with IRS’s Modernization and Information Technology  Services (MITS) division, including those from the IRDM Program  Management Office, and the Investment Planning and Management  Office, which includes the Estimation Program Office (EPO).
We conducted this performance audit from August 2011 through January  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. We determined that the  IRS data we used were sufficiently reliable for our purposes. We also  made appropriate attribution indicating the sources of the data. We are  making recommendations to IRS to improve data reliability in the future.  See appendix I for more information on our scope and methodology.
subsections: section_title: Background
subsections: section_title: IRDM Program
paragraphs: IRS initiated the IRDM program, in part, to implement new information  reporting requirements, but more generally to increase voluntary  compliance with tax laws by expanding and maximizing IRS’s ability to  match existing and future information returns with tax return data and  establishing a new business information matching program. Previously,  IRS had only matched information returns to individuals’ and sole  proprietors’ tax returns. Under IRDM, IRS plans to build several new IT  systems and enhance some existing systems as well as implement  numerous organizational and process changes. IRS plans for IRDM to  use information returns to identify individual and business tax returns that  are likely sources of revenue, which the current individual tax return  matching system is not designed to identify. IRDM implementation is led  by IRS’s Small Business/Self Employed division and MITS, which is  leading the IRDM IT system development.
section_title: IRS Cost Estimation Process and Guidance
paragraphs: Cost estimates are a vital factor for sound management decision making  and they aid in the formation of a project’s budget. IRS uses cost  estimates, in part, to justify budget requests and prioritize the selection of  IT projects for possible funding. After an IT project is approved, the cost  estimate is later used as a starting point for developing the performance  measurement baseline for EVM, a project management approach that, if  implemented appropriately, provides management important tools such  as objective reports of project status and early warning signs of  impending schedule delays and cost overruns. Data from a reliable  performance management system, such as EVM, are necessary inputs  for an updated cost estimate, among other things.
OMB issued guidance on managing IT projects, which discusses cost  estimation and refers to our cost guide for how to meet cost estimating  requirements.continuously updated based on the latest information available to ensure  that they are current, accurate, and valid. According to our cost guide,  effective program and cost control requires ongoing revisions to the cost  estimate, budget, and projected estimates at completion. Specifically, our  guide states that estimates should be continuously updated with actual  costs incurred to that point so that significant cost, schedule, or  performance variances can be examined. In addition, it says that cost  estimates should be updated to reflect significant changes to a project’s   OMB guidelines state that cost estimates should be  scope or specifications and when certain projects approach key  milestones.
Within MITS, project managers and EPO are involved in estimating  program costs. EPO is an independent group of cost estimation experts  that assists project teams by developing and updating cost estimates  using a standard documented process. Project managers are responsible  for maintaining a program’s cost estimate. EPO only becomes involved in  updating a program’s cost estimate at the request of project managers,  according to EPO officials.
IRS procedures for developing, using, and updating cost estimates and  EVM are described in several guidance documents, specifically:
EPO’s Estimator’s Reference Guide, which is used by EPO staff, is  the general resource on the processes and procedures for developing  and delivering IT cost estimates. It discusses the technical aspects of  updating cost estimates, such as what documents are used in cost  modeling once a project has begun.  IRS’s Information Technology Investment Planning and Management  Guide (Investment Guide) outlines the framework for selecting,  managing, and evaluating IRS IT projects. The Investment Guide  includes discussions of how IT projects are selected using cost  information, and how managers should use cost information to  monitor a project. Project managers are responsible for managing  cost, schedule, and performance for a project.
MITS’s Estimation Procedures document describes IRS’s  organizational approach to cost estimation, applicable to all IRS  projects. The document is directed at project managers and it includes  discussions of the steps and staff roles necessary to develop an  estimate, and the circumstances when EPO typically becomes  involved with updating a cost estimate. The document states that  project managers are responsible for monitoring project progress and  suggests initiating assistance from EPO if a project meets certain  thresholds.
The Department of the Treasury’s Earned Value Management Guide  provides guidance for implementing EVM on major IRS projects.  Program or project managers have the ultimate responsibility for  implementing and monitoring the EVM system for their program or  project.
section_title: IRDM Program Cost Estimates
paragraphs: IRS developed two cost estimates for IRDM early in the program. In 2007,  IRS developed a preliminary cost estimate for budgetary purposes when  very little program information was available (referred to in this report as  the 2007 preliminary cost estimate). As shown in table 1, in 2007, IRDM  system development was estimated to cost about $5 million in fiscal year  2009 and about $23 million per year thereafter. In 2009, EPO developed  a solution concept-based estimate (SCBE, referred to in this report as the  2009 SCBE), which was more rigorous than the 2007 preliminary  estimate. The 2009 SCBE was developed before program implementation  began, when MITS had more information than it did in 2007, but system  design plans were still under development. The 2009 SCBE was about  $36 million less through the first 4 years of the project than the 2007  preliminary estimate.
IRS used the 2007 preliminary estimate to justify the initial IRDM budget.  According to a MITS official, the 2009 SCBE was not used for budgetary  purposes because program specifications were undergoing modification  in fiscal year 2010, requiring the full $23-million per-year funding. For  example, in 2010, IRS made several changes to the complexity of the  IRDM program, which included dividing it into four projects. One of the  projects required restructuring as a separate development effort, using  different resources and technologies. Changes also necessitated using  new software development methods not previously used within IRS, and  additional contracting support. According to MITS officials, such changes  increased IRDM funding needs above the amounts supported by the  2009 SCBE. As a result, IRS did not revise its initial funding request for  future fiscal years using the SCBE and instead relied on the 2007  preliminary estimate.
From fiscal years 2009 through 2011, IRS received about $52 million in  total funding for IRDM, of which the IRS had spent $46 million (when  accounting for $2.6 million carried over from fiscal year 2009 funds and  $5.8 million carried over from fiscal year 2010), through fiscal year 2011,  as shown in table 2.
In our May 2011 report,the most rigorous IRDM cost estimate available at the time and the 2007  preliminary estimate lacked documentation for a complete review. We  found that the 2009 SCBE did not fully follow best practices. We  recommended that if IRS updated the cost estimate, it should follow best  practices from our cost guide. In response to our report, IRS said it would  update the 2009 SCBE. IRS subsequently decided not to revise the  estimate because, according to officials, they already have a plan,  schedule and funding, the program is not over-budget, and the risks  associated with IRDM, and the program’s size, do not warrant an update.   we assessed the 2009 SCBE because it was  Over the summer of 2011, MITS provided us with additional cost  information. Officials referred to these documents as IRDM’s new cost  estimate, and they were used in supporting IRS’s fiscal year 2012 IRDM  budget. Consequently, in this report, we refer to the materials provided to  us as the 2011 cost estimate. This estimate was not a SCBE or  developed by EPO. The 2011 cost estimate was based on several data  sources, including IRDM’s Exhibit 300, EVM data, spend plans, and  schedule with work breakdown structures (WBS). MITS projects IRDM to cost $115 million for fiscal years 2012 through 2016, or about $23  million per year.
section_title: IRDM’s 2011 Cost Estimate Does Not Meet Best Practices for Reliability and Does Not Fully Support the Program’s Budget
subsections: section_title: IRDM’s Fiscal Year 2012 and Projected Budget Requests Are Not Supported by a Reliable Cost Estimate
paragraphs: According to best practices established by our cost guide, a cost estimate   We  should be comprehensive, well documented, accurate, and credible.assessed the 2011 cost estimate against cost estimation best practices  because IRS told us the estimate was used to support its budget requests  for fiscal year 2012 and beyond. While the 2011 IRDM cost estimate  shows some characteristics of a reliable cost estimate, it does not fully  meet best practices.
The estimate partially meets best practices for a comprehensive cost  estimate, as shown in figure 1. It reflects the current program schedule  and contains information about the program’s technical characteristics.  The estimate provided some details about costs for IRDM’s fiscal year  2012 budget request, but the cost estimate does not cover the program’s  entire life-cycle. Without fully accounting for life-cycle costs,  management may have difficulty successfully planning program resource  requirements and making informed resource-planning decisions. IRS  defined assumptions used to estimate some IRDM costs, but did not  provide the assumptions used to estimate labor or program operations  costs. Furthermore, IRS did not include ground rules used to develop  the estimate. Unless ground rules and assumptions are clearly defined,  the cost estimate will not have a basis to identify and mitigate areas of  potential risk.
The estimate minimally meets best practices for a well documented cost  estimate, as shown in figure 2. IRS provided supporting information for  some staff resources, but detailed data for the staffing level requested for  fiscal year 2012 was missing. The cost estimate documentation says that  the labor cost justification was captured in the resource loaded project  schedules, but we found that these schedules only justified about 6 out  of the 86 requested full-time equivalent (FTE) staff for IRDM.  Furthermore, although IRS officials cited the WBS as the basis for cost  projections, we found no evidence linking the WBS to cost. Multiple  documents linked software and hardware specifications to cost, but they  did not provide consistent cost information. As a best practice,  documentation should describe the source data used, the estimating  methodology, and show step-by-step how the estimate was developed.  Without a well documented cost estimate, the program’s credibility may  suffer because the documentation cannot explain the rationale of the  methodology or the calculations underlying the cost elements.
The estimate minimally meets best practices for an accurate cost  estimate, as shown in figure 3. Calculations in the estimate are  mathematically correct. However, documentation that IRS provided to  support estimated costs for IRDM hardware and software’s estimated  costs did not match estimates in IRDM’s spend plans. IRS officials said  the discrepancies occurred because the spend plans were developed  using more recent cost information for software purchases that were not  included in the supporting documentation. Additionally, the estimate does  not list any confidence levels or provide a range of possible costs.  According to best practices, unless an estimate is based on an  assessment of the most likely costs and reflects the degree of uncertainty  given all of the risks considered, management will not be able to make  informed decisions. IRDM uses EVM to identify variances between  planned and actual costs, but as discussed below, we found IRDM’s EVM  data to be unreliable, and there was no evidence that IRS uses actual  cost data to evaluate whether cost projections are realistic.
The estimate does not meet best practices for a credible cost estimate, as  shown in figure 4. For example, the estimate was not crosschecked or  assessed for risk and uncertainty. According to best practices, an  estimate without risk and uncertainty analysis can be unrealistic  because it does not assess how the cost estimate would be affected if, for  example, the schedule slipped, the mission changed, or a proposed  solution did not meet users’ needs. In addition, IRS did not perform a  sensitivity analysis. Further, there is no evidence that another office  performed a separate cost estimate—referred to as an independent cost  estimate—to validate the 2011 cost estimate. In previous work we found  that because of limited resources, IRS generally only does an additional  independent cost estimate for its largest programs, and according to  officials, IRDM is not considered a large enough program in terms of its  funding level. While some of MITS’s cost estimates are done by EPO— which is independent of program management offices—the 2011 cost  estimate was done by the IRDM program office.
Because the 2011 cost estimate does not meet best practices, it does not  provide reliable support for IRDM’s fiscal year 2012 budget request, or  any of the projected budget requests. IRS officials said current IRS policy  does not require projects to routinely re-estimate project cost. The IRDM  program office—which does not use the same software or modeling  techniques as EPO—relied on spend plans, EVM, and other documents  to estimate costs. In July 2011, MITS officials said that it would take 90  days for IRDM and EPO staff to complete a new cost estimate for IRDM.  When considering FTEs and time, a new cost estimate developed by  IRDM and EPO staff would require a total of about eight staff months.
section_title: IRDM Could Benefit From EPO Assistance on Updating Cost Estimates
paragraphs: EPO has specialized cost estimation tools, such as software that  incorporates many best practices from our cost guide, and expertise that  project teams can leverage to update cost estimates. If used correctly,  EPO estimation procedures could help IRDM management to maintain  reliable cost information for use in budget requests. EPO officials said  they did not work with the IRDM team to maintain an updated SCBE  because the team did not seek their assistance. IRDM was not required  to do so because MITS guidance, as of September 2011, does not  require project teams to consult with EPO when updating a cost estimate.  Without EPO involvement, IRS has less assurance that cost estimate  updates will follow best practices. Our cost guide states that cost  estimates should be (1) updated to reflect actual costs and changes (i.e.,  significant modifications to a project’s scope or specifications) in order to  keep the estimate current as the program passes through new phases  and milestones and (2) updated if there are significant cost, schedule or  performance variances. The continual updating of the cost estimate as  the program matures not only results in a higher-quality estimate, but also  gives cost estimators the opportunity to collect data for use in future  estimates as well as incorporate lessons learned.
Our cost guide also states that cost estimation work should be done by a  central independent estimating organization, and estimators should  monitor programs to determine whether preliminary information and  assumptions remain relevant and accurate. EPO has the following  characteristics—unlike the IRDM project team—that could help provide  more reliable cost estimate updates:
EPO is able to use robust cost estimation techniques, including the  SEER-SEM software cost estimation model.project histories and cost relationships to produce cost estimates and  can estimate costs consistent with best practices—such as adjusting  for risk and incorporating the results of a sensitivity analysis. When  EPO estimators validate or update a project’s funding requirements,  they tailor SEER-SEM to the project and use it to consider actual cost  data from the project team, according to EPO’s Estimator’s Reference  Guide. Estimators calibrate the model to include the schedule for  remaining work and evaluate and revise key cost drivers, according to  the guidance. As mentioned previously, the 2011 IRDM cost estimate  did not use a cost estimation model.
EPO, as a whole, has more cost estimation experience than the IRDM  project team. EPO’s six cost estimators have 43 years of combined  cost estimation experience. They all have received training in SEER- SEM and other cost estimation models. In addition, EPO officials said  that project teams generally would not have the technical skills to  update a SCBE using cost estimation models. Although the IRDM  project team has some cost estimation experience and relevant  training, IRDM officials do not have SEER-SEM training or  experience. Further, our analysis of IRDM’s 2011 cost estimate  illustrates that estimate updates done by project teams may not result  in reliable cost information. According to EPO officials, an updated  estimate developed by EPO would also be independent, more holistic,  and would include elements that project teams may miss.
EPO-produced updates can help build a historical record of IRS cost  estimate data. According to our cost guide, historical data are crucial  to developing high-quality cost estimates because estimators usually  develop estimates for new programs by relying on data from programs  that already exist and adjusting for any differences. EPO officials told  us that they are working to build a historical database that compares  estimated costs to actual costs. As IRS’s central estimating  organization, EPO is uniquely qualified to use cost estimate updates  as an opportunity to obtain data that are consistent with other  estimates and to use the data to build a historical cost database,  which can ensure that future cost estimates are credible.
According to MITS officials, it is up to project teams to seek EPO  assistance. However, MITS’s Estimation Procedures document, which  provides cost estimation guidance to project teams, suggests some cost  and schedule variance and project size thresholds that, if exceeded,  should cause project managers to contact EPO for an updated estimate.  It recommends EPO involvement in updating estimates when:  cost or schedule variance are 10 percent or greater for major projects;  cost or schedule variance are 25 percent or greater for non-major  projects; or a project with development/modernization/enhancement (DME) costs greater than $5 million reaches milestone 3.
IRDM meets the first threshold. Specifically, IRDM meets the IRS’s  criteria for a “major” project because IRDM’s projected life-cycle costs are  about $166 million, based on funding projections submitted to OMB. Also,  according to IRDM EVM data, as of September 30, 2011, the program  had a greater than 18 percent cost variance and an almost 13 percent  schedule variance. IRDM officials said they did not work with EPO to  update the 2009 SCBE because it was not required by current MITS  guidance.
DME is a term used by OMB to describe the program cost for new investments, changes  or modifications to existing systems to improve capability or performance, changes  mandated by the Congress or agency leadership, personnel costs for investment  management, and direct support. For major IT investments, this amount should equal the  sum of amounts reported for planning and acquisition plus the associated FTE costs  reported in the Exhibit 300. developing cost estimates in 2006. Most of its work has focused on  developing estimates for proposed projects, rather than updating  estimates for existing projects. The Estimation Procedures document was  developed in September 2011, and as of October 2011, officials said  EPO has six cost estimators on staff. As a result, EPO officials said that  project teams generally update IT project cost estimates without EPO  assistance and that, as of October 2011, EPO has been involved in few  cost estimate updates. However, senior MITS and EPO officials said they  would like for EPO to have a greater role in cost estimate updates. If IRS  does not have reliable cost estimate updates, projects may face risks and  their budget requests may not be adequately justified to inform decision  making; these outcomes could be even more significant for projects with  cost or schedule variances or high DME costs, such as IRDM.
section_title: MITS Guidance Has Inconsistencies and Could Better Link Cost Estimation Procedures with Budget Requests
paragraphs: MITS guidance documents used by project managers do not clearly  discuss the appropriate uses of different types of cost estimates.  According to current guidance used by EPO estimators, non-SCBE cost  estimates are less rigorous and are not for use in budgets, but as stated  above, neither the IRDM initial budget request nor the current and  projected budgets were developed using information from the 2009  SCBE.
Three IRS guidance documents describe the relationship between cost  estimates and budgets. However, the documents are directed at different  audiences, do not present consistent information, and contain different  levels of detail. Specifically:
EPO’s Estimator’s Reference Guide, used by EPO estimators, states  that budgets for IT projects should be established using SCBEs. IRS’s  SCBEs rely on cost estimation methods that incorporate best  practices from our cost guide, including considerations of risk, and  provide a level of confidence associated with the estimate. According  to our cost guide, for management to make good decisions, the  program estimate must reflect the degree of uncertainty, so that a  level of confidence can be given about the estimate. The Estimator’s  Reference Guide also discusses techniques that estimators can use  to update SCBEs, and aligns that process with annual budget  submissions. Although this guidance contains many best practices, it  is directed at cost estimators; therefore, project managers do not  typically have access to it or use it.
The MITS Investment Guide, directed at MITS project managers,  discusses the role of EPO in developing initial rough estimates and  budget-ready SCBEs and states that, if a project does not yet have an  SCBE, a rough estimate may be used as a placeholder in a budget  request. The guide requires that MITS staff should work to ensure that  if an SCBE exceeds an initial rough estimate, the project’s scope and  SCBE fit within the appropriated budget. However, the guide does not  discuss how, if at all, a budget request should be adjusted if an SCBE  provides an estimated cost that is lower than the budget, or how any  future cost information should be incorporated into budgets.
The Estimation Procedures document, directed at MITS project  managers, does not define types of cost estimates or discuss whether  they are appropriate for budget decisions. EPO officials said they did  not believe it is necessary to characterize the different types of cost  estimates in the Estimation Procedures document because they are  not necessary for defining the organizational approach to estimation,  which is the intent of the document. The document states that if  updated cost estimates indicate that a project’s budget needs to  change, the changes must be approved. However, it does not specify  who should approve the estimates.
Without consistent guidance about what types of cost estimates are  appropriate for budget requests, project teams may not use the best  information available. Our cost guide states that, as a best practice, an  estimate intended to support budgetary decisions should cover the  project’s entire life-cycle and should be supported by a description of the  program’s technical characteristics, which would be found in an SCBE.  Using a cost estimate that lacks sufficient rigor—such as a preliminary  cost estimate, instead of an SCBE—could lead to budget requests that do  not accurately reflect program funding needs. For example, the 2007  preliminary IRDM estimate lacks an uncertainty analysis, which would  provide a basis for adjusting the estimate to reflect unknown facts and  circumstances that could affect costs, and as a result, IRDM managers do  not have assurance that the program’s funding level remains appropriate.  Further, not providing project managers with guidance on how to  incorporate new cost information—either from an SCBE that has replaced  a preliminary estimate or from an updated cost estimate—into budget  requests could result in requests that do not reflect current or accurate  funding needs for a project.
section_title: Unreliable EVM Data Raise Additional Concerns about IRDM Cost Estimate
paragraphs: IRS provided EVM data in the 2011 IRDM cost estimate to justify its  budget requests, but we found that the program’s EVM data are not  reliable in any of the areas we reviewed. Reliable data on actual  performance, obtained from an EVM system, are a necessary input if an  updated cost estimate is to be considered accurate and credible.  Because IRDM’s 2011 cost estimate is based on unreliable EVM data, it  does not provide adequate support for IRDM’s budget requests. Until IRS  addresses deficiencies in its EVM data, it cannot provide reliable cost  estimate updates for IRDM. EVM data reliability deficiencies, such as  those we observed for IRDM, are common in federal agencies, and we  have also previously reported on inconsistencies in implementation of  EVM for IT projects at Treasury bureaus.
See GAO-09-3SP and Office of Management and Budget, Capital Programming Guide:  Supplement to Circular A-11, Planning, Budgeting, and Acquisition of Capital Assets  (Executive Office of the President, Washington, D.C.: August 2011). determine whether all program requirements have been addressed,  risks have been identified, mitigation plans are in place, and available  and planned resources are sufficient to complete the work.
Using qualified staff, conduct surveillance on the EVM system.
Surveillance is reviewing a contractor’s EVM system to observe ANSI  compliance and how well a contractor is using its EVM system to  manage cost, schedule, and technical performance.
Treasury’s EVM guidance requires a project of IRDM’s size to follow an  abbreviated set of 10 ANSI guidelines, and to conduct surveillance on the  EVM system. Other departments also scale ANSI guidelines according to  the size of projects, which could result in some agencies not fully  following certain best practices. Further, projects like IRDM, according to  Treasury guidance, only need to complete an independent baseline  validation—which although not defined in the guidance, appears to be a  less rigorous version of an integrated baseline review. Following ANSI  guidelines and conducting an integrated baseline review and surveillance  can help ensure that EVM data can indicate how well a program is  performing in terms of cost, schedule, and technical matters. This  performance information is necessary for proactive program management  and risk mitigation, and to maintain a reliable cost estimate. Where  applicable, we assessed IRDM’s EVM data against the standards cited in  the Treasury guidance.
subsections: section_title: IRDM’s EVM System Is Not Compliant with Key Required Guidelines
paragraphs: We assessed IRDM on three ANSI guidelines, which are fundamental  elements for an EVM system and are included in Treasury’s abbreviated  10 EVM guidelines. We found that IRDM’s EVM system is not compliant  with these guidelines. For an overview of our findings on IRDM’s EVM  data reliability, see table 4 in appendix III.
For each selected guideline we found:
WBS: This ANSI guideline states that authorized work elements for  the program should be defined, which typically includes using a WBS  tailored for effective internal control. Further, a project’s schedule,  cost estimate, and EVM system should be based on the same WBS,  according to our cost guide. The WBSs used in IRDM’s schedules do  not match the WBS used for EVM. The WBSs in four of the five IRDM  schedules reflect detailed project-level tasks, while the WBS used for  the EVM data is only broken down by contractor and government  efforts, and does not include any project-level data. The WBS used to  inform the 2011 IRDM cost estimate was not broken down by  contractor or government data, and it did not provide costs for detailed  tasks. Without a WBS, from which to measure progress and to serve  as a consistent framework for the schedules and EVM, there is no  basis for reliable EVM data, according to our cost guide. An IRS  official said that the WBS in the IRDM schedules is not the primary  source of financial information for IRDM. Instead, officials use IRS’s  financial tracking system, which is much less detailed than the  schedules’ WBS, to obtain project level data for EVM. This technique  for gathering EVM data at a project level is contrary to the purpose of  EVM, which is to integrate cost, schedule, and technical data from  detailed work packages that can be monitored for variances against  the original plan. Since a resource-loaded schedule forms the  foundation for the EVM baseline, both the schedule and the EVM data  should be based on the same WBS, according to our cost guide.Because the financial tracking system only provides project level data,  and financial information cannot be traced to the WBS elements in the  schedules, the cost associated with the project tasks is unknown.
Sequencing: This ANSI guideline states that projects should have a  schedule that describes the sequence of work—that is, a list of  activities in the order in which they are to be carried out—and  identifies significant task interdependencies required to meet project  requirements. All of the IRDM schedules had significant problems with  sequencing. For example, many predecessor and successor tasks  were not linked to one another, which are necessary for properly  sequencing work so that the schedule will update in response to  changes. Because the schedule is missing so many of these “logic  links,” it will not automatically recalculate forecasted start and finish  dates of remaining activities. Thus, any activities that are late will not  automatically recalculate the dates for affected successor activities.  IRS officials said they are aware of some issues with missing links. As  a result of these missing links in all of the IRDM schedules we  reviewed, IRDM’s schedules are not reliable. Because the schedules  form the basis for the performance measurement baseline used to  track cost and schedule variances in an EVM system, the data from  the IRDM EVM system are not reliable.
Time-phased budget baseline: This ANSI guideline states that a  program should establish and maintain a time-phased budget  baseline, at the control account level, against which performance  can be measured. Resources must be accounted for in a schedule in  order to develop this baseline, according to our cost guide. IRS was  unable to show evidence that it has established and maintained a  time-phased budget baseline for IRDM. Specifically, the program does  not have one overall schedule, and none of the IRDM schedules we  reviewed were completely resource-loaded.
IRS officials said although they do not have a program level schedule for  IRDM, the individual project schedules are linked through  interdependencies. However, we could not identify these links in our  analysis. IRS officials said the resource-loaded schedules do not show all  project resources because some resources, such as contractor personnel  being used across projects, are used in more than one IRDM project.  Without schedules that include the resources needed to complete tasks,  IRS was unable to prove that it had established and maintained a time- phased budget baseline. This baseline is a critical EVM component for  measuring IRDM’s performance.
section_title: IRS Did Not Validate the Baseline for IRDM’s EVM System
paragraphs: According to Treasury EVM guidance, an independent baseline validation  should be conducted for a DME project like IRDM. As stated above, GAO  and OMB consider an integrated baseline review to be a key element of  EVM data reliability, while Treasury guidance allows projects like IRDM to  complete a less-rigorous independent baseline validation. IRS did not  conduct an integrated baseline review or an independent baseline  validation for IRDM. IRS officials told us that many of the activities  typically done during an integrated baseline validation were performed— such as developing a WBS and schedules. However, as previously  discussed we identified problems with the WBSs and schedules and the  baseline process. Without a comprehensive integrated baseline review or  independent baseline validation, and resolving any issues, IRS has not  sufficiently evaluated the validity of IRDM’s baseline. This calls into  question the reliability of IRDM’s EVM data, and could affect the  program’s ability to identify and mitigate risks.
section_title: IRS Is Not Conducting Surveillance on IRDM’s EVM System
paragraphs: OMB and Treasury require surveillance on EVM systems, and, according  to our cost guide, surveillance should be conducted to check whether the  EVM system summarizes timely and reliable cost, schedule, and  performance information, among other things. IRS officials said IRS is not  performing surveillance on the IRDM EVM system because they did not  believe it was required and because, according to officials, IRS does not  have staff with the necessary technical skills to conduct surveillance. IRS  officials said Treasury is reviewing whether Department-level surveillance  would be efficient. It is important for the agency to conduct surveillance of  EVM systems to ensure that contractors are following their own  processes and satisfying ANSI guidelines.
section_title: Conclusions
paragraphs: The 2011 IRDM cost estimate does not fully meet best practices for a  reliable estimate. It is important for IRDM to have a cost estimate that  meets best practices to inform budgetary decisions and ensure that the  program is implemented as planned. This standard is particularly  important in a budgetary environment with scarce resources. IRDM’s  2011 cost estimate could be improved by using EPO’s expertise to  ensure the cost estimate follows best practices from our cost guide.  Additionally, more consistent guidance on using cost estimates to develop  budget requests could help program managers for IRDM, as well as other  programs, make budget decisions that are supported by reliable cost  information.
IRS could increase the credibility of IRDM’s 2011 cost estimate by  ensuring that IRDM’s EVM data are reliable. Such reliability could also  allow for IRS to update projected costs for the remainder of IRDM’s  implementation. Using a WBS that is developed using best practices from  our cost guide could provide a baseline from which to measure progress,  a key component to an EVM system. Similarly, developing a single  integrated schedule for IRDM, that contains all resources needed to  implement the program, could provide more meaningful EVM data.  Finally, providing oversight of the EVM system, through an independent  baseline validation and EVM surveillance, could help identify potential  program risks and any possible issues with contractor performance.
section_title: Recommendations for Executive Action
paragraphs: To improve the quality of cost and budget information for IRS IT projects,  we recommend that the Commissioner of Internal Revenue take the  following four actions:  1.  Ensure that the IRDM life-cycle cost estimate is reliable and that  budget requests are justified by a reliable cost estimate that follows  best practices.  2.  Require project managers to consult with EPO to determine if projects  could benefit from EPO assistance in updating cost estimates for  programs that exceed thresholds recommended by MITS’s Estimation  Procedures document. For those projects where EPO does not  update the cost estimate, IRS should require that the decision and  rationale be documented. EPO should use the information from  updated cost estimates to develop a historical repository of cost  estimation data.  3.  Review all guidance applicable to cost estimates and take steps to  ensure that they are consistent. As a first step, IRS guidance should  require the use of current and reliable project cost estimates to inform  budget requests, in accordance with the Estimator’s Reference Guide.  4.   Improve the reliability of IRDM’s EVM data, specifically: address WBS issues by developing an EVM baseline for IRDM  that reflects the same WBS as the detailed schedule and IRDM  cost estimate; address sequencing issues and enable the development of a time- phased budget baseline by creating a single integrated master  schedule for IRDM that is properly sequenced and resource- loaded so that effective and meaningful EVM data can be  obtained to better manage the program;  conduct an independent baseline validation for the IRDM EVM  baseline; and  conduct independent surveillance of EVM systems to ensure that  data are reliable.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Commissioner of Internal  Revenue for his review and comment. We received written comments  from the Deputy Commissioner for Operations Support, which are  reprinted in appendix IV. We sought clarification on IRS’s written  response in regards to whether it agreed with two of our  recommendations, and on a reference to OMB guidance. IRS provided us  with additional comments, which are summarized below. In addition, the  agency provided technical comments, which we incorporated into the  report as appropriate. IRS agreed with one of our four recommendations,  partially agreed with another, and disagreed with two.
While IRS’s comment letter did not address the recommendation, the  Director of Risk Management in MITS’s Strategy and Planning Office told  us the agency agrees with the recommendation to require certain IT  project managers to consult with EPO about updating cost estimates,  documenting decisions not to update cost estimates, and placing data  from updated cost estimates in a repository.  Similarly, IRS’s comment  letter did not address our recommendation to ensure that its cost  estimation guidance is consistent. However, IRS officials said they  partially agree with this recommendation and have taken steps to ensure  that their estimation practices and procedures follow consistent,  documented guidance. They noted that in all instances, however, IRS IT  cost estimates will be based on the best information available at the time  the estimate is requested or required as opposed to our recommendation  that IRS require the use of current and reliable cost estimates to inform all  budget requests. We note in our report that using an unreliable cost  estimate could lead to budget requests that do not accurately reflect  program funding needs. Once done in conformance with guidance and  best practices, current and reliable cost estimates can be maintained  through normal required monitoring and cost tracking procedures, unless  significant changes in project circumstances warrant updating the cost  estimate.
IRS disagreed with our recommendation to ensure that the IRDM life- cycle cost estimate is reliable and that budget requests are justified by a  reliable cost estimate that follows best practices.  In its comment letter,  IRS wrote that implementing the recommendation would require it to  spend resources that do not directly contribute to the successful  implementation of the IRDM program, and that the IRDM program is  within its budget and schedule. At the end of fiscal year 2011, IRDM was  under budget by more than 18 percent and behind schedule by almost 13  percent, equal to over 4 months behind schedule, according to IRDM’s  EVM data. Because the IRDM program does not have a reliable cost  estimate, budget authorities do not have reliable information to determine  an appropriate funding level. Such variances indicate that the current  funding level may not be appropriate. As we reported, IRS estimated in  July 2011 that it would take about 90 days, comprising 8 staff months (or  a direct staff cost likely less than $200,000, according to our calculations  using a top government salary rate and general benefits rate), to  complete a new cost estimate for IRDM.  While we agree that federal  resources are tight, we believe that such an investment could produce  benefits that not only improve the reliability of the IRDM cost estimate, but  also better ensure that IRS requests the correct amount of resources to  ensure IRDM fully achieves successful implementation.  Further, benefits  of a new estimate would also stretch beyond that program and provide an  important foundation for improving IRS cost estimates in general. The  OMB Capital Programming Guide directs agencies to develop sound cost  estimates based on our cost guide and states that during the budget  process, the credibility of the costs will be examined, and OMB and the  Congress will hold agencies accountable for meeting the schedule and  performance goals within the cost estimates. More reliable cost estimates  enable Congress and other budget authorities to make more complete  and informed decisions.
IRS also disagreed with our recommendation to improve the reliability of  EVM data, and stated that OMB’s revisions to Circular A-11 remove EVM  system requirements due to negative cost benefit.  We disagree with IRS,  as Circular A-11, Appendix J and the Capital Programming Guide still  contain language that directs agencies to use EVM for major projects.  Former Circular A-11, section 300.7, instructed agencies to use EVM  system requirements to identify areas where problems are occurring  when reporting on ongoing investments. While current guidance for  Exhibit 300 no longer explicitly discusses EVM reporting on ongoing  investments under section 300.7, other sections of the guidance still  direct the use of EVM for managing IT capital assets and state that in  general cost, schedule and performance goals are to be controlled and  monitored by using an EVM system.  Moreover, our report assessed the  reliability of IRDM’s EVM data because the data were included in IRDM’s  2011 cost estimate and the data are used to track the program’s  progress.  Regardless of OMB requirements, any data used for cost  estimation and program management, particularly when it helps to  support a budget request, should be reliable.
We will send copies of this report to the Chairmen and Ranking Members  of Senate and House committees and subcommittees that have  appropriation, authorization, and oversight responsibilities for IRS. We will  also send copies to the Commissioner of Internal Revenue, the Secretary  of the Treasury, the Chairman of the IRS Oversight Board, and the Acting  Director of the Office of Management and Budget. Copies are also  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions concerning this report, please  contact me at (202) 512-9110 or brostekm@gao.gov.  Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: This report builds on our May 2011 reportService’s (IRS) Information Reporting and Document Matching (IRDM)  program and analysis performed during a budget justification review of  the IRDM program, conducted from June 2011 to August 2011, which we  provided to Congress as technical assistance. In the May 2011 report, we  assessed IRDM’s 2009 solution-concept based cost estimate. For the  budget justification work and this report, we assessed IRDM’s 2007  preliminary cost estimate. Also for this report, we assessed IRDM’s 2011  cost estimate.
See GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and  Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). documentation provided to us. Once we determined that the 2011 cost  estimate did not fully meet best practices, we determined why this  occurred. To this end, we interviewed officials in IRDM’s Program  Management Office and MITS’s Estimation Program Office (EPO). We  also compared IRS guidance that addresses cost estimation—the EPO’s  Estimator’s Reference Guide, MITS’s Estimation Procedures document,  and IRS’s Information Technology Investment Planning and Management   Guide—to criteria in our cost guide.
To assess the extent to which IRS’s practices for capturing IRDM’s actual  costs and comparing them to estimated costs, or EVM, generate reliable  performance data, we compared the EVM data for IRDM and IRS’s  process for maintaining the data to the high-level EVM data reliability  tasks outlined in our cost guide. We assessed the extent to which  IRDM’s EVM data adhered to three of the American National Standard  Institute’s (ANSI) 32 guidelines; we selected the three guidelines to  represent some of the fundamental steps for maintaining a reliable EVM  system, as identified in our cost guide, and because these guidelines are  also included in the Department of the Treasury’s Earned Value  Management Guide, which applies to IRS (see appendix III for a list of the  data reliability tasks and ANSI guidelines we reviewed). In situations  where Treasury’s Earned Value Management Guide did not require  certain OMB or GAO best practices, we assessed IRDM’s EVM practices  against the Treasury guidance. To do this analysis, we compared the  work breakdown structures used in IRDM’s EVM system, schedules, and  cost estimates and identified differences in each. We also assessed each  of the five IRDM schedules against scheduling best practices for  ensuring that the activities are sequenced and related using network  logic, as identified in our cost guide.
For both objectives, we interviewed IRS officials in the MITS division,  specifically, officials from the IRDM Program Management Office and the  Investment Planning and Management Office, which includes EPO. We  spoke primarily with officials at IRS Headquarters in Washington, D.C.  and IRS’s division office in New Carrollton, Maryland, where the officials  responsible for IRDM are located. To assess the reliability of the cost  estimate data that we used to support findings in this report, we reviewed  relevant program documentation, such as cost estimation spreadsheets,  as available, to substantiate evidence obtained from interviews with  knowledgeable agency officials. We found the data we used to be  sufficiently reliable for the purposes of our report. As appropriate, we  attributed the sources of the data. We are making recommendations to  IRS to improve data reliability in the future.
We conducted this performance audit from August 2011 through January  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Assessment of MITS’s Current IRDM Cost Estimate
paragraphs: The following table outlines our assessment of the extent to which the  Internal Revenue Service’s (IRS) 2011 Information Reporting and  Document Matching (IRDM) program cost estimate meets best practices,  depicted in figures 1-4.
section_title: Appendix III: High Level Assessment of the Reliability of IRDM’s EVM System
section_title: Appendix IV: Comments from the Internal Revenue Service
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Libby Mixon, Assistant Director;  Laurel Ball; Amy Bowser; Bill Cordrey; Jennifer Echard; Robert Gebhart;  Paul Middleton; Donna Miller; Sabine Paul; Karen Richey; Stacy Steele;  and Lindsay Swenson made key contributions to this report.