Taxpayer Account Strategy: IRS Should Finish Defining Benefits and Improve Cost Estimates
Released: 2011-03-24T00:00:00

Letter

Background

IRS’s Business Systems Modernization program, which began in 1999, is a multibillion-dollar, high-risk, highly complex effort that involves the development and delivery of a number of modernized tax administration and internal management systems, as well as core infrastructure projects, which are intended to replace the agency’s aging business and tax processing systems and provide improved and expanded service to taxpayers and internal business efficiencies for IRS. CADE, one of the core systems of the Business Systems Modernization program, was intended to provide a modernized system of taxpayer accounts, with the ultimate goal of eventually replacing the Individual Master File (IMF), a 1960s legacy tax master file that maintains all taxpayer records for individual taxpayers.

The IMF is the authoritative data source for individual tax account data. Tax data and related information pertaining to individual taxpayers are posted to the IMF, and the IMF is updated annually to incorporate new tax law procedures and changes. The IMF is a critical component of IRS’s ability to process tax returns, as all the other IRS information system applications use data from this source. According to IRS, the IMF is a complex computer system that maintains all individual taxpayer records but has inherent limitations that significantly constrain IRS’s ability to achieve its mission. The delays in weekly updates of taxpayer accounts, the lack of synchronization of taxpayer data across dozens of systems, the complexity of the data, the large volumes of data, and the ongoing need to embed new tax laws and business rules contribute to IRS systems storing, sharing, and processing data that is often incomplete, untimely, or inaccurate.

With CADE, IRS started establishing a database and related functionality for posting, settlement, maintenance, refund processing, and issue detection for taxpayer account and return data for relatively simple tax returns for individual taxpayers. Since 2005, CADE has processed and recorded tax return and tax account information for increasing numbers of individual taxpayers. IRS reported that, in 2010, CADE processed about 41 million relatively simple returns (about 30 percent of the total individual income tax returns received); and issued about 36 million refunds that totaled in excess of $66 billion (approximately 14 percent of total refunds), and processed over 7 million taxpayer payments amounting to approximately $90 billion—with zero defects in balancing.

In addition, IRS reported that it issues refunds with CADE approximately 5 days faster than refunds processed by the IMF.

Challenges and Risks Facing CADE That Led to CADE 2

Although IRS made progress in processing individual taxpayer returns with CADE, IRS found that each successive release of the system was far more difficult as more complex accounts were to be supported, raising concerns about the effort required to address these increasing complexities. Also, since the CADE system operates concurrently with IMF processing, work must be done across the two environments, increasing the complexity of filing season operations and resulting in a greater risk for errors in processing taxpayer account information. IRS was concerned that, without a significant change in approach, CADE would not be completed until at least 2020.

These challenges led the IRS Commissioner to form a team of senior IRS technologists and external advisers to review CADE and IRS’s modernization strategy for individual taxpayer accounts. The review identified six risks with the CADE strategy. As a result of the review, in August 2008, IRS began defining a new strategy to address the challenges confronting CADE and deliver improved individual tax processing capabilities sooner. Specifically, the new strategy called for accelerating completion of a modernized database and converting to a single processing system sooner than the current approach would allow. The new strategy is referred to as CADE 2. Table 1 describes the CADE risks reported by IRS and how CADE 2 intends to address them.

As noted in the table, IRS used prototypes to test and prove complex technical and performance concepts early and provide guidance and information needed for detailed design. These prototypes were completed in 2010, although some remaining tasks in one of them—the Database Performance Test Prototype—will be completed later.

CADE 2 Is Expected to Deliver Functionality in Three Phases (Transition States)

CADE 2 is expected to deliver its functionality incrementally through three phases known as transition states. Table 2 shows the target completion dates and key characteristics for transition state 1 (TS1), transition state 2 (TS2), and the target state.

IRS Has Established Two Projects to Achieve TS1 Goals

TS1 consists of the following two projects: Daily Processing: According to IRS, this project is to enable IRS to process and post all individual taxpayer returns filed and other transactions by updating and settling individual taxpayer accounts in 24-48 hours with current, complete, and authoritative data, and provide employees with timely access. More specifically, through the Daily Processing project, IRS is expected to modify the IMF to run daily, i.e., modify current transaction processing to occur daily versus weekly; make changes to the Integrated Data Retrieval System (IDRS) and the notice generation process to support the daily vs. weekly processing cycle; and prepare daily “loads” to update the CADE 2 database for IDRS and other downstream systems as they are able to support daily processing.

Database Implementation: According to IRS, this project is to establish the CADE 2 database, a relational database that will house data on individual taxpayers and their accounts; develop a capability to transfer data from IMF to the database; and provide for the transfer of data from the database to downstream IRS financial, customer service and compliance systems. Specifically, the project is expected to enable IRS to perform a one-time initialization of the database when TS1 begins (this will involve loading all taxpayer account data from IMF and the CADE database into the CADE 2 database and then validating it); perform daily updates to the database after TS1 begins by extracting, transforming, and loading daily processing changes to IMF data; and utilize the database to provide individual taxpayer account information to selected downstream systems to assist IRS service and compliance personnel. For example, the CADE 2 database is intended to provide daily updates to IDRS, as well as weekly updates to the Integrated Production Model (IPM)—a downstream data repository designed to support IRS compliance functions that house IMF and Business Master File data, information returns, and other data. IPM is expected to include current individual taxpayer data from the CADE 2 database.

Regarding the status of these two projects, IRS completed efforts to define the TS1 projects’ logical design in December 2010 and began the detailed design phase, which includes activities such as documenting the physical design of the solution. IRS expects to complete this phase by April 2011.

IRS has also defined overall objectives for TS2 and the target state, but detailed planning of TS2 is in the early stages, with the formation of a planning team that expects to define a high-level approach by early May 2011.

Treasury Inspector General for Tax Administration and GAO Have Recently Reported on CADE 2

In December 2009, the Treasury Inspector General for Tax Administration (TIGTA) identified several challenges that IRS needed to address to effectively manage identified CADE 2 risks, including, among other things, implementing a governance structure for the PMO to provide oversight and direction for the implementation of CADE 2 and developing contingency plans in the event that CADE 2 cannot be fully implemented. IRS officials completed a contingency strategy in November 2010.

In May 2010, we reported that, while much had been done to define CADE 2’s transition states, IRS had not identified time frames for completing key planning activities for TS2. Consequently, we recommended IRS establish these time frames. In response, IRS stated that it would develop a plan for launching TS2 activities that will outline the approach and a high-level plan for the transition state. IRS also noted that business requirements would be completed in parallel with the development of the plan. In December 2010, IRS initiated TS2 planning activities and set a target of May 2011 for agreeing on the scope and plan for the transition state.

In November 2010, TIGTA reported on the prototypes undertaken by IRS to gain confidence in the CADE 2 solution. TIGTA found that the five prototype teams that were established generally managed their objectives effectively, and that the teams also identified risks that faced the successful execution of the prototype plans and took steps to overcome the barriers. TIGTA recommended that IRS take several actions to reemphasize compliance with the elements of the CADE 2 Prototype Process. In its response, IRS agreed with TIGTA’s recommendations and described steps to address them.

IRS Has Established a New Management Structure for the CADE 2 Program

While the MITS organization has primary responsibility for developing, managing, and delivering the CADE 2 program, IRS has established a new governance approach for the program, including the CADE 2 PMO that is to manage the program and the relationship with the program’s stakeholders, define the solution specifications to meet the requirements and the delivery of the program scope, and specify the roles and responsibilities of all parties. The new governance approach is intended to foster rapid decision-making and proactive risk management and issue resolution, as well as ensure that the appropriate stakeholders are involved, engaged, and collaborating as a unified team in making CADE 2 decisions.

The CADE 2 PMO is headed by an Associate Chief Information Officer and made up of several groups, each responsible for a different aspect of the program. These groups include Delivery Management. Responsible for overseeing and coordinating activities across the projects that are implementing the solution for each of the transition states.

Program Management and Control. Responsible for executing the program management processes, CADE 2 solution planning, and governance and control activities.

Chief Architect. Responsible for setting the overall technical direction in alignment with IRS’s business and technical architecture, as defined in the enterprise architecture.

Chief Engineer. Responsible for reviewing detailed design and interfaces to ensure the CADE 2 solution will work, is secure, and integrates with other systems and infrastructure.

Business Operations. Responsible for providing the administrative support for all branches of the PMO.

These groups are to work with (1) delivery partners who are other MITS organizations (e.g., Applications Development and Enterprise Operations) that work to deliver the program scope, (2) business partners (e.g., W&I and the Chief Financial Officer) who are organizations who either use the system or define the business requirements and, as such, play a key role in overseeing CADE 2, and (3) other stakeholder groups who support, influence, or oversee the program.

To oversee and guide the PMO, IRS also established several executive- level boards and advisory councils composed of business and IT officials. The executive-level boards include the following: The CADE 2 Executive Steering Committee, which consists of senior executives from MITS, W&I, and Department of the Treasury, and serves as an oversight group that ensures the program stays aligned with the IRS strategic plan and approves decisions with significant organizational or external impact; and The CADE 2 Governance Board, which consists of Associate Chief Information Officers from CADE 2 and Applications Development and the business modernization executive from W&I, and ensures that objectives are met; decisions and issues are resolved in a timely manner; risks are managed appropriately; and the expenditure of resources allocated is fiscally sound. The CADE 2 Governance Board also approves program risk response plans, milestone exits, and resolves escalated issues.

The advisory councils include the Executive Oversight Team, which consists of executives from the CADE 2 PMO, delivery partners, and senior-level business executives, and serves as the oversight for the day-to-day execution of the CADE 2 program; Associate Chief Information Officer Advisory Council representing leaders of each of the MITS technical units; Program Leadership Advisory Council, a broad group of executives from W&I, Agency Wide Shared Services, and across MITS that consult on key issues, risks, analyses, and recommendations on an as needed basis; and Architecture/Engineering Review Council recently created to provide architectural and engineering leadership across the program to ensure design and development efforts are complete and adhere to solution architecture and engineering best practices.

In addition, while the projects chartered under CADE 2 are expected to follow the IRS enterprise life cycle methodology that other projects are required to follow, IRS also defined a program management framework for CADE 2 that defines the phases, activities, deliverables, milestones, and reviews necessary to manage both the program and each of its component projects in a coordinated and integrated manner. The PMO and supporting boards are to play key roles in reviewing deliverables and approving milestone exits for the program and assuring that the projects are properly aligned and integrated with the program.

Finally, IRS has also defined and started implementing several processes to support its management of the CADE 2 program. They include processes for managing risks, the integrated master schedule, the prototype process, and other activities. While several of these processes were already established for projects, IRS enhanced them for CADE 2 to take into account the complexities associated with integrating projects into a program.

IRS Has Identified Benefits and Set Most Related Targets for the First Phase but Has Yet to Do So for the Second Phase

For TS1, IRS has identified 20 benefits. These benefits span three categories: service; compliance; and other benefits, including reduced system costs and improved security benefits.

Our past work at IRS established that quantitative targets can be useful for tracking program performance. While it may not always be possible to quantify targets, doing so helps to objectively measure the extent to which expected benefits have been realized.

As shown in table 3, IRS has set quantitative targets for 15 of the 20 benefits for TS1. (Table 7 in app. II describes all the benefits IRS has identified and the related quantitative targets where they exist.)

An example of a benefit with a quantitative target is the percentage of refunds IRS will process on a daily basis. IRS expects CADE 2 to increase this from 30 to 80 percent, resulting in faster refunds to many taxpayers. Another example is the security benefit of reducing the number of IRS databases that contain extracts of data from CADE 2. IRS expects this to be reduced by 20 percent from the 2011 baseline.

IRS officials have stated that they have not set quantitative targets for five TS1 benefits for two reasons. First, officials told us that two benefits related to improved security and updated account information cannot be quantified and therefore do not have numeric targets. (See app. II for more details.) Second, IRS officials stated that competing priorities have impeded their ability to define the remaining targets. While they stated they plan to define these targets, they have not committed to specific time frames for doing so. Until IRS has set quantitative targets where feasible in TS1, it and other stakeholders including Congress will not have complete information on the specific benefits that can be expected from funds spent on this phase. In addition, it will be more difficult to objectively measure whether expected benefits have been delivered.

In contrast with TS1, IRS has not finalized the benefits expected in TS2 or defined related quantitative targets. Specifically, IRS has identified a list of 22 service, compliance, and other benefits including addressing financial material weaknesses it expects to achieve beyond TS1. These benefits are summarized in table 4 and described in greater detail in table 7 in appendix II. However, IRS has not yet determined whether these benefits are to be delivered in TS2 or the target state. According to IRS, committing to what can be delivered in TS2 is contingent upon a number of decisions yet to be made. Specifically, officials stated that IRS must first finalize its approach for TS2 (as noted earlier, the agency expects to do this by May 2011). Officials also stated that funding decisions for the upcoming fiscal year could impact IRS’s plans for TS2.

In addition, IRS has not finished defining the full range of possible benefits it expects beyond TS1. For example, one possible compliance benefit that IRS is exploring is requiring providers of information returns to send those returns to IRS and taxpayers at the same time, which would allow IRS to use this data earlier to conduct some compliance checks. Conducting earlier analyses of tax return information using CADE 2 and IPM could allow IRS to stop an erroneous refund before it goes out. IRS currently waits until well after the filing season to conduct some compliance checks, such as computerized matching of information returns and tax returns and examinations of taxpayers’ books and records. Such compliance checks can lead to costly collection efforts from noncompliant taxpayers.

Finally, as summarized in table 4 and shown in greater detail in appendix II, IRS has not yet set quantitative targets for any of the benefits defined or committed to any time frames for doing so. As noted for TS1, until IRS has set quantitative targets where feasible in TS2, it and other stakeholders will not have complete information on the specific benefits than can be expected from funds spent, and it will be difficult to objectively measure whether expected benefits have been delivered.

We acknowledge that thinking through the expected TS2 benefits (including those from the table in app. II and other potential benefits) and related targets as the approach and design are being considered may result in having to make adjustments later. However, the benefits and targets generated at this stage can influence design decisions and allow IRS to identify early how associated systems and business processes might be affected.

IRS’s Estimates the First Two Phases of CADE 2 Will Total About $1.3 Billion; Process for Developing Estimates Was Largely Consistent with Best Practices

In July 2009, IRS reported preliminary life cycle cost estimates for TS1 and TS2 of about $1.3 billion through 2024, including about $377 million for development and $922 million for operations and maintenance. The estimates include costs for eight categories of work needed to achieve the goals defined for TS1 and TS2. The majority of the costs are for the Operational Framework category ($663 million or 51 percent of total costs), which is to provide for the infrastructure on which to develop, test, and deploy the new and enhanced applications under the CADE 2 program. The CADE 2 preliminary cost estimates are summarized in table 5.

The estimated annual costs from 2009 through 2024 are shown in figure 1. At the time they were developed, IRS estimated CADE 2 development costs to end in 2015 and total costs to peak during 2011 at about $162 million. Estimated costs between 2015 and 2024 are all recurring operations and maintenance costs.

IRS’s estimates did not include expected costs for the following items: operations and maintenance of the IMF system to include enhancements needed to achieve the CADE 2 target state, prototype development and assessments, technical training (estimated at $1 million to $3 million), work required to reach the final target state, and process reengineering costs/business costs associated with implementing the current weekly batch processing of accounts to the daily processing required at the completion of TS1.

With Some Exceptions, Process for Developing Preliminary Cost Estimates Was Consistent with Best Practices

The process IRS used to develop the CADE 2 preliminary cost estimates was generally consistent with the best practices outlined in the GAO cost estimating guide. Specifically, IRS followed a well-defined process for developing a comprehensive, documented, accurate, and mostly credible life cycle cost estimate for the first two phases of CADE 2. For example, the estimation process was performed in accordance with a well-defined plan by an independent Estimation Program Office. Parametric estimation models were used that contained catalogs based on historical data, along with cost estimating relationships, designed to produce estimates analogous to similar programs. In addition, IRS used simulation techniques to incorporate risks and determine confidence bounds on the estimates.

Further, IRS plans to update its estimates, consistent with the best practice to update estimates to reflect changes in technical or program assumptions. Specifically, since the estimates were prepared, IRS has developed a better understanding of the work required to meet CADE 2 objectives for TS1 and the current work breakdown structure (i.e., definition of work to be performed) no longer matches the one used to develop the preliminary estimates. IRS officials told us that, consistent with IRS’s Estimation Program Office Estimator’s Guide, they plan to revise their estimates after detailed design information for the database implementation and daily processing projects is completed and expect them to be available by the completion of our audit. CADE 2 PMO officials stated they did not expect the revised estimates to differ significantly from the current estimate. It is important to note that, even when all TS1 cost elements have been updated to accurately reflect the current understanding of the work to be performed, the total cost of ownership through TS2 could still significantly differ from the new estimate, given that, as previously noted, the approach for TS2 has yet to be defined.

We identified the following three practices IRS did not follow that could strengthen its methods for estimating CADE 2 costs to ensure they are more credible: First, according to best practices, items excluded from estimates should be documented and explained. While IRS documented four of the five items that were excluded from the estimates, it did not provide an explanation for excluding these costs. In addition, IRS did not document that it excluded the business costs associated with moving to daily processing. IRS officials stated that, at the time the estimates were developed, these business costs were not well understood by the estimators and now believe that these costs would be relatively small, representing mostly costs for staff already being paid out of existing budgets. Nevertheless, consistently documenting excluded costs and providing a rationale for excluding them would improve credibility.

Second, IRS’s estimates did not include inflation. According to best practices, adjusting costs for inflation correctly is necessary if the cost estimate is to be credible. Inflation reflects the fact that the cost of an item usually continues to rise over time. If a mistake is made in applying inflation, or if inflation is not included, cost overruns can result.

Third, IRS did not conduct a sensitivity analysis to examine the effects of changing assumptions and ground rules on its estimates. While IRS developed risk-adjusted estimates, it did not perform the formal sensitivity analysis necessary to understand which variables most affect the cost estimate. IRS officials stated they intend to include a sensitivity analysis in the revised cost estimates expected to be available by the completion of our audit. Performing this analysis will not only improve the credibility of the estimate, but it will provide management a better understanding of the relationship of key factors to cost, which is important in evaluating different options as part of subsequent risk and issue mitigation strategies and contingency plans.

IRS’s Risk Management Process Is Generally Consistent with Best Practices

Given the size and significance of the CADE 2 program, it is important that IRS have an effective risk management process. According to best practices, an effective risk management process identifies potential problems before they occur, so that risk-mitigating activities may be planned and invoked as needed across the life of the product and project in order to mitigate adverse impacts on achieving objectives. It includes assigning resources, identifying and analyzing risks, and developing risk mitigation plans and milestones for key mitigation deliverables.

IRS’s risk management process for CADE 2 is generally consistent with best practices. For example, regarding resources assigned to risk management, the CADE 2 PMO’s Director for Program Management and Control is responsible for “executing” the risk management process. Biweekly meetings are held where the Director for Delivery Management and project officials discuss project status and risks and determine whether risks need to be escalated as candidate risks to the program level. Monthly risk and issue meetings are held to discuss the program-level risks. Finally, those risks that require additional oversight are then presented to the Chief Technology Officer for review. Program-level and project-level risks are scored and tracked in a consistent manner using a risk repository tool. Risk mitigation plans and associated milestones are defined for each risk. Officials from the PMO have also told us that the management structure and processes that have been established for the program are intended to help mitigate the risks associated with developing such a large and complex program.

While IRS is generally effectively carrying out the activities associated with the risk management process, the two key documents that support this process are in some cases inconsistent and do not fully reflect actions being taken. For example, the Risk and Issue Management Process document calls for automatically escalating those projects with a high to very high risk exposure score to the next level of management while the Risk and Issue Management Plan calls for escalating risks based primarily upon the judgment of the risk owner and responsible manager. The Risk and Issue Management Process identifies a specific set of actions to mitigate risks depending on the risk exposure score while the Risk and Issue Management Plan provides high-level guidance describing various alternatives that can be taken to mitigate risks. As another example, both documents identify the risk management roles but do not specify when in the process these roles are assigned or what the required qualifications are.

IRS recognizes its risk management procedures could be strengthened, and the Director for Program Management and Control intends to review these procedures by the completion of our audit, to ensure they are consistent and clearly reflect the process in place. This action will help IRS achieve optimal effectiveness in identifying, prioritizing, or managing risks that could affect the cost and schedule of the program.

IRS Has Identified Significant Risks Confronting CADE 2

Using its risk management process, IRS has identified several significant program risks for CADE 2. As of November 2010, six of these were considered “top risks.” These “top risks,” which include TS1 contingency planning and resource contention among IRS priorities are listed in table 6 along with the mitigation strategies that have been defined for them. (For a complete list of program risks, see app. III.)

While IRS has taken many steps to reduce the risks associated with CADE 2, the agency will nevertheless be challenged in meeting its January 2012 schedule for delivering the benefits associated with daily processing and database implementation given the amount of work needed to complete TS1 over the next year. This includes completing logical design and detailed design activities, testing and integration of core daily applications and their development, and testing and population of the database with current CADE and IMF data. Further, as already noted, waiting to finish (1) defining targets for the TS1 benefits and (2) finalizing benefits and related targets for TS2 creates a risk of costly or time-consuming late design changes.

Recognizing the challenges it faces, IRS has taken additional steps to increase the likelihood of meeting the schedule. These include moving certain activities up, performing others concurrently and adding checkpoints to monitor the program’s status. In addition, IRS has recently completed a TS1 contingency strategy. While these actions may increase the likelihood of meeting the schedule, some of them, such as performing activities concurrently, could potentially introduce more risk to CADE 2’s successful development and implementation.

Conclusions

Defining benefits and related quantitative targets where feasible is important to understanding what to expect and how to measure performance against expectations. For TS1, while IRS has identified numerous expected benefits, it has not yet defined quantitative targets for all of them. Given that IRS is finalizing the design and preparing to implement TS1, it is important for the agency to complete defining the targets associated with TS1 benefits. Delaying the setting of targets could result in costly and time-consuming changes to the design and associated requirements. Similarly, addressing TS2 benefits and related targets as the design is being considered could influence design decisions and help identify early on how systems and processes might be affected.

In developing cost estimates for the first two phases of CADE 2, IRS did not disclose all excluded costs, provide a rationale for documented excluded costs, adjust for inflation, or perform a sensitivity analysis to examine the effects of changing assumptions and ground rules. IRS stated it would perform a sensitivity analysis in the revised estimates it expects to release by the completion of our audit.

To its credit, IRS has established a risk management process that is generally consistent with best practices and has identified mitigation strategies to address each identified risk. Several of the risks identified are significant and will require continued attention from IRS.

Recommendations for Executive Action

We recommend that the Commissioner of Internal Revenue direct the appropriate officials to take the following five actions: to provide a better basis for measuring CADE 2 performance in achieving TS1 benefits, set the remaining quantitative targets where feasible as soon as possible in 2011; in conjunction with developing the approach for TS2 expected in May 2011, finalize the phase’s expected benefits, set quantitative targets where feasible, and identify how related systems and business processes might be affected; and to ensure the revised estimates for CADE 2 are credible, include inflation when calculating costs, include the costs IRS explicitly excluded or provide a rationale for excluding them, and include any business costs associated with moving to daily processing or document that these costs were excluded and provide a rationale for excluding them.

Agency Comments and Our Evaluation

IRS’s Commissioner provided written comments on a draft of this report (reprinted in app. IV.) In its comments, IRS stated that it appreciated that the report recognized the progress the agency had made and that it is following best practices on important disciplines like risk management and cost estimation. IRS also agreed with our recommendations and stated it would provide a detailed corrective action plan addressing each one along with its response to our final report. IRS also provided revised cost estimates for CADE 2 that address our finding related to the use of a sensitivity analysis.

We are sending copies of this report to the appropriate congressional committees, and to the Chairmen and Ranking Members of other Senate and House committees and subcommittees that have appropriation, authorization, and oversight responsibilities for IRS. We will also send copies to the Commissioner of Internal Revenue, the Secretary of the Treasury, the Chairman of the IRS Oversight Board, and the Director of the Office of Management and Budget. The report also is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions or wish to discuss the material in this report further, please contact us at (202) 512-9286 or pownerd@gao.gov or at (202) 512-9110 or whitej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

As agreed, our objectives were to 1. determine whether the Internal Revenue Service (IRS) has identified the expected benefits of Customer Account Data Engine (CADE) 2 and set targets for measuring success, 2. examine the estimated costs of CADE 2 and assess IRS’s process for 3. assess IRS’s process for managing the risks associated with CADE 2 and describe the risks IRS has identified using this process.

For our first objective, we reviewed relevant documents, including the CADE 2 Program Business Case and Benefits Management Plan, and interviewed officials from IRS offices: Wage and Investment and Modernization and Information Technology Services (MITS). We also used criteria from our prior work to identify attributes of successful performance measures. We reviewed documents including the Program Roadmap, Business Case and Benefits Management Plan, and Benefits Roadmap, and the Office of Management and Budget Exhibit 300.

For our second objective, we summarized the estimated costs reported in the July 2009 CADE 2 Estimate Summary/ Basis of Estimate and totaled costs by year for each of the major categories for which estimates were developed. We reviewed several documents, including the July 2009 CADE 2 Estimate Summary/ Basis of Estimate and the Estimation Breakdown Summary reports and capabilities and solution concept documents (e.g., the Integrated Master File Enhancements: Capabilities Definition and Solution Concept document and the Financial Settlement: Capabilities Definition and Solution Concept document), which support the Basis of Estimate. We also reviewed IRS’s Estimators’ Reference Guide and aspects of the SEER-SEM methodology that was used to develop the estimates. We also interviewed staff from the Estimation Program Office, as well as members of the Program Management and Control function of the CADE 2 Program Management Office (PMO) responsible for tracking the program’s cost. We compared what we learned with the 12 steps for developing well-documented, comprehensive, accurate, and credible cost estimates identified in the GAO Cost Estimating and Assessment Guide.

For our third objective, we reviewed documented procedures supporting the risk management process, including the Risk and Issue Management Process, the Risk and Issue Management Plan, the user guide for the Item Tracking Reporting and Control system that is used to track risks and the MITS Risk Identification Procedure. We also reviewed artifacts of the risk management process, including program risk logs and detailed risk reports. We participated in meetings in which risks were discussed and reviewed agendas and meeting minutes for others. We also interviewed several officials, including the CADE 2 Program Director, the Director of the CADE 2 PMO Program Management and Control office, and the Director of the CADE 2 PMO Delivery Management office. Through document reviews and interviews with appropriate officials, we also examined the following areas in greater depth: (1) schedule, (2) contractor oversight, (3) requirements management, (4) prototypes, and (5) human capital. We selected these areas because, based on our experience, they represent areas of risk or have been problematic for IRS in the past. In addition, we examined IRS’s prototyping efforts to understand how the agency was using them to mitigate risks. We evaluated what we learned about IRS’s risk management process against relevant criteria for Software Engineering Institute’s Capability Maturity Model Integration (CMMI) to determine the effectiveness of the process.

To obtain background information on the program, we reviewed a host of documents developed by the CADE 2 PMO, including the CADE 2 Program Charter, Solution Architecture, Program Roadmap, Program Management Plan, the charters of the two CADE 2 transition state 1 projects (Daily Processing and Database Implementation), as well as governance team meeting minutes. We also observed CADE 2 meetings, conducted interviews with and received briefings from CADE 2 officials, including the IRS Chief Technology Officer, the Deputy Chief Information Officer for Strategy and Modernization, the Associate Chief Information Officer who serves as executive-in-charge of the CADE 2 program, and the Chief Architect of the CADE 2 program.

We conducted this performance audit from January 2010 to March 2011 primarily at the New Carrollton Federal Building in New Carrollton, Maryland, where the CADE 2 staff are located, and Atlanta, Georgia, where the Wage and Investment division staff are located, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Additional Information about CADE 2 Benefits

Table 7 shows a list of benefits the IRS has defined to date for transition state 1 (TS1) and beyond, a related quantitative target for a benefit if applicable, and the transition state in which IRS expects to realize the benefit. According to IRS, defining which benefits will be delivered in transition state 2 (TS2) is contingent upon a number of decisions yet to be made and actual benefits may differ from those currently expected depending on the outcome of these decisions.

Appendix III: CADE 2 Program Risks

The following (see table 8) are the IRS’s reported CADE 2 program risks and their planned mitigation strategies as of November 2010.

Appendix IV: Comments from the Internal Revenue Service

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the individuals named above, Sabine R. Paul, Assistant Director; Joanna Stamatiades, Assistant Director; Rebecca Eyler; Mary D. Fike; Nancy Glover; Sairah R. Ijaz; Robert Kershaw; Paul B. Middleton; John Ockay; and Jennifer Wong made key contributions to this report.