Taxpayer Account Strategy: IRS Should Finish Defining Benefits and Improve Cost Estimates
Published: 2011-03-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In August 2008, the Internal Revenue Service (IRS) began defining a new strategy for modernizing the way it manages individual taxpayer accounts. The strategy, known as Customer Account Data Engine (CADE) 2, is expected to provide service, compliance, and other benefits to IRS and to taxpayers beginning in 2012. IRS expects to implement CADE 2 in three phases. The first phase is expected to be delivered in 2012, the second in 2014, and the third at a later yet to be determined date. GAO was asked to (1) determine whether IRS has identified the expected benefits of CADE 2 and set targets for measuring success, (2) examine the estimated costs and assess IRS's process for developing them, and (3) assess IRS's process for managing the risks associated with CADE 2 and describe the risks IRS has identified using this process. To do so, GAO reviewed relevant documentation, attended program review meetings, and interviewed IRS officials.
section_title: What GAO Found
paragraphs: IRS has identified 20 service, compliance, and other benefits for the first phase of CADE 2, including increasing the percentage of refunds processed daily and reducing the number of erroneous notices due to better account information, and has set quantitative targets for most of these benefits. GAO has previously reported that quantitative targets can be useful for tracking program performance. While it may not always be possible to quantify targets, doing so helps to objectively measure the extent to which expected benefits have been realized. However, IRS has not yet finalized expected benefits for the second phase or set related quantitative targets, because, according to officials, these are contingent upon yet to be made design and funding decisions. Nevertheless, addressing the second phase's benefits and related targets as the design is being considered could influence design decisions and help identify early on how systems and processes might be affected. IRS reported preliminary life cycle cost estimates for the first two phases of the CADE 2 program of about $1.3 billion through 2024. This includes about $377 million for development and $922 million for operations and maintenance. IRS's process for developing the preliminary estimates was generally consistent with best practices. However, the agency did not follow three practices intended to improve the credibility of cost estimates. Specifically, IRS did not (1) consistently document excluded costs or provide a rationale for excluding them; (2) use inflation in calculating costs; and (3) perform an analysis to examine the effects of changing ground rules and assumptions. While IRS stated it would perform the analysis of changing ground rules and assumptions in revised estimates to be available by the completion of our audit, until the agency implements all these practices its estimates may not be credible. IRS's process for managing the risks associated with CADE 2 is generally consistent with best practices. Through its process, IRS identified significant risks facing CADE 2, including that filing season and other top information technology investment priorities may result in contention for key resources, the delivery of the first phase of CADE 2 may be delayed if deficiencies identified in requirements are not corrected in a timely manner, and the risk that technical challenges and other risks to implementing the database identified as a result of prototyping efforts may not be addressed. To its credit, IRS has developed mitigation strategies for each identified risk. While IRS is working to ensure CADE 2 is successfully managed, the schedule for delivering the initial phase is nevertheless ambitious. IRS officials have acknowledged this and are taking actions to increase their chances of meeting it, including moving certain activities up, performing others concurrently, and adding checkpoints to monitor the program's status. While these actions may increase the likelihood of meeting the schedule, some of them, such as performing activities concurrently, could potentially introduce more risk to CADE 2's successful development and implementation. GAO's recommendations include (1) identifying all of the second phase benefits, setting the related targets, and identifying how systems and business processes might be affected; and (2) improving the credibility of revised cost estimates by including all costs or providing a rationale for excluded costs, and adjusting costs for inflation. In its comments on a draft of this report, IRS agreed with GAO's recommendations.

section_title: Letter
paragraphs: In administering federal tax laws, the Internal Revenue Service (IRS)  collected approximately $2.35 trillion in 2010 and is projected to collect  over $3 trillion beginning in 2013. To do so, IRS depends heavily on  information technology (IT) systems, including legacy tax processing  systems that were developed in the 1960s that are complex and difficult to  maintain.
In 1999, IRS initiated the Customer Account Data Engine (CADE) project  to replace legacy systems for storing, managing, and accessing taxpayer  accounts. Starting in 2005, CADE processed and recorded tax return and  tax account information for millions of individual taxpayers with simple  returns. However, IRS found that the development of the system was more  complex and taking longer than initially anticipated and decided to  undertake a broad review of its IT systems modernization efforts. The  results of the review led IRS to adopt a new strategy for managing  individual taxpayer accounts that would provide benefits to IRS and the  taxpayers sooner than CADE. The new strategy is referred to as CADE 2.
To better understand the implications of moving to the new strategy, you  asked us to (1) determine whether IRS has identified the expected benefits  of CADE 2 and set targets for measuring success, (2) examine the  estimated costs of the program and assess IRS’s process for developing  them, and (3) assess IRS’s process for managing the risks associated with  CADE 2 and describe the risks IRS has identified using this process.
To describe the expected benefits, we reviewed relevant documents,  including the CADE 2 Program Business Case and Benefits Management  Plan, and interviewed IRS officials from the offices of Wage and  Investment (W&I) and Modernization and Information Technology  Services (MITS). We also used program performance criteria established  in our prior reports. To understand the estimated costs and assess IRS’s  process for developing them, we reviewed agency cost documentation and  compared it to the best practices identified in the GAO Cost Estimating  and Assessment Guide and interviewed officials from IRS’s Estimation  Program Office. To determine CADE 2 key risks and IRS’s management of  them, we interviewed officials from the CADE 2 Program Management  Office (PMO) and reviewed documentation associated with the risk  management process and the identification of key risks. We evaluated  IRS’s risk management process against relevant criteria in the Software  Engineering Institute’s Capability Maturity Model Integration (CMMI) best  practice guidance.
We conducted this performance audit from January 2010 to March 2011, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Appendix I contains further details on our  objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: IRS’s Business Systems Modernization program, which began in 1999, is a  multibillion-dollar, high-risk, highly complex effort that involves the  development and delivery of a number of modernized tax administration  and internal management systems, as well as core infrastructure projects,  which are intended to replace the agency’s aging business and tax  processing systems and provide improved and expanded service to  taxpayers and internal business efficiencies for IRS. CADE, one of the core  systems of the Business Systems Modernization program, was intended to  provide a modernized system of taxpayer accounts, with the ultimate goal  of eventually replacing the Individual Master File (IMF), a 1960s legacy tax  master file that maintains all taxpayer records for individual taxpayers.
The IMF is the authoritative data source for individual tax account data.  Tax data and related information pertaining to individual taxpayers are  posted to the IMF, and the IMF is updated annually to incorporate new tax  law procedures and changes. The IMF is a critical component of IRS’s  ability to process tax returns, as all the other IRS information system  applications use data from this source. According to IRS, the IMF is a  complex computer system that maintains all individual taxpayer records  but has inherent limitations that significantly constrain IRS’s ability to  achieve its mission. The delays in weekly updates of taxpayer accounts,  the lack of synchronization of taxpayer data across dozens of systems, the  complexity of the data, the large volumes of data, and the ongoing need to  embed new tax laws and business rules contribute to IRS systems storing,  sharing, and processing data that is often incomplete, untimely, or  inaccurate.
With CADE, IRS started establishing a database and related functionality  for posting, settlement, maintenance, refund processing, and issue  detection for taxpayer account and return data for relatively simple tax  returns for individual taxpayers. Since 2005, CADE has processed and  recorded tax return and tax account information for increasing numbers  of individual taxpayers. IRS reported that, in 2010, CADE    processed about 41 million relatively simple returns (about 30 percent of  the total individual income tax returns received); and issued about 36 million refunds that totaled in excess of $66 billion  (approximately 14 percent of total refunds), and processed over 7 million  taxpayer payments amounting to approximately $90 billion—with zero  defects in balancing.
In addition, IRS reported that it issues refunds with CADE approximately 5  days faster than refunds processed by the IMF.
subsections: section_title: Challenges and Risks Facing CADE That Led to CADE 2
paragraphs: Although IRS made progress in processing individual taxpayer returns  with CADE, IRS found that each successive release of the system was far  more difficult as more complex accounts were to be supported, raising  concerns about the effort required to address these increasing  complexities. Also, since the CADE system operates concurrently with  IMF processing, work must be done across the two environments,  increasing the complexity of filing season operations and resulting in a  greater risk for errors in processing taxpayer account information. IRS  was concerned that, without a significant change in approach, CADE  would not be completed until at least 2020.
These challenges led the IRS Commissioner to form a team of senior IRS  technologists and external advisers to review CADE and IRS’s  modernization strategy for individual taxpayer accounts. The review  identified six risks with the CADE strategy. As a result of the review, in  August 2008, IRS began defining a new strategy to address the challenges  confronting CADE and deliver improved individual tax processing  capabilities sooner. Specifically, the new strategy called for accelerating  completion of a modernized database and converting to a single  processing system sooner than the current approach would allow. The  new strategy is referred to as CADE 2. Table 1 describes the CADE risks  reported by IRS and how CADE 2 intends to address them.
As noted in the table, IRS used prototypes to test and prove complex  technical and performance concepts early and provide guidance and  information needed for detailed design. These prototypes were completed  in 2010, although some remaining tasks in one of them—the Database  Performance Test Prototype—will be completed later.
section_title: CADE 2 Is Expected to Deliver Functionality in Three Phases (Transition States)
paragraphs: CADE 2 is expected to deliver its functionality incrementally through three  phases known as transition states. Table 2 shows the target completion  dates and key characteristics for transition state 1 (TS1), transition state 2  (TS2), and the target state.
section_title: IRS Has Established Two Projects to Achieve TS1 Goals
paragraphs: TS1 consists of the following two projects:  Daily Processing: According to IRS, this project is to enable IRS to  process and post all individual taxpayer returns filed and other  transactions by updating and settling individual taxpayer accounts in 24-48  hours with current, complete, and authoritative data, and provide  employees with timely access. More specifically, through the Daily  Processing project, IRS is expected to    modify the IMF to run daily, i.e., modify current transaction processing to  occur daily versus weekly;   make changes to the Integrated Data Retrieval System (IDRS) and the  notice generation process to support the daily vs. weekly processing cycle;  and    prepare daily “loads” to update the CADE 2 database for IDRS and other  downstream systems as they are able to support daily processing.
Database Implementation: According to IRS, this project is to establish  the CADE 2 database, a relational database that will house data on  individual taxpayers and their accounts; develop a capability to transfer  data from IMF to the database; and provide for the transfer of data from  the database to downstream IRS financial, customer service and  compliance systems. Specifically, the project is expected to enable IRS to    perform a one-time initialization of the database when TS1 begins (this  will involve loading all taxpayer account data from IMF and the CADE  database into the CADE 2 database and then validating it);   perform daily updates to the database after TS1 begins by extracting,  transforming, and loading daily processing changes to IMF data; and    utilize the database to provide individual taxpayer account information to  selected downstream systems to assist IRS service and compliance  personnel. For example, the CADE 2 database is intended to provide daily  updates to IDRS, as well as weekly updates to the Integrated Production  Model (IPM)—a downstream data repository designed to support IRS  compliance functions that house IMF and Business Master File data,  information returns, and other data. IPM is expected to include current  individual taxpayer data from the CADE 2 database.
Regarding the status of these two projects, IRS completed efforts to define  the TS1 projects’ logical design in December 2010 and began the detailed  design phase, which includes activities such as documenting the physical  design of the solution. IRS expects to complete this phase by April 2011.
IRS has also defined overall objectives for TS2 and the target state, but  detailed planning of TS2 is in the early stages, with the formation of a  planning team that expects to define a high-level approach by early May  2011.
section_title: Treasury Inspector General for Tax Administration and GAO Have Recently Reported on CADE 2
paragraphs: In December 2009, the Treasury Inspector General for Tax Administration  (TIGTA) identified several challenges that IRS needed to address to  effectively manage identified CADE 2 risks, including, among other things,  implementing a governance structure for the PMO to provide oversight  and direction for the implementation of CADE 2 and developing  contingency plans in the event that CADE 2 cannot be fully implemented.  IRS officials completed a contingency strategy in November 2010.
In May 2010, we reported that, while much had been done to define CADE  2’s transition states, IRS had not identified time frames for completing key  planning activities for TS2. Consequently, we recommended IRS establish  these time frames. In response, IRS stated that it would develop a plan for  launching TS2 activities that will outline the approach and a high-level  plan for the transition state. IRS also noted that business requirements  would be completed in parallel with the development of the plan. In  December 2010, IRS initiated TS2 planning activities and set a target of  May 2011 for agreeing on the scope and plan for the transition state.
In November 2010, TIGTA reported on the prototypes undertaken by IRS  to gain confidence in the CADE 2 solution. TIGTA found that the five  prototype teams that were established generally managed their objectives  effectively, and that the teams also identified risks that faced the  successful execution of the prototype plans and took steps to overcome  the barriers. TIGTA recommended that IRS take several actions to  reemphasize compliance with the elements of the CADE 2 Prototype  Process. In its response, IRS agreed with TIGTA’s recommendations and  described steps to address them.
section_title: IRS Has Established a New Management Structure for the CADE 2 Program
paragraphs: While the MITS organization has primary responsibility for developing,  managing, and delivering the CADE 2 program, IRS has established a new  governance approach for the program, including the CADE 2 PMO that is  to manage the program and the relationship with the program’s  stakeholders, define the solution specifications to meet the requirements  and the delivery of the program scope, and specify the roles and  responsibilities of all parties. The new governance approach is intended to  foster rapid decision-making and proactive risk management and issue  resolution, as well as ensure that the appropriate stakeholders are  involved, engaged, and collaborating as a unified team in making CADE 2  decisions.
The CADE 2 PMO is headed by an Associate Chief Information Officer and  made up of several groups, each responsible for a different aspect of the  program. These groups include    Delivery Management. Responsible for overseeing and coordinating  activities across the projects that are implementing the solution for each  of the transition states.
Program Management and Control. Responsible for executing the  program management processes, CADE 2 solution planning, and  governance and control activities.
Chief Architect. Responsible for setting the overall technical direction in  alignment with IRS’s business and technical architecture, as defined in the  enterprise architecture.
Chief Engineer. Responsible for reviewing detailed design and interfaces  to ensure the CADE 2 solution will work, is secure, and integrates with  other systems and infrastructure.
Business Operations. Responsible for providing the administrative  support for all branches of the PMO.
These groups are to work with (1) delivery partners who are other MITS  organizations (e.g., Applications Development and Enterprise Operations)  that work to deliver the program scope, (2) business partners (e.g., W&I  and the Chief Financial Officer) who are organizations who either use the  system or define the business requirements and, as such, play a key role in  overseeing CADE 2, and (3) other stakeholder groups who support,  influence, or oversee the program.
To oversee and guide the PMO, IRS also established several executive- level boards and advisory councils composed of business and IT officials.  The executive-level boards include the following:    The CADE 2 Executive Steering Committee, which consists of senior  executives from MITS, W&I, and Department of the Treasury, and serves  as an oversight group that ensures the program stays aligned with the IRS  strategic plan and approves decisions with significant organizational or  external impact; and   The CADE 2 Governance Board, which consists of Associate Chief  Information Officers from CADE 2 and Applications Development and the  business modernization executive from W&I, and ensures that objectives  are met; decisions and issues are resolved in a timely manner; risks are  managed appropriately; and the expenditure of resources allocated is  fiscally sound. The CADE 2 Governance Board also approves program risk  response plans, milestone exits, and resolves escalated issues.
The advisory councils include the    Executive Oversight Team, which consists of executives from the CADE 2  PMO, delivery partners, and senior-level business executives, and serves  as the oversight for the day-to-day execution of the CADE 2 program;    Associate Chief Information Officer Advisory Council representing leaders  of each of the MITS technical units;   Program Leadership Advisory Council, a broad group of executives from  W&I, Agency Wide Shared Services, and across MITS that consult on key  issues, risks, analyses, and recommendations on an as needed basis; and   Architecture/Engineering Review Council recently created to provide  architectural and engineering leadership across the program to ensure  design and development efforts are complete and adhere to solution  architecture and engineering best practices.
In addition, while the projects chartered under CADE 2 are expected to  follow the IRS enterprise life cycle methodology that other projects are  required to follow, IRS also defined a program management framework for  CADE 2 that defines the phases, activities, deliverables, milestones, and  reviews necessary to manage both the program and each of its component  projects in a coordinated and integrated manner. The PMO and supporting  boards are to play key roles in reviewing deliverables and approving  milestone exits for the program and assuring that the projects are properly  aligned and integrated with the program.
Finally, IRS has also defined and started implementing several processes  to support its management of the CADE 2 program. They include  processes for managing risks, the integrated master schedule, the  prototype process, and other activities. While several of these processes  were already established for projects, IRS enhanced them for CADE 2 to  take into account the complexities associated with integrating projects  into a program.
section_title: IRS Has Identified Benefits and Set Most Related Targets for the First Phase but Has Yet to Do So for the Second Phase
paragraphs: For TS1, IRS has identified 20 benefits. These benefits span three  categories: service; compliance; and other benefits, including reduced  system costs and improved security benefits.
Our past work at IRS established that quantitative targets can be useful for  tracking program performance. While it may not always be possible to  quantify targets, doing so helps to objectively measure the extent to which  expected benefits have been realized.
As shown in table 3, IRS has set quantitative targets for 15 of the 20  benefits for TS1. (Table 7 in app. II describes all the benefits IRS has  identified and the related quantitative targets where they exist.)
An example of a benefit with a quantitative target is the percentage of  refunds IRS will process on a daily basis. IRS expects CADE 2 to increase  this from 30 to 80 percent, resulting in faster refunds to many taxpayers.  Another example is the security benefit of reducing the number of IRS  databases that contain extracts of data from CADE 2. IRS expects this to  be reduced by 20 percent from the 2011 baseline.
IRS officials have stated that they have not set quantitative targets for five  TS1 benefits for two reasons. First, officials told us that two benefits  related to improved security and updated account information cannot be  quantified and therefore do not have numeric targets. (See app. II for more  details.) Second, IRS officials stated that competing priorities have  impeded their ability to define the remaining targets. While they stated  they plan to define these targets, they have not committed to specific time  frames for doing so. Until IRS has set quantitative targets where feasible in  TS1, it and other stakeholders including Congress will not have complete  information on the specific benefits that can be expected from funds spent  on this phase. In addition, it will be more difficult to objectively measure  whether expected benefits have been delivered.
In contrast with TS1, IRS has not finalized the benefits expected in TS2 or  defined related quantitative targets. Specifically, IRS has identified a list of  22 service, compliance, and other benefits including addressing financial  material weaknesses it expects to achieve beyond TS1. These benefits  are summarized in table 4 and described in greater detail in table 7 in  appendix II. However, IRS has not yet determined whether these benefits  are to be delivered in TS2 or the target state. According to IRS, committing  to what can be delivered in TS2 is contingent upon a number of decisions  yet to be made. Specifically, officials stated that IRS must first finalize its  approach for TS2 (as noted earlier, the agency expects to do this by May  2011). Officials also stated that funding decisions for the upcoming fiscal  year could impact IRS’s plans for TS2.
In addition, IRS has not finished defining the full range of possible benefits  it expects beyond TS1. For example, one possible compliance benefit that  IRS is exploring is requiring providers of information returns to send those  returns to IRS and taxpayers at the same time, which would allow IRS to  use this data earlier to conduct some compliance checks. Conducting  earlier analyses of tax return information using CADE 2 and IPM could  allow IRS to stop an erroneous refund before it goes out. IRS currently  waits until well after the filing season to conduct some compliance checks,  such as computerized matching of information returns and tax returns and  examinations of taxpayers’ books and records. Such compliance checks  can lead to costly collection efforts from noncompliant taxpayers.
Finally, as summarized in table 4 and shown in greater detail in appendix  II, IRS has not yet set quantitative targets for any of the benefits defined or  committed to any time frames for doing so. As noted for TS1, until IRS has  set quantitative targets where feasible in TS2, it and other stakeholders  will not have complete information on the specific benefits than can be  expected from funds spent, and it will be difficult to objectively measure  whether expected benefits have been delivered.
We acknowledge that thinking through the expected TS2 benefits  (including those from the table in app. II and other potential benefits) and  related targets as the approach and design are being considered may result  in having to make adjustments later. However, the benefits and targets  generated at this stage can influence design decisions and allow IRS to  identify early how associated systems and business processes might be  affected.
section_title: IRS’s Estimates the First Two Phases of CADE 2 Will Total About $1.3 Billion; Process for Developing Estimates Was Largely Consistent with Best Practices
paragraphs: In July 2009, IRS reported preliminary life cycle cost estimates for TS1 and  TS2 of about $1.3 billion through 2024, including about $377 million for  development and $922 million for operations and maintenance. The  estimates include costs for eight categories of work needed to achieve the  goals defined for TS1 and TS2. The majority of the costs are for the  Operational Framework category ($663 million or 51 percent of total  costs), which is to provide for the infrastructure on which to develop, test,  and deploy the new and enhanced applications under the CADE 2  program. The CADE 2 preliminary cost estimates are summarized in   table 5.
The estimated annual costs from 2009 through 2024 are shown in figure 1.  At the time they were developed, IRS estimated CADE 2 development  costs to end in 2015 and total costs to peak during 2011 at about $162  million. Estimated costs between 2015 and 2024 are all recurring  operations and maintenance costs.
IRS’s estimates did not include expected costs for the following items:     operations and maintenance of the IMF system to include enhancements  needed to achieve the CADE 2 target state,   prototype development and assessments,  technical training (estimated at $1 million to $3 million),    work required to reach the final target state, and    process reengineering costs/business costs associated with implementing  the current weekly batch processing of accounts to the daily processing  required at the completion of TS1.
subsections: section_title: With Some Exceptions, Process for Developing Preliminary Cost Estimates Was Consistent with Best Practices
paragraphs: The process IRS used to develop the CADE 2 preliminary cost estimates  was generally consistent with the best practices outlined in the GAO cost  estimating guide. Specifically, IRS followed a well-defined process for  developing a comprehensive, documented, accurate, and mostly credible  life cycle cost estimate for the first two phases of CADE 2. For example,  the estimation process was performed in accordance with a well-defined  plan by an independent Estimation Program Office. Parametric estimation  models were used that contained catalogs based on historical data, along  with cost estimating relationships, designed to produce estimates  analogous to similar programs. In addition, IRS used simulation techniques  to incorporate risks and determine confidence bounds on the estimates.
Further, IRS plans to update its estimates, consistent with the best  practice to update estimates to reflect changes in technical or program  assumptions. Specifically, since the estimates were prepared, IRS has  developed a better understanding of the work required to meet CADE 2  objectives for TS1 and the current work breakdown structure (i.e.,  definition of work to be performed) no longer matches the one used to  develop the preliminary estimates. IRS officials told us that, consistent  with IRS’s Estimation Program Office Estimator’s Guide, they plan to  revise their estimates after detailed design information for the database  implementation and daily processing projects is completed and expect  them to be available by the completion of our audit. CADE 2 PMO officials  stated they did not expect the revised estimates to differ significantly from  the current estimate. It is important to note that, even when all TS1 cost  elements have been updated to accurately reflect the current  understanding of the work to be performed, the total cost of ownership  through TS2 could still significantly differ from the new estimate, given  that, as previously noted, the approach for TS2 has yet to be defined.
We identified the following three practices IRS did not follow that could  strengthen its methods for estimating CADE 2 costs to ensure they are  more credible:  First, according to best practices, items excluded from estimates should  be documented and explained. While IRS documented four of the five  items that were excluded from the estimates, it did not provide an  explanation for excluding these costs. In addition, IRS did not document  that it excluded the business costs associated with moving to daily  processing. IRS officials stated that, at the time the estimates were  developed, these business costs were not well understood by the  estimators and now believe that these costs would be relatively small,  representing mostly costs for staff already being paid out of existing  budgets. Nevertheless, consistently documenting excluded costs and  providing a rationale for excluding them would improve credibility.
Second, IRS’s estimates did not include inflation. According to best  practices, adjusting costs for inflation correctly is necessary if the cost  estimate is to be credible. Inflation reflects the fact that the cost of an item  usually continues to rise over time. If a mistake is made in applying  inflation, or if inflation is not included, cost overruns can result.
Third, IRS did not conduct a sensitivity analysis to examine the effects of  changing assumptions and ground rules on its estimates. While IRS  developed risk-adjusted estimates, it did not perform the formal sensitivity  analysis necessary to understand which variables most affect the cost  estimate. IRS officials stated they intend to include a sensitivity analysis in  the revised cost estimates expected to be available by the completion of  our audit. Performing this analysis will not only improve the credibility of  the estimate, but it will provide management a better understanding of the  relationship of key factors to cost, which is important in evaluating  different options as part of subsequent risk and issue mitigation strategies  and contingency plans.
section_title: IRS’s Risk Management Process Is Generally Consistent with Best Practices
paragraphs: Given the size and significance of the CADE 2 program, it is important that  IRS have an effective risk management process. According to best  practices, an effective risk management process identifies potential  problems before they occur, so that risk-mitigating activities may be  planned and invoked as needed across the life of the product and project  in order to mitigate adverse impacts on achieving objectives. It includes  assigning resources, identifying and analyzing risks, and developing risk  mitigation plans and milestones for key mitigation deliverables.
IRS’s risk management process for CADE 2 is generally consistent with  best practices. For example, regarding resources assigned to risk  management, the CADE 2 PMO’s Director for Program Management and  Control is responsible for “executing” the risk management process.  Biweekly meetings are held where the Director for Delivery Management  and project officials discuss project status and risks and determine  whether risks need to be escalated as candidate risks to the program level.  Monthly risk and issue meetings are held to discuss the program-level  risks. Finally, those risks that require additional oversight are then  presented to the Chief Technology Officer for review. Program-level and  project-level risks are scored and tracked in a consistent manner using a  risk repository tool. Risk mitigation plans and associated milestones are  defined for each risk. Officials from the PMO have also told us that the  management structure and processes that have been established for the  program are intended to help mitigate the risks associated with developing  such a large and complex program.
While IRS is generally effectively carrying out the activities associated  with the risk management process, the two key documents that support  this process are in some cases inconsistent and do not fully reflect actions  being taken. For example, the Risk and Issue Management Process  document calls for automatically escalating those projects with a high to  very high risk exposure score to the next level of management while the  Risk and Issue Management Plan calls for escalating risks based  primarily upon the judgment of the risk owner and responsible manager.  The Risk and Issue Management Process identifies a specific set of  actions to mitigate risks depending on the risk exposure score while the  Risk and Issue Management Plan provides high-level guidance describing  various alternatives that can be taken to mitigate risks. As another  example, both documents identify the risk management roles but do not  specify when in the process these roles are assigned or what the required  qualifications are.
IRS recognizes its risk management procedures could be strengthened,  and the Director for Program Management and Control intends to review  these procedures by the completion of our audit, to ensure they are  consistent and clearly reflect the process in place. This action will help  IRS achieve optimal effectiveness in identifying, prioritizing, or managing  risks that could affect the cost and schedule of the program.
subsections: section_title: IRS Has Identified Significant Risks Confronting CADE 2
paragraphs: Using its risk management process, IRS has identified several significant  program risks for CADE 2. As of November 2010, six of these were  considered “top risks.” These “top risks,” which include TS1 contingency  planning and resource contention among IRS priorities are listed in table 6  along with the mitigation strategies that have been defined for them. (For  a complete list of program risks, see app. III.)
While IRS has taken many steps to reduce the risks associated with CADE  2, the agency will nevertheless be challenged in meeting its January 2012  schedule for delivering the benefits associated with daily processing and  database implementation given the amount of work needed to complete  TS1 over the next year. This includes completing logical design and  detailed design activities, testing and integration of core daily applications  and their development, and testing and population of the database with  current CADE and IMF data. Further, as already noted, waiting to finish  (1) defining targets for the TS1 benefits and (2) finalizing benefits and  related targets for TS2 creates a risk of costly or time-consuming late  design changes.
Recognizing the challenges it faces, IRS has taken additional steps to  increase the likelihood of meeting the schedule. These include moving  certain activities up, performing others concurrently and adding  checkpoints to monitor the program’s status. In addition, IRS has recently  completed a TS1 contingency strategy. While these actions may increase  the likelihood of meeting the schedule, some of them, such as performing  activities concurrently, could potentially introduce more risk to CADE 2’s  successful development and implementation.
section_title: Conclusions
paragraphs: Defining benefits and related quantitative targets where feasible is  important to understanding what to expect and how to measure  performance against expectations. For TS1, while IRS has identified  numerous expected benefits, it has not yet defined quantitative targets for  all of them. Given that IRS is finalizing the design and preparing to  implement TS1, it is important for the agency to complete defining the  targets associated with TS1 benefits. Delaying the setting of targets could  result in costly and time-consuming changes to the design and associated  requirements. Similarly, addressing TS2 benefits and related targets as the  design is being considered could influence design decisions and help  identify early on how systems and processes might be affected.
In developing cost estimates for the first two phases of CADE 2, IRS did  not disclose all excluded costs, provide a rationale for documented  excluded costs, adjust for inflation, or perform a sensitivity analysis to  examine the effects of changing assumptions and ground rules. IRS stated  it would perform a sensitivity analysis in the revised estimates it expects  to release by the completion of our audit.
To its credit, IRS has established a risk management process that is  generally consistent with best practices and has identified mitigation  strategies to address each identified risk. Several of the risks identified are  significant and will require continued attention from IRS.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Commissioner of Internal Revenue direct the  appropriate officials to take the following five actions:  to provide a better basis for measuring CADE 2 performance in achieving  TS1 benefits, set the remaining quantitative targets where feasible as soon  as possible in 2011; in conjunction with developing the approach for TS2 expected in May  2011, finalize the phase’s expected benefits, set quantitative targets where  feasible, and identify how related systems and business processes might  be affected; and to ensure the revised estimates for CADE 2 are credible, include inflation when calculating costs, include the costs IRS explicitly excluded or provide a rationale for  excluding them, and include any business costs associated with moving to daily processing  or document that these costs were excluded and provide a rationale  for excluding them.
section_title: Agency Comments and Our Evaluation
paragraphs: IRS’s Commissioner provided written comments on a draft of this report  (reprinted in app. IV.) In its comments, IRS stated that it appreciated that  the report recognized the progress the agency had made and that it is  following best practices on important disciplines like risk management  and cost estimation. IRS also agreed with our recommendations and stated  it would provide a detailed corrective action plan addressing each one  along with its response to our final report. IRS also provided revised cost  estimates for CADE 2 that address our finding related to the use of a  sensitivity analysis.
We are sending copies of this report to the appropriate congressional  committees, and to the Chairmen and Ranking Members of other Senate  and House committees and subcommittees that have appropriation,  authorization, and oversight responsibilities for IRS. We will also send  copies to the Commissioner of Internal Revenue, the Secretary of the  Treasury, the Chairman of the IRS Oversight Board, and the Director of the  Office of Management and Budget. The report also is available at no  charge on the GAO Web site at http://www.gao.gov.
If you or your staff members have any questions or wish to discuss the  material in this report further, please contact us at (202) 512-9286 or  pownerd@gao.gov or at (202) 512-9110 or whitej@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: As agreed, our objectives were to  1.  determine whether the Internal Revenue Service (IRS) has identified  the expected benefits of Customer Account Data Engine (CADE) 2 and  set targets for measuring success, 2.  examine the estimated costs of CADE 2 and assess IRS’s process for  3.  assess IRS’s process for managing the risks associated with CADE 2  and describe the risks IRS has identified using this process.
For our first objective, we reviewed relevant documents, including the  CADE 2 Program Business Case and Benefits Management Plan, and  interviewed officials from IRS offices: Wage and Investment and  Modernization and Information Technology Services (MITS). We also used  criteria from our prior work to identify attributes of successful  performance measures. We reviewed documents including the Program  Roadmap, Business Case and Benefits Management Plan, and Benefits  Roadmap, and the Office of Management and Budget Exhibit 300.
For our second objective, we summarized the estimated costs reported in  the July 2009 CADE 2 Estimate Summary/ Basis of Estimate and totaled  costs by year for each of the major categories for which estimates were  developed. We reviewed several documents, including the July 2009 CADE  2 Estimate Summary/ Basis of Estimate and the Estimation Breakdown  Summary reports and capabilities and solution concept documents (e.g.,  the Integrated Master File Enhancements: Capabilities Definition and  Solution Concept document and the Financial Settlement: Capabilities  Definition and Solution Concept document), which support the Basis of  Estimate. We also reviewed IRS’s Estimators’ Reference Guide and  aspects of the SEER-SEM methodology that was used to develop the  estimates. We also interviewed staff from the Estimation Program Office,  as well as members of the Program Management and Control function of  the CADE 2 Program Management Office (PMO) responsible for tracking  the program’s cost. We compared what we learned with the 12 steps for  developing well-documented, comprehensive, accurate, and credible cost  estimates identified in the GAO Cost Estimating and Assessment Guide.
For our third objective, we reviewed documented procedures supporting  the risk management process, including the Risk and Issue Management  Process, the Risk and Issue Management Plan, the user guide for the Item  Tracking Reporting and Control system that is used to track risks and the  MITS Risk Identification Procedure. We also reviewed artifacts of the risk  management process, including program risk logs and detailed risk  reports. We participated in meetings in which risks were discussed and  reviewed agendas and meeting minutes for others. We also interviewed  several officials, including the CADE 2 Program Director, the Director of  the CADE 2 PMO Program Management and Control office, and the  Director of the CADE 2 PMO Delivery Management office. Through  document reviews and interviews with appropriate officials, we also  examined the following areas in greater depth: (1) schedule, (2) contractor  oversight, (3) requirements management, (4) prototypes, and (5) human  capital. We selected these areas because, based on our experience, they  represent areas of risk or have been problematic for IRS in the past. In  addition, we examined IRS’s prototyping efforts to understand how the  agency was using them to mitigate risks. We evaluated what we learned  about IRS’s risk management process against relevant criteria for Software  Engineering Institute’s Capability Maturity Model Integration (CMMI) to  determine the effectiveness of the process.
To obtain background information on the program, we reviewed a host of  documents developed by the CADE 2 PMO, including the CADE 2  Program Charter, Solution Architecture, Program Roadmap, Program  Management Plan, the charters of the two CADE 2 transition state 1  projects (Daily Processing and Database Implementation), as well as  governance team meeting minutes. We also observed CADE 2 meetings,  conducted interviews with and received briefings from CADE 2 officials,  including the IRS Chief Technology Officer, the Deputy Chief Information  Officer for Strategy and Modernization, the Associate Chief Information  Officer who serves as executive-in-charge of the CADE 2 program, and the  Chief Architect of the CADE 2 program.
We conducted this performance audit from January 2010 to March 2011  primarily at the New Carrollton Federal Building in New Carrollton,  Maryland, where the CADE 2 staff are located, and Atlanta, Georgia, where  the Wage and Investment division staff are located, in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions   based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
section_title: Appendix II: Additional Information about CADE 2 Benefits
paragraphs: Table 7 shows a list of benefits the IRS has defined to date for transition  state 1 (TS1) and beyond, a related quantitative target for a benefit if  applicable, and the transition state in which IRS expects to realize the  benefit. According to IRS, defining which benefits will be delivered in  transition state 2 (TS2) is contingent upon a number of decisions yet to be  made and actual benefits may differ from those currently expected  depending on the outcome of these decisions.
section_title: Appendix III: CADE 2 Program Risks
paragraphs: The following (see table 8) are the IRS’s reported CADE 2 program risks  and their planned mitigation strategies as of November 2010.
section_title: Appendix IV: Comments from the Internal Revenue Service
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the individuals named above, Sabine R. Paul, Assistant  Director; Joanna Stamatiades, Assistant Director; Rebecca Eyler; Mary D.  Fike; Nancy Glover; Sairah R. Ijaz; Robert Kershaw; Paul B. Middleton;  John Ockay; and Jennifer Wong made key contributions to this report.