Information Technology: Additional Executive Review Sessions Needed to Address Troubled Projects
Released: 2013-06-13T00:00:00

Letter

Background

If done correctly, investments in IT have the potential to make organizations more efficient in fulfilling their missions. For example, Department of Defense (DOD) officials recently reported that an IT system supporting military logistics has improved the organization’s performance by providing real-time information about road conditions, construction, incidents, and weather to facilitate rapid deployment of military assets. Also, Federal Aviation Administration officials reported that an IT system supporting weather data processing has improved aviation operations by integrating terminal and aircraft sensor data with forecast data from the National Weather Service, and providing it to air traffic controllers. These officials estimated that the system allows them to increase airspace capacity by 25 percent in certain weather conditions.

However, as we have previously reported, federal IT projects too frequently incur cost overruns and schedule slippages while contributing little to mission-related outcomes. For example, in May 2010, we reported that after spending $127 million over 9 years on an outpatient scheduling system, the Department of Veterans Affairs has not implemented any of the system’s capabilities and was essentially starting over. Further, in May 2012, we reported that while IT should enable government to better serve the American people, the federal government had not achieved expected productivity improvements—despite spending more than $600 billion on IT over the past decade.

OMB plays a key role in overseeing how federal agencies manage their IT investments by working with them to better plan, justify, and determine how to manage them. Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT projects and how these funds are to be allocated. OMB also guides agencies in developing sound business cases for IT investments and establishing management processes for overseeing these investments throughout their life cycles. The scope of this undertaking is quite large: in planning for fiscal year 2014, 27 federal agencies reported plans to spend about $76.5 billion on 8,142 IT investments.

Roles and Responsibilities for Overseeing Federal IT Investments

Over the last three decades, Congress has enacted several laws to assist agencies and the federal government in managing IT investments. For example, the Paperwork Reduction Act of 1995 specifies OMB and agency responsibilities for managing IT. Among its provisions, this law establishes agency responsibility for maximizing the value and assessing It also and managing the risks of major information systems initiatives.requires that OMB develop and oversee policies, principles, standards, and guidelines for federal agency IT functions, including periodic evaluations of major information systems.

In addition, to assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996. This law requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by federal agencies. It also requires that OMB report to Congress on the net program performance benefits achieved as a result of these investments.

As set out in these laws, OMB is to play a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects. Within OMB, the Office of E-government and Information Technology, headed by the Federal Chief Information Officer (CIO), directs the policy and strategic planning of federal IT investments and is responsible for oversight of federal technology spending.

Agency CIOs are also expected to have a key role in IT management. Federal law, specifically the Clinger-Cohen Act, has defined the role of the CIO as the focal point for IT management, requiring agency heads to designate CIOs to lead reforms that would help control system development risks; better manage technology spending; and achieve real, measurable improvements in agency performance. In September 2011, we reported that federal CIOs are not consistently responsible for all of the areas assigned by law or identified as critical to effective IT management. For example, although most of the CIOs were responsible for capital planning and investment management, we found that CIOs are less frequently responsible for information management duties such as records management and privacy requirements. In an August 2011 memo, OMB reiterated the primary areas of responsibility for agency CIOs. This memo detailed four areas in which the CIO should have a lead role: IT governance, program management, commodity services, and information security. It emphasized the role of the CIO in driving the investment review process, including TechStats, and the CIO’s responsibility over the entire IT portfolio for an agency.

OMB Established the IT Dashboard to Better Oversee and Manage Federal IT Investments

To help carry out its oversight role and further improve the transparency into and oversight of agencies’ IT investments, in June 2009 OMB publicly deployed a website, known as the IT Dashboard. The Dashboard displays federal agencies’ cost, schedule, and performance data for over 700 major federal IT investments at 27 federal agencies that comprise about $40 billion of the federal budget. According to OMB, these data are intended to provide both a historical and a near-real-time perspective on the performance of these investments. OMB analysts are expected to use the Dashboard to identify IT investments that are experiencing performance problems.

Using data drawn from federal agency budget submissions, the IT Dashboard provides information on an IT investment’s primary function, as defined by the Federal Enterprise Architecture. For fiscal year 2012 submissions, agencies were required to select a primary function from categories within the Federal Enterprise Architecture business reference models. The primary functions available for IT investments in fiscal year 2012 submissions were: administrative management community and social services controls and oversight correctional activities defense and national security disaster management economic development education energy environmental management financial management general government general science and innovation health homeland security human resource management information and technology management intelligence operations internal risk management and mitigation international affairs and commerce law enforcement legislative relations litigation and judicial activities natural resources planning and budgeting public affairs workforce management regulatory development revenue collection supply chain management transportation The Dashboard visually presents performance ratings for individual investments using metrics that OMB has defined—cost, schedule, and the CIO’s evaluation of risk. To develop the CIO’s risk evaluation, OMB instructed agency CIOs to assess their IT investments against a set of six evaluation factors, including risk management, requirements management, contractor oversight, historical performance, and human capital. The CIO assigns a rating of 1 to 5 based on his or her best judgment of the level of risk facing the investment. OMB then translates the agency CIO’s numerical assignment for an investment into a color for depiction on the Dashboard, with green signifying low or moderately low risk, yellow signifying medium risk, and red signifying moderately high or high risk (see table 1).

In January 2010, OMB began conducting TechStats to enable the federal government to intervene to turnaround, halt, or terminate IT projects that are failing or are not producing results. TechStats are face-to-face, evidence-based reviews of an at-risk IT investment. OMB used CIO ratings from the IT Dashboard, among other sources, to select at-risk investments for the TechStats it conducted from 2010 through 2011.

Subsequently, as part of the Federal CIO’s 25-point IT Reform Plan,OMB empowered agency CIOs to hold their own TechStat sessions within their respective agencies, and required federal agencies to hold at least one TechStat by March 2011. The IT Reform Plan also required agencies to roll the TechStat model out to its component-level agencies and bureaus (bureaus) by June 2012. To do this, agencies were required to make agency CIOs responsible for deploying the necessary tools and training on how to conduct TechStat reviews and have at least one bureau conduct TechStat reviews by June 2012. In August 2011, OMB required agency CIOs to continue holding TechStat sessions.

In establishing and rolling out the TechStat sessions, OMB stated that it expects that the sessions will help strengthen IT governance, improve line-of-sight between project teams and senior executives, increase the precision of ongoing measurement of IT program health, and boost the quality and timing of interventions to keep projects on track. We have found that the TechStat model is consistent with government and industry best practices for overseeing IT investments, including our own guidance on IT investment management processes. by focusing management attention on troubled projects and establishing clear action items to turn the projects around or terminate them.

GAO Has Previously Reported on Federal IT Investment Oversight and Reform

GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: Mar. 2004). deficiencies and use that list to report to Congress on progress made in correcting high-risk problems.

As a result, OMB started publicly releasing aggregate data on its internal list of mission-critical projects that needed to improve (called its Management Watch List) and disclosing the projects’ deficiencies. The agency also established a High-Risk List, which consisted of projects identified as requiring special attention from oversight authorities and the highest levels of agency management. In June 2009, OMB replaced the Management Watch List and the High-Risk List when it deployed a public website—the IT Dashboard—to further improve the transparency and oversight of agencies’ IT investments.

In 2010 and 2011, we reported that while the Dashboard was an important tool for monitoring major IT projects, the cost and schedule ratings were not always accurate for selected agencies.recommendations to improve the accuracy of the data and, more recently, found that the accuracy had improved.

In April 2012, we reported on the progress of OMB and selected federal agencies on action items in OMB’s IT Reform Plan. We found that, of 10 selected action items, agencies had completed 3 and made progress on the other 7. However, we found that agencies lacked time frames for completing 5 of those 7 in-progress action items, and only had performance measures for 4 of the 10 selected action items. Thus, we recommended, among other things, that OMB ensure that action items are completed prior to the IT Reform Plan’s June 2012 deadline and establish time frames and performance measures for the action items. OMB agreed with our recommendations to complete action items and provide deadlines, but disagreed with our recommendation to establish performance measures. OMB has made progress with completing incomplete action items, but more remains to be done. For example, one action item involved issuing contracting guidance to support modular development and OMB completed this action in June 2012. However, another action item involved ensuring agency data center consolidation plans were in place so agencies could close 800 data centers by 2015, and we recently reported that all but 1 of the 24 agencies’ plans were incomplete.

Most recently, in October 2012, we reported specifically on the CIO rating portion of the IT Dashboard at six selected agencies. CIOs at six federal agencies rated the majority of their IT investments as low risk, and that many ratings remained constant over time. For ratings that did change, we found two agencies reported more investments with reduced risk compared to earlier risk ratings; the other four agencies reported more investments with increased risk. In addition, we reported instances where the CIO ratings did not appropriately reflect significant cost, schedule, and performance issues reported by GAO and others. We recommended that the Federal CIO analyze agency trends reflected in Dashboard CIO ratings, and report the results of this analysis in future budget submissions. OMB agreed with the recommendation and recently released some information on trends as part of the 2014 budget. We have work under way to evaluate this information.

GAO, Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies, GAO-13-98 (Washington, D.C.: Oct. 16, 2012).

OMB and Selected Agencies Have Held Multiple TechStats on a Diverse Set of IT Investments, but Other At-Risk Investments Have Not Yet Been Assessed

OMB and the four selected agencies have held multiple TechStats on IT investments that varied in terms of function, significance, amount spent to date, and risk level. Specifically, from January 2010 through April 2013, OMB reported leading 79 TechStat sessions, which focused on 55 IT investments at 23 federal agencies. These investments covered 21 functional areas (such as information and technology management, law enforcement, and health), and consist of 45 major, 8 non-major, and 2 unrated investments. For the 45 major investments that OMB reviewed, almost 70 percent had a CIO rating of medium to high risk at the time of the TechStat review.

The four agencies selected for our review—Agriculture, Commerce, DHS, and HHS—held a total of 37 TechStat sessions covering 28 investments from January 2011 through March 2013. Most of the investments underwent a TechStat session at the agency level, but 8 underwent a TechStat review at the bureau level, and 2 underwent TechStat reviews at both. The agency-led TechStats covered 14 functional areas, and consist of 21 major and 7 non-major investments. For the 21 major investments, about 76 percent of the agency-led TechStats were on investments that had a CIO rating of medium- to high-risk.

While both OMB and agencies have made progress in holding TechStat sessions, there is more that could be done. Specifically, the number of TechStats held to date is relatively small compared to the total number of medium- and high-risk IT investments. Further, there are multiple high- risk IT investments spending millions of dollars that have not yet been assessed. Two of the selected agencies—Agriculture and Commerce— had reviewed all of their high-risk investments, and DHS has plans to review its remaining high-risk investments. However, HHS has not yet established plans to review all of its high-risk investments. Also, OMB does not have plans or schedules for assessing the other high-risk IT investments. Until OMB and agencies intervene to turn around these at- risk projects, the government will continue to spend limited IT investment dollars on underperforming projects.

OMB Reported Holding 79 TechStats for a Variety of IT Investments

As of April 2013, OMB reported conducting 79 TechStat reviews, with 59 reviews occurring in 2010, 8 in 2011, 11 in 2012, and 1 so far in 2013. OMB conducted fewer TechStats in recent years because it expected the agencies to increase the number of agency-led TechStats. Most of the OMB-led TechStats were at DHS and DOD. OMB has conducted at least one TechStat at 23 agencies (including the Office of the Director of National Intelligence). It has not held a TechStat at the Department of Labor, the National Aeronautics and Space Administration, the National Science Foundation, the Nuclear Regulatory Commission, or the Smithsonian Institution. Figure 1 shows the total number of reported OMB-led TechStats at each agency as of April 2013.

These 79 TechStat reviews included 55 IT investments from 23 federal agencies. OMB also reported leading follow-up sessions for many of these investments. For example, OMB led 4 TechStats on a DHS investment called the Federal Emergency Management Agency— National Flood Insurance Program Information Technology Systems and Services. Table 2 identifies the investments for which OMB led TechStats, the date of the first session, and the total number of sessions that OMB has held on that investment.

These 55 investments span 21 primary functional areas.common functional area was information and technology management, followed by homeland security and human resource management. Figure 2 provides a numerical depiction of the functions of those investments that were subject to a TechStat.

OMB held most of its TechStats on major IT investments, with 45 of its 55 investments rated as major investments. OMB also held 8 TechStats on non-major investments and 2 on investments lacking a major/non-major designation. Figure 3 shows the breakdown of major and non-major investments.

Thirty-eight of the 45 major investments that underwent an OMB-led TechStat have reportedly cost the federal government about $16 billion through fiscal year 2012. DHS’s investments that were the subject of a TechStat session accounted for most of the cost, with a reported $4.9 billion spent. DOT follows with investments that have reportedly cost $2.8 billion through September 2012. The reported cost of individual investments ranged from $4.18 million to $3.2 billion. Table 3 details how much each agency has spent on its major investment.

Of the 45 major IT investments that were subject to a TechStat, 67 percent were considered medium- to high-risk investments at the time they were chosen for a TechStat. Specifically, 9 had high- or moderately high-risk (red) CIO ratings on the IT Dashboard, 21 had medium-risk (yellow) ratings, and 9 had low- or moderately low-risk (green) ratings. In addition, 6 investments’ risk levels were not identified. Table 4 shows the Dashboard ratings for the major IT investments that underwent a TechStat.

From January 2011 through March 2013, the four selected agencies— Agriculture, Commerce, DHS, and HHS—held 37 TechStats that covered 28 different investments. Commerce held the most, with 15 TechStat sessions, followed by DHS, HHS, and Agriculture, with 8, 7, and 7 sessions, respectively. Table 5 lists the investments, the date of the first TechStat session, and the total number of TechStats by agency.

Although most of the agency-led TechStat reviews were held at the agency level, three of the agencies in our review have also been holding them at the bureau or component agency level. Specifically, of the agencies in our review, 18 of the 28 investments underwent at least one TechStat review at the agency level, 8 underwent a TechStat review at the bureau level, and 2 underwent TechStat reviews at both. Of the selected agencies, Agriculture had not held a bureau-level TechStat (see fig. 4).

Like the OMB-led reviews, the investments selected by agencies for TechStat reviews had diverse primary functional areas. Specifically, 25 of the investments that underwent agency-led TechStats at the four selected agencies cover 14 different functional areas. The most common area was information and technology management (6 investments), followed by environmental management (4 investments) (see fig. 5).

Most of the investments that underwent a TechStat review at the four agencies were major investments, with a quarter of the sessions focusing on non-major investments (see fig. 6).

The four agencies reported spending $7.8 billion through the end of fiscal year 2012 on 20 of the 21 major investments that underwent an agency- led review.for a new Commerce investment to $3.7 billion for a DHS investment. Table 6 provides a summary of the amount spent on major investments through September 2012.

Selected Agencies are Generally Conducting TechStats in Accordance with OMB Guidance, but Areas for Improvement Exist

The four selected agencies are generally conducting TechStats in accordance with OMB guidance. In 2011, when OMB decided to move beyond conducting its own TechStats and to have agencies conduct them too, OMB provided agencies guidance through the IT Reform Plan, a memorandum, and the TechStat Toolkit, which is available on the CIO Council’s website. These documents include 15 key requirements, including when TechStats should be implemented by the agencies, what participants should be included, how at-risk investments should be chosen, and how outcomes should be tracked and reported. The requirements can be grouped into four broad categories: scope, governance, process, and outcomes. Table 10 shows key requirements for TechStat reviews.

The four selected agencies implemented most of OMB’s 15 key requirements. Specifically, DHS implemented all 15 requirements, Commerce fully implemented 14 requirements and partially implemented 1 requirement, HHS fully implemented 11 of the requirements and partially implemented 4 requirements, and Agriculture fully implemented 10 of the requirements, partially implemented 3 requirements, and did not implement 2 requirements.

All four agencies fully implemented eight of OMB’s requirements: (1) the appropriate people are invited to the TechStat meetings, (2) sessions are led by the CIO, (3) the preparation for the TechStat is performed by the TechStat team and the CIO, (4) investments selected by TechStats are done so by the prescribed criteria, (5) all action items are tracked in a consolidated repository, (6) each investment is tracked into an outcome, (7) outcomes are shared with OMB, and (8) results are published in an collaborative tool. The requirement with the least implementation involves documenting action items, deadlines, and responsible parties in a memorandum following each TechStat. Table 11 provides details regarding the agencies’ implementation of OMB guidance on TechStats.

While the four selected agencies have largely implemented OMB’s guidance on conducting TechStats, three agencies have selected areas where they can improve. For example, Agriculture created memorandums following a TechStat, but did not consistently include responsible parties; Commerce created memorandums, but did not include deadlines; and HHS did not always create memorandums or monitor all of its action items to closure.

Agency officials noted several reasons for not fully implementing OMB’s guidance. Specifically, Agriculture officials noted that they are updating their capital planning guidance to include TechStat reviews, but that this guidance is in the process of being reviewed and approved. Also, Commerce and HHS officials noted that OMB gave agencies flexibility in exactly how to implement their guidance. While OMB did provide agencies flexibility in selecting investments for TechStat reviews and conducting those reviews, the requirements are clearly delineated in OMB instructions and training. Fully implementing OMB’s TechStat guidance could better position the agencies to realize the benefits of the TechStat initiative—including strengthening overall IT governance and oversight, and proactively identifying and resolving problems before investments experience delays or cost overruns.

OMB and Agencies Reported Positive Results and Cost Savings from Holding TechStats, but Are Not Ensuring Results Are Valid

The IT Reform Plan and related OMB guidance instructed agencies to track and report on the outcomes of their TechStat sessions (including improved governance, accelerated deliveries, and terminated projects), and on any associated cost implications (such as cost savings or cost avoidances).

OMB and the four agencies we reviewed have tracked and reported positive results from TechStats, with most resulting in improved governance or accelerated deliveries. OMB also reported that federal agencies achieved over $3 billion in cost savings or avoidances as a result of the OMB-led TechStats in 2010 and $900 million from agency- led TechStats in 2011. Using a different calculation formula, OMB also reported that TechStats resulted in $63.5 million in cost implications in 2012.

However, we were unable to validate the reported outcomes and associated savings because OMB did not provide supporting artifacts or demonstrate the steps that OMB analysts took to verify the agencies’ data. Without documentation or an explanation of its method in validating agencies’ reported results and cost savings, it will be difficult for OMB to provide a sufficient level of confidence to Congress and the public that the information it has presented is credible.

OMB and Agencies Reported Positive Results from TechStats

Both OMB and agencies reported achieving positive results from their respective TechStat sessions. Specifically, in 2011 OMB staff reported achieving a variety of positive outcomes from the OMB-led TechStat sessions, including 11 investments being reduced in scope, four investments that were cancelled, and multiple investments with accelerated program delivery. We also identified four investments that were the focus of an OMB-led TechStat that were subsequently terminated and two other investments that were split into multiple smaller investments to improve governance and accelerate the delivery of discrete capabilities. Further, we previously found that the June 2010 TechStat on the National Archives and Records Administration’s Electronic Records Archives investment resulted in six corrective actions, including halting fiscal year 2012 development funding pending the completion of a strategic plan.likely experience cost overruns of between $205 and $405 million if the agency completed the program as originally designed.

We estimated that the program would Seeking to improve the reporting of outcomes, OMB instructed federal agencies to track the results of each agency-led TechStat session into one of six outcomes: accelerated delivery, improved governance, reduced scope, eliminated duplication, halted, or terminated. In December 2011, OMB reported a summary of the outcomes of 294 agency-led TechStats across the federal government. Specifically, OMB reported the following outcomes:

49 percent resulted in accelerated delivery,

42 percent resulted in improved governance,

3 percent did not report results,

2 percent resulted in terminations,

1 percent resulted in reducing the scope,

1 percent eliminated duplication, and

1 percent halted the investment.

In addition, the four agencies in our review reported on the results of their agency-led TechStats, with the majority resulting in improved governance. Specifically, out of 36 TechStat reviews, 28 resulted in improved governance, 4 in accelerated delivery, 2 in terminations, and 2 in a reduced scope. Table 12 provides a summary of reported results, by agency.

In conjunction with the reported outcomes of agency-led TechStats, OMB instructed agencies to provide information on the cost implications of the outcomes. These implications could include cost savings (a reduction in actual expenditures) or cost avoidances (an action taken immediately that will reduce costs in the future). In addition, OMB’s guidance on performance reporting notes that performance data should be appropriately accurate and reliable for their intended use. This guidance further describes verification and validation techniques that OMB encourages agencies to use in internal assessments, including ensuring that supporting documentation is maintained and readily available, data are verified as appropriate to the needed level of accuracy, and data limitations are explained and documented.

OMB has reported cost savings and avoidances from OMB- and agency- led TechStat reviews. Specifically, OMB has reported that federal agencies achieved about $3 billion in cost savings or avoidances as a result of the OMB-led TechStats held in 2010 and 2011, and $900 million from agency-led TechStats in 2011. See figure 7 for the reported cost implications for both OMB-led and agency-led TechStats as of November 2011.

In addition, OMB reported that the cost savings and avoidances for the agency-led TechStat reviews, as of November 2011, came from 10 of the 27 agencies under their purview. Department of Transportation had the most cost implications, with $510 million. See figure 8 for a depiction of the cost implications by agency.

Two of the agencies in our review—DHS and HHS—reported $23 million in cost implications from three agency-led TechStats:

DHS reported a $14 million cost avoidance after deciding to decommission one part of the Federal Protective Service’s Risk Assessment and Management Program;

HHS reported saving $8.2 million by reengineering redundant business processes supporting the Food and Drug Administration’s Mission Accomplishment and Regulatory Compliance Service project; and

HHS reported $800,000 in cost implications associated with improving governance and reducing the scope of the One Stop Service Solution project (now called GovZone).

More recently, OMB has been reporting cost savings from agency-led TechStat reviews in quarterly reports to Congress. From December 2011 through December 2012, OMB identified a total of $63.5 million in cost implications from agency-led TechStats. OMB staff stated that they are calculating the cost savings in these quarterly reports differently than their prior reports on cost savings, and thus the costs should not be compared.

OMB and Agencies Are Not Ensuring Reported Outcomes and Cost Implications Are Valid

When collecting data, it is important to have assurance that the data are accurate. Best practices in implementing the Government Performance and Results Act of 1993 emphasize the need for agencies, when providing information, to explain the procedures used to verify or validate their data. Specifically, agencies should ensure that reported data are sufficiently complete, accurate, and consistent, and also identify any significant data limitations. Explaining the limitations of the information can provide a context for understanding and assessing the challenges agencies face in gathering, processing, and analyzing needed data. Such a presentation of data limitation can also help identify the actions needed to improve the agency’s ability to measure its performance. More recently, we have reiterated the importance of providing OMB with complete and accurate data.

DHS and HHS, the two agencies in our review that reported cost savings and avoidances, generally perform analyses to establish these estimates. DHS provided supporting documentation for how the cost implications for its investment were calculated and HHS provided supporting documentation for its larger investment. We were unable to validate the cost savings for HHS’s smaller investment because the agency was unable to provide supporting documentation.

From a governmentwide perspective, OMB staff explained that they review agencies’ cost implication data for completeness and quality. However, we were unable to validate OMB’s reported outcomes and almost $4 billion in cost implications because OMB did not provide documentation on any steps that it or the agencies took to ensure the validity of the agencies’ outcome and cost data. For example, OMB staff did not provide memorandums documenting action items from the OMB- led TechStats, the outcomes identified for OMB- and agency-led TechStats, documentation identifying which investments resulted in cost savings, documentation demonstrating the methodology used to calculate the cost savings, or a summary of steps the agencies took to validate reported cost savings. Moreover, OMB does not require agencies to report on what steps they took to verify their reported outcomes and cost savings.

By not requiring agencies to report on their efforts to validate reported outcomes and cost savings, it is not evident that OMB is following its own guidance for ensuring that performance data are reliable and accurate. Moreover, OMB is not providing reasonable assurance to Congress and the public that the information it has presented is credible.

One entity that was formed to assist OMB in its oversight of the TechStat results was the CIO Council’s subcommittee on IT Governance and TechStats, but according to OMB and agency officials this subcommittee has already been dissolved. The CIO Council’s Management Best Practices Committee formed the subcommittee to report on the outcomes and lessons learned from implementing the TechStat process. This subcommittee was to play a key role in continuing to mature the TechStat process. However, in 2012, the CIO Council dissolved the subcommittee because the committee chairs determined that it was no longer needed.

Conclusions

Consistent with government and industry best practices for overseeing IT investments, TechStat sessions hold value by focusing management attention on troubled projects and establishing clear action items to turn the projects around or terminate them. While OMB and agencies are reporting positive results from holding TechStat sessions, neither are doing enough to ensure that at-risk investments are undergoing review, sound processes are in place, and reported results are valid. Specifically, agencies are reviewing only about a third of their at-risk IT investments. Until OMB and agencies develop plans and schedules for addressing these at-risk investments, the investments will likely remain at risk.

The four agencies we reviewed had implemented most of the OMB- required TechStat processes, but three had shortfalls in selected processes. For example, Agriculture had not yet incorporated TechStats in its investment management processes, Commerce had not consistently included deadlines for action items in its TechStat memoranda, and HHS had not consistently created action item memoranda following TechStats or tracked its action items to completion. Addressing these shortfalls could better position these agencies to realize the full benefits that TechStats offer.

OMB regularly reports on cost savings associated with TechStats, but it has not taken basic steps to provide reasonable assurance to Congress and the public that these data are valid. Until OMB requires agencies to report on what they did to validate cost savings data and shares this information, neither Congress nor the public can be assured that TechStats are as effective as reported. While the CIO Council recently dissolved its subcommittee responsible for reviewing TechStats, the council is one entity that is uniquely positioned to assist OMB in its oversight of the TechStat results.

Recommendations for Executive Action

To ensure that TechStat sessions are having the appropriate impact in the oversight of underperforming projects, we are making three recommendations to OMB. Specifically, we recommend that the Director of the Office of Management and Budget direct the Federal Chief Information Officer to require agencies to conduct TechStats for each IT investment rated with a moderately high- or high-risk CIO rating on the IT Dashboard, unless there is a clear reason for not doing so; require agencies to report to OMB on efforts to validate the outcomes, cost savings, and cost avoidances resulting from TechStat sessions; this information should be summarized when OMB reports on governmentwide outcomes; and direct the Federal CIO Council to track the outcome of TechStat sessions and to support OMB’s efforts to validate the resulting cost savings it reports to Congress.

In addition, we are making a recommendation to the Secretaries of Agriculture and Commerce to address the weaknesses in agency- and bureau-led TechStat processes and management outlined in this report.

We are also making two recommendations to the Secretary of Health and Human Services to establish a plan and schedule for addressing each IT investment rated with a moderately high- or high-risk CIO rating on the IT Dashboard; such a plan could include conducting a TechStat session, and address the weaknesses in agency- and bureau-led TechStat processes and management outlined in this report.

Agency Comments and Our Evaluation

We requested comments from OMB, Agriculture, Commerce, DHS, and HHS on a draft of our report. In that draft report, we had made recommendations to the Secretaries of Agriculture, Commerce, and DHS to establish a plan and schedule for addressing their high-risk investments that had not yet undergone TechStat reviews. However, these agencies completed these steps prior to the issuance of this report. Therefore, we removed the applicable recommendations.

OMB and three agencies provided comments on our draft report. OMB generally agreed with our recommendations; Commerce agreed with our recommendation; and Agriculture and HHS did not agree or disagree with our recommendations. DHS declined to provide comments. Each agency’s comments are discussed in more detail below: In comments provided via e-mail, staff from OMB’s Office of E- Government and Information Technology generally concurred with our findings and recommendations and stated that OMB and the agencies are currently taking appropriate steps to meet the recommendations. OMB provided additional details as follows: In commenting on our finding that it did not ensure the validity of the outcomes and savings it reported, OMB noted that it is confident in the agency validation of cost-savings and avoidances through the TechStat sessions, and that it was speculative to conclude that the reported cost savings data were not valid because we could not actually assess these figures. OMB did not provide supporting documentation for the cost savings calculations and explained that the process used to validate the cost figures was deliberative. However, part of a sound process for ensuring validity involves documenting the procedures used to verify or validate the data; as such, we were unable to validate OMB’s reported outcomes and almost $4 billion in cost implications from 2010 and 2011 because OMB did not provide documentation on any steps that it or the agencies took to ensure the validity of the agencies’ outcome and cost data. For example, OMB staff did not provide memorandums documenting action items from the OMB-led TechStats, the outcomes identified for OMB- and agency-led TechStats, documentation identifying which investments resulted in cost savings, documentation demonstrating the methodology used to calculate the cost savings, or a summary of steps the agencies took to validate reported cost savings. Moreover, OMB did not require agencies to report on what steps they took to verify their reported outcomes and cost savings. Without disclosing the steps it took to validate the outcomes and cost savings, OMB has not provided reasonable assurance to Congress and the public that the information it has presented is credible.

OMB also noted that since June 2012, OMB and agencies have validated and transmitted cost-savings and avoidance data associated with TechStat sessions to Congress on a quarterly basis through the Integrated, Efficient, and Effective Use of IT Report. These reports summarize the validation methods OMB and agencies undertake. We acknowledge that these reports summarize methods through which the costs savings and avoidances were documented; however, due to a lack of supporting documentation, we could only validate $22.2 million of the $63.5 million cost savings and avoidances OMB recently reported.

In written comments, the Department of Agriculture’s Chief Information Officer partially agreed with our assessment of the agency’s TechStat process but did not specify the part of the assessment with which she disagreed. The department stated that it would continue to conduct periodic reviews of its IT investments to ensure a review of all major IT investments. In addition, the department stated that IT Dashboard ratings alone are not true indicators that an investment is poorly performing or underperforming and that it takes into consideration other factors, such as earned value management data, when determining an investment’s performance. We agree that the CIO rating should reflect the CIO’s assessment of the risk and should use many factors in determining the rating. The department’s written comments are provided in appendix II.

In written comments, the Department of Commerce’s Acting Secretary concurred with our recommendation and stated that the department is in full compliance with the recommendation. The department plans to ensure that all TechStat memos have a specific point of contact and a specific due date. The department’s written comments are provided in appendix III.

In comments provided via e-mail, a management analyst within the Department of Health and Human Services’ Office of the Assistant Secretary for Legislation stated that the department had no general comments. The department provided technical comments, which we have incorporated as appropriate. In its technical comments, HHS commented on our finding that the agency did not satisfy OMB’s requirement to hold its first TechStat session by March 2011. The HHS officials stated that the agency had appropriately initiated the process for holding a TechStat prior to OMB’s March 2011 deadline. While we acknowledge that the agency initiated the process prior to the deadline, the actual session was held after the deadline.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) identify key characteristics of TechStat Accountability Sessions (TechStats) conducted by the Office of Management and Budget (OMB) and selected agencies, (2) evaluate whether selected agencies are conducting TechStats in accordance with OMB guidance, and (3) analyze the extent to which reported results from TechStat review sessions are documented, tracked, and validated.

In conducting our review, we selected four agencies based on two factors: the number of investments with medium-, moderately high-, and high-risk Chief Information Officer (CIO) ratings on the information technology (IT) Dashboard as of August 2012, and a determination on whether the agency had led its own TechStats in 2011. The four agencies we selected—the Departments of Agriculture (Agriculture), Commerce (Commerce), Health and Human Services (HHS), and Homeland Security (DHS)—had the highest number of medium-, moderately high-, and high- risk investments. In addition, in December 2011, the CIO Council reported We that all four agencies had held their own TechStat sessions in 2011.assessed the reliability of the IT Dashboard data and the data on agency- led TechStats by comparing them to agency documents, and found that the data were sufficient for our purpose of selecting agencies for further review.

We reviewed exhibit 300 and 53 data linked on the IT Dashboard as of the last update in August 2012. data for each of these investments. We also interviewed OMB, Agriculture, Commerce, DHS, and HHS officials regarding the TechStat sessions. We assessed the reliability of OMB’s and the four agencies’ lists of reported TechStat sessions by seeking corroboration for the sessions held at our selected agencies. We determined that OMB’s and the four agencies’ lists were sufficient for our purposes.

To address the second objective, we analyzed OMB’s guidance to agencies on conducting TechStat reviews and identified15 key requirements. We then compared these requirements to the selected agencies’ documentation, including capital planning and investment control plans and guidance, agency-specific TechStat guidance and training materials, meeting minutes from investment review board meetings, and electronic submissions from the agency to OMB regarding the TechStat sessions. To identify whether agencies were documenting, tracking, and monitoring action items—one of the 15 key requirements— we conducted a random sample of 10 percent of each agency’s action items and sought documentation from each of the agencies to support the status of the selected action items. This sample is not generalizable to the entire population of action items from each agency, but we determined it was sufficiently reliable for our purposes of determining whether the agencies were tracking and monitoring action items. We also interviewed officials from OMB and our four selected agencies about their respective TechStat processes.

To address the third objective, we identified OMB’s guidance to agencies on reporting the results of agency-led TechStat sessions. We analyzed the selected agencies’ and OMB’s reported results in each of OMB’s outcome categories and attempted to validate OMB’s and the agencies’ reported cost implications. We also reviewed quarterly reports from OMB to Congress to identify additional cost implication information. As noted in the body of this report, we were unable to determine the reliability of the reported cost implications because OMB did not provide artifacts demonstrating how it validated the data or evidence that it obtained agency validation of the data.

We conducted this performance audit from September 2012 to June 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Agriculture

Appendix III: Comments from the Department of Commerce

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Colleen Phillips (Assistant Director), Rebecca Eyler, Kate Feild, and Jessica Waselkow.