Information Technology: Additional Executive Review Sessions Needed to Address Troubled Projects
Published: 2013-06-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: While IT investments have the potential to make organizations more efficient, many federal IT projects experience cost overruns, schedule delays, and performance shortfalls. To help address these shortfalls, OMB established TechStats--face-to-face meetings to terminate or turnaround IT investments that are failing or are not producing results.
GAO was asked to evaluate the implementation of TechStats. GAO's objectives were to (1) identify key characteristics of TechStats held to date; (2) evaluate whether selected agencies are conducting TechStats in accordance with OMB guidance, and (3) analyze the extent to which reported TechStat results are tracked and validated. To do so, GAO selected four agencies--Agriculture, Commerce, HHS, and DHS--because these were the agencies with the highest number of at-risk investments. GAO analyzed OMB and agency documentation, compared agency processes to TechStat guidance, compared efforts to validate reported outcomes to leading practices, and interviewed OMB and agency officials.
section_title: What GAO Found
paragraphs: Since January 2010, the Office of Management and Budget (OMB) and selected agencies have held multiple TechStat Accountability Sessions (TechStats) on information technology (IT) investments that varied in terms of function, significance, and risk. As of April 2013, OMB reported conducting 79 TechStats, which focused on 55 investments at 23 federal agencies. The four agencies conducted 37 TechStats covering 28 investments. About 70 percent of the OMB- and 76 percent of agency-led TechStats on major investments were considered medium- to high-risk at the time of the TechStat. However, the number of at-risk TechStats held to date is relatively small compared to the current number of medium- and high-risk IT investments. Until OMB and agencies develop plans to address these investments, the investments will likely remain at risk.
The selected agencies are generally conducting TechStats in accordance with OMB guidance. OMB's TechStat guidance includes 15 key requirements, such as when TechStats should be implemented, what participants should be included, and how outcomes should be tracked and reported. DHS implemented all of the TechStat requirements. Commerce, HHS, and Agriculture implemented a majority of the requirements, but each had shortcomings. For example, these agencies did not consistently create memorandums with responsible parties and due dates for action items. Fully implementing OMB's guidance could better position agencies to effectively manage and resolve problems on IT investments.
OMB and selected agencies have tracked and reported positive results from TechStats, with most resulting in improved governance. OMB also reported in 2011 that federal agencies achieved almost $4 billion in life-cycle cost savings as a result of TechStat sessions. However, GAO was unable to validate OMB's reported results because OMB did not provide artifacts showing that it ensured the results were valid. From GAO's selected agencies, three investments had cost implications. Agencies provided supporting documentation for about $22.2 million in cost savings and avoidances. Until OMB obtains and shares information on the methods used to validate reported results, it will be difficult for the results to be independently validated and for OMB to provide assurance to Congress and the public that TechStats are achieving their intended impact.
section_title: What GAO Recommends
paragraphs: GAO is making recommendations to OMB to require agencies to address high-risk investments and to report on how they validated the outcomes. GAO is also making recommendations to selected agencies to address weaknesses in following OMB's TechStat guidance. OMB and Commerce officials generally agreed with GAO's recommendations. Agriculture partially agreed with GAO's assessment; neither it nor HHS commented on the recommendations.

section_title: Letter
paragraphs: Many information technology (IT) projects undertaken by the federal  government have cost hundreds of millions of dollars more than  anticipated and taken years longer to deploy than was originally  expected. To help address these issues, the Office of Management and  Budget (OMB) launched TechStat Accountability Sessions (TechStats) in  January 2010. A TechStat is a face-to-face, evidence-based  accountability review of an IT investment that enables the federal  government to intervene to turnaround, halt, or terminate projects that are  failing or are not producing results. OMB began leading TechStat  sessions on agency IT projects in 2010, and subsequently required  federal agencies to start holding them too.
You asked us to evaluate how OMB and selected federal agencies are  implementing TechStats. Our objectives were to (1) identify key  characteristics of TechStats conducted by OMB and selected agencies,  (2) evaluate whether selected agencies are conducting TechStats in  accordance with OMB guidance, and (3) analyze the extent to which  reported results from TechStat review sessions are documented, tracked,  and validated.
To do so, we selected four agencies—the Department of Agriculture  (Agriculture), the Department of Commerce (Commerce), the Department  of Health and Human Services (HHS), and the Department of Homeland  Security (DHS)—based on two factors: the number of at-risk IT  investments shown on OMB’s IT Dashboard as of August 2012 and a  determination on whether the agency had led TechStat sessions during  2011. We analyzed OMB’s list of the TechStat sessions it had led through  February 2013 as well as the latest budget data for each of these  investments to determine key characteristics of the investments that  underwent OMB-led TechStats. We also analyzed the four selected  agencies’ lists of the agency-led TechStat reviews held as of March 2013  as well as budgetary and other supporting documentation for each  investment and review to determine key characteristics of the investments  that underwent agency-led TechStats. We compared the OMB and  agency TechStat documentation against OMB guidance to identify  strengths or weaknesses in the agencies’ approaches. We also compared  OMB’s and the agencies’ reported TechStat results to OMB’s guidance   In  on reporting agency-led outcomes and other relevant best practices.addition, we interviewed officials at OMB and our four selected agencies  to better understand the TechStat process and associated results from  these reviews. While we were able to corroborate information about the IT  investments, we were unable to determine the reliability of the reported  outcomes and cost savings for the OMB-led TechStats because OMB did  not provide evidence that would allow us to verify its reported data.
We conducted this performance audit from September 2012 to June 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I provides additional  details on our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: If done correctly, investments in IT have the potential to make  organizations more efficient in fulfilling their missions. For example,  Department of Defense (DOD) officials recently reported that an IT  system supporting military logistics has improved the organization’s  performance by providing real-time information about road conditions,  construction, incidents, and weather to facilitate rapid deployment of  military assets. Also, Federal Aviation Administration officials reported  that an IT system supporting weather data processing has improved  aviation operations by integrating terminal and aircraft sensor data with  forecast data from the National Weather Service, and providing it to air  traffic controllers. These officials estimated that the system allows them to  increase airspace capacity by 25 percent in certain weather conditions.
However, as we have previously reported, federal IT projects too  frequently incur cost overruns and schedule slippages while contributing  little to mission-related outcomes. For example, in May 2010, we reported  that after spending $127 million over 9 years on an outpatient scheduling  system, the Department of Veterans Affairs has not implemented any of  the system’s capabilities and was essentially starting over. Further, in  May 2012, we reported that while IT should enable government to better  serve the American people, the federal government had not achieved  expected productivity improvements—despite spending more than $600  billion on IT over the past decade.
OMB plays a key role in overseeing how federal agencies manage their  IT investments by working with them to better plan, justify, and determine  how to manage them. Each year, OMB and federal agencies work  together to determine how much the government plans to spend on IT  projects and how these funds are to be allocated. OMB also guides  agencies in developing sound business cases for IT investments and  establishing management processes for overseeing these investments  throughout their life cycles. The scope of this undertaking is quite large: in  planning for fiscal year 2014, 27 federal agencies reported plans to spend  about $76.5 billion on 8,142 IT investments.
subsections: section_title: Roles and Responsibilities for Overseeing Federal IT Investments
paragraphs: Over the last three decades, Congress has enacted several laws to assist  agencies and the federal government in managing IT investments. For  example, the Paperwork Reduction Act of 1995 specifies OMB and  agency responsibilities for managing IT. Among its provisions, this law  establishes agency responsibility for maximizing the value and assessing   It also  and managing the risks of major information systems initiatives.requires that OMB develop and oversee policies, principles, standards,  and guidelines for federal agency IT functions, including periodic  evaluations of major information systems.
In addition, to assist agencies in managing their investments, Congress  enacted the Clinger-Cohen Act of 1996. This law requires OMB to establish processes to analyze, track, and evaluate the risks and results  of major capital investments in information systems made by federal  agencies. It also requires that OMB report to Congress on the net  program performance benefits achieved as a result of these investments.
As set out in these laws, OMB is to play a key role in helping federal  agencies manage their investments by working with them to better plan,  justify, and determine how much they need to spend on projects and how  to manage approved projects. Within OMB, the Office of E-government  and Information Technology, headed by the Federal Chief Information  Officer (CIO), directs the policy and strategic planning of federal IT  investments and is responsible for oversight of federal technology  spending.
Agency CIOs are also expected to have a key role in IT management.  Federal law, specifically the Clinger-Cohen Act, has defined the role of  the CIO as the focal point for IT management, requiring agency heads to  designate CIOs to lead reforms that would help control system  development risks; better manage technology spending; and achieve real,  measurable improvements in agency performance. In September 2011,  we reported that federal CIOs are not consistently responsible for all of  the areas assigned by law or identified as critical to effective IT  management. For example, although most of the CIOs were responsible  for capital planning and investment management, we found that CIOs are  less frequently responsible for information management duties such as  records management and privacy requirements. In an August 2011  memo, OMB reiterated the primary areas of responsibility for agency  CIOs. This memo detailed four areas in which the CIO should have a  lead role: IT governance, program management, commodity services,  and information security. It emphasized the role of the CIO in driving the  investment review process, including TechStats, and the CIO’s  responsibility over the entire IT portfolio for an agency.
section_title: OMB Established the IT Dashboard to Better Oversee and Manage Federal IT Investments
paragraphs: To help carry out its oversight role and further improve the transparency  into and oversight of agencies’ IT investments, in June 2009 OMB  publicly deployed a website, known as the IT Dashboard. The  Dashboard displays federal agencies’ cost, schedule, and performance  data for over 700 major federal IT investments at 27 federal agencies that  comprise about $40 billion of the federal budget. According to OMB,  these data are intended to provide both a historical and a near-real-time  perspective on the performance of these investments. OMB analysts are  expected to use the Dashboard to identify IT investments that are  experiencing performance problems.
Using data drawn from federal agency budget submissions, the IT  Dashboard provides information on an IT investment’s primary function,  as defined by the Federal Enterprise Architecture. For fiscal year 2012  submissions, agencies were required to select a primary function from  categories within the Federal Enterprise Architecture business reference  models. The primary functions available for IT investments in fiscal year  2012 submissions were: administrative management community and social services controls and oversight correctional activities defense and national security disaster management economic development education energy environmental management financial management general government general science and innovation health homeland security human resource management information and technology  management  intelligence operations  internal risk management and  mitigation  international affairs and  commerce  law enforcement  legislative relations  litigation and judicial activities natural resources planning and budgeting public affairs workforce management  regulatory development  revenue collection  supply chain management  transportation  The Dashboard visually presents performance ratings for individual  investments using metrics that OMB has defined—cost, schedule, and  the CIO’s evaluation of risk. To develop the CIO’s risk evaluation, OMB  instructed agency CIOs to assess their IT investments against a set of six  evaluation factors, including risk management, requirements  management, contractor oversight, historical performance, and human  capital. The CIO assigns a rating of 1 to 5 based on his or her best  judgment of the level of risk facing the investment. OMB then translates  the agency CIO’s numerical assignment for an investment into a color for  depiction on the Dashboard, with green signifying low or moderately low  risk, yellow signifying medium risk, and red signifying moderately high or  high risk (see table 1).
In January 2010, OMB began conducting TechStats to enable the federal  government to intervene to turnaround, halt, or terminate IT projects that  are failing or are not producing results. TechStats are face-to-face,  evidence-based reviews of an at-risk IT investment. OMB used CIO  ratings from the IT Dashboard, among other sources, to select at-risk  investments for the TechStats it conducted from 2010 through 2011.
Subsequently, as part of the Federal CIO’s 25-point IT Reform Plan,OMB empowered agency CIOs to hold their own TechStat sessions  within their respective agencies, and required federal agencies to hold at  least one TechStat by March 2011. The IT Reform Plan also required  agencies to roll the TechStat model out to its component-level agencies  and bureaus (bureaus) by June 2012. To do this, agencies were required  to make agency CIOs responsible for deploying the necessary tools and  training on how to conduct TechStat reviews and have at least one  bureau conduct TechStat reviews by June 2012. In August 2011, OMB  required agency CIOs to continue holding TechStat sessions.
In establishing and rolling out the TechStat sessions, OMB stated that it  expects that the sessions will help strengthen IT governance, improve  line-of-sight between project teams and senior executives, increase the  precision of ongoing measurement of IT program health, and boost the  quality and timing of interventions to keep projects on track. We have  found that the TechStat model is consistent with government and industry  best practices for overseeing IT investments, including our own guidance  on IT investment management processes. by focusing management attention on troubled projects and establishing  clear action items to turn the projects around or terminate them.
section_title: GAO Has Previously Reported on Federal IT Investment Oversight and Reform
paragraphs: GAO, Information Technology Investment Management: A Framework for Assessing  and Improving Process Maturity, GAO-04-394G (Washington, D.C.: Mar. 2004). deficiencies and use that list to report to Congress on progress made in  correcting high-risk problems.
As a result, OMB started publicly releasing aggregate data on its internal  list of mission-critical projects that needed to improve (called its  Management Watch List) and disclosing the projects’ deficiencies. The  agency also established a High-Risk List, which consisted of projects  identified as requiring special attention from oversight authorities and the  highest levels of agency management. In June 2009, OMB replaced the  Management Watch List and the High-Risk List when it deployed a public  website—the IT Dashboard—to further improve the transparency and  oversight of agencies’ IT investments.
In 2010 and 2011, we reported that while the Dashboard was an  important tool for monitoring major IT projects, the cost and schedule  ratings were not always accurate for selected agencies.recommendations to improve the accuracy of the data and, more recently,  found that the accuracy had improved.
In April 2012, we reported on the progress of OMB and selected federal  agencies on action items in OMB’s IT Reform Plan. We found that, of 10  selected action items, agencies had completed 3 and made progress on  the other 7. However, we found that agencies lacked time frames for  completing 5 of those 7 in-progress action items, and only had  performance measures for 4 of the 10 selected action items. Thus, we  recommended, among other things, that OMB ensure that action items  are completed prior to the IT Reform Plan’s June 2012 deadline and  establish time frames and performance measures for the action items.  OMB agreed with our recommendations to complete action items and  provide deadlines, but disagreed with our recommendation to establish  performance measures. OMB has made progress with completing  incomplete action items, but more remains to be done. For example, one  action item involved issuing contracting guidance to support modular  development and OMB completed this action in June 2012. However,  another action item involved ensuring agency data center consolidation  plans were in place so agencies could close 800 data centers by 2015,  and we recently reported that all but 1 of the 24 agencies’ plans were  incomplete.
Most recently, in October 2012, we reported specifically on the CIO rating  portion of the IT Dashboard at six selected agencies. CIOs at six federal agencies rated the majority of their IT investments as  low risk, and that many ratings remained constant over time. For ratings  that did change, we found two agencies reported more investments with  reduced risk compared to earlier risk ratings; the other four agencies  reported more investments with increased risk. In addition, we reported  instances where the CIO ratings did not appropriately reflect significant  cost, schedule, and performance issues reported by GAO and others. We  recommended that the Federal CIO analyze agency trends reflected in  Dashboard CIO ratings, and report the results of this analysis in future  budget submissions. OMB agreed with the recommendation and recently  released some information on trends as part of the 2014 budget. We have  work under way to evaluate this information.
GAO, Information Technology Dashboard: Opportunities Exist to Improve Transparency  and Oversight of Investment Risk at Select Agencies, GAO-13-98 (Washington, D.C.: Oct.  16, 2012).
section_title: OMB and Selected Agencies Have Held Multiple TechStats on a Diverse Set of IT Investments, but Other At-Risk Investments Have Not Yet Been Assessed
paragraphs: OMB and the four selected agencies have held multiple TechStats on IT  investments that varied in terms of function, significance, amount spent to  date, and risk level. Specifically, from January 2010 through April 2013,  OMB reported leading 79 TechStat sessions, which focused on 55 IT  investments at 23 federal agencies. These investments covered 21  functional areas (such as information and technology management, law  enforcement, and health), and consist of 45 major, 8 non-major, and 2  unrated investments. For the 45 major investments that OMB reviewed,  almost 70 percent had a CIO rating of medium to high risk at the time of  the TechStat review.
The four agencies selected for our review—Agriculture, Commerce, DHS,  and HHS—held a total of 37 TechStat sessions covering 28 investments  from January 2011 through March 2013. Most of the investments  underwent a TechStat session at the agency level, but 8 underwent a  TechStat review at the bureau level, and 2 underwent TechStat reviews  at both. The agency-led TechStats covered 14 functional areas, and  consist of 21 major and 7 non-major investments. For the 21 major  investments, about 76 percent of the agency-led TechStats were on  investments that had a CIO rating of medium- to high-risk.
While both OMB and agencies have made progress in holding TechStat  sessions, there is more that could be done. Specifically, the number of  TechStats held to date is relatively small compared to the total number of  medium- and high-risk IT investments. Further, there are multiple high- risk IT investments spending millions of dollars that have not yet been  assessed. Two of the selected agencies—Agriculture and Commerce— had reviewed all of their high-risk investments, and DHS has plans to  review its remaining high-risk investments. However, HHS has not yet  established plans to review all of its high-risk investments. Also, OMB  does not have plans or schedules for assessing the other high-risk IT  investments. Until OMB and agencies intervene to turn around these at- risk projects, the government will continue to spend limited IT investment  dollars on underperforming projects.
subsections: section_title: OMB Reported Holding 79 TechStats for a Variety of IT Investments
paragraphs: As of April 2013, OMB reported conducting 79 TechStat reviews, with 59  reviews occurring in 2010, 8 in 2011, 11 in 2012, and 1 so far in 2013.  OMB conducted fewer TechStats in recent years because it expected the  agencies to increase the number of agency-led TechStats. Most of the  OMB-led TechStats were at DHS and DOD. OMB has conducted at least  one TechStat at 23 agencies (including the Office of the Director of  National Intelligence). It has not held a TechStat at the Department of  Labor, the National Aeronautics and Space Administration, the National  Science Foundation, the Nuclear Regulatory Commission, or the  Smithsonian Institution. Figure 1 shows the total number of reported  OMB-led TechStats at each agency as of April 2013.
These 79 TechStat reviews included 55 IT investments from 23 federal  agencies. OMB also reported leading follow-up sessions for many of  these investments. For example, OMB led 4 TechStats on a DHS  investment called the Federal Emergency Management Agency— National Flood Insurance Program Information Technology Systems and  Services. Table 2 identifies the investments for which OMB led  TechStats, the date of the first session, and the total number of sessions  that OMB has held on that investment.
These 55 investments span 21 primary functional areas.common functional area was information and technology management,  followed by homeland security and human resource management. Figure  2 provides a numerical depiction of the functions of those investments  that were subject to a TechStat.
OMB held most of its TechStats on major IT investments, with 45 of its 55  investments rated as major investments. OMB also held 8 TechStats on  non-major investments and 2 on investments lacking a major/non-major  designation. Figure 3 shows the breakdown of major and non-major  investments.
Thirty-eight of the 45 major investments that underwent an OMB-led  TechStat have reportedly cost the federal government about $16 billion  through fiscal year 2012. DHS’s investments that were the subject of a  TechStat session accounted for most of the cost, with a reported $4.9  billion spent. DOT follows with investments that have reportedly cost $2.8  billion through September 2012. The reported cost of individual  investments ranged from $4.18 million to $3.2 billion. Table 3 details how  much each agency has spent on its major investment.
Of the 45 major IT investments that were subject to a TechStat, 67  percent were considered medium- to high-risk investments at the time  they were chosen for a TechStat. Specifically, 9 had high- or moderately  high-risk (red) CIO ratings on the IT Dashboard, 21 had medium-risk  (yellow) ratings, and 9 had low- or moderately low-risk (green) ratings. In  addition, 6 investments’ risk levels were not identified. Table 4 shows the  Dashboard ratings for the major IT investments that underwent a  TechStat.
From January 2011 through March 2013, the four selected agencies—  Agriculture, Commerce, DHS, and HHS—held 37 TechStats that covered  28 different investments. Commerce held the most, with 15 TechStat  sessions, followed by DHS, HHS, and Agriculture, with 8, 7, and 7  sessions, respectively. Table 5 lists the investments, the date of the first  TechStat session, and the total number of TechStats by agency.
Although most of the agency-led TechStat reviews were held at the  agency level, three of the agencies in our review have also been holding  them at the bureau or component agency level. Specifically, of the  agencies in our review, 18 of the 28 investments underwent at least one  TechStat review at the agency level, 8 underwent a TechStat review at  the bureau level, and 2 underwent TechStat reviews at both. Of the  selected agencies, Agriculture had not held a bureau-level TechStat (see  fig. 4).
Like the OMB-led reviews, the investments selected by agencies for  TechStat reviews had diverse primary functional areas. Specifically, 25 of  the investments that underwent agency-led TechStats at the four selected  agencies cover 14 different functional areas. The most common area  was information and technology management (6 investments), followed  by environmental management (4 investments) (see fig. 5).
Most of the investments that underwent a TechStat review at the four  agencies were major investments, with a quarter of the sessions focusing  on non-major investments (see fig. 6).
The four agencies reported spending $7.8 billion through the end of fiscal  year 2012 on 20 of the 21 major investments that underwent an agency- led review.for a new Commerce investment to $3.7 billion for a DHS investment.  Table 6 provides a summary of the amount spent on major investments  through September 2012.
section_title: Selected Agencies are Generally Conducting TechStats in Accordance with OMB Guidance, but Areas for Improvement Exist
paragraphs: The four selected agencies are generally conducting TechStats in  accordance with OMB guidance. In 2011, when OMB decided to move  beyond conducting its own TechStats and to have agencies conduct them  too, OMB provided agencies guidance through the IT Reform Plan, a  memorandum, and the TechStat Toolkit, which is available on the CIO  Council’s website. These documents include 15 key requirements,  including when TechStats should be implemented by the agencies, what  participants should be included, how at-risk investments should be  chosen, and how outcomes should be tracked and reported. The  requirements can be grouped into four broad categories: scope,  governance, process, and outcomes. Table 10 shows key requirements  for TechStat reviews.
The four selected agencies implemented most of OMB’s 15 key  requirements. Specifically, DHS implemented all 15 requirements,  Commerce fully implemented 14 requirements and partially implemented  1 requirement, HHS fully implemented 11 of the requirements and  partially implemented 4 requirements, and Agriculture fully implemented  10 of the requirements, partially implemented 3 requirements, and did not  implement 2 requirements.
All four agencies fully implemented eight of OMB’s requirements: (1) the  appropriate people are invited to the TechStat meetings, (2) sessions are  led by the CIO, (3) the preparation for the TechStat is performed by the  TechStat team and the CIO, (4) investments selected by TechStats are  done so by the prescribed criteria, (5) all action items are tracked in a  consolidated repository, (6) each investment is tracked into an outcome,  (7) outcomes are shared with OMB, and (8) results are published in an  collaborative tool. The requirement with the least implementation involves  documenting action items, deadlines, and responsible parties in a  memorandum following each TechStat. Table 11 provides details  regarding the agencies’ implementation of OMB guidance on TechStats.
While the four selected agencies have largely implemented OMB’s  guidance on conducting TechStats, three agencies have selected areas  where they can improve. For example, Agriculture created memorandums  following a TechStat, but did not consistently include responsible parties;  Commerce created memorandums, but did not include deadlines; and  HHS did not always create memorandums or monitor all of its action  items to closure.
Agency officials noted several reasons for not fully implementing OMB’s  guidance. Specifically, Agriculture officials noted that they are updating  their capital planning guidance to include TechStat reviews, but that this  guidance is in the process of being reviewed and approved. Also,  Commerce and HHS officials noted that OMB gave agencies flexibility in  exactly how to implement their guidance. While OMB did provide  agencies flexibility in selecting investments for TechStat reviews and  conducting those reviews, the requirements are clearly delineated in OMB  instructions and training. Fully implementing OMB’s TechStat guidance  could better position the agencies to realize the benefits of the TechStat  initiative—including strengthening overall IT governance and oversight,  and proactively identifying and resolving problems before investments  experience delays or cost overruns.
section_title: OMB and Agencies Reported Positive Results and Cost Savings from Holding TechStats, but Are Not Ensuring Results Are Valid
paragraphs: The IT Reform Plan and related OMB guidance instructed agencies to  track and report on the outcomes of their TechStat sessions (including  improved governance, accelerated deliveries, and terminated projects),  and on any associated cost implications (such as cost savings or cost  avoidances).
OMB and the four agencies we reviewed have tracked and reported  positive results from TechStats, with most resulting in improved  governance or accelerated deliveries. OMB also reported that federal  agencies achieved over $3 billion in cost savings or avoidances as a  result of the OMB-led TechStats in 2010 and $900 million from agency- led TechStats in 2011. Using a different calculation formula, OMB also  reported that TechStats resulted in $63.5 million in cost implications in  2012.
However, we were unable to validate the reported outcomes and  associated savings because OMB did not provide supporting artifacts or  demonstrate the steps that OMB analysts took to verify the agencies’  data. Without documentation or an explanation of its method in validating  agencies’ reported results and cost savings, it will be difficult for OMB to  provide a sufficient level of confidence to Congress and the public that the  information it has presented is credible.
subsections: section_title: OMB and Agencies Reported Positive Results from TechStats
paragraphs: Both OMB and agencies reported achieving positive results from their  respective TechStat sessions. Specifically, in 2011 OMB staff reported  achieving a variety of positive outcomes from the OMB-led TechStat  sessions, including 11 investments being reduced in scope, four  investments that were cancelled, and multiple investments with  accelerated program delivery. We also identified four investments that  were the focus of an OMB-led TechStat that were subsequently  terminated and two other investments that were split into multiple smaller  investments to improve governance and accelerate the delivery of  discrete capabilities. Further, we previously found that the June 2010  TechStat on the National Archives and Records Administration’s  Electronic Records Archives investment resulted in six corrective actions,  including halting fiscal year 2012 development funding pending the  completion of a strategic plan.likely experience cost overruns of between $205 and $405 million if the  agency completed the program as originally designed.
We estimated that the program would  Seeking to improve the reporting of outcomes, OMB instructed federal  agencies to track the results of each agency-led TechStat session into  one of six outcomes: accelerated delivery, improved governance, reduced  scope, eliminated duplication, halted, or terminated. In December 2011,  OMB reported a summary of the outcomes of 294 agency-led TechStats  across the federal government. Specifically, OMB reported the following outcomes:
49 percent resulted in accelerated delivery,
42 percent resulted in improved governance,
3 percent did not report results,
2 percent resulted in terminations,
1 percent resulted in reducing the scope,
1 percent eliminated duplication, and
1 percent halted the investment.
In addition, the four agencies in our review reported on the results of their  agency-led TechStats, with the majority resulting in improved  governance. Specifically, out of 36 TechStat reviews, 28 resulted in  improved governance, 4 in accelerated delivery, 2 in terminations, and 2  in a reduced scope. Table 12 provides a summary of reported results, by  agency.
In conjunction with the reported outcomes of agency-led TechStats, OMB  instructed agencies to provide information on the cost implications of the  outcomes. These implications could include cost savings (a reduction in  actual expenditures) or cost avoidances (an action taken immediately that  will reduce costs in the future). In addition, OMB’s guidance on  performance reporting notes that performance data should be  appropriately accurate and reliable for their intended use. This guidance  further describes verification and validation techniques that OMB  encourages agencies to use in internal assessments, including ensuring  that supporting documentation is maintained and readily available, data  are verified as appropriate to the needed level of accuracy, and data  limitations are explained and documented.
OMB has reported cost savings and avoidances from OMB- and agency- led TechStat reviews. Specifically, OMB has reported that federal agencies achieved about $3 billion in cost savings or avoidances as a  result of the OMB-led TechStats held in 2010 and 2011, and $900 million  from agency-led TechStats in 2011. See figure 7 for the reported cost  implications for both OMB-led and agency-led TechStats as of November  2011.
In addition, OMB reported that the cost savings and avoidances for the  agency-led TechStat reviews, as of November 2011, came from 10 of the  27 agencies under their purview. Department of Transportation had the  most cost implications, with $510 million. See figure 8 for a depiction of  the cost implications by agency.
Two of the agencies in our review—DHS and HHS—reported $23 million  in cost implications from three agency-led TechStats:
DHS reported a $14 million cost avoidance after deciding to  decommission one part of the Federal Protective Service’s Risk  Assessment and Management Program;
HHS reported saving $8.2 million by reengineering redundant  business processes supporting the Food and Drug Administration’s  Mission Accomplishment and Regulatory Compliance Service project;  and
HHS reported $800,000 in cost implications associated with improving  governance and reducing the scope of the One Stop Service Solution  project (now called GovZone).
More recently, OMB has been reporting cost savings from agency-led  TechStat reviews in quarterly reports to Congress. From December  2011 through December 2012, OMB identified a total of $63.5 million in  cost implications from agency-led TechStats. OMB staff stated that they  are calculating the cost savings in these quarterly reports differently than  their prior reports on cost savings, and thus the costs should not be  compared.
section_title: OMB and Agencies Are Not Ensuring Reported Outcomes and Cost Implications Are Valid
paragraphs: When collecting data, it is important to have assurance that the data are  accurate. Best practices in implementing the Government Performance  and Results Act of 1993 emphasize the need for agencies, when  providing information, to explain the procedures used to verify or validate  their data. Specifically, agencies should ensure that reported data are  sufficiently complete, accurate, and consistent, and also identify any  significant data limitations. Explaining the limitations of the information  can provide a context for understanding and assessing the challenges  agencies face in gathering, processing, and analyzing needed data. Such  a presentation of data limitation can also help identify the actions needed  to improve the agency’s ability to measure its performance. More  recently, we have reiterated the importance of providing OMB with  complete and accurate data.
DHS and HHS, the two agencies in our review that reported cost savings  and avoidances, generally perform analyses to establish these estimates.  DHS provided supporting documentation for how the cost implications for  its investment were calculated and HHS provided supporting  documentation for its larger investment. We were unable to validate the  cost savings for HHS’s smaller investment because the agency was  unable to provide supporting documentation.
From a governmentwide perspective, OMB staff explained that they  review agencies’ cost implication data for completeness and quality.  However, we were unable to validate OMB’s reported outcomes and  almost $4 billion in cost implications because OMB did not provide  documentation on any steps that it or the agencies took to ensure the  validity of the agencies’ outcome and cost data. For example, OMB staff  did not provide memorandums documenting action items from the OMB- led TechStats, the outcomes identified for OMB- and agency-led  TechStats, documentation identifying which investments resulted in cost  savings, documentation demonstrating the methodology used to calculate  the cost savings, or a summary of steps the agencies took to validate  reported cost savings. Moreover, OMB does not require agencies to  report on what steps they took to verify their reported outcomes and cost  savings.
By not requiring agencies to report on their efforts to validate reported  outcomes and cost savings, it is not evident that OMB is following its own  guidance for ensuring that performance data are reliable and accurate.  Moreover, OMB is not providing reasonable assurance to Congress and  the public that the information it has presented is credible.
One entity that was formed to assist OMB in its oversight of the TechStat  results was the CIO Council’s subcommittee on IT Governance and  TechStats, but according to OMB and agency officials this subcommittee  has already been dissolved. The CIO Council’s Management Best  Practices Committee formed the subcommittee to report on the outcomes  and lessons learned from implementing the TechStat process. This  subcommittee was to play a key role in continuing to mature the TechStat  process. However, in 2012, the CIO Council dissolved the subcommittee  because the committee chairs determined that it was no longer needed.
section_title: Conclusions
paragraphs: Consistent with government and industry best practices for overseeing IT  investments, TechStat sessions hold value by focusing management  attention on troubled projects and establishing clear action items to turn  the projects around or terminate them. While OMB and agencies are  reporting positive results from holding TechStat sessions, neither are  doing enough to ensure that at-risk investments are undergoing review,  sound processes are in place, and reported results are valid. Specifically,  agencies are reviewing only about a third of their at-risk IT investments.  Until OMB and agencies develop plans and schedules for addressing  these at-risk investments, the investments will likely remain at risk.
The four agencies we reviewed had implemented most of the OMB- required TechStat processes, but three had shortfalls in selected  processes. For example, Agriculture had not yet incorporated TechStats  in its investment management processes, Commerce had not consistently  included deadlines for action items in its TechStat memoranda, and HHS  had not consistently created action item memoranda following TechStats  or tracked its action items to completion. Addressing these shortfalls  could better position these agencies to realize the full benefits that  TechStats offer.
OMB regularly reports on cost savings associated with TechStats, but it  has not taken basic steps to provide reasonable assurance to Congress  and the public that these data are valid. Until OMB requires agencies to  report on what they did to validate cost savings data and shares this  information, neither Congress nor the public can be assured that  TechStats are as effective as reported. While the CIO Council recently  dissolved its subcommittee responsible for reviewing TechStats, the  council is one entity that is uniquely positioned to assist OMB in its  oversight of the TechStat results.
section_title: Recommendations for Executive Action
paragraphs: To ensure that TechStat sessions are having the appropriate impact in  the oversight of underperforming projects, we are making three  recommendations to OMB. Specifically, we recommend that the Director  of the Office of Management and Budget direct the Federal Chief  Information Officer to  require agencies to conduct TechStats for each IT investment rated  with a moderately high- or high-risk CIO rating on the IT Dashboard,  unless there is a clear reason for not doing so; require agencies to report to OMB on efforts to validate the outcomes,  cost savings, and cost avoidances resulting from TechStat   sessions; this information should be summarized when OMB reports  on governmentwide outcomes; and direct the Federal CIO Council to track the outcome of TechStat  sessions and to support OMB’s efforts to validate the resulting cost  savings it reports to Congress.
In addition, we are making a recommendation to the Secretaries of  Agriculture and Commerce to address the weaknesses in agency- and bureau-led TechStat  processes and management outlined in this report.
We are also making two recommendations to the Secretary of Health and  Human Services to establish a plan and schedule for addressing each IT investment rated  with a moderately high- or high-risk CIO rating on the IT Dashboard;  such a plan could include conducting a TechStat session, and address the weaknesses in agency- and bureau-led TechStat  processes and management outlined in this report.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments from OMB, Agriculture, Commerce, DHS, and  HHS on a draft of our report. In that draft report, we had made  recommendations to the Secretaries of Agriculture, Commerce, and DHS  to establish a plan and schedule for addressing their high-risk  investments that had not yet undergone TechStat reviews. However,  these agencies completed these steps prior to the issuance of this report.  Therefore, we removed the applicable recommendations.
OMB and three agencies provided comments on our draft report. OMB  generally agreed with our recommendations; Commerce agreed with our  recommendation; and Agriculture and HHS did not agree or disagree with  our recommendations. DHS declined to provide comments. Each  agency’s comments are discussed in more detail below:  In comments provided via e-mail, staff from OMB’s Office of E- Government and Information Technology generally concurred with our  findings and recommendations and stated that OMB and the agencies  are currently taking appropriate steps to meet the recommendations.  OMB provided additional details as follows:  In commenting on our finding that it did not ensure the validity of  the outcomes and savings it reported, OMB noted that it is  confident in the agency validation of cost-savings and avoidances  through the TechStat sessions, and that it was speculative to  conclude that the reported cost savings data were not valid  because we could not actually assess these figures. OMB did not  provide supporting documentation for the cost savings  calculations and explained that the process used to validate the  cost figures was deliberative. However, part of a sound process  for ensuring validity involves documenting the procedures used to  verify or validate the data; as such, we were unable to validate  OMB’s reported outcomes and almost $4 billion in cost  implications from 2010 and 2011 because OMB did not provide  documentation on any steps that it or the agencies took to ensure  the validity of the agencies’ outcome and cost data. For example,  OMB staff did not provide memorandums documenting action  items from the OMB-led TechStats, the outcomes identified for  OMB- and agency-led TechStats, documentation identifying which  investments resulted in cost savings, documentation  demonstrating the methodology used to calculate the cost  savings, or a summary of steps the agencies took to validate  reported cost savings. Moreover, OMB did not require agencies to  report on what steps they took to verify their reported outcomes  and cost savings. Without disclosing the steps it took to validate  the outcomes and cost savings, OMB has not provided  reasonable assurance to Congress and the public that the  information it has presented is credible.
OMB also noted that since June 2012, OMB and agencies have  validated and transmitted cost-savings and avoidance data  associated with TechStat sessions to Congress on a quarterly  basis through the Integrated, Efficient, and Effective Use of IT  Report. These reports summarize the validation methods OMB  and agencies undertake. We acknowledge that these reports  summarize methods through which the costs savings and  avoidances were documented; however, due to a lack of  supporting documentation, we could only validate $22.2 million of  the $63.5 million cost savings and avoidances OMB recently  reported.
In written comments, the Department of Agriculture’s Chief  Information Officer partially agreed with our assessment of the  agency’s TechStat process but did not specify the part of the  assessment with which she disagreed. The department stated that it  would continue to conduct periodic reviews of its IT investments to  ensure a review of all major IT investments. In addition, the  department stated that IT Dashboard ratings alone are not true  indicators that an investment is poorly performing or underperforming  and that it takes into consideration other factors, such as earned value  management data, when determining an investment’s performance.  We agree that the CIO rating should reflect the CIO’s assessment of  the risk and should use many factors in determining the rating. The  department’s written comments are provided in appendix II.
In written comments, the Department of Commerce’s Acting Secretary  concurred with our recommendation and stated that the department is  in full compliance with the recommendation. The department plans to  ensure that all TechStat memos have a specific point of contact and a  specific due date. The department’s written comments are provided in  appendix III.
In comments provided via e-mail, a management analyst within the  Department of Health and Human Services’ Office of the Assistant  Secretary for Legislation stated that the department had no general  comments. The department provided technical comments, which we  have incorporated as appropriate. In its technical comments, HHS  commented on our finding that the agency did not satisfy OMB’s  requirement to hold its first TechStat session by March 2011. The  HHS officials stated that the agency had appropriately initiated the  process for holding a TechStat prior to OMB’s March 2011 deadline.  While we acknowledge that the agency initiated the process prior to  the deadline, the actual session was held after the deadline.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or at pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) identify key characteristics of TechStat  Accountability Sessions (TechStats) conducted by the Office of  Management and Budget (OMB) and selected agencies, (2) evaluate  whether selected agencies are conducting TechStats in accordance with  OMB guidance, and (3) analyze the extent to which reported results from  TechStat review sessions are documented, tracked, and validated.
In conducting our review, we selected four agencies based on two  factors: the number of investments with medium-, moderately high-, and  high-risk Chief Information Officer (CIO) ratings on the information  technology (IT) Dashboard as of August 2012, and a determination on  whether the agency had led its own TechStats in 2011. The four agencies  we selected—the Departments of Agriculture (Agriculture), Commerce  (Commerce), Health and Human Services (HHS), and Homeland Security  (DHS)—had the highest number of medium-, moderately high-, and high- risk investments. In addition, in December 2011, the CIO Council reported   We  that all four agencies had held their own TechStat sessions in 2011.assessed the reliability of the IT Dashboard data and the data on agency- led TechStats by comparing them to agency documents, and found that  the data were sufficient for our purpose of selecting agencies for further  review.
We reviewed exhibit 300 and 53 data linked on the IT Dashboard as of the last update in  August 2012. data for each of these investments. We also interviewed OMB,  Agriculture, Commerce, DHS, and HHS officials regarding the TechStat  sessions. We assessed the reliability of OMB’s and the four agencies’  lists of reported TechStat sessions by seeking corroboration for the  sessions held at our selected agencies. We determined that OMB’s and  the four agencies’ lists were sufficient for our purposes.
To address the second objective, we analyzed OMB’s guidance to  agencies on conducting TechStat reviews and identified15 key  requirements. We then compared these requirements to the selected  agencies’ documentation, including capital planning and investment  control plans and guidance, agency-specific TechStat guidance and  training materials, meeting minutes from investment review board  meetings, and electronic submissions from the agency to OMB regarding  the TechStat sessions. To identify whether agencies were documenting,  tracking, and monitoring action items—one of the 15 key requirements— we conducted a random sample of 10 percent of each agency’s action  items and sought documentation from each of the agencies to support the  status of the selected action items. This sample is not generalizable to the  entire population of action items from each agency, but we determined it  was sufficiently reliable for our purposes of determining whether the  agencies were tracking and monitoring action items. We also interviewed  officials from OMB and our four selected agencies about their respective  TechStat processes.
To address the third objective, we identified OMB’s guidance to agencies  on reporting the results of agency-led TechStat sessions. We analyzed  the selected agencies’ and OMB’s reported results in each of OMB’s  outcome categories and attempted to validate OMB’s and the agencies’  reported cost implications. We also reviewed quarterly reports from OMB  to Congress to identify additional cost implication information. As noted  in the body of this report, we were unable to determine the reliability of  the reported cost implications because OMB did not provide artifacts  demonstrating how it validated the data or evidence that it obtained  agency validation of the data.
We conducted this performance audit from September 2012 to June 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Agriculture
section_title: Appendix III: Comments from the Department of Commerce
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Colleen Phillips (Assistant Director), Rebecca  Eyler, Kate Feild, and Jessica Waselkow.