IT Dashboard: Agencies Are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available
Published: 2013-12-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: OMB launched the Dashboard in June 2009 as a public Web site that reports performance for major IT investments--on which the federal government plans to invest over $38 billion in fiscal year 2014. The Dashboard is to provide transparency for these investments and to facilitate public monitoring of them. After its launch, OMB began using it to identify at-risk investments.
This report (1) characterizes the CIO ratings for selected federal agencies' IT investments as reported on the Dashboard over time, (2) determines the extent to which selected agencies' CIO ratings are consistent with investment risk, and (3) determines the extent to which selected agencies are addressing at-risk investments. GAO selected the eight agencies with the most reported major IT spending in fiscal year 2012 (excluding those GAO recently reviewed) and selected 10 investments at each agency. GAO reviewed the investments' documentation, compared it to the CIO ratings, and reviewed processes used for the highest-risk investments. GAO also interviewed appropriate officials.
section_title: What GAO Found
paragraphs: As of August 2013, the Chief Information Officers (CIO) at the eight selected agencies rated 198 of their 244 major information technology (IT) investments listed on the Federal IT Dashboard (Dashboard) as low risk or moderately low risk, 41 as medium risk, and 5 as high risk or moderately high risk. However, the total number of investments reported by these agencies has varied over time, which impacts the number of investments receiving CIO ratings. For example, Energy reclassified several of its supercomputer investments from IT to facilities and Commerce decided to reclassify its satellite ground system investments. Both decisions resulted in the removal of the investments from the Dashboard, even though the investments were clearly IT. In addition, the Office of Management and Budget (OMB) does not update the public version of the Dashboard as the President's budget request is being created. As a result, the public version of the Dashboard was not updated for 15 of the past 24 months, and so was not available as a tool for investment oversight and decision making.
Of the 80 investments reviewed, 53 of the CIO ratings were consistent with the investment risk, 20 were partially consistent, and 7 were inconsistent. While two agencies' CIO ratings were entirely consistent, other agencies' ratings were inconsistent for a variety of reasons, including delays in updating the Dashboard and how investment performance was tracked. For example, the Department of Justice downgraded an investment in July 2012, but the Dashboard was not updated to reflect this until April 2013. Further, the Social Security Administration resets investment cost and schedule performance baselines annually, an approach that increases the risk of undetected cost or schedule variances that will impact investment success.
Of the eight investments that were at highest risk in 2012, seven were reviewed by their agencies using tools such as TechStat sessions--evidence-based reviews intended to improve investment performance and other high-level reviews. Each of these resulted in action items intended to improve performance. The final investment was scheduled to have a TechStat, but instead, according to department officials, a decision was made to modify its program cost and schedule commitments to better reflect the investment's actual performance.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB make Dashboard information available independent of the budget process, and that agencies appropriately categorize IT investments and address identified weaknesses. OMB neither agreed nor disagreed. Six agencies generally agreed with the report or had no comments and two others did not agree, believing their categorizations were appropriate. GAO continues to believe its recommendations remain valid, as discussed.

section_title: Letter
paragraphs: In accordance with its responsibilities set forth in law, the Office of  Management and Budget (OMB) launched the Federal Information  Technology (IT) Dashboard in June 2009 as a public Web site that  reports performance and supporting data for major IT investments—on  which the federal government plans to invest over $38 billion in fiscal year  2014. The Dashboard is to provide transparency for these investments in  order to facilitate public monitoring of government operations and  accountability for investment performance by the federal Chief  Information Officers (CIO) who oversee them. According to OMB, it began  using the Dashboard to identify at-risk investments with its launch in June  2009. These investments became the focus of joint OMB-agency  TechStat Accountability Sessions (TechStats)—evidence-based reviews  intended to improve investment performance through concrete actions.
This report responds to your request that we review the consistency of  investment risk information submitted to the Dashboard. Our objectives  for this engagement were to (1) characterize the CIO ratings for selected  federal agencies’ IT investments as reported on the Dashboard over time,  (2) determine the extent to which selected agencies’ CIO ratings are  consistent with reported investment risk, and (3) determine the extent to  which selected agencies are addressing at-risk investments.
To address our first objective, we selected the eight agencies with the  most reported major IT spending in fiscal year 2012, after excluding  agencies reviewed in our most recent report on the Dashboard.reviewed publicly available data from the Dashboard from its inception in  June 2009 through August 2013. We also interviewed officials from OMB  and the selected agencies.
To accomplish the second objective, we selected the top 10 major  investments with the highest reported IT spending at each of the 8  selected agencies (for a total of 80 investments). We reviewed investment  documentation (such as executive-level briefings, operational analyses,  and the results of performance reviews) and interviewed officials from  each of the agencies to understand how they rated and monitored  investments. We compared these documents to the CIO ratings  submitted to the Dashboard in calendar year 2012 to determine whether  the ratings on the Dashboard were accurately portraying the risk of these  investments. We elected to review the investments’ ratings during  calendar year 2012 so that our assessment would span multiple fiscal  years.
Finally, for our third objective, we reviewed and described each of the  processes used by the eight selected agencies to address the highest- risk major investments (such as TechStat guides and investment review  board results). We also identified investments within the 80 investments  selected for the second objective which had received high or moderately  high ratings of risk on the Dashboard during 2012. We analyzed both the  processes used to address these investments and the results of  performance reviews. We also interviewed officials to determine whether  the agencies implemented the appropriate risk management processes to  oversee and review the selected investments.
We conducted this performance audit from February to December 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Further details of our  objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: While IT can enrich people’s lives and improve organizational  performance, we have previously reported that federal IT projects too  frequently incur cost overruns and schedule slippages while contributing  little to mission-related outcomes. For example, in July 2013, we testified  that the federal government continued to spend billions of dollars on  troubled IT investments and we identified billions of dollars worth of failed  and challenged IT projects.
OMB plays a key role in overseeing how federal agencies manage their  IT investments by working with them to better plan, justify, and determine  how to manage them. Each year, OMB and federal agencies work  together to determine how much the government plans to spend on IT  projects and how these funds are to be allocated. OMB also guides  agencies in developing sound business cases for IT investments and  establishing management processes for overseeing these investments  throughout their life cycles. The scope of this undertaking is quite large: in  planning for fiscal year 2014, 27 federal agencies reported plans to spend  about $82 billion on IT.investments, $37.6 billion is to be spent on non-major IT investments, and  $5.5 billion is to be spent on classified Department of Defense IT  investments.
Of that, $38.7 billion is to be spent on major IT  Within OMB, the Office of E-Government and Information Technology,  headed by the Federal CIO, directs the policy and strategic planning of  federal IT investments and is responsible for oversight of federal  information technology spending.
subsections: section_title: OMB’s Dashboard Is Intended to Provide Visibility into the Performance of Federal IT Investments
paragraphs: In June 2009, OMB deployed a public Web site to further improve the  transparency and oversight of agencies’ IT investments. Known as the IT  Dashboard,performance data for over 700 major federal IT investments at 27 federal  agencies, accounting for $38.7 billion of those agencies’ planned $82  billion budget for fiscal year 2014. According to OMB, these data are  intended to provide a near-real-time perspective on the performance of  these investments, as well as a historical perspective. Further, the public  display of these data is intended to allow OMB; other oversight bodies,  including Congress; and the general public to hold the government  agencies accountable for progress and results.   this site displays federal agencies’ cost, schedule, and  The Dashboard visually presents performance ratings for agencies and  for individual investments using metrics that OMB has defined—cost,  schedule, and CIO evaluation. The Web site also provides the capability  to download certain data. Figure 1 is an example of VA’s portfolio page as  recently depicted on the Dashboard. (The ratings are explained in the  narrative following the figure.)
The Dashboard’s data spans the period from its June 2009 inception to  the present, and is based, in part, on agency assessments of individual  investment performance and each agency’s budget submissions to  OMB.
The cost and schedule data agencies are required to submit to the  Dashboard have changed over time, as have the related calculations. For  example, in response to our recommendations (further discussed in the  following section), OMB changed how the Dashboard calculates the cost  and schedule ratings in July 2010 to include “in progress” milestones  rather than just “completed” ones for a more accurate reflection of current  investment status. While the required cost and schedule data have  changed, the thresholds for assigning cost and schedule ratings have  remained constant over the life of the Dashboard. Specifically, the  Dashboard assigns rating colors (red, yellow, green) based on agency- submitted cost and schedule variances, as shown in table 1.
Unlike the variance-based cost and schedule ratings, the Dashboard’s  “Investment Evaluation by Agency CIO” (also called the CIO rating) is  determined by agency officials. According to OMB’s instructions, each  agency CIO is to assess his or her IT investments against a set of six pre- established evaluation factors identified by OMB (shown in table 2) and  then assign a rating of 1 (high risk) to 5 (low risk) based on the CIO’s best  judgment of the level of risk facing the investment.
OMB recommends that CIOs consult with appropriate stakeholders in  making their evaluation, including Chief Acquisition Officers, program  managers, and other interested parties. According to an OMB staff  member, agency CIOs are responsible for determining appropriate  thresholds for the risk levels and for applying them to investments when  assigning CIO ratings. OMB requires agencies to update these ratings as  soon as new information becomes available which will affect an  investment’s assessment. After agencies assign a level of risk to each  investment, the Dashboard assigns colors to CIO ratings according to a  five-point scale, as illustrated in table 3.
An OMB staff member from the Office of E-Government and Information  Technology noted that the CIO rating should be a current assessment of  future performance based on historical results and is the only Dashboard  performance indicator that has been defined and produced the same way  since the Dashboard’s inception.
Furthermore, OMB issued guidance in August 2011other things, that agency CIOs shall be held accountable for the  performance of IT program managers based on their governance process  and the data reported on the IT Dashboard, which includes the CIO  rating. According to OMB, the addition of CIO names and photos on  Dashboard investments is intended to highlight this accountability and link  it to the Dashboard’s reporting on investment performance.
section_title: GAO Has Previously Reported on the Dashboard’s Value, Data Quality, and Improvements
paragraphs: We have previously reported that OMB has taken significant steps to  enhance the oversight, transparency, and accountability of federal IT  investments by creating its IT Dashboard, and by improving the accuracy  of investment ratings. We also found issues with the accuracy and data  reliability of cost and schedule data, and recommended steps that OMB  should take to improve these data.
In July 2010, we reportedOMB’s Dashboard were not always accurate for the investments we   that the cost and schedule ratings on  reviewed, because these ratings did not take into consideration  current performance. As a result, the ratings were based on outdated  information. We recommended that OMB report on its planned  changes to the Dashboard to improve the accuracy of performance  information and provide guidance to agencies to standardize  milestone reporting. OMB agreed with our recommendations and, as  a result, updated the Dashboard’s cost and schedule calculations to  include both ongoing and completed activities.
Similarly, our report in March 2011 noted that OMB had initiated  several efforts to increase the Dashboard’s value as an oversight tool,  and had used its data to improve federal IT management. We also  reported, however, that agency practices and the Dashboard’s  calculations contributed to inaccuracies in the reported investment  performance data. For instance, we found missing data submissions  or erroneous data at each of the five agencies we reviewed, along  with instances of inconsistent program baselines and unreliable  source data. As a result, we recommended that the agencies take  steps to improve the accuracy and reliability of their Dashboard  information, and that OMB improve how it rates investments relative  to current performance and schedule variance. Most agencies  generally concurred with our recommendations and three have taken  steps to address our recommendation. OMB agreed with our  recommendation for improving ratings for schedule variance. It  disagreed with our recommendation to improve how it reflects current  performance in cost and schedule ratings, but more recently made  changes to Dashboard calculations to address this while also noting  challenges in comprehensively evaluating cost and schedule data for  these investments.
We also reported on OMB’s guidance to agencies for reporting their IT  investments in September 2011 and found that it did not ensure  complete reporting. Specifically, agencies differed on what investments they included as an IT investment. Among other things,  we recommended that OMB clarify its guidance on reporting IT  investments to specify whether certain types of systems—such as  space systems—are to be included. OMB did not agree that further  efforts were needed to clarify reporting in regard to the types of  systems.
Subsequently, in November 2011, we noted that the accuracy of  investment cost and schedule ratings had improved since our July  2010 report because OMB had refined the Dashboard’s cost and  schedule calculations. Most of the ratings for the eight investments we  reviewed as part of our November 2011 report were accurate,  although we noted that more could be done to inform oversight and  decision making by emphasizing recent performance in the ratings.  We recommended that the General Services Administration comply  with OMB’s guidance for updating its ratings when new information  becomes available (including when investments are rebaselined). The  agency concurred and has since taken actions to address this  recommendation. Since we previously recommended that OMB  improve how it rates investments, we did not make any further  recommendations.
More recently, in October 2012 we reported that CIOs at six agencies  rated a majority of investments listed on the federal IT Dashboard as  low or moderately low risk from June 2009 through March 2012. Additionally, two agencies, the Department of Defense and the  National Science Foundation, rated no investments as high or  moderately high risk during this time period. Agencies generally  followed OMB’s instructions for assigning CIO ratings, although the  Department of Defense’s ratings were unique in reflecting additional  considerations, such as the likelihood of OMB review. Most of the  selected agencies reported various benefits associated with producing  and reporting CIO ratings, such as increased quality of their  performance data and greater transparency and visibility of  investments. We recommended that OMB analyze agencies’  investment risk over time as reflected in the Dashboard’s CIO ratings  and present its analysis with the President’s annual budget  submission, and that the Department of Defense ensure that its CIO  ratings reflect available investment performance assessments and its  risk management guidance. Both agencies concurred with our  recommendations, and OMB reported on CIO rating trends in the  fiscal year 2014 budget submission; however, the Department of  Defense has not updated any of its ratings since September 2012.
Further, we studied OMB’s efforts to help agencies address IT  projects with cost overruns, schedule delays, and performance  shortfalls in June 2013. In particular, we reported that OMB used  CIO ratings from the Dashboard, among other sources, to select at- risk investments for reviews known as TechStats. OMB initiated  these reviews in January 2010 to further improve investment  performance, and subsequently incorporated the TechStat model into  its 25-point plan for reforming federal IT management. We reported  that OMB and selected agencies had held multiple TechStat sessions  but additional OMB oversight was needed to ensure that these  meetings were having the appropriate impact on underperforming  projects and that resulting cost savings were valid. Among other  things, we recommended that OMB require agencies to address their  highest-risk investments and to report on how they validated the  outcomes. OMB generally agreed with our recommendations, and  stated that it and the agencies were taking appropriate steps to  address them.
section_title: CIOs at Selected Agencies’ Rated Majority of IT Investments as Low or Moderately Low Risk
paragraphs: As of August 2013, CIOs at the eight selected agencies rated 198 of their  244 major investments listed on the Dashboard as low risk or moderately  low risk. Of the remaining 46 investments, 41 were rated medium risk,  and 5 were rated high risk or moderately high risk. Historically, over the  life of the Dashboard from June 2009 to August 2013, low or moderately  low risk investments accounted for an average of 72 percent of all ratings  at the eight agencies, medium risk ratings an average of 22 percent, and  high risk or moderately high risk ratings an average of 6 percent. The  CIO ratings and associated number of investments at each the eight  agencies as reported on the Dashboard over the past 4 years are  presented in figure 2.
When comparing the investments’ first and most recent ratings, the  agencies we reviewed showed generally positive results. Specifically, of  the 383 total investments listed on the Dashboard from June 2009 to  August 2013 for the selected agencies, 121 increased (meaning risk is  lower) and 81 decreased (meaning risk is higher). Additionally, a  significant portion of investments had returned to their original rating (74)  and about one third of investments’ ratings had never changed (107).  (See fig. 3.)
When considered individually, five of the eight selected agencies— Agriculture, Commerce, Energy, Treasury, and VA—had more  investments’ ratings increase than decrease, whereas the other three— Justice, Transportation, and SSA—had more investments decrease.
Figure 4 presents the net changes in investments’ ratings for each  agency during the reporting period of June 2009 to August 2013.
The agencies cited additional oversight or program reviews as factors that  contributed to improved ratings. Furthermore, Agriculture and Commerce  both attributed improved ratings to enhanced timeliness and sufficiency of  investment reporting. Both of these agencies factor investment reporting  into CIO ratings and increased ratings as better performance and risk  information was provided by investment management. This suggests that  caution should be exercised when interpreting changing risk levels for  investments, as rating changes by agencies may not be entirely due to  investment performance. Agencies also commented that the CIO ratings  and Dashboard reporting had spurred improved investment management  and risk mitigation.
Additionally, the total number of investments that the eight agencies  reported on the Dashboard has varied over time, which impacts the  number of investments receiving CIO ratings (see fig. 5).
The variation in the number of investments reported on the Dashboard  can be attributed, in part, to agencies’ ability to add, downgrade, and  remove investments throughout the annual federal budget process.  However, as we concluded in September 2011, OMB’s guidance also did  not ensure complete reporting of IT investments by agencies.  Specifically, we found that OMB’s definition of an IT investment is broad,  and the 10 agencies we evaluated differed on what systems they included  as IT investments. For example, 5 of the 10 agencies we reviewed  consistently considered investments in research and development  systems as IT, and 5 did not. Consequently, we recommended that OMB  clarify its guidance on reporting IT investments to specify whether certain  types of systems—such as space systems—were to be included. OMB  did not agree that further efforts were needed to clarify reporting in regard  to the types of systems.
Now, 2 years later, and given the continuing lack of clarity in investment  reporting guidance, agencies have elected to withdraw investments from  the Dashboard that are clearly IT. Because we have continued to identify  inappropriate reclassifications of IT investments, we continue to believe  this recommendation has merit. For example, as part of the most recent  budget process, Energy officials reported several of Energy’s  supercomputer investments as facilities rather than IT, thus removing  those investments from the Dashboard and accounting for a portion of the  recent decrease in investments. Energy officials also stated that OMB  approved this change. These investments account for $368 million, or  almost 25 percent, of Energy’s fiscal year 2012 IT spending, and include  the National Nuclear Security Administration’s Sequoia system, which  Energy reported as the most powerful computing system in the world as  of June 2012. According to Energy officials, these investments were  recategorized because they include both supercomputers and laboratory  facilities (which are not IT).
As another example, the Deputy Secretary of Commerce issued a  directive in September 2012 which resulted in the removal of Commerce’s  satellite investments from the Dashboard. As we found in 2011,  Commerce only reported the ground systems of a spacecraft as IT  investments, and not the technology components on the spacecraft  itself. With this directive, Commerce removed the ground-based  portions from IT investment reporting as well, accounting for $447 million,  or 26 percent, of the department’s fiscal year 2012 IT spending. In making  this decision, Commerce determined that it needed to refocus oversight  efforts to a more appropriate level and consequently minimized the role of  the CIO and others in the oversight of satellites. A Commerce official  stated that Commerce plans to exclude all such investments from the  department’s fiscal year 2015 IT budget submission.
Clinger-Cohen Act of 1996 (40 U.S.C. § 11101(6)). and transmit satellite data. A staff member from the Office of E- Government stated that OMB could not stop agencies from making such  recategorizations and that OMB had no control over such agency  decisions. By gathering incomplete information on IT investments, OMB  increases the risk of not fulfilling its oversight responsibilities, of agencies  making inefficient and ineffective investment decisions, and of Congress  and the public being misinformed as to the performance of federal IT  investments.
As part of the budget process, OMB is required to analyze, track, and  evaluate the risks and results of all major IT investments. The  Dashboard gives the public access to the same tools and analysis that  the government uses in conducting this performance oversight. According  to OMB, Dashboard data are intended to provide a near-real-time  perspective on the performance of major investments. http://www.itdashboard.gov/faq. year, OMB decreases the utility of the Dashboard as a tool for greater IT  investment oversight and transparency.
section_title: Most CIO Ratings Are Consistent with Reported Investment Risk
paragraphs: Of the 80 selected investments at the eight agencies we reviewed, 53 of  the CIO ratings were consistent with the risks portrayed in the supporting  investment performance documents, 20 were partially consistent, and 7  VA investments were inconsistent. Those that were partially consistent  had few instances of discrepancies during the 12-month period. For  example, both of Commerce’s inconsistent investments had a  discrepancy during 1 of the 12 months which we reviewed. As previously  mentioned, a CIO rating should reflect the level of risk facing an  investment on a scale from 1 (high risk) to 5 (low risk) relative to that  investment’s ability to accomplish its goals. However, seven of the eight  agencies we reviewed had at least one instance wherein the Dashboard’s  CIO ratings did not consistently reflect the risks identified in agency  documents. Table 4 summarizes our assessment of the 10 investments at  each of the selected agencies during the 12-month period from January  2012 through December 2012 and a discussion of the analysis of each  agency’s documentation follows the table.
Agriculture: Eight of the 10 investments at Agriculture had some  inconsistencies with reported risks. In particular, Agriculture  experienced technical issues uploading its ratings to the Dashboard in  early 2012, which impacted the ratings of six investments. Agriculture  officials identified an issue with these investments after observing that  submitted changes were not reflected on the Dashboard. They  addressed this issue by working with their contractor to determine the  cause of the issue, modifying their process to prevent the issue from  recurring, and establishing procedures to identify future technical  issues. Additionally, Agriculture’s Financial Management  Modernization Initiative—one of the six investments that experienced  the technical issue described above—and Human Resources Line of  Business’s October ratings should have been lower (higher risk) in  October 2012. Officials stated that these CIO ratings were updated at  the beginning of November 2012 and attributed the delays in the  rating submissions to the annual budget process.
Additionally, Agriculture’s CIO rated the Farm Program Modernization  investment as moderately low risk until November 2012, despite the  investment showing indications of significantly increased risk as early  as May 2012. For example, the number of high and very high risks  increased every month in a series of briefings from May to July 2012,  at which point there was 1 high-impact risk and 2 very high-impact  risks each with an estimated 70 percent probability of occurring.  Further, in September 2012 a senior management oversight  committee determined the investment to be off-track and unable to  resolve the issues. However, the investment remained rated on the  Dashboard as moderately low risk until November 2012, when it was  updated to medium risk. As a result, the CIO rating did not reflect the  results of the oversight committee’s review for 3 months. The  investment was ultimately given a high-risk rating in December 2012.
Commerce: Two of the 10 selected Commerce investments differed  from risk levels identified in underlying documentation for 1 month  each. In particular, Commerce’s Census - Economic Census and  Surveys investment should have been green rather than yellow in  March 2012, and its National Oceanic and Atmospheric Administration  IT Infrastructure should have been yellow rather than green in June  2012. In both cases Commerce officials recognized that there were  problems and took steps to correct them the following month.
Energy: Five of the 10 selected Energy investments remained listed  on the Dashboard after the department had recategorized them as  investment types that would have no longer been displayed.  Specifically, in October 2012, as part of its annual budget process,  Energy downgraded its SR Mission Support Systems from major to  non-major and, as discussed previously, changed four of the selected  supercomputer investments from IT to non-IT. An SR Mission Support  Systems official explained that Energy downgraded the investment  after realizing that significant portions of it could be interpreted as  non-IT under the Federal Acquisition Regulation, and the remainder  did not meet the department’s threshold for being a major IT  investment. Further, although Energy made these decisions in  October 2012, OMB officials explained that they do not publish  budgetary data such as these until the release of the President’s  budget, which did not happen until April 2013 at which time the  Dashboard reflected Energy’s changes.
Justice: One of the 10 selected Justice investments remained on the  Dashboard after the department recategorized it as an investment  type that would have no longer been displayed, similar to Energy’s  situation. Specifically, Justice downgraded its Law Enforcement  Wireless Communications investment from major to non-major as part  of its annual budget process in July 2012, but the Dashboard did not  reflect the change until April 2013. As noted above, OMB officials  explained that they do not publish budgetary data until the release of  the President’s budget, which occurred in April 2013.
Transportation: There were no inconsistencies between CIO ratings  and reported investments’ risks. Continuing to report consistent and  timely data should help ensure that Transportation’s Dashboard’s CIO  ratings are accurate.
Treasury: We did not identify inconsistencies between CIO ratings  and reported investments’ risks. Such attention to reporting consistent  and timely data should help ensure the accuracy of the Dashboard’s  CIO ratings.
VA: Seven of the 10 selected investments were never updated during  our evaluation period, and 3 were updated once. VA did not update its  ratings because it did not have the ability to automatically submit  ratings from September 2011 to March 2013, a period of 19 months.  Instead, VA elected to build the capability to submit ratings to the  Dashboard, rather than purchase this capability. VA completed  development of this tool in March 2013, at which point it resumed  making the required updates to CIO ratings on the Dashboard. For the  3 that VA updated during this period, VA used a manual budget  process to update their ratings in September 2012.
SSA, Information Resources Management Strategic Plan: FY 2012-2016 (May 2012). and schedule variances, prevent the agency from monitoring year-to- year investment performance, and make it difficult to estimate and  understand life-cycle costs. While this only affected one investment  we reviewed, it has the potential to impact SSA’s entire IT investment  portfolio. As described in our Cost Estimating and Assessment Guide,  the purpose of such baseline changes should be to restore  management’s control of the remaining effort by providing a  meaningful basis for performance management. Further, these  changes should be infrequent, and recurrent changes may indicate  that the scope is not well understood or simply that program  management is unable to develop realistic estimates.SSA, frequent rebaselining increases the risk that its investments  have undetected cost or schedule variances that will impact the  success of the associated investment.
While agencies experienced several issues with reporting the risk of their  investments, such as technical problems and delayed updates to the  Dashboard, the CIO ratings were mostly or completely consistent with  investment risk at seven of the eight selected agencies. Additionally, the  agencies had already addressed several of the discrepancies that we  identified. The final agency, VA, did not update 7 of its 10 selected  investments because it elected to build, rather than buy, the ability to  automatically update the Dashboard, and has now resumed updating all  investments. To their credit, agencies’ continued attention to reporting the  risk of their major investments supports the Dashboard’s goal of providing  transparency and oversight of federal IT investments. Nevertheless, the  rating issues that we identified with performance reporting and annual  baselining, some of which are now corrected, serve to highlight the need  for agencies’ continued attention to the timeliness and accuracy of  submitted information, in order to allow the Dashboard to continue to fulfill  its stated purpose.
section_title: Highest-Risk Investments Were Appropriately Addressed by Selected Agencies
paragraphs: We have previously concluded that, consistent with government and  industry best practices, including our own guidance on IT investment  management, agencies’ highest-risk investments should receive  additional management attention. In particular, agency leaders should  use data-driven reviews as a leadership strategy to drive performance  improvement. Correspondingly, OMB requires reviews, known as  TechStat sessions, to enable the federal government to intervene to  turnaround, halt, or terminate IT projects that are failing or are not  producing results. Of the 80 investments we reviewed, there were 8  investments at four of the eight selected agencies that received a red  (high or moderately high risk) rating in 2012. See figure 6 for a list of  those at-risk investments and the associated CIO ratings.
Of the eight investments receiving a red rating in 2012, the agencies  reviewed seven using tools such as TechStat sessions,investment review boards, and other high-level reviews. Each of these  resulted in action items intended to improve performance. The final  investment, Agriculture’s Human Resources Line of Business: Service   department  Center was scheduled to be reviewed using a TechStat session, but in  August 2013 officials from Agriculture’s Office of the CIO stated that this  investment was in the process of going through a rebaseline. As noted  earlier, such changes should be infrequent and reinforce the need for  agencies to review the highest-risk investments to ensure that root  causes of baseline changes are effectively addressed.
Each of these agencies we reviewed had similar approaches to  identifying and conducting these reviews. The agencies with red-rated  investments in 2012—Agriculture, Commerce, Justice, and  Transportation—monitored investment performance, identified troubled  investments using a variety of criteria, reviewed investments in need of  attention, and assigned and tracked action items. For example, both  Agriculture and Commerce identified investments for review using the  Dashboard’s CIO ratings. All four agencies further ensured that  investments implemented corrective actions resulting from management  reviews. For example, as a result of its November 2012 TechStat,  Agriculture leaders assigned 17 action items to the Farm Program  Modernization investment, which were to be completed by January 2013.  Additionally, the four remaining agencies without red-rated investments— Energy, Treasury, VA, and SSA—have documented processes which  provide comparable monitoring and oversight capabilities. For example,  VA’s process relies on missed deliverable dates as the key requirement  to hold a review, after which senior management makes a determination  as to the future of the investment. Alternatively, Energy reviews all major  IT investments on a quarterly basis and requires those which fall outside  expected thresholds to complete corrective actions. As such, all of the  agencies have focused management attention on troubled investments  and once identified, have established and followed through on clear  action items to turn around or terminate such investments.
section_title: Conclusions
paragraphs: Since its inception in 2009, the IT Dashboard has increased the  transparency of the performance of major federal IT investments. Its CIO  ratings, in particular, have improved visibility into changes in the risk level  of investments over time, as well as agencies’ ability to accomplish their  goals. To that end, over the past 4 years, the agencies we reviewed have  reported lower risk for more than one quarter of all their investments.  However, the effectiveness of the Dashboard depends on the quality of  the information that agencies submit. We previously recommended that  OMB clarify guidance on whether certain types of systems should be  included in IT reporting, but OMB did not agree. Agencies’ subsequent  decisions to remove investments from the Dashboard by changing  investments’ categorizations represent a troubling trend toward  decreased transparency and accountability. Thus, we continue to believe  that our prior recommendation remains valid and should be implemented.  Additionally, OMB’s annual freeze of the Dashboard for as long as 8  months negates one of the primary purposes of this valuable tool—to  facilitate transparency and oversight of the government’s billions of  dollars in IT investments.
Beyond the transparency they promote, CIO ratings present an  opportunity to improve the data and processes agencies use to assess  investment risk. Each of the agencies we examined had established such  processes, and most of the resulting Dashboard ratings were consistent  with underlying documentation. While two agencies, Transportation and  Treasury, had ratings that accurately reflected their investments’  supporting documentation, other agencies’ Dashboard ratings were  inconsistent with the actual risk of the associated investment. These  inconsistencies—such as inaccurate reflection of negative performance,  questionable decisions about the recategorization of investments, and  too-frequent changes to performance baselines—highlight the continued  need for agencies to populate the Dashboard with an accurate  representation of the full breadth and health of their investments. In  providing this information, agencies will help the Dashboard accomplish  its goal of providing transparency and oversight of millions of dollars in  federal IT investments.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that the Dashboard provides meaningful ratings and  reliable investment data, we are recommending that the Director of OMB  direct the Federal CIO to make accessible regularly updated portions of  the public version of the Dashboard (such as CIO ratings) independent of  the annual budget process.
In addition, to better ensure that the Dashboard provides accurate ratings,  we are making three recommendations to the heads of three of the  selected agencies. Specifically, we are recommending that:
The Secretary of Commerce direct the department CIO to ensure that  the department’s investments are appropriately categorized in  accordance with existing statutes and that major IT investments are  included on the Dashboard.
The Secretary of Energy direct the department CIO to ensure that the  department’s investments are appropriately categorized in  accordance with existing statutes and that major IT investments are  included on the Dashboard.
The Commissioner of the SSA direct the CIO to revise the agency’s  investment management processes to ensure that they are consistent  with the baselining best practices identified in our published guidance  on cost estimating and assessment.
section_title: Agency Comments and Our Evaluation
paragraphs: We received comments on a draft of this report from OMB and all eight  departments and agencies in our review. OMB neither agreed nor  disagreed with our recommendation; Agriculture agreed with the report’s  findings; Commerce disagreed with our recommendation; Energy  concurred with our recommendation with one exception related to  supercomputers; Justice, Treasury, and Transportation neither agreed  nor disagreed with the report’s findings; VA agreed with the report’s  findings; and SSA agreed with our recommendation. Each agency’s  comments are discussed in more detail below.
In written comments, the Federal CIO neither agreed nor disagreed  with our recommendation for OMB to make accessible regularly  updated portions of the public version of the Dashboard (such as CIO  ratings) independent of the annual budget process. However, OMB  also noted that the manner in which agencies submit Dashboard  information to OMB makes it difficult to separate materials it believes  are predecisional or deliberative. Despite this, OMB agreed to explore  whether it would be practical to make the Dashboard more accessible,  and to consider how it could separate predecisional or deliberative  materials. We support OMB in its efforts to increase public  transparency and accountability.
OMB continued to disagree with the 2011 recommendation that we  believe continues to have merit, related to the clarity of its guidance  on whether certain types of systems should be included in IT  reporting. OMB believes that the existing guidance is appropriate and  does not have plans to review it. However, as noted earlier in this  report, we believe that the recommendation is appropriate because  the existing guidance does not address key categories of IT  investments where we continued to find inconsistencies.
OMB also disputed our characterization of two issues. First, OMB  noted that up-to-date Dashboard information is available to the  agencies and OMB during the budget development period, who use it  as a tool for investment oversight and decision making. While we  acknowledge that OMB and federal agencies continue to have access  to the Dashboard during the budget process, the public display of  these data is intended to allow other oversight bodies, including  Congress and the general public, to hold government agencies  accountable for progress and results. Secondly, OMB stated that it  began using the IT Dashboard to help identify at-risk investments  starting with its launch in June 2009, rather than 2010. We modified  the report to reflect OMB’s statement. OMB also provided technical  comments, which we have incorporated in the report as appropriate.
OMB’s written comments are provided in appendix III.
In written comments, Agriculture’s CIO stated that the department  concurred with the content of the report and had no comments.  Agriculture’s written comments are provided in appendix IV.
In written comments, Commerce agreed with most of the report’s  findings but disagreed with our recommendation to ensure that the  department’s investments are appropriately categorized in  accordance with existing statutes and that major IT investments are  included on the Dashboard. Specifically, Commerce stated that  although it is no longer reporting publicly on its satellite ground system  investments through the Dashboard, neither the department CIO nor  OMB has relinquished its oversight role. Moreover, Commerce stated  that it is reviewing its satellites more frequently and in more detail; as  an example, the department noted that Commerce’s CIO conducts  monthly Dashboard-like assessments that cover the status of the  satellite investments. However, regardless of these additional efforts,  the removal of the satellite investments from the Dashboard prevents  the public display of these data intended to allow OMB and other  oversight bodies, including Congress, to hold the government  agencies accountable for progress and results. Additionally, as  discussed in this report, Commerce’s recategorization of its satellite  ground system investments is contrary to the definition of IT as set  forth in the Clinger-Cohen Act. Based on these facts, we continue to  believe that our recommendation to Commerce that its CIO ensure  that the department’s investments are appropriately categorized in  accordance with existing statutes and that major IT investments are  included on the Dashboard has merit and should be implemented.  Commerce’s written comments are provided in appendix V. The  department also provided technical comments related to our  characterization of the department’s rating process, which we have  incorporated in the report as appropriate.
In written comments, Energy concurred, but with a comment  regarding our recommendation to ensure that the department’s  investments are appropriately categorized in accordance with existing  statutes and that major IT investments are included on the  Dashboard. Specifically, while agreeing that Energy should ensure  that its IT investments are appropriately categorized, the department  stated that it would continue to report investments that it believes  should be categorized and managed on the Dashboard, noting that  supercomputers would be an exception to this policy. However, this  exception runs contrary to the finding on which this recommendation  is based, namely that Energy removed $368 million in investments  from the Dashboard, including a supercomputer reported in 2012 as  the most powerful in the world. While Energy agreed to partner with  OMB to develop an alternate mechanism to track and report  supercomputer investments to OMB, it is not clear whether this  information will be publicly available. Additionally, Energy’s  recategorization of its supercomputer investments is contrary to the  definition of IT as set forth in the Clinger-Cohen Act, as discussed in  this report. Therefore, we disagree that the department’s planned  actions adequately address existing requirements for open and  transparent reporting of major IT investments, and we stand by our  assessment that Energy’s CIO should ensure that the department’s  investments are appropriately categorized in accordance with existing  statutes and that major IT investments are included on the  Dashboard. Energy’s written comments are provided in appendix VI.
In comments provided via e-mail on November 6, 2013, a  representative of Justice’s Justice Management Division stated that  the department had no comments.
In comments provided via e-mail on November 1, 2013,  Transportation’s Deputy Director of Audit Relations stated that the  department had no comments. The department also provided  technical comments, which we have incorporated in the report as  appropriate.
In written comments, the Treasury’s CIO stated that the department  had no comments. Treasury’s written comments are provided in  appendix VII.
In written comments, VA’s Chief of Staff stated that the department  generally agreed with the findings of the report and provided general  comments in which it confirmed that issues identified in the report had  been addressed. Specifically, VA confirmed that the department has  implemented the capability to automatically submit CIO ratings to the  Dashboard and now uses more specific data to generate the CIO  rating, such as the number of “red flags” associated with an  investment or the number of TechStat reviews held. Finally, while  stating that the department will continue its efforts to improve timely  and accurate reporting to the Dashboard, VA also noted that it is  important to recognize that the department manages investment  delivery at the project level in comparison to the investment level  found in the Dashboard. VA’s written comments are provided in  appendix VIII.
In written comments, SSA’s Deputy Chief of Staff agreed with our  recommendation to revise the agency’s investment management  processes to ensure that they are consistent with the baselining  practices, and discussed planned actions to address it. SSA’s written  comments are provided in appendix IX.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees; the Secretaries of Agriculture, Commerce, Energy,  Transportation, Treasury, Veterans Affairs, Attorney General of the United  States, Acting Commissioner of Social Security, the Director of the Office  of Management and Budget; and other interested parties. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix X.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives for this engagement were to (1) characterize the Chief  Information Officer (CIO) ratings for selected federal agencies’  information technology (IT) investments as reported on the Dashboard  over time, (2) determine the extent to which selected agencies’ CIO  ratings are consistent with reported investment risk, and (3) determine the  extent to which selected agencies are addressing at-risk investments.
To address our first objective, we selected the eight agencies with the  most reported major IT spending in fiscal year 2012, after excluding  agencies reviewed in our most recent Dashboard report. The eight  agencies are the Departments of Agriculture (Agriculture), Commerce  (Commerce), Energy (Energy), Justice (Justice), Transportation  (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), and  the Social Security Administration (SSA). The results in this report  represent only these agencies. We downloaded and examined the  Dashboard’s CIO ratings for all major investments at the eight agencies  (a total of 383 investments reported by these agencies). To characterize  the numbers and percentages of major IT investments at each risk level  at each of our subject agencies, we analyzed, summarized, and—where  appropriate—graphically depicted average CIO ratings for investments by  agencies over time during the period from June 2009 to August 2013.  Specifically, we compared each investment’s first and last CIO ratings  (including those investments that were not on the Dashboard at its  inception, and those that were downgraded or eliminated) and  summarized the data by agency. To describe whether CIO ratings  indicated higher or lower investment risk over time, we calculated the  numbers and percentages of investments (by agency and collectively for  all the agencies) that maintained a constant rating over the entire  performance period, and those that experienced a change to their CIO  rating in at least one rating period. Then we analyzed the subset of  investments that experienced at least one changed rating and compared  the first CIO rating with the latest CIO rating (no later than August 2013)  to determine the numbers and percentages of investments (by agency  and collectively for all the agencies) that experienced a net rating  increase, a net rating decrease, or no net change. We also reviewed  investments which had been removed from the Dashboard due to  recategorization and compared their definitions to the Clinger-Cohen  Act’s definition of IT. We presented our results to each agency and the  Office of Management and Budget (OMB) and solicited their input,  explanations for the results, and additional corroborating documentation,  where appropriate.
To accomplish the second objective, we selected the top 10 major  investments at each of the eight selected agencies (for a total of 80  investments), with the highest reported IT spending in fiscal year 2012.  We reviewed investment documentation (such as executive-level  briefings, operational analyses, and TechStat results) and interviewed  officials from each of the agencies to understand how they rate and  monitor investments. of investment risk and performance (such as cost and schedule  variances), which could impact the success of the associated investment.  We then organized the results by month, and compared these results to  the relevant investment Dashboard CIO ratings for calendar year 2012.
Within these artifacts, we identified representations  We then evaluated whether each investment’s monthly CIO rating was  consistent with reported investment risks, and categorized each  investment by the number of months which were inconsistent.  Investments which were consistent with underlying documentation for  every month in 2012 were categorized as “consistent,” those which were  inconsistent for 1 or more months were categorized as “partially  inconsistent,” and those with inconsistencies in each month were  “inconsistent.”
Because we were only evaluating the consistency of Dashboard ratings with reported  risk, we did not evaluate the accuracy of the investment documentation. moderately high) rating on the Dashboard during 2012. We reviewed the  processes used to address these investments and interviewed relevant  officials. We also examined the results of performance reviews and  interviewed officials to determine whether the agencies implemented the  appropriate risk management processes to oversee and review the  selected investments.
We conducted this performance audit from February to December 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Selected Investments
paragraphs: Below is the list of the investments that are included in this review, as well  as the reported fiscal year 2012 IT spending.
section_title: Appendix III: Comments from the Office of Management and Budget
section_title: Appendix IV: Comments from the Department of Agriculture
section_title: Appendix V: Comments from the Department of Commerce
section_title: Appendix VI: Comments from the Department of Energy
section_title: Appendix VII: Comments from the Department of the Treasury
section_title: Appendix VIII: Comments from the Department of Veterans Affairs
section_title: Appendix IX: Comments from the Social Security Administration
section_title: Appendix X: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Dave Hinchman (Assistant Director), Rebecca  Eyler, Mike Mithani, Kevin Walsh, and Shawn Ward.