Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies
Published: 2012-10-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In June 2009, OMB launched the federal IT Dashboard, a public website that reports performance data for over 700 major IT investments that represent about $40 billion of the estimated $80 billion budgeted for IT in fiscal year 2012. The Dashboard is to provide transparency for these investments to aid public monitoring of government operations. It does so by reporting, among other things, how agency CIOs rate investment risk. GAO was asked to (1) characterize the CIO ratings for selected federal agencies' IT investments as reported over time on the Dashboard, (2) determine how agencies' approaches for assigning and updating CIO ratings vary, and (3) describe the benefits and challenges associated with agencies' approaches to the CIO rating.
To do so, GAO selected six agencies spanning a range of 2011 IT spending levels and analyzed data reported for each of their investments on the Dashboard. GAO also interviewed agency officials and analyzed related documentation and written responses to questions about ratings and evaluation approaches, as well as agency views on the benefits and challenges related to the CIO rating.
section_title: What GAO Found
paragraphs: Chief Information Officers (CIO) at six federal agencies rated the majority of their information technology (IT) investments as low risk, and many ratings remained constant over time. Specifically, CIOs at the selected agencies rated a majority of investments listed on the federal IT Dashboard as low risk or moderately low risk from June 2009 through March 2012; at five of these agencies, these risk levels accounted for at least 66 percent of investments. These agencies also rated no more than 12 percent of their investments as high or moderately high risk, and two agencies (Department of Defense (DOD) and the National Science Foundation (NSF)) rated no investments at these risk levels. Over time, about 47 percent of the agencies' Dashboard investments received the same rating in every rating period. For ratings that changed, the Department of Homeland Security (DHS) and Office of Personnel Management (OPM) reported more investments with reduced risk when initial ratings were compared with those in March 2012; the other four agencies reported more investments with increased risk. In the past, the Office of Management and Budget (OMB) reported trends for risky IT investments needing management attention as part of its annual budget submission, but discontinued this reporting in fiscal year 2010.
Agencies generally followed OMB's instructions for assigning CIO ratings, which included considering stakeholder input, updating ratings when new data become available, and applying OMB's six evaluation factors. DOD's ratings were unique in reflecting additional considerations, such as the likelihood of OMB review, and consequently DOD did not rate any of its investments as high risk. However, in selected cases, these ratings did not appropriately reflect significant cost, schedule, and performance issues reported by GAO and others. Moreover, DOD did not apply its own risk management guidance to the ratings, which reduces their value for investment management and oversight.
Various benefits were associated with producing and reporting CIO ratings. Most agencies reported (1) increased quality of their performance data, (2) greater transparency and visibility of investments, and (3) increased focus on project management practices. Agencies also noted challenges, such as (1) the effort required to gather, validate, and gain internal approval for CIO ratings; and (2) obtaining information from OMB to execute required changes to the Dashboard. OMB has taken steps to improve its communications with agencies.
section_title: What GAO Recommends
paragraphs: GAO is recommending that OMB analyze agencies' investment risk over time as reflected in the Dashboard's CIO ratings and present its analysis with the President's annual budget submission, and that DOD ensure that its CIO ratings reflect available investment performance assessments and its risk management guidance. Both OMB and DOD concurred with our recommendations.

section_title: Letter
paragraphs: Spending on information technology (IT) represents a significant portion  of the federal budget—estimated at $80 billion for fiscal year 2012. More  than 700 major investments account for approximately $40 billion of this  IT spending. The Clinger-Cohen Act of 1996 charges the Director of the  Office of Management and Budget (OMB) with responsibility for  analyzing, tracking, and evaluating the risks and results of all major IT  investments as part of the federal budget process, and reporting to  Congress on the performance benefits achieved by these investments.  The act also places responsibility for managing investments with the  heads of agencies and establishes chief information officers (CIOs) to  advise and assist agency heads in carrying out this responsibility.
OMB launched the Federal IT Dashboard in June 2009 as a public  website that reports performance and supporting data for the major IT  investments. The Dashboard is to provide transparency for these  investments in order to facilitate public monitoring of government  operations and accountability for investment performance by the federal  CIOs who oversee them. In January 2010, OMB began using the  Dashboard as one of several tools to identify troubled investments. These  investments became the focus of joint OMB-agency TechStat  Accountability Sessions (TechStats)—evidence-based reviews intended  to improve investment performance through concrete actions. In  December 2010, OMB reported that these sessions resulted in $3 billion  in reduced life-cycle costs and subsequently incorporated the TechStat  model into its 25-point plan for reforming federal IT management. With  this plan, agency CIOs became responsible for leading TechStat sessions  at the department level, analyzing investments using data from the  Dashboard, and terminating or turning around at least one-third of  underperforming IT projects within 18 months. OMB reported its progress  on the plan, improvements to the Dashboard, and results of TechStat  sessions in the analytical perspectives it provided for the President’s 2012  and 2013 budget submissions.
In response to your request, our objectives for this review were to (1)  characterize the CIO ratings for selected federal agencies’ IT investments  as reported over time on the Dashboard, (2) determine how agencies’  approaches for assigning and updating CIO ratings vary, and (3) describe  the benefits and challenges associated with agencies’ approaches to the  CIO rating.
To establish the scope of our review, we selected six agencies that  spanned a range of IT spending for fiscal year 2011, including the three  highest spending agencies, two of the lowest, and an agency in the  middle. Collectively, these agencies accounted for approximately $51  billion, or 65 percent, of 2011 spending on IT investments. The six  agencies are the Department of Defense (DOD), Department of  Homeland Security (DHS), Department of Health and Human Services  (HHS), Department of the Interior (DOI), National Science Foundation  (NSF), and Office of Personnel Management (OPM).
To address our objectives, we downloaded CIO ratings and related data  reported for investments on the Dashboard and analyzed these data for  the period June 2009 to March 2012. We did not independently evaluate  the ratings as reported by the agencies, but determined that they were  sufficiently complete and accurate for our analyses. We interviewed  agency officials, including CIOs where possible, and obtained written  responses and supporting documents, related agency policies,  procedures, reported data, artifacts, as well as agency views on the  benefits and challenges associated with performing these ratings and  reporting them to the Dashboard. We also utilized recent GAO and DOD  Inspector General reviews of DOD’s major IT investments and compared  findings in these reports to the CIO ratings that the department submitted  to the Dashboard. In addition, we analyzed OMB documentation and  interviewed OMB staff to update our information on how the Dashboard  has evolved, identify the guidance agencies received about CIO ratings,  determine the efforts OMB has under way to improve the Dashboard, and  describe the ways in which OMB is using the data to improve IT  management.
We conducted this performance audit from January 2012 through  September 2012 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives. Further details of  our objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: The Clinger-Cohen Act of 1996 requires OMB to establish processes to  analyze, track, and evaluate the risks and results of major capital  investments in information systems made by federal agencies and report  to Congress on the net program performance benefits achieved as a  result of these IT investments. Further, the act places responsibility for  managing investments with the heads of agencies and establishes CIOs  to advise and assist agency heads in carrying out this responsibility.
OMB established the Management Watch List in 2003 to help carry out its  oversight role. The Management Watch List included mission-critical  projects that needed to improve performance measures, project  management, IT security, or overall justification for inclusion in the  President’s budget submission. Further, in August 2005, OMB  established a High-Risk List, which consisted of projects identified by  federal agencies, with OMB’s input, as requiring special attention from  oversight authorities and the highest levels of agency management.  Between 2005 and 2009, OMB described its efforts to monitor and  manage risky federal IT investments in the annual budget submission.
Over the past several years, we have reported and testified on OMB’s  initiatives to highlight troubled IT projects, justify investments, and use  project management tools. For instance, in 2006 we recommended that  OMB develop a single aggregated list of high-risk projects and their  deficiencies and use that list to report to Congress on progress made in  correcting high-risk problems. As a result, OMB started publicly releasing  aggregate data on its Management Watch List and disclosing the  projects’ deficiencies. Moreover, between 2007 and 2009, the President’s  budget submission included an overview of investment performance over  several budget years, including the number of federal IT projects in need  of management attention. Such information helped Congress stay better   With  informed of high-risk projects and make related funding decisions.the advent of its IT Dashboard in 2009, OMB discontinued this type of  reporting in the fiscal year 2010 budget submission.
subsections: section_title: OMB’s Dashboard Provides Visibility into the Performance of Federal IT Investments
paragraphs: to allow OMB; other oversight bodies, including Congress; and the  general public to hold the government agencies accountable for progress  and results.
OMB reported on plans and implementation progress for this  management tool in the “Analytical Perspectives” section of the  President’s budget submissions for fiscal years 2012 and 2013,  including planned updates to the Dashboard during 2012 to support  closer executive oversight and intervention to prevent schedule delays,  cost overruns, and failures in delivering key functionality needed by  federal programs. For example, it reported using the Dashboard to  identify investments for TechStat reviews.
The Dashboard visually presents performance ratings for agencies overall  and for individual investments using metrics that OMB has defined—cost,  schedule, and CIO evaluation. The website also provides the capability to  download certain data. Figure 1 is an example of an agency’s (OPM)  portfolio page as recently depicted on the Dashboard.
The Dashboard’s data spans the period from June 2009 to the present,  and is based, in part, on each agency’s exhibit 53 and exhibit 300   to OMB, as well as on agency assessments and  submissionssupporting information on each investment. Over the life of the  Dashboard, OMB has issued guidance to agencies on, among other  things, what data to report, how those data need to be structured and  formatted for upload to the Dashboard, and procedures for using the  Dashboard’s submission tools. For instance, OMB instructed agencies to  update and submit investment cost and schedule data monthly. OMB has  made various changes to the organization, available data, and features of  the Dashboard over time, including  improvements to Dashboard calculations to incorporate the variance  of “in progress” milestones rather than just “completed” milestones; web pages containing data on historical ratings and rebaselines of  eliminated and downgraded investments; added data on awarded contracts, with links to USAspending.gov;  release of IT Dashboard source code and documentation to an open  source hosting provider; enhancements to baseline history, which give users the ability to see  field-by-field changes for each rebaseline; a mechanism for OMB analysts to provide feedback to agencies on mobile-friendly formatting of Dashboard displays.
Once OMB has received agency-reported investment data, it converts  these into investment performance ratings for display on the dashboard  according to calculations and protocols described on its website. OMB  assigns cost and schedule performance ratings by using data submitted  by agencies to calculate variances between the planned cost or schedule  targets and the actual or projected cost or schedule values. OMB  converts these variances to percentages, and assigns the ratings to be  presented on the Dashboard within three ranges, red, yellow, and green,  as shown in table 1.
Although the thresholds for assigning cost and schedule variance ratings  has remained constant over the life of the Dashboard, the cost and  schedule data agencies are required to submit have changed in several  ways, as have the variance calculations. For example, in response to our  recommendations (further discussed in the next section), OMB changed  how the Dashboard calculates the cost and schedule ratings in July 2010,  to include “in progress” milestones rather than just “completed” ones for  more accurate reflection of current investment status.
section_title: GAO Has Previously Reported on the Dashboard’s Value, Data Quality, and Recent Improvements
paragraphs: We have previously reported that OMB has taken significant steps to  enhance the oversight, transparency, and accountability of federal IT  investments by creating its IT Dashboard, and by improving the accuracy  of investment ratings. We also found issues with the accuracy and data  reliability of cost and schedule data, and recommended steps that OMB  should take to improve these data.
In July 2010, we reportedDashboard were not always accurate for the investments we reviewed,  because these ratings did not take into consideration current  performance. As a result, the ratings were based on outdated information.  We recommended that OMB report on its planned changes to the  Dashboard to improve the accuracy of performance information and  provide guidance to agencies to standardize milestone reporting. OMB  agreed with our recommendations and, as a result, updated the  Dashboard’s cost and schedule calculations to include both ongoing and   that the cost and schedule ratings on OMB’s  completed activities. Similarly, in March 2011, we reportedhad initiated several efforts to increase the Dashboard’s value as an  oversight tool, and had used its data to improve federal IT management.  We also reported, however, that agency practices and the Dashboard’s  calculations contributed to inaccuracies in the reported investment  performance data. For instance, we found missing data submissions or  erroneous data at each of the five agencies we reviewed, along with  instances of inconsistent program baselines and unreliable source data.  As a result, we recommended that the agencies take steps to improve the  accuracy and reliability of their Dashboard information, and that OMB  improve how it rates investments relative to current performance and  schedule variance. Most agencies generally concurred with our  recommendations; OMB agreed with our recommendation for improving  ratings for schedule variance. It disagreed with our recommendation to  improve how it reflects current performance in cost and schedule ratings,  but more recently made changes to Dashboard calculations to address  this while also noting challenges in comprehensively evaluating cost and  schedule data for these investments.
More recently, in November 2011, we reported investment cost and schedule ratings had improved since our July 2010  report because OMB had refined the Dashboard’s cost and schedule  calculations. Most of the ratings for the eight investments we reviewed  were accurate, although we noted that more could be done to inform  oversight and decision making by emphasizing recent performance in the  ratings. We recommended that the General Services Administration  comply with OMB’s guidance for updating its ratings when new  information becomes available (including when investments are  rebaselined) and the agency concurred. Since we previously  recommended that OMB improve how it rates investments, we did not  make any further recommendations.
section_title: CIO Ratings Are Important for OMB’s IT Reform and Management Initiatives
paragraphs: GAO-12-210. investment into a color for depiction on the Dashboard. An OMB staff  member from the Office of E-Government and Information Technology  noted that the CIO rating should be a current assessment of future  performance based on historical results and is the only Dashboard  performance indicator that has been defined and produced the same way  since the Dashboard’s inception. According to OMB’s instructions, a CIO  rating should reflect the level of risk facing an investment on a scale from  1 (high risk) to 5 (low risk) relative to that investment’s ability to  accomplish its goals. Each agency CIO is to assess their IT investments  against a set of six preestablished evaluation factors identified by OMB  (shown in table 2) and then assign a rating of 1 to 5 based on his or her  best judgment of the level of risk facing the investment. According to an  OMB staff member, agency CIOs are responsible for determining  appropriate thresholds for the risk levels and for applying them to  investments when assigning CIO ratings.
OMB recommends that CIOs consult with appropriate stakeholders in  making their evaluation, including Chief Acquisition Officers, program  managers, and other interested parties. Ultimately, CIO ratings are  assigned colors for presentation on the Dashboard, according to the five- point rating scale, as illustrated in table 3.
OMB has made the CIO’s evaluation and rating a key component of its  larger IT Reform Initiative and 25 Point Plan. In its plan, OMB reported  that it used agencies’ CIO ratings to select investments for the TechStat  review sessions it conducted between 2010 and 2011. These sessions  are data-driven assessments of IT investments by agency leaders that  are intended to result in concrete action to improve performance. OMB  reported that the TechStats it conducted on selected investments resulted  in approximately $3 billion in reduced costs. Building on the results of  those sessions, the plan articulates a strategy for strengthening IT  governance, in part, through the adoption of the TechStat model by  federal agencies. In conducting TechStats, agencies are to rely, in part,  on CIO ratings from the IT Dashboard. The TechStat Toolkit, developed  by OMB and a task force of agency leads, provides sample questions  regarding an investment’s CIO rating and associated risks for use in  TechStat sessions.
Furthermore, OMB issued guidance in August 2011other things, that agency CIOs shall be held accountable for the  performance of IT program managers based on their governance process  and the data reported on the IT Dashboard, which includes the CIO   that stated, among  rating. According to OMB, the addition of CIO names and photos on  Dashboard investments is intended to highlight this accountability and link  it to the Dashboard’s reporting on investment performance. Figure 2  illustrates the CIO rating information presented on the Dashboard for an  example IT investment.
section_title: CIOs Rated Most IT Investments as Low Risk or Moderately Low Risk; Agencies Had Mixed Results in Reducing Higher Risk Levels
paragraphs: As of March 2012, CIO ratings for most investments listed on the  Dashboard for the six agencies we reviewed indicated either low risk or  moderately low risk (223 out of 313 investments across all the selected  agencies). High risk or moderately high risk ratings were assigned to  fewer investments (12 out of 313 investments across all the selected  agencies). Figure 3 presents the total number of IT investments rated on  the Dashboard for each of the selected agencies according to their risk  levels, as of March 2012, and illustrates the predominance of low risk  investments for the agencies in our review. The figure also reports  agencies’ budgets for their major IT investments for fiscal year 2012, as  presented on the Dashboard.
Historically, over the life of the Dashboard from June 2009 to March 2012,  low or moderately low risk ratings accounted for at least 66 percent of all  ratings at five of the six agencies (the exception is DHS with 51 percent).
Medium risk ratings accounted for between 0 to 38 percent of all reported  ratings across agencies during this period. The maximum percentage of  ratings in the high risk or moderately high risk categories for any agency  during this 34-month period was 12 percent, with two agencies—DOD   DOD stated in written  and NSF—reporting no high risk investments.comments that this was because they did not deem any of their  investments to be high risk. (DOD’s investment risks are further  discussed in the next section.) An NSF official from the Division of  Information Systems stated that there were no high risk investments  because most of their investments were in the operations and  maintenance phase. Table 4 presents the average composition of ratings  for each agency during the reporting period of June 2009 to March 2012.  Appendix III depicts each agency’s CIO ratings by risk level on a monthly  basis during the reporting period.
Overall, the CIO rating remained constant for 147 of 313 investments that  were active as of March 2012 (about 47 percent of the investments we  reviewed). These investments were rated at the same risk level during  every rating period (see fig. 4).
Four of the six agencies did not change the CIO rating for a majority of  their investments (excluding any investments that were downgraded or  eliminated) during the time frame we examined. In contrast, the other two  agencies—OPM and HHS—changed the CIO rating for more than 70  percent of their investments at least once between the investment’s initial  rating and the rating reported as of March 2012. Table 5 lists the number  of each agency’s investments whose ratings were constant and changed  over time.
Agencies offered several reasons for why many investments had no  changes in their CIO ratings during their entire time on the Dashboard.  Five of the six selected agencies indicated that many investments were in  a steady-state or operations and maintenance phase with no new  development. One agency reported that their investments’ CIO ratings  remained constant because the investments consistently met all  requirements and deadlines and were using project management best  practices.
The agencies we reviewed showed mixed results in reducing the number   For investments  of higher risk investments during the rating period.whose rating changed at least once during the period, 40 percent (67  investments) received a lower risk rating in March 2012 than they  received initially, 41 percent of investments (68 investments) received a  higher risk rating, and the remaining 19 percent (31 investments) received  the same rating in March 2012 as they had initially received, despite  whatever interim changes may have occurred (i.e., there was no “net”  change to their reported risk levels). (See fig. 5.)
Two agencies—DHS and OPM—reported more investments with reduced  risk in March 2012, as compared with initial ratings. The other four  agencies reported more investments with increased risk. Table 6 presents  net changes in risk levels at each of the selected agencies (among  investments that were not downgraded or eliminated). Appendix III  graphically summarizes these data for all six agencies.
Agencies most commonly cited additional oversight or program reviews  as factors that contributed to decreased risk levels. Specifically, agencies  commented that the CIO ratings and Dashboard reporting had spurred  improved program management and risk mitigation. For example, one  agency’s officials commented that the CIO now closely monitors the  monthly performance and risk data generated by their investments, and  that the additional oversight has brought about strengthened processes  and more focused attention to issues.
In contrast, several agencies cited generally poor risk management at the  investment level, the introduction of new investment/programs risks, as  well as instances of poor project management as factors contributing to  increased risk for investments. For example, one agency responded that  internal review findings revealed new risks that caused an investment’s  risk level to increase. Another agency’s officials reported that various  technical issues caused one of their investments to fall behind schedule,  thus increasing risk.
Both OMB and several agencies suggested caution in interpreting  changing risk levels for investments. They noted that an increase in an  investment’s risk level can sometimes indicate better management by the  program or CIO because previously unidentified risks have been  assessed and included in the CIO evaluation. Conversely, a decrease in  an investment’s risk level may not indicate improved management if the  data and analysis on which the CIO rating are based is incomplete,  inconsistent, or outdated.
Further analysis of the characteristics and causes of Dashboard’s CIO  ratings, and reporting on the patterns of risk within and among agencies,  could provide Congress and the public with additional perspectives on  federal IT investment risk over time. However, for the past four budget  submissions, OMB has not summarized the extent of risk represented by  major federal IT investments in the analysis it prepares annually for the  President’s budget submission, as it did prior to the fiscal year 2010  submission. As a result, OMB is missing an opportunity to integrate such  risk assessments into its evaluation of major capital investments in  reporting to Congress.
section_title: Most Agencies Established Approaches for CIO Ratings That Follow OMB’s Instructions
paragraphs: OMB has provided agencies with instructions for assigning CIO ratings for  the major IT investments reported on the Dashboard. Specifically, OMB’s  instructions state that agency CIOs should rate each investment based on  his/her best judgment and should  include input from stakeholders, such as Chief Acquisition Officers,  program managers, and others;   update the rating as soon as new information becomes available that  might affect the assessment of a given investment; and   utilize OMB’s investment rating factors, including: risk management,  requirements management, contractor oversight, historical  performance, and human capital, as well as any other factors deemed  relevant by the CIO.
Despite differences in the specific inputs and processes used, agencies  generally followed OMB’s instructions for assigning CIO ratings. However,  DOD’s ratings reflected additional considerations beyond OMB’s  instructions and did not reflect available information about significant risks  for certain investments. The sections that follow describe how each  agency addressed OMB’s instructions.
Include input from stakeholders. Each of the six agencies we reviewed  relied on stakeholder input, at least in part, when assigning CIO ratings.  Agencies also cited a variety of review boards, data from program and  financial systems, and other investment assessments as inputs to the  rating. Table 7 describes the data and processes that agencies reported  using when they derived their CIO ratings.
Update CIO ratings. All six agencies established guidelines for  periodically reviewing and updating their CIO ratings. Specifically, HHS,  NSF, DOI, and OPM reported that they update CIO ratings on a monthly  basis. DOD has adopted a quarterly update cycle, although an official  noted that the actual process of collecting information and evaluating  investments for the ratings takes slightly longer than 3 months. DHS  officials with the Office of the CIO stated that the frequency of its updates  varies based on the risk level of an investment’s previous rating:  investments with a previous CIO rating of green are to be reviewed  semiannually; yellow investments are to be reviewed quarterly; and red  investments are to be reviewed monthly.
Utilize OMB’s investment rating factors. Most of the selected agencies  use OMB’s investment rating factors when evaluating their investments.  Only one agency (HHS) does not use all of them. Specifically, an HHS  official from the Office of the CIO told us that human capital issues are not  explicitly covered in their CIO rating criteria because investment owners  are to provide adequate IT human capital, and that these owners will  reflect any issues that arise when providing input for the CIO rating.
Among the agencies we reviewed, DOD was unique in that its ratings  reflected additional considerations beyond OMB’s instructions. For  example, briefing slides prepared for DOD’s 2011 CIO rating exercise  identified the need to “balance” CIO ratings, and advised that yellow or  red ratings could lead to an OMB review. In addition, DOD officials  explained that the department rated investments green (or low risk) if the  risk of the investment not meeting its performance goals is low; yellow (or  medium risk) if the investment is facing difficulty; and red (high risk) only if  the department planned to restructure or cancel the investment, or had  already done so. DOD officials further stated that their CIO ratings  provide a measured assessment of how DOD believes an investment will  perform in the future.
Although the CIO ratings submitted by DOD to the Dashboard are  consistent with their ratings approach, they do not reflect other available  information about the risk of these investments. As we previously noted,  none of DOD’s investments that were active in March 2012 were rated as  high risk, and approximately 85 percent were rated as either low risk or  moderately low risk throughout their time on the Dashboard. However,  these ratings did not always reflect significant schedule delays, cost  increases, and other weaknesses identified for certain investments in our  recent reviews, or problems with those investments identified in a recent  report by the DOD Inspector General.
Based on the department’s long-standing difficulties with such programs,  we designated DOD business systems modernization as a high-risk area  in 1995 and it remains a high-risk area today. More recently, we reported  weaknesses in several of the department’s business system  investments.effectively ensured that these systems would deliver capabilities on time  and within budget; that acquisition delays required extended funding for  duplicative legacy systems; that delays and cost overruns were likely to  erode the cost savings these systems were to provide; and that,  ultimately, DOD’s management of these investments was putting the  department’s transformation of business operations at risk.
Specifically, we reported that the department had not  Although the following selected examples of DOD investments  experienced significant performance problems and were included with  those considered to be high-risk business system investments in our  recent reviews of those systems, they were all rated low risk or  moderately low risk by the DOD CIO.
Air Force’s Defense Enterprise Accounting and Management  System (DEAMS): DEAMS is the Air Force’s target accounting  system designed to provide accurate, reliable, and timely financial  information. In early 2012, GAO reported that DEAMS faced a 2-year  deployment delay, an estimated cost increase of about $500 million  for an original life-cycle cost estimate of $1.1 billion (an increase of  approximately 45 percent), and that assessments by DOD users had  identified operational problems with the system, such as data  accuracy issues, an inability to generate auditable financial reports,  and the need for manual workarounds.Inspector General reported that the DEAMS’ schedule delays were  likely to diminish the cost savings it was to provide, and would  jeopardize the department’s goals for attaining an auditable financial  statement. DOD’s CIO rated DEAMS low risk or moderately low risk  from July 2009 through March 2012.
In July 2012, the DOD
Army’s General Fund Enterprise Business System (GFEBS):  GFEBS is an Army financial management system intended to improve  the timeliness and reliability of financial information and to support the  department’s auditability goals. In early 2012, we reported that  GFEBS faced a 10-month implementation delay, and that DOD users  reported operational problems, including deficiencies in data accuracy  and an inability to generate auditable financial reports. These concerns were reiterated by the DOD Inspector General in July 2012.  DOD’s CIO rated GFEBS as moderately low risk from July 2009  through March 2012.
Army’s Global Combat Support System-Army (GCSS-Army):  GCSS-Army is intended to improve the Army’s supply chain  management capabilities and provide accurate equipment readiness  status reports, among other things. In March 2012, we reported that  GCSS-Army was experiencing a cost overrun of approximately $300  million on an original life-cycle cost estimate of $3.9 billion (an  increase of approximately 8 percent) and a deployment delay of  approximately 2 years. DOD rated GCSS-Army as low or moderately low risk from July 2009 through March 2012.
Explanations submitted by DOD with the CIO ratings for these  investments did not provide meaningful insight for why they were rated at  the lowest risk levels in the face of known issues. DOD officials told us  that they rated these investments as low risk because, in their view, the  cost and schedule variances listed above did not constitute significant  risks. Officials explained that: (1) the cost variances were not that large  compared to DOD’s overall size and large amount of IT spending; (2) the  schedule variance needed to be understood in the context that the  average DOD large-scale IT program takes 7 years (or 84 months) to  implement; and (3) that each of those programs had risk mitigation plans  in place. However, the first two reasons are inconsistent with DOD’s own   which recommends that risks be assessed  risk management guidance,against the program’s own cost and schedule estimates, not other  department investments. In addition, completing risk mitigation plans  does not necessarily lower investment risk. DOD’s guidance calls for  implementing the mitigation plan and then reassessing resulting changes  to the risk. Even if the department adopts these elements of its own  guidance, the CIO’s evaluation will be incomplete unless it also reflects  the assessments of investment performance and risks identified by us  and others. Until the department does so, CIO ratings for DOD’s  Dashboard investments may not be sufficiently accurate or useful for its  TechStat sessions or OMB’s management and oversight.
section_title: CIO Ratings Present Benefits and Challenges for Agencies’ Investment Management
paragraphs: Selected agencies identified various benefits associated with performing  CIO ratings and Dashboard reporting in general. Almost all of the  agencies (five of six) reported the following three benefits.
Increased quality of investment performance data. For example, one  agency also reported that the Dashboard has made information about  investments more understandable.
Greater transparency and visibility for CIOs and their staff into  investment- and program-level performance data. One agency  reported that its CIO was better able to conduct reviews with actual  investment numbers, as opposed to self-reported data presented by  the investment’s program managers. Agencies could also compare  their investments’ ratings to those of other agencies and departments.
Increased focus on project management practices. Two agencies  reported improved investment performance as a direct result of their  Dashboard rating and reporting activities; another stated that  Dashboard reporting supported and reinforced their existing IT  governance, capital planning, and program management processes.
Some of these benefits were interrelated. Several agencies viewed the  improved data quality as a by-product of greater scrutiny brought about  by having to report such data to the Dashboard on a regular basis. One  agency response noted that their program managers were surprised to  see the extent to which investment data were visible to the public, and  that this visibility motivated their staff to provide accurate and timely data  (which has improved data quality). Another agency noted that the visibility  of the IT Dashboard has increased awareness among investment and  project managers about the need to improve the planning of project  activities and the definition of operational performance metrics (which  support program management).
Nevertheless, agencies also identified challenges associated with  producing and reporting CIO ratings. First, three agencies reported a  challenge associated with the time and effort required to gather, validate,  and gain internal approval for CIO ratings and other data reported to the  Dashboard. For example, one agency reported that, due to the number of  organizations involved and the number of investments being evaluated, it  generally takes 90 to 120 days to develop and update its CIO ratings. The  agency further reported that this effort was separate from (and in addition  to) time it already spends on its own internal processes for managing and  overseeing acquisition programs.
Second, four of the six agencies identified challenges with the number of  changes OMB has made to the Dashboard, as well as with the timeliness  and clarity of OMB’s communication regarding those changes. For  example, officials at one agency commented that the frequency of  changes has actually hindered their efforts to improve data quality, since  errors sometimes resulted when it adapted to changes required by OMB.  Officials at another agency stated that OMB allowed insufficient time for  agencies to test their systems’ interfaces with the Dashboard when  changes were made, which they said resulted in data errors and  challenges for staff. These officials also noted that OMB’s guidance for  agency submissions has, at times, not matched the technical data  schemas implemented by OMB, impeding agencies’ efforts to  successfully upload their data. An OMB staff member commented that  their office releases changes to the Dashboard as early in the fiscal year  as possible to give agencies time to adjust and that OMB announces  planned changes to agencies before they are implemented via the  Dashboard’s interagency web portal. OMB has recently held meetings  with agency officials to discuss these issues and determine ways to better  communicate going forward.
Finally, one agency responded that while monthly updates to the  Dashboard have increased investment and project managers’ attention to  the performance of their investments and projects, this regular scrutiny  could encourage investment and project managers to “perform to the test”  rather than concentrate on effective investment and project management.  However, based on the interrelationships of the benefits of CIO ratings  identified by some agencies, the process of generating and reporting CIO  ratings does not have to be just a grading exercise. As previously noted,  the benefit of improved investment performance data for the CIO’s  investment evaluation can lead to more effective management, which  could, in turn, improve investment performance. Executives and staff who  can envision these results from the Dashboard’s CIO evaluations may be  less likely to view the additional time and effort required to generate the  CIO ratings as a challenge, but as an opportunity for more efficient and  effective management.
section_title: Conclusions
paragraphs: Since its inception in 2009, the Federal IT Dashboard has increased the  transparency of the performance of major federal IT investments. Its CIO  ratings, in particular, have improved visibility into changes in the risk  levels of agencies’ investments over time. Determining whether such  changes represent improvements or deficiencies in management and  oversight can be difficult without additional information on investment  performance and the rating process, but analyzing and reporting the  ratings for investments and agencies over time for the President’s budget  submission could help OMB ensure that risk is accurately assessed and  that patterns of risk deserving of special management attention are  identified.
DOD demonstrated one such pattern of interest in its CIO ratings. During  the 34-month life of the Dashboard, none of the 87 investments that were  active as of March 2012 were rated high risk or moderately high risk, and  approximately 85 percent of ratings were low risk or moderately low risk.  Although DOD implemented OMB’s broad instructions for producing CIO  ratings, it also considered how the ratings might increase the likelihood of  an OMB review of an investment and minimized the effects of significant  schedule delays and cost increases, which were identified in our reviews  and those of DOD’s Inspector General. As a result, DOD is masking  significant investment risks, has not employed its own risk management  guidance, and has not delivered the transparency intended by the  Dashboard. By incorporating the results of external reviews into its  evaluations, DOD can further improve the quality of the information on  which investment risk ratings are based.
Beyond the transparency they promote, CIO ratings present an  opportunity to improve the data and processes agencies use to assess  investment risk. Some agencies have already experienced collateral  benefits and management results from their risk evaluations. Continuing  focus from OMB and agencies on how to accurately portray and derive  value from the ratings and the associated processes could enable  agencies to experience such benefits.
section_title: Recommendations for Executive Action
paragraphs: To ensure that OMB’s preparation of the President’s budget submission  accurately reflects the risks associated with all major IT investments, we  are recommending that the Federal CIO analyze agency trends reflected  in Dashboard CIO ratings, and present the results of this analysis with the  President’s annual budget submission.
To ensure that DOD’s CIO evaluations of investment risk for its major IT  Dashboard investments reflect all available performance assessments  and are consistent with the department’s own guidance for managing risk,  we are recommending that the Secretary of Defense direct the  department’s CIO to reassess the department’s considerations for  assigning CIO risk levels for Dashboard investments, including  assessments of investment performance and risk from outside the  programs, and apply the appropriate elements of the department’s risk  management guidance to OMB’s evaluation factors in determining CIO  ratings.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our report to the six agencies selected for our  review and to OMB. In oral comments, staff from OMB’s Office of E- Government & Information Technology stated that OMB concurred with  our recommendation that the Federal CIO analyze agency trends  reflected in Dashboard CIO ratings and present the results of this analysis  with the President’s annual budget submission. OMB staff also provided  technical comments, which we incorporated as appropriate. In a written  response, DOD’s Deputy Chief Information Officer for Information  Enterprise agreed with our recommendation that the department’s CIO  reassess considerations for assigning CIO risk levels for Dashboard  investments, and committed to updating the department’s CIO ratings  process to better report risk and improve the timeliness and transparency  of reporting. DOD’s written response is reprinted in Appendix IV. Officials  at DOI provided technical comments, which we incorporated as  appropriate. The remaining agencies had no comment on the draft report.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees; the Secretaries of Defense, Interior, Homeland Security,  Health and Human Services, the Director of the National Science  Foundation, the Director of the Office of Personnel Management, the  Director of the Office of Management and Budget; and other interested  parties. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact David A. Powner at (202) 512-9286 or by e-mail at  pownerd@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) characterize the Chief Information Officer (CIO)  ratings for selected federal agencies’ information technology (IT)  investments as reported over time on the federal IT Dashboard; (2)  determine how agencies’ approaches for assigning and updating CIO  ratings vary; and (3) describe the benefits and challenges associated with  agencies’ approaches to the CIO rating.
To establish the scope of our review, we downloaded and examined data  on total IT spending for fiscal year 2011 for the 27 agencies reported on  the IT Dashboard. (The Office of Management and Budget (OMB)  extracts these data based on exhibit 300 forms submitted by each  agency.) We then selected six agencies that spanned a range of IT  spending for fiscal year 2011, including the three highest spending  agencies, two of the lowest, and an agency in the middle. Collectively,  these agencies accounted for approximately $51 billion, or 65 percent, of  2011 spending on IT investments. The six agencies are the Department  of Defense, Department of Homeland Security, Department of Health and  Human Services, Department of the Interior, National Science  Foundation, and Office of Personnel Management. The results in this  report represent only these agencies.
To address the first objective, we downloaded and examined the  Dashboard’s CIO ratings for all investments at the six agencies we  selected (a total of approximately 308 investments reported by these  agencies). To characterize the numbers and percentages of major IT  investments at each risk level at each of our subject agencies, we  analyzed, summarized, and—where appropriate—graphically depicted  average CIO ratings for investments by agencies over time during the  period from June 2009 to March 2012. Specifically, we compared the CIO  ratings in June 2009 (or whenever an individual investment was first  rated) up through and including each investment’s rating as of March  2012 and summarized the data by agency. To describe whether CIO  ratings indicated higher or lower investment risk over time, we calculated  the numbers and percentages of investments (by agency and collectively  for all the agencies) that maintained a constant rating over the entire  performance period, and those that experienced a change to their CIO  rating in at least one rating period. Then we analyzed the subset of  investments that experienced at least one changed rating and compared  the first CIO rating with the latest CIO rating (no later than March 2012) to  determine the numbers and percentages of investments (by agency and  collectively for all the agencies) that experienced a net rating increase, a  net rating decrease, or no net change. We also examined the comments  provided with the ratings to determine whether such comments were  useful in understanding the ratings. We presented our results to each  agency and OMB and solicited their input, explanations for the results,  and additional corroborating documentation, where appropriate.
To address our second objective, we reviewed available documentation,  obtained written responses to questions we posed to all agencies, and  interviewed OMB and agency officials to determine their policies and  practices related to assigning and updating the CIO ratings and related  data for the Dashboard. Specifically, we gathered descriptions about the  data, participants, and processes used to generate CIO ratings for  investments; when and under what circumstances each agency updates  its ratings; the specific factors agencies used in assigning their ratings;  and the reason(s) for their approaches to assigning and reporting the  ratings. We reviewed our results with agency officials to ensure that our  presentation of their approach was accurate. In addition, we utilized our  prior work and a report by the Department of Defense’s Office of the  Inspector General related to the department’s major IT investments. We  compared the findings in these reports to the CIO ratings the department  submitted to the Dashboard for investments that had been rated  consistently low or moderately low risk, and discussed our results with  department officials.
To address our third objective, we reviewed written and oral descriptions  of the benefits and challenges that agencies and OMB have experienced  in developing, submitting, updating, and utilizing CIO ratings. We sought  specific examples, corroborating documentation, and causal factors,  where available. After obtaining this information from individual agencies,  we compared their responses to identify benefits and challenges common  to multiple agencies and applied our judgment in determining whether any  additional benefits or challenges were present.
We conducted this performance audit from January 2012 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Risk Levels for Investments at Selected Agencies, as of March 2012
paragraphs: The table below lists the total number of major information technology (IT)  investments rated on the federal IT Dashboard as of March 2012 for each  agency selected for this review, with the numbers of investments rated at  each of the risk levels specified by the Office of Management and Budget  (OMB) for the chief information officer (CIO) rating. The last line in the  table reports each agency’s total budget for fiscal year 2012 for their  major IT investments, as also reported on the Dashboard in March 2012.
section_title: Appendix III: CIO Ratings and Net Changes for Major IT Investments at Selected Agencies, June 2009 through March 2012
paragraphs: This appendix provides additional information about chief information  officer (CIO) ratings for major IT information technology (IT) investments  at each of the agencies selected for this review. The first figure for each  agency depicts the number of investments at each rating level for the end  of each month, as reported on the federal IT Dashboard. The second  figure depicts the number of investments whose risk level demonstrated a  net increase, net decrease, no net change, or remained constant during  the investment’s entire time on the Dashboard.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff also made key  contributions to this report: Paula Moore (Assistant Director), Neil  Doherty, Lynn Espedido, Rebecca Eyler, Kate Feild, Dan Gordon,  Andrew Stavisky, Sonya Vartivarian, Shawn Ward, Kevin Walsh, Jessica  Waselkow, and Monique Williams.