Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies
Released: 2012-11-15T00:00:00

Letter

Background

The Clinger-Cohen Act of 1996 requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by federal agencies and report to Congress on the net program performance benefits achieved as a result of these IT investments. Further, the act places responsibility for managing investments with the heads of agencies and establishes CIOs to advise and assist agency heads in carrying out this responsibility.

OMB established the Management Watch List in 2003 to help carry out its oversight role. The Management Watch List included mission-critical projects that needed to improve performance measures, project management, IT security, or overall justification for inclusion in the President’s budget submission. Further, in August 2005, OMB established a High-Risk List, which consisted of projects identified by federal agencies, with OMB’s input, as requiring special attention from oversight authorities and the highest levels of agency management. Between 2005 and 2009, OMB described its efforts to monitor and manage risky federal IT investments in the annual budget submission.

Over the past several years, we have reported and testified on OMB’s initiatives to highlight troubled IT projects, justify investments, and use project management tools. For instance, in 2006 we recommended that OMB develop a single aggregated list of high-risk projects and their deficiencies and use that list to report to Congress on progress made in correcting high-risk problems. As a result, OMB started publicly releasing aggregate data on its Management Watch List and disclosing the projects’ deficiencies. Moreover, between 2007 and 2009, the President’s budget submission included an overview of investment performance over several budget years, including the number of federal IT projects in need of management attention. Such information helped Congress stay better With informed of high-risk projects and make related funding decisions.the advent of its IT Dashboard in 2009, OMB discontinued this type of reporting in the fiscal year 2010 budget submission.

OMB’s Dashboard Provides Visibility into the Performance of Federal IT Investments

to allow OMB; other oversight bodies, including Congress; and the general public to hold the government agencies accountable for progress and results.

OMB reported on plans and implementation progress for this management tool in the “Analytical Perspectives” section of the President’s budget submissions for fiscal years 2012 and 2013, including planned updates to the Dashboard during 2012 to support closer executive oversight and intervention to prevent schedule delays, cost overruns, and failures in delivering key functionality needed by federal programs. For example, it reported using the Dashboard to identify investments for TechStat reviews.

The Dashboard visually presents performance ratings for agencies overall and for individual investments using metrics that OMB has defined—cost, schedule, and CIO evaluation. The website also provides the capability to download certain data. Figure 1 is an example of an agency’s (OPM) portfolio page as recently depicted on the Dashboard.

The Dashboard’s data spans the period from June 2009 to the present, and is based, in part, on each agency’s exhibit 53 and exhibit 300 to OMB, as well as on agency assessments and submissionssupporting information on each investment. Over the life of the Dashboard, OMB has issued guidance to agencies on, among other things, what data to report, how those data need to be structured and formatted for upload to the Dashboard, and procedures for using the Dashboard’s submission tools. For instance, OMB instructed agencies to update and submit investment cost and schedule data monthly. OMB has made various changes to the organization, available data, and features of the Dashboard over time, including improvements to Dashboard calculations to incorporate the variance of “in progress” milestones rather than just “completed” milestones; web pages containing data on historical ratings and rebaselines of eliminated and downgraded investments; added data on awarded contracts, with links to USAspending.gov; release of IT Dashboard source code and documentation to an open source hosting provider; enhancements to baseline history, which give users the ability to see field-by-field changes for each rebaseline; a mechanism for OMB analysts to provide feedback to agencies on mobile-friendly formatting of Dashboard displays.

Once OMB has received agency-reported investment data, it converts these into investment performance ratings for display on the dashboard according to calculations and protocols described on its website. OMB assigns cost and schedule performance ratings by using data submitted by agencies to calculate variances between the planned cost or schedule targets and the actual or projected cost or schedule values. OMB converts these variances to percentages, and assigns the ratings to be presented on the Dashboard within three ranges, red, yellow, and green, as shown in table 1.

Although the thresholds for assigning cost and schedule variance ratings has remained constant over the life of the Dashboard, the cost and schedule data agencies are required to submit have changed in several ways, as have the variance calculations. For example, in response to our recommendations (further discussed in the next section), OMB changed how the Dashboard calculates the cost and schedule ratings in July 2010, to include “in progress” milestones rather than just “completed” ones for more accurate reflection of current investment status.

GAO Has Previously Reported on the Dashboard’s Value, Data Quality, and Recent Improvements

We have previously reported that OMB has taken significant steps to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard, and by improving the accuracy of investment ratings. We also found issues with the accuracy and data reliability of cost and schedule data, and recommended steps that OMB should take to improve these data.

In July 2010, we reportedDashboard were not always accurate for the investments we reviewed, because these ratings did not take into consideration current performance. As a result, the ratings were based on outdated information. We recommended that OMB report on its planned changes to the Dashboard to improve the accuracy of performance information and provide guidance to agencies to standardize milestone reporting. OMB agreed with our recommendations and, as a result, updated the Dashboard’s cost and schedule calculations to include both ongoing and that the cost and schedule ratings on OMB’s completed activities. Similarly, in March 2011, we reportedhad initiated several efforts to increase the Dashboard’s value as an oversight tool, and had used its data to improve federal IT management. We also reported, however, that agency practices and the Dashboard’s calculations contributed to inaccuracies in the reported investment performance data. For instance, we found missing data submissions or erroneous data at each of the five agencies we reviewed, along with instances of inconsistent program baselines and unreliable source data. As a result, we recommended that the agencies take steps to improve the accuracy and reliability of their Dashboard information, and that OMB improve how it rates investments relative to current performance and schedule variance. Most agencies generally concurred with our recommendations; OMB agreed with our recommendation for improving ratings for schedule variance. It disagreed with our recommendation to improve how it reflects current performance in cost and schedule ratings, but more recently made changes to Dashboard calculations to address this while also noting challenges in comprehensively evaluating cost and schedule data for these investments.

More recently, in November 2011, we reported investment cost and schedule ratings had improved since our July 2010 report because OMB had refined the Dashboard’s cost and schedule calculations. Most of the ratings for the eight investments we reviewed were accurate, although we noted that more could be done to inform oversight and decision making by emphasizing recent performance in the ratings. We recommended that the General Services Administration comply with OMB’s guidance for updating its ratings when new information becomes available (including when investments are rebaselined) and the agency concurred. Since we previously recommended that OMB improve how it rates investments, we did not make any further recommendations.

CIO Ratings Are Important for OMB’s IT Reform and Management Initiatives

GAO-12-210. investment into a color for depiction on the Dashboard. An OMB staff member from the Office of E-Government and Information Technology noted that the CIO rating should be a current assessment of future performance based on historical results and is the only Dashboard performance indicator that has been defined and produced the same way since the Dashboard’s inception. According to OMB’s instructions, a CIO rating should reflect the level of risk facing an investment on a scale from 1 (high risk) to 5 (low risk) relative to that investment’s ability to accomplish its goals. Each agency CIO is to assess their IT investments against a set of six preestablished evaluation factors identified by OMB (shown in table 2) and then assign a rating of 1 to 5 based on his or her best judgment of the level of risk facing the investment. According to an OMB staff member, agency CIOs are responsible for determining appropriate thresholds for the risk levels and for applying them to investments when assigning CIO ratings.

OMB recommends that CIOs consult with appropriate stakeholders in making their evaluation, including Chief Acquisition Officers, program managers, and other interested parties. Ultimately, CIO ratings are assigned colors for presentation on the Dashboard, according to the five- point rating scale, as illustrated in table 3.

OMB has made the CIO’s evaluation and rating a key component of its larger IT Reform Initiative and 25 Point Plan. In its plan, OMB reported that it used agencies’ CIO ratings to select investments for the TechStat review sessions it conducted between 2010 and 2011. These sessions are data-driven assessments of IT investments by agency leaders that are intended to result in concrete action to improve performance. OMB reported that the TechStats it conducted on selected investments resulted in approximately $3 billion in reduced costs. Building on the results of those sessions, the plan articulates a strategy for strengthening IT governance, in part, through the adoption of the TechStat model by federal agencies. In conducting TechStats, agencies are to rely, in part, on CIO ratings from the IT Dashboard. The TechStat Toolkit, developed by OMB and a task force of agency leads, provides sample questions regarding an investment’s CIO rating and associated risks for use in TechStat sessions.

Furthermore, OMB issued guidance in August 2011other things, that agency CIOs shall be held accountable for the performance of IT program managers based on their governance process and the data reported on the IT Dashboard, which includes the CIO that stated, among rating. According to OMB, the addition of CIO names and photos on Dashboard investments is intended to highlight this accountability and link it to the Dashboard’s reporting on investment performance. Figure 2 illustrates the CIO rating information presented on the Dashboard for an example IT investment.

CIOs Rated Most IT Investments as Low Risk or Moderately Low Risk; Agencies Had Mixed Results in Reducing Higher Risk Levels

As of March 2012, CIO ratings for most investments listed on the Dashboard for the six agencies we reviewed indicated either low risk or moderately low risk (223 out of 313 investments across all the selected agencies). High risk or moderately high risk ratings were assigned to fewer investments (12 out of 313 investments across all the selected agencies). Figure 3 presents the total number of IT investments rated on the Dashboard for each of the selected agencies according to their risk levels, as of March 2012, and illustrates the predominance of low risk investments for the agencies in our review. The figure also reports agencies’ budgets for their major IT investments for fiscal year 2012, as presented on the Dashboard.

Historically, over the life of the Dashboard from June 2009 to March 2012, low or moderately low risk ratings accounted for at least 66 percent of all ratings at five of the six agencies (the exception is DHS with 51 percent).

Medium risk ratings accounted for between 0 to 38 percent of all reported ratings across agencies during this period. The maximum percentage of ratings in the high risk or moderately high risk categories for any agency during this 34-month period was 12 percent, with two agencies—DOD DOD stated in written and NSF—reporting no high risk investments.comments that this was because they did not deem any of their investments to be high risk. (DOD’s investment risks are further discussed in the next section.) An NSF official from the Division of Information Systems stated that there were no high risk investments because most of their investments were in the operations and maintenance phase. Table 4 presents the average composition of ratings for each agency during the reporting period of June 2009 to March 2012. Appendix III depicts each agency’s CIO ratings by risk level on a monthly basis during the reporting period.

Overall, the CIO rating remained constant for 147 of 313 investments that were active as of March 2012 (about 47 percent of the investments we reviewed). These investments were rated at the same risk level during every rating period (see fig. 4).

Four of the six agencies did not change the CIO rating for a majority of their investments (excluding any investments that were downgraded or eliminated) during the time frame we examined. In contrast, the other two agencies—OPM and HHS—changed the CIO rating for more than 70 percent of their investments at least once between the investment’s initial rating and the rating reported as of March 2012. Table 5 lists the number of each agency’s investments whose ratings were constant and changed over time.

Agencies offered several reasons for why many investments had no changes in their CIO ratings during their entire time on the Dashboard. Five of the six selected agencies indicated that many investments were in a steady-state or operations and maintenance phase with no new development. One agency reported that their investments’ CIO ratings remained constant because the investments consistently met all requirements and deadlines and were using project management best practices.

The agencies we reviewed showed mixed results in reducing the number For investments of higher risk investments during the rating period.whose rating changed at least once during the period, 40 percent (67 investments) received a lower risk rating in March 2012 than they received initially, 41 percent of investments (68 investments) received a higher risk rating, and the remaining 19 percent (31 investments) received the same rating in March 2012 as they had initially received, despite whatever interim changes may have occurred (i.e., there was no “net” change to their reported risk levels). (See fig. 5.)

Two agencies—DHS and OPM—reported more investments with reduced risk in March 2012, as compared with initial ratings. The other four agencies reported more investments with increased risk. Table 6 presents net changes in risk levels at each of the selected agencies (among investments that were not downgraded or eliminated). Appendix III graphically summarizes these data for all six agencies.

Agencies most commonly cited additional oversight or program reviews as factors that contributed to decreased risk levels. Specifically, agencies commented that the CIO ratings and Dashboard reporting had spurred improved program management and risk mitigation. For example, one agency’s officials commented that the CIO now closely monitors the monthly performance and risk data generated by their investments, and that the additional oversight has brought about strengthened processes and more focused attention to issues.

In contrast, several agencies cited generally poor risk management at the investment level, the introduction of new investment/programs risks, as well as instances of poor project management as factors contributing to increased risk for investments. For example, one agency responded that internal review findings revealed new risks that caused an investment’s risk level to increase. Another agency’s officials reported that various technical issues caused one of their investments to fall behind schedule, thus increasing risk.

Both OMB and several agencies suggested caution in interpreting changing risk levels for investments. They noted that an increase in an investment’s risk level can sometimes indicate better management by the program or CIO because previously unidentified risks have been assessed and included in the CIO evaluation. Conversely, a decrease in an investment’s risk level may not indicate improved management if the data and analysis on which the CIO rating are based is incomplete, inconsistent, or outdated.

Further analysis of the characteristics and causes of Dashboard’s CIO ratings, and reporting on the patterns of risk within and among agencies, could provide Congress and the public with additional perspectives on federal IT investment risk over time. However, for the past four budget submissions, OMB has not summarized the extent of risk represented by major federal IT investments in the analysis it prepares annually for the President’s budget submission, as it did prior to the fiscal year 2010 submission. As a result, OMB is missing an opportunity to integrate such risk assessments into its evaluation of major capital investments in reporting to Congress.

Most Agencies Established Approaches for CIO Ratings That Follow OMB’s Instructions

OMB has provided agencies with instructions for assigning CIO ratings for the major IT investments reported on the Dashboard. Specifically, OMB’s instructions state that agency CIOs should rate each investment based on his/her best judgment and should include input from stakeholders, such as Chief Acquisition Officers, program managers, and others;  update the rating as soon as new information becomes available that might affect the assessment of a given investment; and  utilize OMB’s investment rating factors, including: risk management, requirements management, contractor oversight, historical performance, and human capital, as well as any other factors deemed relevant by the CIO.

Despite differences in the specific inputs and processes used, agencies generally followed OMB’s instructions for assigning CIO ratings. However, DOD’s ratings reflected additional considerations beyond OMB’s instructions and did not reflect available information about significant risks for certain investments. The sections that follow describe how each agency addressed OMB’s instructions.

Include input from stakeholders. Each of the six agencies we reviewed relied on stakeholder input, at least in part, when assigning CIO ratings. Agencies also cited a variety of review boards, data from program and financial systems, and other investment assessments as inputs to the rating. Table 7 describes the data and processes that agencies reported using when they derived their CIO ratings.

Update CIO ratings. All six agencies established guidelines for periodically reviewing and updating their CIO ratings. Specifically, HHS, NSF, DOI, and OPM reported that they update CIO ratings on a monthly basis. DOD has adopted a quarterly update cycle, although an official noted that the actual process of collecting information and evaluating investments for the ratings takes slightly longer than 3 months. DHS officials with the Office of the CIO stated that the frequency of its updates varies based on the risk level of an investment’s previous rating: investments with a previous CIO rating of green are to be reviewed semiannually; yellow investments are to be reviewed quarterly; and red investments are to be reviewed monthly.

Utilize OMB’s investment rating factors. Most of the selected agencies use OMB’s investment rating factors when evaluating their investments. Only one agency (HHS) does not use all of them. Specifically, an HHS official from the Office of the CIO told us that human capital issues are not explicitly covered in their CIO rating criteria because investment owners are to provide adequate IT human capital, and that these owners will reflect any issues that arise when providing input for the CIO rating.

Among the agencies we reviewed, DOD was unique in that its ratings reflected additional considerations beyond OMB’s instructions. For example, briefing slides prepared for DOD’s 2011 CIO rating exercise identified the need to “balance” CIO ratings, and advised that yellow or red ratings could lead to an OMB review. In addition, DOD officials explained that the department rated investments green (or low risk) if the risk of the investment not meeting its performance goals is low; yellow (or medium risk) if the investment is facing difficulty; and red (high risk) only if the department planned to restructure or cancel the investment, or had already done so. DOD officials further stated that their CIO ratings provide a measured assessment of how DOD believes an investment will perform in the future.

Although the CIO ratings submitted by DOD to the Dashboard are consistent with their ratings approach, they do not reflect other available information about the risk of these investments. As we previously noted, none of DOD’s investments that were active in March 2012 were rated as high risk, and approximately 85 percent were rated as either low risk or moderately low risk throughout their time on the Dashboard. However, these ratings did not always reflect significant schedule delays, cost increases, and other weaknesses identified for certain investments in our recent reviews, or problems with those investments identified in a recent report by the DOD Inspector General.

Based on the department’s long-standing difficulties with such programs, we designated DOD business systems modernization as a high-risk area in 1995 and it remains a high-risk area today. More recently, we reported weaknesses in several of the department’s business system investments.effectively ensured that these systems would deliver capabilities on time and within budget; that acquisition delays required extended funding for duplicative legacy systems; that delays and cost overruns were likely to erode the cost savings these systems were to provide; and that, ultimately, DOD’s management of these investments was putting the department’s transformation of business operations at risk.

Specifically, we reported that the department had not Although the following selected examples of DOD investments experienced significant performance problems and were included with those considered to be high-risk business system investments in our recent reviews of those systems, they were all rated low risk or moderately low risk by the DOD CIO.

Air Force’s Defense Enterprise Accounting and Management System (DEAMS): DEAMS is the Air Force’s target accounting system designed to provide accurate, reliable, and timely financial information. In early 2012, GAO reported that DEAMS faced a 2-year deployment delay, an estimated cost increase of about $500 million for an original life-cycle cost estimate of $1.1 billion (an increase of approximately 45 percent), and that assessments by DOD users had identified operational problems with the system, such as data accuracy issues, an inability to generate auditable financial reports, and the need for manual workarounds.Inspector General reported that the DEAMS’ schedule delays were likely to diminish the cost savings it was to provide, and would jeopardize the department’s goals for attaining an auditable financial statement. DOD’s CIO rated DEAMS low risk or moderately low risk from July 2009 through March 2012.

In July 2012, the DOD

Army’s General Fund Enterprise Business System (GFEBS): GFEBS is an Army financial management system intended to improve the timeliness and reliability of financial information and to support the department’s auditability goals. In early 2012, we reported that GFEBS faced a 10-month implementation delay, and that DOD users reported operational problems, including deficiencies in data accuracy and an inability to generate auditable financial reports. These concerns were reiterated by the DOD Inspector General in July 2012. DOD’s CIO rated GFEBS as moderately low risk from July 2009 through March 2012.

Army’s Global Combat Support System-Army (GCSS-Army): GCSS-Army is intended to improve the Army’s supply chain management capabilities and provide accurate equipment readiness status reports, among other things. In March 2012, we reported that GCSS-Army was experiencing a cost overrun of approximately $300 million on an original life-cycle cost estimate of $3.9 billion (an increase of approximately 8 percent) and a deployment delay of approximately 2 years. DOD rated GCSS-Army as low or moderately low risk from July 2009 through March 2012.

Explanations submitted by DOD with the CIO ratings for these investments did not provide meaningful insight for why they were rated at the lowest risk levels in the face of known issues. DOD officials told us that they rated these investments as low risk because, in their view, the cost and schedule variances listed above did not constitute significant risks. Officials explained that: (1) the cost variances were not that large compared to DOD’s overall size and large amount of IT spending; (2) the schedule variance needed to be understood in the context that the average DOD large-scale IT program takes 7 years (or 84 months) to implement; and (3) that each of those programs had risk mitigation plans in place. However, the first two reasons are inconsistent with DOD’s own which recommends that risks be assessed risk management guidance,against the program’s own cost and schedule estimates, not other department investments. In addition, completing risk mitigation plans does not necessarily lower investment risk. DOD’s guidance calls for implementing the mitigation plan and then reassessing resulting changes to the risk. Even if the department adopts these elements of its own guidance, the CIO’s evaluation will be incomplete unless it also reflects the assessments of investment performance and risks identified by us and others. Until the department does so, CIO ratings for DOD’s Dashboard investments may not be sufficiently accurate or useful for its TechStat sessions or OMB’s management and oversight.

CIO Ratings Present Benefits and Challenges for Agencies’ Investment Management

Selected agencies identified various benefits associated with performing CIO ratings and Dashboard reporting in general. Almost all of the agencies (five of six) reported the following three benefits.

Increased quality of investment performance data. For example, one agency also reported that the Dashboard has made information about investments more understandable.

Greater transparency and visibility for CIOs and their staff into investment- and program-level performance data. One agency reported that its CIO was better able to conduct reviews with actual investment numbers, as opposed to self-reported data presented by the investment’s program managers. Agencies could also compare their investments’ ratings to those of other agencies and departments.

Increased focus on project management practices. Two agencies reported improved investment performance as a direct result of their Dashboard rating and reporting activities; another stated that Dashboard reporting supported and reinforced their existing IT governance, capital planning, and program management processes.

Some of these benefits were interrelated. Several agencies viewed the improved data quality as a by-product of greater scrutiny brought about by having to report such data to the Dashboard on a regular basis. One agency response noted that their program managers were surprised to see the extent to which investment data were visible to the public, and that this visibility motivated their staff to provide accurate and timely data (which has improved data quality). Another agency noted that the visibility of the IT Dashboard has increased awareness among investment and project managers about the need to improve the planning of project activities and the definition of operational performance metrics (which support program management).

Nevertheless, agencies also identified challenges associated with producing and reporting CIO ratings. First, three agencies reported a challenge associated with the time and effort required to gather, validate, and gain internal approval for CIO ratings and other data reported to the Dashboard. For example, one agency reported that, due to the number of organizations involved and the number of investments being evaluated, it generally takes 90 to 120 days to develop and update its CIO ratings. The agency further reported that this effort was separate from (and in addition to) time it already spends on its own internal processes for managing and overseeing acquisition programs.

Second, four of the six agencies identified challenges with the number of changes OMB has made to the Dashboard, as well as with the timeliness and clarity of OMB’s communication regarding those changes. For example, officials at one agency commented that the frequency of changes has actually hindered their efforts to improve data quality, since errors sometimes resulted when it adapted to changes required by OMB. Officials at another agency stated that OMB allowed insufficient time for agencies to test their systems’ interfaces with the Dashboard when changes were made, which they said resulted in data errors and challenges for staff. These officials also noted that OMB’s guidance for agency submissions has, at times, not matched the technical data schemas implemented by OMB, impeding agencies’ efforts to successfully upload their data. An OMB staff member commented that their office releases changes to the Dashboard as early in the fiscal year as possible to give agencies time to adjust and that OMB announces planned changes to agencies before they are implemented via the Dashboard’s interagency web portal. OMB has recently held meetings with agency officials to discuss these issues and determine ways to better communicate going forward.

Finally, one agency responded that while monthly updates to the Dashboard have increased investment and project managers’ attention to the performance of their investments and projects, this regular scrutiny could encourage investment and project managers to “perform to the test” rather than concentrate on effective investment and project management. However, based on the interrelationships of the benefits of CIO ratings identified by some agencies, the process of generating and reporting CIO ratings does not have to be just a grading exercise. As previously noted, the benefit of improved investment performance data for the CIO’s investment evaluation can lead to more effective management, which could, in turn, improve investment performance. Executives and staff who can envision these results from the Dashboard’s CIO evaluations may be less likely to view the additional time and effort required to generate the CIO ratings as a challenge, but as an opportunity for more efficient and effective management.

Conclusions

Since its inception in 2009, the Federal IT Dashboard has increased the transparency of the performance of major federal IT investments. Its CIO ratings, in particular, have improved visibility into changes in the risk levels of agencies’ investments over time. Determining whether such changes represent improvements or deficiencies in management and oversight can be difficult without additional information on investment performance and the rating process, but analyzing and reporting the ratings for investments and agencies over time for the President’s budget submission could help OMB ensure that risk is accurately assessed and that patterns of risk deserving of special management attention are identified.

DOD demonstrated one such pattern of interest in its CIO ratings. During the 34-month life of the Dashboard, none of the 87 investments that were active as of March 2012 were rated high risk or moderately high risk, and approximately 85 percent of ratings were low risk or moderately low risk. Although DOD implemented OMB’s broad instructions for producing CIO ratings, it also considered how the ratings might increase the likelihood of an OMB review of an investment and minimized the effects of significant schedule delays and cost increases, which were identified in our reviews and those of DOD’s Inspector General. As a result, DOD is masking significant investment risks, has not employed its own risk management guidance, and has not delivered the transparency intended by the Dashboard. By incorporating the results of external reviews into its evaluations, DOD can further improve the quality of the information on which investment risk ratings are based.

Beyond the transparency they promote, CIO ratings present an opportunity to improve the data and processes agencies use to assess investment risk. Some agencies have already experienced collateral benefits and management results from their risk evaluations. Continuing focus from OMB and agencies on how to accurately portray and derive value from the ratings and the associated processes could enable agencies to experience such benefits.

Recommendations for Executive Action

To ensure that OMB’s preparation of the President’s budget submission accurately reflects the risks associated with all major IT investments, we are recommending that the Federal CIO analyze agency trends reflected in Dashboard CIO ratings, and present the results of this analysis with the President’s annual budget submission.

To ensure that DOD’s CIO evaluations of investment risk for its major IT Dashboard investments reflect all available performance assessments and are consistent with the department’s own guidance for managing risk, we are recommending that the Secretary of Defense direct the department’s CIO to reassess the department’s considerations for assigning CIO risk levels for Dashboard investments, including assessments of investment performance and risk from outside the programs, and apply the appropriate elements of the department’s risk management guidance to OMB’s evaluation factors in determining CIO ratings.

Agency Comments and Our Evaluation

We provided a draft of our report to the six agencies selected for our review and to OMB. In oral comments, staff from OMB’s Office of E- Government & Information Technology stated that OMB concurred with our recommendation that the Federal CIO analyze agency trends reflected in Dashboard CIO ratings and present the results of this analysis with the President’s annual budget submission. OMB staff also provided technical comments, which we incorporated as appropriate. In a written response, DOD’s Deputy Chief Information Officer for Information Enterprise agreed with our recommendation that the department’s CIO reassess considerations for assigning CIO risk levels for Dashboard investments, and committed to updating the department’s CIO ratings process to better report risk and improve the timeliness and transparency of reporting. DOD’s written response is reprinted in Appendix IV. Officials at DOI provided technical comments, which we incorporated as appropriate. The remaining agencies had no comment on the draft report.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees; the Secretaries of Defense, Interior, Homeland Security, Health and Human Services, the Director of the National Science Foundation, the Director of the Office of Personnel Management, the Director of the Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact David A. Powner at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) characterize the Chief Information Officer (CIO) ratings for selected federal agencies’ information technology (IT) investments as reported over time on the federal IT Dashboard; (2) determine how agencies’ approaches for assigning and updating CIO ratings vary; and (3) describe the benefits and challenges associated with agencies’ approaches to the CIO rating.

To establish the scope of our review, we downloaded and examined data on total IT spending for fiscal year 2011 for the 27 agencies reported on the IT Dashboard. (The Office of Management and Budget (OMB) extracts these data based on exhibit 300 forms submitted by each agency.) We then selected six agencies that spanned a range of IT spending for fiscal year 2011, including the three highest spending agencies, two of the lowest, and an agency in the middle. Collectively, these agencies accounted for approximately $51 billion, or 65 percent, of 2011 spending on IT investments. The six agencies are the Department of Defense, Department of Homeland Security, Department of Health and Human Services, Department of the Interior, National Science Foundation, and Office of Personnel Management. The results in this report represent only these agencies.

To address the first objective, we downloaded and examined the Dashboard’s CIO ratings for all investments at the six agencies we selected (a total of approximately 308 investments reported by these agencies). To characterize the numbers and percentages of major IT investments at each risk level at each of our subject agencies, we analyzed, summarized, and—where appropriate—graphically depicted average CIO ratings for investments by agencies over time during the period from June 2009 to March 2012. Specifically, we compared the CIO ratings in June 2009 (or whenever an individual investment was first rated) up through and including each investment’s rating as of March 2012 and summarized the data by agency. To describe whether CIO ratings indicated higher or lower investment risk over time, we calculated the numbers and percentages of investments (by agency and collectively for all the agencies) that maintained a constant rating over the entire performance period, and those that experienced a change to their CIO rating in at least one rating period. Then we analyzed the subset of investments that experienced at least one changed rating and compared the first CIO rating with the latest CIO rating (no later than March 2012) to determine the numbers and percentages of investments (by agency and collectively for all the agencies) that experienced a net rating increase, a net rating decrease, or no net change. We also examined the comments provided with the ratings to determine whether such comments were useful in understanding the ratings. We presented our results to each agency and OMB and solicited their input, explanations for the results, and additional corroborating documentation, where appropriate.

To address our second objective, we reviewed available documentation, obtained written responses to questions we posed to all agencies, and interviewed OMB and agency officials to determine their policies and practices related to assigning and updating the CIO ratings and related data for the Dashboard. Specifically, we gathered descriptions about the data, participants, and processes used to generate CIO ratings for investments; when and under what circumstances each agency updates its ratings; the specific factors agencies used in assigning their ratings; and the reason(s) for their approaches to assigning and reporting the ratings. We reviewed our results with agency officials to ensure that our presentation of their approach was accurate. In addition, we utilized our prior work and a report by the Department of Defense’s Office of the Inspector General related to the department’s major IT investments. We compared the findings in these reports to the CIO ratings the department submitted to the Dashboard for investments that had been rated consistently low or moderately low risk, and discussed our results with department officials.

To address our third objective, we reviewed written and oral descriptions of the benefits and challenges that agencies and OMB have experienced in developing, submitting, updating, and utilizing CIO ratings. We sought specific examples, corroborating documentation, and causal factors, where available. After obtaining this information from individual agencies, we compared their responses to identify benefits and challenges common to multiple agencies and applied our judgment in determining whether any additional benefits or challenges were present.

We conducted this performance audit from January 2012 to September 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Risk Levels for Investments at Selected Agencies, as of March 2012

The table below lists the total number of major information technology (IT) investments rated on the federal IT Dashboard as of March 2012 for each agency selected for this review, with the numbers of investments rated at each of the risk levels specified by the Office of Management and Budget (OMB) for the chief information officer (CIO) rating. The last line in the table reports each agency’s total budget for fiscal year 2012 for their major IT investments, as also reported on the Dashboard in March 2012.

Appendix III: CIO Ratings and Net Changes for Major IT Investments at Selected Agencies, June 2009 through March 2012

This appendix provides additional information about chief information officer (CIO) ratings for major IT information technology (IT) investments at each of the agencies selected for this review. The first figure for each agency depicts the number of investments at each rating level for the end of each month, as reported on the federal IT Dashboard. The second figure depicts the number of investments whose risk level demonstrated a net increase, net decrease, no net change, or remained constant during the investment’s entire time on the Dashboard.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contact name above, the following staff also made key contributions to this report: Paula Moore (Assistant Director), Neil Doherty, Lynn Espedido, Rebecca Eyler, Kate Feild, Dan Gordon, Andrew Stavisky, Sonya Vartivarian, Shawn Ward, Kevin Walsh, Jessica Waselkow, and Monique Williams.