IT Dashboard: Agencies Are Managing Investment Risk, but Related Ratings Need to Be More Accurate and Available
Released: 2014-01-13T00:00:00

Letter

Background

While IT can enrich people’s lives and improve organizational performance, we have previously reported that federal IT projects too frequently incur cost overruns and schedule slippages while contributing little to mission-related outcomes. For example, in July 2013, we testified that the federal government continued to spend billions of dollars on troubled IT investments and we identified billions of dollars worth of failed and challenged IT projects.

OMB plays a key role in overseeing how federal agencies manage their IT investments by working with them to better plan, justify, and determine how to manage them. Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT projects and how these funds are to be allocated. OMB also guides agencies in developing sound business cases for IT investments and establishing management processes for overseeing these investments throughout their life cycles. The scope of this undertaking is quite large: in planning for fiscal year 2014, 27 federal agencies reported plans to spend about $82 billion on IT.investments, $37.6 billion is to be spent on non-major IT investments, and $5.5 billion is to be spent on classified Department of Defense IT investments.

Of that, $38.7 billion is to be spent on major IT Within OMB, the Office of E-Government and Information Technology, headed by the Federal CIO, directs the policy and strategic planning of federal IT investments and is responsible for oversight of federal information technology spending.

OMB’s Dashboard Is Intended to Provide Visibility into the Performance of Federal IT Investments

In June 2009, OMB deployed a public Web site to further improve the transparency and oversight of agencies’ IT investments. Known as the IT Dashboard,performance data for over 700 major federal IT investments at 27 federal agencies, accounting for $38.7 billion of those agencies’ planned $82 billion budget for fiscal year 2014. According to OMB, these data are intended to provide a near-real-time perspective on the performance of these investments, as well as a historical perspective. Further, the public display of these data is intended to allow OMB; other oversight bodies, including Congress; and the general public to hold the government agencies accountable for progress and results. this site displays federal agencies’ cost, schedule, and The Dashboard visually presents performance ratings for agencies and for individual investments using metrics that OMB has defined—cost, schedule, and CIO evaluation. The Web site also provides the capability to download certain data. Figure 1 is an example of VA’s portfolio page as recently depicted on the Dashboard. (The ratings are explained in the narrative following the figure.)

The Dashboard’s data spans the period from its June 2009 inception to the present, and is based, in part, on agency assessments of individual investment performance and each agency’s budget submissions to OMB.

The cost and schedule data agencies are required to submit to the Dashboard have changed over time, as have the related calculations. For example, in response to our recommendations (further discussed in the following section), OMB changed how the Dashboard calculates the cost and schedule ratings in July 2010 to include “in progress” milestones rather than just “completed” ones for a more accurate reflection of current investment status. While the required cost and schedule data have changed, the thresholds for assigning cost and schedule ratings have remained constant over the life of the Dashboard. Specifically, the Dashboard assigns rating colors (red, yellow, green) based on agency- submitted cost and schedule variances, as shown in table 1.

Unlike the variance-based cost and schedule ratings, the Dashboard’s “Investment Evaluation by Agency CIO” (also called the CIO rating) is determined by agency officials. According to OMB’s instructions, each agency CIO is to assess his or her IT investments against a set of six pre- established evaluation factors identified by OMB (shown in table 2) and then assign a rating of 1 (high risk) to 5 (low risk) based on the CIO’s best judgment of the level of risk facing the investment.

OMB recommends that CIOs consult with appropriate stakeholders in making their evaluation, including Chief Acquisition Officers, program managers, and other interested parties. According to an OMB staff member, agency CIOs are responsible for determining appropriate thresholds for the risk levels and for applying them to investments when assigning CIO ratings. OMB requires agencies to update these ratings as soon as new information becomes available which will affect an investment’s assessment. After agencies assign a level of risk to each investment, the Dashboard assigns colors to CIO ratings according to a five-point scale, as illustrated in table 3.

An OMB staff member from the Office of E-Government and Information Technology noted that the CIO rating should be a current assessment of future performance based on historical results and is the only Dashboard performance indicator that has been defined and produced the same way since the Dashboard’s inception.

Furthermore, OMB issued guidance in August 2011other things, that agency CIOs shall be held accountable for the performance of IT program managers based on their governance process and the data reported on the IT Dashboard, which includes the CIO rating. According to OMB, the addition of CIO names and photos on Dashboard investments is intended to highlight this accountability and link it to the Dashboard’s reporting on investment performance.

GAO Has Previously Reported on the Dashboard’s Value, Data Quality, and Improvements

We have previously reported that OMB has taken significant steps to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard, and by improving the accuracy of investment ratings. We also found issues with the accuracy and data reliability of cost and schedule data, and recommended steps that OMB should take to improve these data.

In July 2010, we reportedOMB’s Dashboard were not always accurate for the investments we that the cost and schedule ratings on reviewed, because these ratings did not take into consideration current performance. As a result, the ratings were based on outdated information. We recommended that OMB report on its planned changes to the Dashboard to improve the accuracy of performance information and provide guidance to agencies to standardize milestone reporting. OMB agreed with our recommendations and, as a result, updated the Dashboard’s cost and schedule calculations to include both ongoing and completed activities.

Similarly, our report in March 2011 noted that OMB had initiated several efforts to increase the Dashboard’s value as an oversight tool, and had used its data to improve federal IT management. We also reported, however, that agency practices and the Dashboard’s calculations contributed to inaccuracies in the reported investment performance data. For instance, we found missing data submissions or erroneous data at each of the five agencies we reviewed, along with instances of inconsistent program baselines and unreliable source data. As a result, we recommended that the agencies take steps to improve the accuracy and reliability of their Dashboard information, and that OMB improve how it rates investments relative to current performance and schedule variance. Most agencies generally concurred with our recommendations and three have taken steps to address our recommendation. OMB agreed with our recommendation for improving ratings for schedule variance. It disagreed with our recommendation to improve how it reflects current performance in cost and schedule ratings, but more recently made changes to Dashboard calculations to address this while also noting challenges in comprehensively evaluating cost and schedule data for these investments.

We also reported on OMB’s guidance to agencies for reporting their IT investments in September 2011 and found that it did not ensure complete reporting. Specifically, agencies differed on what investments they included as an IT investment. Among other things, we recommended that OMB clarify its guidance on reporting IT investments to specify whether certain types of systems—such as space systems—are to be included. OMB did not agree that further efforts were needed to clarify reporting in regard to the types of systems.

Subsequently, in November 2011, we noted that the accuracy of investment cost and schedule ratings had improved since our July 2010 report because OMB had refined the Dashboard’s cost and schedule calculations. Most of the ratings for the eight investments we reviewed as part of our November 2011 report were accurate, although we noted that more could be done to inform oversight and decision making by emphasizing recent performance in the ratings. We recommended that the General Services Administration comply with OMB’s guidance for updating its ratings when new information becomes available (including when investments are rebaselined). The agency concurred and has since taken actions to address this recommendation. Since we previously recommended that OMB improve how it rates investments, we did not make any further recommendations.

More recently, in October 2012 we reported that CIOs at six agencies rated a majority of investments listed on the federal IT Dashboard as low or moderately low risk from June 2009 through March 2012. Additionally, two agencies, the Department of Defense and the National Science Foundation, rated no investments as high or moderately high risk during this time period. Agencies generally followed OMB’s instructions for assigning CIO ratings, although the Department of Defense’s ratings were unique in reflecting additional considerations, such as the likelihood of OMB review. Most of the selected agencies reported various benefits associated with producing and reporting CIO ratings, such as increased quality of their performance data and greater transparency and visibility of investments. We recommended that OMB analyze agencies’ investment risk over time as reflected in the Dashboard’s CIO ratings and present its analysis with the President’s annual budget submission, and that the Department of Defense ensure that its CIO ratings reflect available investment performance assessments and its risk management guidance. Both agencies concurred with our recommendations, and OMB reported on CIO rating trends in the fiscal year 2014 budget submission; however, the Department of Defense has not updated any of its ratings since September 2012.

Further, we studied OMB’s efforts to help agencies address IT projects with cost overruns, schedule delays, and performance shortfalls in June 2013. In particular, we reported that OMB used CIO ratings from the Dashboard, among other sources, to select at- risk investments for reviews known as TechStats. OMB initiated these reviews in January 2010 to further improve investment performance, and subsequently incorporated the TechStat model into its 25-point plan for reforming federal IT management. We reported that OMB and selected agencies had held multiple TechStat sessions but additional OMB oversight was needed to ensure that these meetings were having the appropriate impact on underperforming projects and that resulting cost savings were valid. Among other things, we recommended that OMB require agencies to address their highest-risk investments and to report on how they validated the outcomes. OMB generally agreed with our recommendations, and stated that it and the agencies were taking appropriate steps to address them.

CIOs at Selected Agencies’ Rated Majority of IT Investments as Low or Moderately Low Risk

As of August 2013, CIOs at the eight selected agencies rated 198 of their 244 major investments listed on the Dashboard as low risk or moderately low risk. Of the remaining 46 investments, 41 were rated medium risk, and 5 were rated high risk or moderately high risk. Historically, over the life of the Dashboard from June 2009 to August 2013, low or moderately low risk investments accounted for an average of 72 percent of all ratings at the eight agencies, medium risk ratings an average of 22 percent, and high risk or moderately high risk ratings an average of 6 percent. The CIO ratings and associated number of investments at each the eight agencies as reported on the Dashboard over the past 4 years are presented in figure 2.

When comparing the investments’ first and most recent ratings, the agencies we reviewed showed generally positive results. Specifically, of the 383 total investments listed on the Dashboard from June 2009 to August 2013 for the selected agencies, 121 increased (meaning risk is lower) and 81 decreased (meaning risk is higher). Additionally, a significant portion of investments had returned to their original rating (74) and about one third of investments’ ratings had never changed (107). (See fig. 3.)

When considered individually, five of the eight selected agencies— Agriculture, Commerce, Energy, Treasury, and VA—had more investments’ ratings increase than decrease, whereas the other three— Justice, Transportation, and SSA—had more investments decrease.

Figure 4 presents the net changes in investments’ ratings for each agency during the reporting period of June 2009 to August 2013.

The agencies cited additional oversight or program reviews as factors that contributed to improved ratings. Furthermore, Agriculture and Commerce both attributed improved ratings to enhanced timeliness and sufficiency of investment reporting. Both of these agencies factor investment reporting into CIO ratings and increased ratings as better performance and risk information was provided by investment management. This suggests that caution should be exercised when interpreting changing risk levels for investments, as rating changes by agencies may not be entirely due to investment performance. Agencies also commented that the CIO ratings and Dashboard reporting had spurred improved investment management and risk mitigation.

Additionally, the total number of investments that the eight agencies reported on the Dashboard has varied over time, which impacts the number of investments receiving CIO ratings (see fig. 5).

The variation in the number of investments reported on the Dashboard can be attributed, in part, to agencies’ ability to add, downgrade, and remove investments throughout the annual federal budget process. However, as we concluded in September 2011, OMB’s guidance also did not ensure complete reporting of IT investments by agencies. Specifically, we found that OMB’s definition of an IT investment is broad, and the 10 agencies we evaluated differed on what systems they included as IT investments. For example, 5 of the 10 agencies we reviewed consistently considered investments in research and development systems as IT, and 5 did not. Consequently, we recommended that OMB clarify its guidance on reporting IT investments to specify whether certain types of systems—such as space systems—were to be included. OMB did not agree that further efforts were needed to clarify reporting in regard to the types of systems.

Now, 2 years later, and given the continuing lack of clarity in investment reporting guidance, agencies have elected to withdraw investments from the Dashboard that are clearly IT. Because we have continued to identify inappropriate reclassifications of IT investments, we continue to believe this recommendation has merit. For example, as part of the most recent budget process, Energy officials reported several of Energy’s supercomputer investments as facilities rather than IT, thus removing those investments from the Dashboard and accounting for a portion of the recent decrease in investments. Energy officials also stated that OMB approved this change. These investments account for $368 million, or almost 25 percent, of Energy’s fiscal year 2012 IT spending, and include the National Nuclear Security Administration’s Sequoia system, which Energy reported as the most powerful computing system in the world as of June 2012. According to Energy officials, these investments were recategorized because they include both supercomputers and laboratory facilities (which are not IT).

As another example, the Deputy Secretary of Commerce issued a directive in September 2012 which resulted in the removal of Commerce’s satellite investments from the Dashboard. As we found in 2011, Commerce only reported the ground systems of a spacecraft as IT investments, and not the technology components on the spacecraft itself. With this directive, Commerce removed the ground-based portions from IT investment reporting as well, accounting for $447 million, or 26 percent, of the department’s fiscal year 2012 IT spending. In making this decision, Commerce determined that it needed to refocus oversight efforts to a more appropriate level and consequently minimized the role of the CIO and others in the oversight of satellites. A Commerce official stated that Commerce plans to exclude all such investments from the department’s fiscal year 2015 IT budget submission.

Clinger-Cohen Act of 1996 (40 U.S.C. § 11101(6)). and transmit satellite data. A staff member from the Office of E- Government stated that OMB could not stop agencies from making such recategorizations and that OMB had no control over such agency decisions. By gathering incomplete information on IT investments, OMB increases the risk of not fulfilling its oversight responsibilities, of agencies making inefficient and ineffective investment decisions, and of Congress and the public being misinformed as to the performance of federal IT investments.

As part of the budget process, OMB is required to analyze, track, and evaluate the risks and results of all major IT investments. The Dashboard gives the public access to the same tools and analysis that the government uses in conducting this performance oversight. According to OMB, Dashboard data are intended to provide a near-real-time perspective on the performance of major investments. http://www.itdashboard.gov/faq. year, OMB decreases the utility of the Dashboard as a tool for greater IT investment oversight and transparency.

Most CIO Ratings Are Consistent with Reported Investment Risk

Of the 80 selected investments at the eight agencies we reviewed, 53 of the CIO ratings were consistent with the risks portrayed in the supporting investment performance documents, 20 were partially consistent, and 7 VA investments were inconsistent. Those that were partially consistent had few instances of discrepancies during the 12-month period. For example, both of Commerce’s inconsistent investments had a discrepancy during 1 of the 12 months which we reviewed. As previously mentioned, a CIO rating should reflect the level of risk facing an investment on a scale from 1 (high risk) to 5 (low risk) relative to that investment’s ability to accomplish its goals. However, seven of the eight agencies we reviewed had at least one instance wherein the Dashboard’s CIO ratings did not consistently reflect the risks identified in agency documents. Table 4 summarizes our assessment of the 10 investments at each of the selected agencies during the 12-month period from January 2012 through December 2012 and a discussion of the analysis of each agency’s documentation follows the table.

Agriculture: Eight of the 10 investments at Agriculture had some inconsistencies with reported risks. In particular, Agriculture experienced technical issues uploading its ratings to the Dashboard in early 2012, which impacted the ratings of six investments. Agriculture officials identified an issue with these investments after observing that submitted changes were not reflected on the Dashboard. They addressed this issue by working with their contractor to determine the cause of the issue, modifying their process to prevent the issue from recurring, and establishing procedures to identify future technical issues. Additionally, Agriculture’s Financial Management Modernization Initiative—one of the six investments that experienced the technical issue described above—and Human Resources Line of Business’s October ratings should have been lower (higher risk) in October 2012. Officials stated that these CIO ratings were updated at the beginning of November 2012 and attributed the delays in the rating submissions to the annual budget process.

Additionally, Agriculture’s CIO rated the Farm Program Modernization investment as moderately low risk until November 2012, despite the investment showing indications of significantly increased risk as early as May 2012. For example, the number of high and very high risks increased every month in a series of briefings from May to July 2012, at which point there was 1 high-impact risk and 2 very high-impact risks each with an estimated 70 percent probability of occurring. Further, in September 2012 a senior management oversight committee determined the investment to be off-track and unable to resolve the issues. However, the investment remained rated on the Dashboard as moderately low risk until November 2012, when it was updated to medium risk. As a result, the CIO rating did not reflect the results of the oversight committee’s review for 3 months. The investment was ultimately given a high-risk rating in December 2012.

Commerce: Two of the 10 selected Commerce investments differed from risk levels identified in underlying documentation for 1 month each. In particular, Commerce’s Census - Economic Census and Surveys investment should have been green rather than yellow in March 2012, and its National Oceanic and Atmospheric Administration IT Infrastructure should have been yellow rather than green in June 2012. In both cases Commerce officials recognized that there were problems and took steps to correct them the following month.

Energy: Five of the 10 selected Energy investments remained listed on the Dashboard after the department had recategorized them as investment types that would have no longer been displayed. Specifically, in October 2012, as part of its annual budget process, Energy downgraded its SR Mission Support Systems from major to non-major and, as discussed previously, changed four of the selected supercomputer investments from IT to non-IT. An SR Mission Support Systems official explained that Energy downgraded the investment after realizing that significant portions of it could be interpreted as non-IT under the Federal Acquisition Regulation, and the remainder did not meet the department’s threshold for being a major IT investment. Further, although Energy made these decisions in October 2012, OMB officials explained that they do not publish budgetary data such as these until the release of the President’s budget, which did not happen until April 2013 at which time the Dashboard reflected Energy’s changes.

Justice: One of the 10 selected Justice investments remained on the Dashboard after the department recategorized it as an investment type that would have no longer been displayed, similar to Energy’s situation. Specifically, Justice downgraded its Law Enforcement Wireless Communications investment from major to non-major as part of its annual budget process in July 2012, but the Dashboard did not reflect the change until April 2013. As noted above, OMB officials explained that they do not publish budgetary data until the release of the President’s budget, which occurred in April 2013.

Transportation: There were no inconsistencies between CIO ratings and reported investments’ risks. Continuing to report consistent and timely data should help ensure that Transportation’s Dashboard’s CIO ratings are accurate.

Treasury: We did not identify inconsistencies between CIO ratings and reported investments’ risks. Such attention to reporting consistent and timely data should help ensure the accuracy of the Dashboard’s CIO ratings.

VA: Seven of the 10 selected investments were never updated during our evaluation period, and 3 were updated once. VA did not update its ratings because it did not have the ability to automatically submit ratings from September 2011 to March 2013, a period of 19 months. Instead, VA elected to build the capability to submit ratings to the Dashboard, rather than purchase this capability. VA completed development of this tool in March 2013, at which point it resumed making the required updates to CIO ratings on the Dashboard. For the 3 that VA updated during this period, VA used a manual budget process to update their ratings in September 2012.

SSA, Information Resources Management Strategic Plan: FY 2012-2016 (May 2012). and schedule variances, prevent the agency from monitoring year-to- year investment performance, and make it difficult to estimate and understand life-cycle costs. While this only affected one investment we reviewed, it has the potential to impact SSA’s entire IT investment portfolio. As described in our Cost Estimating and Assessment Guide, the purpose of such baseline changes should be to restore management’s control of the remaining effort by providing a meaningful basis for performance management. Further, these changes should be infrequent, and recurrent changes may indicate that the scope is not well understood or simply that program management is unable to develop realistic estimates.SSA, frequent rebaselining increases the risk that its investments have undetected cost or schedule variances that will impact the success of the associated investment.

While agencies experienced several issues with reporting the risk of their investments, such as technical problems and delayed updates to the Dashboard, the CIO ratings were mostly or completely consistent with investment risk at seven of the eight selected agencies. Additionally, the agencies had already addressed several of the discrepancies that we identified. The final agency, VA, did not update 7 of its 10 selected investments because it elected to build, rather than buy, the ability to automatically update the Dashboard, and has now resumed updating all investments. To their credit, agencies’ continued attention to reporting the risk of their major investments supports the Dashboard’s goal of providing transparency and oversight of federal IT investments. Nevertheless, the rating issues that we identified with performance reporting and annual baselining, some of which are now corrected, serve to highlight the need for agencies’ continued attention to the timeliness and accuracy of submitted information, in order to allow the Dashboard to continue to fulfill its stated purpose.

Highest-Risk Investments Were Appropriately Addressed by Selected Agencies

We have previously concluded that, consistent with government and industry best practices, including our own guidance on IT investment management, agencies’ highest-risk investments should receive additional management attention. In particular, agency leaders should use data-driven reviews as a leadership strategy to drive performance improvement. Correspondingly, OMB requires reviews, known as TechStat sessions, to enable the federal government to intervene to turnaround, halt, or terminate IT projects that are failing or are not producing results. Of the 80 investments we reviewed, there were 8 investments at four of the eight selected agencies that received a red (high or moderately high risk) rating in 2012. See figure 6 for a list of those at-risk investments and the associated CIO ratings.

Of the eight investments receiving a red rating in 2012, the agencies reviewed seven using tools such as TechStat sessions,investment review boards, and other high-level reviews. Each of these resulted in action items intended to improve performance. The final investment, Agriculture’s Human Resources Line of Business: Service department Center was scheduled to be reviewed using a TechStat session, but in August 2013 officials from Agriculture’s Office of the CIO stated that this investment was in the process of going through a rebaseline. As noted earlier, such changes should be infrequent and reinforce the need for agencies to review the highest-risk investments to ensure that root causes of baseline changes are effectively addressed.

Each of these agencies we reviewed had similar approaches to identifying and conducting these reviews. The agencies with red-rated investments in 2012—Agriculture, Commerce, Justice, and Transportation—monitored investment performance, identified troubled investments using a variety of criteria, reviewed investments in need of attention, and assigned and tracked action items. For example, both Agriculture and Commerce identified investments for review using the Dashboard’s CIO ratings. All four agencies further ensured that investments implemented corrective actions resulting from management reviews. For example, as a result of its November 2012 TechStat, Agriculture leaders assigned 17 action items to the Farm Program Modernization investment, which were to be completed by January 2013. Additionally, the four remaining agencies without red-rated investments— Energy, Treasury, VA, and SSA—have documented processes which provide comparable monitoring and oversight capabilities. For example, VA’s process relies on missed deliverable dates as the key requirement to hold a review, after which senior management makes a determination as to the future of the investment. Alternatively, Energy reviews all major IT investments on a quarterly basis and requires those which fall outside expected thresholds to complete corrective actions. As such, all of the agencies have focused management attention on troubled investments and once identified, have established and followed through on clear action items to turn around or terminate such investments.

Conclusions

Since its inception in 2009, the IT Dashboard has increased the transparency of the performance of major federal IT investments. Its CIO ratings, in particular, have improved visibility into changes in the risk level of investments over time, as well as agencies’ ability to accomplish their goals. To that end, over the past 4 years, the agencies we reviewed have reported lower risk for more than one quarter of all their investments. However, the effectiveness of the Dashboard depends on the quality of the information that agencies submit. We previously recommended that OMB clarify guidance on whether certain types of systems should be included in IT reporting, but OMB did not agree. Agencies’ subsequent decisions to remove investments from the Dashboard by changing investments’ categorizations represent a troubling trend toward decreased transparency and accountability. Thus, we continue to believe that our prior recommendation remains valid and should be implemented. Additionally, OMB’s annual freeze of the Dashboard for as long as 8 months negates one of the primary purposes of this valuable tool—to facilitate transparency and oversight of the government’s billions of dollars in IT investments.

Beyond the transparency they promote, CIO ratings present an opportunity to improve the data and processes agencies use to assess investment risk. Each of the agencies we examined had established such processes, and most of the resulting Dashboard ratings were consistent with underlying documentation. While two agencies, Transportation and Treasury, had ratings that accurately reflected their investments’ supporting documentation, other agencies’ Dashboard ratings were inconsistent with the actual risk of the associated investment. These inconsistencies—such as inaccurate reflection of negative performance, questionable decisions about the recategorization of investments, and too-frequent changes to performance baselines—highlight the continued need for agencies to populate the Dashboard with an accurate representation of the full breadth and health of their investments. In providing this information, agencies will help the Dashboard accomplish its goal of providing transparency and oversight of millions of dollars in federal IT investments.

Recommendations for Executive Action

To better ensure that the Dashboard provides meaningful ratings and reliable investment data, we are recommending that the Director of OMB direct the Federal CIO to make accessible regularly updated portions of the public version of the Dashboard (such as CIO ratings) independent of the annual budget process.

In addition, to better ensure that the Dashboard provides accurate ratings, we are making three recommendations to the heads of three of the selected agencies. Specifically, we are recommending that:

The Secretary of Commerce direct the department CIO to ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard.

The Secretary of Energy direct the department CIO to ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard.

The Commissioner of the SSA direct the CIO to revise the agency’s investment management processes to ensure that they are consistent with the baselining best practices identified in our published guidance on cost estimating and assessment.

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB and all eight departments and agencies in our review. OMB neither agreed nor disagreed with our recommendation; Agriculture agreed with the report’s findings; Commerce disagreed with our recommendation; Energy concurred with our recommendation with one exception related to supercomputers; Justice, Treasury, and Transportation neither agreed nor disagreed with the report’s findings; VA agreed with the report’s findings; and SSA agreed with our recommendation. Each agency’s comments are discussed in more detail below.

In written comments, the Federal CIO neither agreed nor disagreed with our recommendation for OMB to make accessible regularly updated portions of the public version of the Dashboard (such as CIO ratings) independent of the annual budget process. However, OMB also noted that the manner in which agencies submit Dashboard information to OMB makes it difficult to separate materials it believes are predecisional or deliberative. Despite this, OMB agreed to explore whether it would be practical to make the Dashboard more accessible, and to consider how it could separate predecisional or deliberative materials. We support OMB in its efforts to increase public transparency and accountability.

OMB continued to disagree with the 2011 recommendation that we believe continues to have merit, related to the clarity of its guidance on whether certain types of systems should be included in IT reporting. OMB believes that the existing guidance is appropriate and does not have plans to review it. However, as noted earlier in this report, we believe that the recommendation is appropriate because the existing guidance does not address key categories of IT investments where we continued to find inconsistencies.

OMB also disputed our characterization of two issues. First, OMB noted that up-to-date Dashboard information is available to the agencies and OMB during the budget development period, who use it as a tool for investment oversight and decision making. While we acknowledge that OMB and federal agencies continue to have access to the Dashboard during the budget process, the public display of these data is intended to allow other oversight bodies, including Congress and the general public, to hold government agencies accountable for progress and results. Secondly, OMB stated that it began using the IT Dashboard to help identify at-risk investments starting with its launch in June 2009, rather than 2010. We modified the report to reflect OMB’s statement. OMB also provided technical comments, which we have incorporated in the report as appropriate.

OMB’s written comments are provided in appendix III.

In written comments, Agriculture’s CIO stated that the department concurred with the content of the report and had no comments. Agriculture’s written comments are provided in appendix IV.

In written comments, Commerce agreed with most of the report’s findings but disagreed with our recommendation to ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Specifically, Commerce stated that although it is no longer reporting publicly on its satellite ground system investments through the Dashboard, neither the department CIO nor OMB has relinquished its oversight role. Moreover, Commerce stated that it is reviewing its satellites more frequently and in more detail; as an example, the department noted that Commerce’s CIO conducts monthly Dashboard-like assessments that cover the status of the satellite investments. However, regardless of these additional efforts, the removal of the satellite investments from the Dashboard prevents the public display of these data intended to allow OMB and other oversight bodies, including Congress, to hold the government agencies accountable for progress and results. Additionally, as discussed in this report, Commerce’s recategorization of its satellite ground system investments is contrary to the definition of IT as set forth in the Clinger-Cohen Act. Based on these facts, we continue to believe that our recommendation to Commerce that its CIO ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard has merit and should be implemented. Commerce’s written comments are provided in appendix V. The department also provided technical comments related to our characterization of the department’s rating process, which we have incorporated in the report as appropriate.

In written comments, Energy concurred, but with a comment regarding our recommendation to ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Specifically, while agreeing that Energy should ensure that its IT investments are appropriately categorized, the department stated that it would continue to report investments that it believes should be categorized and managed on the Dashboard, noting that supercomputers would be an exception to this policy. However, this exception runs contrary to the finding on which this recommendation is based, namely that Energy removed $368 million in investments from the Dashboard, including a supercomputer reported in 2012 as the most powerful in the world. While Energy agreed to partner with OMB to develop an alternate mechanism to track and report supercomputer investments to OMB, it is not clear whether this information will be publicly available. Additionally, Energy’s recategorization of its supercomputer investments is contrary to the definition of IT as set forth in the Clinger-Cohen Act, as discussed in this report. Therefore, we disagree that the department’s planned actions adequately address existing requirements for open and transparent reporting of major IT investments, and we stand by our assessment that Energy’s CIO should ensure that the department’s investments are appropriately categorized in accordance with existing statutes and that major IT investments are included on the Dashboard. Energy’s written comments are provided in appendix VI.

In comments provided via e-mail on November 6, 2013, a representative of Justice’s Justice Management Division stated that the department had no comments.

In comments provided via e-mail on November 1, 2013, Transportation’s Deputy Director of Audit Relations stated that the department had no comments. The department also provided technical comments, which we have incorporated in the report as appropriate.

In written comments, the Treasury’s CIO stated that the department had no comments. Treasury’s written comments are provided in appendix VII.

In written comments, VA’s Chief of Staff stated that the department generally agreed with the findings of the report and provided general comments in which it confirmed that issues identified in the report had been addressed. Specifically, VA confirmed that the department has implemented the capability to automatically submit CIO ratings to the Dashboard and now uses more specific data to generate the CIO rating, such as the number of “red flags” associated with an investment or the number of TechStat reviews held. Finally, while stating that the department will continue its efforts to improve timely and accurate reporting to the Dashboard, VA also noted that it is important to recognize that the department manages investment delivery at the project level in comparison to the investment level found in the Dashboard. VA’s written comments are provided in appendix VIII.

In written comments, SSA’s Deputy Chief of Staff agreed with our recommendation to revise the agency’s investment management processes to ensure that they are consistent with the baselining practices, and discussed planned actions to address it. SSA’s written comments are provided in appendix IX.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees; the Secretaries of Agriculture, Commerce, Energy, Transportation, Treasury, Veterans Affairs, Attorney General of the United States, Acting Commissioner of Social Security, the Director of the Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix X.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to (1) characterize the Chief Information Officer (CIO) ratings for selected federal agencies’ information technology (IT) investments as reported on the Dashboard over time, (2) determine the extent to which selected agencies’ CIO ratings are consistent with reported investment risk, and (3) determine the extent to which selected agencies are addressing at-risk investments.

To address our first objective, we selected the eight agencies with the most reported major IT spending in fiscal year 2012, after excluding agencies reviewed in our most recent Dashboard report. The eight agencies are the Departments of Agriculture (Agriculture), Commerce (Commerce), Energy (Energy), Justice (Justice), Transportation (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), and the Social Security Administration (SSA). The results in this report represent only these agencies. We downloaded and examined the Dashboard’s CIO ratings for all major investments at the eight agencies (a total of 383 investments reported by these agencies). To characterize the numbers and percentages of major IT investments at each risk level at each of our subject agencies, we analyzed, summarized, and—where appropriate—graphically depicted average CIO ratings for investments by agencies over time during the period from June 2009 to August 2013. Specifically, we compared each investment’s first and last CIO ratings (including those investments that were not on the Dashboard at its inception, and those that were downgraded or eliminated) and summarized the data by agency. To describe whether CIO ratings indicated higher or lower investment risk over time, we calculated the numbers and percentages of investments (by agency and collectively for all the agencies) that maintained a constant rating over the entire performance period, and those that experienced a change to their CIO rating in at least one rating period. Then we analyzed the subset of investments that experienced at least one changed rating and compared the first CIO rating with the latest CIO rating (no later than August 2013) to determine the numbers and percentages of investments (by agency and collectively for all the agencies) that experienced a net rating increase, a net rating decrease, or no net change. We also reviewed investments which had been removed from the Dashboard due to recategorization and compared their definitions to the Clinger-Cohen Act’s definition of IT. We presented our results to each agency and the Office of Management and Budget (OMB) and solicited their input, explanations for the results, and additional corroborating documentation, where appropriate.

To accomplish the second objective, we selected the top 10 major investments at each of the eight selected agencies (for a total of 80 investments), with the highest reported IT spending in fiscal year 2012. We reviewed investment documentation (such as executive-level briefings, operational analyses, and TechStat results) and interviewed officials from each of the agencies to understand how they rate and monitor investments. of investment risk and performance (such as cost and schedule variances), which could impact the success of the associated investment. We then organized the results by month, and compared these results to the relevant investment Dashboard CIO ratings for calendar year 2012.

Within these artifacts, we identified representations We then evaluated whether each investment’s monthly CIO rating was consistent with reported investment risks, and categorized each investment by the number of months which were inconsistent. Investments which were consistent with underlying documentation for every month in 2012 were categorized as “consistent,” those which were inconsistent for 1 or more months were categorized as “partially inconsistent,” and those with inconsistencies in each month were “inconsistent.”

Because we were only evaluating the consistency of Dashboard ratings with reported risk, we did not evaluate the accuracy of the investment documentation. moderately high) rating on the Dashboard during 2012. We reviewed the processes used to address these investments and interviewed relevant officials. We also examined the results of performance reviews and interviewed officials to determine whether the agencies implemented the appropriate risk management processes to oversee and review the selected investments.

We conducted this performance audit from February to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Investments

Below is the list of the investments that are included in this review, as well as the reported fiscal year 2012 IT spending.

Appendix III: Comments from the Office of Management and Budget

Appendix IV: Comments from the Department of Agriculture

Appendix V: Comments from the Department of Commerce

Appendix VI: Comments from the Department of Energy

Appendix VII: Comments from the Department of the Treasury

Appendix VIII: Comments from the Department of Veterans Affairs

Appendix IX: Comments from the Social Security Administration

Appendix X: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Rebecca Eyler, Mike Mithani, Kevin Walsh, and Shawn Ward.