IT Dashboard: Agencies Need to Fully Consider Risks When Rating Their Major Investments
Released: 2016-06-02T00:00:00

Letter

Background

OMB plays a key role in overseeing how federal agencies manage their IT investments by working with them to better plan, justify, and determine how to manage them. To provide visibility into the performance of such investments, OMB deployed the IT Dashboard in 2009, which displays federal agencies’ cost, schedule, and performance data for over 770 major federal IT investments at 26 federal agencies, accounting for $42 billion of those agencies’ planned $82 billion in IT spending for fiscal year 2017. According to OMB, these data are intended to provide a near-real- time perspective on the performance of these investments, as well as a historical perspective. The Dashboard’s data span the period from its June 2009 inception to the present, and are based, in part, on agency assessments of individual investment performance and each agency’s budget request to OMB. Further, the public display of these data is intended to allow OMB; other oversight bodies, including Congress; and the general public to hold government agencies accountable for progress and results.

OMB’s Dashboard Is Intended to Provide Visibility into the Performance of Federal IT Investments

The Dashboard visually presents performance ratings for agencies and for individual investments using metrics that OMB has defined—cost, schedule, and CIO evaluation.

Cost and schedule ratings. The Dashboard calculates these ratings by determining cost and schedule variances based on agency- submitted data, such as planned versus actual costs or planned versus actual completion dates. The Dashboard then assigns rating colors (red, yellow, green) based on the magnitude of the variances. Specifically, a variance greater than 30 percent is red, a variance between 10 percent and 30 percent is yellow, and a variance less than 10 percent is green.

CIO ratings. Unlike the cost and schedule ratings, the Dashboard’s “Investment Evaluation by Agency CIO” (also called the CIO rating) is determined by the agency CIO. According to OMB’s instructions, each agency CIO is to assess his or her IT investments against a set of pre-established evaluation factors and then assign a rating of 1 (high risk) to 5 (low risk) based on the CIO’s best judgment of the level of risk facing the investment. OMB suggests six evaluation factors, as shown in table 1.

OMB recommends that CIOs consult with appropriate stakeholders in making their evaluations, including Chief Acquisition Officers, program managers, and other interested parties. According to an OMB staff member, agency CIOs are responsible for determining appropriate thresholds for the risk levels and for applying them to investments when assigning CIO ratings. OMB requires agencies to update these ratings as soon as new information becomes available which will affect an investment’s assessment and, since June 2015, has required that this be at least once each calendar month. After agencies assign a level of risk to each investment, the Dashboard assigns colors to CIO ratings according to a five-point scale: high risk and moderately high risk are red, medium risk is yellow, and moderately low risk and low risk are green.

Recent Legislation Reinforced the Importance of the Dashboard

Recognizing the importance of government-wide management of IT, in December 2014, Congress enacted IT acquisition reform legislation, FITARA. The law was designed to improve agencies’ acquisition of IT and enable Congress to monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings. FITARA contains specific requirements related to seven areas, including one titled “Enhanced Transparency and Improved Risk Management in Information Technology Investments.” Among other things, that area requires OMB and agencies to make publicly available detailed information on federal IT investments, and agency CIOs to categorize their IT investments by risk. This requirement is addressed by OMB’s IT Dashboard.

GAO Has Previously Reported on the Dashboard

Over the past 5 years, we have issued a series of reports about the IT Dashboard that noted both significant steps OMB has taken to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard, as well as issues with the accuracy and reliability of data. Further, we have reported on the Dashboard’s CIO ratings: In October 2012, we reported that CIOs at six agencies rated a majority of investments listed on the IT Dashboard as low or moderately low risk from June 2009—when the Dashboard was implemented—through March 2012 and two agencies, the Department of Defense (Defense) and the National Science Foundation, rated no investments as high or moderately high risk during this time period (categorized as “red” by the Dashboard). Additionally, agencies generally followed OMB’s instructions for assigning CIO ratings, although Defense’s ratings were unique in reflecting additional considerations, such as the likelihood of OMB review. Most of the selected agencies reported various benefits associated with producing and reporting CIO ratings, such as increased quality of their performance data and greater transparency and visibility of investments. We recommended that OMB analyze and report on agencies’ CIO ratings over time, and that Defense ensure that its CIO ratings reflect available investment performance assessments and its risk management guidance. Both agencies concurred with our recommendations. Subsequently, OMB reported on CIO rating trends. Further, Defense now identifies red investments on the Dashboard.

More recently in December 2013, we reported that, as of August 2013, the CIOs at eight selected agencies had rated 198 of their 244 major IT investments listed on the Dashboard as low risk or moderately low risk, 41 as medium risk, and 5 as high risk or moderately high risk. However, the total number of investments reported by these agencies had varied over time, which impacted the number of investments receiving CIO ratings. For example, the Department of Energy (Energy) reclassified several of its supercomputer investments from IT to facilities, and the Department of Commerce (Commerce) decided to reclassify its satellite ground system investments. Both decisions resulted in the removal of the investments from the Dashboard, even though the investments were clearly IT. We recommended that these agencies appropriately categorize all investments, but they disagreed with our recommendation.

In that same report, we reviewed 80 investments and found that 53 of the CIO ratings were consistent with the investment risk, 20 were partially consistent, and 7 were inconsistent. While two agencies’ CIO ratings were entirely consistent, other agencies’ ratings were inconsistent for a variety of reasons, including delays in updating the Dashboard and how investment performance was tracked. For example, the Social Security Administration (SSA) resets investment cost and schedule performance baselines annually, an approach that increases the risk of undetected cost or schedule variances that will impact investment success. As such, we recommended that SSA revise its investment management approach. The agency agreed with our recommendation and discussed planned actions to address it.

Additionally, we reported that OMB does not update the public version of the Dashboard as the President’s annual budget request is being created. Consequently, the public version of the Dashboard was not updated for 15 of the past 24 months. We recommended that OMB make Dashboard information available independent of the budget process. OMB recently updated the Dashboard with a number of changes, and intends for the Dashboard to be able to show updates throughout the year.

Agencies Use a Variety of Processes to Determine Investments’ CIO Ratings

Agencies determine investments’ CIO ratings using a variety of processes, which include OMB’s suggested factors. However, their interpretation of these factors varies significantly. In addition, the majority of agencies base their ratings on qualitative assessments, but several base theirs on formulas. Further, 13 agencies’ process guidance calls for at least monthly updates to CIO ratings, 1 agency (the Department of Homeland Security (DHS)) schedules its reviews based on risk, and 3 agencies require less frequent updates.

Agencies Use Many of OMB’s Suggested Factors to Determine CIO Ratings

As described earlier, OMB requires that each agency CIO rate the risk of his or her IT investments. OMB gives the CIOs the flexibility to use their judgment and suggests six evaluation factors.

As noted above, we reviewed data reported to OMB as part of the federal budget process to identify major investments which planned to spend at least 80 percent of their fiscal year 2015 funding on development, modernization, and enhancement activities. This selection produced a list of 17 agencies and 107 investments. Each of the 17 agencies has incorporated at least 2 of OMB’s suggested factors into their CIO’s risk rating processes and 9 use all of the factors. OMB requires that agencies provide CIO evaluations for all major IT investments which reflect the CIO’s best judgment of the current level of risk for the investment in terms of its ability to accomplish its goals. According to OMB’s guidance, the evaluation can be informed by the following factors, including, but not limited to: risk management, requirements management, contractor oversight, historical performance, human capital, and other factors that the CIO deems important to forecasting future success.

Table 2 summarizes the extent to which the 17 selected agencies incorporate OMB’s suggested evaluation factors into their CIO’s risk rating processes. Appendix II provides more information on the selected agencies’ CIO rating processes.

While the factors suggested by OMB were considered in the agencies’ CIO rating processes, their interpretation of these factors varied. In particular, most agencies considered active risks when rating investments, but others only evaluated compliance with risk processes. For example:

Defense, the Department of the Treasury (Treasury), and the General Services Administration (GSA) consider individual active risks, rather than investments’ compliance with risk guidance. For example, Defense considered the risk caused by budget cuts and their potential impact through the following fiscal year when rating its Base Information Transport Infrastructure Wired investment. In addition, GSA, when rating its Integrated Award Environment investment, considered the risk involved with transitioning from an existing contract to one that could provide better expertise and greater oversight.

The Departments of Agriculture (Agriculture), Education (Education), Energy, the Interior (Interior), State (State), and Veterans Affairs (VA), and the Office of Personnel Management (OPM) review compliance with risk management practices, but do not assess active risks. For example, compliance may include whether mitigation plans exist, risk logs are current, and risks are clearly prioritized.

The rest of the agencies that include the risk factor—Commerce, DHS, the Department of Labor (Labor), the Department of Transportation (Transportation), as well as the Environmental Protection Agency (EPA) and SSA—considered both process compliance and reviews of active risks. For example, Commerce reviews at least the top three active risks for investments, verifies that these risks are specific to the investment, appropriately managed and mitigated, and that the risk register is updated regularly. DHS also considers active investment risks, ensures that they are current, and that risk mitigation plans are in place.

Furthermore, the selected agencies considered different types of historical data when rating their IT investments. While all of the agencies considered performance measures and cost and schedule variances, five considered changes to the investment’s baseline; eight considered the accomplishment of milestones; and three considered relevant news, GAO, or Inspector General reports. While the details of these approaches vary, they align with OMB’s suggested factors.

Most Agencies Used Qualitative Rather than Quantitative Methodologies

Of the 17 selected agencies, 6 used formulas to create CIO ratings. Specifically, Agriculture, the Department of Health and Human Services (HHS), DHS, Education, Treasury, and VA determined their ratings by quantifying and combining inputs such as cost and schedule variances, risk exposure values, and compliance with agency processes. Metrics for compliance with agency processes included those related to program and project management, project execution, the quality of investment documentation, and whether the investment is regularly updating risk management plans and logs.

The remaining 11 agencies based their CIO ratings on qualitative assessments of performance metrics, risks, investment documentation, and informal investment knowledge. In particular, they assign ratings based on metrics such as investment performance, discussions with management staff, and the quality of investment documentation.

Most Selected Agencies Require Monthly CIO Rating Updates

Thirteen agencies’ process guidance calls for at least monthly updates to CIO ratings, one agency (DHS) schedules its reviews based on risk, and three agencies require less frequent updates. Although a monthly review and update process was not previously required by OMB, the fiscal year 2017 Capital Planning Guidance issued in June 2015 requires agencies to update their CIO ratings at least once per month. Table 3 summarizes the frequency of CIO rating updates called for by the selected agencies’ processes.

The three selected agencies that do not comply with OMB’s current requirement for monthly rating updates are Defense, Education, and SSA. In particular:

Defense updates CIO ratings semi-annually.

Education updates CIO ratings based on bi-monthly reviews of investments.

SSA conducts monthly investment reviews, but updates CIO ratings on a quarterly basis.

Additionally, DHS staggers its updates based on investment risk, with high risk (red) investments reviewed monthly, moderate (yellow) investments reviewed quarterly, and low (green) investments reviewed semi-annually. These four agencies’ practices are inconsistent with OMB’s guidance and can limit the transparency and oversight of the government’s IT investments.

However, staff from OMB told us that the Capital Planning Guidance for fiscal year 2018 would not contain the monthly reporting requirement and would instead encourage agencies to keep their CIO ratings accurate and current, rather than mandate reporting frequency. Moving forward, it will be important for any such revised guidance to encourage the frequent and appropriate updating of agencies’ CIO ratings while also remaining compliant with relevant provisions of FITARA. These provisions require that agencies report at least semi-annually to OMB on each major IT investment and include data on cost, schedule, and performance. These provisions also require joint OMB and agency reviews of any investment that has been evaluated as high risk for four consecutive quarters.

Our Assessments Reflected More Risk than Most Selected Investments’ CIO Ratings

As discussed earlier, to assess the risk of individual assessments, we reviewed the 107 investments that we originally selected, but excluded 12 that were inactive, not in development, lacked a key risk document, or were managed as part of a larger development program. This resulted in 95 investments at 15 agencies. Our assessments of these investments generally showed more risk than the associated CIO ratings. In particular, of the 95 investments we reviewed, our assessments matched the CIO ratings 22 times, showed more risk 60 times, and showed less risk 13 times. We identified three factors which contributed to these differences: (1) 40 of the 95 CIO ratings were not updated in April 2015, (2) three agencies’ rating processes span longer than 1 month, and (3) seven agencies’ rating processes did not focus on active risks (as previously discussed).

According to OMB’s guidance, CIO ratings “should reflect the CIO’s assessment of the risk and the investment’s ability to accomplish its goals.” Such assessments of risk inherently involve a great deal of human judgment. Consequently, risk assessments should be expected to vary both across and within organizations. For example, Defense’s CIO ratings process documentation states that, since its major investments are “inherently high risk,” its ratings are “assessments of relative risk implemented within this risk baseline.” That is, when measuring risk, Defense is more tolerant and uses a different scale than other agencies. Similarly, risk assessments can vary within agencies. For example, officials at several agencies expressed concerns that the assignment of risk scores (probability and impact) were not consistent across investments. Officials at DHS also noted that program managers may score risks higher to flag an issue for management attention. Further, in many cases, agency CIOs could have more information than we examined in our assessments.

We attempted to minimize the subjectivity in our risk assessments by using the agencies’ own lists of risks, known as risk registers, as the basis of our assessments (see appendix I for additional details on our methodology). We also augmented our ratings with agencies’ cost and schedule data, briefings to review boards, and relevant reports. Our calculations are only intended to provide a standardized view of risk across all the departments and investments we reviewed and this methodology is not intended to serve as a prescriptive approach to the agencies’ evaluation of investment risk.

Our Assessments Showed More Risk than Almost Two Thirds of Selected Investments’ CIO Ratings

While the variety of methodologies and inputs meant that some differences were inevitable, almost two thirds of our assessments showed more risk than the associated CIO ratings for our 95 selected investments. Figure 1 summarizes how our assessments compared to the select investments’ CIO ratings.

Of the 95 investments we reviewed, our assessments showed less risk 13 times, matched the CIO ratings 22 times, and showed more risk 60 times. Additionally, our assessments showed more risk for at least 1 investment at 13 of the 15 agencies we assessed. Table 4 summarizes these comparisons by agency, and appendix III lists the April 2015 CIO ratings and our assessments for each of the selected investments.

Overall, our assessments reflected more risk than 63 percent of the associated CIO ratings, and 13 of the 15 agencies reported less risk for at least 1 investment. Of the 13, 11 reported less risk for at least half of the selected investments and the remaining 2 reported less risk for just under half of those agencies’ selected investments. For example, we identified more risk at 3 of DHS’s 7 investments and 4 of SSA’s 9 investments.

Our Assessments Reflected Less Risk than the Agencies’ CIO Ratings for 13 Investments

Our assessments showed less risk than the CIO ratings for 13 of the 95 selected investments (14 percent). Specifically, we assessed 5 green that the agencies rated yellow and 8 yellow that the agencies rated red; there were no instances where we assessed an investment green that the agencies rated red. These investments belonged to 8 of the 15 selected agencies: HHS (3 investments), Defense (2 investments), DHS (2 investments), Education (2 investments), Commerce (1 investment), Energy (1 investment), GSA (1 investment), and SSA (1 investment). Table 5 lists those investments, the April 2015 CIO rating, and our associated assessment.

The reasons why our assessments showed less risk varied among these investments. For instance:

Commerce rated its Integrated Dissemination Program as yellow, but we assessed it as green. A department official explained that the rating was because (1) the investment had not been transparent in its activities making it difficult to determine whether a milestone was achieved or services were provided and (2) the investment had only identified generic, non-specific risks. Further, the official noted that the department had even debated rating this investment red. We assessed this investment as green because of the low overall level of risk and low cost and schedule variances.

DHS rated its Continuous Diagnostics and Mitigation investment as yellow, but we assessed it as green. DHS officials stated that the investment was rated yellow because (1) it was considered complex and higher risk due to the involvement of several civilian government agencies, (2) cost and schedule variances exceeded OMB’s thresholds, and (3) concerns about the schedule and availability of resources. In contrast, we assessed it as green because it did not have any risks with both high impact and high probability scores and more than half of the risks scored low in overall risk exposure.

HHS rated its Medicaid and Children’s Health Insurance Program Business Information and Solutions investment as red, but we assessed it as yellow. HHS officials rated this investment red because the investment team did not submit required data and a rebaseline caused a large cost variance. Even though the investment’s overall risk score was low, we assessed the investment as yellow based upon documented cost and schedule variances and program issues identified in review board briefings.

GSA rated its Integrated Award Environment as yellow, but we assessed it as green. GSA officials stated that the investment was rated yellow as a precaution: the investment was undergoing a contract transition and the CIO knew that problems could develop. GSA officials stated that another contributing factor was the investment’s late contract award, which had residual impact on investment performance. We assessed this investment green because of its overall low risk score.

As noted earlier, CIO ratings are intended to reflect the CIO’s assessment of the risk and may be based on additional programmatic information not included in our assessment methodology, which focused primarily on investments’ risk registers. As such, the inherently judgmental nature of the CIOs’ assessments may reflect broader considerations that, in their organization’s view, better represent the overall risk of an investment.

Our Assessments Matched the Agencies’ CIO Ratings for 22 Investments

For 22 of the 95 selected investments (23 percent), our assessments matched the CIO rating. Specifically, we matched 10 green ratings, 8 yellow ratings, and 4 red ratings. Table 6 lists those investments, the April 2015 CIO rating, and our associated assessment.

In particular, there were 4 investments at Defense, 4 at SSA, 4 at Transportation, 3 at Commerce, 2 at GSA, 2 at DHS, 1 at HHS, 1 at Interior, and 1 at State that had CIO ratings that matched our assessments. These investments were a mix of red, yellow, and green ratings. However, the reasoning behind the CIO ratings and our individual assessments differed. For example, Interior rated its Integrated Reporting of Wildland-Fire Information investment as yellow because the investment’s required documentation did not meet agency standards.

Specifically, the investment’s most recent artifact review before the April 2015 CIO rating period showed that the investment lacked required documentation, including a risk management plan. However, we assessed the investment as yellow because the IT Dashboard showed significant cost and schedule variances at the time of our review.

Our Assessments Reflected More Risk than the Agencies’ CIO Ratings for 60 Investments

For 60 of the 95 selected investments (63 percent), our assessments reflected more risk than agencies’ CIO ratings. Specifically, we assessed 9 red that the agencies rated yellow, 28 yellow that the agencies rated green, and 23 red that the agencies rated green. Further, these investments were at 13 of the 15 agencies we assessed. Table 7 lists those investments, the April 2015 CIO rating, and our associated assessment.

The agencies’ explanations as to why our assessments showed more risk varied in these 60 cases. For example,

Agriculture rated its Optimized Computing Environment investment as green, but we assessed it as red. While Agriculture officials noted that this investment’s funding from partner agencies was uncertain, the investment received a green CIO rating because the funding uncertainties were a recurring concern that had been previously managed without issue. Conversely, we assessed the investment as red because 21 of the 44 risks in the investment’s risk register had high overall risk scores.

DHS rated its Next Generation Networks Priority Services investment as green, but we assessed it as red. DHS officials stated that they rated this investment green because the investment was progressing well and because it had successfully mitigated its high impact/high probability risks. However, our assessment was partly based on two risks with both high probability and high impact scores that the investment team categorized as potentially causing investment failure. These risk scores and descriptions indicated that the program believed that it was likely that these risks would be realized and cause critical, perhaps investment-threatening problems. DHS officials questioned the probability and impact scores and explained that investment teams may inflate such scores to flag potential issues for management.

SSA rated its Supplemental Security Income Modernization investment as green, but we assessed it as red. SSA officials stated that their review did not see significant reason to lower the rating, even though part of the investment was working through significant technical challenges. We assessed the investment red because half of its risks had high risk scores. Included in the most critical risks were those pertaining to requirements changes, system complexity, and staffing losses. When asked about these risks, SSA officials explained that, generally, they were common risks faced by every investment. Consequently, they expressed doubt that these risks necessitated such high risk scores.

Treasury rated its Post Payment System investment as green, but we assessed it as red. Treasury officials stated that this investment was rated green because it was well-run and had previously kept its risks from becoming realized issues. Conversely, we rated this investment as red because 20 out of its 26 risks had high overall risk scores, including 2 that indicated a high probability of schedule delays. Treasury officials stated that they were monitoring the risks that we identified.

As noted earlier, the judgmental nature of a CIO’s assessment may reflect a broader organizational view of investment risk beyond the contents of the investment’s risk register. However, unlike the CIO ratings that reflected more risk than our assessments, many of these CIO ratings minimized the potential severity and impact of high risk scores. Our past work has shown that such an approach to risk management can often lead to cost and schedule overruns or failed projects.

Three Issues Contributed to Discrepancies between Agencies’ CIO Ratings and Our Assessments

In addition to the previously discussed issue of rating subjectivity, we identified three factors which contributed to differences between our assessments and CIO ratings at 10 of the 15 selected agencies. In particular, ratings were not updated in April 2015, rating processes spanned longer than 1 month, and rating processes did not focus on active risks.

Rather than pertaining to the CIOs’ personal evaluations of risk, these additional issues relate to the 10 agencies’ update practices or rating processes. Because these issues are with underlying practices and processes, they have the potential to impact all investment ratings— whether or not we reviewed them as part of our assessment. Specifically, we found that 40 of the 95 CIO ratings were not updated in April 2015, 3 agencies’ rating processes span longer than 1 month, and 7 agencies’ rating processes did not focus on active risks (see table 8). Following the table is a further description of these issues.

Of the 95 investments we selected, we found that agencies had not updated CIO ratings for 40 in April 2015 (see table 9).

Further, we found that our assessments were more likely to match those investments updated in April because these recently updated ratings reflected the then-current investment information that we also used in our assessments. In particular, 17 of the 22 investments where our assessment matched the CIOs’ ratings were updated in April 2015, whereas we matched only 5 of the 40 without April 2015 updates. Of the 40 ratings that were not updated, 15 were at 6 agencies, which provided the following explanations for the lack of April updates:

Agriculture officials stated that they were in the midst of switching systems used to update ratings to the Dashboard, so they were unable to update ratings that month.

Commerce officials stated that they did not receive quality data from the investment until the following month and thus did not have a valid basis for changing the rating or CIO comments.

DHS officials stated that they did not update 2 of the selected investments’ ratings because they were undergoing TechStat reviews. Additionally, 2 investments’ CIO ratings were not updated in April even though they initially received April reviews. According to officials, 1 investment’s review occurred too late in the month and the other investment’s update was delayed. They also stated that they were revising their process so that rating updates would occur during such reviews. Additionally, DHS’s risk-based review cycle meant that two investments were not due to be rated in April. In particular, those investments were rated yellow less than 3 months prior and DHS reviews yellow-rated investments on a quarterly basis.

An Education official explained that the agency only updates its ratings if an investment’s status has changed enough to warrant an update.

EPA officials told us that its selected investment was not updated in April 2015 because the investment was in a state of flux during that time and a review would not have been useful.

HHS officials stated that a combination of system and human errors kept 2 investment updates from posting.

The remaining 25 investments which did not have an April 2015 update were at Defense, which updates its ratings on a semi-annual basis (see earlier discussion of CIO rating update frequency). If we had used Defense’s subsequent update from June 2015, the number of instances in which the CIO ratings matched our assessments would have increased from 4 to 10.

The preceding examples underscore the importance of OMB’s June 2015 policy change, requiring that agencies provide monthly rating updates. Such updates will help to ensure that the information on the Dashboard is timely and accurately reflects recent changes. Without such updates, the CIO ratings on the Dashboard may not reflect the current level of investment risk.

Three Selected Agencies’ Rating Processes Took Longer than 1 Month

The duration of agencies’ CIO rating processes also impacted the comparison between the CIO ratings and our assessment. Since we used March 2015 risk registers as the basis for our assessment, process times longer than a month mean that the data we used for our assessment would not have been included in agencies’ April 2015 CIO ratings. Fourteen of the 17 selected agencies indicated that it takes 1 month or less to process investment data and update the IT Dashboard. However, processes at 3 agencies—Education, State, and Treasury—can be longer than 1 month. For example, Treasury officials stated that late invoices from contractors can delay their processes past the 1 month time frame. State officials explained that their conversion of certain cost and schedule data into the format required by the Dashboard can take between 1 and 2 months to complete. Longer processes mean that CIO ratings are based upon older data and may not reflect the current level of investment risk. Further, these longer processes may prevent the agencies from meeting OMB’s requirement to update ratings monthly.

Seven Selected Agencies Did Not Factor Active Risks into Ratings

When developing CIO ratings, seven agencies did not consider active risks, as discussed earlier. Six of those agencies (Agriculture, Education, Energy, Interior, State, and VA) instead chose to focus on investments’ risk management processes, such as whether a process was in place or whether a risk log was current. Such approaches did not consider individual risks, such as funding cuts or staffing changes, which detail the probability and impact of pending threats to success. Instead, VA’s CIO rating process considers several specific risk management criteria: whether an investment (1) has a risk management strategy, (2) keeps the risk register current and complete, (3) clearly prioritizes risks, and (4) puts mitigation plans in place to address risks.

Considering process compliance, rather than active risks, contributed to our assessments only matching 2 of the CIO ratings for the 14 selected investments at these six agencies. The remaining agency, HHS, did not factor risk into its CIO ratings (as discussed earlier). This contributed to our assessments matching only 1 of the 9 selected HHS investments. In all cases, CIO ratings that do not incorporate active risks increase the chance that ratings do not reflect the true likelihood of investment success.

Conclusions

Since its inception in 2009, the IT Dashboard has increased the transparency of the government’s multi-billion dollar spending on major IT investments. The Dashboard’s CIO ratings, in particular, have improved visibility into the risks facing these critically important efforts. To that end, agency CIOs have developed a variety of processes to assess and report the risk of their investments. Although the effectiveness of the Dashboard depends on the quality of the CIOs’ ratings, selected agencies’ rating methods do not provide an accurate assessment of investment risk and thus reduce the value of this important tool for transparency and oversight. Further, multiple agencies’ infrequent submissions raise concerns that those updates are not reflecting timely and accurate risk information, contrary to OMB’s current policy requiring monthly updates. Such practices limit the transparency and oversight of the government’s billions of dollars in IT investments.

Beyond the transparency they provide, CIO ratings present an opportunity to improve CIOs’ understanding of their IT portfolio and identify those investments in need of additional oversight. While these ratings are by definition inherently judgmental, our assessments of the selected investments generally showed more risk than almost 65 percent of the associated CIO ratings. Consequently, the associated risk rating processes used by the agencies generally are understating the level of risk, raising the likelihood that critical federal investments in IT are not receiving the appropriate levels of oversight. Finally, agencies that do not factor active risks into their CIO ratings trigger additional questions about the degree to which information reported on the Dashboard provides full and accurate information about an investment’s risk. While agencies’ consideration of active risk is not explicitly called for by OMB’s guidance, this represents a gap in the agencies’ processes that is understating the amount of risk reflected in the Dashboard’s CIO ratings.

Recommendations for Executive Action

To better ensure that the Dashboard ratings more accurately reflect risk, we recommend that: the Secretaries of the Departments of Agriculture, Education, Energy, Health and Human Services, the Interior, State, and Veterans Affairs; and the Director of the Office of Personnel Management direct their CIOs to factor active risks into their IT Dashboard CIO ratings; the Secretaries of the Departments of Defense, Education, and Homeland Security; and the Commissioner of the Social Security Administration direct their CIOs to update their CIO ratings at least as frequently as required in OMB’s guidance; and the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, State, Transportation, the Treasury, Veterans Affairs; the Administrator of the Environmental Protection Agency; and the Commissioner of the Social Security Administration direct their CIOs to ensure that their CIO ratings reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals.

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB, the 15 agencies to which we made recommendations, and the remaining 2 to which we did not make recommendations. Specifically, 9 agencies agreed with our recommendations, 2 (Education and SSA) agreed with one or more while partially agreeing with another, 1 (DHS) agreed with one and disagreed with another, 1 (Defense) partially agreed with one and disagreed with another, 1 (EPA) disagreed, 1 (OMB) did not agree or disagree, and 1 (Treasury) did not comment on our recommendation. The 2 agencies without recommendations (Labor and GSA) stated that they had no comments. Multiple agencies also provided technical comments, which we have incorporated as appropriate. The following is a detailed discussion of each agency’s comments.

In comments provided via e-mail on April 25, 2016, an OMB official from the Office of General Counsel did not agree or disagree with our recommendations. OMB also provided technical comments, which we have incorporated as appropriate.

Our draft report to OMB for comment included a recommendation that Defense, Education, DHS, and SSA update their agencies’ CIO ratings on a monthly basis, as required by OMB’s fiscal year 2017 IT Budget Capital Planning Guidance. Subsequently, OMB informed us that the fiscal year 2018 Capital Planning Guidance will be revised to only require that agencies update the Dashboard as soon as new information becomes available or when CIO reviews are performed. Taking into account OMB’s planned course of action, we have modified our recommendation to those four agencies to reflect that they should, at a minimum, comply with OMB’s required reporting frequency.

In comments provided via e-mail on May 16, 2016, a Senior Advisor for Oversight and Compliance from Agriculture’s Office of the CIO stated that the department concurred with our recommendation. Agriculture also provided technical comments, which we incorporated as appropriate.

In written comments, Commerce concurred with our recommendation and committed to ensuring that the department’s CIO ratings will reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals. Commerce’s written comments are provided in appendix IV.

In written comments, Defense partially concurred with our recommendation to reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals. While Defense agreed with the need of CIO ratings to reflect risk, the department stated that its current CIO rating process already incorporates this factor. While we also agree that risk plays a role in Defense’s documented CIO ratings process, our findings indicate that the CIO ratings for the selected investments may have underreported investment risk. Specifically, our assessments for 19 of the selected 25 Defense investments (or 76 percent) show more risk than the CIO ratings on the Dashboard in April 2015. We therefore believe that our recommendation is appropriate.

In addition, the department did not concur with our recommendation to update its CIO ratings on a monthly basis. Specifically, the department states that its semi-annual reporting is consistent with FITARA requirements and is documented in the department’s OMB- approved FITARA implementation plan. As noted earlier, we recognize OMB’s plans to remove the monthly reporting requirement for CIO ratings and have modified this recommendation to reflect that planned change. We acknowledge that when this new policy is finalized, Defense’s semi-annual reporting may be in compliance with the new requirement. Until such time, agencies are still required by existing policy to report monthly and consequently, we believe that our recommendation is appropriate. Defense’s written comments are provided in appendix V.

In written comments, Education concurred with our recommendations to factor active risks into its CIO ratings and to have CIO ratings reflect the level of risk facing an investment relative to that investments’ ability to accomplish its goals, and described plans to implement those recommendations. Specifically, the department will include consideration of active risks when formulating its CIO ratings and its investment review board chair will provide specific guidance that the CIO should ensure the ratings reflect the level of risk facing an investment.

The department partially concurred with our recommendation to update CIO ratings monthly, stating that OMB’s fiscal year 2017 IT budget guidance addresses the required frequency of updates in several places, and the section specific to CIO evaluations only requires agencies to update their ratings as soon as new information becomes available. While we agree that OMB’s fiscal year 2017 guidance does address Dashboard reporting frequency in several places, the requirement for monthly updates is nonetheless explicitly stated and was confirmed by OMB staff. However, as noted earlier, we recognize OMB’s plans to remove the monthly reporting requirement for CIO ratings and have modified this recommendation to reflect that planned change. We acknowledge that when this new policy is finalized, Education’s reporting may be in compliance with the new requirement. Until such time, agencies are still required by existing policy to report monthly and consequently, we believe that our recommendation is appropriate. Education’s written comments are provided in appendix VI.

In written comments, Energy concurred with our recommendations and noted that the Office of the CIO would work collaboratively with IT executives to address the recommendations. Energy also provided technical comments, which we have incorporated as appropriate. Energy’s written comments are provided in appendix VII.

In written comments, HHS concurred with our recommendations, but noted that the recommendations were based on a now-outdated department methodology and that a new methodology, which went into effect in January 2016, addresses our recommendations. HHS also provided technical comments, which we have incorporated as appropriate. HHS’s written comments are provided in appendix VIII.

In written comments, DHS concurred with our recommendation to reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals.

The department did not concur with our recommendation to update its CIO ratings on a monthly basis, specifically noting that its risk-based process complies with OMB’s Fiscal Year 2017 Capital Planning Guidance with regards to the frequency of its updates to the IT Dashboard, and that investments receive health assessments on a risk-based basis, either monthly, quarterly, or semi-annually. However, we disagree with the assertion that this update frequency aligns with current OMB guidance, which explicitly requires that the Dashboard be updated at least monthly. However, as noted earlier, we recognize OMB’s plans to remove the monthly reporting requirement for CIO ratings and have modified this recommendation to reflect that planned change. We acknowledge that when this new policy is finalized, DHS’s reporting may be in compliance with the new requirement. Until such time, agencies are still required by existing policy to report monthly and consequently, we believe that our recommendation is appropriate. DHS also provided technical comments, which we have incorporated as appropriate. DHS’s written comments are provided in appendix IX.

In written comments, Interior concurred with our recommendation and noted it is currently enhancing the department’s CIO ratings process to maximize standardization across investment CIO ratings and to strengthen the assessment of active risks. Interior’s written comments are provided in appendix X.

In written comments, State agreed with our recommendations, noting that the department currently analyzes active risks and reviews those risks on a monthly basis. However, our review found that while State’s CIO ratings process included compliance with risk management issues (e.g., ensuring that the risk register is being updated and that mitigation strategies are properly planned), the department did not review active risks by evaluating the probability and impact of individual investment risks and applying that knowledge to the CIO ratings. By considering active risks, State can increase the chance that the ratings will better reflect the true likelihood of investment success. State’s written comments are presented in appendix XI.

In comments provided via e-mail on April 27, 2016, Transportation’s Director of Audit Relations and Program Improvement in the Office of the Secretary stated that the department concurred with our recommendation.

In comments provided via e-mail on May 10, 2016, a GAO liaison from Treasury’s Office of the CIO did not comment on our recommendation. Treasury also provided technical comments, which we have incorporated as appropriate.

In written comments, VA concurred with our recommendations and described actions the department is taking to address both of them. For our recommendation to factor active risks into its IT Dashboard CIO ratings, VA indicated that it would amend its current monthly process to include a requirement for investment managers to review at least the top three active operational risks. Additionally, for our recommendation to ensure that CIO ratings reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals, VA plans, among other things, to amend its current monthly process to include a requirement for investment managers to assess operational risks that detail the probability and impact of pending threats to success. VA’s written comments are presented in appendix XII.

In written comments, EPA did not agree with our recommendation. While agreeing that CIO ratings should reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals, the agency asserted that its current process already allows for this through the criteria used to determine an investment’s CIO rating. EPA specifically cited an EPA investment that was rated yellow, but we assessed as red. Further, EPA indicated that this disagreement does not mean that EPA’s process does not consider risks. We agree that the difference between EPA’s CIO rating and our assessment does not necessarily mean that EPA does not factor risk or the investment’s ability to accomplish its goals. However, it does indicate that the level of risk may be underreported. Consequently, we believe that our recommendation is appropriate. EPA also provided technical comments, which we have incorporated as appropriate. EPA’s written comments are provided in appendix XIII.

In written comments, OPM concurred with our recommendation and stated the agency will begin factoring active risks into CIO ratings. OPM’s written comments are provided in appendix XIV.

In written comments, SSA agreed with our recommendation to update its CIO ratings on a monthly basis. However, as noted earlier, we recognize OMB’s plans to remove the monthly reporting requirement for CIO ratings and have modified this recommendation to reflect that planned change.

SSA partially agreed with our recommendation to reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals. Noting that the agency head and the CIO should work together to appropriately consider investment risk, the agency disagreed with the implication that these individuals were not doing so. Additionally, SSA stated that it was too early in the implementation of FITARA to conclude that following OMB’s related guidance would result in agencies misestimating risk and that our report should not imply that SSA’s risk assessments do not fulfill the legislative intent of FITARA. While we recognize the collaborative efforts of SSA’s executives, our assessments nevertheless showed more risk for four out of nine selected SSA investments. Further, three of the four were rated green by SSA but assessed as red by us, indicating the possibility that these CIO ratings did not fully reflect the risk being faced by these investments. Additionally, our report does not conclude that OMB’s guidance leads to agencies misestimating risk, but rather that agencies’ processes are understating risk. As such, we believe that our recommendation is warranted. SSA’s written comments are provided in appendix XV.

Comments from the agencies to which we did not make recommendations are discussed in more detail here.

In comments provided via e-mail on April 18, 2016, a representative from Labor’s Office of the Assistant Secretary for Administration and Management stated that the department had no comments on the report.

In comments provided via e-mail on April 15, 2016, a representative from GSA’s GAO/IG Audit Response Division stated that the agency had no comments on the report.

We are sending copies of this report to interested congressional committees; the Secretaries of Agriculture, Commerce, Defense, Homeland Security, Education, Energy, Health and Human Services, the Interior, Labor, State, Transportation, the Treasury, and Veterans Affairs; the Administrator of the Environmental Protection Agency; the Administrator of the General Services Administration; the Director of the Office of Personnel Management; the Commissioner of the Social Security Administration; the Director of the Office of Management and Budget; and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov.

If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe agencies’ processes for determining the Chief Information Officer (CIO) risk ratings for major information technology (IT) investments and (2) assess the risk of federal IT investments and analyze any differences with the investments’ CIO risk ratings.

To select the agencies and investments, we reviewed data reported to OMB as part of the federal budget process to identify major investments that planned to spend at least 80 percent of their fiscal year 2015 funding on development, modernization, and enhancement activities. This produced a list 17 agencies and 107 selected investments. These agencies were: the Departments of Agriculture (Agriculture), Commerce (Commerce), Defense (Defense), Homeland Security (DHS), Education (Education), Energy (Energy), Health and Human Services (HHS), the Interior (Interior), Labor (Labor), State (State), Transportation (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), as well as the Environmental Protection Agency (EPA), the General Services Administration (GSA), the Social Security Administration (SSA), and the Office of Personnel Management (OPM). Appendix III contains a complete listing of the selected agencies and investments.

To address our first objective, we met with the 17 selected agencies to discuss their CIO rating processes and collected relevant documentation, such as capital planning and investment control guides, program health assessment guidance, and rating processes. We then compared the agencies’ processes to the Office of Management and Budget’s (OMB) suggested evaluation factors for creating CIO ratings. These evaluation factors are: (1) risk management, (2) requirements management, (3) contractor oversight, (4) historical performance, (5) human capital, and (6) other (i.e., the other factors that the CIO deems important to forecasting future success). We also analyzed the agencies’ documents and interviewed officials to determine how the agencies’ use of OMB’s factors varied.

To address our second objective, we reviewed the 107 investments, but excluded 12 after agencies told us that: 2 were inactive in our period of review, 5 were not primarily in development, 4 were too new to have a risk register (a key document for our assessments), and 1 was managed as part of a larger development program and did not have its own risk register. These exclusions eliminated the 2 selected investments from Labor and the 1 from OPM. This resulted in 95 investments at 15 agencies.

We made the decision to review the ratings from April 2015, the month that our audit work began, in order to minimize any influence that our ongoing work could have on the agencies’ processes and resulting ratings. We first downloaded the April ratings, interviewed appropriate officials at the 15 selected agencies, and collected March 2015 risk documentation (the data we would expect to be reflected in the April ratings), performance data, review board briefings, and relevant reports (e.g., GAO and Inspector General reports). In cases where agencies were unable to provide March documentation, we used documents from the closest available date. We did not consider risks that were introduced after March in these documents. We used this information to assess each selected investment’s overall investment risk and compared the result to the April 2015 CIO ratings.

According to OMB’s guidance, CIO ratings “should reflect the CIO’s assessment of the risk and the investment’s ability to accomplish its goals.” To create our assessments of risk, we combined each investment’s detailed risk lists, known as risk registers, with several additional metrics. Specifically, we combined the probability and impact of every active risk in the risk registers of each of the selected investments to determine what is known as the exposure of each risk. These exposure scores ranged from “very low” to “very high” and were based upon industry and government best practices. Table 10 shows how probability and impact values derived from these sources were combined to determine risk exposure.

We then weighted each risk exposure, placing significantly increased emphasis on higher risks so that they were not canceled out by lower risks. Table 11 lists the weights we assigned to the exposures.

We then averaged these weights and translated the result into green, yellow, and red grades according to the following scale.

For example, we would assess the following risk register as yellow.

We then lowered the assessments based on consideration of the following elements: (1) cost and schedule data from the IT Dashboard, (2) relevant review board briefings, and (3) relevant reports (e.g., GAO and Inspector General reports). Specifically, we first reviewed each investment’s project-level cost and schedule data from the Dashboard as of April 2015 and if more than half of its projects’ variances were colored red by the Dashboard, we lowered our assessment one level (i.e., from green to yellow or from yellow to red). As a result of this review, we identified a total of 19 investments; however, 11 of those could not be lowered further as we had already assessed them as red. The grades for the remaining 8 investments were lowered one level. In particular, 6 grades were lowered from yellow to red and 2 were lowered from green to yellow.

Then, we examined review board briefings covering March 2015, as well as relevant GAO and Inspector General reports, and lowered our assessment if we deemed the identified issues represented serious risks to the investment and remained relevant in April 2015. Using this approach, we considered lowering 12 of our assessments due to such information; however, 11 of the 12 assessments were already either red or were already lowered due to cost and schedule issues. Consequently, the only investment for which we lowered our assessment was Defense’s Warfighter Information Network-Tactical Increment 2, which was reduced from yellow to red. A briefing for that investment showed that the program’s cost variance triggered what is known as a Nunn-McCurdy breach, that the program was being restructured, that its scope was being reduced, and we had recently reported that the program struggled to demonstrate required performance and reliability during operational testing.

We then compared our assessment to the CIO ratings on the Dashboard, discussed our findings with agency officials, and corroborated the Dashboard’s data with agency officials. Our calculations are only intended to provide a standardized view of risk across all the departments and investments we reviewed, and this methodology is not intended to serve as a prescriptive approach to the agencies’ evaluation of investment risk.

We conducted this performance audit from April 2015 to June 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Agencies’ CIO Rating Processes

Table 14 describes the processes that the selected agencies reported as using to create the Chief Information Officer (CIO) ratings for their investments.

Appendix III: Agencies and Investments Selected for Review

Table 15 lists the selected agencies and investments, including those which we exempted (as discussed in appendix I and shaded in gray), as well as the associated Chief Information Officer (CIO) ratings and our assessments.

Appendix IV: Comments from the Department of Commerce

Appendix V: Comments from the Department of Defense

Appendix VI: Comments from the Department of Education

Appendix VII: Comments from the Department of Energy

Appendix VIII: Comments from the Department of Health and Human Services

Appendix IX: Comments from the Department of Homeland Security

Appendix X: Comments from the Department of the Interior

Appendix XI: Comments from the Department of State

Appendix XII: Comments from the Department of Veterans Affairs

Appendix XIII: Comments from the Environmental Protection Agency

Appendix XIV: Comments from the Office of Personnel Management

Appendix XV: Comments from the Social Security Administration

Appendix XVI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Karl Seifert (Assistant Director), Kevin Walsh (Assistant Director), Andrew Banister, Rebecca Eyler, Sandra Kerr, and Meredith Raymond.