Background

OMB plays a key role in overseeing how federal agencies manage their  IT investments by working with them to better plan, justify, and determine  how to manage them. To provide visibility into the performance of such  investments, OMB deployed the IT Dashboard in 2009, which displays  federal agencies’ cost, schedule, and performance data for over 770  major federal IT investments at 26 federal agencies, accounting for $42  billion of those agencies’ planned $82 billion in IT spending for fiscal year  2017. According to OMB, these data are intended to provide a near-real- time perspective on the performance of these investments, as well as a  historical perspective. The Dashboard’s data span the period from its  June 2009 inception to the present, and are based, in part, on agency  assessments of individual investment performance and each agency’s  budget request to OMB. Further, the public display of these data is  intended to allow OMB; other oversight bodies, including Congress; and  the general public to hold government agencies accountable for progress  and results.

		OMB’s Dashboard Is Intended to Provide Visibility into the Performance of Federal IT Investments

The Dashboard visually presents performance ratings for agencies and  for individual investments using metrics that OMB has defined—cost,  schedule, and CIO evaluation.
Cost and schedule ratings. The Dashboard calculates these ratings  by determining cost and schedule variances based on agency- submitted data, such as planned versus actual costs or planned  versus actual completion dates. The Dashboard then assigns rating  colors (red, yellow, green) based on the magnitude of the variances.  Specifically, a variance greater than 30 percent is red, a variance  between 10 percent and 30 percent is yellow, and a variance less  than 10 percent is green.
CIO ratings. Unlike the cost and schedule ratings, the Dashboard’s  “Investment Evaluation by Agency CIO” (also called the CIO rating) is  determined by the agency CIO. According to OMB’s instructions, each  agency CIO is to assess his or her IT investments against a set of  pre-established evaluation factors and then assign a rating of 1 (high  risk) to 5 (low risk) based on the CIO’s best judgment of the level of  risk facing the investment. OMB suggests six evaluation factors, as  shown in table 1.
OMB recommends that CIOs consult with appropriate stakeholders in  making their evaluations, including Chief Acquisition Officers, program  managers, and other interested parties. According to an OMB staff  member, agency CIOs are responsible for determining appropriate  thresholds for the risk levels and for applying them to investments when  assigning CIO ratings. OMB requires agencies to update these ratings as  soon as new information becomes available which will affect an  investment’s assessment and, since June 2015, has required that this be  at least once each calendar month. After agencies assign a level of risk to  each investment, the Dashboard assigns colors to CIO ratings according  to a five-point scale: high risk and moderately high risk are red, medium  risk is yellow, and moderately low risk and low risk are green.

		Recent Legislation Reinforced the Importance of the Dashboard

Recognizing the importance of government-wide management of IT, in  December 2014, Congress enacted IT acquisition reform legislation,  FITARA. The law was designed to improve agencies’ acquisition of IT and  enable Congress to monitor agencies’ progress and hold them  accountable for reducing duplication and achieving cost savings. FITARA  contains specific requirements related to seven areas, including one titled  “Enhanced Transparency and Improved Risk Management in Information  Technology Investments.” Among other things, that area requires OMB  and agencies to make publicly available detailed information on federal IT  investments, and agency CIOs to categorize their IT investments by risk.  This requirement is addressed by OMB’s IT Dashboard.

		GAO Has Previously Reported on the Dashboard

Over the past 5 years, we have issued a series of reports about the IT  Dashboard that noted both significant steps OMB has taken to enhance  the oversight, transparency, and accountability of federal IT investments  by creating its IT Dashboard, as well as issues with the accuracy and  reliability of data. Further, we have reported on the Dashboard’s CIO  ratings:  In October 2012, we reported that CIOs at six agencies rated a  majority of investments listed on the IT Dashboard as low or  moderately low risk from June 2009—when the Dashboard was  implemented—through March 2012 and two agencies, the  Department of Defense (Defense) and the National Science  Foundation, rated no investments as high or moderately high risk  during this time period (categorized as “red” by the Dashboard).  Additionally, agencies generally followed OMB’s instructions for  assigning CIO ratings, although Defense’s ratings were unique in  reflecting additional considerations, such as the likelihood of OMB  review. Most of the selected agencies reported various benefits  associated with producing and reporting CIO ratings, such as  increased quality of their performance data and greater transparency  and visibility of investments. We recommended that OMB analyze and  report on agencies’ CIO ratings over time, and that Defense ensure  that its CIO ratings reflect available investment performance  assessments and its risk management guidance. Both agencies  concurred with our recommendations. Subsequently, OMB reported  on CIO rating trends. Further, Defense now identifies red investments  on the Dashboard.
More recently in December 2013, we reported that, as of August  2013, the CIOs at eight selected agencies had rated 198 of their 244  major IT investments listed on the Dashboard as low risk or  moderately low risk, 41 as medium risk, and 5 as high risk or  moderately high risk. However, the total number of investments  reported by these agencies had varied over time, which impacted the  number of investments receiving CIO ratings. For example, the  Department of Energy (Energy) reclassified several of its  supercomputer investments from IT to facilities, and the Department  of Commerce (Commerce) decided to reclassify its satellite ground  system investments. Both decisions resulted in the removal of the  investments from the Dashboard, even though the investments were  clearly IT. We recommended that these agencies appropriately  categorize all investments, but they disagreed with our  recommendation.
In that same report, we reviewed 80 investments and found that 53 of  the CIO ratings were consistent with the investment risk, 20 were  partially consistent, and 7 were inconsistent. While two agencies’ CIO  ratings were entirely consistent, other agencies’ ratings were  inconsistent for a variety of reasons, including delays in updating the  Dashboard and how investment performance was tracked. For  example, the Social Security Administration (SSA) resets investment  cost and schedule performance baselines annually, an approach that  increases the risk of undetected cost or schedule variances that will  impact investment success. As such, we recommended that SSA  revise its investment management approach. The agency agreed with  our recommendation and discussed planned actions to address it.
Additionally, we reported that OMB does not update the public version  of the Dashboard as the President’s annual budget request is being  created. Consequently, the public version of the Dashboard was not  updated for 15 of the past 24 months. We recommended that OMB  make Dashboard information available independent of the budget  process. OMB recently updated the Dashboard with a number of  changes, and intends for the Dashboard to be able to show updates  throughout the year.

	Agencies Use a Variety of Processes to Determine Investments’ CIO Ratings

Agencies determine investments’ CIO ratings using a variety of  processes, which include OMB’s suggested factors. However, their  interpretation of these factors varies significantly. In addition, the majority  of agencies base their ratings on qualitative assessments, but several  base theirs on formulas. Further, 13 agencies’ process guidance calls for  at least monthly updates to CIO ratings, 1 agency (the Department of  Homeland Security (DHS)) schedules its reviews based on risk, and 3  agencies require less frequent updates.

		Agencies Use Many of OMB’s Suggested Factors to Determine CIO Ratings

As described earlier, OMB requires that each agency CIO rate the risk of  his or her IT investments. OMB gives the CIOs the flexibility to use their  judgment and suggests six evaluation factors.
As noted above, we reviewed data reported to OMB as part of the federal  budget process to identify major investments which planned to spend at  least 80 percent of their fiscal year 2015 funding on development,  modernization, and enhancement activities. This selection produced a list  of 17 agencies and 107 investments. Each of the 17 agencies has  incorporated at least 2 of OMB’s suggested factors into their CIO’s risk  rating processes and 9 use all of the factors. OMB requires that agencies  provide CIO evaluations for all major IT investments which reflect the  CIO’s best judgment of the current level of risk for the investment in terms  of its ability to accomplish its goals. According to OMB’s guidance, the  evaluation can be informed by the following factors, including, but not  limited to: risk management, requirements management, contractor  oversight, historical performance, human capital, and other factors that  the CIO deems important to forecasting future success.
Table 2 summarizes the extent to which the 17 selected agencies  incorporate OMB’s suggested evaluation factors into their CIO’s risk  rating processes. Appendix II provides more information on the selected  agencies’ CIO rating processes.
While the factors suggested by OMB were considered in the agencies’  CIO rating processes, their interpretation of these factors varied. In  particular, most agencies considered active risks when rating  investments, but others only evaluated compliance with risk processes.  For example:
Defense, the Department of the Treasury (Treasury), and the General  Services Administration (GSA) consider individual active risks, rather  than investments’ compliance with risk guidance. For example,  Defense considered the risk caused by budget cuts and their potential  impact through the following fiscal year when rating its Base  Information Transport Infrastructure Wired investment. In addition,  GSA, when rating its Integrated Award Environment investment,  considered the risk involved with transitioning from an existing  contract to one that could provide better expertise and greater  oversight.
The Departments of Agriculture (Agriculture), Education (Education),  Energy, the Interior (Interior), State (State), and Veterans Affairs (VA),  and the Office of Personnel Management (OPM) review compliance  with risk management practices, but do not assess active risks. For  example, compliance may include whether mitigation plans exist, risk  logs are current, and risks are clearly prioritized.
The rest of the agencies that include the risk factor—Commerce, DHS,  the Department of Labor (Labor), the Department of Transportation  (Transportation), as well as the Environmental Protection Agency (EPA)  and SSA—considered both process compliance and reviews of active  risks. For example, Commerce reviews at least the top three active risks  for investments, verifies that these risks are specific to the investment,  appropriately managed and mitigated, and that the risk register is updated  regularly. DHS also considers active investment risks, ensures that they  are current, and that risk mitigation plans are in place.
Furthermore, the selected agencies considered different types of  historical data when rating their IT investments. While all of the agencies  considered performance measures and cost and schedule variances, five  considered changes to the investment’s baseline; eight considered the  accomplishment of milestones; and three considered relevant news,  GAO, or Inspector General reports. While the details of these approaches  vary, they align with OMB’s suggested factors.

		Most Agencies Used Qualitative Rather than Quantitative Methodologies

Of the 17 selected agencies, 6 used formulas to create CIO ratings.  Specifically, Agriculture, the Department of Health and Human Services  (HHS), DHS, Education, Treasury, and VA determined their ratings by  quantifying and combining inputs such as cost and schedule variances,  risk exposure values, and compliance with agency processes. Metrics for  compliance with agency processes included those related to program and  project management, project execution, the quality of investment  documentation, and whether the investment is regularly updating risk  management plans and logs.
The remaining 11 agencies based their CIO ratings on qualitative  assessments of performance metrics, risks, investment documentation,  and informal investment knowledge. In particular, they assign ratings  based on metrics such as investment performance, discussions with  management staff, and the quality of investment documentation.

		Most Selected Agencies Require Monthly CIO Rating Updates

Thirteen agencies’ process guidance calls for at least monthly updates to  CIO ratings, one agency (DHS) schedules its reviews based on risk, and  three agencies require less frequent updates. Although a monthly review  and update process was not previously required by OMB, the fiscal year  2017 Capital Planning Guidance issued in June 2015 requires agencies  to update their CIO ratings at least once per month. Table 3 summarizes  the frequency of CIO rating updates called for by the selected agencies’  processes.
The three selected agencies that do not comply with OMB’s current  requirement for monthly rating updates are Defense, Education, and SSA.  In particular:
Defense updates CIO ratings semi-annually.
Education updates CIO ratings based on bi-monthly reviews of  investments.
SSA conducts monthly investment reviews, but updates CIO ratings  on a quarterly basis.
Additionally, DHS staggers its updates based on investment risk, with  high risk (red) investments reviewed monthly, moderate (yellow)  investments reviewed quarterly, and low (green) investments reviewed  semi-annually. These four agencies’ practices are inconsistent with  OMB’s guidance and can limit the transparency and oversight of the  government’s IT investments.
However, staff from OMB told us that the Capital Planning Guidance for  fiscal year 2018 would not contain the monthly reporting requirement and  would instead encourage agencies to keep their CIO ratings accurate and  current, rather than mandate reporting frequency. Moving forward, it will  be important for any such revised guidance to encourage the frequent  and appropriate updating of agencies’ CIO ratings while also remaining  compliant with relevant provisions of FITARA. These provisions require  that agencies report at least semi-annually to OMB on each major IT  investment and include data on cost, schedule, and performance. These  provisions also require joint OMB and agency reviews of any investment  that has been evaluated as high risk for four consecutive quarters.

	Our Assessments Reflected More Risk than Most Selected Investments’ CIO Ratings

As discussed earlier, to assess the risk of individual assessments, we  reviewed the 107 investments that we originally selected, but excluded 12  that were inactive, not in development, lacked a key risk document, or  were managed as part of a larger development program. This resulted in  95 investments at 15 agencies. Our assessments of these investments  generally showed more risk than the associated CIO ratings. In particular,  of the 95 investments we reviewed, our assessments matched the CIO  ratings 22 times, showed more risk 60 times, and showed less risk 13  times. We identified three factors which contributed to these differences:  (1) 40 of the 95 CIO ratings were not updated in April 2015, (2) three  agencies’ rating processes span longer than 1 month, and (3) seven  agencies’ rating processes did not focus on active risks (as previously  discussed).
According to OMB’s guidance, CIO ratings “should reflect the CIO’s  assessment of the risk and the investment’s ability to accomplish its  goals.” Such assessments of risk inherently involve a great deal of  human judgment. Consequently, risk assessments should be expected to  vary both across and within organizations. For example, Defense’s CIO  ratings process documentation states that, since its major investments  are “inherently high risk,” its ratings are “assessments of relative risk  implemented within this risk baseline.” That is, when measuring risk,  Defense is more tolerant and uses a different scale than other agencies.  Similarly, risk assessments can vary within agencies. For example,  officials at several agencies expressed concerns that the assignment of  risk scores (probability and impact) were not consistent across  investments. Officials at DHS also noted that program managers may  score risks higher to flag an issue for management attention. Further, in  many cases, agency CIOs could have more information than we  examined in our assessments.
We attempted to minimize the subjectivity in our risk assessments by  using the agencies’ own lists of risks, known as risk registers, as the  basis of our assessments (see appendix I for additional details on our  methodology). We also augmented our ratings with agencies’ cost and  schedule data, briefings to review boards, and relevant reports. Our  calculations are only intended to provide a standardized view of risk  across all the departments and investments we reviewed and this  methodology is not intended to serve as a prescriptive approach to the  agencies’ evaluation of investment risk.

		Our Assessments Showed More Risk than Almost Two Thirds of Selected Investments’ CIO Ratings

While the variety of methodologies and inputs meant that some  differences were inevitable, almost two thirds of our assessments showed  more risk than the associated CIO ratings for our 95 selected  investments. Figure 1 summarizes how our assessments compared to the  select investments’ CIO ratings.
Of the 95 investments we reviewed, our assessments showed less risk 13  times, matched the CIO ratings 22 times, and showed more risk 60 times.  Additionally, our assessments showed more risk for at least 1 investment  at 13 of the 15 agencies we assessed. Table 4 summarizes these  comparisons by agency, and appendix III lists the April 2015 CIO ratings  and our assessments for each of the selected investments.
Overall, our assessments reflected more risk than 63 percent of the  associated CIO ratings, and 13 of the 15 agencies reported less risk for at  least 1 investment. Of the 13, 11 reported less risk for at least half of the  selected investments and the remaining 2 reported less risk for just under  half of those agencies’ selected investments. For example, we identified  more risk at 3 of DHS’s 7 investments and 4 of SSA’s 9 investments.

			Our Assessments Reflected Less Risk than the Agencies’ CIO Ratings for 13 Investments

Our assessments showed less risk than the CIO ratings for 13 of the 95  selected investments (14 percent). Specifically, we assessed 5 green that  the agencies rated yellow and 8 yellow that the agencies rated red; there  were no instances where we assessed an investment green that the  agencies rated red. These investments belonged to 8 of the 15 selected  agencies: HHS (3 investments), Defense (2 investments), DHS (2  investments), Education (2 investments), Commerce (1 investment),  Energy (1 investment), GSA (1 investment), and SSA (1 investment).  Table 5 lists those investments, the April 2015 CIO rating, and our  associated assessment.
The reasons why our assessments showed less risk varied among these  investments. For instance:
Commerce rated its Integrated Dissemination Program as yellow, but  we assessed it as green. A department official explained that the  rating was because (1) the investment had not been transparent in its  activities making it difficult to determine whether a milestone was  achieved or services were provided and (2) the investment had only  identified generic, non-specific risks. Further, the official noted that the  department had even debated rating this investment red. We  assessed this investment as green because of the low overall level of  risk and low cost and schedule variances.
DHS rated its Continuous Diagnostics and Mitigation investment as  yellow, but we assessed it as green. DHS officials stated that the  investment was rated yellow because (1) it was considered complex  and higher risk due to the involvement of several civilian government  agencies, (2) cost and schedule variances exceeded OMB’s  thresholds, and (3) concerns about the schedule and availability of  resources. In contrast, we assessed it as green because it did not  have any risks with both high impact and high probability scores and  more than half of the risks scored low in overall risk exposure.
HHS rated its Medicaid and Children’s Health Insurance Program  Business Information and Solutions investment as red, but we  assessed it as yellow. HHS officials rated this investment red because  the investment team did not submit required data and a rebaseline  caused a large cost variance. Even though the investment’s overall  risk score was low, we assessed the investment as yellow based  upon documented cost and schedule variances and program issues  identified in review board briefings.
GSA rated its Integrated Award Environment as yellow, but we  assessed it as green. GSA officials stated that the investment was  rated yellow as a precaution: the investment was undergoing a  contract transition and the CIO knew that problems could develop.  GSA officials stated that another contributing factor was the  investment’s late contract award, which had residual impact on  investment performance. We assessed this investment green  because of its overall low risk score.
As noted earlier, CIO ratings are intended to reflect the CIO’s assessment  of the risk and may be based on additional programmatic information not  included in our assessment methodology, which focused primarily on  investments’ risk registers. As such, the inherently judgmental nature of  the CIOs’ assessments may reflect broader considerations that, in their  organization’s view, better represent the overall risk of an investment.

			Our Assessments Matched the Agencies’ CIO Ratings for 22 Investments

For 22 of the 95 selected investments (23 percent), our assessments  matched the CIO rating. Specifically, we matched 10 green ratings, 8  yellow ratings, and 4 red ratings. Table 6 lists those investments, the April  2015 CIO rating, and our associated assessment.
In particular, there were 4 investments at Defense, 4 at SSA, 4 at  Transportation, 3 at Commerce, 2 at GSA, 2 at DHS, 1 at HHS, 1 at  Interior, and 1 at State that had CIO ratings that matched our  assessments. These investments were a mix of red, yellow, and green  ratings. However, the reasoning behind the CIO ratings and our individual  assessments differed. For example, Interior rated its Integrated Reporting  of Wildland-Fire Information investment as yellow because the  investment’s required documentation did not meet agency standards.
Specifically, the investment’s most recent artifact review before the April  2015 CIO rating period showed that the investment lacked required  documentation, including a risk management plan. However, we  assessed the investment as yellow because the IT Dashboard showed  significant cost and schedule variances at the time of our review.

			Our Assessments Reflected More Risk than the Agencies’ CIO Ratings for 60 Investments

For 60 of the 95 selected investments (63 percent), our assessments  reflected more risk than agencies’ CIO ratings. Specifically, we assessed  9 red that the agencies rated yellow, 28 yellow that the agencies rated  green, and 23 red that the agencies rated green. Further, these  investments were at 13 of the 15 agencies we assessed. Table 7 lists  those investments, the April 2015 CIO rating, and our associated  assessment.
The agencies’ explanations as to why our assessments showed more risk  varied in these 60 cases. For example,
Agriculture rated its Optimized Computing Environment investment as  green, but we assessed it as red. While Agriculture officials noted that  this investment’s funding from partner agencies was uncertain, the  investment received a green CIO rating because the funding  uncertainties were a recurring concern that had been previously  managed without issue. Conversely, we assessed the investment as  red because 21 of the 44 risks in the investment’s risk register had  high overall risk scores.
DHS rated its Next Generation Networks Priority Services investment  as green, but we assessed it as red. DHS officials stated that they  rated this investment green because the investment was progressing  well and because it had successfully mitigated its high impact/high  probability risks. However, our assessment was partly based on two  risks with both high probability and high impact scores that the  investment team categorized as potentially causing investment failure.  These risk scores and descriptions indicated that the program  believed that it was likely that these risks would be realized and cause  critical, perhaps investment-threatening problems. DHS officials  questioned the probability and impact scores and explained that  investment teams may inflate such scores to flag potential issues for  management.
SSA rated its Supplemental Security Income Modernization  investment as green, but we assessed it as red. SSA officials stated  that their review did not see significant reason to lower the rating,  even though part of the investment was working through significant  technical challenges. We assessed the investment red because half  of its risks had high risk scores. Included in the most critical risks were  those pertaining to requirements changes, system complexity, and  staffing losses. When asked about these risks, SSA officials explained  that, generally, they were common risks faced by every investment.  Consequently, they expressed doubt that these risks necessitated  such high risk scores.
Treasury rated its Post Payment System investment as green, but we  assessed it as red. Treasury officials stated that this investment was  rated green because it was well-run and had previously kept its risks  from becoming realized issues. Conversely, we rated this investment  as red because 20 out of its 26 risks had high overall risk scores,  including 2 that indicated a high probability of schedule delays.  Treasury officials stated that they were monitoring the risks that we  identified.
As noted earlier, the judgmental nature of a CIO’s assessment may  reflect a broader organizational view of investment risk beyond the  contents of the investment’s risk register. However, unlike the CIO ratings  that reflected more risk than our assessments, many of these CIO ratings  minimized the potential severity and impact of high risk scores. Our past  work has shown that such an approach to risk management can often  lead to cost and schedule overruns or failed projects.

		Three Issues Contributed to Discrepancies between Agencies’ CIO Ratings and Our Assessments

In addition to the previously discussed issue of rating subjectivity, we  identified three factors which contributed to differences between our  assessments and CIO ratings at 10 of the 15 selected agencies. In  particular,  ratings were not updated in April 2015,  rating processes spanned longer than 1 month, and  rating processes did not focus on active risks.
Rather than pertaining to the CIOs’ personal evaluations of risk, these  additional issues relate to the 10 agencies’ update practices or rating  processes. Because these issues are with underlying practices and  processes, they have the potential to impact all investment ratings— whether or not we reviewed them as part of our assessment. Specifically,  we found that 40 of the 95 CIO ratings were not updated in April 2015, 3  agencies’ rating processes span longer than 1 month, and 7 agencies’  rating processes did not focus on active risks (see table 8). Following the  table is a further description of these issues.
Of the 95 investments we selected, we found that agencies had not  updated CIO ratings for 40 in April 2015 (see table 9).
Further, we found that our assessments were more likely to match those  investments updated in April because these recently updated ratings  reflected the then-current investment information that we also used in our  assessments. In particular, 17 of the 22 investments where our  assessment matched the CIOs’ ratings were updated in April 2015,  whereas we matched only 5 of the 40 without April 2015 updates. Of the  40 ratings that were not updated, 15 were at 6 agencies, which provided  the following explanations for the lack of April updates:
Agriculture officials stated that they were in the midst of switching  systems used to update ratings to the Dashboard, so they were  unable to update ratings that month.
Commerce officials stated that they did not receive quality data from  the investment until the following month and thus did not have a valid  basis for changing the rating or CIO comments.
DHS officials stated that they did not update 2 of the selected  investments’ ratings because they were undergoing TechStat  reviews.  Additionally, 2 investments’ CIO ratings were not updated  in April even though they initially received April reviews. According to  officials, 1 investment’s review occurred too late in the month and the  other investment’s update was delayed. They also stated that they  were revising their process so that rating updates would occur during  such reviews. Additionally, DHS’s risk-based review cycle meant that  two investments were not due to be rated in April. In particular, those  investments were rated yellow less than 3 months prior and DHS  reviews yellow-rated investments on a quarterly basis.
An Education official explained that the agency only updates its  ratings if an investment’s status has changed enough to warrant an  update.
EPA officials told us that its selected investment was not updated in  April 2015 because the investment was in a state of flux during that  time and a review would not have been useful.
HHS officials stated that a combination of system and human errors  kept 2 investment updates from posting.
The remaining 25 investments which did not have an April 2015 update  were at Defense, which updates its ratings on a semi-annual basis (see  earlier discussion of CIO rating update frequency). If we had used  Defense’s subsequent update from June 2015, the number of instances  in which the CIO ratings matched our assessments would have increased  from 4 to 10.
The preceding examples underscore the importance of OMB’s June 2015  policy change, requiring that agencies provide monthly rating updates.  Such updates will help to ensure that the information on the Dashboard is  timely and accurately reflects recent changes. Without such updates, the  CIO ratings on the Dashboard may not reflect the current level of  investment risk.

			Three Selected Agencies’ Rating Processes Took Longer than 1 Month

The duration of agencies’ CIO rating processes also impacted the  comparison between the CIO ratings and our assessment. Since we used  March 2015 risk registers as the basis for our assessment, process times  longer than a month mean that the data we used for our assessment  would not have been included in agencies’ April 2015 CIO ratings.  Fourteen of the 17 selected agencies indicated that it takes 1 month or  less to process investment data and update the IT Dashboard. However,  processes at 3 agencies—Education, State, and Treasury—can be longer  than 1 month. For example, Treasury officials stated that late invoices  from contractors can delay their processes past the 1 month time frame.  State officials explained that their conversion of certain cost and schedule  data into the format required by the Dashboard can take between 1 and 2  months to complete. Longer processes mean that CIO ratings are based  upon older data and may not reflect the current level of investment risk.  Further, these longer processes may prevent the agencies from meeting  OMB’s requirement to update ratings monthly.

			Seven Selected Agencies Did Not Factor Active Risks into Ratings

When developing CIO ratings, seven agencies did not consider active  risks, as discussed earlier. Six of those agencies (Agriculture, Education,  Energy, Interior, State, and VA) instead chose to focus on investments’  risk management processes, such as whether a process was in place or  whether a risk log was current. Such approaches did not consider  individual risks, such as funding cuts or staffing changes, which detail the  probability and impact of pending threats to success. Instead, VA’s CIO  rating process considers several specific risk management criteria:  whether an investment (1) has a risk management strategy, (2) keeps the  risk register current and complete, (3) clearly prioritizes risks, and (4) puts  mitigation plans in place to address risks.
Considering process compliance, rather than active risks, contributed to  our assessments only matching 2 of the CIO ratings for the 14 selected  investments at these six agencies. The remaining agency, HHS, did not  factor risk into its CIO ratings (as discussed earlier). This contributed to  our assessments matching only 1 of the 9 selected HHS investments. In  all cases, CIO ratings that do not incorporate active risks increase the  chance that ratings do not reflect the true likelihood of investment  success.

	Conclusions

Since its inception in 2009, the IT Dashboard has increased the  transparency of the government’s multi-billion dollar spending on major IT  investments. The Dashboard’s CIO ratings, in particular, have improved  visibility into the risks facing these critically important efforts. To that end,  agency CIOs have developed a variety of processes to assess and report  the risk of their investments. Although the effectiveness of the Dashboard  depends on the quality of the CIOs’ ratings, selected agencies’ rating  methods do not provide an accurate assessment of investment risk and  thus reduce the value of this important tool for transparency and  oversight. Further, multiple agencies’ infrequent submissions raise  concerns that those updates are not reflecting timely and accurate risk  information, contrary to OMB’s current policy requiring monthly updates.  Such practices limit the transparency and oversight of the government’s  billions of dollars in IT investments.
Beyond the transparency they provide, CIO ratings present an opportunity  to improve CIOs’ understanding of their IT portfolio and identify those  investments in need of additional oversight. While these ratings are by  definition inherently judgmental, our assessments of the selected  investments generally showed more risk than almost 65 percent of the  associated CIO ratings. Consequently, the associated risk rating  processes used by the agencies generally are understating the level of  risk, raising the likelihood that critical federal investments in IT are not  receiving the appropriate levels of oversight. Finally, agencies that do not  factor active risks into their CIO ratings trigger additional questions about  the degree to which information reported on the Dashboard provides full  and accurate information about an investment’s risk. While agencies’  consideration of active risk is not explicitly called for by OMB’s guidance,  this represents a gap in the agencies’ processes that is understating the  amount of risk reflected in the Dashboard’s CIO ratings.

	Recommendations for Executive Action

To better ensure that the Dashboard ratings more accurately reflect risk,  we recommend that:  the Secretaries of the Departments of Agriculture, Education, Energy,  Health and Human Services, the Interior, State, and Veterans Affairs;  and the Director of the Office of Personnel Management direct their  CIOs to factor active risks into their IT Dashboard CIO ratings;  the Secretaries of the Departments of Defense, Education, and  Homeland Security; and the Commissioner of the Social Security  Administration direct their CIOs to update their CIO ratings at least as  frequently as required in OMB’s guidance; and  the Secretaries of the Departments of Agriculture, Commerce,  Defense, Education, Energy, Health and Human Services, Homeland  Security, State, Transportation, the Treasury, Veterans Affairs; the  Administrator of the Environmental Protection Agency; and the  Commissioner of the Social Security Administration direct their CIOs  to ensure that their CIO ratings reflect the level of risk facing an  investment relative to that investment’s ability to accomplish its goals.

	Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB, the 15  agencies to which we made recommendations, and the remaining 2 to  which we did not make recommendations. Specifically, 9 agencies agreed  with our recommendations, 2 (Education and SSA) agreed with one or  more while partially agreeing with another, 1 (DHS) agreed with one and  disagreed with another, 1 (Defense) partially agreed with one and  disagreed with another, 1 (EPA) disagreed, 1 (OMB) did not agree or  disagree, and 1 (Treasury) did not comment on our recommendation. The  2 agencies without recommendations (Labor and GSA) stated that they  had no comments. Multiple agencies also provided technical comments,  which we have incorporated as appropriate. The following is a detailed  discussion of each agency’s comments.
In comments provided via e-mail on April 25, 2016, an OMB official  from the Office of General Counsel did not agree or disagree with our  recommendations. OMB also provided technical comments, which we  have incorporated as appropriate.
Our draft report to OMB for comment included a recommendation that  Defense, Education, DHS, and SSA update their agencies’ CIO  ratings on a monthly basis, as required by OMB’s fiscal year 2017 IT  Budget Capital Planning Guidance. Subsequently, OMB informed us  that the fiscal year 2018 Capital Planning Guidance will be revised to  only require that agencies update the Dashboard as soon as new  information becomes available or when CIO reviews are performed.  Taking into account OMB’s planned course of action, we have  modified our recommendation to those four agencies to reflect that  they should, at a minimum, comply with OMB’s required reporting  frequency.
In comments provided via e-mail on May 16, 2016, a Senior Advisor  for Oversight and Compliance from Agriculture’s Office of the CIO  stated that the department concurred with our recommendation.  Agriculture also provided technical comments, which we incorporated  as appropriate.
In written comments, Commerce concurred with our recommendation  and committed to ensuring that the department’s CIO ratings will  reflect the level of risk facing an investment relative to that  investment’s ability to accomplish its goals. Commerce’s written  comments are provided in appendix IV.
In written comments, Defense partially concurred with our  recommendation to reflect the level of risk facing an investment  relative to that investment’s ability to accomplish its goals. While  Defense agreed with the need of CIO ratings to reflect risk, the  department stated that its current CIO rating process already  incorporates this factor. While we also agree that risk plays a role in  Defense’s documented CIO ratings process, our findings indicate that  the CIO ratings for the selected investments may have underreported  investment risk. Specifically, our assessments for 19 of the selected  25 Defense investments (or 76 percent) show more risk than the CIO  ratings on the Dashboard in April 2015. We therefore believe that our  recommendation is appropriate.
In addition, the department did not concur with our recommendation to  update its CIO ratings on a monthly basis. Specifically, the  department states that its semi-annual reporting is consistent with  FITARA requirements and is documented in the department’s OMB- approved FITARA implementation plan. As noted earlier, we  recognize OMB’s plans to remove the monthly reporting requirement  for CIO ratings and have modified this recommendation to reflect that  planned change. We acknowledge that when this new policy is  finalized, Defense’s semi-annual reporting may be in compliance with  the new requirement. Until such time, agencies are still required by  existing policy to report monthly and consequently, we believe that our  recommendation is appropriate. Defense’s written comments are  provided in appendix V.
In written comments, Education concurred with our recommendations  to factor active risks into its CIO ratings and to have CIO ratings  reflect the level of risk facing an investment relative to that  investments’ ability to accomplish its goals, and described plans to  implement those recommendations. Specifically, the department will  include consideration of active risks when formulating its CIO ratings  and its investment review board chair will provide specific guidance  that the CIO should ensure the ratings reflect the level of risk facing  an investment.
The department partially concurred with our recommendation to  update CIO ratings monthly, stating that OMB’s fiscal year 2017 IT  budget guidance addresses the required frequency of updates in  several places, and the section specific to CIO evaluations only  requires agencies to update their ratings as soon as new information  becomes available. While we agree that OMB’s fiscal year 2017  guidance does address Dashboard reporting frequency in several  places, the requirement for monthly updates is nonetheless explicitly  stated and was confirmed by OMB staff. However, as noted earlier,  we recognize OMB’s plans to remove the monthly reporting  requirement for CIO ratings and have modified this recommendation  to reflect that planned change. We acknowledge that when this new  policy is finalized, Education’s reporting may be in compliance with  the new requirement. Until such time, agencies are still required by  existing policy to report monthly and consequently, we believe that our  recommendation is appropriate. Education’s written comments are  provided in appendix VI.
In written comments, Energy concurred with our recommendations  and noted that the Office of the CIO would work collaboratively with IT  executives to address the recommendations. Energy also provided  technical comments, which we have incorporated as appropriate.  Energy’s written comments are provided in appendix VII.
In written comments, HHS concurred with our recommendations, but  noted that the recommendations were based on a now-outdated  department methodology and that a new methodology, which went  into effect in January 2016, addresses our recommendations. HHS  also provided technical comments, which we have incorporated as  appropriate. HHS’s written comments are provided in appendix VIII.
In written comments, DHS concurred with our recommendation to  reflect the level of risk facing an investment relative to that  investment’s ability to accomplish its goals.
The department did not concur with our recommendation to update its  CIO ratings on a monthly basis, specifically noting that its risk-based  process complies with OMB’s Fiscal Year 2017 Capital Planning  Guidance with regards to the frequency of its updates to the IT  Dashboard, and that investments receive health assessments on a  risk-based basis, either monthly, quarterly, or semi-annually.  However, we disagree with the assertion that this update frequency  aligns with current OMB guidance, which explicitly requires that the  Dashboard be updated at least monthly. However, as noted earlier,  we recognize OMB’s plans to remove the monthly reporting  requirement for CIO ratings and have modified this recommendation  to reflect that planned change. We acknowledge that when this new  policy is finalized, DHS’s reporting may be in compliance with the new  requirement. Until such time, agencies are still required by existing  policy to report monthly and consequently, we believe that our  recommendation is appropriate. DHS also provided technical  comments, which we have incorporated as appropriate. DHS’s written  comments are provided in appendix IX.
In written comments, Interior concurred with our recommendation and  noted it is currently enhancing the department’s CIO ratings process  to maximize standardization across investment CIO ratings and to  strengthen the assessment of active risks. Interior’s written comments  are provided in appendix X.
In written comments, State agreed with our recommendations, noting  that the department currently analyzes active risks and reviews those  risks on a monthly basis. However, our review found that while State’s  CIO ratings process included compliance with risk management  issues (e.g., ensuring that the risk register is being updated and that  mitigation strategies are properly planned), the department did not  review active risks by evaluating the probability and impact of  individual investment risks and applying that knowledge to the CIO  ratings. By considering active risks, State can increase the chance  that the ratings will better reflect the true likelihood of investment  success. State’s written comments are presented in appendix XI.
In comments provided via e-mail on April 27, 2016, Transportation’s  Director of Audit Relations and Program Improvement in the Office of  the Secretary stated that the department concurred with our  recommendation.
In comments provided via e-mail on May 10, 2016, a GAO liaison  from Treasury’s Office of the CIO did not comment on our  recommendation. Treasury also provided technical comments, which  we have incorporated as appropriate.
In written comments, VA concurred with our recommendations and  described actions the department is taking to address both of them.  For our recommendation to factor active risks into its IT Dashboard  CIO ratings, VA indicated that it would amend its current monthly  process to include a requirement for investment managers to review  at least the top three active operational risks. Additionally, for our  recommendation to ensure that CIO ratings reflect the level of risk  facing an investment relative to that investment’s ability to accomplish  its goals, VA plans, among other things, to amend its current monthly  process to include a requirement for investment managers to assess  operational risks that detail the probability and impact of pending  threats to success. VA’s written comments are presented in appendix  XII.
In written comments, EPA did not agree with our recommendation.  While agreeing that CIO ratings should reflect the level of risk facing  an investment relative to that investment’s ability to accomplish its  goals, the agency asserted that its current process already allows for  this through the criteria used to determine an investment’s CIO rating.  EPA specifically cited an EPA investment that was rated yellow, but  we assessed as red. Further, EPA indicated that this disagreement  does not mean that EPA’s process does not consider risks. We agree  that the difference between EPA’s CIO rating and our assessment  does not necessarily mean that EPA does not factor risk or the  investment’s ability to accomplish its goals. However, it does indicate  that the level of risk may be underreported. Consequently, we believe  that our recommendation is appropriate. EPA also provided technical  comments, which we have incorporated as appropriate. EPA’s written  comments are provided in appendix XIII.
In written comments, OPM concurred with our recommendation and  stated the agency will begin factoring active risks into CIO ratings.  OPM’s written comments are provided in appendix XIV.
In written comments, SSA agreed with our recommendation to update  its CIO ratings on a monthly basis. However, as noted earlier, we  recognize OMB’s plans to remove the monthly reporting requirement  for CIO ratings and have modified this recommendation to reflect that  planned change.
SSA partially agreed with our recommendation to reflect the level of  risk facing an investment relative to that investment’s ability to  accomplish its goals. Noting that the agency head and the CIO should  work together to appropriately consider investment risk, the agency  disagreed with the implication that these individuals were not doing  so. Additionally, SSA stated that it was too early in the implementation  of FITARA to conclude that following OMB’s related guidance would  result in agencies misestimating risk and that our report should not  imply that SSA’s risk assessments do not fulfill the legislative intent of  FITARA. While we recognize the collaborative efforts of SSA’s  executives, our assessments nevertheless showed more risk for four  out of nine selected SSA investments. Further, three of the four were  rated green by SSA but assessed as red by us, indicating the  possibility that these CIO ratings did not fully reflect the risk being  faced by these investments. Additionally, our report does not conclude  that OMB’s guidance leads to agencies misestimating risk, but rather  that agencies’ processes are understating risk. As such, we believe  that our recommendation is warranted. SSA’s written comments are  provided in appendix XV.
Comments from the agencies to which we did not make  recommendations are discussed in more detail here.
In comments provided via e-mail on April 18, 2016, a representative  from Labor’s Office of the Assistant Secretary for Administration and  Management stated that the department had no comments on the  report.
In comments provided via e-mail on April 15, 2016, a representative  from GSA’s GAO/IG Audit Response Division stated that the agency  had no comments on the report.
We are sending copies of this report to interested congressional  committees; the Secretaries of Agriculture, Commerce, Defense,  Homeland Security, Education, Energy, Health and Human Services, the  Interior, Labor, State, Transportation, the Treasury, and Veterans Affairs;  the Administrator of the Environmental Protection Agency; the  Administrator of the General Services Administration; the Director of the  Office of Personnel Management; the Commissioner of the Social  Security Administration; the Director of the Office of Management and  Budget; and other interested parties. This report will also be available at  no charge on our website at http://www.gao.gov.
If you or your staff have any questions on matters discussed in this report,  please contact me at (202) 512-9286 or pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix XVI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe agencies’ processes for determining  the Chief Information Officer (CIO) risk ratings for major information  technology (IT) investments and (2) assess the risk of federal IT  investments and analyze any differences with the investments’ CIO risk  ratings.
To select the agencies and investments, we reviewed data reported to  OMB as part of the federal budget process to identify major investments  that planned to spend at least 80 percent of their fiscal year 2015 funding  on development, modernization, and enhancement activities. This  produced a list 17 agencies and 107 selected investments. These  agencies were: the Departments of Agriculture (Agriculture), Commerce  (Commerce), Defense (Defense), Homeland Security (DHS), Education  (Education), Energy (Energy), Health and Human Services (HHS), the  Interior (Interior), Labor (Labor), State (State), Transportation  (Transportation), the Treasury (Treasury), and Veterans Affairs (VA), as  well as the Environmental Protection Agency (EPA), the General Services  Administration (GSA), the Social Security Administration (SSA), and the  Office of Personnel Management (OPM). Appendix III contains a  complete listing of the selected agencies and investments.
To address our first objective, we met with the 17 selected agencies to  discuss their CIO rating processes and collected relevant documentation,  such as capital planning and investment control guides, program health  assessment guidance, and rating processes. We then compared the  agencies’ processes to the Office of Management and Budget’s (OMB)  suggested evaluation factors for creating CIO ratings. These evaluation  factors are: (1) risk management, (2) requirements management, (3)  contractor oversight, (4) historical performance, (5) human capital, and (6)  other (i.e., the other factors that the CIO deems important to forecasting  future success). We also analyzed the agencies’ documents and  interviewed officials to determine how the agencies’ use of OMB’s factors  varied.
To address our second objective, we reviewed the 107 investments, but  excluded 12 after agencies told us that: 2 were inactive in our period of  review, 5 were not primarily in development, 4 were too new to have a  risk register (a key document for our assessments), and 1 was managed  as part of a larger development program and did not have its own risk  register. These exclusions eliminated the 2 selected investments from  Labor and the 1 from OPM. This resulted in 95 investments at 15  agencies.
We made the decision to review the ratings from April 2015, the month  that our audit work began, in order to minimize any influence that our  ongoing work could have on the agencies’ processes and resulting  ratings. We first downloaded the April ratings, interviewed appropriate  officials at the 15 selected agencies, and collected March 2015 risk  documentation (the data we would expect to be reflected in the April  ratings), performance data, review board briefings, and relevant reports  (e.g., GAO and Inspector General reports). In cases where agencies were  unable to provide March documentation, we used documents from the  closest available date. We did not consider risks that were introduced  after March in these documents. We used this information to assess each  selected investment’s overall investment risk and compared the result to  the April 2015 CIO ratings.
According to OMB’s guidance, CIO ratings “should reflect the CIO’s  assessment of the risk and the investment’s ability to accomplish its  goals.” To create our assessments of risk, we combined each  investment’s detailed risk lists, known as risk registers, with several  additional metrics. Specifically, we combined the probability and impact of  every active risk in the risk registers of each of the selected investments  to determine what is known as the exposure of each risk. These  exposure scores ranged from “very low” to “very high” and were based  upon industry and government best practices. Table 10 shows how  probability and impact values derived from these sources were combined  to determine risk exposure.
We then weighted each risk exposure, placing significantly increased  emphasis on higher risks so that they were not canceled out by lower  risks. Table 11 lists the weights we assigned to the exposures.
We then averaged these weights and translated the result into green,  yellow, and red grades according to the following scale.
For example, we would assess the following risk register as yellow.
We then lowered the assessments based on consideration of the  following elements: (1) cost and schedule data from the IT Dashboard, (2)  relevant review board briefings, and (3) relevant reports (e.g., GAO and  Inspector General reports). Specifically, we first reviewed each  investment’s project-level cost and schedule data from the Dashboard as  of April 2015 and if more than half of its projects’ variances were colored  red by the Dashboard, we lowered our assessment one level (i.e., from  green to yellow or from yellow to red). As a result of this review, we  identified a total of 19 investments; however, 11 of those could not be  lowered further as we had already assessed them as red. The grades for  the remaining 8 investments were lowered one level. In particular, 6  grades were lowered from yellow to red and 2 were lowered from green to  yellow.
Then, we examined review board briefings covering March 2015, as well  as relevant GAO and Inspector General reports, and lowered our  assessment if we deemed the identified issues represented serious risks  to the investment and remained relevant in April 2015. Using this  approach, we considered lowering 12 of our assessments due to such  information; however, 11 of the 12 assessments were already either red  or were already lowered due to cost and schedule issues. Consequently,  the only investment for which we lowered our assessment was Defense’s  Warfighter Information Network-Tactical Increment 2, which was reduced  from yellow to red. A briefing for that investment showed that the  program’s cost variance triggered what is known as a Nunn-McCurdy  breach, that the program was being restructured, that its scope was  being reduced, and we had recently reported that the program struggled  to demonstrate required performance and reliability during operational  testing.
We then compared our assessment to the CIO ratings on the Dashboard,  discussed our findings with agency officials, and corroborated the  Dashboard’s data with agency officials. Our calculations are only intended  to provide a standardized view of risk across all the departments and  investments we reviewed, and this methodology is not intended to serve  as a prescriptive approach to the agencies’ evaluation of investment risk.
We conducted this performance audit from April 2015 to June 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Selected Agencies’ CIO Rating Processes

Table 14 describes the processes that the selected agencies reported as  using to create the Chief Information Officer (CIO) ratings for their  investments.

Appendix III: Agencies and Investments Selected for Review

Table 15 lists the selected agencies and investments, including those  which we exempted (as discussed in appendix I and shaded in gray), as  well as the associated Chief Information Officer (CIO) ratings and our  assessments.

Appendix IV: Comments from the Department of Commerce

Appendix V: Comments from the Department of Defense

Appendix VI: Comments from the Department of Education

Appendix VII: Comments from the Department of Energy

Appendix VIII: Comments from the Department of Health and Human Services

Appendix IX: Comments from the Department of Homeland Security

Appendix X: Comments from the Department of the Interior

Appendix XI: Comments from the Department of State

Appendix XII: Comments from the Department of Veterans Affairs

Appendix XIII: Comments from the Environmental Protection Agency

Appendix XIV: Comments from the Office of Personnel Management

Appendix XV: Comments from the Social Security Administration

Appendix XVI: GAO Contact and Staff Acknowledgments

	GAO Contact

	Staff Acknowledgments

In addition to the contact named above, individuals making contributions  to this report included Dave Hinchman (Assistant Director), Karl Seifert  (Assistant Director), Kevin Walsh (Assistant Director), Andrew Banister,  Rebecca Eyler, Sandra Kerr, and Meredith Raymond.

Summary:
Why GAO Did This Study

Although the government spends more than $80 billion in information technology (IT) annually, many of the investments have failed or have been troubled. In December 2014, provisions commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA) were enacted. Among other things, FITARA states that OMB shall make available to the public a list of each major IT investment including data on cost, schedule, and performance. OMB does so via the Federal IT Dashboard—its public website that reports on major IT investments, including ratings from CIOs which should reflect the level of risk facing an investment.
GAO's objectives were to (1) describe agencies' processes for determining CIO risk ratings for major federal IT investments primarily in development and (2) assess the risk of federal IT investments and analyze any differences with the investments' CIO risk ratings. To do so, GAO selected major IT investments with at least 80 percent of their fiscal year 2015 budget allocated to development (resulting in 95 investments across 15 agencies) and compared CIO rating processes to OMB guidance. GAO also analyzed data on those investments to create its own risk assessments.

What GAO Found

Agencies determined investments' Chief Information Officer (CIO) ratings using a variety of processes, which included the Office of Management and Budget's (OMB) six suggested factors (including risk management, requirements management, and historical performance). Specifically, all 17 selected agencies incorporated at least two of OMB's factors into their risk rating processes and 9 used all of the factors. However, agencies' interpretations of these factors varied. For example, most agencies considered active risks, such as funding cuts or staffing changes, when rating investments, but others only evaluated compliance with the agency's risk management processes. Further, 13 agencies required monthly updates to CIO ratings as does OMB (as of June 2015), 1 agency scheduled its reviews based on risk, and 3 agencies required updates less often than on a monthly basis.
GAO's assessments generally showed more risk than the associated CIO ratings. In particular, of the 95 investments assessed, GAO's assessments matched the CIO ratings 22 times, showed more risk 60 times, and showed less risk 13 times (see graphic).
Forty of the 95 CIO ratings were not updated during the month GAO reviewed, which led to more differences between GAO's assessments and the CIOs' ratings. This underscores the importance of frequent rating updates, which help to ensure that the information on the Dashboard is timely and accurately reflects recent changes to investment status.
Three agencies' rating processes span longer than 1 month. Longer processes mean that CIO ratings are based upon older data and may not reflect the current level of investment risk.
Seven agencies' rating processes did not focus on active risks. According to OMB's guidance, CIO ratings should reflect the CIO's assessment of the risk and the investment's ability to accomplish its goals. CIO ratings that do not incorporate active risks increase the chance that ratings overstate the likelihood of investment success.

What GAO Recommends

GAO is making 25 recommendations to 15 agencies to improve the quality and frequency of CIO ratings. Twelve agencies generally agreed with or did not comment on the recommendations and three agencies disagreed, stating their CIO ratings were adequate. GAO continues to believe these recommendations are valid.