Information Technology: OMB's Dashboard Has Increased Transparency and Oversight, but Improvements Needed
Published: 2010-07-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal IT spending has risen to an estimated $79 billion for fiscal year 2011. To improve transparency and oversight of this spending, in June 2009 the Office of Management and Budget (OMB) deployed a public website, known as the IT Dashboard, which provides information on federal agencies' major IT investments, including assessments of actual performance against cost and schedule targets (referred to as ratings). According to OMB, these data are intended to provide both a near real-time and historical perspective of the performance of these investments. GAO was asked to (1) examine the accuracy of the cost and schedule performance ratings on the Dashboard for selected investments and (2) determine whether the data on the Dashboard are used as a management tool to make improvements to IT investments. To do so, GAO selected 8 major investments from 5 agencies with large IT budgets, compared its analyses of the selected investments' performance to the ratings on the Dashboard, and interviewed agency officials about their use of the Dashboard to manage investments.
section_title: What GAO Found
paragraphs: The cost and schedule ratings on OMB's Dashboard were not always accurate for the selected investments. GAO found that 4 of the 8 selected investments had notable discrepancies on either their cost or schedule ratings. For example, the Dashboard indicated one investment had a less than 5 percent variance on cost every month from July 2009 through January 2010. GAO's analysis shows the investment's cost performance in December 2009 through January 2010 had a variance of 10 percent to less than 15 percent. Additionally, another investment on the Dashboard reported that it had been less than 30 days behind schedule since July 2009. However, investment data GAO examined showed that from September to December 2009 it was behind schedule greater than or equal to 30 days and less than 90 days. A primary reason for the data inaccuracies was that while the Dashboard was intended to represent near real-time performance information, the cost and schedule ratings did not take into consideration current performance. As a result, the ratings were based on outdated information. For example, cost ratings for each of the investments were based on data between 2 months and almost 2 years old. As of July 1, 2010, OMB plans to release an updated version of the Dashboard in July that includes ratings that factor in the performance of ongoing milestones. Another issue with the ratings was the wide variation in the number of milestones agencies reported, which was partly because OMB's guidance to agencies was too general. Having too many milestones can mask recent performance problems because the performance of every milestone (dated and recent) is equally averaged into the ratings. Specifically, investments that perform well during many previously completed milestones and then start performing poorly on a few recently completed milestones can maintain ratings that still reflect good performance. Conversely, having too few milestones limits the amount of information available to rate performance and allows agencies to potentially skew the ratings. OMB officials stated that they have recently chartered a working group with the intention of developing guidance for standardizing milestone reporting. However, until such guidance is available, the ratings may continue to have accuracy issues. Officials at three of the five agencies stated they were not using the Dashboard to manage their investments because they maintain they already had existing means to do so; officials at the other two agencies indicated that they were using the Dashboard to supplement their existing management processes. OMB officials indicated that they relied on the Dashboard as a management tool, including using the Dashboard's investment trend data to identify and address issues with investments' performance. According to OMB officials, the Dashboard was one of the key sources of information that they used to determine if an investment requires additional oversight. In addition, the Federal Chief Information Officer (CIO) stated that the Dashboard has greatly improved oversight capabilities compared to previously used mechanisms. He also stated that the Dashboard has increased the accountability of agencies' CIOs and established much needed visibility.
section_title: What GAO Recommends
paragraphs: GAO recommends that OMB report on its planned changes to the Dashboard to improve the accuracy of performance information and provide guidance to agencies that standardizes milestone reporting. OMB agreed with these recommendations, but disagreed with aspects of the draft report that GAO addressed, as appropriate.

section_title: Letter
paragraphs: Billions of taxpayer dollars are spent on information technology (IT)  investments each year; federal IT spending has now risen to an estimated  $79 billion for fiscal year 2011. During the past several years, we have  issued several reports and testimonies and made numerous  recommendations to the Office of Management and Budget (OMB) to  improve the transparency, oversight, and management of the federal  government’s IT investments. In June 2009, OMB deployed a public  website, known as the IT Dashboard, which provides detailed information  on federal agencies’ major IT investments, including assessments of actual  performance against cost and schedule targets (referred to as ratings) for  approximately 800 major federal IT investments. The Dashboard aims to  improve the transparency and oversight of these investments.
This report responds to your request that we (1) examine the accuracy of  the cost and schedule performance ratings on the Dashboard for selected  investments and (2) determine whether the data on the Dashboard are  used as a management tool to make improvements to IT investments.
To address our first objective, we selected five agencies—the Departments  of Agriculture (USDA), Defense (DOD), Energy (DOE), Health and Human  Services (HHS), and Justice (DOJ)—and ten investments to review. To  select these agencies and investments, we first identified ten agencies with  large IT budgets, and then identified the five largest investments at each of  the ten agencies. In narrowing the list to five agencies and ten total  investments, we considered several factors to ensure there were two  viable investments at each agency, such as selecting investments that were  not part of our ongoing audit work and providing a balance of investment  sizes. We then collected and analyzed monthly investment performance  reports from the ten investments. We compared our analyses of each  investment’s performance to the ratings on the Dashboard to determine if  the information was consistent. We also reviewed and analyzed OMB’s and  the selected agencies’ processes for populating and updating the  Dashboard. Additionally, we interviewed officials from OMB and the  agencies to obtain further information on their efforts to ensure the  accuracy of the data on the Dashboard. We did not test the adequacy of  the agency or contractor cost-accounting systems. Our evaluation of these  cost data was based on the documentation the agencies provided.
To address our second objective, we interviewed officials and analyzed  documentation at the selected agencies to determine the extent to which  they use the data on the Dashboard to make management decisions. We  also attended one of OMB’s TechStat sessions, which are reviews of  selected IT investments between OMB and agencies.
We conducted this performance audit from January to July 2010, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Further details of our objectives, scope, and  methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: Each year, OMB and federal agencies work together to determine how  much the government plans to spend on IT projects and how these funds  are to be allocated. Planned federal IT spending has now risen to an  estimated $79.4 billion for fiscal year 2011, a 1.2 percent increase from the  2010 level of $78.4 billion. OMB plays a key role in helping federal agencies  manage their investments by working with them to better plan, justify, and  determine how much they need to spend on projects and how to manage  approved projects.
To assist agencies in managing their investments, Congress enacted the  Clinger-Cohen Act of 1996, which requires OMB to establish processes to  analyze, track, and evaluate the risks and results of major capital  investments in information systems made by federal agencies and report  to Congress on the net program performance benefits achieved as a result  of these investments. Further, the act places responsibility for managing  investments with the heads of agencies and establishes chief information  officers (CIO) to advise and assist agency heads in carrying out this  responsibility. Another key law is the E-Government Act of 2002, which  requires OMB to report annually to Congress on the status of e- government. In these reports, referred to as the Implementation of the E- Government Act reports, OMB is to describe the Administration’s use of e- government principles to improve government performance and the  delivery of information and services to the public.
To help carry out its oversight role and assist the agencies in carrying out  their responsibilities as assigned by the Clinger-Cohen Act, OMB  developed a Management Watch List in 2003 and a High Risk List in 2005  to focus executive attention and to ensure better planning and tracking of  major IT investments. Consistent with the Clinger-Cohen Act, OMB  reported on the status of investments on the Management Watch List and  High Risk List in its annual budget documents.
Over the past several years, we have reported and testified on OMB’s  initiatives to highlight troubled projects, justify investments, and use  project management tools. We have made multiple recommendations to  OMB and federal agencies to improve these initiatives to further enhance  the oversight and transparency of federal projects. Among other things, we  recommended that OMB develop a central list of projects and their  deficiencies and analyze that list to develop governmentwide and agency  assessments of the progress and risks of the investments, identifying  opportunities for continued improvement. In addition, in 2006 we also  recommended that OMB develop a single aggregate list of high-risk  projects and their deficiencies and use that list to report to Congress on  progress made in correcting high-risk problems. As a result, OMB started  publicly releasing aggregate data on its Management Watch List and  disclosing the projects’ deficiencies. Furthermore, OMB issued  governmentwide and agency assessments of the projects on the  Management Watch List and identified risks and opportunities for  improvement, including risk management and security.
Table 1 provides a historical perspective of the number of projects on the  Management Watch List and their associated budgets for the period of  time during which OMB updated the Management Watch List. The table  shows that while the number of projects and their associated budgets on  the list generally decreased, the number of projects on the Management  Watch List increased by 239 projects and $13 billion for fiscal year 2009,  and represented a significant percentage of the total budget.
subsections: section_title: OMB’s Dashboard Publicizes Investment Details and Performance Status
paragraphs: More recently, to further improve the transparency into and oversight of  agencies’ IT investments, and to address data quality issues, in June 2009,  OMB publicly deployed a Web site, known as the IT Dashboard, which  replaced the Management Watch List and High Risk List. It displays  information on federal agencies’ cost, schedule, and performance  information for the approximately 800 major federal IT investments at 28  federal agencies. According to OMB, these data are intended to provide a  near real-time perspective of the performance of these investments, as  well as a historical perspective. Further, the public display of these data  are intended to allow OMB, other oversight bodies, and the general public  to hold the government agencies accountable for results and progress.
The Dashboard was initially deployed in June 2009 based on each agency’s  Exhibit 53 and Exhibit 300 submissions. After the initial population of  data, agency CIOs have been responsible for updating cost, schedule, and  performance fields on a monthly basis, which is a major improvement  from the quarterly reporting cycle OMB previously used for the  Management Watch List and High Risk List.
For each major investment, the Dashboard provides performance ratings  on cost and schedule, a CIO evaluation, and an overall rating which is  based on the cost, schedule, and CIO ratings. The cost rating is determined  by a formula that calculates the amount by which an investment’s  aggregated actual costs deviate from the aggregated planned costs. Table 2  displays the rating scale and associated category for cost variations.
An investment’s schedule rating is calculated by determining the average  days late or early. Table 3 displays the rating scale and associated category  for schedule deviations.
Each major investment on the Dashboard also includes a rating  determined by the agency CIO, which is based on his or her evaluation of  the performance of each investment. The rating is expected to take into  consideration the following criteria: risk management, requirements  management, contractor oversight, historical performance, and human  capital. This rating is to be updated when new information becomes  available that would impact the assessment of a given investment.
Lastly, the Dashboard calculates an overall rating for each major  investment. Figure 1 identifies the Dashboard’s overall ratings scale. This  overall rating is an average of the cost, schedule, and CIO ratings, with  each representing one-third of the overall rating. However, when the CIO’s  rating is lower than both the cost and schedule ratings, the CIO’s rating  will be the overall rating. Of the 792 major investments on the Dashboard  as of May 2010, 540 (68 percent) were green, 204 (26 percent) were yellow,  and 48 (6 percent) were red.
section_title: Earned Value Management Provides Additional Insight on Program Cost and Schedule
paragraphs: Earned value management is a technique that integrates the technical,  cost, and schedule parameters of a development contract and measures  progress against them. During the planning phase, a performance  measurement baseline is developed by assigning and scheduling budget  resources for defined work. As work is performed and measured against  the baseline, the corresponding budget value is “earned.” Using this earned  value metric, cost and schedule variances, as well as cost and time to  complete estimates, can be determined and analyzed.
Without knowing the planned cost of completed work and work in  progress (i.e., the earned value), it is difficult to determine a program’s  true status. Earned value allows for this key information, which provides  an objective view of program status and is necessary for understanding the  health of a program. As a result, earned value management can alert  program managers to potential problems sooner than using expenditures  alone, thereby reducing the chance and magnitude of cost overruns and  schedule slippages. Moreover, earned value management directly supports  the institutionalization of key processes for acquiring and developing  systems and the ability to effectively manage investments—areas that are  often found to be inadequate on the basis of our assessments of major IT  investments. In August 2005, OMB issued guidance that all major and high- risk development projects, among other things, develop comprehensive  policies to ensure that their major IT investments use earned value  management to manage their investments.
section_title: Performance Ratings on the Dashboard Were Not Always Accurate
paragraphs: Cost and schedule performance ratings were not always accurate for the  selected investments we reviewed. A key reason for the inaccuracies is  that the Dashboard’s cost and schedule ratings do not reflect current  performance. Another issue with the ratings is that large inconsistencies  exist in the number of milestones that agencies report on the Dashboard.
subsections: section_title: Cost and Schedule Performance Ratings Were Not Always Accurate
paragraphs: The cost and schedule performance ratings of selected investments were  not always accurate. There were several instances of inaccurate cost  ratings; however, two investments experienced notable discrepancies  while the other discrepancies were not as dramatic. Specifically, 5 of the 8  selected investments on the Dashboard had inaccurate cost ratings:  BioSense, Financial Management Modernization Initiative, Joint Precision  Approach and Landing System, Law Enforcement Wireless  Communication, and Unified Financial Management System. For example,  the Dashboard rated the Law Enforcement Wireless Communication  investment a 10 for cost (less than 5 percent variance) every month from  July 2009 through January 2010. However, our analysis shows the  investment’s cost rating during December 2009 and January 2010 is  equivalent to an 8 (a variance of 10 percent to less than 15 percent).  Accordingly, this investment’s cost performance should have been rated a  “yellow” instead of a “green,” meaning it needed attention. Further, the  Dashboard’s cost rating for the Financial Management Modernization  Initiative reported that this investment was “yellow,” while it should have  been “green” for 7 months. Maneuver Control System, Sequoia Platform,  and Risk Management Agency-13 are the three investments that had  accurate cost ratings. Figure 2 shows the comparison of selected  investments’ Dashboard cost ratings to GAO’s ratings for the months of  July 2009-January 2010.
There were fewer instances of discrepancies with the schedule ratings;  however, these discrepancies were also notable. Specifically, of the 8  selected investments, the Dashboard’s schedule ratings were inaccurate  for 2 investments: Risk Management Agency-13 and the Unified Financial  Management System. The Unified Financial Management System’s last  completed milestone was in May 2009 and the Dashboard rating for the  investment’s schedule has been a 10 since July 2009. However, investment  data we examined showed the schedule rating should have been a 5  (greater than or equal to 30 days and less than 90 days behind schedule)  from September 2009 through December 2009. As a result, this  investment’s schedule performance should have been rated a “yellow”  instead of a “green” for those months. Additionally, the Dashboard’s  schedule rating for Risk Management Agency-13 reported that this  investment was “red” for two months, while it should have been “green,”  and “yellow” for four months, when it should have been “green.” BioSense,  Financial Management Modernization Initiative, Joint Precision Approach  and Landing System, Law Enforcement Wireless Communication,  Maneuver Control System, and Sequoia Platform are the 6 investments  that had accurate schedule ratings. Figure 3 shows the comparison of  selected investments’ Dashboard schedule ratings to GAO’s ratings for the  months of July 2009-January 2010.
In addition to determining that cost and schedule ratings are not always  accurate, we found other data inaccuracies. Specifically, rebaseline  information on the Dashboard was not always accurate. Best practices  and GAO’s Cost Estimating Guide state that a rebaseline should occur  when the current cost and schedule baseline does not adequately  represent the amount of work to be completed, causing difficulty in  monitoring progress of the program. However, OMB reports all major  and minor corrections to planned information on the Dashboard, includ typographical fixes, as a rebaseline. More specifically, while the  Dashboard allows agencies to provide reasons for baseline changes, the  current version of the Dashboard, at a high level, identifies all changes to  planned information as rebaselines. For example, according to the  Dashboard, DOJ’s Law Enforcement Wireless Communication investment  has been rebaselined four times. However, program officials stated that  the program has only been rebaselined once. Similarly, the Dashboard  shows that the Sequoia Platform and Integrated Management Navigation  System investments at DOE have both been rebaselined four times.  However, program officials stated that neither of these programs had  actually been rebaselined. Rather, they stated that this number represents  instances in which they made minor corrections to the data on the  Dashboard. Table 4 shows the selected investments whose program  officials reported a lower number of rebaselines than what was reported  on the Dashboard.
section_title: Cost and Schedule Ratings Do Not Reflect Current Performance and Wide Variation in Milestone Reporting Exists
paragraphs: A primary reason why the cost and schedule ratings were not always  accurate is that the cost and schedule ratings do not take current  performance into consideration for many investments on the Dashboard,  though it is intended to represent near real-time performance information  on all major IT investments. Specifically, as of April 2010, the formula to  calculate the cost ratings on the Dashboard intentionally only factored in  completed portions of the investments (referred to as milestones) to  determine cost ratings. As such, milestones that are currently under way  are not taken into account. Table 5 identifies each selected investment’s  last completed milestone and the number of days that the Dashboard’s  cost rating is out of date for each selected investment.
OMB officials agreed that the ratings not factoring in current performance  is an area needing improvement and said that they are planning on  upgrading the Dashboard application in July 2010 to include updated cost  and schedule formulas that factor in the performance of ongoing  milestones; however, they have not yet made this change. One step OMB  has taken toward collecting the information needed for the new formulas  is that it now requires agencies to provide information on their investment  milestones’ planned and actual start dates. In addition, OMB officials  stated that they plan to use a previously unused data field—percent  complete. These are key data points necessary to calculate the  performance of ongoing milestones.
Another issue with the ratings is that there were wide variations in the  number of milestones agencies reported. For example, DOE’s Integrated  Management Navigation System investment lists 314 milestones, whereas  DOD’s Joint Precision Approach and Landing System investment lists 6.  Having too many milestones may mask recent performance problems  because the performance of every milestone (i.e., historical and recently  completed) is equally averaged into the ratings. Specifically, investments  that perform well during many previously completed milestones and then  start performing poorly on a few recently completed milestones can  maintain ratings that still reflect good performance. A more appropriate  approach could be to give additional weight to recently completed and  ongoing milestones when calculating the ratings. Too many detailed  milestones also defeat the purpose of an executive-level reporting tool.  Conversely, having too few milestones can limit the amount of information  available to track work and rate performance and allows agencies to  potentially skew the performance ratings.
In commenting on a draft of this report, the Federal CIO stated that OMB  has a new version of the Dashboard that implements updated cost and  schedule calculations. He stated that the new calculations greatly increase  the weight of current activities. As of July 1, 2010, this updated Dashboard  had not been released. An OMB analyst subsequently told us that the  agency plans to release the new version in July 2010. Additionally, OMB  officials have provided us with documentation of the new calculations and  demonstrated the new version of the Dashboard that will be released  soon. The Federal CIO also added that OMB will consider additional  changes to the ratings in the future.
Table 6 demonstrates the large inconsistencies in the number of  milestones reported for each selected investment.
In June 2009, OMB issued guidance that agencies are responsible for  providing quality data and, at minimum, should provide milestones that  consist of major segments of the investment, referred to as work  breakdown structure level 2, but prefers that agencies provide lower-level  milestones within each segment (work breakdown structure level 3). A  work breakdown structure is the cornerstone of every program because it  defines in detail the work necessary to accomplish a program’s objectives.  Standardizing a work breakdown structure is considered a best practice  because it enables an organization to collect and share data among  programs. Further, standardizing work breakdown structures allows data  to be shared across organizations.
However, certain agencies are not following OMB’s guidance and list  milestones that they consider to be at work breakdown structure level 1,  which are high-level milestones. Specifically, of the 5 agencies we  reviewed, officials at DOD, USDA, and DOE stated that they were  reporting work breakdown structure level 1 milestones to the Dashboard  for each of their selected investments. OMB officials acknowledge that not  all agencies are following their guidance, but stated that OMB analysts are  working with agencies to try to improve compliance. Furthermore, the  guidance that OMB has provided is not clear on the level of detail that it  wants agencies to report in their milestones and has left it to the agencies  to individually interpret their general guidance. Specifically, while OMB  states that agencies should report milestones that are, at a minimum, work  breakdown structure level 2, there is no commonly accepted definition  among federal agencies on the level of detail that should comprise each of  these levels. OMB officials acknowledged that they have not provided  clear guidance, but recently stated that they have begun exploring ways to  ensure more uniformity across agencies’ reporting. Specifically, in  commenting on a draft of this report, the Federal CIO stated that OMB has  recently chartered a working group comprised of representatives from  several federal agencies, with the intention of developing clear guidance  for standardizing and improving investment activity reporting.
OMB and agencies acknowledge that additional improvements can be  made beyond the cost and schedule ratings and have taken certain steps to  try to improve the accuracy of the data. For example, OMB implemented  an automated monthly data upload process and created a series of data  validation rules that detect common data entry errors, such as investment  milestone start dates that occur after completion dates. In addition, four of  the five agencies we reviewed indicated that they have processes in place  aimed at improving the accuracy of the data. For instance, HHS has  established a process wherein an official has been assigned responsibility  for ensuring the Dashboard is accurately updated. Further, DOJ has  developed an automated process to find missing data elements in the  information to be uploaded on the Dashboard.
Despite these efforts, until OMB upgrades the Dashboard application to  improve the accuracy of the cost and schedule ratings to include ongoing  milestones, explains the outcome of these improvements in its next annual  report to Congress on the Implementation of the E-Government Act  (which is a key mechanism for reporting on the implementation of the  Dashboard), provides clear and consistent guidance to agencies that  standardizes milestone reporting, and ensures agencies comply with the  new guidance, the Dashboard’s cost and schedule ratings will likely  continue to experience data accuracy issues.
section_title: Use of the Dashboard as a Management Tool Varies
paragraphs: Officials at three of the five agencies we reviewed—DOD, DOJ, and HHS— stated that they are not using the Dashboard to manage their investments,  and the other two agencies, DOE and USDA, indicated that they are using  the Dashboard to manage their investments.
Specifically, officials from the three agencies are not using the Dashboard  to manage their investments because they have other existing means to do  so:    DOD officials indicated that they use the department’s Capital Planning  and Investment Control process to track IT investment data—including  cost and schedule.
DOJ uses an internal dashboard that the office of the CIO developed that  provides for more detailed management of investments than OMB’s  Dashboard.
HHS officials said they use a portfolio investment management tool, which  they indicated provides greater insight into their investments.
Officials from the other two agencies—DOE and USDA— noted that they  are using the Dashboard as a management tool to supplement their  existing internal processes to manage their IT investments.
DOE officials stated that since their current process is based on a  quarterly review cycle, the monthly reporting nature of the Dashboard has  allowed officials to gain more frequent insight into investment  performance. As a result, DOE officials say that they are able to identify  potential issues before these issues present problems for investments.
USDA officials stated that they use the ratings on the Dashboard to  identify investments that appear to be problematic and hold meetings with  the investments’ program managers to discuss corrective actions.
Additionally, in OMB’s fiscal year 2009 Report to Congress on the  Implementation of the E-Government Act of 2002, 11 agencies reported on  how the Dashboard has increased their visibility and awareness of IT  investments. For example, the Department of Veterans’ Affairs terminated  12 IT projects, partly because of the increased visibility that the CIO  obtained from the Dashboard.
OMB indicated that it is using the Dashboard to manage IT investments.  Specifically, OMB analysts are using the Dashboard’s investment trend  data to track changes and identify issues with investments’ performance in  a timely manner and are also using the Dashboard to identify and drive  investment data quality issues. The Federal CIO stated that the Dashboard  has greatly improved oversight capabilities compared to previously used  mechanisms. He also stated that the Dashboard has increased the  accountability of agencies’ CIOs and established much-needed visibility.  According to OMB officials, the Dashboard is one of the key sources of  information that OMB analysts use to identify investments that are  experiencing performance problems and select them for a TechStat  session—a review of selected IT investments between OMB and agency  leadership that is led by the Federal CIO. OMB has identified factors that  may result in a TechStat session, such as policy interests, Dashboard data  inconsistencies, recurring patterns of problems, and/or an OMB analyst’s  concerns with an investment. As of June 2010, OMB officials indicated that  27 TechStat sessions have been held with federal agencies. According to  OMB, this program enables the government to improve or terminate IT  investments that are experiencing performance problems.
section_title: Conclusions
paragraphs: OMB has taken significant steps to enhance the oversight, transparency,  and accountability of federal IT investments by creating its IT Dashboard.  However, the cost and schedule ratings on the Dashboard were not always  accurate. Further, the rebaseline data were not always accurate. The cost  and schedule inaccuracies were due, in part, to calculations of ratings that  did not factor in current performance. Additionally, there were large  inconsistencies in the number of milestones that agencies report on the  Dashboard because OMB has not fully defined the level of detail that  federal agencies should use to populate the Dashboard and several  selected agencies decided not to follow OMB’s general guidance.  Moreover, the performance of historical and recently completed  milestones are equally averaged in the cost and schedule ratings, which is  counter to OMB’s goal to report near real-time performance on the  Dashboard. While the use of the Dashboard as a management tool varies,  OMB has efforts under way to include the performance of ongoing  milestones and its officials acknowledge that additional improvements are  needed. Nevertheless, until OMB explains in its next annual  Implementation of the E-Government Act report how the upgrade to the  Dashboard application has improved the accuracy of the cost and  schedule ratings, and provides clear and consistent guidance that enables  agencies to report standardized information on their milestones, the  accuracy of the data on the Dashboard may continue to be in question.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that the IT Dashboard provides meaningful ratings and  accurate investment data, we are recommending that the Director of OMB  take the following two actions:  include in its next annual Implementation of the E-Government Act report  the effect of planned formula changes on the accuracy of data; and   develop and issue clear guidance that standardizes milestone reporting on  the Dashboard.
In addition, we are recommending that the Secretaries of the Departments  of Agriculture, Defense, and Energy direct their Chief Information Officers  to ensure that they comply with OMB’s guidance on standardized  milestone reporting, once it is available.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Federal  CIO and DOE’s Associate CIO for IT Planning, Architecture, and E- Government. Letters from these agencies are reprinted in appendixes III  and IV. In addition, we received technical comments via e-mail from a  Coordinator at HHS, which we incorporated where appropriate. In  addition, the Deputy CIO from USDA, the Principal Director to the Deputy  Assistant Secretary of Defense for Resources from DOD, and an Audit  Liaison Specialist from DOJ indicated via e-mail that they had reviewed  the draft report and did not have any comments.
In OMB’s comments on our draft report, which contained four  recommendations to the OMB Director, the Federal CIO stated that he  agreed with two recommendations and disagreed with two because of  actions OMB has recently taken. After reviewing these actions, we agreed  that they addressed our concerns and will not make these two  recommendations.
OMB agreed with our recommendation that it include in its next annual  Implementation of the E-Government Act report how the planned formula  changes have improved the accuracy of data.
OMB agreed with our recommendation that it develop and issue clear  guidance that standardizes milestone reporting on the Dashboard.  Additionally, the Federal CIO asked that we update the report to reflect  that they have recently chartered a working group comprised of  representatives from several federal agencies, with the intention of  developing clear guidance for standardizing and improving investment  activity reporting. We have incorporated this additional information into  the report.
In response to our draft recommendation that OMB revise the IT  Dashboard and its guidance so that only major changes to investments are  considered to be rebaselines, OMB provided us with its new guidance on  managing IT baselines, which was issued on June 28, 2010. The guidance,  among other things, describes when agencies should report baseline  changes on the Dashboard. OMB also provided documentation of the  specific modifications that will be made in an upcoming release of the  Dashboard to improve the way baseline changes are displayed. We agree  that these recent changes address our recommendation. As such, we  updated the report to acknowledge and include this additional  information, where appropriate.
Regarding our recommendation that OMB consider weighing recently  completed and ongoing milestones more heavily than historical milestones  in the cost and schedule ratings, the Federal CIO stated that OMB has a  new version of the Dashboard that implements updated cost and schedule  calculations. He stated that the new calculations greatly increase the  weight of current activities. As previously stated, as of July 1, 2010, this  updated Dashboard had not been released. An OMB analyst subsequently  told us that the agency plans to release the new version in July 2010.  Additionally, OMB officials have provided us with documentation of the  new calculations and demonstrated the new version of the Dashboard that  will be released soon. The Federal CIO also added that OMB will consider  additional changes to the ratings in the future. We agree that these recent  changes address our recommendation. As such, we updated the report to  acknowledge and include this additional information, where appropriate.  Additionally, OMB will report on the effect of the upcoming changes to the  calculations in its next annual Implementation of the E-Government Act  report.
OMB also provided additional comments, which we address in appendix  III.
In DOE’s comments on our draft report, the Associate CIO for IT Planning,  Architecture, and E-Government indicated that she agreed with our  assessment of the implementation of the IT Dashboard across federal  agencies and with the recommendations presented to OMB. Additionally,  in response to our recommendation that the CIO of DOE comply with  OMB guidance on milestone reporting once it is available, the Associate  CIO stated that once OMB releases the additional guidance, DOE officials  will work to ensure the appropriate level of detail is reported on the  Dashboard. DOE also provided an additional comment, which we address  in appendix IV.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees; the Director of the Office of Management and  Budget; the Secretaries of the Departments of Agriculture, Defense,  Energy, Health and Human Services, and Justice; and other interested  parties. In addition, the report will be available at no charge on our Web  site at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) examine the accuracy of the cost and schedule  performance ratings on the Dashboard for selected investments and (2)  determine whether the data on the Dashboard are used as a management  tool to make improvements to IT investments.
To address both objectives, we selected five agencies and ten investments  to review. To select these agencies and investments, we first identified ten  agencies with large IT budgets as reported in the Office of Management  and Budget’s (OMB) fiscal year 2010 Exhibit 53. We then identified the five  largest investments at each of the ten agencies, according to the fiscal year  2010 budget, that were spending more than half of their budget on IT  development, modernization, and enhancement work, and were primarily  carried out by contractors. In narrowing the list to five agencies and ten  total investments, we considered several factors to ensure there were two  viable investments at each agency:    The investment is not part of our ongoing audit work related to cost,  schedule, and technical performance.
The investment is not part of a recent governmentwide earned value  management review.
The investment has not been highlighted as an investment needing  significant attention.
The collective list of investments creates a balance of investment sizes to  include both larger and smaller investments.
The five agencies are: the Departments of Agriculture (USDA), Defense  (DOD), Energy (DOE), Health and Human Services (HHS), and Justice  (DOJ).
The ten investments are: USDA’s Financial Management Modernization  Initiative and Risk Management Agency-13 Program; DOD’s Joint Precision  Approach and Landing System and Maneuver Control System; DOE’s  Integrated Management Navigation System and Sequoia Platform; HHS’s  BioSense Program and Electronic Research Administration System; DOJ’s  Law Enforcement Wireless Communication and Unified Financial  Management System (see appendix II for descriptions of each investment).
To address the first objective, we evaluated earned value data of the  selected investments to determine their cost and schedule performance  and compared it to the ratings on the Dashboard. The investment earned  value data was contained in contractor earned value management  performance reports obtained from the programs. To perform this  analysis, we compared the investment’s cumulative cost variance for each  month from July 2009 through January 2010 to the cost variance reported  on the Dashboard for those months. Similarly, we calculated the number  of months each investment was ahead or behind schedule over the same  period on the Dashboard. We also assessed 13 months of investment data  to analyze trends in cost and schedule performances. To further assess the  accuracy of the cost data, we compared it with other available supporting  program documents, including monthly and quarterly investment program  management reports; electronically tested the data to identify obvious  problems with completeness or accuracy; and interviewed agency and  program officials about the data and earned value management systems.  For the purposes of this report, we determined that the cost data at eight  of the investments were sufficiently reliable to use for our assessment. For  the two remaining investments, we determined that based on their  methods of earned value management, the data would not allow us to  sufficiently assess and rate monthly investment performance. We did not  test the adequacy of the agency or contractor cost-accounting systems.  Our evaluation of these cost data was based on the documentation the  agency provided.
We also reviewed and analyzed OMB’s and the selected agencies’  processes for populating and updating the Dashboard. Additionally, we  interviewed officials from OMB and the selected agencies and reviewed  OMB guidance to obtain additional information on OMB’s and agencies’  efforts to ensure the accuracy of the investment performance data and  cost and schedule performance ratings on the Dashboard. We used the  information provided by OMB and agency officials to identify the factors  contributing to inaccurate cost and schedule performance ratings on the  Dashboard. Moreover, we used this information to examine the accuracy  of the rebaseline information on the Dashboard, we interviewed agency  and program officials about the number of rebaselines each investment  has had, and compared these data with the rebaseline information listed  on the Dashboard.
To address our second objective, we analyzed related agency  documentation to assess what policies or procedures they have  implemented for using the data on the Dashboard to make management  decisions. We also interviewed agency and program officials regarding the  extent to which they use the data on the Dashboard as a management tool.  Additionally, we attended one of OMB’s TechStat sessions, which are  reviews of selected IT investments between OMB and agencies.
We conducted this performance audit from January to July 2010 at the  selected agencies’ offices in the Washington, D.C., metropolitan area. Our  work was done in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Selected Investment Descriptions
paragraphs: Below are descriptions of each of the selected investments that are  included in this review.
subsections: section_title: Department of Agriculture (USDA)
subsections: section_title: Financial Management Modernization Initiative
paragraphs: The Financial Management Modernization Initiative is USDA’s financial  management system modernization program. It is intended to be the  central financial system for USDA and is to consolidate the current  financial management system environment from 19 legacy systems into  one Web-based system.
section_title: Risk Management Agency-13
paragraphs: USDA’s Risk Management Agency-13 program is intended to support the  reengineering of all business systems associated with the crop insurance  program and provide a central financial system that will provide Web- based tools and applications for accessing Risk Management Agency data.
section_title: Department of Defense (DOD)
subsections: section_title: Joint Precision Approach and Landing System
paragraphs: DOD’s Joint Precision Approach and Landing System investment is  intended to provide a precision approach and landing capability for all  DOD ground and airborne systems. It is intended to enable U.S. forces to  safely land aircraft on any suitable surface worldwide (land and sea), with  ceiling and/or visibility the limiting factor.
section_title: Maneuver Control System
paragraphs: DOD’s maneuver control system investment is intended to provide, among  other things, the warfighter environment and collaborative and situational  awareness tools used to support executive decision making, planning,  rehearsal, and execution management. This system is to be used  throughout the Army to provide a common view of critical information.
section_title: Department of Energy (DOE)
subsections: section_title: Integrated Management Navigation System
paragraphs: DOE’s Integrated Management Navigation System consists of 5 major  projects and is intended to standardize and integrate accounting, data  warehouse, human resource, procurement, and budget processes  throughout DOE. The Integrated Management Navigation System  incorporates enterprisewide projects from DOE’s Office of the Chief  Financial Officer, Office of Human Capital Management, and Office of  Management.
section_title: Sequoia Platform
paragraphs: DOE’s Sequoia Platform is a supercomputer being developed for use by  three weapons laboratories—Los Alamos, Lawrence Livermore, and  Sandia National Laboratories—to contribute dramatically to the national  security enterprise. This supercomputer will also be used in maintaining  the nuclear deterrence and areas of nonproliferation, nuclear  counterterrorism, and support to the intelligence community.
section_title: Department of Health and Human Services (HHS)
paragraphs: HHS’s BioSense program is intended to improve the nation’s capabilities  for disease detection, monitoring, and near real-time health situational  awareness by creating a system that uses data from existing health-related  databases to identify patterns of disease symptoms prior to specific  diagnoses.
subsections: section_title: Electronic Research Administration
paragraphs: HHS’s Electronic Research Administration program is the National  Institutes of Health’s system for conducting interactive electronic  transactions for the receipt, review, monitoring, and administration of  grant awards to biomedical investigators worldwide. It is also intended to  provide the technology capabilities for the agency to efficiently and  effectively perform grants administration functions.
section_title: Department of Justice (DOJ)
subsections: section_title: Law Enforcement Wireless Communication
paragraphs: DOJ’s Law Enforcement Wireless Communication System, also known as  the Integrated Wireless Network, is to support the replacement and  modernization of failing radio systems and achieve communication  standards at DOJ’s law enforcement agencies. This program is intended to  provide all four law enforcement components with a shared unified radio  network, which should eliminate redundant coverage and duplicative  radio sites, while providing efficient and comparable coverage.
section_title: Unified Financial Management System
paragraphs: DOJ’s Unified Financial Management System is to improve the existing  and future financial management and procurement operations across DOJ.  Upon full implementation, the Unified Financial Management System will  replace five financial management systems and multiple procurement  systems with an integrated commercial off-the-shelf solution. This is to  streamline and standardize business processes and procedures across the  DOJ components.
Table 7 provides additional details for each of the selected investments in  our review.
section_title: Appendix III: Comments from the Office of Management and Budget
paragraphs: The following is GAO’s response to the Office of Management and  Budget’s (OMB) additional comments.
subsections: section_title: GAO Comments
paragraphs: 1.  We agree that the Dashboard has increased transparency,  accountability, and oversight; therefore, we updated the report to  discuss additional uses of the Dashboard, such as the use of trend  data, improved oversight capabilities, and enhancements to agencies’  investment management processes. We also updated the number of  Techstat sessions that have taken place.  2.  While additional data quality issues need to be addressed in the  Dashboard, we agree that the Dashboard is an improvement when  compared to OMB’s previous oversight tools such as the Management  Watch List and High Risk List. As such, we modified the report to  highlight these improvements. For example, we added to the report  that the Dashboard’s monthly reporting cycle is a significant  improvement in the quality of the data from the Management Watch  List and High Risk List, which were updated on a quarterly basis. 3.  As stated in the report, we found that the ratings were not always  accurate. We based this characterization on the fact that there were  several instances in which the ratings were inconsistent with the  performance indicated in our analysis of the investments’ earned value  management (EVM) reports and were notably different (e.g., ratings of  “green” versus “yellow”). We agree that EVM data generally only  covers the contracted development parts of the investments. As such,  as part of our methodology, we specifically selected investments  where the majority of each investment was focused on development  efforts (versus operational) and primarily carried out by contractors.  As such, we maintain that the comparison between the selected  investments’ Dashboard ratings and the performance indicated in their  EVM reports is a fair assessment. 4.  We acknowledge that the quality of EVM reports can vary. As such, we  took steps to ensure that the EVM reports we used were reliable  enough to evaluate the ratings on the Dashboard, and as OMB’s  comments indicate, we discounted two of the ten selected investments  after determining that their data was insufficient for our needs. We do  not state that OMB should base their ratings solely on EVM data. 5.  We agree that the original cost and schedule calculations are  performing as planned (i.e., are not defective) and we further clarified  this point in the report. We also note that planned changes to the rating  calculations will incorporate current performance. However, these  calculations, as originally planned and implemented, do not factor in  the performance of ongoing milestones, which we and OMB agree is an  area for improvement.  6.  We agree that the severity of the discrepancies were not always  dramatic. However, 4 of the 8 investments had notable discrepancies  on either their cost or schedule ratings. Specifically, as demonstrated  in the report, there were multiple instances in which the ratings were  discrepant enough to change the color of the ratings. The difference  between a “green” rating (i.e., normal performance) and a “yellow”  rating (i.e., needs attention) is the difference between whether an  investment is flagged for needing attention or not, which we believe is  an important point to highlight. 7.  We agree that agencies have a responsibility to provide quality  milestone data; however, we maintain that OMB’s existing guidance on  which milestones to report is too general for agencies to ensure they  are reporting consistently. OMB acknowledges that this is an area for  improvement and has established a working group to address this  issue. 8.  As previously discussed, on June 28, 2010, OMB issued its new  guidance on managing IT baselines. This guidance, among other things,  describes when agencies should report baselines changes to the  Dashboard. Officials also provided information on the upcoming  release of the Dashboard—which is intended to be released in July  2010—that will change the way baseline changes are displayed. We  agree that these recent changes address the issues we identified. 9.  We acknowledge that the Dashboard has made significant  improvements to oversight and transparency, in comparison to OMB’s  previous methods of overseeing IT investments, and we have added  additional information to the background of the report to highlight this  point.
section_title: Appendix IV: Comments from the Department of Energy
paragraphs: The following is GAO’s response to the Department of Energy’s (DOE)  additional comment.
subsections: section_title: GAO Comment
paragraphs: OMB’s guidance required agencies to provide data at one consistent work  breakdown structure level, rather than a mix of multiple levels. OMB and  others confirmed that agencies were able to transmit milestones at a single  consistent level. For this report, we observed agencies uploading at levels  1 through 4 and, thus, disagree that agencies were unable to transmit  milestones lower than level 1.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff also made key  contributions to this report: Shannin O’Neill, Assistant Director; Carol Cha;  Eric Costello; Rebecca Eyler; Emily Longcore; Bradley Roach; and Kevin  Walsh.