Information Technology: OMB Has Made Improvements to Its Dashboard, but Further Work Is Needed by Agencies and OMB to Ensure Data Accuracy
Published: 2011-03-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Each year the federal government spends billions of dollars on information technology (IT) investments. Given the importance of oversight, the Office of Management and Budget (OMB) established a public Web site, referred to as the IT Dashboard, that provides detailed information on about 800 federal IT investments, including assessments of actual performance against cost and schedule targets (referred to as ratings). In the second of a series of Dashboard reviews, GAO was asked to (1) determine OMB's efforts to improve the Dashboard and how it is using data from the Dashboard, and (2) examine the accuracy of the Dashboard's cost and schedule performance ratings. To do so, GAO analyzed documentation on OMB oversight efforts and Dashboard improvement plans, compared the performance of 10 major investments from five agencies with large IT budgets against the ratings on the Dashboard, and interviewed OMB and agency officials.
section_title: What GAO Found
paragraphs: Since GAO's first review, in July 2010, OMB has initiated several efforts to increase the Dashboard's value as an oversight tool, and has used the Dashboard's data to improve federal IT management. These efforts include streamlining key OMB investment reporting tools, eliminating manual monthly submissions, coordinating with agencies to improve data, and improving the Dashboard's user interface. Recent changes provide new views of historical data and rating changes. OMB anticipates that these efforts will increase the reliability of the data on the Dashboard. To improve IT management, OMB analysts use Dashboard data to track investment changes and identify issues with performance. OMB officials stated that they use these data to identify poorly performing IT investments for review sessions by OMB and agency leadership. OMB reported that these sessions and other management reviews have resulted in a $3 billion reduction in life-cycle costs, as of December 2010. While the efforts above as well as initial actions taken to address issues GAO identified in its prior review--such as OMB's updated ratings calculations to factor in ongoing milestones to better reflect current status--have contributed to data quality improvements, performance data inaccuracies remain. The ratings of selected IT investments on the Dashboard did not always accurately reflect current performance, which is counter to the Web site's purpose of reporting near real-time performance. Specifically, GAO found that cost ratings were inaccurate for six of the investments that GAO reviewed and schedule ratings were inaccurate for nine. For example, the Dashboard rating for a Department of Homeland Security investment reported significant cost variances for 3 months in 2010; however, GAO's analysis showed lesser variances from cost targets for the same months. Conversely, a Department of Transportation investment was reported as on schedule on the Dashboard, which does not reflect the significant delays GAO has identified in recent work. These inaccuracies can be attributed to weaknesses in how agencies report data to the Dashboard, such as providing erroneous data submissions, as well as limitations in how OMB calculates the ratings. Until the selected agencies and OMB resolve these issues, ratings will continue to often be inaccurate and may not reflect current program performance.
section_title: What GAO Recommends
paragraphs: GAO is recommending that selected agencies take steps to improve the accuracy and reliability of Dashboard information and OMB improve how it rates investments relative to current performance and schedule variance. Agencies generally concurred with the recommendations; OMB did not concur with the first recommendation but concurred with the second. GAO maintains that until OMB implements both, performance may continue to be inaccurately represented on the Dashboard.

section_title: Letter
paragraphs: Billions of taxpayer dollars are spent on information technology (IT)  investments each year; federal IT spending has now risen to an estimated  $79 billion for fiscal year 2011. During the past several years, we have  issued multiple reports and testimonies and made numerous  recommendations to the Office of Management and Budget (OMB) to  improve the transparency, oversight, and management of the federal  government’s IT investments. As part of its response to our prior work,  OMB deployed a public Web site in June 2009, known as the IT Dashboard,  which provides detailed information on federal agencies’ major IT  investments, including assessments of actual performance against cost  and schedule targets (referred to as ratings) for approximately 800 major  federal IT investments. The Dashboard aims to improve the transparency  and oversight of these investments.
In July 2010, we completed our first review of the Dashboard and reported  that the cost and schedule ratings on OMB’s Dashboard were not always  accurate because of limitations with OMB’s calculations. We  recommended that OMB report to Congress on the effect of its planned  Dashboard calculation changes on the accuracy of performance  information and provide guidance to agencies that standardizes milestone  reporting.
This is the second report in our series of Dashboard reviews and responds  to your request that we (1) determine what efforts OMB has under way to  improve the Dashboard and the ways in which it is using data from the  Dashboard to improve IT management and (2) examine the accuracy of  the cost and schedule performance ratings on the Dashboard for selected  investments.
To address our first objective, we interviewed OMB officials and analyzed  supporting OMB guidance and documentation to determine the efforts  OMB has under way to improve the Dashboard and the ways in which  OMB is using the data to improve IT management.
To address our second objective, we selected five agencies—the  Departments of Homeland Security (DHS), Transportation (DOT), the  Treasury (Treasury), and Veterans Affairs (VA), as well as the Social  Security Administration (SSA)—and 10 investments to review. The five  agencies account for 22 percent of the planned IT spending for fiscal year  2011. The 10 investments selected for case study represent about $1.27  billion in total planned spending in fiscal year 2011. We analyzed monthly  cost and schedule performance reports and program management  documents for the 10 investments to assess program performance against  planned cost and schedule targets. We then compared our analyses of  investment performance against the corresponding ratings on the  Dashboard to determine if the ratings were accurate. Additionally, we  interviewed officials from OMB and the agencies to obtain further  information on their efforts to ensure the accuracy of the data used to rate  investment performance on the Dashboard. We did not test the adequacy  of the agency or contractor cost-accounting systems. Our evaluation of  these cost data was based on the documentation the agencies provided.
We conducted this performance audit from July 2010 to March 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Further details of our objectives, scope, and  methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: Each year, OMB and federal agencies work together to determine how  much the government plans to spend on IT investments and how these  funds are to be allocated. According to the President’s Budget for Fiscal  Year 2011, the total planned spending on IT in fiscal year 2011 is an  estimated $79.4 billion, a 1.2 percent increase from the fiscal year 2010  budget level of $78.4 billion. OMB plays a key role in helping federal  agencies manage their investments by working with them to better plan,  justify, and determine how much they need to spend on projects and how  to manage approved projects.
To assist agencies in managing their investments, Congress enacted the  Clinger-Cohen Act of 1996, which requires OMB to establish processes to  analyze, track, and evaluate the risks and results of major capital  investments in information systems made by federal agencies and report  to Congress on the net program performance benefits achieved as a result  of these investments. Further, the act places responsibility for managing  investments with the heads of agencies and establishes chief information  officers (CIO) to advise and assist agency heads in carrying out this  responsibility. Another key law is the E-Government Act of 2002, which  requires OMB to report annually to Congress on the status of e- government. In these reports, referred to as Implementation of the E- Government Act reports, OMB is to describe the administration’s use of e- government principles to improve government performance and the  delivery of information and services to the public.
To help carry out its oversight role, in 2003, OMB established the  Management Watch List, which included mission-critical projects that  needed to improve performance measures, project management, IT  security, or overall justification for inclusion in the federal budget.  Further, in August 2005, OMB established a High-Risk List, which  consisted of projects identified by federal agencies, with the assistance of  OMB, as requiring special attention from oversight authorities and the  highest levels of agency management.
Over the past several years, we have reported and testified on OMB’s  initiatives to highlight troubled IT projects, justify investments, and use  project management tools. We have made multiple recommendations to  OMB and federal agencies to improve these initiatives to further enhance  the oversight and transparency of federal projects. Among other things, we  recommended that OMB develop a central list of projects and their  deficiencies and analyze that list to develop governmentwide and agency  assessments of the progress and risks of the investments, identifying  opportunities for continued improvement. In addition, in 2006 we also  recommended that OMB develop a single aggregate list of high-risk  projects and their deficiencies and use that list to report to Congress on  progress made in correcting high-risk problems. As a result, OMB started  publicly releasing aggregate data on its Management Watch List and  disclosing the projects’ deficiencies. Furthermore, OMB issued  governmentwide and agency assessments of the projects on the  Management Watch List and identified risks and opportunities for  improvement, including risk management and security.
subsections: section_title: OMB’s Dashboard Publicizes Investment Details and Performance Status
paragraphs: More recently, to further improve the transparency and oversight of  agencies’ IT investments, and to address data quality issues, in June 2009,  OMB publicly deployed a Web site, known as the IT Dashboard, which  replaced the Management Watch List and High-Risk List. It displays federal  agencies’ cost, schedule, and performance data for the approximately 800  major federal IT investments at 27 federal agencies. According to OMB,  these data are intended to provide a near real-time perspective on the  performance of these investments, as well as a historical perspective.  Further, the public display of these data is intended to allow OMB; other  oversight bodies, including Congress; and the general public to hold the  government agencies accountable for results and progress.
The Dashboard was initially deployed in June 2009 based on each agency’s  Exhibit 53 and Exhibit 300 submissions. After the initial population of  data, agency CIOs have been responsible for updating cost, schedule, and  performance fields on a monthly basis, which is a major improvement  from the quarterly reporting cycle OMB previously used for the  Management Watch List and High-Risk List.
For each major investment, the Dashboard provides performance ratings  on cost and schedule, a CIO evaluation, and an overall rating, which is  based on the cost, schedule, and CIO ratings. As of July 2010, the cost  rating was determined by a formula that calculates the amount by which  an investment’s total actual costs deviate from the total planned costs.  Similarly, the schedule rating is the variance between the investment’s  planned and actual progress to date. Figure 1 displays the rating scale and  associated categories for cost and schedule variations.
Each major investment on the Dashboard also includes a rating  determined by the agency CIO, which is based on his or her evaluation of  the performance of each investment. The rating is expected to take into  consideration the following criteria: risk management, requirements  management, contractor oversight, historical performance, and human  capital. This rating is to be updated when new information becomes  available that would affect the assessment of a given investment.
Last, the Dashboard calculates an overall rating for each major investment.  This overall rating is an average of the cost, schedule, and CIO ratings,  with each representing one-third of the overall rating. However, when the  CIO’s rating is lower than both the cost and schedule ratings, the CIO’s  rating will be the overall rating. Figure 2 shows the overall performance  ratings of the 805 major investments on the Dashboard as of March 2011.
section_title: Earned Value Management Provides Additional Insight on Program Cost and Schedule
paragraphs: To better manage IT investments, OMB issued guidance directing agencies  to develop comprehensive policies to ensure that their major IT  investments and high-risk development projects use earned value  management to manage their investments. Earned value management is a  technique that integrates the technical, cost, and schedule parameters of a  development contract and measures progress against them. During the  planning phase, a performance measurement baseline is developed by  assigning and scheduling budget resources for defined work. As work is  performed and measured against the baseline, the corresponding bud value is “earned.” Using this earned value metric, cost and schedule  variances, as well as cost a nd time to complete estimates, can be  determined and analyzed.
Without knowing the planned cost of completed work and work in  progress (i.e., the earned value), it is difficult to determine a program’s  true status. Earned value allows for this key information, which provides  an objective view of program status and is necessary for understandin health of a program. As a result, earned value management can alert  program managers to potential problems sooner than using expenditures  alone, thereby reducing the chance and magnitude of cost overruns and  schedule slippages. Moreover, earned value management directly supp the institutionalization of key processes for acquiring and developing  systems and the ability to effectively manage investments—areas that are  often found to be inadequate on the basis of our assessments of major IT  investments.
section_title: OMB Has Taken Step Address Prior GAO Recommendations on Dashboard Improving Accuracy
paragraphs: In July 2010, we reported that the cost and schedule ratings on OMB’s  Dashboard were not always accurate for selected agencies. Specifically, we found that several selected investments had notable discrepancies in  their cost or schedule ratings, the cost and schedule ratings did not takeinto consideration current performance, and the number of milestones  (activities) reported by agencies varied widely. We made a number of  recommendations to OMB to better ensure that the Dashboard prov meaningful ratings and accurate investment data. In particular, we  recommended that OMB report on its planned Dashboard changes to  improve the accuracy of performance information and provide guidance agencies that standardizes activity reporting. OMB agreed with the two recommendations and reported it had initiated work to address them.
section_title: OMB Has Multiple Efforts Under Way to Further Refine the Dashboard and Uses the Dashboard to Improve IT Management
paragraphs: Since our last report, OMB has initiated multiple efforts to increase the  Dashboard’s value as a management and oversight tool, and has used data  in the Dashboard to improve the management of federal IT investments.  Specifically, OMB is focusing its efforts in four main areas: streamlining  key OMB investment reporting tools, eliminating manual monthly  submissions, coordinating with agencies to improve data, and improving  the user interface.
OMB’s plan to reform federal IT management commits OMB to  streamlining two of the Dashboard’s sources of information—specifically,  the OMB Exhibits 53 and 300. OMB has committed, by May 2011, to  reconstruct the exhibits around distinct data elements that drive value for  agencies and provide the information necessary for meaningful oversight.  OMB anticipates that these changes will also alleviate the reporting burden  and increase data accuracy, and that the revised exhibits will serve as its  authoritative management tools.
According to OMB officials, the Dashboard no longer accepts manual data  submissions. Instead, the Dashboard allows only system-to-system  submissions. Officials explained that this update allows the Dashboard to  reject incomplete submissions and those that do not meet the Dashboard’s  data validation rules. By eliminating direct manual submissions, this effort  is expected to improve the reliability of the data shown on the Dashboard.
Further, OMB officials stated that they work to improve the Dashboard  through routine interactions with agencies and IT portfolio management  tool vendors, training courses, working groups, and data quality letters to  agencies. Specifically, OMB officials stated that they held 58 TechStat  reviews (discussed later in this report), hosted four online training  sessions (recordings of which OMB officials stated are also available  online), collaborated with several Dashboard working groups, and sent  letters to agency CIOs identifying specific data quality issues on the  Dashboard that their agencies could improve. Further, OMB officials  explained that in December 2010, OMB analysts informed agency CIOs  about specific data quality issues and provided analyses of agency data, a  comparison of agency Dashboard performance with that of the rest of the  government, and expected remedial actions. OMB anticipates these efforts  will increase the Dashboard’s data reliability by ensuring that the agencies  are aware of and are working to address issues.
Finally, OMB continues to improve the Dashboard’s user interface. For  instance, in November 2010, OMB updated the Dashboard to provide new  views of historical data and rating changes and provide new functionality  allowing agencies to make corrections to activities and performance  metrics (conforming to rebaselining guidance). Officials also described a  planned future update, which is intended to contain updated budget data,  display corrections and changes made to activities, and reflect increased  validation of agency-submitted data. OMB anticipates these efforts will  increase the transparency and reliability of investment information on the  Dashboard by providing agencies and users additional ways to view  investment information and by improving validation of submitted data.
Additionally, OMB uses the Dashboard to improve the management of IT  investments. Specifically, OMB analysts are using the Dashboard’s  investment trend data to track changes and identify issues with  investments’ performance in a timely manner. OMB analysts also use the  Dashboard to identify data quality issues and drive improvements to the  data. The Federal CIO stated that the Dashboard has greatly improved  oversight capabilities compared with those of previously used  mechanisms, such as the annual capital asset plan and business case  (Exhibit 300) process. Additionally, according to OMB officials, the  Dashboard is one of the key sources of information that OMB analysts use  to identify IT investments that are experiencing performance problems  and to select them for a TechStat session—a review of selected IT  investments between OMB and agency leadership that is led by the  Federal CIO. As of December 2010, OMB officials stated that 58 TechStat  sessions have been held with federal agencies. According to OMB, these  sessions have enabled the government to improve or terminate IT  investments that are experiencing performance problems. Information  from the TechStat sessions and the Dashboard was used by OMB to  identify, halt, and review all federal financial IT systems modernization  projects. Furthermore, according to OMB, these sessions and other OMB  management reviews have resulted in a $3 billion reduction in life-cycle  costs, as of December 2010. OMB officials stated that, as of December  2010, 11 investments have been reduced in scope and 4 have been  terminated as a result of these sessions. For example,    The TechStat on the Department of Housing and Urban Development’s  Transformation Initiative investment found that the department lacked the  skills and resources necessary and would not be positioned to succeed. As  a result, the department agreed to reduce the number of projects from 29  to 7 and to limit fiscal year 2010 funds for these 7 priority projects to $85.7  million (from the original $138 million).
The TechStat on the National Archives and Records Administration’s  Electronic Records Archives investment resulted in six corrective actions,  including halting fiscal year 2012 development funding pending the  completion of a strategic plan.
According to OMB officials, OMB and agency CIOs also used the  Dashboard data and TechStat sessions, in addition to other forms of  research (such as reviewing program documentation, news articles, and  inspector general reports), to identify 26 high-risk IT projects and, in turn,  coordinate with agencies to develop corrective actions for these projects  at TechStat sessions. For example, the Department of the Interior is to  establish incremental deliverables for its Incident Management Analysis  and Reporting System, which will accelerate delivery of services that will  help 6,000 law enforcement officers protect the nation’s natural resources  and cultural monuments.
section_title: Dashboard Ratings Did Not Always Reflect True Investment Cost and Schedule Performance
paragraphs: While the efforts previously described are important steps to improving  the quality of the information on the Dashboard, cost and schedule  performance data inaccuracies remain. The Dashboard’s cost and  schedule ratings were not always reflective of the true performance for  selected investments from the five agencies in our review. More  specifically, while the Dashboard is intended to present near real-time  performance, the ratings did not always reflect the current performance of  these investments. Dashboard rating inaccuracies were the result of  weaknesses in agency practices, such as the Dashboard not reflecting  baseline changes and the reporting of erroneous data, as well as  limitations of the Dashboard’s calculations. Until the agencies submit  complete, reliable, and timely data to the Dashboard and OMB revises its  Dashboard calculations, performance ratings will continue to be  inaccurate and may not reflect current program performance.
subsections: section_title: Cost and Schedule Performance Was Not Always Accurately Depicted in Dashboard Ratings
paragraphs: Most of the Dashboard’s cost ratings of the nine selected investments did  not match the results of our analyses over a 3-month period. Specifically,  four investments had inaccurate ratings for 2 or more months, and two  were inaccurate for 1 month, while three investments were accurately  depicted for all 3 months. For example, Intelligent Disability’s cost  performance was rated “red” on the Dashboard for July 2010 and “green”  for August 2010, whereas our analysis showed its current cost  performance was “yellow” for those months. Further, Medical Legacy’s  cost ratings were “red” on the Dashboard for June through August 2010,  while the department’s internal rating showed that the cost performance  for 105 of the 107 projects that constitute the investment was “green” in  August 2010; similar ratings were also seen for June and July 2010. Overall,  the Dashboard’s cost ratings generally showed poorer performance than  our assessments. Figure 3 shows the comparison of the selected  investments’ Dashboard cost ratings with GAO’s ratings based on analysis  of agency data for the months of June 2010 through August 2010.
Regarding schedule, most of the Dashboard’s ratings of the nine selected  investments did not match the results of our analyses over a 3-month  period. Specifically, seven investments had inaccurate ratings for 2 or  more months, and two were inaccurate for 1 month. For example,  Automatic Dependent Surveillance-Broadcast’s schedule performance was  rated “green” on the Dashboard in July 2010, but our analysis showed its  current performance was “yellow” that month. Additionally, the “green”  schedule ratings for En Route Automation Modernization did not  represent how this program is actually performing. Specifically, we  recently reported that the program is experiencing significant schedule  delays, and the CIO evaluation of the program on the Dashboard has  indicated schedule delays since February 2010. As with the cost ratings,  the Dashboard’s schedule ratings generally showed poorer performance  than our assessments. Figure 4 shows the comparison of the selected  investments’ Dashboard schedule ratings with GAO’s ratings based on  analysis of agency data for the months of June 2010 through August 2010.
section_title: Dashboard Rating Inaccuracies Are a Result of Weaknesses in Agencies’ Practices and Limitations with OMB’s Calculations
paragraphs: OMB guidance, as of June 2010, states that agencies are responsible for  maintaining consistency between the data in their internal systems and the  data on the Dashboard. Furthermore, the guidance states that agency  CIOs should update their evaluation on the Dashboard as soon as new  information becomes available that affects the assessment of a given  investment. According to our assessment of the nine selected investments,  agencies did not always follow this guidance. In particular, there were four  primary weaknesses in agency practices that resulted in inaccurate cost  and schedule ratings on the Dashboard: the investment baseline on the  Dashboard was not reflective of the investment’s actual baseline, agencies  did not report data to the Dashboard, agencies reported erroneous data,  and unreliable earned value data were reported to the Dashboard. In  addition, two limitations of OMB’s Dashboard calculations contributed to  ratings inaccuracies: a lack of emphasis on current performance and an  understatement of schedule variance. Table 1 shows the causes of  inaccurate ratings for the selected investments.
Inconsistent program baseline: Three of the selected investments  reported baselines on the Dashboard that did not match the actual  baselines tracked by the agencies. Agency officials responsible for each of  these investments acknowledged this issue. For example, according to  Modernized e-File officials, the investment was in the process of a  rebaseline in June 2010; thus, officials were unable to update the baseline  on the Dashboard until July 2010. For another investment—HealtheVet  Core—officials stated that it was stopped in August, and thus the  HealtheVet Core baseline on the Dashboard is incorrect. As such, the CIO  investment evaluation should have been updated to reflect that the  investment was stopped. In June 2010, OMB issued new guidance on  rebaselining, which stated that agencies should update investment  baselines on the Dashboard within 30 days of internal approval of a  baseline change and that this update will be considered notification to  OMB. However, agencies still must go through their internal processes to  approve a new baseline, and during this process the baseline on the  Dashboard will be inaccurate. As such, investment CIO ratings should  disclose that performance data on the Dashboard are unreliable because  of baseline changes. However, the CIO evaluation ratings for these  investments did not include such information. Without proper disclosure  of pending baseline changes and resulting data reliability weaknesses,  OMB and other external oversight groups will not have the appropriate  information to make informed decisions about these investments.
Missing data submissions: Three investments did not upload complete  and timely data submissions to the Dashboard. For example, DHS officials  did not submit data to the Dashboard for the C4ISR investment from June  through August 2010. According to DHS officials, C4ISR investment  officials did not provide data for DHS to upload for these months. Further  compounding the performance rating issues of this investment is that in  March 2010, inaccurate data were submitted for nine of its activities; these  data were not corrected until September 2010. Until officials submit  complete, accurate, and timely data to the Dashboard, performance ratings  may continue to be inaccurate.
Erroneous data submissions: Seven investments reported erroneous data  to the Dashboard. For example, SSA submitted start dates for Intelligent  Disability and Disability Case Processing System activities that had not  actually started yet. SSA officials stated that, because of SSA’s internal  processes, their start dates always correspond to the beginning of the  fiscal year. In addition, according to a Treasury official, Internal Revenue  Service officials for the Modernized e-File investment provided inaccurate  data for the investment’s “actual percent complete” fields for some  activities. Until officials submit accurate data to the Dashboard,  performance ratings may continue to be inaccurate.
Unreliable source data: Treasury’s Payment Application Modernization  investment used unreliable earned value data as the sole source of data on  the Dashboard. As such, this raises questions about the accuracy of the  performance ratings reported on the Dashboard. Investment officials  stated that they have taken steps to address weaknesses with the earned  value management system and are currently evaluating other adjustments  to investment management processes. However, without proper disclosure  about data reliability in the CIO assessment, OMB and other external  oversight groups will not have the appropriate information to make  informed decisions about this investment.
Additionally, two limitations in the Dashboard method for calculating  ratings contributed to inaccuracies:    Current performance calculation: The Dashboard is intended to represent  near real-time performance information on all major IT investments, as  previously discussed. To OMB’s credit, in July 2010, it updated the  Dashboard’s cost and schedule calculations to include both ongoing and  completed activities in order to accomplish this. However, the  performance of ongoing activities is combined with the performance of  completed activities, which can mask recent performance. As such, the  cost and schedule performance ratings on the Dashboard may not always  reflect current performance. Until OMB updates the Dashboard’s cost and  schedule calculations to focus on current performance, the performance  ratings may not reflect performance problems that the investments are  presently facing, and OMB and agencies are thus missing an opportunity to  identify solutions to such problems.
Schedule variance calculation: Another contributing factor to certain  schedule inaccuracies is that OMB’s schedule calculation for in-progress  activities understates the schedule variance for activities that are overdue.  Specifically, OMB’s schedule calculation does not recognize the full  variance of an overdue activity until it has actually completed. For  example, as of September 13, 2010, the Dashboard reported a 21-day  schedule variance for an En Route Automation Modernization activity that  was actually 256 days overdue. Until OMB updates its in-progress schedule  calculation to be more reflective of the actual schedule variance of  ongoing activities, schedule ratings for these activities may be  understated.
section_title: Conclusions
paragraphs: The Dashboard has enhanced OMB’s and agency CIOs’ oversight of federal  IT investments. Among other things, performance data from the  Dashboard are being used to identify poorly performing investments for  executive leadership review sessions. Since the establishment of the  Dashboard, OMB has worked to continuously refine it, with multiple  planned improvement efforts under way for improving the data quality and  Dashboard usability.
However, the quality of the agency data reported to the Dashboard  continues to be a challenge. Specifically, the cost and schedule ratings on  the Dashboard were not always accurate in depicting current program  performance for most of the selected investments, which is counter to  OMB’s goal to report near real-time performance. The Dashboard rating  inaccuracies were due, in part, to weaknesses in agencies’ practices and  limitations in OMB’s calculations. More specifically, the agency  practices—including the inconsistency between Dashboard and program  baselines, reporting of erroneous data, and unreliable source data—and  OMB’s formulas to track current performance have collectively impaired  data quality. Until agencies provide more reliable data and OMB improves  the calculations of the ratings on the Dashboard, the accuracy of the  ratings will continue to be in question and the ratings may not reflect  current program performance.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that the Dashboard provides accurate cost and schedule  performance ratings, we are making eleven recommendations to the heads  of each of the five selected agencies. Specifically, we are recommending  that:    The Secretary of the Department of Homeland Security direct the CIO to    ensure that investment data submissions include complete and  accurate investment information for all required fields;    comply with OMB’s guidance on updating the CIO rating as soon as  new information becomes available that affects the assessment of a  given investment, including when an investment is in the process of a  rebaseline; and    work with C4ISR officials to comply with OMB’s guidance on updating  investment cost and schedule data on the Dashboard at least monthly.
The Secretary of the Department of Transportation direct the CIO to work  with Automatic Dependent Surveillance-Broadcast officials to comply with  OMB’s guidance on updating investment cost and schedule data on the  Dashboard at least monthly.
The Secretary of the Department of the Treasury direct the CIO to    comply with OMB’s guidance on updating the CIO rating as soon as  new information becomes available that affects the assessment of a  given investment, including when an investment is in the process of a  rebaseline;    work with Modernized e-File officials to report accurate actual percent  complete data for each of the investment’s activities; and    work with Payment Application Modernization officials to disclose the  extent of this investment’s data reliability issues in the CIO rating  assessment on the Dashboard.
The Secretary of the Department of Veterans Affairs direct the CIO to    comply with OMB’s guidance on updating the CIO rating as soon as  new information becomes available that affects the assessment of a  given investment, including when an investment is in the process of a  rebaseline;    work with Medical Legacy officials to comply with OMB’s guidance on  updating investment cost and schedule data on the Dashboard at least  monthly; and    ensure Medical Legacy investment data submitted to the Dashboard are  consistent with the investment’s internal performance information.
The Commissioner of the Social Security Administration direct the CIO to  ensure that data submissions to the Dashboard include accurate  investment information for all required fields.
In addition, to better ensure that the Dashboard provides meaningful  ratings and reliable investment data, we are recommending that the  Director of OMB direct the Federal CIO to take the following two actions:    develop cost and schedule rating calculations that better reflect current    update the Dashboard’s schedule calculation for in-progress activities to  more accurately represent the variance of ongoing, overdue activities.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our report to the five agencies in our review and to  OMB. In commenting on the draft, four agencies generally concurred with  our recommendations. One agency, the Department of Transportation,  agreed to consider our recommendation. OMB agreed with one of our  recommendations and disagreed with the other. In addition, OMB raised  concerns about the methodology used in our report. Agencies also  provided technical comments, which we incorporated as appropriate.  Each agency’s comments are discussed in more detail below.
In e-mail comments on a draft of the report, DHS’s Departmental Audit  Liaison stated that the department concurred with our recommendations.
In e-mail comments, DOT’s Director of Audit Relations stated that DOT  would consider our recommendation; however, he also stated that the  department disagreed with the way its investments were portrayed in the  draft. Specifically, department officials stated that our assessment was not  reasonable because our methodology only incorporated the most recent 6  months of performance rather than using cumulative investment  performance. As discussed in this report, combining the performance of  ongoing and completed activities can mask recent performance. As such,  we maintain that our methodology is a reasonable means of deriving near  real-time performance, which the Dashboard is intended to represent.
In oral comments, Treasury’s Chief Architect stated that the department  generally concurred with our recommendations and added that the  department would work to update its Dashboard ratings for the two  selected investments.
In written comments, VA’s Chief of Staff stated that the department  generally concurred with our recommendations and agreed with our  conclusions. Further, he outlined the department’s planned process  improvements to address the weaknesses identified in this report. VA’s  comments are reprinted in appendix III.
In written comments, SSA’s Deputy Chief of Staff stated that the  Administration agreed with our recommendation and had taken corrective  actions intended to prevent future data quality errors. SSA’s comments are  reprinted in appendix IV.
Officials from OMB’s Office of E-Government & Information Technology  provided the following oral comments on the draft:    OMB officials agreed with our recommendation to update the Dashboard’s  schedule calculation for in-progress activities to more accurately represent  the variance of ongoing, overdue activities. These officials stated that the  agency has long-term plans to update the Dashboard’s calculations, which  they believe will provide a solution to the concern identified in this report.
OMB officials disagreed with our recommendation to develop cost and  schedule rating calculations that better reflect current investment  performance. According to OMB, real-time performance is always  reflected in the ratings since current investment performance data are  uploaded to the Dashboard on a monthly basis.
Regarding OMB’s comments, our point is not that performance data on the  Dashboard are infrequently updated, but that the use of historical data  going back to an investment’s inception can mask more recent  performance. For this reason, current investment performance may not  always be as apparent as it should be, as this report has shown. Until the  agency places less emphasis on the historical data factored into the  Dashboard’s calculations, it will be passing up an opportunity to more  efficiently and effectively identify and oversee investments that either  currently are or soon will be experiencing problems.
OMB officials also described the agency’s plans for enhancing Dashboard  data quality and performance calculations. According to OMB, plans were  developed in February 2011 with stakeholders from other agencies to  standardize the reporting structure for investment activities. Further, OMB  officials said that their plans also call for the Dashboard’s performance  calculations to be updated to more accurately reflect activities that are  delayed. In doing so, OMB stated that agencies will be expected to report  new data elements associated with investment activities. Additionally,  OMB officials noted that new agency requirements associated with these  changes will be included in key OMB guidance (Circular A-11) no later  than September 2011.
OMB officials also raised two concerns regarding our methodology.  Specifically,    OMB stated that our reliance on earned value data as the primary source  for determining investment performance was questionable. These officials  stated that, on the basis of their experience collecting earned value data,  the availability and quality of these data vary significantly across agencies.  As such, according to these officials, OMB developed its Dashboard cost  and schedule calculations to avoid relying on earned value data.
We acknowledge that the quality of earned value data can vary. As such,  we took steps to ensure that the data we used were reliable enough to  evaluate the ratings on the Dashboard, and discounted the earned value  data of one of the selected investments after determining its data were  insufficient for our needs. While we are not critical of OMB’s decision to  develop its own method for calculating performance ratings, we maintain  that our use of earned value data is sound. Furthermore, earned value data  were not the only source for our analysis; we also based our findings on  other program management documentation, such as inspector general  reports and internal performance management system performance  ratings, as discussed in appendix I.
OMB also noted that, because we used earned value data to determine  investment performance, our ratings were not comparable to the ratings  on the Dashboard. Specifically, OMB officials said that the Dashboard  requires reporting of all activities under an investment, including  government resources or operations and maintenance activities. OMB  further said that this is more comprehensive than earned value data, which  only account for contractor-led development activities.
We acknowledge and support the Dashboard’s requirement for a  comprehensive accounting of investment performance. Further, we agree  that earned value data generally only cover development work associated  with the investments (thus excluding other types of work, such as  planning and operations and maintenance). For this reason, as part of our  methodology, we specifically selected investments for which the majority  of the work being performed was development work. We did this because  earned value management is a proven technique for providing objective  quantitative data on program performance, and alternative approaches do  not always provide a comparable substitute for such data. Additionally, as  discussed above, we did not base our analysis solely upon earned value  data, but evaluated other available program performance documentation  to ensure that we captured performance for the entire investment. As  such, we maintain that the use of earned value data (among other sources)  and the comparison of selected investments’ Dashboard ratings with our  analyses resulted in a fair assessment.
We are sending copies of this report to interested congressional  committees; the Secretaries of the Departments of Homeland Security,  Transportation, the Treasury, and Veterans Affairs, as well as the  Commissioner of the Social Security Administration; and other interested  parties. In addition, the report will be available at no charge on GAO’s Web  site at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) determine what efforts the Office of  Management and Budget (OMB) has under way to improve the Dashboard  and the ways in which it is using data from the Dashboard to improve  information technology (IT) management and (2) examine the accuracy of  the cost and schedule performance ratings on OMB’s Dashboard.
To address the first objective, we examined related OMB guidance and  documentation to determine the ongoing and planned improvements OMB  has made to the Dashboard and discussed these improvements with OMB  officials. Additionally, we evaluated OMB documentation of current and  planned efforts to oversee and improve the management of IT investments  and the Dashboard, such as memos detailing the results of investment  management review sessions, and interviewed OMB officials regarding  these efforts.
To address the second objective, we selected 5 agencies and 10 investments  to review. To select these agencies and investments, we first identified the  12 agencies with the largest IT budgets as reported in OMB’s fiscal year 2011  Exhibit 53. This list of agencies was narrowed down to 10 because 2  agencies did not have enough investments that met our criteria (as defined  in the following text). We then excluded agencies that were assessed in our  previous review of the Dashboard. As a result, we selected the  Departments of Homeland Security (DHS), Transportation (DOT), the  Treasury, and Veterans Affairs (VA), and the Social Security Administration  (SSA). In selecting the specific investments at each agency, we identified the  10 largest investments that, according to the fiscal year 2011 budget, were  spending more than half of their budget on IT development, modernization,  and enhancement work. To narrow this list, we excluded investments  whose four different Dashboard ratings (overall, cost, schedule, and chief  information officer) were generally “red” because they were likely already  receiving significant scrutiny. We then selected 2 investments per agency.  As part of this selection process, we considered the following: investments  that use earned value management techniques to monitor cost and schedule  performance, and investments whose four different Dashboard ratings  appeared to be in conflict (e.g., cost and schedule ratings were “green,” yet  the overall rating was “red”). The 10 final investments were DHS’s U.S.
Citizenship and Immigration Service (USCIS)-Transformation program and  U.S. Coast Guard-Command, Control, Communications, Computers,  Intelligence, Surveillance & Reconnaissance (C4ISR) program; DOT’s  Automatic Dependent Surveillance-Broadcast system and En Route  Automation Modernization system; Treasury’s Modernized e-File system  and Payment Application Modernization investment; VA’s HealtheVet Core  and Medical Legacy investments; and SSA’s Disability Case Processing  System and Intelligent Disability program. The 5 agencies account for 22  percent of the planned IT spending for fiscal year 2011. The 10 investments  selected for case study represent about $1.27 billion in total planned  spending in fiscal year 2011.
To assess the accuracy of the cost and schedule performance ratings on  the Dashboard, we evaluated earned value data of 7 of the selected  investments to determine their current cost and schedule performances  and compared them with the performance ratings on the Dashboard. The  investment earned value data were contained in contractor earned value  management performance reports obtained from the programs. To  perform the current performance analysis, we averaged the cost and  schedule variances over the last 6 months and compared the averages with  the performance ratings on the Dashboard. To assess the accuracy of the  cost data, we compared them with data from other available supporting  program documents, including program management reports and  inspector general reports; electronically tested the data to identify obvious  problems with completeness or accuracy; and interviewed agency and  program officials about the earned value management systems. For the  purposes of this report, we determined that the cost data for these 7  investments were sufficiently reliable. For the 3 remaining investments,  we did not use earned value data because the investments either did not  measure performance using earned value management or the earned value  data were determined to be insufficiently reliable. Instead, we used other  program documentation, such as inspector general reports and internal  performance management system performance ratings, to assess the  accuracy of the cost and schedule ratings on the Dashboard. We did not  test the adequacy of the agency or contractor cost-accounting systems.  Our evaluation of these cost data was based on what we were told by each  agency and the information it could provide.
We also interviewed officials from OMB and the selected agencies and  reviewed OMB guidance to obtain additional information on OMB’s and  agencies’ efforts to ensure the accuracy of the data used to rate investment  performance on the Dashboard. We used the information provided by  OMB and agency officials to identify the factors contributing to inaccurate  cost and schedule performance ratings on the Dashboard.
We conducted this performance audit from July 2010 to March 2011 at the  selected agencies’ offices in the Washington, D.C., metropolitan area. Our  work was done in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Selected Investment Descriptions
paragraphs: Below are descriptions of each of the selected investments that are  included in this review.
subsections: section_title: Department of Homeland Security
paragraphs: USCIS-Transformation is a bureauwide program to move from a paper- based filing system to a centralized, consolidated, electronic adjudication  filing system.
The C4ISR Common Operating Picture collects and fuses relevant  information for Coast Guard commanders to allow them to efficiently  exercise authority, while directing and monitoring all assigned forces and  first responders, across the range of Coast Guard operations.
section_title: Department of Transportation
subsections: section_title: Automatic Dependent Surveillance-Broadcast
paragraphs: The Automatic Dependent Surveillance-Broadcast system is intended to be  an underlying technology in the Federal Aviation Administration’s plan to  transform air traffic control from the current radar-based system to a  satellite-based system. The Automatic Dependent Surveillance-Broadcast  system is to bring the precision and reliability of satellite-based  surveillance to the nation’s skies.
section_title: En Route Automation Modernization
paragraphs: The En Route Automation Modernization system is to replace the current  computer system used at the Federal Aviation Administration’s high- altitude en route centers. The current system is considered the backbone  of the nation’s airspace system and processes flight radar data, provides  communications, and generates display data to air traffic controllers.
section_title: Department of the Treasury
subsections: section_title: Modernized e-File
paragraphs: The current Modernized e-File system is a Web-based platform that  supports electronic tax returns and annual information returns for large  corporations and certain tax-exempt organizations, as well as individual  Form 1040 and other schedules and supporting forms. This system is  being updated to include the electronic filing of the more than 120  remaining 1040 forms and schedules. Combining these efforts is intended  to streamline tax return filing processes and reduce the costs associated  with paper tax returns.
section_title: Payment Application Modernization
paragraphs: The Payment Application Modernization investment is an effort to  modernize the current mainframe-based software applications that are  used to disburse approximately 1 billion federal payments annually. The  existing payment system is a configuration of numerous software  applications that generate check, wire transfer, and Automated Clearing  House payments for federal program agencies, including the Social  Security Administration, Internal Revenue Service, Department of  Veterans Affairs, and others.
section_title: Department of Veterans Affairs
subsections: section_title: HealtheVet Core
paragraphs: HealtheVet Core was a set of initiatives to improve health care delivery,  provide the platform for health information sharing, and update outdated  technology. The investment was to support veterans, their beneficiaries,  and providers by advancing the use of health care information and leading  edge IT to provide a patient-centric, longitudinal, computable health  record. According to department officials, the HealtheVet Core investment  was “stopped” in August 2010.
section_title: Medical Legacy
paragraphs: The Medical Legacy program is an effort to provide software applications  necessary to maintain and modify the department’s Veterans Health  Information Systems and Technology Architecture.
section_title: Social Security Administration
subsections: section_title: Disability Case Processing System
paragraphs: The Disability Case Processing System is intended to provide common  functionality and consistency to support the business processes of each  state’s Disability Determination Services. Ultimately, it is to provide  analysis functionality, integrate health IT, improve case processing,  simplify maintenance, and reduce infrastructure growth costs.
section_title: Intelligent Disability
paragraphs: The Intelligent Disability program is intended to reduce the backlog of  disability claims, develop an electronic case processing system, and  support efficiencies in the claims process.
Table 2 provides additional details for each of the selected investments in  our review.
section_title: Appendix III: Comments from the Department of Veterans Affairs
section_title: Appendix IV: Comments from the Social Security Administration
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff also made key  contributions to this report: Carol Cha, Assistant Director; Shannin O’Neill,  Assistant Director; Alina Johnson; Emily Longcore; Lee McCracken; and  Kevin Walsh.