Information Technology: OMB Has Made Improvements to Its Dashboard, but Further Work Is Needed by Agencies and OMB to Ensure Data Accuracy
Released: 2011-03-15T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT investments and how these funds are to be allocated. According to the President’s Budget for Fiscal Year 2011, the total planned spending on IT in fiscal year 2011 is an estimated $79.4 billion, a 1.2 percent increase from the fiscal year 2010 budget level of $78.4 billion. OMB plays a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects.

To assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by federal agencies and report to Congress on the net program performance benefits achieved as a result of these investments. Further, the act places responsibility for managing investments with the heads of agencies and establishes chief information officers (CIO) to advise and assist agency heads in carrying out this responsibility. Another key law is the E-Government Act of 2002, which requires OMB to report annually to Congress on the status of e- government. In these reports, referred to as Implementation of the E- Government Act reports, OMB is to describe the administration’s use of e- government principles to improve government performance and the delivery of information and services to the public.

To help carry out its oversight role, in 2003, OMB established the Management Watch List, which included mission-critical projects that needed to improve performance measures, project management, IT security, or overall justification for inclusion in the federal budget. Further, in August 2005, OMB established a High-Risk List, which consisted of projects identified by federal agencies, with the assistance of OMB, as requiring special attention from oversight authorities and the highest levels of agency management.

Over the past several years, we have reported and testified on OMB’s initiatives to highlight troubled IT projects, justify investments, and use project management tools. We have made multiple recommendations to OMB and federal agencies to improve these initiatives to further enhance the oversight and transparency of federal projects. Among other things, we recommended that OMB develop a central list of projects and their deficiencies and analyze that list to develop governmentwide and agency assessments of the progress and risks of the investments, identifying opportunities for continued improvement. In addition, in 2006 we also recommended that OMB develop a single aggregate list of high-risk projects and their deficiencies and use that list to report to Congress on progress made in correcting high-risk problems. As a result, OMB started publicly releasing aggregate data on its Management Watch List and disclosing the projects’ deficiencies. Furthermore, OMB issued governmentwide and agency assessments of the projects on the Management Watch List and identified risks and opportunities for improvement, including risk management and security.

OMB’s Dashboard Publicizes Investment Details and Performance Status

More recently, to further improve the transparency and oversight of agencies’ IT investments, and to address data quality issues, in June 2009, OMB publicly deployed a Web site, known as the IT Dashboard, which replaced the Management Watch List and High-Risk List. It displays federal agencies’ cost, schedule, and performance data for the approximately 800 major federal IT investments at 27 federal agencies. According to OMB, these data are intended to provide a near real-time perspective on the performance of these investments, as well as a historical perspective. Further, the public display of these data is intended to allow OMB; other oversight bodies, including Congress; and the general public to hold the government agencies accountable for results and progress.

The Dashboard was initially deployed in June 2009 based on each agency’s Exhibit 53 and Exhibit 300 submissions. After the initial population of data, agency CIOs have been responsible for updating cost, schedule, and performance fields on a monthly basis, which is a major improvement from the quarterly reporting cycle OMB previously used for the Management Watch List and High-Risk List.

For each major investment, the Dashboard provides performance ratings on cost and schedule, a CIO evaluation, and an overall rating, which is based on the cost, schedule, and CIO ratings. As of July 2010, the cost rating was determined by a formula that calculates the amount by which an investment’s total actual costs deviate from the total planned costs. Similarly, the schedule rating is the variance between the investment’s planned and actual progress to date. Figure 1 displays the rating scale and associated categories for cost and schedule variations.

Each major investment on the Dashboard also includes a rating determined by the agency CIO, which is based on his or her evaluation of the performance of each investment. The rating is expected to take into consideration the following criteria: risk management, requirements management, contractor oversight, historical performance, and human capital. This rating is to be updated when new information becomes available that would affect the assessment of a given investment.

Last, the Dashboard calculates an overall rating for each major investment. This overall rating is an average of the cost, schedule, and CIO ratings, with each representing one-third of the overall rating. However, when the CIO’s rating is lower than both the cost and schedule ratings, the CIO’s rating will be the overall rating. Figure 2 shows the overall performance ratings of the 805 major investments on the Dashboard as of March 2011.

Earned Value Management Provides Additional Insight on Program Cost and Schedule

To better manage IT investments, OMB issued guidance directing agencies to develop comprehensive policies to ensure that their major IT investments and high-risk development projects use earned value management to manage their investments. Earned value management is a technique that integrates the technical, cost, and schedule parameters of a development contract and measures progress against them. During the planning phase, a performance measurement baseline is developed by assigning and scheduling budget resources for defined work. As work is performed and measured against the baseline, the corresponding bud value is “earned.” Using this earned value metric, cost and schedule variances, as well as cost a nd time to complete estimates, can be determined and analyzed.

Without knowing the planned cost of completed work and work in progress (i.e., the earned value), it is difficult to determine a program’s true status. Earned value allows for this key information, which provides an objective view of program status and is necessary for understandin health of a program. As a result, earned value management can alert program managers to potential problems sooner than using expenditures alone, thereby reducing the chance and magnitude of cost overruns and schedule slippages. Moreover, earned value management directly supp the institutionalization of key processes for acquiring and developing systems and the ability to effectively manage investments—areas that are often found to be inadequate on the basis of our assessments of major IT investments.

OMB Has Taken Step Address Prior GAO Recommendations on Dashboard Improving Accuracy

In July 2010, we reported that the cost and schedule ratings on OMB’s Dashboard were not always accurate for selected agencies. Specifically, we found that several selected investments had notable discrepancies in their cost or schedule ratings, the cost and schedule ratings did not takeinto consideration current performance, and the number of milestones (activities) reported by agencies varied widely. We made a number of recommendations to OMB to better ensure that the Dashboard prov meaningful ratings and accurate investment data. In particular, we recommended that OMB report on its planned Dashboard changes to improve the accuracy of performance information and provide guidance agencies that standardizes activity reporting. OMB agreed with the two recommendations and reported it had initiated work to address them.

OMB Has Multiple Efforts Under Way to Further Refine the Dashboard and Uses the Dashboard to Improve IT Management

Since our last report, OMB has initiated multiple efforts to increase the Dashboard’s value as a management and oversight tool, and has used data in the Dashboard to improve the management of federal IT investments. Specifically, OMB is focusing its efforts in four main areas: streamlining key OMB investment reporting tools, eliminating manual monthly submissions, coordinating with agencies to improve data, and improving the user interface.

OMB’s plan to reform federal IT management commits OMB to streamlining two of the Dashboard’s sources of information—specifically, the OMB Exhibits 53 and 300. OMB has committed, by May 2011, to reconstruct the exhibits around distinct data elements that drive value for agencies and provide the information necessary for meaningful oversight. OMB anticipates that these changes will also alleviate the reporting burden and increase data accuracy, and that the revised exhibits will serve as its authoritative management tools.

According to OMB officials, the Dashboard no longer accepts manual data submissions. Instead, the Dashboard allows only system-to-system submissions. Officials explained that this update allows the Dashboard to reject incomplete submissions and those that do not meet the Dashboard’s data validation rules. By eliminating direct manual submissions, this effort is expected to improve the reliability of the data shown on the Dashboard.

Further, OMB officials stated that they work to improve the Dashboard through routine interactions with agencies and IT portfolio management tool vendors, training courses, working groups, and data quality letters to agencies. Specifically, OMB officials stated that they held 58 TechStat reviews (discussed later in this report), hosted four online training sessions (recordings of which OMB officials stated are also available online), collaborated with several Dashboard working groups, and sent letters to agency CIOs identifying specific data quality issues on the Dashboard that their agencies could improve. Further, OMB officials explained that in December 2010, OMB analysts informed agency CIOs about specific data quality issues and provided analyses of agency data, a comparison of agency Dashboard performance with that of the rest of the government, and expected remedial actions. OMB anticipates these efforts will increase the Dashboard’s data reliability by ensuring that the agencies are aware of and are working to address issues.

Finally, OMB continues to improve the Dashboard’s user interface. For instance, in November 2010, OMB updated the Dashboard to provide new views of historical data and rating changes and provide new functionality allowing agencies to make corrections to activities and performance metrics (conforming to rebaselining guidance). Officials also described a planned future update, which is intended to contain updated budget data, display corrections and changes made to activities, and reflect increased validation of agency-submitted data. OMB anticipates these efforts will increase the transparency and reliability of investment information on the Dashboard by providing agencies and users additional ways to view investment information and by improving validation of submitted data.

Additionally, OMB uses the Dashboard to improve the management of IT investments. Specifically, OMB analysts are using the Dashboard’s investment trend data to track changes and identify issues with investments’ performance in a timely manner. OMB analysts also use the Dashboard to identify data quality issues and drive improvements to the data. The Federal CIO stated that the Dashboard has greatly improved oversight capabilities compared with those of previously used mechanisms, such as the annual capital asset plan and business case (Exhibit 300) process. Additionally, according to OMB officials, the Dashboard is one of the key sources of information that OMB analysts use to identify IT investments that are experiencing performance problems and to select them for a TechStat session—a review of selected IT investments between OMB and agency leadership that is led by the Federal CIO. As of December 2010, OMB officials stated that 58 TechStat sessions have been held with federal agencies. According to OMB, these sessions have enabled the government to improve or terminate IT investments that are experiencing performance problems. Information from the TechStat sessions and the Dashboard was used by OMB to identify, halt, and review all federal financial IT systems modernization projects. Furthermore, according to OMB, these sessions and other OMB management reviews have resulted in a $3 billion reduction in life-cycle costs, as of December 2010. OMB officials stated that, as of December 2010, 11 investments have been reduced in scope and 4 have been terminated as a result of these sessions. For example, The TechStat on the Department of Housing and Urban Development’s Transformation Initiative investment found that the department lacked the skills and resources necessary and would not be positioned to succeed. As a result, the department agreed to reduce the number of projects from 29 to 7 and to limit fiscal year 2010 funds for these 7 priority projects to $85.7 million (from the original $138 million).

The TechStat on the National Archives and Records Administration’s Electronic Records Archives investment resulted in six corrective actions, including halting fiscal year 2012 development funding pending the completion of a strategic plan.

According to OMB officials, OMB and agency CIOs also used the Dashboard data and TechStat sessions, in addition to other forms of research (such as reviewing program documentation, news articles, and inspector general reports), to identify 26 high-risk IT projects and, in turn, coordinate with agencies to develop corrective actions for these projects at TechStat sessions. For example, the Department of the Interior is to establish incremental deliverables for its Incident Management Analysis and Reporting System, which will accelerate delivery of services that will help 6,000 law enforcement officers protect the nation’s natural resources and cultural monuments.

Dashboard Ratings Did Not Always Reflect True Investment Cost and Schedule Performance

While the efforts previously described are important steps to improving the quality of the information on the Dashboard, cost and schedule performance data inaccuracies remain. The Dashboard’s cost and schedule ratings were not always reflective of the true performance for selected investments from the five agencies in our review. More specifically, while the Dashboard is intended to present near real-time performance, the ratings did not always reflect the current performance of these investments. Dashboard rating inaccuracies were the result of weaknesses in agency practices, such as the Dashboard not reflecting baseline changes and the reporting of erroneous data, as well as limitations of the Dashboard’s calculations. Until the agencies submit complete, reliable, and timely data to the Dashboard and OMB revises its Dashboard calculations, performance ratings will continue to be inaccurate and may not reflect current program performance.

Cost and Schedule Performance Was Not Always Accurately Depicted in Dashboard Ratings

Most of the Dashboard’s cost ratings of the nine selected investments did not match the results of our analyses over a 3-month period. Specifically, four investments had inaccurate ratings for 2 or more months, and two were inaccurate for 1 month, while three investments were accurately depicted for all 3 months. For example, Intelligent Disability’s cost performance was rated “red” on the Dashboard for July 2010 and “green” for August 2010, whereas our analysis showed its current cost performance was “yellow” for those months. Further, Medical Legacy’s cost ratings were “red” on the Dashboard for June through August 2010, while the department’s internal rating showed that the cost performance for 105 of the 107 projects that constitute the investment was “green” in August 2010; similar ratings were also seen for June and July 2010. Overall, the Dashboard’s cost ratings generally showed poorer performance than our assessments. Figure 3 shows the comparison of the selected investments’ Dashboard cost ratings with GAO’s ratings based on analysis of agency data for the months of June 2010 through August 2010.

Regarding schedule, most of the Dashboard’s ratings of the nine selected investments did not match the results of our analyses over a 3-month period. Specifically, seven investments had inaccurate ratings for 2 or more months, and two were inaccurate for 1 month. For example, Automatic Dependent Surveillance-Broadcast’s schedule performance was rated “green” on the Dashboard in July 2010, but our analysis showed its current performance was “yellow” that month. Additionally, the “green” schedule ratings for En Route Automation Modernization did not represent how this program is actually performing. Specifically, we recently reported that the program is experiencing significant schedule delays, and the CIO evaluation of the program on the Dashboard has indicated schedule delays since February 2010. As with the cost ratings, the Dashboard’s schedule ratings generally showed poorer performance than our assessments. Figure 4 shows the comparison of the selected investments’ Dashboard schedule ratings with GAO’s ratings based on analysis of agency data for the months of June 2010 through August 2010.

Dashboard Rating Inaccuracies Are a Result of Weaknesses in Agencies’ Practices and Limitations with OMB’s Calculations

OMB guidance, as of June 2010, states that agencies are responsible for maintaining consistency between the data in their internal systems and the data on the Dashboard. Furthermore, the guidance states that agency CIOs should update their evaluation on the Dashboard as soon as new information becomes available that affects the assessment of a given investment. According to our assessment of the nine selected investments, agencies did not always follow this guidance. In particular, there were four primary weaknesses in agency practices that resulted in inaccurate cost and schedule ratings on the Dashboard: the investment baseline on the Dashboard was not reflective of the investment’s actual baseline, agencies did not report data to the Dashboard, agencies reported erroneous data, and unreliable earned value data were reported to the Dashboard. In addition, two limitations of OMB’s Dashboard calculations contributed to ratings inaccuracies: a lack of emphasis on current performance and an understatement of schedule variance. Table 1 shows the causes of inaccurate ratings for the selected investments.

Inconsistent program baseline: Three of the selected investments reported baselines on the Dashboard that did not match the actual baselines tracked by the agencies. Agency officials responsible for each of these investments acknowledged this issue. For example, according to Modernized e-File officials, the investment was in the process of a rebaseline in June 2010; thus, officials were unable to update the baseline on the Dashboard until July 2010. For another investment—HealtheVet Core—officials stated that it was stopped in August, and thus the HealtheVet Core baseline on the Dashboard is incorrect. As such, the CIO investment evaluation should have been updated to reflect that the investment was stopped. In June 2010, OMB issued new guidance on rebaselining, which stated that agencies should update investment baselines on the Dashboard within 30 days of internal approval of a baseline change and that this update will be considered notification to OMB. However, agencies still must go through their internal processes to approve a new baseline, and during this process the baseline on the Dashboard will be inaccurate. As such, investment CIO ratings should disclose that performance data on the Dashboard are unreliable because of baseline changes. However, the CIO evaluation ratings for these investments did not include such information. Without proper disclosure of pending baseline changes and resulting data reliability weaknesses, OMB and other external oversight groups will not have the appropriate information to make informed decisions about these investments.

Missing data submissions: Three investments did not upload complete and timely data submissions to the Dashboard. For example, DHS officials did not submit data to the Dashboard for the C4ISR investment from June through August 2010. According to DHS officials, C4ISR investment officials did not provide data for DHS to upload for these months. Further compounding the performance rating issues of this investment is that in March 2010, inaccurate data were submitted for nine of its activities; these data were not corrected until September 2010. Until officials submit complete, accurate, and timely data to the Dashboard, performance ratings may continue to be inaccurate.

Erroneous data submissions: Seven investments reported erroneous data to the Dashboard. For example, SSA submitted start dates for Intelligent Disability and Disability Case Processing System activities that had not actually started yet. SSA officials stated that, because of SSA’s internal processes, their start dates always correspond to the beginning of the fiscal year. In addition, according to a Treasury official, Internal Revenue Service officials for the Modernized e-File investment provided inaccurate data for the investment’s “actual percent complete” fields for some activities. Until officials submit accurate data to the Dashboard, performance ratings may continue to be inaccurate.

Unreliable source data: Treasury’s Payment Application Modernization investment used unreliable earned value data as the sole source of data on the Dashboard. As such, this raises questions about the accuracy of the performance ratings reported on the Dashboard. Investment officials stated that they have taken steps to address weaknesses with the earned value management system and are currently evaluating other adjustments to investment management processes. However, without proper disclosure about data reliability in the CIO assessment, OMB and other external oversight groups will not have the appropriate information to make informed decisions about this investment.

Additionally, two limitations in the Dashboard method for calculating ratings contributed to inaccuracies: Current performance calculation: The Dashboard is intended to represent near real-time performance information on all major IT investments, as previously discussed. To OMB’s credit, in July 2010, it updated the Dashboard’s cost and schedule calculations to include both ongoing and completed activities in order to accomplish this. However, the performance of ongoing activities is combined with the performance of completed activities, which can mask recent performance. As such, the cost and schedule performance ratings on the Dashboard may not always reflect current performance. Until OMB updates the Dashboard’s cost and schedule calculations to focus on current performance, the performance ratings may not reflect performance problems that the investments are presently facing, and OMB and agencies are thus missing an opportunity to identify solutions to such problems.

Schedule variance calculation: Another contributing factor to certain schedule inaccuracies is that OMB’s schedule calculation for in-progress activities understates the schedule variance for activities that are overdue. Specifically, OMB’s schedule calculation does not recognize the full variance of an overdue activity until it has actually completed. For example, as of September 13, 2010, the Dashboard reported a 21-day schedule variance for an En Route Automation Modernization activity that was actually 256 days overdue. Until OMB updates its in-progress schedule calculation to be more reflective of the actual schedule variance of ongoing activities, schedule ratings for these activities may be understated.

Conclusions

The Dashboard has enhanced OMB’s and agency CIOs’ oversight of federal IT investments. Among other things, performance data from the Dashboard are being used to identify poorly performing investments for executive leadership review sessions. Since the establishment of the Dashboard, OMB has worked to continuously refine it, with multiple planned improvement efforts under way for improving the data quality and Dashboard usability.

However, the quality of the agency data reported to the Dashboard continues to be a challenge. Specifically, the cost and schedule ratings on the Dashboard were not always accurate in depicting current program performance for most of the selected investments, which is counter to OMB’s goal to report near real-time performance. The Dashboard rating inaccuracies were due, in part, to weaknesses in agencies’ practices and limitations in OMB’s calculations. More specifically, the agency practices—including the inconsistency between Dashboard and program baselines, reporting of erroneous data, and unreliable source data—and OMB’s formulas to track current performance have collectively impaired data quality. Until agencies provide more reliable data and OMB improves the calculations of the ratings on the Dashboard, the accuracy of the ratings will continue to be in question and the ratings may not reflect current program performance.

Recommendations for Executive Action

To better ensure that the Dashboard provides accurate cost and schedule performance ratings, we are making eleven recommendations to the heads of each of the five selected agencies. Specifically, we are recommending that: The Secretary of the Department of Homeland Security direct the CIO to ensure that investment data submissions include complete and accurate investment information for all required fields; comply with OMB’s guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline; and work with C4ISR officials to comply with OMB’s guidance on updating investment cost and schedule data on the Dashboard at least monthly.

The Secretary of the Department of Transportation direct the CIO to work with Automatic Dependent Surveillance-Broadcast officials to comply with OMB’s guidance on updating investment cost and schedule data on the Dashboard at least monthly.

The Secretary of the Department of the Treasury direct the CIO to comply with OMB’s guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline; work with Modernized e-File officials to report accurate actual percent complete data for each of the investment’s activities; and work with Payment Application Modernization officials to disclose the extent of this investment’s data reliability issues in the CIO rating assessment on the Dashboard.

The Secretary of the Department of Veterans Affairs direct the CIO to comply with OMB’s guidance on updating the CIO rating as soon as new information becomes available that affects the assessment of a given investment, including when an investment is in the process of a rebaseline; work with Medical Legacy officials to comply with OMB’s guidance on updating investment cost and schedule data on the Dashboard at least monthly; and ensure Medical Legacy investment data submitted to the Dashboard are consistent with the investment’s internal performance information.

The Commissioner of the Social Security Administration direct the CIO to ensure that data submissions to the Dashboard include accurate investment information for all required fields.

In addition, to better ensure that the Dashboard provides meaningful ratings and reliable investment data, we are recommending that the Director of OMB direct the Federal CIO to take the following two actions: develop cost and schedule rating calculations that better reflect current update the Dashboard’s schedule calculation for in-progress activities to more accurately represent the variance of ongoing, overdue activities.

Agency Comments and Our Evaluation

We provided a draft of our report to the five agencies in our review and to OMB. In commenting on the draft, four agencies generally concurred with our recommendations. One agency, the Department of Transportation, agreed to consider our recommendation. OMB agreed with one of our recommendations and disagreed with the other. In addition, OMB raised concerns about the methodology used in our report. Agencies also provided technical comments, which we incorporated as appropriate. Each agency’s comments are discussed in more detail below.

In e-mail comments on a draft of the report, DHS’s Departmental Audit Liaison stated that the department concurred with our recommendations.

In e-mail comments, DOT’s Director of Audit Relations stated that DOT would consider our recommendation; however, he also stated that the department disagreed with the way its investments were portrayed in the draft. Specifically, department officials stated that our assessment was not reasonable because our methodology only incorporated the most recent 6 months of performance rather than using cumulative investment performance. As discussed in this report, combining the performance of ongoing and completed activities can mask recent performance. As such, we maintain that our methodology is a reasonable means of deriving near real-time performance, which the Dashboard is intended to represent.

In oral comments, Treasury’s Chief Architect stated that the department generally concurred with our recommendations and added that the department would work to update its Dashboard ratings for the two selected investments.

In written comments, VA’s Chief of Staff stated that the department generally concurred with our recommendations and agreed with our conclusions. Further, he outlined the department’s planned process improvements to address the weaknesses identified in this report. VA’s comments are reprinted in appendix III.

In written comments, SSA’s Deputy Chief of Staff stated that the Administration agreed with our recommendation and had taken corrective actions intended to prevent future data quality errors. SSA’s comments are reprinted in appendix IV.

Officials from OMB’s Office of E-Government & Information Technology provided the following oral comments on the draft: OMB officials agreed with our recommendation to update the Dashboard’s schedule calculation for in-progress activities to more accurately represent the variance of ongoing, overdue activities. These officials stated that the agency has long-term plans to update the Dashboard’s calculations, which they believe will provide a solution to the concern identified in this report.

OMB officials disagreed with our recommendation to develop cost and schedule rating calculations that better reflect current investment performance. According to OMB, real-time performance is always reflected in the ratings since current investment performance data are uploaded to the Dashboard on a monthly basis.

Regarding OMB’s comments, our point is not that performance data on the Dashboard are infrequently updated, but that the use of historical data going back to an investment’s inception can mask more recent performance. For this reason, current investment performance may not always be as apparent as it should be, as this report has shown. Until the agency places less emphasis on the historical data factored into the Dashboard’s calculations, it will be passing up an opportunity to more efficiently and effectively identify and oversee investments that either currently are or soon will be experiencing problems.

OMB officials also described the agency’s plans for enhancing Dashboard data quality and performance calculations. According to OMB, plans were developed in February 2011 with stakeholders from other agencies to standardize the reporting structure for investment activities. Further, OMB officials said that their plans also call for the Dashboard’s performance calculations to be updated to more accurately reflect activities that are delayed. In doing so, OMB stated that agencies will be expected to report new data elements associated with investment activities. Additionally, OMB officials noted that new agency requirements associated with these changes will be included in key OMB guidance (Circular A-11) no later than September 2011.

OMB officials also raised two concerns regarding our methodology. Specifically, OMB stated that our reliance on earned value data as the primary source for determining investment performance was questionable. These officials stated that, on the basis of their experience collecting earned value data, the availability and quality of these data vary significantly across agencies. As such, according to these officials, OMB developed its Dashboard cost and schedule calculations to avoid relying on earned value data.

We acknowledge that the quality of earned value data can vary. As such, we took steps to ensure that the data we used were reliable enough to evaluate the ratings on the Dashboard, and discounted the earned value data of one of the selected investments after determining its data were insufficient for our needs. While we are not critical of OMB’s decision to develop its own method for calculating performance ratings, we maintain that our use of earned value data is sound. Furthermore, earned value data were not the only source for our analysis; we also based our findings on other program management documentation, such as inspector general reports and internal performance management system performance ratings, as discussed in appendix I.

OMB also noted that, because we used earned value data to determine investment performance, our ratings were not comparable to the ratings on the Dashboard. Specifically, OMB officials said that the Dashboard requires reporting of all activities under an investment, including government resources or operations and maintenance activities. OMB further said that this is more comprehensive than earned value data, which only account for contractor-led development activities.

We acknowledge and support the Dashboard’s requirement for a comprehensive accounting of investment performance. Further, we agree that earned value data generally only cover development work associated with the investments (thus excluding other types of work, such as planning and operations and maintenance). For this reason, as part of our methodology, we specifically selected investments for which the majority of the work being performed was development work. We did this because earned value management is a proven technique for providing objective quantitative data on program performance, and alternative approaches do not always provide a comparable substitute for such data. Additionally, as discussed above, we did not base our analysis solely upon earned value data, but evaluated other available program performance documentation to ensure that we captured performance for the entire investment. As such, we maintain that the use of earned value data (among other sources) and the comparison of selected investments’ Dashboard ratings with our analyses resulted in a fair assessment.

We are sending copies of this report to interested congressional committees; the Secretaries of the Departments of Homeland Security, Transportation, the Treasury, and Veterans Affairs, as well as the Commissioner of the Social Security Administration; and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) determine what efforts the Office of Management and Budget (OMB) has under way to improve the Dashboard and the ways in which it is using data from the Dashboard to improve information technology (IT) management and (2) examine the accuracy of the cost and schedule performance ratings on OMB’s Dashboard.

To address the first objective, we examined related OMB guidance and documentation to determine the ongoing and planned improvements OMB has made to the Dashboard and discussed these improvements with OMB officials. Additionally, we evaluated OMB documentation of current and planned efforts to oversee and improve the management of IT investments and the Dashboard, such as memos detailing the results of investment management review sessions, and interviewed OMB officials regarding these efforts.

To address the second objective, we selected 5 agencies and 10 investments to review. To select these agencies and investments, we first identified the 12 agencies with the largest IT budgets as reported in OMB’s fiscal year 2011 Exhibit 53. This list of agencies was narrowed down to 10 because 2 agencies did not have enough investments that met our criteria (as defined in the following text). We then excluded agencies that were assessed in our previous review of the Dashboard. As a result, we selected the Departments of Homeland Security (DHS), Transportation (DOT), the Treasury, and Veterans Affairs (VA), and the Social Security Administration (SSA). In selecting the specific investments at each agency, we identified the 10 largest investments that, according to the fiscal year 2011 budget, were spending more than half of their budget on IT development, modernization, and enhancement work. To narrow this list, we excluded investments whose four different Dashboard ratings (overall, cost, schedule, and chief information officer) were generally “red” because they were likely already receiving significant scrutiny. We then selected 2 investments per agency. As part of this selection process, we considered the following: investments that use earned value management techniques to monitor cost and schedule performance, and investments whose four different Dashboard ratings appeared to be in conflict (e.g., cost and schedule ratings were “green,” yet the overall rating was “red”). The 10 final investments were DHS’s U.S.

Citizenship and Immigration Service (USCIS)-Transformation program and U.S. Coast Guard-Command, Control, Communications, Computers, Intelligence, Surveillance & Reconnaissance (C4ISR) program; DOT’s Automatic Dependent Surveillance-Broadcast system and En Route Automation Modernization system; Treasury’s Modernized e-File system and Payment Application Modernization investment; VA’s HealtheVet Core and Medical Legacy investments; and SSA’s Disability Case Processing System and Intelligent Disability program. The 5 agencies account for 22 percent of the planned IT spending for fiscal year 2011. The 10 investments selected for case study represent about $1.27 billion in total planned spending in fiscal year 2011.

To assess the accuracy of the cost and schedule performance ratings on the Dashboard, we evaluated earned value data of 7 of the selected investments to determine their current cost and schedule performances and compared them with the performance ratings on the Dashboard. The investment earned value data were contained in contractor earned value management performance reports obtained from the programs. To perform the current performance analysis, we averaged the cost and schedule variances over the last 6 months and compared the averages with the performance ratings on the Dashboard. To assess the accuracy of the cost data, we compared them with data from other available supporting program documents, including program management reports and inspector general reports; electronically tested the data to identify obvious problems with completeness or accuracy; and interviewed agency and program officials about the earned value management systems. For the purposes of this report, we determined that the cost data for these 7 investments were sufficiently reliable. For the 3 remaining investments, we did not use earned value data because the investments either did not measure performance using earned value management or the earned value data were determined to be insufficiently reliable. Instead, we used other program documentation, such as inspector general reports and internal performance management system performance ratings, to assess the accuracy of the cost and schedule ratings on the Dashboard. We did not test the adequacy of the agency or contractor cost-accounting systems. Our evaluation of these cost data was based on what we were told by each agency and the information it could provide.

We also interviewed officials from OMB and the selected agencies and reviewed OMB guidance to obtain additional information on OMB’s and agencies’ efforts to ensure the accuracy of the data used to rate investment performance on the Dashboard. We used the information provided by OMB and agency officials to identify the factors contributing to inaccurate cost and schedule performance ratings on the Dashboard.

We conducted this performance audit from July 2010 to March 2011 at the selected agencies’ offices in the Washington, D.C., metropolitan area. Our work was done in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Investment Descriptions

Below are descriptions of each of the selected investments that are included in this review.

Department of Homeland Security

USCIS-Transformation is a bureauwide program to move from a paper- based filing system to a centralized, consolidated, electronic adjudication filing system.

The C4ISR Common Operating Picture collects and fuses relevant information for Coast Guard commanders to allow them to efficiently exercise authority, while directing and monitoring all assigned forces and first responders, across the range of Coast Guard operations.

Department of Transportation

Automatic Dependent Surveillance-Broadcast

The Automatic Dependent Surveillance-Broadcast system is intended to be an underlying technology in the Federal Aviation Administration’s plan to transform air traffic control from the current radar-based system to a satellite-based system. The Automatic Dependent Surveillance-Broadcast system is to bring the precision and reliability of satellite-based surveillance to the nation’s skies.

En Route Automation Modernization

The En Route Automation Modernization system is to replace the current computer system used at the Federal Aviation Administration’s high- altitude en route centers. The current system is considered the backbone of the nation’s airspace system and processes flight radar data, provides communications, and generates display data to air traffic controllers.

Department of the Treasury

Modernized e-File

The current Modernized e-File system is a Web-based platform that supports electronic tax returns and annual information returns for large corporations and certain tax-exempt organizations, as well as individual Form 1040 and other schedules and supporting forms. This system is being updated to include the electronic filing of the more than 120 remaining 1040 forms and schedules. Combining these efforts is intended to streamline tax return filing processes and reduce the costs associated with paper tax returns.

Payment Application Modernization

The Payment Application Modernization investment is an effort to modernize the current mainframe-based software applications that are used to disburse approximately 1 billion federal payments annually. The existing payment system is a configuration of numerous software applications that generate check, wire transfer, and Automated Clearing House payments for federal program agencies, including the Social Security Administration, Internal Revenue Service, Department of Veterans Affairs, and others.

Department of Veterans Affairs

HealtheVet Core

HealtheVet Core was a set of initiatives to improve health care delivery, provide the platform for health information sharing, and update outdated technology. The investment was to support veterans, their beneficiaries, and providers by advancing the use of health care information and leading edge IT to provide a patient-centric, longitudinal, computable health record. According to department officials, the HealtheVet Core investment was “stopped” in August 2010.

Medical Legacy

The Medical Legacy program is an effort to provide software applications necessary to maintain and modify the department’s Veterans Health Information Systems and Technology Architecture.

Social Security Administration

Disability Case Processing System

The Disability Case Processing System is intended to provide common functionality and consistency to support the business processes of each state’s Disability Determination Services. Ultimately, it is to provide analysis functionality, integrate health IT, improve case processing, simplify maintenance, and reduce infrastructure growth costs.

Intelligent Disability

The Intelligent Disability program is intended to reduce the backlog of disability claims, develop an electronic case processing system, and support efficiencies in the claims process.

Table 2 provides additional details for each of the selected investments in our review.

Appendix III: Comments from the Department of Veterans Affairs

Appendix IV: Comments from the Social Security Administration

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, the following staff also made key contributions to this report: Carol Cha, Assistant Director; Shannin O’Neill, Assistant Director; Alina Johnson; Emily Longcore; Lee McCracken; and Kevin Walsh.