Information Technology: OMB's Dashboard Has Increased Transparency and Oversight, but Improvements Needed
Released: 2010-07-20T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much the government plans to spend on IT projects and how these funds are to be allocated. Planned federal IT spending has now risen to an estimated $79.4 billion for fiscal year 2011, a 1.2 percent increase from the 2010 level of $78.4 billion. OMB plays a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects.

To assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by federal agencies and report to Congress on the net program performance benefits achieved as a result of these investments. Further, the act places responsibility for managing investments with the heads of agencies and establishes chief information officers (CIO) to advise and assist agency heads in carrying out this responsibility. Another key law is the E-Government Act of 2002, which requires OMB to report annually to Congress on the status of e- government. In these reports, referred to as the Implementation of the E- Government Act reports, OMB is to describe the Administration’s use of e- government principles to improve government performance and the delivery of information and services to the public.

To help carry out its oversight role and assist the agencies in carrying out their responsibilities as assigned by the Clinger-Cohen Act, OMB developed a Management Watch List in 2003 and a High Risk List in 2005 to focus executive attention and to ensure better planning and tracking of major IT investments. Consistent with the Clinger-Cohen Act, OMB reported on the status of investments on the Management Watch List and High Risk List in its annual budget documents.

Over the past several years, we have reported and testified on OMB’s initiatives to highlight troubled projects, justify investments, and use project management tools. We have made multiple recommendations to OMB and federal agencies to improve these initiatives to further enhance the oversight and transparency of federal projects. Among other things, we recommended that OMB develop a central list of projects and their deficiencies and analyze that list to develop governmentwide and agency assessments of the progress and risks of the investments, identifying opportunities for continued improvement. In addition, in 2006 we also recommended that OMB develop a single aggregate list of high-risk projects and their deficiencies and use that list to report to Congress on progress made in correcting high-risk problems. As a result, OMB started publicly releasing aggregate data on its Management Watch List and disclosing the projects’ deficiencies. Furthermore, OMB issued governmentwide and agency assessments of the projects on the Management Watch List and identified risks and opportunities for improvement, including risk management and security.

Table 1 provides a historical perspective of the number of projects on the Management Watch List and their associated budgets for the period of time during which OMB updated the Management Watch List. The table shows that while the number of projects and their associated budgets on the list generally decreased, the number of projects on the Management Watch List increased by 239 projects and $13 billion for fiscal year 2009, and represented a significant percentage of the total budget.

OMB’s Dashboard Publicizes Investment Details and Performance Status

More recently, to further improve the transparency into and oversight of agencies’ IT investments, and to address data quality issues, in June 2009, OMB publicly deployed a Web site, known as the IT Dashboard, which replaced the Management Watch List and High Risk List. It displays information on federal agencies’ cost, schedule, and performance information for the approximately 800 major federal IT investments at 28 federal agencies. According to OMB, these data are intended to provide a near real-time perspective of the performance of these investments, as well as a historical perspective. Further, the public display of these data are intended to allow OMB, other oversight bodies, and the general public to hold the government agencies accountable for results and progress.

The Dashboard was initially deployed in June 2009 based on each agency’s Exhibit 53 and Exhibit 300 submissions. After the initial population of data, agency CIOs have been responsible for updating cost, schedule, and performance fields on a monthly basis, which is a major improvement from the quarterly reporting cycle OMB previously used for the Management Watch List and High Risk List.

For each major investment, the Dashboard provides performance ratings on cost and schedule, a CIO evaluation, and an overall rating which is based on the cost, schedule, and CIO ratings. The cost rating is determined by a formula that calculates the amount by which an investment’s aggregated actual costs deviate from the aggregated planned costs. Table 2 displays the rating scale and associated category for cost variations.

An investment’s schedule rating is calculated by determining the average days late or early. Table 3 displays the rating scale and associated category for schedule deviations.

Each major investment on the Dashboard also includes a rating determined by the agency CIO, which is based on his or her evaluation of the performance of each investment. The rating is expected to take into consideration the following criteria: risk management, requirements management, contractor oversight, historical performance, and human capital. This rating is to be updated when new information becomes available that would impact the assessment of a given investment.

Lastly, the Dashboard calculates an overall rating for each major investment. Figure 1 identifies the Dashboard’s overall ratings scale. This overall rating is an average of the cost, schedule, and CIO ratings, with each representing one-third of the overall rating. However, when the CIO’s rating is lower than both the cost and schedule ratings, the CIO’s rating will be the overall rating. Of the 792 major investments on the Dashboard as of May 2010, 540 (68 percent) were green, 204 (26 percent) were yellow, and 48 (6 percent) were red.

Earned Value Management Provides Additional Insight on Program Cost and Schedule

Earned value management is a technique that integrates the technical, cost, and schedule parameters of a development contract and measures progress against them. During the planning phase, a performance measurement baseline is developed by assigning and scheduling budget resources for defined work. As work is performed and measured against the baseline, the corresponding budget value is “earned.” Using this earned value metric, cost and schedule variances, as well as cost and time to complete estimates, can be determined and analyzed.

Without knowing the planned cost of completed work and work in progress (i.e., the earned value), it is difficult to determine a program’s true status. Earned value allows for this key information, which provides an objective view of program status and is necessary for understanding the health of a program. As a result, earned value management can alert program managers to potential problems sooner than using expenditures alone, thereby reducing the chance and magnitude of cost overruns and schedule slippages. Moreover, earned value management directly supports the institutionalization of key processes for acquiring and developing systems and the ability to effectively manage investments—areas that are often found to be inadequate on the basis of our assessments of major IT investments. In August 2005, OMB issued guidance that all major and high- risk development projects, among other things, develop comprehensive policies to ensure that their major IT investments use earned value management to manage their investments.

Performance Ratings on the Dashboard Were Not Always Accurate

Cost and schedule performance ratings were not always accurate for the selected investments we reviewed. A key reason for the inaccuracies is that the Dashboard’s cost and schedule ratings do not reflect current performance. Another issue with the ratings is that large inconsistencies exist in the number of milestones that agencies report on the Dashboard.

Cost and Schedule Performance Ratings Were Not Always Accurate

The cost and schedule performance ratings of selected investments were not always accurate. There were several instances of inaccurate cost ratings; however, two investments experienced notable discrepancies while the other discrepancies were not as dramatic. Specifically, 5 of the 8 selected investments on the Dashboard had inaccurate cost ratings: BioSense, Financial Management Modernization Initiative, Joint Precision Approach and Landing System, Law Enforcement Wireless Communication, and Unified Financial Management System. For example, the Dashboard rated the Law Enforcement Wireless Communication investment a 10 for cost (less than 5 percent variance) every month from July 2009 through January 2010. However, our analysis shows the investment’s cost rating during December 2009 and January 2010 is equivalent to an 8 (a variance of 10 percent to less than 15 percent). Accordingly, this investment’s cost performance should have been rated a “yellow” instead of a “green,” meaning it needed attention. Further, the Dashboard’s cost rating for the Financial Management Modernization Initiative reported that this investment was “yellow,” while it should have been “green” for 7 months. Maneuver Control System, Sequoia Platform, and Risk Management Agency-13 are the three investments that had accurate cost ratings. Figure 2 shows the comparison of selected investments’ Dashboard cost ratings to GAO’s ratings for the months of July 2009-January 2010.

There were fewer instances of discrepancies with the schedule ratings; however, these discrepancies were also notable. Specifically, of the 8 selected investments, the Dashboard’s schedule ratings were inaccurate for 2 investments: Risk Management Agency-13 and the Unified Financial Management System. The Unified Financial Management System’s last completed milestone was in May 2009 and the Dashboard rating for the investment’s schedule has been a 10 since July 2009. However, investment data we examined showed the schedule rating should have been a 5 (greater than or equal to 30 days and less than 90 days behind schedule) from September 2009 through December 2009. As a result, this investment’s schedule performance should have been rated a “yellow” instead of a “green” for those months. Additionally, the Dashboard’s schedule rating for Risk Management Agency-13 reported that this investment was “red” for two months, while it should have been “green,” and “yellow” for four months, when it should have been “green.” BioSense, Financial Management Modernization Initiative, Joint Precision Approach and Landing System, Law Enforcement Wireless Communication, Maneuver Control System, and Sequoia Platform are the 6 investments that had accurate schedule ratings. Figure 3 shows the comparison of selected investments’ Dashboard schedule ratings to GAO’s ratings for the months of July 2009-January 2010.

In addition to determining that cost and schedule ratings are not always accurate, we found other data inaccuracies. Specifically, rebaseline information on the Dashboard was not always accurate. Best practices and GAO’s Cost Estimating Guide state that a rebaseline should occur when the current cost and schedule baseline does not adequately represent the amount of work to be completed, causing difficulty in monitoring progress of the program. However, OMB reports all major and minor corrections to planned information on the Dashboard, includ typographical fixes, as a rebaseline. More specifically, while the Dashboard allows agencies to provide reasons for baseline changes, the current version of the Dashboard, at a high level, identifies all changes to planned information as rebaselines. For example, according to the Dashboard, DOJ’s Law Enforcement Wireless Communication investment has been rebaselined four times. However, program officials stated that the program has only been rebaselined once. Similarly, the Dashboard shows that the Sequoia Platform and Integrated Management Navigation System investments at DOE have both been rebaselined four times. However, program officials stated that neither of these programs had actually been rebaselined. Rather, they stated that this number represents instances in which they made minor corrections to the data on the Dashboard. Table 4 shows the selected investments whose program officials reported a lower number of rebaselines than what was reported on the Dashboard.

Cost and Schedule Ratings Do Not Reflect Current Performance and Wide Variation in Milestone Reporting Exists

A primary reason why the cost and schedule ratings were not always accurate is that the cost and schedule ratings do not take current performance into consideration for many investments on the Dashboard, though it is intended to represent near real-time performance information on all major IT investments. Specifically, as of April 2010, the formula to calculate the cost ratings on the Dashboard intentionally only factored in completed portions of the investments (referred to as milestones) to determine cost ratings. As such, milestones that are currently under way are not taken into account. Table 5 identifies each selected investment’s last completed milestone and the number of days that the Dashboard’s cost rating is out of date for each selected investment.

OMB officials agreed that the ratings not factoring in current performance is an area needing improvement and said that they are planning on upgrading the Dashboard application in July 2010 to include updated cost and schedule formulas that factor in the performance of ongoing milestones; however, they have not yet made this change. One step OMB has taken toward collecting the information needed for the new formulas is that it now requires agencies to provide information on their investment milestones’ planned and actual start dates. In addition, OMB officials stated that they plan to use a previously unused data field—percent complete. These are key data points necessary to calculate the performance of ongoing milestones.

Another issue with the ratings is that there were wide variations in the number of milestones agencies reported. For example, DOE’s Integrated Management Navigation System investment lists 314 milestones, whereas DOD’s Joint Precision Approach and Landing System investment lists 6. Having too many milestones may mask recent performance problems because the performance of every milestone (i.e., historical and recently completed) is equally averaged into the ratings. Specifically, investments that perform well during many previously completed milestones and then start performing poorly on a few recently completed milestones can maintain ratings that still reflect good performance. A more appropriate approach could be to give additional weight to recently completed and ongoing milestones when calculating the ratings. Too many detailed milestones also defeat the purpose of an executive-level reporting tool. Conversely, having too few milestones can limit the amount of information available to track work and rate performance and allows agencies to potentially skew the performance ratings.

In commenting on a draft of this report, the Federal CIO stated that OMB has a new version of the Dashboard that implements updated cost and schedule calculations. He stated that the new calculations greatly increase the weight of current activities. As of July 1, 2010, this updated Dashboard had not been released. An OMB analyst subsequently told us that the agency plans to release the new version in July 2010. Additionally, OMB officials have provided us with documentation of the new calculations and demonstrated the new version of the Dashboard that will be released soon. The Federal CIO also added that OMB will consider additional changes to the ratings in the future.

Table 6 demonstrates the large inconsistencies in the number of milestones reported for each selected investment.

In June 2009, OMB issued guidance that agencies are responsible for providing quality data and, at minimum, should provide milestones that consist of major segments of the investment, referred to as work breakdown structure level 2, but prefers that agencies provide lower-level milestones within each segment (work breakdown structure level 3). A work breakdown structure is the cornerstone of every program because it defines in detail the work necessary to accomplish a program’s objectives. Standardizing a work breakdown structure is considered a best practice because it enables an organization to collect and share data among programs. Further, standardizing work breakdown structures allows data to be shared across organizations.

However, certain agencies are not following OMB’s guidance and list milestones that they consider to be at work breakdown structure level 1, which are high-level milestones. Specifically, of the 5 agencies we reviewed, officials at DOD, USDA, and DOE stated that they were reporting work breakdown structure level 1 milestones to the Dashboard for each of their selected investments. OMB officials acknowledge that not all agencies are following their guidance, but stated that OMB analysts are working with agencies to try to improve compliance. Furthermore, the guidance that OMB has provided is not clear on the level of detail that it wants agencies to report in their milestones and has left it to the agencies to individually interpret their general guidance. Specifically, while OMB states that agencies should report milestones that are, at a minimum, work breakdown structure level 2, there is no commonly accepted definition among federal agencies on the level of detail that should comprise each of these levels. OMB officials acknowledged that they have not provided clear guidance, but recently stated that they have begun exploring ways to ensure more uniformity across agencies’ reporting. Specifically, in commenting on a draft of this report, the Federal CIO stated that OMB has recently chartered a working group comprised of representatives from several federal agencies, with the intention of developing clear guidance for standardizing and improving investment activity reporting.

OMB and agencies acknowledge that additional improvements can be made beyond the cost and schedule ratings and have taken certain steps to try to improve the accuracy of the data. For example, OMB implemented an automated monthly data upload process and created a series of data validation rules that detect common data entry errors, such as investment milestone start dates that occur after completion dates. In addition, four of the five agencies we reviewed indicated that they have processes in place aimed at improving the accuracy of the data. For instance, HHS has established a process wherein an official has been assigned responsibility for ensuring the Dashboard is accurately updated. Further, DOJ has developed an automated process to find missing data elements in the information to be uploaded on the Dashboard.

Despite these efforts, until OMB upgrades the Dashboard application to improve the accuracy of the cost and schedule ratings to include ongoing milestones, explains the outcome of these improvements in its next annual report to Congress on the Implementation of the E-Government Act (which is a key mechanism for reporting on the implementation of the Dashboard), provides clear and consistent guidance to agencies that standardizes milestone reporting, and ensures agencies comply with the new guidance, the Dashboard’s cost and schedule ratings will likely continue to experience data accuracy issues.

Use of the Dashboard as a Management Tool Varies

Officials at three of the five agencies we reviewed—DOD, DOJ, and HHS— stated that they are not using the Dashboard to manage their investments, and the other two agencies, DOE and USDA, indicated that they are using the Dashboard to manage their investments.

Specifically, officials from the three agencies are not using the Dashboard to manage their investments because they have other existing means to do so: DOD officials indicated that they use the department’s Capital Planning and Investment Control process to track IT investment data—including cost and schedule.

DOJ uses an internal dashboard that the office of the CIO developed that provides for more detailed management of investments than OMB’s Dashboard.

HHS officials said they use a portfolio investment management tool, which they indicated provides greater insight into their investments.

Officials from the other two agencies—DOE and USDA— noted that they are using the Dashboard as a management tool to supplement their existing internal processes to manage their IT investments.

DOE officials stated that since their current process is based on a quarterly review cycle, the monthly reporting nature of the Dashboard has allowed officials to gain more frequent insight into investment performance. As a result, DOE officials say that they are able to identify potential issues before these issues present problems for investments.

USDA officials stated that they use the ratings on the Dashboard to identify investments that appear to be problematic and hold meetings with the investments’ program managers to discuss corrective actions.

Additionally, in OMB’s fiscal year 2009 Report to Congress on the Implementation of the E-Government Act of 2002, 11 agencies reported on how the Dashboard has increased their visibility and awareness of IT investments. For example, the Department of Veterans’ Affairs terminated 12 IT projects, partly because of the increased visibility that the CIO obtained from the Dashboard.

OMB indicated that it is using the Dashboard to manage IT investments. Specifically, OMB analysts are using the Dashboard’s investment trend data to track changes and identify issues with investments’ performance in a timely manner and are also using the Dashboard to identify and drive investment data quality issues. The Federal CIO stated that the Dashboard has greatly improved oversight capabilities compared to previously used mechanisms. He also stated that the Dashboard has increased the accountability of agencies’ CIOs and established much-needed visibility. According to OMB officials, the Dashboard is one of the key sources of information that OMB analysts use to identify investments that are experiencing performance problems and select them for a TechStat session—a review of selected IT investments between OMB and agency leadership that is led by the Federal CIO. OMB has identified factors that may result in a TechStat session, such as policy interests, Dashboard data inconsistencies, recurring patterns of problems, and/or an OMB analyst’s concerns with an investment. As of June 2010, OMB officials indicated that 27 TechStat sessions have been held with federal agencies. According to OMB, this program enables the government to improve or terminate IT investments that are experiencing performance problems.

Conclusions

OMB has taken significant steps to enhance the oversight, transparency, and accountability of federal IT investments by creating its IT Dashboard. However, the cost and schedule ratings on the Dashboard were not always accurate. Further, the rebaseline data were not always accurate. The cost and schedule inaccuracies were due, in part, to calculations of ratings that did not factor in current performance. Additionally, there were large inconsistencies in the number of milestones that agencies report on the Dashboard because OMB has not fully defined the level of detail that federal agencies should use to populate the Dashboard and several selected agencies decided not to follow OMB’s general guidance. Moreover, the performance of historical and recently completed milestones are equally averaged in the cost and schedule ratings, which is counter to OMB’s goal to report near real-time performance on the Dashboard. While the use of the Dashboard as a management tool varies, OMB has efforts under way to include the performance of ongoing milestones and its officials acknowledge that additional improvements are needed. Nevertheless, until OMB explains in its next annual Implementation of the E-Government Act report how the upgrade to the Dashboard application has improved the accuracy of the cost and schedule ratings, and provides clear and consistent guidance that enables agencies to report standardized information on their milestones, the accuracy of the data on the Dashboard may continue to be in question.

Recommendations for Executive Action

To better ensure that the IT Dashboard provides meaningful ratings and accurate investment data, we are recommending that the Director of OMB take the following two actions: include in its next annual Implementation of the E-Government Act report the effect of planned formula changes on the accuracy of data; and develop and issue clear guidance that standardizes milestone reporting on the Dashboard.

In addition, we are recommending that the Secretaries of the Departments of Agriculture, Defense, and Energy direct their Chief Information Officers to ensure that they comply with OMB’s guidance on standardized milestone reporting, once it is available.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Federal CIO and DOE’s Associate CIO for IT Planning, Architecture, and E- Government. Letters from these agencies are reprinted in appendixes III and IV. In addition, we received technical comments via e-mail from a Coordinator at HHS, which we incorporated where appropriate. In addition, the Deputy CIO from USDA, the Principal Director to the Deputy Assistant Secretary of Defense for Resources from DOD, and an Audit Liaison Specialist from DOJ indicated via e-mail that they had reviewed the draft report and did not have any comments.

In OMB’s comments on our draft report, which contained four recommendations to the OMB Director, the Federal CIO stated that he agreed with two recommendations and disagreed with two because of actions OMB has recently taken. After reviewing these actions, we agreed that they addressed our concerns and will not make these two recommendations.

OMB agreed with our recommendation that it include in its next annual Implementation of the E-Government Act report how the planned formula changes have improved the accuracy of data.

OMB agreed with our recommendation that it develop and issue clear guidance that standardizes milestone reporting on the Dashboard. Additionally, the Federal CIO asked that we update the report to reflect that they have recently chartered a working group comprised of representatives from several federal agencies, with the intention of developing clear guidance for standardizing and improving investment activity reporting. We have incorporated this additional information into the report.

In response to our draft recommendation that OMB revise the IT Dashboard and its guidance so that only major changes to investments are considered to be rebaselines, OMB provided us with its new guidance on managing IT baselines, which was issued on June 28, 2010. The guidance, among other things, describes when agencies should report baseline changes on the Dashboard. OMB also provided documentation of the specific modifications that will be made in an upcoming release of the Dashboard to improve the way baseline changes are displayed. We agree that these recent changes address our recommendation. As such, we updated the report to acknowledge and include this additional information, where appropriate.

Regarding our recommendation that OMB consider weighing recently completed and ongoing milestones more heavily than historical milestones in the cost and schedule ratings, the Federal CIO stated that OMB has a new version of the Dashboard that implements updated cost and schedule calculations. He stated that the new calculations greatly increase the weight of current activities. As previously stated, as of July 1, 2010, this updated Dashboard had not been released. An OMB analyst subsequently told us that the agency plans to release the new version in July 2010. Additionally, OMB officials have provided us with documentation of the new calculations and demonstrated the new version of the Dashboard that will be released soon. The Federal CIO also added that OMB will consider additional changes to the ratings in the future. We agree that these recent changes address our recommendation. As such, we updated the report to acknowledge and include this additional information, where appropriate. Additionally, OMB will report on the effect of the upcoming changes to the calculations in its next annual Implementation of the E-Government Act report.

OMB also provided additional comments, which we address in appendix III.

In DOE’s comments on our draft report, the Associate CIO for IT Planning, Architecture, and E-Government indicated that she agreed with our assessment of the implementation of the IT Dashboard across federal agencies and with the recommendations presented to OMB. Additionally, in response to our recommendation that the CIO of DOE comply with OMB guidance on milestone reporting once it is available, the Associate CIO stated that once OMB releases the additional guidance, DOE officials will work to ensure the appropriate level of detail is reported on the Dashboard. DOE also provided an additional comment, which we address in appendix IV.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees; the Director of the Office of Management and Budget; the Secretaries of the Departments of Agriculture, Defense, Energy, Health and Human Services, and Justice; and other interested parties. In addition, the report will be available at no charge on our Web site at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) examine the accuracy of the cost and schedule performance ratings on the Dashboard for selected investments and (2) determine whether the data on the Dashboard are used as a management tool to make improvements to IT investments.

To address both objectives, we selected five agencies and ten investments to review. To select these agencies and investments, we first identified ten agencies with large IT budgets as reported in the Office of Management and Budget’s (OMB) fiscal year 2010 Exhibit 53. We then identified the five largest investments at each of the ten agencies, according to the fiscal year 2010 budget, that were spending more than half of their budget on IT development, modernization, and enhancement work, and were primarily carried out by contractors. In narrowing the list to five agencies and ten total investments, we considered several factors to ensure there were two viable investments at each agency: The investment is not part of our ongoing audit work related to cost, schedule, and technical performance.

The investment is not part of a recent governmentwide earned value management review.

The investment has not been highlighted as an investment needing significant attention.

The collective list of investments creates a balance of investment sizes to include both larger and smaller investments.

The five agencies are: the Departments of Agriculture (USDA), Defense (DOD), Energy (DOE), Health and Human Services (HHS), and Justice (DOJ).

The ten investments are: USDA’s Financial Management Modernization Initiative and Risk Management Agency-13 Program; DOD’s Joint Precision Approach and Landing System and Maneuver Control System; DOE’s Integrated Management Navigation System and Sequoia Platform; HHS’s BioSense Program and Electronic Research Administration System; DOJ’s Law Enforcement Wireless Communication and Unified Financial Management System (see appendix II for descriptions of each investment).

To address the first objective, we evaluated earned value data of the selected investments to determine their cost and schedule performance and compared it to the ratings on the Dashboard. The investment earned value data was contained in contractor earned value management performance reports obtained from the programs. To perform this analysis, we compared the investment’s cumulative cost variance for each month from July 2009 through January 2010 to the cost variance reported on the Dashboard for those months. Similarly, we calculated the number of months each investment was ahead or behind schedule over the same period on the Dashboard. We also assessed 13 months of investment data to analyze trends in cost and schedule performances. To further assess the accuracy of the cost data, we compared it with other available supporting program documents, including monthly and quarterly investment program management reports; electronically tested the data to identify obvious problems with completeness or accuracy; and interviewed agency and program officials about the data and earned value management systems. For the purposes of this report, we determined that the cost data at eight of the investments were sufficiently reliable to use for our assessment. For the two remaining investments, we determined that based on their methods of earned value management, the data would not allow us to sufficiently assess and rate monthly investment performance. We did not test the adequacy of the agency or contractor cost-accounting systems. Our evaluation of these cost data was based on the documentation the agency provided.

We also reviewed and analyzed OMB’s and the selected agencies’ processes for populating and updating the Dashboard. Additionally, we interviewed officials from OMB and the selected agencies and reviewed OMB guidance to obtain additional information on OMB’s and agencies’ efforts to ensure the accuracy of the investment performance data and cost and schedule performance ratings on the Dashboard. We used the information provided by OMB and agency officials to identify the factors contributing to inaccurate cost and schedule performance ratings on the Dashboard. Moreover, we used this information to examine the accuracy of the rebaseline information on the Dashboard, we interviewed agency and program officials about the number of rebaselines each investment has had, and compared these data with the rebaseline information listed on the Dashboard.

To address our second objective, we analyzed related agency documentation to assess what policies or procedures they have implemented for using the data on the Dashboard to make management decisions. We also interviewed agency and program officials regarding the extent to which they use the data on the Dashboard as a management tool. Additionally, we attended one of OMB’s TechStat sessions, which are reviews of selected IT investments between OMB and agencies.

We conducted this performance audit from January to July 2010 at the selected agencies’ offices in the Washington, D.C., metropolitan area. Our work was done in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Investment Descriptions

Below are descriptions of each of the selected investments that are included in this review.

Department of Agriculture (USDA)

Financial Management Modernization Initiative

The Financial Management Modernization Initiative is USDA’s financial management system modernization program. It is intended to be the central financial system for USDA and is to consolidate the current financial management system environment from 19 legacy systems into one Web-based system.

Risk Management Agency-13

USDA’s Risk Management Agency-13 program is intended to support the reengineering of all business systems associated with the crop insurance program and provide a central financial system that will provide Web- based tools and applications for accessing Risk Management Agency data.

Department of Defense (DOD)

Joint Precision Approach and Landing System

DOD’s Joint Precision Approach and Landing System investment is intended to provide a precision approach and landing capability for all DOD ground and airborne systems. It is intended to enable U.S. forces to safely land aircraft on any suitable surface worldwide (land and sea), with ceiling and/or visibility the limiting factor.

Maneuver Control System

DOD’s maneuver control system investment is intended to provide, among other things, the warfighter environment and collaborative and situational awareness tools used to support executive decision making, planning, rehearsal, and execution management. This system is to be used throughout the Army to provide a common view of critical information.

Department of Energy (DOE)

Integrated Management Navigation System

DOE’s Integrated Management Navigation System consists of 5 major projects and is intended to standardize and integrate accounting, data warehouse, human resource, procurement, and budget processes throughout DOE. The Integrated Management Navigation System incorporates enterprisewide projects from DOE’s Office of the Chief Financial Officer, Office of Human Capital Management, and Office of Management.

Sequoia Platform

DOE’s Sequoia Platform is a supercomputer being developed for use by three weapons laboratories—Los Alamos, Lawrence Livermore, and Sandia National Laboratories—to contribute dramatically to the national security enterprise. This supercomputer will also be used in maintaining the nuclear deterrence and areas of nonproliferation, nuclear counterterrorism, and support to the intelligence community.

Department of Health and Human Services (HHS)

HHS’s BioSense program is intended to improve the nation’s capabilities for disease detection, monitoring, and near real-time health situational awareness by creating a system that uses data from existing health-related databases to identify patterns of disease symptoms prior to specific diagnoses.

Electronic Research Administration

HHS’s Electronic Research Administration program is the National Institutes of Health’s system for conducting interactive electronic transactions for the receipt, review, monitoring, and administration of grant awards to biomedical investigators worldwide. It is also intended to provide the technology capabilities for the agency to efficiently and effectively perform grants administration functions.

Department of Justice (DOJ)

Law Enforcement Wireless Communication

DOJ’s Law Enforcement Wireless Communication System, also known as the Integrated Wireless Network, is to support the replacement and modernization of failing radio systems and achieve communication standards at DOJ’s law enforcement agencies. This program is intended to provide all four law enforcement components with a shared unified radio network, which should eliminate redundant coverage and duplicative radio sites, while providing efficient and comparable coverage.

Unified Financial Management System

DOJ’s Unified Financial Management System is to improve the existing and future financial management and procurement operations across DOJ. Upon full implementation, the Unified Financial Management System will replace five financial management systems and multiple procurement systems with an integrated commercial off-the-shelf solution. This is to streamline and standardize business processes and procedures across the DOJ components.

Table 7 provides additional details for each of the selected investments in our review.

Appendix III: Comments from the Office of Management and Budget

The following is GAO’s response to the Office of Management and Budget’s (OMB) additional comments.

GAO Comments

1. We agree that the Dashboard has increased transparency, accountability, and oversight; therefore, we updated the report to discuss additional uses of the Dashboard, such as the use of trend data, improved oversight capabilities, and enhancements to agencies’ investment management processes. We also updated the number of Techstat sessions that have taken place. 2. While additional data quality issues need to be addressed in the Dashboard, we agree that the Dashboard is an improvement when compared to OMB’s previous oversight tools such as the Management Watch List and High Risk List. As such, we modified the report to highlight these improvements. For example, we added to the report that the Dashboard’s monthly reporting cycle is a significant improvement in the quality of the data from the Management Watch List and High Risk List, which were updated on a quarterly basis. 3. As stated in the report, we found that the ratings were not always accurate. We based this characterization on the fact that there were several instances in which the ratings were inconsistent with the performance indicated in our analysis of the investments’ earned value management (EVM) reports and were notably different (e.g., ratings of “green” versus “yellow”). We agree that EVM data generally only covers the contracted development parts of the investments. As such, as part of our methodology, we specifically selected investments where the majority of each investment was focused on development efforts (versus operational) and primarily carried out by contractors. As such, we maintain that the comparison between the selected investments’ Dashboard ratings and the performance indicated in their EVM reports is a fair assessment. 4. We acknowledge that the quality of EVM reports can vary. As such, we took steps to ensure that the EVM reports we used were reliable enough to evaluate the ratings on the Dashboard, and as OMB’s comments indicate, we discounted two of the ten selected investments after determining that their data was insufficient for our needs. We do not state that OMB should base their ratings solely on EVM data. 5. We agree that the original cost and schedule calculations are performing as planned (i.e., are not defective) and we further clarified this point in the report. We also note that planned changes to the rating calculations will incorporate current performance. However, these calculations, as originally planned and implemented, do not factor in the performance of ongoing milestones, which we and OMB agree is an area for improvement. 6. We agree that the severity of the discrepancies were not always dramatic. However, 4 of the 8 investments had notable discrepancies on either their cost or schedule ratings. Specifically, as demonstrated in the report, there were multiple instances in which the ratings were discrepant enough to change the color of the ratings. The difference between a “green” rating (i.e., normal performance) and a “yellow” rating (i.e., needs attention) is the difference between whether an investment is flagged for needing attention or not, which we believe is an important point to highlight. 7. We agree that agencies have a responsibility to provide quality milestone data; however, we maintain that OMB’s existing guidance on which milestones to report is too general for agencies to ensure they are reporting consistently. OMB acknowledges that this is an area for improvement and has established a working group to address this issue. 8. As previously discussed, on June 28, 2010, OMB issued its new guidance on managing IT baselines. This guidance, among other things, describes when agencies should report baselines changes to the Dashboard. Officials also provided information on the upcoming release of the Dashboard—which is intended to be released in July 2010—that will change the way baseline changes are displayed. We agree that these recent changes address the issues we identified. 9. We acknowledge that the Dashboard has made significant improvements to oversight and transparency, in comparison to OMB’s previous methods of overseeing IT investments, and we have added additional information to the background of the report to highlight this point.

Appendix IV: Comments from the Department of Energy

The following is GAO’s response to the Department of Energy’s (DOE) additional comment.

GAO Comment

OMB’s guidance required agencies to provide data at one consistent work breakdown structure level, rather than a mix of multiple levels. OMB and others confirmed that agencies were able to transmit milestones at a single consistent level. For this report, we observed agencies uploading at levels 1 through 4 and, thus, disagree that agencies were unable to transmit milestones lower than level 1.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, the following staff also made key contributions to this report: Shannin O’Neill, Assistant Director; Carol Cha; Eric Costello; Rebecca Eyler; Emily Longcore; Bradley Roach; and Kevin Walsh.