Information Technology: Further Improvements Needed to identify and Oversee Poorly Planned and Performing Projects
Released: 2007-09-20T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much the government plans to spend for IT and how these funds are to be allocated. Federal IT spending has risen to an estimated $65 billion in fiscal year 2008.

OMB plays a key role in overseeing the implementation and management of federal IT investments. To improve this oversight, Congress enacted the Clinger-Cohen Act in 1996, expanding the responsibilities delegated to OMB and agencies under the Paperwork Reduction Act. Among other things, Clinger-Cohen requires agency heads, acting through agency chief information officers, to better link their IT planning and investment decisions to program missions and goals and to implement and enforce IT management policies, procedures, standards, and guidelines. The act also requires that agencies engage in capital planning and performance and results-based management. OMB’s responsibilities under the act include establishing processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. OMB must also report to Congress on the net program performance benefits achieved as a result of major capital investments in information systems that are made by executive agencies.

In response to the Clinger-Cohen Act and other statutes, OMB developed policy for planning, budgeting, acquisition, and management of federal capital assets. This policy is set forth in OMB Circular A-11 (section 300) and in OMB’s Capital Programming Guide (supplement to Part 7 of Circular A-11), which directs agencies to develop, implement, and use a capital programming process to build their capital asset portfolios. Among other things, OMB’s Capital Programming Guide directs agencies to evaluate and select capital asset investments that will support core mission functions that must be performed by the federal government and demonstrate projected returns on investment that are clearly equal to or better than alternative uses of available public resources; institute performance measures and management processes that monitor actual performance and compare to planned results; and establish oversight mechanisms that require periodic review of operational capital assets to determine how mission requirements might have changed and whether the asset continues to fulfill mission requirements and deliver intended benefits to the agency and customers.

To further support the implementation of IT capital planning practices as required by statute and directed in OMB’s Capital Programming Guide, we have developed an IT investment management framework that agencies can use in developing a stable and effective capital planning process. Consistent with the statutory focus on selecting, controlling, and evaluating investments, this framework focuses on these processes in relation to IT investments specifically. It is a tool that can be used to determine both the status of an agency’s current IT investment management capabilities and the additional steps that are needed to establish more effective processes. Mature and effective management of IT investments can vastly improve government performance and accountability. Without good management, such investments can result in wasteful spending and lost opportunities for improving delivery of services to the public.

Prior Reviews on Federal IT Investment Management Have Identified Weaknesses

Only by effectively and efficiently managing their IT resources through a robust investment management process can agencies gain opportunities to make better allocation decisions among many investment alternatives and further leverage their investments. However, the federal government faces enduring IT challenges in this area. For example, in January 2004 we reported on mixed results of federal agencies’ use of IT investment management practices. Specifically, we reported that although most of the agencies had IT investment boards responsible for defining and implementing the agencies’ investment management processes, agencies did not always have important mechanisms in place for these boards to effectively control investments, including decision-making rules for project oversight, early warning mechanisms, and/or requirements that corrective actions for underperforming projects be agreed upon and tracked. Executive-level oversight of project-level management activities provides organizations with increased assurance that each investment will achieve the desired cost, benefit, and schedule results. Accordingly, we made several recommendations to agencies to improve their practices.

In previous work using our investment management framework, we reported that the use of IT investment management practices by agencies was mixed. For example, a few agencies that have followed the framework in implementing capital planning processes have made significant improvements. In contrast, however, we and others have continued to identify weaknesses at agencies in many areas, including immature management processes to support both the selection and oversight of major IT investments and the measurement of actual versus expected performance in meeting established performance measures. For example, we recently reported that the Department of Homeland Security and the Department of Treasury did not have the processes in place to effectively select and oversee their major investments.

OMB’s Management Watch List Is Intended to Correct Project Weaknesses and Business Case Deficiencies

To help ensure that investments of public resources are justified and that public resources are wisely invested, OMB began using its Management Watch List in the President’s fiscal year 2004 budget request, as a means to oversee the justification for and planning of agencies’ IT investments. This list was derived based on a detailed review of the investments’ Capital Asset Plan and Business Case, also known as the exhibit 300.

The exhibit 300 is a reporting mechanism intended to enable an agency to demonstrate to its own management, as well as OMB, that a major project is well planned in that it has employed the disciplines of good project management; developed a strong business case for the investment; and met other Administration priorities in defining the cost, schedule, and performance goals proposed for the investment.

We reported in 2005 that OMB analysts evaluate agency exhibit 300s by assigning scores to each exhibit 300 based on guidance presented in OMB Circular A-11. As described in this circular, the scoring of a business case consists of individual scoring for 10 categories, as well as a total composite score of all the categories. The 10 categories are project (investment) management, performance-based management system (including the earned value life-cycle costs formulation, and support of the President’s Management Agenda.

Projects are placed on the Management Watch List if they receive low scores (3 or less on a scale from 1 to 5) in the areas of performance goals, performance-based management systems, security and privacy or a low composite score.

According to OMB, agencies with weaknesses in these three areas are to submit remediation plans addressing the weaknesses. OMB officials also stated that decisions on follow-up and monitoring the progress are typically made by staff with responsibility for reviewing individual agency budget submissions, depending on the staff’s insights into agency operations and objectives. According to OMB officials, those Management Watch List projects that receive specific follow-up attention receive feedback through the passback process, targeted evaluation of remediation plans designed to address weaknesses, the apportioning of funds so that the use of budgeted dollars was conditional on appropriate remediation plans being in place, and the quarterly e-Gov Scorecards. OMB removes projects from the Management Watch List as agencies remediate the weaknesses identified with these projects’ business cases.

OMB’s High Risk Projects Process Intended to Correct and Improve Project Performance

As originally defined in OMB Circular A-11 and subsequently reiterated in an August 2005 memorandum, high risk projects are those that require special attention from oversight authorities and the highest levels of agency management. These projects are not necessarily “at risk” of failure, but may be on the list because of one or more of the following four reasons: The agency has not consistently demonstrated the ability to manage complex projects.

The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

Delay or failure of the project would introduce for the first time unacceptable or inadequate performance or failure of an essential mission function of the agency, a component of the agency, or another organization.

Most agencies reported that to identify high risk projects, staff from the Office of the Chief Information Officer compare the criteria against their current portfolio to determine which projects met OMB’s definition. They then submit the list to OMB for review. According to OMB and agency officials, after the submission of the initial list, examiners at OMB work with individual agencies to identify or remove projects as appropriate. According to most agencies, the final list is then approved by their Chief Information Officer.

For the identified high risk projects, beginning September 15, 2005, and quarterly thereafter, Chief Information Officers are to assess, confirm, and document projects’ performance. Specifically, agencies are required to determine, for each of their high risk projects, whether the project was meeting one or more of four performance evaluation criteria: establishing baselines with clear cost, schedule, and performance goals; maintaining the project’s cost and schedule variances within 10 percent; assigning a qualified project manager; and avoiding duplication by leveraging inter-agency and governmentwide investments.

If a high risk project meets any of these four performance evaluation criteria, agencies are instructed to document this using a standard template provided by OMB and provide this template to oversight authorities (e.g., OMB, agency inspectors general, agency management, and GAO) on request. Upon submission, according to OMB staff, individual analysts review the quarterly performance reports of projects with shortfalls to determine how well the projects are progressing and whether the actions described in the planned improvement efforts are adequate using other performance data already received on IT projects such as the e-Gov Scorecards, earned value management data, and the exhibit 300.

Poorly Planned and Performing Projects Total at Least $10 Billion in Estimated Expenditures for Fiscal Year 2008

OMB and federal agencies have identified approximately 227 IT projects— totaling at least $10.4 billion in expenditures for fiscal year 2008—as being poorly planned, poorly performing, or both. Figure 1 shows the distribution of these projects and their associated dollar values.

Hundreds of Projects Totaling Billions of Dollars Are Placed on the Management Watch List Annually

Each year, OMB places hundreds of projects totaling billions of dollars on the Management Watch List. Table 1 provides a historical perspective of the number of these projects and their associated budget since OMB started reporting on the Management Watch List in the President’s budget request for 2004. The table shows that while the number of projects and their associated budget have generally decreased since then, they increased by 83 projects this year, and represent a significant percentage of the total budget.

As of July 2007, 136 projects, representing $8.6 billion, still remained on the Management Watch List (see appendix 1 for complete list). We determined that 29 of these projects were on the Management Watch List as of September 2006.

Poorly Performing Projects Total About $6 Billion in Estimated Expenditures for Fiscal Year 2008

As of June 2007, when agencies last reported on their high risk projects to OMB, the 24 major agencies identified 438 IT projects as high risk, of which 124 had performance shortfalls collectively totaling about $6.0 billion in funding requested for fiscal year 2008. Table 2 shows that the number of projects, as well as the number of projects with shortfalls increased this year. OMB attributes this rise to increased management oversight by agencies.

The majority of projects were not reported to have had performance shortfalls. In addition, five agencies—the departments of Energy, Housing and Urban Development, Labor, and State, and the National Science Foundation—reported that none of their high risk projects experienced any performance shortfalls. Figure 2 illustrates the number of high risk projects by agency as of June 2007, with and without shortfalls.

Agencies reported cost and schedule variances that exceeded 10 percent as the greatest shortfall. This is consistent with what they reported about a year ago, and the distribution of shortfalls types is similar to last year. Figure 3 illustrates the reported number and type of performance shortfalls associated with high risk projects.

Appendix II identifies the shortfalls associated with each of the poorly performing projects.

Twenty-two high risk projects have experienced performance shortfalls for the past four quarters (see figure 4).

Of these projects, the following six have had shortfalls since the High Risk List was established in September 2005.

Department of Homeland Security’s (DHS) Secure Border Initiative Net Technology Program, which is expected to provide on-scene agents near real-time information on attempted border crossings by illegal aliens, terrorists, or smugglers; Department of Agriculture’s (USDA) Modernize and Innovate the Delivery of Agricultural Systems, which is intended to modernize the delivery of farm program benefits by deploying an internet-based self-service capabilities for customers, and eliminating the department’s reliance on aging technology and service centers as the sole means of delivering program benefits; Department of Veterans Affairs’ (VA) VistA Imaging, which should provide complete online patient data to health care providers, increase clinician productivity, facilitate medical decision-making, and improve quality of care; DHS’s Transportation Worker Identification Credentialing, which is to establish a system-wide common secure biometric credential, used by all transportation modes, for personnel requiring unescorted physical and/or logical access to secure areas of the transportation system; Department of Justice’s (DOJ) Regional Data Exchange, which is expected to combine and share regional investigative information and provide powerful tools for analyzing the integrated data sets; and VA’s Patient Financial Services System, which is expected create a comprehensive business solution for revenue improvement utilizing improved business practices, commercial software, and enhanced VA clinical applications.

Several Projects are Both Poorly Planned and Poorly Performing

Thirty-three projects are on both the Management Watch List and list of high risk projects with shortfalls, meaning that they are both poorly planned and poorly performing. They total about $4.1 billion in estimated expenditures for fiscal year 2008. These projects are listed in table 3 below.

OMB Has Taken Steps to Improve the Identification and Oversight of Management Watch List and High Risk Projects, but Additional Efforts Are Needed

OMB has taken steps to improve the identification and oversight of the Management Watch List and high risk projects by addressing some of the recommendations we previously made, but additional efforts are needed to more effectively perform these activities and ultimately ensure that potentially billions of taxpayer dollars are not wasted. Specifically, we previously recommended that OMB take action to improve the accuracy and reliability of exhibit 300s and application of the high risk projects criteria, and perform governmentwide tracking and analysis of Management Watch List and high risk project information. While OMB took steps to address our concerns, more can be done.

Exhibit 300s Are Now Reported Publicly but Their Accuracy and Reliability Issues Remain

In January 2006, we noted that the underlying support for information provided in the exhibit 300s was often inadequate and that, as a result, the Management Watch List may be undermined by inaccurate and unreliable data. Specifically, we noted that documentation either did not exist or did not fully agree with specific areas of all exhibit 300s; agencies did not always demonstrate that they complied with federal or departmental requirements or policies with regard to management and reporting processes; for example, no exhibit 300 had cost analyses that fully complied with OMB requirements for cost-benefit and cost- effectiveness analyses; and data for actual costs were unreliable because they were not derived from cost-accounting systems with adequate controls; in the absence of such systems, agencies generally derived cost information from ad hoc processes.

We recommended, among other things, that OMB direct agencies to improve the accuracy and reliability of exhibit 300 information.

To address our recommendation, in June 2006, OMB directed agencies to post their exhibit 300s on their website within two weeks of the release of the President’s budget request for fiscal year 2008. While this is a step in the right direction, the accuracy and reliability of exhibit 300 information is still a significant weakness among the 24 major agencies, as evidenced by a March 2007 President’s Council on Integrity and Efficiency and Executive Council on Integrity and Efficiency study commissioned by OMB to ascertain the validity of exhibit 300s. Specifically, according to individual agency reports contained within the study, Inspectors General found that the documents supporting agencies’ exhibit 300s continue to have accuracy and reliability issues. For example, according to these reports, the Agency for International Development did not maintain the documentation supporting exhibit 300s cost figures. In addition, at the Internal Revenue Service, the exhibit 300s were unreliable because, among other things, project costs were being reported inaccurately and progress on projects in development was measured inaccurately.

High Risk Criteria Are Being Applied More Consistently, but Questions Remain as to Whether All Projects Are Identified, Including Projects with Shortfalls

In June 2006, we noted that OMB did not always consistently apply the criteria for identifying high risk projects. For example, we identified projects that appeared to meet the criteria but that were not designated as high risk. Accordingly, we recommended that OMB apply their high risk criteria consistently. OMB has since designated as high risk the projects that we identified. Further, OMB officials stated that they have worked with agencies to ensure a more consistent application of the high risk criteria. These are positive steps, as they result in more projects receiving the management attention they deserve.

However, questions remain as to whether all high risk projects with shortfalls are being reported by agencies. For example, we have reported in our high risk series that the Department of Defense’s efforts to modernize its business systems have been hampered because of weaknesses in practices for (1) developing and using an enterprise architecture, (2) instituting effective investment management processes, and (3) establishing and implementing effective systems acquisition processes. We concluded that the department remains far from where it needs to be to effectively and efficiently manage an undertaking of such size, complexity, and significance. Despite these problems, Department of Defense (DOD), which accounts for $31 billion of the government’s $65 billion in IT expenditures, only reported three projects as being high risk with shortfalls representing a total of about $1 million. The dollar value of DOD’s three projects represents less than one tenth of one percent of high risk projects with shortfalls. In light of the problems we and others have identified with many of DOD’s projects, this appears to be an underestimation. Given the critical nature of high risk projects, it is particularly important to identify early on those that are performing poorly, before their shortfalls become overly costly to address.

Management Watch List and High Risk Projects Made Public, but Governmentwide Analyses Still Not Performed

Finally, to improve the oversight of the Management Watch List projects, we recommended in our April 2005 report that the Director of OMB report to Congress on projects’ deficiencies, agencies’ progress in addressing risks of major IT investments, and management areas needing attention. In addition, to fully realize the potential benefits of using the Management Watch List, we recommended that OMB use the list as the basis for selecting projects for follow-up, tracking follow-up activities and analyze the prioritized list to develop governmentwide and agency assessments of the progress and risks of IT investments, identifying opportunities for continued improvement. We also made similar recommendations to the Director of OMB regarding high risk projects. Specifically, we recommended that OMB develop a single aggregate list of high risk projects and their deficiencies and use that list to report to Congress progress made in correcting high risk problems, actions under way, and further actions that may be needed.

To its credit, OMB started publicly releasing aggregate lists of the Management Watch List and high risk projects in September 2006, and has been releasing updated versions on a quarterly basis by posting them on their website. While this is a positive step, OMB does not publish the specific reasons that each project is placed on the Management Watch List, nor does it specifically identify why high risk projects are poorly performing, as we have done in appendix II. Providing this information would allow OMB and others to better analyze the reasons projects are poorly planned and performing and take corrective actions and track these projects on a governmentwide basis. Such information would also help to highlight progress made by agencies or projects, identify management issues that transcend individual agencies, and highlight the root causes of governmentwide issues and trends. Such analysis would be valuable to agencies in planning future IT projects, and could enable OMB to prioritize follow-up actions and ensure that high-priority deficiencies are addressed.

In summary, the Management Watch List and high risk projects processes play important roles in improving the management of federal IT investments by helping to identify poorly planned and poorly performing projects that require management attention. As of June 2007, the 24 major agencies had 227 such projects totaling at least $10 billion. OMB has taken steps to improve the identification of these projects, including implementing recommendations related to improving the accuracy of exhibit 300s and the application of the high risk projects criteria. However, the number of projects may be understated because issues concerning the accuracy and reliability of the budgetary documents the Management Watch List is derived from still remain and high risk projects with shortfalls may not be consistently identified.

While OMB can act to further improve the identification and oversight of poorly planned and poorly performing projects, we recognize that agencies must also take action to fulfill their responsibilities in these areas. We have addressed this in previous reports and made related recommendations. Until further improvements in the identification and oversight of poorly planned and poorly performing IT projects, potentially billions in taxpayer dollars are at risk of being wasted.

GAO Contact and Acknowledgments

If you should have any questions about this testimony, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Individuals who made key contributions to this testimony are Sabine Paul, Assistant Director; Neil Doherty; Amos Tevelow; Kevin Walsh and Eric Winter.

Appendix I: Management Watch List Projects

The following provides additional detail on the investments comprising OMB’s Management Watch List as of July 2007. Under the Clinger-Cohen Act of 1996, agencies are required to submit business plans for IT investments to OMB. If the agency’s investment plan contains one or more planning weaknesses, it is placed on OMB’s Management Watch List and targeted for follow-up action to correct potential problems prior to execution.

We estimated the fiscal year 2008 request based on the data in the Report on IT Spending for Fiscal Years 2006, 2007, and 2008 (generally referred to as exhibit 53), and data provided by agencies.

Appendix II: High Risk Projects with Shortfalls

The following provides additional detail on the high risk projects that have performance shortfalls as of June 2007.

We estimated the fiscal year 2008 request based on the data in the Report on IT Spending for Fiscal Years 2006, 2007, and 2008 (generally referred to as exhibit 53), and data provided by agencies.