Information Technology: Improvements Needed to More Accurately Identify and Better Oversee Risky Projects Totaling Billions of Dollars
Released: 2006-09-07T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much the government plans to spend for IT and how these funds are to be allocated. Federal IT spending has risen to an estimated $64 billion in fiscal year 2007.

OMB plays a key role in overseeing federal IT investments and how they are managed. To drive improvement in the implementation and management of IT projects, Congress enacted the Clinger-Cohen Act in 1996 to further expand the responsibilities of OMB and the agencies under the Paperwork Reduction Act. In particular, the act requires agency heads, acting through agency chief information officers (CIOs), to, among other things, better link their IT planning and investment decisions to program missions and goals and to implement and enforce IT management policies, procedures, standards, and guidelines. The Clinger-Cohen Act requires that agencies engage in capital planning and performance and results- based management. The act also requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. OMB is also required to report to Congress on the net program performance benefits achieved as a result of major capital investments in information systems that are made by executive agencies.

In response to the Clinger-Cohen Act and other statutes, OMB developed policy for planning, budgeting, acquisition, and management of federal capital assets. This policy is set forth in OMB Circular A-11 (section 300) and in OMB’s Capital Programming Guide (supplement to Part 7 of Circular A-11), which directs agencies to develop, implement, and use a capital programming process to build their capital asset portfolios. Among other things, OMB’s Capital Programming Guide directs agencies to ● evaluate and select capital asset investments that will support core mission functions that must be performed by the federal government and demonstrate projected returns on investment that are clearly equal to or better than alternative uses of available public resources; institute performance measures and management processes that monitor actual performance and compare to planned results; and ● establish oversight mechanisms that require periodic review of operational capital assets to determine how mission requirements might have changed and whether the asset continues to fulfill mission requirements and deliver intended benefits to the agency and customers.

To further support the implementation of IT capital planning practices, we have developed an IT investment management framework that agencies can use in developing a stable and effective capital planning process, as required by statute and directed in OMB’s Capital Programming Guide. Consistent with the statutory focus on selecting, controlling, and evaluating investments, this framework focuses on these processes in relation to IT investments specifically. It is a tool that can be used to determine both the status of an agency’s current IT investment management capabilities and the additional steps that are needed to establish more effective processes. Mature and effective management of IT investments can vastly improve government performance and accountability. Without good management, such investments can result in wasteful spending and lost opportunities for improving delivery of services to the public.

Prior Reviews on Federal IT Investment Management Have Identified Weaknesses

Only by effectively and efficiently managing their IT resources through a robust investment management process can agencies gain opportunities to make better allocation decisions among many investment alternatives and further leverage their investments. However, the federal government faces enduring IT challenges in this area. For example, in January 2004 we reported on mixed results of federal agencies’ use of IT investment management practices. Specifically, we reported that although most of the agencies had IT investment boards responsible for defining and implementing the agencies’ IT investment management processes, agencies did not always have important mechanisms in place for these boards to effectively control investments, including decision- making rules for project oversight, early warning mechanisms, and/or requirements that corrective actions for underperforming projects be agreed upon and tracked. Executive-level oversight of project-level management activities provides organizations with increased assurance that each investment will achieve the desired cost, benefit, and schedule results. Accordingly, we made several recommendations to agencies to improve their practices.

In previous work using our investment management framework, we reported that the use of IT investment management practices by agencies was mixed. For example, a few agencies that have followed the framework in implementing capital planning processes have made significant improvements. In contrast, however, we and others have continued to identify weaknesses at agencies in many areas, including immature management processes to support both the selection and oversight of major IT investments and the measurement of actual versus expected performance in meeting established performance measures.

OMB’s Management Watch List Intended to Correct Project Weaknesses and Business Case Deficiencies

In helping to ensure that investments of public resources are justified and that public resources are wisely invested, OMB began using the Management Watch List, in the President’s fiscal year 2004 budget request, as a means to oversee the justification for and planning of agencies’ IT investments. This list was derived based on a detailed review of the investments’ Capital Asset Plan and Business Case, also known as the exhibit 300.

The exhibit 300 is a reporting mechanism intended to enable an agency to demonstrate to its own management, as well as OMB, that a major project is well planned in that it has employed the disciplines of good project management; developed a strong business case for the investment; and met other Administration priorities in defining the cost, schedule, and performance goals proposed for the investment.

We reported in 2005 that OMB analysts evaluate agency exhibit 300s by assigning scores to each exhibit 300 based on guidance presented in OMB Circular A-11. As described in this circular, the scoring of a business case consists of individual scoring for 10 categories, as well as a total composite score of all the categories. The 10 categories are ● project (investment) management, ● security and privacy, ● performance-based management system (including the earned value management system), life-cycle costs formulation, and ● support of the President’s Management Agenda.

Using these scores, projects were placed on the Management Watch List if their exhibit 300 business case received a total composite score of 3 or less, or if it received a score of 3 or less in the areas of performance goals, performance-based management systems, or security and privacy, even if its overall score was a 4 or 5. To derive the total number of projects on the list that were reported for fiscal year 2005, OMB polled the individual analysts and compiled the numbers.

According to OMB, agencies with weaknesses in these three areas were to submit remediation plans addressing the weaknesses. OMB officials also stated that decisions on follow-up and monitoring the progress were typically made by staff with responsibility for reviewing individual agency budget submissions, depending on the staff’s insights into agency operations and objectives. According to OMB officials, those Management Watch List projects that did receive specific follow-up attention received feedback through the passback process, through targeted evaluation of remediation plans designed to address weaknesses, and through the apportioning of funds so that the use of budgeted dollars was conditional on appropriate remediation plans being in place, and through the quarterly e-Gov Scorecards.

OMB Issued August 2005 Memorandum on Improving Performance of High Risk IT Projects

To improve IT project execution, OMB issued a memorandum in August 2005 to all federal CIOs, directing them to begin taking steps to identify IT projects that are high risk and to report quarterly on their performance. As originally defined in OMB Circular A-11 and subsequently reiterated in the August 2005 memorandum, high risk projects are those that require special attention from oversight authorities and the highest levels of agency management because of one or more of the following four reasons: ● The agency has not consistently demonstrated the ability to manage complex projects. ● The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio. ● The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization. ● Delay or failure of the project would introduce for the first time unacceptable or inadequate performance or failure of an essential mission function of the agency, a component of the agency, or another organization.

As directed in the memorandum, agencies are to work with OMB to identify their high risk IT projects using these criteria. Most agencies reported that, to identify high risk projects, CIO office staff compared the criteria against their current portfolio to determine which projects met OMB’s definition. They then submitted the list to OMB for review. According to OMB and agency officials, after the submission of the initial list, examiners at OMB worked with individual agencies to identify or remove projects as appropriate. According to most agencies, the final list was then approved by their CIO.

For the identified high risk projects, beginning September 15, 2005, and quarterly thereafter, CIOs were to assess, confirm, and document projects’ performance. Specifically, agencies were required to determine, for each of their high risk projects, whether the project was meeting one or more of four performance evaluation criteria: (1) establishing baselines with clear cost, schedule, and performance goals; (2) maintaining the project’s cost and schedule variances within 10 percent; (3) assigning a qualified project manager; and (4) avoiding duplication by leveraging inter-agency and governmentwide investments. If a high risk project met any of these four performance evaluation criteria, agencies were instructed to document this using a standard template provided by OMB and provide this template to oversight authorities (e.g., OMB, agency inspectors general, agency management, and GAO) on request. Upon submission, according to OMB staff, individual analysts review the quarterly performance reports of projects with shortfalls to determine how well the projects are progressing and whether the actions described in the planned improvement efforts are adequate using other performance data already received on IT projects such as the e-Gov Scorecards, earned value management data, and the exhibit 300.

Poorly Planned and Performing Projects Identified, Totaling About $12 Billion in Estimated Expenditures for Fiscal Year 2007

About 300 projects totaling about $12 billion in estimated IT expenditures for fiscal year 2007 have been placed on OMB’s Management Watch List or as a high risk project with performance shortfalls. Specifically, the President’s budget for fiscal year 2007 included 857 major IT projects, totaling approximately $64 billion. Of this, OMB reported that there were 263 proposed major projects that were poorly planned, totaling $10 billion. In addition, agencies reported that 79 of the 226 high-risk projects identified as of March 2006, collectively totaling about $2.2 billion had a performance shortfall primarily associated with cost and schedule variances that exceeded 10 percent.

OMB first reported on the Management Watch List in the President’s budget request for 2004. While the number of projects and their associated budget have decreased since then, they still represent a significant percentage of the total IT budget. Table 1 shows the budget information for projects on the Management Watch List for fiscal years 2004, 2005, 2006, and 2007.

Table 2 provides the number of projects on the Management Watch List for fiscal years 2004, 2005, 2006, and 2007.

In addition, in response to OMB’s August 2005 memorandum, the 24 major agencies identified 226 IT projects as high risk, totaling about $6.4 billion in funding requested for fiscal year 2007. Agencies identified most projects as high risk because their delay or failure would impact the essential business functions of the agency. In addition, agencies reported that about 35 percent of the high risk projects—or 79 investments, collectively totaling about $2.2 billion in fiscal year 2007—had a performance shortfall, primarily associated with cost and schedule variances that exceeded 10 percent.

Figure 1 illustrates the number of agency high risk projects as of March 2006 with and without shortfalls. The majority of the agencies reported that their high risk projects did not have performance shortfalls in any of the four areas identified by OMB. In addition, six agencies—the departments of Commerce, Energy, Housing and Urban Development, and Labor, and the National Aeronautics and Space Administration and the National Science Foundation— reported that none of their high risk projects experienced any performance shortfalls.

Improvements Needed to Identify and Oversee Management Watch List and High Risk Projects

While the Management Watch List and high risk processes serve to highlight poorly planned and performing projects and focus attention on them, our reviews identified opportunities to strengthen the identification and oversight of projects for each.

Management Watch List May Be Based on Unreliable Data and High Risk Project Criteria Are Not Always Consistently Applied

OMB’s Management Watch List may be undermined by inaccurate and unreliable data. While OMB uses the exhibit 300s as the basis for designating projects as poorly planned, we have recently reported that the underlying support was often inadequate for information provided in the exhibit 300s GAO reviewed. Three general types of weaknesses were evident: ● All exhibit 300s had documentation weaknesses. Documentation either did not exist or did not fully agree with specific areas of the exhibit 300. ● Agencies did not always demonstrate that they complied with federal or departmental requirements or policies with regard to management and reporting processes. Also, none had cost analyses that fully complied with OMB requirements for cost-benefit and cost-effectiveness analyses. In contrast, most investments did demonstrate compliance with information security planning and training requirements. ● In sections that required actual cost data, these data were unreliable because they were not derived from cost-accounting systems with adequate controls. In the absence of such systems, agencies generally derived cost information from ad hoc processes.

Moreover, although agencies, with OMB’s assistance, generally identified their high risk projects using criteria specified by OMB, these criteria were not always consistently applied. ● In several cases, agencies did not use OMB’s criteria to identify high risk projects. Some agencies reported using other reasons to identify a total of 31 high risk projects. For example, the Department of Homeland Security reported investments that were high risk because they had weaknesses associated with their business cases based on the evaluation by OMB. The Department of Transportation reported projects as high risk because two did not have approved baselines, and four had incomplete or poor earned value management assessments. ● Regarding the criterion for high risk designation that the agency has not consistently demonstrated the ability to manage complex projects, only three agencies reported having projects meeting this criterion. This appears to be somewhat low, considering that we and others have previously reported on weaknesses in numerous agencies’ ability to manage complex projects. For example, we have reported in our high risk series on major programs and operations that need urgent attention and transformation in order to ensure that our national government functions in the most economical, efficient, and effective manner possible. Specifically, the Department of Defense’s efforts to modernize its business systems have been hampered because of weaknesses in practices for (1) developing and using an enterprise architecture, (2) instituting effective investment management processes, and (3) establishing and implementing effective systems acquisition processes. We concluded that the Department of Defense, as a whole, remains far from where it needs to be to effectively and efficiently manage an undertaking with the size, complexity, and significance of its departmentwide business systems modernization. We also reported that, after almost 25 years and $41 billion, efforts to modernize the air traffic control program of the Federal Aviation Administration, the Department of Transportation’s largest component, are far from complete and that projects continue to face challenges in meeting cost, schedule, and performance expectations. However, neither the Department of Defense nor the Department of Transportation cited the “inability to consistently manage complex projects” criteria for any projects as being high risk. ● Finally, while agencies have reported a significant number of IT projects as high risk, we identified other projects on which we have reported and testified that appear to meet one or more of OMB’s criteria for high risk designation including high development or operating costs and recognized deficiencies in the adequate performance but were not identified as high risk. Examples we have recently reported include the following projects: ● The Decennial Response Integration System of the Census Bureau, is intended to integrate paper, Internet, and telephone responses. Its high development and operating costs are expected to make up a large portion of the $1.8 billion program to develop, test, and implement decennial census systems. In March 2006, we testified that the component agency has established baseline requirements for the acquisition, but the bureau has not yet validated them or implemented a process for managing the requirements. We concluded that, until these and other basic contract management activities are fully implemented, this project faced increased risks that the system would experience cost overruns, schedule delays, and performance shortfalls.

System—an initiative managed by the Department of Commerce, the Department of Defense, and the National Aeronautics and Space Administration—is to converge two satellite programs into a single satellite program capable of satisfying both civilian and military requirements. In November 2005, we reported that the system was a troubled program because of technical problems on critical sensors, escalating costs, poor management at multiple levels, and the lack of a decision on how to proceed with the program. Over the last several years, this system has experienced continual cost increases to about $10 billion and schedule delays, requiring difficult decisions about the program’s direction and capabilities. More recently, we testified that the program is still in trouble and that its future direction is not yet known. While the program office has corrective actions under way, we concluded that, as the project continues, it will be critical to ensure that the management issues of the past are not repeated. ● Rescue 21, is a planned coastal communications system of the Department of Homeland Security. We recently reported that inadequacies in several areas contributed to Rescue 21 cost overruns and schedule delays. These inadequacies occurred in requirements management, project monitoring, risk management, contractor cost and schedule estimation and delivery, and executive level oversight. Accordingly, the estimated total acquisition cost has increased from $250 million in 1999 to $710.5 million in 2005, and the timeline for achieving full operating capability has been extended from 2006 to 2011.

For the projects we identified as appearing to meet OMB’s criteria for high risk, the responsible agencies reported that they did not consider these investments to be high risk projects for such reasons as (1) the project was not a major investment; (2) agency management is experienced in overseeing projects; or (3) the project did not have weaknesses in its business case. In particular, one agency stated that their list does not include all high risk projects, only those that are the highest priority of the high risk investments. However, none of the reasons provided are associated with OMB’s high risk definition. Without consistent application of the criteria, OMB and executives cannot have the assurance that all projects that require special attention have been identified.

OMB Does Not Use an Aggregate List to Perform Its Oversight of the Management Watch List or High Risk Projects

While OMB’s Management Watch List identified opportunities to strengthen investments and promote improvements in IT management, OMB did not develop a single, aggregate list identifying the projects and their weaknesses. According to OMB officials, they did not construct a single list of projects meeting their watch list criteria because they did not see such an activity as necessary in performing OMB’s predominant mission: to assist in overseeing the preparation of the federal budget and to supervise agency budget administration. Thus, OMB did not exploit the opportunity to use the list as a tool for analyzing IT investments on a govermentwide basis, limiting its ability to identify and report on the full set of IT investments requiring corrective actions.

In addition, while OMB asked agencies to take corrective actions to address weaknesses associated with projects on the Management Watch List, it did not develop a structured, consistent process or criteria for deciding how to follow up on these actions. We also reported that because it did not consistently monitor the follow-up performed, OMB could not tell us which of the 621 projects identified on the fiscal year 2005 list received follow-up attention, and it did not know whether the specific project risks that it identified through its Management Watch List were being managed effectively. This approach could leave resources at risk of being committed to poorly planned and managed projects. Thus, OMB was not using its Management Watch List as a tool for improving IT investments on a governmentwide basis and focusing attention where it was most needed.

Similar to the Management Watch List, we reported in June 2006 that while OMB analysts review the quarterly performance reports on high risk projects, they did not compile a single aggregate list of high risk projects. According to OMB staff they did not see such an activity as necessary in achieving the intent of the guidance—to improve project planning and execution. Consistent with our Management Watch list observations and recommendations, we believe that by not having a single list, OMB is limiting its ability to identity and report on the full set of IT investments across the federal government that require special oversight and greater agency management attention.

Implementation of Recommendations Can Lead to Improved Processes to Identify and Oversee Management Watch List and High Risk Projects

To address our key findings, we made several recommendations to the Director of OMB. For example, to improve how the Management Watch List projects are identified, we have made several recommendations to improve the accuracy and validity of exhibit 300s for major IT investments, including that the Director require agencies to determine the extent to which the information contained in each exhibit 300 is accurate and reliable, and, where weaknesses in accuracy and reliability are identified, disclose them and explain the agency’s approach to mitigating them. We also recommended that the Director provide for training of agency personnel responsible for completing exhibit 300s, and specified that, in developing the training, OMB consult with agencies to identify deficiencies that the training should address. Likewise, to improve how high risk projects are identified, we recommended that the Director direct federal agency CIOs to ensure that they are consistently applying the high risk criteria defined by OMB.

To improve how the Management Watch List is provided oversight, in our April 2005 report, we recommended that the Director of OMB develop a central list of projects and their deficiencies and report to Congress on progress made in addressing risks of major IT investments and management areas needing attention. In addition, to fully realize the potential benefits of using the Management Watch List, we recommended that OMB use the list as the basis for selecting projects for follow-up, tracking follow-up activities and analyze the prioritized list to develop governmentwide and agency assessments of the progress and risks of IT investments, identifying opportunities for continued improvement. We also made similar recommendations to the Director of OMB regarding high risk projects. Specifically, we recommended that OMB develop a single aggregate list of high risk projects and their deficiencies and use that list to report to Congress progress made in correcting high risk problems, actions under way, and further actions that may be needed.

OMB generally disagreed with our recommendations for strengthening the Management Watch List and high risk projects processes. Specifically, OMB’s Administrator of the Office of E- Government and Information Technology stated that the ultimate responsibility to improve the accuracy and reliability of the exhibit 300s lies with the agencies. While this is true, OMB also has statutory responsibility for providing IT guidance governmentwide, especially when it involves an OMB-required budget document. Regarding the consistent application of the high risk criteria, the Administrator stated that some flexibility in the application of the criteria is essential. While some flexibility may be appropriate, we believe that these criteria should be more consistently applied so that projects that clearly meet them are identified and provided oversight. The Administrator also disagreed with our recommendations that an aggregated governmentwide Management Watch List and high risk project list is necessary to perform adequate oversight. However, we continue to believe that these lists are needed to facilitate OMB’s ability to track progress. Addressing these recommendations would provide increased assurance that poorly planned and performing projects are accurately identified and more effectively provided oversight. ------------------------------- In summary, the Management Watch List and High Risk processes play important roles in improving the management of federal IT investments by helping to identify poorly planned and performing projects totaling at least $12 billion that require management attention. However, the number of projects identified on both lists is likely understated because the Management Watch List is derived from budgetary documents that are not always accurate and reliable and the high risk projects are not always identified consistently using OMB criteria. In addition, we noted areas where oversight of both sets of projects could be strengthened primarily by reporting the results in the aggregate so that governmentwide analyses can be performed, progress can be tracked, and Congress can be informed. The recommendations we made to agencies and OMB to address these issues are aimed at providing greater assurance that poorly planned and performing projects are more accurately identified and receiving adequate oversight, and ultimately ensuring that potentially billions of taxpayers dollars are not wasted.

Contacts and Acknowledgements

If you should have any questions about this testimony, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Other individuals who made key contributions to this testimony are Sabine Paul and Niti Tandon.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.