Information Technology: Agencies and OMB Should Strengthen Processes for Identifying and Overseeing High Risk Projects
Released: 2006-07-17T00:00:00

Letter

Background

Each year, OMB and federal agencies work together to determine how much government plans to spend for IT and how these funds are to be allocated. Over the past decade, federal IT spending has risen to an estimated $64 billion in fiscal year 2007.

OMB plays a key role in overseeing these IT investments and how they are managed, stemming from its predominant mission: to assist the President in overseeing the preparation of the federal budget and to supervise budget administration in Executive Branch agencies. In helping to formulate the President’s spending plans, OMB is responsible for evaluating the effectiveness of agency programs, policies, and procedures; assessing competing funding demands among agencies; and setting funding priorities. OMB also ensures that agency reports, rules, testimony, and proposed legislation are consistent with the President’s budget and with administration policies. In carrying out these responsibilities, OMB depends on agencies to collect and report accurate and complete information; these activities depend, in turn, on agencies having effective IT management practices.

To drive improvement in the implementation and management of IT projects, Congress enacted the Clinger-Cohen Act in 1996 to further expand the responsibilities of OMB and the agencies under the Paperwork Reduction Act. In particular, the act requires agency heads, acting through agency chief information officers (CIO), to, among other things, better link their IT planning and investment decisions to program missions and goals and to implement and enforce IT management policies, procedures, standards, and guidelines. OMB is required by the Clinger-Cohen Act to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. OMB is also required to report to Congress on the net program performance benefits achieved as a result of major capital investments in information systems that are made by executive agencies.

OMB is aided in its responsibilities by the Chief Information Officers Council as described by the E-Government Act of 2002. The council is designated the principal interagency forum for improving agency practices related to the design, acquisition, development, modernization, use, operation, sharing, and performance of federal government information resources. Among the specific functions of the CIO Council are the development of recommendations for the Director of OMB on government information resources management policies and requirements and the sharing of experiences, ideas, best practices, and innovative approaches related to information resources management.

Prior Review on Governmentwide IT Investment Management Has Identified Weaknesses

Only by effectively and efficiently managing their IT resources through a robust investment management process can agencies gain opportunities to make better allocation decisions among many investment alternatives and further leverage their investments. However, the federal government faces enduring IT challenges in this area. For example, in January 2004 we reported on mixed results of federal agencies’ use of IT investment management practices. Specifically, we reported that although most of the agencies had IT investment boards responsible for defining and implementing the agencies’ IT investment management processes, no agency had fully implemented practices for monitoring the progress of its investments. Executive-level oversight of project-level management activities provides organizations with increased assurance that each investment will achieve the desired cost, benefit, and schedule results. Accordingly, we made several recommendations to agencies to improve their practices.

OMB’s Management Watch List Intended to Correct Project Weaknesses and Business Case Deficiencies

In carrying out its responsibilities to assist the President in overseeing the preparation of the federal budget, OMB reported in the President’s fiscal year 2004 budget that there were 771 IT investment projects on what was called the At-Risk List (later referred to as the Management Watch List). This list included mission-critical projects that did not successfully demonstrate sufficient potential for success based on the agency Capital Asset Plan and Business Case, also known as the exhibit 300, or did not adequately address IT security. To identify projects for inclusion on the Management Watch List, OMB used scoring criteria contained in OMB Circular A-11 that the agency established for evaluating the justifications for funding that federal agencies submitted for major investments and for ensuring that agency planning and management of capital assets is consistent with OMB policy and guidance. This evaluation is carried out as part of OMB’s responsibility to help ensure that investments of public resources are justified and that public resources are wisely invested.

In presenting the fiscal year 2005 budget, OMB reported that there were 621 major projects on the Management Watch List, consisting of mission- critical projects that needed to improve performance measures, project management, and IT security. OMB staff described this assessment as again being based on evaluations of the exhibit 300s that agencies submitted to justify project funding. Agencies were required to successfully correct identified project weaknesses and business case deficiencies; otherwise, they risked OMB’s placing limits on their spending.

In April 2005, we reported on OMB’s development of its Management Watch List. We concluded that OMB’s scoring of the exhibit 300s addressed many critical IT management areas and promoted the improvement of investments. However, because OMB did not compile a single aggregate list and had not developed a structured, consistent process for deciding how to follow up on corrective actions being taken by the agencies, the agency missed the opportunity to use its scoring process more effectively to identify management issues that transcended individual agencies, to prioritize follow-up actions, and to ensure that high- priority deficiencies were addressed. To take advantage of this potential benefit, we recommended that OMB compile a single aggregate list and use the list as the basis for selecting projects for follow up and for tracking follow-up activities by developing specific criteria for prioritizing the IT projects included on the list.

OMB has continued to report on its Management Watch List in the most recent President’s budget request. Table 1 shows the budget information for projects on the Management Watch List for fiscal years 2004, 2005, 2006, and 2007.

Table 2 shows the number of projects on the Management Watch List for fiscal years 2004, 2005, 2006, and 2007.

OMB’s August 2005 Memorandum on Improving Performance of High Risk IT Projects

To continue improving IT project planning and execution, OMB issued a memorandum in August 2005 to all federal chief information officers, directing them to begin taking steps to identify IT projects that are high risk and to report quarterly on their performance. As originally defined in OMB Circular A-11 and subsequently reiterated in the August 2005 memorandum, high risk projects are those that require special attention from oversight authorities and the highest levels of agency management because of one or more of the following four reasons: The agency has not consistently demonstrated the ability to manage complex projects.

The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

Delay or failure of the project would introduce for the first time unacceptable or inadequate performance or failure of an essential mission function of the agency, a component of the agency, or another organization.

As directed in the memorandum, by August 15, 2005, agencies in collaboration with OMB were required to initially identify their high risk IT projects using these criteria. In addition, OMB subsequently provided additional instructions through e-mails to agencies. Through these instructions, OMB directed agencies to declare all e-government and line of business (LOB) initiatives managed by their agency as high risk. In addition, the instructions specified that partner agencies consider investments associated with migrations to an e-government or LOB initiative as high risk until they have completed migration or OMB determines they should no longer be designated as high risk.

For the identified high risk projects, beginning September 15, 2005, and quarterly thereafter, CIOs were to assess, confirm, and document projects’ performance. Specifically, agencies were required to determine, for each of their high risk projects, whether the project was meeting one or more of four performance evaluation criteria: (1) establishing baselines with clear cost, schedule, and performance goals; (2) maintaining the project’s cost and schedule variances within 10 percent; (3) assigning a qualified project manager; and (4) avoiding duplication by leveraging inter-agency and governmentwide investments. If a high risk project meets these four performance evaluation criteria, agencies are instructed to document this using a standard template provided by OMB and provide this template to oversight authorities (e.g., OMB, agency inspectors general, agency management, and GAO) on request.

If any of the identified high risk projects have performance shortfalls, meaning that the project did not meet one or more of the four performance evaluation criteria, agencies are required to document the information on these projects on the standard template and provide it to OMB along with copies to the agency inspector general. For each of these projects, agencies must specify, using the template, (1) the specific performance shortfalls, (2) the specific cause of the shortfall, (3) a plan of action and milestones actions needed to correct each shortfall, and (4) the amount and source of additional funding needed to improve performance.

Federal Agencies Identified 226 Projects as High Risk

In response to OMB’s August 2005 memorandum, as of March 2006, the 24 CFO agencies identified 226 IT projects as high risk, totaling about $6.4 billion and representing about 10 percent of the President’s total IT budget request for fiscal year 2007. According to the agencies, these projects were identified as such mainly because of one or more of the four reasons provided in OMB’s memorandum. About 70 percent of the projects identified were reported as high risk because their delay or failure would impact the agency’s essential business functions. Moreover, about 35 percent of the high risk projects—or 79 investments, totaling about $2.2 billion in fiscal year 2007 planned funding, were reported as having performance shortfalls primarily because of cost and schedule variances exceeding 10 percent.

High Risk Projects Identified Total About $6.4 Billion for Fiscal Year 2007

As of March 2006, the 24 CFO agencies identified 226 IT investments as high risk. Collectively, five agencies—the Small Business Administration, National Aeronautics and Space Administration, Office of Personnel Management, and the Departments of Veterans Affairs and Homeland Security—identified about 100 of these projects.

According to the President’s most recent budget, about $6.4 billion has been requested for fiscal year 2007 by the 24 CFO agencies for the 226 high risk projects. Five of these agencies—the Departments of Defense, Homeland Security, Transportation, Veterans Affairs, and Justice, account for about 70 percent of the total high risk budget, totaling about $4.5 billion. Table 3 shows the number of high risk projects and associated funding reported by each of the 24 CFO agencies.

Most Projects Reported as High Risk Because Their Delay or Failure Could Impact Mission Performance

Agencies reported 195 of the 226 projects as meeting one or more of the reasons defined by OMB. Specifically, more than half of the agencies reported that their IT projects were identified as high risk because delay or failure of the project would result in inadequate performance or failure of an essential mission function. About one fourth of the projects were determined to be high risk because of high development, operating, or maintenance costs. In addition, three agencies identified 11 projects as high risk because of the inability to manage complex projects. Table 4 summarizes the OMB reasons for high risk designations.

A total of 31 projects were identified as high risk using rationale other than OMB’s four criteria. In these cases, agencies reasons included that the business cases had weaknesses or approved baselines were not established.

Agencies Identified 79 Projects with Performance Shortfalls

Agencies identified about 35 percent of the high risk projects as having performance shortfalls. Specifically, for the last reporting quarter—March 2006—agencies identified 79 investments, totaling about $2.2 billion in fiscal year 2007 planned funding, as having performance shortfalls. The most frequent reason provided for the shortfalls was cost and schedule variances exceeding 10 percent. By contrast, only two projects were reported by agencies as having an overlapping or duplicative IT investment.

Since September 2005, the number of projects with performance shortfalls has increased—from 58 projects in September 2005 to 67 projects in December 2005 to the 79 in March 2006. For the September and December 2005 and March 2006 reporting periods, figure 1 illustrates that agencies have reported that most of the weaknesses were in cost and schedule variances not within 10 percent and that there was an increase in projects that do not have clear baseline information on cost, schedule, and performance goals.

Figure 2 illustrates the number of agency high risk projects with and without shortfalls as of March 2006. The majority of the agencies reported that their high risk projects did not have performance shortfalls in any of the four areas identified by OMB. In addition, six agencies—the departments of Commerce, Energy, Housing and Urban Development, and Labor, and the National Aeronautics and Space Administration and the National Science Foundation—reported that none of their high risk projects experienced any performance shortfalls.

For the identification of all high risk projects by agency including funding, reasons for the high risk designation, specific performance shortfalls, and planned improvement efforts, see appendix III.

Processes Exist to Identify and Oversee High Risk Projects, but Opportunities Exist to Improve These Processes

Although agencies, with OMB’s assistance, generally identified their high risk projects by evaluating their IT portfolio against the four criteria specified by OMB, the criteria were not always consistently applied. In addition, OMB did not define a process for updating the list. To oversee high risk projects, agencies reported having investment management practices in place; however, we have previously reported on agencies’ maturing investment management processes and have made several recommendations to improve them. OMB staff perform their oversight of high risk projects by reviewing the quarterly performance reports, but they do not have a single aggregate list to analyze projects and for tracking progress on a governmentwide basis. Unless they address the issues regarding the identification, update, and oversight of high risk projects, OMB and agencies could be missing opportunities to perform these activities more effectively.

High Risk Projects Identified Primarily Using OMB’s Criteria, but Criteria Not Always Consistently Applied

Agencies primarily used the criteria defined in OMB’s August 2005 memorandum in determining the initial list of high risk projects; however, the criteria were not always consistently applied. Specifically, most agencies reported that officials from the Office of the CIO compared the criteria against their current portfolio to determine which projects met OMB’s definition. They then submitted the list to OMB for review. According to OMB and agency officials, after the submission of the initial list, examiners at OMB worked with individual agencies to identify or remove projects as appropriate. According to most agencies, the final list was then approved by their CIO.

However, OMB’s criteria for identifying high risk projects were not always consistently applied.

In several cases, agencies did not use OMB’s criteria to identify high risk projects. As previously discussed, some agencies reported using other reasons to identify a total of 31 high risk projects. For example, the Department of Homeland Security reported investments that were high risk because they had weaknesses associated with their business cases based on the evaluation by OMB. The Department of Transportation reported projects as high risk because two did not have approved baselines, and four had incomplete or poor earned value management (EVM) assessments.

Regarding the first criterion for high risk designation—the agency has not demonstrated the ability to manage complex projects—only three agencies reported having projects meeting this criterion. This appears to be somewhat low, considering that we and others have previously reported on weaknesses in numerous agencies’ ability to manage complex projects. For example, we have reported in our high risk series on major programs and operations that need urgent attention and transformation in order to ensure that our federal government functions in the most economical, efficient, and effective manner possible. Specifically, the Department of Defense’s efforts to modernize its business systems have been hampered because of weaknesses in practices for (1) developing and using an enterprise architecture, (2) instituting effective investment management processes, and (3) establishing and implementing effective systems acquisition processes. We concluded that the Department of Defense, as a whole, remains far from where it needs to be to effectively and efficiently manage an undertaking with the size, complexity, and significance of its departmentwide business systems modernization. We also reported that, after almost 25 years and $41 billion, efforts to modernize the air traffic control program of the Federal Aviation Administration, the Department of Transportation’s largest component, are far from complete and that projects continue to face challenges in meeting cost, schedule, and performance expectations. However, neither the Department of Defense nor the Department of Transportation identified any projects as being high risk because of their inability to manage complex projects.

While agencies have reported a significant number of IT projects as high risk, we identified other projects on which we have reported and testified that appear to meet one or more of OMB’s criteria for high risk designation including high development or operating costs and recognized deficiencies in adequate performance but were not identified as high risk. Examples we have recently reported include the following projects: The Decennial Response Integration System of the Census Bureau is intended to integrate paper, Internet, and telephone responses. Its high development and operating costs are expected to make up a large portion of the $1.8 billion program to develop, test, and implement decennial census systems. In March 2006, we testified that the component agency has established baseline requirements for the acquisition, but the bureau has not yet validated the requirements or implemented a process for managing them. We concluded that, until these and other basic contract management activities are fully implemented, this project faced increased risks that the system would experience cost overruns, schedule delays, and performance shortfalls.

System—an initiative managed by the Departments of Commerce and Defense and the National Aeronautics and Space Administration—is to converge two satellite programs into a single satellite program capable of satisfying both civilian and military requirements. In November 2005, we reported that the system was a troubled program because of technical problems on critical sensors, escalating costs, poor management at multiple levels, and the lack of a decision on how to proceed with the program. Over the last several years, this system has experienced continual cost increases to about $10 billion and schedule delays, requiring difficult decisions about the program’s direction and capabilities. More recently, we testified that the program is still in trouble and that its future direction is not yet known. While the program office has corrective actions under way, we concluded that, as the project continues, it will be critical to ensure that the management issues of the past are not repeated.

The Rescue 21 project is a planned coastal communications system of the Department of Homeland Security. We recently reported that inadequacies in several areas contributed to Rescue 21 cost overruns and schedule delays. These inadequacies occurred in requirements management, project monitoring, risk management, contractor cost and schedule estimation and delivery, and executive level oversight. Accordingly, the estimated total acquisition cost has increased from $250 million in 1999 to $710.5 million in 2005, and the timeline for achieving full operating capability has been extended from 2006 to 2011.

For the projects we identified as appearing to meet OMB’s criteria for high risk, the responsible agencies reported that they did not consider these investments to be high risk projects for reasons such as (1) the project was not a major investment; (2) agency management is experienced in overseeing projects; or (3) the project did not have weaknesses in its business case. In particular, one agency stated that their list does not include all high risk projects, it includes only those that are the highest priority of the high risk investments. However, none of the reasons provided are associated with OMB’s high risk definition.

While OMB staff acknowledged that the process for identifying high risk projects might not catch all projects meeting the criteria, they stated that they have other mechanisms for determining the performance of all IT projects, including high risk projects, such as the review of earned value management data. Nevertheless, without consistent application of the high risk criteria, OMB and executives cannot have the assurance that all projects that require special attention have been identified.

Process for Updating High Risk Projects Is Not Defined

OMB’s guidance does not define a process for updating high risk projects that have been identified including identifying new projects and removing current ones. In the absence of such guidance, agencies use different procedures, for example, for removing projects from the list. Specifically, some agencies reported removing projects from the list if they no longer meet OMB’s criteria and other agencies reported removing a project if it (1) is completed or moves into operations; (2) has become compliant with its cost and schedule baseline goals; (3) is no longer considered a major IT investment; (4) becomes on track and maintains this status within specific cost, schedule and performance for a minimum of two quarters; or (5) addresses major weaknesses such as earned value management requirements.

While OMB staff acknowledge that there is no defined process for updating the set of projects, they stated that agencies are in constant communication with individual analysts at OMB through e-mails, phone calls, or meetings to identify new high risk projects if they meet the definition or remove old ones if they no longer meet the criteria. Nevertheless, without guidance for updating high risk projects on a continuing basis, OMB and agency executives cannot be assured they have identified the appropriate projects that should be designated as high risk.

OMB and Agencies Can Further Improve Oversight of High Risk Projects

All 24 CFO agencies reported having procedures for overseeing high risk projects. While some agencies reported using their current investment management processes for specific oversight, other agencies established additional oversight procedures. For example, one agency developed and documented specific procedures for sending a quarterly data call to the program offices that have high risk investments. The program office then completes a template capturing current performance information and sends it to the Office of the CIO for review and feedback. The CIO office forwards it to OMB, as required. In contrast, some other agencies reported that these projects are managed as part of their current investment review process—requiring the investment review board to perform control reviews along with other investments.

While procedures for overseeing high risk projects are positive steps, we have previously reported that agencies generally have weaknesses in project oversight. In particular, we reported that agencies did not always have important mechanisms in place for agencywide investment management boards to effectively control investments, including decision-making rules for project oversight, early warning mechanisms, and/or requirements that corrective actions for underperforming projects be agreed upon and tracked. To remedy these weaknesses, we have made several recommendations to improve processes for effective oversight, many of which remain open. Until agencies establish the practices needed to effectively manage IT investments including those that are high risk, OMB, agency executives, and Congress cannot be assured that investments are being properly managed.

OMB’s oversight of high risk projects, in turn, entails reviewing the performance reports on a quarterly basis. Specifically, according to OMB staff, individual analysts review the quarterly performance reports of projects with shortfalls to determine how well the projects are progressing and whether the actions described in the planned improvement efforts are adequate. These officials also stated that the OMB analysts review the quarterly reports for completeness and consistency with other performance data already received on IT projects. This includes quarterly e-Gov Scorecards,earned value management data, and the exhibit 300. For projects without shortfalls, officials stated that while the memorandum does not direct agencies to submit these reports, agencies communicate the status of these projects to the appropriate officials. According to OMB, the reporting requirement for high risk projects enhances oversight by capturing all key elements in a single report and providing oversight authorities and agency management early indicators of any problems or shortfalls since the reporting is conducted on a quarterly basis.

However, OMB does not maintain a single aggregate list of high risk projects. OMB staff told us they do not construct a single list because they did not see such an activity as necessary in achieving the intent of the guidance—to improve project planning and execution. Consistent with our Management Watch List observations and recommendations, we believe that by not having a single list, OMB is not fully exploiting the opportunity to use the quarterly reports as a tool for analyzing high risk projects on a governmentwide basis and for tracking governmentwide progress. It is limiting its ability to identify and report on the full set of IT investments across the federal government that require special oversight and greater agency management attention.

High Risk and Management Watch List Projects Identified Using Different Criteria

The high risk projects and Management Watch List projects are identified using different sets of criteria. In addition, while the identification of high risk projects centers on an agency’s oversight of the project’s performance, the Management Watch List focuses more on a project’s planning.

As discussed previously, the high risk list consists of projects identified by the agencies with the assistance of OMB, using specific criteria established by OMB, including memorandum M-05-23. As discussed previously, these projects are reported quarterly by the agencies to OMB on a template focusing on each project’s performance in four specified areas and noted shortfalls. The agencies are also to report planned corrective actions addressing the shortfalls.

On the other hand, OMB determines projects to be included on its Management Watch List based on an evaluation of exhibit 300 business cases that agencies submit for major projects as part of the budget development process. This evaluation is part of OMB’s responsibility for helping to ensure that investments of public resources are justified and that public resources are wisely invested. Each exhibit 300 is assigned a score in 10 different categories, the results of which determine whether an individual project (or investment) warrants being included on the Management Watch List. This may result in OMB’s asking the agency to submit a remediation plan to address the weaknesses identified in the agency’s business case.

While the criteria for identifying the Management Watch List projects and high risk projects differ, Management Watch List projects can also be high risk. For example, of the 226 total number of high risk projects, agencies identified 37 of these projects as being on OMB’s Management Watch List, with 19 of these projects having performance shortfalls. According to OMB staff, identifying and addressing poorly planned projects as part of the Management Watch List process could result in fewer projects with performance shortfalls over time. Nevertheless, both types of projects require close attention because of their importance in supporting critical functions and the likelihood that performance problems associated with them could potentially result in billions of taxpayers’ dollars being wasted if they are not detected early.

Conclusions

OMB and agencies’ efforts to identify 226 high risk projects are important steps in helping focus management attention on critically important IT projects. Although many projects were appropriately identified as high risk initiatives consistent with OMB’s guidance, OMB’s criteria were not always consistently applied. As a result, projects that appear to be high risk were not always identified as such. Further, because OMB has not provided guidance on how the initial set of high risk projects list should be updated, agencies do not have a consistent process for doing so.

Agencies and OMB have both taken actions to ensure oversight of the high risk projects. Specifically, agencies are using existing oversight procedures or ones they have specifically established for the high risk projects and OMB is reviewing quarterly reports. However, weaknesses remain: agencies need to implement specific recommendations we have previously made to improve their practices for overseeing projects. Finally, OMB has not developed a single aggregate list of high risk projects to track progress, perform governmentwide analysis, and report the results to Congress.

While the criteria for high risk projects and those on the Management Watch List differ, both types of projects support critical business functions and could experience performance problems that could become costly to address if they are not detected early. Given this, the Management Watch List projects and the high risk projects both require continued attention.

Recommendations for Executive Action

In order for OMB to take advantage of the potential benefits of using the quarterly performance reports as a tool for identifying and overseeing high risk projects on a governmentwide basis, we are recommending that the Director of OMB take the following three actions: Direct federal agency CIOs to ensure that they are consistently applying the criteria defined by OMB.

Establish a structured, consistent process to update the initial list of high risk projects on a regular basis, including identifying new projects and removing previous ones to ensure the list is current and complete.

Develop a single aggregate list of high risk projects and their deficiencies and use that list to report to Congress progress made in correcting high risk problems, actions under way, and further actions that may be needed. OMB could consider using the information we have developed in appendix III as a starting point for developing this single list. In implementing these recommendations, OMB should consider working with the CIO Council to help ensure governmentwide acceptance of these actions.

Because we have outstanding recommendations aimed at (1) improving agencies’ investment management practices and (2) using the Management Watch List as a tool for analyzing, setting priorities, and following up on IT projects, we are not making any new recommendations in this report regarding these issues.

Agency Comments and Our Evaluation

OMB’s Administrator for the E-Government and Information Technology provided written comments on a draft of this report (reprinted in app. II). In these comments, OMB stated that it appreciated our careful review of OMB’s process for identifying and overseeing high risk projects. However, the agency disagreed with our recommendations and made other observations.

In its comments, OMB stated that it is concerned about our interpretation of the goals and intent of the high risk process in comparison to GAO’s high risk list. Our intent is not to confuse the goals and intent of the two efforts. Nevertheless, as noted in our report, some major programs and operations have been placed on our high risk list because of weaknesses in key agency management practices, and this is consistent with OMB’s first criterion for high risk designation—the agency has not demonstrated the ability to manage complex projects.

In its comments, OMB also observed that the policy for identifying and overseeing high risk projects is separate and apart from OMB’s Management Watch List and presents oversight authorities with information that differs in focus, timing, and expected results. While we agree with OMB that the two policies are different and acknowledge this in our report, we also noted in the report that Management Watch List projects can also be high risk. We believe projects from both lists warrant close attention because of their importance in supporting critical functions and the likelihood that performance problems associated with them could potentially result in billion of taxpayers’ dollars being wasted if they are not detected early.

Regarding our recommendations to direct agencies to consistently apply the criteria for designating projects as high risk and to establish a structured, consistent process to update the initial list of high risk projects, OMB stated that the process and criteria for designating projects as high risk are clear and that some flexibility in the application of the criteria is essential. While some flexibility in the application of the criteria may be appropriate, we believe these criteria should be applied more consistently so that projects that clearly appear to meet them, such as those we mention in the report, are identified.

OMB also disagreed with our recommendation to develop a single aggregate list of projects and their deficiencies to perform adequate oversight and management. As noted in the report, we believe that, by not having this list, OMB is not fully exploiting the opportunity to use the agencies’ quarterly reports as a tool for analyzing high risk projects on a governmentwide basis and for tracking governmentwide progress. In addition, OMB is limiting its ability to identify and report on the full set of IT investments across the federal government that requires special oversight and greater agency management attention.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to other interested congressional committees, the Director of the Office of Management and Budget, and other interested parties. Copies will also be made available at no charge on our Web site at www.gao.gov.

If you have any questions on matters discussed in this report, please contact me at (202) 512-9286 or at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) provide a summary of high risk projects that identifies by agency the number of high risk projects, their proposed budget for fiscal year 2007, agency reasons for the high risk designation, and reported performance shortfalls; (2) determine how high risk projects were identified and updated and what processes and procedures have been established to effectively oversee them; and (3) determine the relationship between the high risk list and OMB’s Management Watch List.

We conducted our work at OMB and the 24 chief financial officer (CFO) agencies in Washington, D.C. The 24 agencies are the departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, the Treasury, and Veterans Affairs; and the Environmental Protection Agency, General Services Administration, National Aeronautics and Space Administration, National Science Foundation, Nuclear Regulatory Commission, Office of Personnel Management, Small Business Administration, Social Security Administration, and U.S. Agency for International Development.

To address the first objective, we requested and reviewed documentation that identifies, for each agency, the number of high risk projects, their proposed budget for fiscal year 2007, agency reasons for the high risk designation, and reported performance shortfalls. In particular, we reviewed agency performance reports on high risk projects for September and December 2005 and March 2006 that identified high risk projects and planned improvement efforts, if any. We did not independently verify the information contained in these performance reports. However, we asked all 24 CFO agencies to confirm the data in appendix III regarding their high risk projects. Furthermore, we obtained the funding information for all high risk projects for fiscal years 2005, 2006, and 2007 from the Report on IT Spending for the Federal Government, Exhibit 53. We did not verify these data.

To address the second objective, we used a structured data collection instrument to better understand the 24 CFO agencies’ processes and procedures for identifying and overseeing high risk projects. All 24 agencies responded to our structured questionnaire. We did not verify the accuracy of the agencies’ responses; however, we reviewed supporting documentation that selected agencies provided to validate their responses. We contacted agency officials when necessary for follow-up information. We then analyzed the agencies’ responses.

Moreover, we identified and reviewed prior GAO reports on projects with weaknesses that met OMB’s high risk definition. Finally, to gain insight into OMB’s processes and procedures to oversee the high risk list, we reviewed related policy guidance, including its Memorandum on Improving IT Project Planning and Execution (M-05-23, dated August 4, 2005), and the Clinger-Cohen Act. We also interviewed OMB staff including the chief of the Information Technology and Policy Branch.

To address the third objective, we interviewed OMB staff who are responsible for developing and monitoring the high risk list and Management Watch List, including the chief of the Information Technology and Policy Branch. In addition, we reviewed our prior work on OMB’s Management Watch List, (GAO-05-276), to better understand the processes for placing projects on the Management Watch List and following up on their corrective actions. Finally, we requested information from the 24 CFO agencies on which of their high risk projects were also on the Management Watch List. Two of the 24 agencies did not identify how many of their high risk projects were also on the Management Watch List.

We conducted our work in Washington, D.C., from October 2005 through May 2006 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Office of Management and Budget

Appendix III: Summary of High Risk IT Projects by Department or Agency

FY2005 actuals (in millions) millions) millions)

Unclear baselines, schedule variance not within 10 percent, and qualified project manager is not in place.

Component agency has 20 people currently enrolled in project management training and revising business case. The investment has been elevated to the Undersecretary level to address management issues.

FY2005 actuals (in millions) millions) millions)

Unclear baselines, schedule variance not within 10 percent, and qualified project manager is not in place.

Revising business case and addressing project management issues.

Component agency has signed agreements for conversion to enterprise human resource integration.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions) millions)

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

Defense Acquisition Executive established a Joint Program Executive Officer with acquisition authority across all product lines in 2nd quarter fiscal year 2005. This officer commissioned an independent assessment of program cost, schedule, and performance, and technical maturity in spring 2005. The Defense Acquisition Executive last reviewed progress on the project’s planning on November 22, 2005.

On December 1, 2005, Deputy Secretary of Defense determined project is a viable solution for Army personnel and pay and transferred the program to the new Business Transformation Agency.

Air Force assessment will be briefed to the Defense Business Systems Management Committee on March 23, 2006. The Navy assessment will start March 13, 2006, followed by the Marine Corps in fiscal year 2007.

Completion date is to be determined.

FY2005 actuals (in millions) millions) millions)

Systems Integrator Source Selection under way. Program will realign schedule subsequent to systems integrator contract award in June 2006.

An Army 3-star level review was conducted on February 1, 2006, and the Office of the Secretary of Defense, Networks and Information Integration, Overarching Integrated Product Team was briefed on February 2, 2006. The program office will undergo another Overarching Integrated Product Team review in June 2006 and will submit for Office of the Secretary of Defense approval a baseline that includes metrics for cost, schedule, and performance.

The prime contract was fully defined on January 2, 2006. The program rebaselining is planned to be completed in the 3rd quarter of fiscal year 2006.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2006 enacted (in millions) millions)

The use of earned value management techniques will closely monitor the project’s development and production schedule. Project schedule agreed to by upper management, constantly overseen.

The project manager is attending IT project manager certification program.

Rebaseline the cost and schedule based on changing requirements.

FY2005 actuals (in millions)

FY2006 enacted (in millions) millions)

Cost and schedule variances not within 10 percent and project manager is not qualified.

The project manager is serving in a temporary capacity as the office is going through reorganization.

Project manager is not qualified.

The project manager is attending IT project manager certification program.

Cost and schedule variances not within 10 percent and project manager is not qualified.

The project manager is scheduled to complete IT project manager certification program.

Project manager is not qualified.

The project manager is attending IT project manager certification program.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2005 actuals (in millions) millions) millions)

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2005 actuals (in millions) millions) millions)

Unclear baselines and project manager is not qualified.

Baseline revision is completed and will be submitted to the agency Investment Review Board for review/approval 3/14/06.

Project manager has completed 2 courses of a 7 course master’s certification program.

FY2005 actuals (in millions) millions) millions)

Governance issues remain unclear. Specifically, it is imperative that a financing strategy be in place and that migrations be adequately funded before the Shared Service Centers start servicing new customers.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2005 actuals (in millions) millions) millions)

Project is in initial phase; therefore, baselines have not been approved and earned value management is not yet required. Program manager is not qualified.

Project manager enrolled in training to achieve level III certification.

Corrective actions not reported.

Program manager is not qualified.

Certification application to be submitted to DHS by 1/31/06.

Unclear baselines and program manager is not qualified.

Conducting internal Investment Review Board making “within threshold adjustments” to key work breakdown structure by 6/1/06 and assign a fully qualified project manager by 3/15/06.

Unclear baselines and program manager is not qualified.

Appropriate resources have been contacted to complete the approval of the baseline documentation and project manager certification by 5/24/06.

FY2005 actuals (in millions) millions) millions)

Unclear baselines, project manager is not qualified, and duplication exists between other investments.

Submit baseline documents by 3/1/06 and project manager certification by 2/15/06 to prepare for the Investment Review Board briefing scheduled for 4/26/06.

Weakness in the area of performance goals.

Create detailed project plans to satisfy earned value management criteria.

Briefing to the component agency’s administrator on need for funding.

Program manager is not qualified.

Project manager has developed and is implementing a training plan to achieve certification.

Program manager is not qualified.

Training plan in place and program office is looking to backfill position.

Revised deployment schedule is contingent on completing the investment review process.

FY2005 actuals (in millions) millions) millions)

Component agency officials are giving technical assistance to develop and present an approved baseline to DHS by 3/15/06 and project manager certification to be granted April 2006.

Corrective actions not reported.

Since current project manager is acting, DHS will hire an individual with appropriate certification level.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

According to agency officials, since Secure Flight and Crew Vetting were considered as one investment in the fiscal year 2007 budget submission, the 2005 actuals, 2006 enacted and 2007 request are the same for both projects. They will be separate investments in fiscal year 2008. millions) millions)

Reasons for high risk designation Supports the presidential initiative for a citizen- centered, results- oriented, market- based government.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

New contract was awarded that includes requirement for contractor to use an ANSI Standard 748-compliant EVMS. An Integrated Baseline Review is under way and will be completed by March 31, 2006. Project will request DOI Investment Review Board approval of new baseline in April.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

The planned contract award of the development contract is January 2007. The ANSI/EIA-748 compliance will occur in April 2007.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

Reasons for high risk designation $14.0 B, D A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2006 enacted (in millions)

FY2007 request (in millions)

On October 17, 2005, the Under Secretary for Management signed a task order authorizing the initiation of a detailed contingency planning effort for this investment. A report on this planning effort was submitted by the Chief Information Officer to the Under Secretary for Management on February 13, 2006.

OMB and the General Services Administration, the managing partner of this e-government initiative, have been consistently apprised of the problems with the vendor’s software and the efforts the Department of State has made to help the vendor design the needed functionality.

The international version of the software is scheduled to be released by the vendor near the end of fiscal year 2006. Department of State anticipates a significant amount of testing prior to using the international capabilities of this software in a production environment. As a result, this will push the first overseas pilot into fiscal year 2007.

FY2005 actuals (in millions)

FY2006 enacted (in millions)

FY2007 request (in millions)

The National Finance Center is assessing the impact of system modifications to meet the Department of State’s payroll processing requirements. System development efforts by the National Finance Center will determine the implementation schedule for the agency and the center’s migration activities and overall costs for both agencies.

The National Finance Center has committed to providing a written cost estimate by March 17, 2006.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The projects is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2005 actuals (in millions)

FY2006 enacted (in millions)

FY2007 request (in millions)

Corrective actions taken to put the program back on track to meet fiscal year 2007 target date for full implementation.

Program rebaselined in December 2005 and corrective actions taken that bring it within variance limits.

FY2005 actuals (in millions)

FY2006 enacted (in millions)

FY2007 request (in millions)

Investment Review Board has not baselined this project.

Investment Review Board has not baselined this project.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

The development, modernization, enhancement costs are expected to fall within tolerance as a result of closeout costs being reported.

The corrective action for the schedule variance is being handled as part of the restructuring and re- planning activity in 1st quarter fiscal year 2006.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

This project is being terminated.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

FY2005 actuals (in millions) millions) millions)

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

This project is being terminated.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

FY2005 actuals (in millions) millions) millions)

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

A new operational baseline with associated cost and schedule variances will be submitted for OMB approval.

An operational baseline with associated cost and schedule variances will be submitted for OMB approval.

No planned improvement efforts reported.

An operational baseline with associated cost and schedule variances will be submitted for OMB approval.

FY2005 actuals (in millions) millions) millions)

A new operational baseline with associated cost and schedule variances will be submitted for OMB approval.

An operational baseline with associated cost and schedule variances will be submitted for OMB approval.

A new performance measurement baseline with associated cost and schedule variances will be submitted for OMB approval.

An operational baseline with associated cost and schedule variances will be submitted for OMB approval.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2006 enacted (in millions)

FY2007 request (in millions)

A rebaseline will be requested and monitored by operational analysis rather than earned value management until development funds are reauthorized.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

This is a large project in the initial stage.

Unclear baselines, cost, and schedule variances not within 10 percent and project manager is not qualified.

Based on the results of an independent assessment, GSA has determined that this investment is not meeting the current and future business objectives. As a result, GSA is terminating this investment.

GSA has initiated a data migration initiative that will enable migration of the two regions to the legacy system. Will provide quarterly updates on progress of migration activity.

FY2005 actuals (in millions) millions) millions)

Cost variance not within 10 percent and project manager is not qualified.

Update task planned start and end dates on protest resolution and project manager will continue required training to meet CIO program manager certification criteria.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions) millions)

N/A millions) millions) millions)

Reasons for high risk designation A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2006 enacted (in millions)

FY2007 request (in millions)

Reasons for high risk designation A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions) millions)

Unclear baselines, cost and schedule variances not within 10 percent, and project manager is not qualified.

The Office of Personnel Management’s project coordinator will work with OMB staff and interagency Information Systems Security Line of Business participants to clarify governmentwide and agency goals. Once the goals are clarified, the baseline cost and schedule will be developed. Agency will assess the project manager against the agency’s qualification guidelines.

This project is still in the planning phase and a baseline is being developed.

Corrective actions not reported.

N/A millions) millions) millions)

The Human Resources Management Line of Business/Human Resource Development Project Management Office will closely monitor the delivery of activities on the enterprise architecture, Workforce Development Roadmap, and performance management sub- projects. OPM requested the completion of remaining baseline corrections to resolve located schedule errors.

Cost and schedule variances not within 10 percent.

For both the cost/ and schedule variances, the agency is updating out estimate to complete to reflect a realistic timeline given the current circumstances with external stakeholders.

N/A millions) millions) millions)

Reasons for high risk designation A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The projects is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other. millions) millions)

FY2005 actuals (in millions) millions) millions)

Project manager is not yet qualified.

Original project deliverable for fiscal year 2006 was deferred, with no project manager required. New project manager is receiving training as part of Office of CIO directed formal training activity.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

According to agency officials, the fiscal year 2006 request was enacted for these investments.

FY2006 enacted (in millions) millions)

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

FY2005 actuals (in millions) millions) millions)

Baselines not yet established and cost and schedule variances not within 10 percent.

To collect information from various sources at the agency and the Department of State in order to validate milestones.

A=The agency has not consistently demonstrated the ability to manage complex projects.

B=The project has exceptionally high development, operating, or maintenance costs, either in absolute terms or as a percentage of the agency’s total IT portfolio.

C=The project is being undertaken to correct recognized deficiencies in the adequate performance of an essential mission program or function of the agency, a component of the agency, or another organization.

E=Other.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, the following people made key contributions to this report: William G. Barrick, Nancy Glover, Nnaemeka Okonkwo, Sabine Paul, and Niti Tandon.