Information Technology: Further Improvements Needed to identify and Oversee Poorly Planned and Performing Projects
Published: 2007-09-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Office of Management and Budget (OMB) plays a key role in overseeing federal information technology (IT) investments. The Clinger-Cohen Act, among other things, requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by agencies and to report to Congress on the net program performance benefits achieved as a result of these investments. OMB has developed several processes to help carry out its role. For example, OMB began using a Management Watch List several years ago as a means of identifying poorly planned projects based on its evaluation of agencies' funding justifications for major projects, known as exhibit 300s. In addition, in August 2005, OMB established a process for agencies to identify high risk projects and to report on those that are performing poorly. GAO testified last year on the Management Watch List and high risk projects, and on GAO's recommendations to improve these processes. GAO was asked to (1) provide an update on the Management Watch List and high risk projects and (2) identify OMB's efforts to improve the identification and oversight of these projects. In preparing this testimony, GAO summarized its previous reports on initiatives for improving the management of federal IT investments. GAO also analyzed current Management Watch List and high risk project information.
section_title: What GAO Found
paragraphs: OMB and federal agencies have identified approximately 227 IT projects--totaling at least $10.4 billion in expenditures for fiscal year 2008--as being poorly planned (on the Management Watch List), poorly performing (on the High Risk List with performance shortfalls), or both. OMB has taken steps to improve the identification and oversight of the Management Watch List and High Risk projects by addressing recommendations previously made by GAO, however, additional efforts are needed to more effectively perform these activities. Specifically, GAO previously recommended that OMB take action to improve the accuracy and reliability of exhibit 300s and consistent application of the high risk projects criteria, and perform governmentwide tracking and analysis of Management Watch List and high risk project information. In response to these recommendations, OMB, for example, started publicly releasing aggregate lists of Management Watch List and high risk projects by agency in September 2006 and has been updating them since then on a quarterly basis. However, OMB does not publish the reasons for placing projects on the Management Watch List, nor does it specifically identify why high risk projects are poorly performing. Providing this information would allow OMB and others to better analyze the reasons projects are poorly planned and performing, take corrective actions, and track these projects on a governmentwide basis. Such information would also help to highlight progress made by agencies or projects, identify management issues that transcend individual agencies, and highlight the root causes of governmentwide issues and trends. Until OMB makes further improvements in the identification and oversight of poorly planned and poorly performing IT projects, potentially billions in taxpayer dollars are at risk of being wasted.

section_title: Letter
paragraphs: I am pleased to be here today to discuss the federal government’s key  processes for improving the management of information technology (IT)  investments totaling $65 billion for fiscal year 2008. Effective management  of these investments is essential to the health, economy, and security of  the nation. The Office of Management and Budget (OMB) plays a key role  in overseeing federal IT investments. In particular, as required by the  Clinger-Cohen Act, OMB must establish processes to analyze, track, and  evaluate the risks and results of major capital investments in information  systems made by executive agencies and to report to Congress on the net  program performance benefits achieved as a result of these investments.
To help carry out its oversight role, OMB has developed several processes,  including its Management Watch List and high risk projects. The  Management Watch List identifies projects that are poorly planned  (projects with weaknesses in their funding justifications, known as exhibit  300s). High risk projects require special attention from oversight  authorities and the highest level of agency management, and include  projects that are poorly performing (projects experiencing performance  shortfalls, meaning that they do not meet one or more of four performance  evaluation criteria). The Management Watch List and high risk processes  are instrumental in helping to identify and improve oversight of poorly  planned and poorly performing projects.
Last September, we testified on OMB’s oversight of federal IT projects. We  highlighted the number and dollar value of the projects identified as poorly  planned and/or poorly performing as a result of the Management Watch  List and high risk processes. Given the importance of OMB’s oversight  processes, you asked us to (1) provide an update on the Management  Watch List and High Risk projects, and (2) identify OMB’s efforts to  improve the identification and oversight of these projects. In preparing  this testimony, we summarized our previous reports on initiatives for  improving the management of federal IT investments and interviewed  OMB staff on their efforts to better identify and oversee Management  Watch List and high risk projects. We also analyzed current Management  Watch List and high risk project information. We performed our work in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Each year, OMB and federal agencies work together to determine how  much the government plans to spend for IT and how these funds are to be  allocated. Federal IT spending has risen to an estimated $65 billion in  fiscal year 2008.
OMB plays a key role in overseeing the implementation and management  of federal IT investments. To improve this oversight, Congress enacted the  Clinger-Cohen Act in 1996, expanding the responsibilities delegated to  OMB and agencies under the Paperwork Reduction Act. Among other  things, Clinger-Cohen requires agency heads, acting through agency chief  information officers, to better link their IT planning and investment  decisions to program missions and goals and to implement and enforce IT  management policies, procedures, standards, and guidelines. The act also  requires that agencies engage in capital planning and performance and  results-based management. OMB’s responsibilities under the act include  establishing processes to analyze, track, and evaluate the risks and results  of major capital investments in information systems made by executive  agencies. OMB must also report to Congress on the net program  performance benefits achieved as a result of major capital investments in  information systems that are made by executive agencies.
In response to the Clinger-Cohen Act and other statutes, OMB developed  policy for planning, budgeting, acquisition, and management of federal  capital assets. This policy is set forth in OMB Circular A-11 (section 300)  and in OMB’s Capital Programming Guide (supplement to Part 7 of  Circular A-11), which directs agencies to develop, implement, and use a  capital programming process to build their capital asset portfolios. Among  other things, OMB’s Capital Programming Guide directs agencies to    evaluate and select capital asset investments that will support core  mission functions that must be performed by the federal government and  demonstrate projected returns on investment that are clearly equal to or  better than alternative uses of available public resources; institute performance measures and management processes that monitor  actual performance and compare to planned results; and   establish oversight mechanisms that require periodic review of operational  capital assets to determine how mission requirements might have changed  and whether the asset continues to fulfill mission requirements and deliver  intended benefits to the agency and customers.
To further support the implementation of IT capital planning practices as  required by statute and directed in OMB’s Capital Programming Guide, we  have developed an IT investment management framework that agencies  can use in developing a stable and effective capital planning process.  Consistent with the statutory focus on selecting, controlling, and  evaluating investments, this framework focuses on these processes in  relation to IT investments specifically. It is a tool that can be used to  determine both the status of an agency’s current IT investment  management capabilities and the additional steps that are needed to  establish more effective processes. Mature and effective management of IT  investments can vastly improve government performance and  accountability. Without good management, such investments can result in  wasteful spending and lost opportunities for improving delivery of services  to the public.
subsections: section_title: Prior Reviews on Federal IT Investment Management Have Identified Weaknesses
paragraphs: Only by effectively and efficiently managing their IT resources through a  robust investment management process can agencies gain opportunities to  make better allocation decisions among many investment alternatives and  further leverage their investments. However, the federal government faces  enduring IT challenges in this area. For example, in January 2004 we  reported on mixed results of federal agencies’ use of IT investment  management practices. Specifically, we reported that although most of  the agencies had IT investment boards responsible for defining and  implementing the agencies’ investment management processes, agencies  did not always have important mechanisms in place for these boards to  effectively control investments, including decision-making rules for  project oversight, early warning mechanisms, and/or requirements that  corrective actions for underperforming projects be agreed upon and  tracked. Executive-level oversight of project-level management activities  provides organizations with increased assurance that each investment will  achieve the desired cost, benefit, and schedule results. Accordingly, we  made several recommendations to agencies to improve their practices.
In previous work using our investment management framework, we  reported that the use of IT investment management practices by agencies  was mixed. For example, a few agencies that have followed the framework  in implementing capital planning processes have made significant  improvements. In contrast, however, we and others have continued to  identify weaknesses at agencies in many areas, including immature  management processes to support both the selection and oversight of  major IT investments and the measurement of actual versus expected  performance in meeting established performance measures. For example,  we recently reported that the Department of Homeland Security and the  Department of Treasury did not have the processes in place to effectively  select and oversee their major investments.
section_title: OMB’s Management Watch List Is Intended to Correct Project Weaknesses and Business Case Deficiencies
paragraphs: To help ensure that investments of public resources are justified and that  public resources are wisely invested, OMB began using its Management  Watch List in the President’s fiscal year 2004 budget request, as a means to  oversee the justification for and planning of agencies’ IT investments. This  list was derived based on a detailed review of the investments’ Capital  Asset Plan and Business Case, also known as the exhibit 300.
The exhibit 300 is a reporting mechanism intended to enable an agency to  demonstrate to its own management, as well as OMB, that a major project  is well planned in that it has employed the disciplines of good project  management; developed a strong business case for the investment; and  met other Administration priorities in defining the cost, schedule, and  performance goals proposed for the investment.
We reported in 2005 that OMB analysts evaluate agency exhibit 300s by  assigning scores to each exhibit 300 based on guidance presented in OMB  Circular A-11. As described in this circular, the scoring of a business case  consists of individual scoring for 10 categories, as well as a total  composite score of all the categories. The 10 categories are    project (investment) management,    performance-based management system (including the earned value  life-cycle costs formulation, and   support of the President’s Management Agenda.
Projects are placed on the Management Watch List if they receive low  scores (3 or less on a scale from 1 to 5) in the areas of performance goals,  performance-based management systems, security and privacy or a low  composite score.
According to OMB, agencies with weaknesses in these three areas are to  submit remediation plans addressing the weaknesses. OMB officials also  stated that decisions on follow-up and monitoring the progress are  typically made by staff with responsibility for reviewing individual agency  budget submissions, depending on the staff’s insights into agency  operations and objectives. According to OMB officials, those Management  Watch List projects that receive specific follow-up attention receive  feedback through the passback process, targeted evaluation of  remediation plans designed to address weaknesses, the apportioning of  funds so that the use of budgeted dollars was conditional on appropriate  remediation plans being in place, and the quarterly e-Gov Scorecards.  OMB removes projects from the Management Watch List as agencies  remediate the weaknesses identified with these projects’ business cases.
section_title: OMB’s High Risk Projects Process Intended to Correct and Improve Project Performance
paragraphs: As originally defined in OMB Circular A-11 and subsequently reiterated in  an August 2005 memorandum, high risk projects are those that require  special attention from oversight authorities and the highest levels of  agency management. These projects are not necessarily “at risk” of failure,  but may be on the list because of one or more of the following four  reasons:    The agency has not consistently demonstrated the ability to manage  complex projects.
The project has exceptionally high development, operating, or  maintenance costs, either in absolute terms or as a percentage of the  agency’s total IT portfolio.
The project is being undertaken to correct recognized deficiencies in the  adequate performance of an essential mission program or function of the  agency, a component of the agency, or another organization.
Delay or failure of the project would introduce for the first time  unacceptable or inadequate performance or failure of an essential mission  function of the agency, a component of the agency, or another  organization.
Most agencies reported that to identify high risk projects, staff from the  Office of the Chief Information Officer compare the criteria against their  current portfolio to determine which projects met OMB’s definition. They  then submit the list to OMB for review. According to OMB and agency  officials, after the submission of the initial list, examiners at OMB work  with individual agencies to identify or remove projects as appropriate.  According to most agencies, the final list is then approved by their Chief  Information Officer.
For the identified high risk projects, beginning September 15, 2005, and  quarterly thereafter, Chief Information Officers are to assess, confirm, and  document projects’ performance. Specifically, agencies are required to  determine, for each of their high risk projects, whether the project was  meeting one or more of four performance evaluation criteria:    establishing baselines with clear cost, schedule, and performance goals;    maintaining the project’s cost and schedule variances within 10 percent;    assigning a qualified project manager; and    avoiding duplication by leveraging inter-agency and governmentwide  investments.
If a high risk project meets any of these four performance evaluation  criteria, agencies are instructed to document this using a standard  template provided by OMB and provide this template to oversight  authorities (e.g., OMB, agency inspectors general, agency management,  and GAO) on request. Upon submission, according to OMB staff,  individual analysts review the quarterly performance reports of projects  with shortfalls to determine how well the projects are progressing and  whether the actions described in the planned improvement efforts are  adequate using other performance data already received on IT projects  such as the e-Gov Scorecards, earned value management data, and the  exhibit 300.
section_title: Poorly Planned and Performing Projects Total at Least $10 Billion in Estimated Expenditures for Fiscal Year 2008
paragraphs: OMB and federal agencies have identified approximately 227 IT projects— totaling at least $10.4 billion in expenditures for fiscal year 2008—as being  poorly planned, poorly performing, or both. Figure 1 shows the  distribution of these projects and their associated dollar values.
subsections: section_title: Hundreds of Projects Totaling Billions of Dollars Are Placed on the Management Watch List Annually
paragraphs: Each year, OMB places hundreds of projects totaling billions of dollars on  the Management Watch List. Table 1 provides a historical perspective of  the number of these projects and their associated budget since OMB  started reporting on the Management Watch List in the President’s budget  request for 2004. The table shows that while the number of projects and  their associated budget have generally decreased since then, they  increased by 83 projects this year, and represent a significant percentage  of the total budget.
As of July 2007, 136 projects, representing $8.6 billion, still remained on  the Management Watch List (see appendix 1 for complete list). We  determined that 29 of these projects were on the Management Watch List  as of September 2006.
section_title: Poorly Performing Projects Total About $6 Billion in Estimated Expenditures for Fiscal Year 2008
paragraphs: As of June 2007, when agencies last reported on their high risk projects to  OMB, the 24 major agencies identified 438 IT projects as high risk, of  which 124 had performance shortfalls collectively totaling about $6.0  billion in funding requested for fiscal year 2008. Table 2 shows that the  number of projects, as well as the number of projects with shortfalls  increased this year. OMB attributes this rise to increased management  oversight by agencies.
The majority of projects were not reported to have had performance  shortfalls. In addition, five agencies—the departments of Energy, Housing  and Urban Development, Labor, and State, and the National Science  Foundation—reported that none of their high risk projects experienced  any performance shortfalls. Figure 2 illustrates the number of high risk  projects by agency as of June 2007, with and without shortfalls.
Agencies reported cost and schedule variances that exceeded 10 percent  as the greatest shortfall. This is consistent with what they reported about a  year ago, and the distribution of shortfalls types is similar to last year.  Figure 3 illustrates the reported number and type of performance  shortfalls associated with high risk projects.
Appendix II identifies the shortfalls associated with each of the poorly  performing projects.
Twenty-two high risk projects have experienced performance shortfalls  for the past four quarters (see figure 4).
Of these projects, the following six have had shortfalls since the High Risk  List was established in September 2005.
Department of Homeland Security’s (DHS) Secure Border Initiative Net  Technology Program, which is expected to provide on-scene agents near  real-time information on attempted border crossings by illegal aliens,  terrorists, or smugglers;    Department of Agriculture’s (USDA) Modernize and Innovate the Delivery  of Agricultural Systems, which is intended to modernize the delivery of  farm program benefits by deploying an internet-based self-service  capabilities for customers, and eliminating the department’s reliance on  aging technology and service centers as the sole means of delivering  program benefits;   Department of Veterans Affairs’ (VA) VistA Imaging, which should provide  complete online patient data to health care providers, increase clinician  productivity, facilitate medical decision-making, and improve quality of  care;   DHS’s Transportation Worker Identification Credentialing, which is to  establish a system-wide common secure biometric credential, used by all  transportation modes, for personnel requiring unescorted physical and/or  logical access to secure areas of the transportation system;   Department of Justice’s (DOJ) Regional Data Exchange, which is expected  to combine and share regional investigative information and provide  powerful tools for analyzing the integrated data sets; and   VA’s Patient Financial Services System, which is expected create a  comprehensive business solution for revenue improvement utilizing  improved business practices, commercial software, and enhanced VA  clinical applications.
section_title: Several Projects are Both Poorly Planned and Poorly Performing
paragraphs: Thirty-three projects are on both the Management Watch List and list of  high risk projects with shortfalls, meaning that they are both poorly  planned and poorly performing. They total about $4.1 billion in estimated  expenditures for fiscal year 2008. These projects are listed in table 3  below.
section_title: OMB Has Taken Steps to Improve the Identification and Oversight of Management Watch List and High Risk Projects, but Additional Efforts Are Needed
paragraphs: OMB has taken steps to improve the identification and oversight of the  Management Watch List and high risk projects by addressing some of the  recommendations we previously made, but additional efforts are needed  to more effectively perform these activities and ultimately ensure that  potentially billions of taxpayer dollars are not wasted. Specifically, we  previously recommended that OMB take action to improve the accuracy  and reliability of exhibit 300s and application of the high risk projects  criteria, and perform governmentwide tracking and analysis of  Management Watch List and high risk project information. While OMB  took steps to address our concerns, more can be done.
subsections: section_title: Exhibit 300s Are Now Reported Publicly but Their Accuracy and Reliability Issues Remain
paragraphs: In January 2006, we noted that the underlying support for information  provided in the exhibit 300s was often inadequate and that, as a result, the  Management Watch List may be undermined by inaccurate and unreliable  data. Specifically, we noted that    documentation either did not exist or did not fully agree with specific  areas of all exhibit 300s;   agencies did not always demonstrate that they complied with federal or  departmental requirements or policies with regard to management and  reporting processes; for example, no exhibit 300 had cost analyses that  fully complied with OMB requirements for cost-benefit and cost- effectiveness analyses; and   data for actual costs were unreliable because they were not derived from  cost-accounting systems with adequate controls; in the absence of such  systems, agencies generally derived cost information from ad hoc  processes.
We recommended, among other things, that OMB direct agencies to  improve the accuracy and reliability of exhibit 300 information.
To address our recommendation, in June 2006, OMB directed agencies to  post their exhibit 300s on their website within two weeks of the release of  the President’s budget request for fiscal year 2008. While this is a step in  the right direction, the accuracy and reliability of exhibit 300 information  is still a significant weakness among the 24 major agencies, as evidenced  by a March 2007 President’s Council on Integrity and Efficiency and  Executive Council on Integrity and Efficiency study commissioned by  OMB to ascertain the validity of exhibit 300s. Specifically, according to  individual agency reports contained within the study, Inspectors General  found that the documents supporting agencies’ exhibit 300s continue to  have accuracy and reliability issues. For example, according to these  reports, the Agency for International Development did not maintain the  documentation supporting exhibit 300s cost figures. In addition, at the  Internal Revenue Service, the exhibit 300s were unreliable because, among  other things, project costs were being reported inaccurately and progress  on projects in development was measured inaccurately.
section_title: High Risk Criteria Are Being Applied More Consistently, but Questions Remain as to Whether All Projects Are Identified, Including Projects with Shortfalls
paragraphs: In June 2006, we noted that OMB did not always consistently apply the  criteria for identifying high risk projects. For example, we identified  projects that appeared to meet the criteria but that were not designated as  high risk. Accordingly, we recommended that OMB apply their high risk  criteria consistently. OMB has since designated as high risk the projects  that we identified. Further, OMB officials stated that they have worked  with agencies to ensure a more consistent application of the high risk  criteria. These are positive steps, as they result in more projects receiving  the management attention they deserve.
However, questions remain as to whether all high risk projects with  shortfalls are being reported by agencies. For example, we have reported  in our high risk series that the Department of Defense’s efforts to  modernize its business systems have been hampered because of  weaknesses in practices for (1) developing and using an enterprise  architecture, (2) instituting effective investment management processes,  and (3) establishing and implementing effective systems acquisition  processes. We concluded that the department remains far from where it  needs to be to effectively and efficiently manage an undertaking of such  size, complexity, and significance. Despite these problems, Department of  Defense (DOD), which accounts for $31 billion of the government’s $65  billion in IT expenditures, only reported three projects as being high risk  with shortfalls representing a total of about $1 million. The dollar value of  DOD’s three projects represents less than one tenth of one percent of high  risk projects with shortfalls. In light of the problems we and others have  identified with many of DOD’s projects, this appears to be an  underestimation. Given the critical nature of high risk projects, it is  particularly important to identify early on those that are performing  poorly, before their shortfalls become overly costly to address.
section_title: Management Watch List and High Risk Projects Made Public, but Governmentwide Analyses Still Not Performed
paragraphs: Finally, to improve the oversight of the Management Watch List projects,  we recommended in our April 2005 report that the Director of OMB  report to Congress on projects’ deficiencies, agencies’ progress in  addressing risks of major IT investments, and management areas needing  attention. In addition, to fully realize the potential benefits of using the  Management Watch List, we recommended that OMB use the list as the  basis for selecting projects for follow-up, tracking follow-up activities and  analyze the prioritized list to develop governmentwide and agency  assessments of the progress and risks of IT investments, identifying  opportunities for continued improvement. We also made similar  recommendations to the Director of OMB regarding high risk projects.  Specifically, we recommended that OMB develop a single aggregate list of  high risk projects and their deficiencies and use that list to report to  Congress progress made in correcting high risk problems, actions under  way, and further actions that may be needed.
To its credit, OMB started publicly releasing aggregate lists of the  Management Watch List and high risk projects in September 2006, and has  been releasing updated versions on a quarterly basis by posting them on  their website. While this is a positive step, OMB does not publish the  specific reasons that each project is placed on the Management Watch  List, nor does it specifically identify why high risk projects are poorly  performing, as we have done in appendix II. Providing this information  would allow OMB and others to better analyze the reasons projects are  poorly planned and performing and take corrective actions and track these  projects on a governmentwide basis. Such information would also help to  highlight progress made by agencies or projects, identify management  issues that transcend individual agencies, and highlight the root causes of  governmentwide issues and trends. Such analysis would be valuable to  agencies in planning future IT projects, and could enable OMB to prioritize  follow-up actions and ensure that high-priority deficiencies are addressed.
In summary, the Management Watch List and high risk projects processes  play important roles in improving the management of federal IT  investments by helping to identify poorly planned and poorly performing  projects that require management attention. As of June 2007, the 24 major  agencies had 227 such projects totaling at least $10 billion. OMB has taken  steps to improve the identification of these projects, including  implementing recommendations related to improving the accuracy of  exhibit 300s and the application of the high risk projects criteria. However,  the number of projects may be understated because issues concerning the  accuracy and reliability of the budgetary documents the Management  Watch List is derived from still remain and high risk projects with  shortfalls may not be consistently identified.
While OMB can act to further improve the identification and oversight of  poorly planned and poorly performing projects, we recognize that  agencies must also take action to fulfill their responsibilities in these  areas. We have addressed this in previous reports and made related  recommendations. Until further improvements in the identification and  oversight of poorly planned and poorly performing IT projects, potentially  billions in taxpayer dollars are at risk of being wasted.
section_title: GAO Contact and Acknowledgments
paragraphs: If you should have any questions about this testimony, please contact me  at (202) 512-9286 or by e-mail at pownerd@gao.gov. Individuals who made  key contributions to this testimony are Sabine Paul, Assistant Director;  Neil Doherty; Amos Tevelow; Kevin Walsh and Eric Winter.
section_title: Appendix I: Management Watch List Projects
paragraphs: The following provides additional detail on the investments comprising  OMB’s Management Watch List as of July 2007. Under the Clinger-Cohen  Act of 1996, agencies are required to submit business plans for IT  investments to OMB. If the agency’s investment plan contains one or more  planning weaknesses, it is placed on OMB’s Management Watch List and  targeted for follow-up action to correct potential problems prior to  execution.
We estimated the fiscal year 2008 request based on the data in the Report  on IT Spending for Fiscal Years 2006, 2007, and 2008 (generally  referred to as exhibit 53), and data provided by agencies.
section_title: Appendix II: High Risk Projects with Shortfalls
paragraphs: The following provides additional detail on the high risk projects that have  performance shortfalls as of June 2007.
We estimated the fiscal year 2008 request based on the data in the Report  on IT Spending for Fiscal Years 2006, 2007, and 2008 (generally  referred to as exhibit 53), and data provided by agencies.