Information Technology: OMB Can Make More Effective Use of Its Investment Reviews
Published: 2005-04-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: For the President's Budget for Fiscal Year 2005, the Office of Management and Budget (OMB) stated that of the nearly 1,200 major information technology (IT) projects in the budget, it had placed approximately half--621 projects, representing about $22 billion--on a Management Watch List, composed of mission-critical projects with identified weaknesses. GAO was asked to describe and assess OMB's processes for (1) placing projects on its Management Watch List and (2) following up on corrective actions established for projects on the list.
section_title: What GAO Found
paragraphs: For the fiscal year 2005 budget, OMB developed processes and criteria for including IT investments on its Management Watch List. In doing so, it identified opportunities to strengthen investments and promote improvements in IT management. However, it did not develop a single, aggregate list identifying the projects and their weaknesses. Instead, OMB officials told GAO that to identify IT projects with weaknesses, individual OMB analysts used scoring criteria that the office established for evaluating the justifications for funding that federal agencies submit for major projects. These analysts, each of whom is typically responsible for several federal agencies, were then responsible for maintaining information on these projects. To derive the total number of projects on the list that OMB reported for fiscal year 2005, OMB polled its individual analysts and compiled the result. However, OMB officials told GAO that they did not compile a list that identified the specific projects and their identified weaknesses. The officials added that they did not construct a single list because they did not see such an activity as necessary. Thus, OMB has not fully exploited the opportunity to use the list as a tool for analyzing IT investments on a governmentwide basis. OMB had not developed a structured, consistent process for deciding how to follow up on corrective actions that its individual analysts asked agencies to take to address weaknesses associated with projects on its Management Watch List. According to OMB officials, decisions on follow-up and monitoring of progress were typically made by the staff with responsibility for reviewing individual agency budget submissions, depending on the staff's insights into agency operations and objectives. Because it did not consistently require or monitor follow-up activities, OMB did not know whether the project risks that it identified through its Management Watch List were being managed effectively, potentially leaving resources at risk of being committed to poorly planned and managed projects. In addition, because it did not consistently monitor the follow-up performed on projects on the Management Watch List, OMB could not readily tell GAO which of the 621 projects received follow-up attention. Thus, OMB was not using its Management Watch List as a tool in setting priorities for improving IT investments on a governmentwide basis and focusing attention where it was most needed.

section_title: Letter
paragraphs: The President’s Budget for Fiscal Year 2005 identified approximately  $60 billion for information technology (IT) projects. In that budget, the  Office of Management and Budget (OMB) stated that, of approximately  1,200 major IT projects, about half—621 projects, representing about $22 billion—were on a ‘‘Management Watch List.’’ This information was  reiterated in testimony in March 2004, during which OMB officials stated  that the list consisted of mission-critical projects that needed to improve  performance measures, project management, and IT security. OMB  identified weaknesses in these three areas, among others, in its analysis of  the business cases that agencies submitted to justify project funding. The  officials added that the fiscal year 2005 budget process required agencies to  successfully correct project weaknesses and business case deficiencies;  otherwise, OMB would limit agencies’ spending on new starts and other  developmental activities.
This report responds to your request that we describe and assess OMB’s  processes for (1) placing projects on its Management Watch List and  (2) following up on corrective actions established for projects on the list.  To accomplish these objectives, we reviewed and analyzed OMB’s policy  and budget guidance for fiscal year 2005 and interviewed OMB officials  (further details on our objectives, scope, and methodology are provided  following the background section).
subsections: section_title: Background
paragraphs: According to OMB, its predominant mission is to assist the President in  overseeing the preparation of the federal budget and to supervise budget  administration in executive branch agencies. In helping to formulate the  President’s spending plans, OMB is responsible for evaluating the  effectiveness of agency programs, policies, and procedures; assessing  competing funding demands among agencies; and setting funding  priorities. OMB also is to ensure that agency reports, rules, testimony, and  proposed legislation are consistent with the President’s budget and with  administration policies.
In addition, OMB is responsible for overseeing and coordinating the  administration’s procurement, financial management, information, and  regulatory policies. In each of these areas, OMB’s role is to help improve  administrative management, to develop better performance measures and  coordinating mechanisms, and to reduce unnecessary burden on the  public.
To drive improvement in the implementation and management of IT  projects, the Congress enacted the Clinger-Cohen Act in 1996 to further  expand the responsibilities of OMB and the agencies under the Paperwork Reduction Act. The act requires that agencies engage in capital planning  and performance- and results-based management. OMB is required by the  Clinger-Cohen Act to establish processes to analyze, track, and evaluate the  risks and results of major capital investments in information systems made  by executive agencies. OMB is also required to report to the Congress on  the net program performance benefits achieved as a result of major capital  investments in information systems that are made by executive agencies.
In response to the Clinger-Cohen Act and other statutes, OMB developed  section 300 of Circular A-11. This section provides policy for planning,  budgeting, acquisition, and management of federal capital assets and  instructs agencies on budget justification and reporting requirements for  major IT investments. Section 300 defines the budget exhibit 300, also  called the Capital Asset Plan and Business Case, as a document that  agencies submit to OMB to justify resource requests for major IT  investments. The exhibit 300 consists of two parts: the first is required of all  assets; the second applies only to information technology. Among other  things, the exhibit 300 requires agencies to provide information  summarizing spending and funding plans; performance goals and  measures; project management plans, goals, and progress; and security  plans and progress. This reporting mechanism, as part of the budget  formulation and review process, is intended to enable an agency to  demonstrate to its own management, as well as OMB, that it has employed  the disciplines of good project management, developed a strong business  case for the investment, and met other Administration priorities in defining  the cost, schedule, and performance goals proposed for the investment.  The types of information included in the exhibit 300, among other things,  are to help OMB and the agencies identify and correct poorly planned or  performing investments (i.e., investments that are behind schedule, over  budget, or not delivering expected results) and real or potential systemic  weaknesses in federal information resource management (e.g., project  manager qualifications).
According to OMB’s description of its processes, agencies’ exhibit 300  business cases are reviewed by OMB analysts from its four statutory  offices—Offices of E-Government and Information Technology (e-Gov),  Information and Regulatory Affairs (OIRA), Federal Financial  Management, and Federal Procurement Policy—and its Resource  Management Offices (RMO). In addition to other responsibilities under  various statutes, e-Gov and OIRA develop and oversee the implementation  of governmentwide policies in the areas of IT, information policy, privacy,  and statistical policy. OIRA and e-Gov analysts also carry out economic and  related analyses, including reviewing exhibit 300s. Each of about 12  analysts is responsible for overseeing IT projects for a specific agency or  (more commonly) several agencies.
OMB’s RMOs are staffed with program examiners, whose responsibility is  to develop and support the President’s Budget and Management Agenda.  RMOs work as liaisons between federal agencies and the presidency. In  formulating the budget, they evaluate agency requests for funding and  evaluate agency management and financial practices. RMOs also evaluate  and make recommendations to the President when agencies seek new  legislation or the issuance of Presidential executive orders that would help  agencies to fulfill their organizational objectives.
According to OMB officials, the OIRA and e-Gov analysts, along with RMO  program examiners, evaluate agency exhibit 300 business cases as part of  the development of the President’s Budget. The results of this review are  provided to agencies through what is called the “passback” process. That  is, OMB passes the requests back to agencies with its evaluation, which  identifies any areas requiring remediation.
The final step in the budget process, occurring after the Congress has  appropriated funds, is apportionment, through which OMB formally  controls agency spending. According to the Antideficiency Act, before the  agency may spend its funding resources, appropriations must be  apportioned by periods within the fiscal year (typically by quarters) or  among the projects to be undertaken. Although apportionment is a  procedure required to allow agencies to access their appropriated funds,  OMB can also use apportionment to impose conditions on agency  spending, such as changes in agency practices; it is one of several  mechanisms that the Clinger-Cohen Act authorizes OMB to use to enforce  an agency head’s accountability for the agency’s IT investments.
The President’s Budget for Fiscal Year 2005 included about 1,200 IT  projects, totaling about $60 billion. Of this total number of projects, OMB  reported in the budget that slightly over half—621 projects, representing  about $22 billion—were on a Management Watch List. According to OMB’s  March 2004 testimony, this list consists of mission-critical projects that  needed to improve performance measures, project management, IT  security, or overall justification. OMB officials described this assessment as  based on evaluations of exhibit 300s submitted to justify inclusion in the  budget. According to OMB’s testimony, the fiscal year 2005 budget required  agencies to successfully correct identified project weaknesses and  business case deficiencies; otherwise, they risked OMB placing limits on  their spending. OMB officials testified in March 2004 that they would  enforce these corrective actions through the apportionment process.
OMB continued its use of a Management Watch List in the recently released  President’s Budget for Fiscal Year 2006. The President’s Budget for Fiscal  Year 2006 includes 1,087 IT projects, totaling about $65 billion. Of this total  number of projects, OMB reported in the budget that 342 projects,  representing about $15 billion, are on the fiscal year 2006 Management  Watch List.
section_title: Objectives, Scope, and Methodology
paragraphs: Our objectives were to describe and assess OMB’s processes for (1) placing  projects on its Management Watch List and (2) following up on corrective  actions established for projects on the list.
To examine OMB’s processes for developing the list, we requested a copy  of the Management Watch List; we reviewed related OMB policy guidance,  including its Circular A-11 and Capital Programming Guide, as well as the  Analytical Perspectives for the President’s Budget submissions for fiscal  years 2005 and 2006; and we interviewed OMB analysts and their managers,  including the Deputy Administrator of OIRA and the Chief of the  Information Technology and Policy Branch, to identify the processes and  criteria they have in place to determine which IT projects to include on the  Management Watch List.
To examine OMB’s follow-up procedures on corrective actions established  for IT projects on the list, we reviewed related policy guidance, including  section 300 of Circular A-11 and OMB’s Capital Programming Guide. We  analyzed OMB’s apportionment documentation, specifically the Standard  Form 132 (Apportionment and Reapportionment Schedule), which  documented special apportionments that specified conditions that had to  be met before the agencies could receive funds. In addition, we interviewed  OMB officials and analysts and reviewed testimony and laws affecting the  management of IT investments, such as the Clinger-Cohen Act.
We conducted our work at OMB headquarters in Washington, D.C., from  August 2004 through March 2005, in accordance with generally accepted  government auditing standards.
section_title: OMB Established Processes and Criteria for Identifying Weak Projects, but It Did Not Use an Aggregate List to Perform Its Analysis or Oversight
paragraphs: According to OMB officials, including the Deputy Administrator of OIRA  and the Chief of the Information Technology and Policy Branch, OMB staff  identified projects for the Management Watch List through their evaluation  of the exhibit 300s that agencies submit for major IT projects as part of the  budget development process. This evaluation is carried out as part of  OMB’s responsibility for helping to ensure that investments of public  resources are justified and that public resources are wisely invested.
The OMB officials added that their analysts evaluate agency exhibit 300s by  assigning scores to each exhibit 300 based on guidance presented in OMB  Circular A-11. According to this circular, the purpose of the scoring is to  ensure that agency planning and management of capital assets are  consistent with OMB policy and guidance.
As described in Circular A-11, the scoring of a business case consists of  individual scoring for 10 categories, as well as a total composite score of all  the categories. The 10 categories are   project (investment) management,   enterprise architecture,   performance-based management system (including the earned value   life-cycle costs formulation, and   support of the President’s Management Agenda.
According to Circular A-11, scores range from 1 to 5, with 5 indicating  investments whose business cases provided the best justification and 1 the  least. For investments with average scores of 3 or below, OMB may ask  agencies for remediation plans to address weaknesses in their business  cases.
OMB officials said that, for fiscal year 2005, an IT project was placed on the  Management Watch List if its exhibit 300 business case received a total  composite score of 3 or less, or if it received a score of 3 or less in the areas  of performance goals, performance-based management systems, or  security and privacy, even if its overall score was a 4 or 5. OMB reported  that agencies with weaknesses in these three areas were to submit  remediation plans addressing the weaknesses.
According to OMB management, individual analysts were responsible for  evaluating projects and determining which projects met the criteria to be  on the Management Watch List for their assigned agencies. To derive the  total number of projects on the list that were reported for fiscal year 2005,  OMB polled the individual analysts and compiled the numbers.
OMB officials said that they did not aggregate these projects into a single  list describing projects and their weaknesses. According to these officials,  they did not construct a single list of projects meeting their watch list  criteria because they did not see such an activity as necessary in  performing OMB’s predominant mission: to assist in overseeing the  preparation of the federal budget and to supervise agency budget  administration. Further, OMB officials stated that the limited number of  analysts involved enabled them to explore governmentwide issues using ad  hoc queries and to develop approaches to address systemic problems  without the use of an aggregate list. They pointed at successes in improving  IT management, such as better compliance with security requirements, as  examples of the effectiveness of their current approach.
Nevertheless, OMB has not fully exploited the opportunity to use its  Management Watch List as a tool for analyzing IT investments on a  governmentwide basis. According to the Clinger-Cohen Act, OMB is  required to establish processes to analyze, track, and evaluate the risks and  results of major IT capital investments by executive agencies, which  aggregation of the Management Watch List would facilitate. Without  aggregation, the list’s visibility was limited at more senior levels of OMB,  constraining its ability to conduct analysis of IT investments on a  governmentwide basis and limiting its ability to identify and report on the  full set of IT investments requiring corrective actions.
section_title: OMB’s Follow-up on Projects Was Inconsistent, and Follow-up Activities Were Not Tracked Centrally
paragraphs: OMB did not develop a structured, consistent process or criteria for  deciding how to follow up on corrective actions that it asked agencies to  take to address weaknesses associated with projects on the Management  Watch List. Instead, OMB officials, including the Deputy Administrator of  OIRA and the Chief of the Information Technology and Policy Branch, said  that the decision on whether and how to follow up on a specific project  was typically made jointly between the OIRA analyst and the RMO program  examiner who had responsibility for the individual agency, and that follow- up on specific projects was driven by a number of factors, only one of  which was inclusion on the Management Watch List.
These officials also said that the decision for follow-up was generally  driven by OMB’s predominant mission to assist in budget preparation and  to supervise budget administration, rather than strictly by the perceived  risk of individual projects. According to these officials, those Management  Watch List projects that did receive specific follow-up attention received  feedback through the passback process, through targeted evaluation of  remediation plans designed to address weaknesses, and through the  apportioning of funds so that the use of budgeted dollars was conditional  on appropriate remediation plans being in place. These officials also said  that follow-up of some Management Watch List projects was done through  quarterly e-Gov Scorecards.
OMB officials also stated that those Management Watch List projects that  did receive follow-up attention were not tracked centrally, but only by the  individual OMB analysts with responsibility for the specific agencies. For  example, if an agency corrected a deficiency or weakness in a specific area  of the exhibit 300 for a Management Watch List project, that change was  not recorded centrally. Accordingly, OMB could not readily tell us which of  the 621 watch list projects for fiscal year 2005 were followed up on, nor  could it use the list to describe the relationship between its follow-up  activities and the changes in the numbers of projects on the watch list  between fiscal year 2005 (621 projects) and fiscal year 2006 (342). Further,  because OMB did not trace follow-up centrally, senior management could  not report which projects received follow-up attention and which did not.
OMB does not have specific criteria for prioritizing follow-up on  Management Watch List projects. Without specific criteria, OMB staff may  be agreeing to commit resources to follow up on projects that did not  represent OMB’s top priorities from a governmentwide perspective. For  example, inconsistent attention to OMB priorities, such as earned value  management, could undermine the objectives that OMB set in these areas.  In addition, major projects with significant management deficiencies may  have continued to absorb critical agency resources.
In order for OMB management to have assurance that IT program  deficiencies are addressed, it is critical that corrective actions associated  with Management Watch List projects be monitored. Follow-up activities  are instrumental in ensuring that agencies address and resolve weaknesses  found in exhibit 300s, which may indicate underlying weaknesses in project  planning or management. Tracking these follow-up activities is essential to  enabling OMB to determine progress on both specific projects and  governmentwide trends. In addition, tracking is necessary for OMB to fully  execute its responsibilities under the Clinger-Cohen Act, which requires  OMB to establish processes to analyze, track, and evaluate the risks and  results of major capital investments made by executive agencies for  information systems. Without tracking specific follow-up activities, OMB  could not know whether the risks that it identified through its Management  Watch List were being managed effectively; if they were not, funds were  potentially being spent on poorly planned and managed projects.
section_title: Conclusions
paragraphs: By scoring agency IT budget submissions and identifying weaknesses that  may indicate investments at risk, OMB is identifying opportunities to  strengthen investments. This scoring addresses many critical IT  management areas and promotes the improvement of IT investments.  However, OMB has not developed a single, aggregate list identifying the  projects and their weaknesses, nor has it developed a structured,  consistent process for deciding how to follow up on corrective actions.  Aggregating the results at a governmentwide level would help OMB take  full advantage of the effort that it puts into reviewing business cases for  hundreds of IT projects. A governmentwide perspective could enable OMB  to use its scoring process more effectively to identify management issues  that transcend individual agencies, to prioritize follow-up actions, and to  ensure that high-priority deficiencies are addressed.
OMB’s follow-up on poorly planned and managed IT projects has been  largely driven by its focus on the imperatives of the overall budget process.  Although this approach is consistent with OMB’s predominant mission, it  does not fully exploit the insights developed through the scoring process,  and it may leave unattended weak projects consuming significant budget  dollars. The Management Watch List described in the President’s Budget  for Fiscal Year 2005 contained projects representing over $20 billion in  budgetary resources that could have remained at risk because of  inadequate planning and project management. Because of the absence of a  consistent and integrated approach to follow-up and tracking, OMB was  unable to use the Management Watch List to ascertain whether progress  was made in addressing governmentwide and project-specific weaknesses  and where resources should be applied to encourage additional progress.  Thus, there is an increased risk that remedial actions were incomplete and  that billions of dollars were invested in IT projects with planning and  management deficiencies. In addition, OMB’s ability to report to the  Congress on progress made in addressing critical issues and areas needing  continued attention is limited by the absence of a consolidated list and  coordinated follow-up activities.
section_title: Recommendations for Executive Action
paragraphs: In order for OMB to take advantage of the potential benefits of using the  Management Watch List as a tool for analyzing and following up on IT  investments on a governmentwide basis, we are recommending that the  Director of OMB take the following four actions:  Develop a central list of projects and their deficiencies.
Use the list as the basis for selecting projects for follow-up and for   to guide follow-up, develop specific criteria for prioritizing the IT  projects included on the list, taking into consideration such factors  as the relative potential financial and program benefits of these IT  projects, as well as potential risks.
Analyze the prioritized list to develop governmentwide and agency  assessments of the progress and risks of IT investments, identifying  opportunities for continued improvement.
Report to the Congress on progress made in addressing risks of major IT  investments and management areas needing attention.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, OMB’s Administrator of the  Office of E-Government and Information Technology expressed  appreciation for our review of OMB’s use of its Management Watch List.  She noted that the report was narrowly focused on the Management Watch  List and the use of exhibit 300s in that context. She added that the report  did not address the more broad budget and policy oversight responsibilities  that OMB carries out or the other strategic tools available to OMB as it  executes those responsibilities. We agree that our review described and  assessed OMB’s processes for (1) placing the 621 projects representing  about $22 billion on its Management Watch List and (2) following up on  corrective actions established for projects on the list.
The Administrator commented that OMB’s oversight activities include the  quarterly President’s Management Agenda Scorecard assessment. We  acknowledge these activities in the report in the context of the e-Gov  scorecard, which measures the results of OMB’s evaluation of the agencies’  implementation of e-government criteria in the President’s Management  Agenda. We also agree with the Administrator that OMB is not the sole  audience of an exhibit 300. As we state in the report, an exhibit 300  justification is intended to enable an agency to demonstrate to its own  management, as well as to OMB, that it has employed the disciplines of  good project management, developed a strong business case for the  investment, and met other Administration priorities in defining the cost,  schedule, and performance goals proposed for the investment.
The Administrator disagreed with our finding that OMB did not have  specific criteria for prioritizing follow-up on exhibit 300s that have been  included on the Management Watch List. She explained that OMB  establishes priorities on a case-by-case basis within the larger context of  OMB’s overall review of agency program and budget performance.  However, our review showed that OMB did not develop a structured,  consistent process or criteria for deciding how it should follow up on  corrective actions that it asked agencies to take to address the weaknesses  of the projects on the Management Watch List. Accordingly, we continue to  believe that OMB should specifically consider those factors that it had  already determined were critical enough that they caused an investment to  be included in the Management Watch List. Without consistent attention to  those IT management areas already deemed as being of the highest priority  by OMB, the office risks focusing on areas of lesser importance.
We agree with the Administrator’s separate point that agencies have the  responsibility for ensuring that investments on the Management Watch List  are successfully brought up to an acceptable level. The follow-up that we  describe in our report consists of those activities that would allow OMB to  ascertain that the deficient investments have, in fact, been successfully  strengthened. We note in the report that the quarterly President’s  Management Agenda Scorecard plays a role in this activity (in the report,  we refer to the e-Gov Scorecard, which contributes to the Management  Agenda Scorecard).
Finally, the Administrator disagrees with our assessment that an  aggregated governmentwide list is necessary to perform adequate oversight  and management, and that OMB does not know whether risks are being  addressed. However, our review indicated that OMB was unable to easily  determine which of the 621 investments on the Management Watch List  remained deficient or how much of the $22 billion cited in the President’s  Budget remained at risk. In our assessment we observed that OMB had  expended considerable resources in the scoring of all exhibit 300s and the  identification of investments requiring corrective action, but that it never  committed the additional resources that would be required to aggregate the  partial management watch lists held by each individual analyst. Because no  complete Management Watch List was formed, OMB lost the opportunity to  analyze the full set of deficient investments as a single set of data. This  undermined its ability to assess governmentwide trends and issues. In  addition, the lack of a complete Management Watch List necessarily  inhibited OMB’s ability to track progress overall and to represent the full  set of investments requiring corrective action. We continue to believe that  these activities could be facilitated by an aggregate Management Watch  List.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  report date. At that time, we will send copies to other interested  congressional committees and to the Director of the Office of Management  and Budget. We also will make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site at  www.gao.gov.
Should you or your offices have questions on matters discussed in this  report, please contact me at (202) 512-9286, or Lester P. Diamond, Assistant  Director, at (202) 512-7957. We can also be reached by e-mail at  pownerd@gao.gov, or diamondl@gao.gov, respectively. Key contributors to  this report were William G. Barrick, Barbara Collier, Sandra Kerr, and Mary  Beth McClanahan.