Information Technology: Federal Agencies Need to Strengthen Investment Board Oversight of Poorly Planned and Performing Projects
Published: 2009-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government expects to spend about $71 billion for information technology (IT) projects for fiscal year 2009. Given the amount of money at stake, it is critical that these projects be planned and managed effectively to ensure that the public's resources are being invested wisely. This includes ensuring that they receive appropriate selection and oversight reviews. Selection involves identifying and analyzing projects' risks and returns and selecting those that will best support the agency's mission needs; oversight includes reviewing the progress of projects against expectations and taking corrective action when these expectations are not being met. GAO was asked to determine whether (1) federal departments and agencies have guidance on the role of their department-level investment review boards in selecting and overseeing IT projects and (2) these boards are performing reviews of poorly planned and poorly performing projects. In preparing this report, GAO reviewed the guidance of 24 major agencies and requested evidence of department-level board reviews for a sample of 41 projects that were identified as being poorly planned or poorly performing.
section_title: What GAO Found
paragraphs: The 24 major federal agencies have guidance calling for department-level investment review boards to select and oversee IT investments. However, while all of the agencies had department-level boards, the board membership for the Departments of Commerce and Labor did not include business unit (i.e., mission) representation as called for by IT investment management best practices. Without business unit representation on their department-level boards, these agencies will not have assurance that the boards include those executives who are in the best position to make the full range of investment decisions necessary for them to carry out their missions most effectively. About half of the projects GAO examined did not receive selection or oversight reviews. Specifically, 12 of the 24 projects GAO reviewed that were identified by OMB as being poorly planned (accounting for $4.9 billion in the President's fiscal year 2008 budget request or two-thirds of the funding represented by the 24 projects) did not receive a selection review, and 13 of 28 poorly performing projects GAO reviewed (amounting to about $4.4 billion or 93 percent of the funding represented by the 28 projects) did not receive an oversight review by a department-level board. Agencies provided several reasons for not performing department-level board reviews, including some which were not consistent with sound management practices. Furthermore, 6 of the 11 projects in the sample identified as being both poorly planned and poorly performing, with over $3.7 billion in funding in the President's fiscal year 2008 budget request, received neither a selection review nor an oversight review. Without consistent involvement of department-level review boards in selecting and overseeing projects that have been identified as poorly planned or poorly performing, agencies incur the risk that these projects will not improve, potentially leading to billions of federal taxpayer dollars being wasted.

section_title: Letter
paragraphs: Federal government expenditures for information technology (IT)  investments have exceeded $60 billion each year since fiscal year 2004,  and the government expects to spend about $71 billion for IT projects in  fiscal year 2009. Given the amount of money at stake, it is critical that IT  projects be planned and managed effectively to ensure that the public’s  resources are being invested wisely.
To this end, the Office of Management and Budget (OMB), which plays a  key role in directing and overseeing the federal government’s IT  investments, established a Management Watch List of major IT projects  identified as poorly planned and also required the major federal  departments and agencies to identify high-risk projects that are  performing poorly. In addition, GAO and OMB have long endorsed having  agencies establish a disciplined process for their executives to participate  in selecting and overseeing projects, among other things. Selecting  projects involves identifying and analyzing risks and returns before  committing any significant funds to them and selecting those that will best  support the agency’s mission needs. Overseeing projects involves  reviewing the progress of projects against expectations and taking  corrective action when these expectations are not being met.
Given the large number and dollar value of projects that are identified as  being poorly planned and poorly performing every year, you asked us to  determine whether (1) federal departments and agencies have guidance on  the role of their department-level investment review boards (IRB) in  selecting and overseeing IT projects and (2) these boards are actually  performing selection and oversight reviews of poorly planned and poorly  performing projects.
To address the first objective, we reviewed the investment management  guidance of 24 major agencies to determine the role department-level  IRBs are expected to play in selecting and overseeing IT projects, updating  the findings from our 2004 governmentwide review of agencies’ use of key  investment management practices. We also reviewed the composition of  the boards to determine whether they included senior executives from  both IT and business units. To address the second objective, we identified  a sample of 48 (subsequently reduced to 41) projects that were identified  as being poorly planned according to OMB’s Management Watch List or  reported as being poorly performing on the High-Risk List. For each  project, we requested and analyzed evidence of department-level IRB  reviews during the time period when the projects were on the OMB lists.
We conducted this performance audit from January 2008 to June 2009 in  Washington, D.C., in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives. Further details on  our objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: OMB plays a key role in helping federal agencies manage their IT  investments by working with them to better plan, justify, and determine  how much they need to spend on IT projects and how to manage approved  projects. In particular, the Clinger-Cohen Act of 1996 requires OMB to  establish processes to analyze, track, and evaluate the risks and results of  major capital investments in information systems made by federal  agencies and report to Congress on the net program performance benefits  achieved as a result of these investments. In addition, the Clinger-Cohen  Act places responsibility for managing IT investments with the heads of  agencies and establishes chief information officers to advise and assist  agency heads in carrying out this responsibility.
To help carry out its oversight role and assist the agencies in carrying out  their responsibilities, OMB developed its Management Watch List in 2003  and its High-Risk List in 2005 to focus executive attention and to ensure  better planning and tracking of the major IT investments. The Management  Watch List identifies projects at federal agencies that are poorly planned,  i.e., projects with weaknesses in their funding justifications, which are  known as exhibit 300s. Because of the focus on the funding justifications,  projects on the Management Watch List specifically concern the process  by which agencies select projects to invest in. OMB places projects on the  High-Risk List when they require special attention from oversight  authorities and the highest level of agency management. These projects  are not necessarily “at risk” of failure, but may be on the list because of  one or more of the following four reasons:    The agency has not consistently demonstrated the ability to manage  complex projects.
The project has exceptionally high development, operating, or  maintenance costs, either in absolute terms or as a percentage of the  agency’s total IT portfolio.
The project is being undertaken to correct recognized deficiencies in the  adequate performance of an essential mission program or function of the  agency, a component of the agency, or another organization.
Delay or failure of the project would introduce for the first time  unacceptable or inadequate performance or failure of an essential mission  function of the agency, a component of the agency, or another  organization.
The High-Risk List also includes projects that are performing poorly (i.e.,  high-risk projects with reported performance shortfalls). High-risk  projects are identified as having performance shortfalls if one or more of  the following performance evaluation criteria are not met: (1) establishing  baselines with clear cost, schedule, and performance goals; (2)  maintaining the project’s cost and schedule variances within 10 percent;  (3) assigning a qualified project manager; and (4) avoiding duplication by  leveraging inter-agency and governmentwide investments. Projects on the  High-Risk List, therefore, require disciplined and effective oversight to  ensure that performance shortfalls, if any, are addressed.
GAO, Information Technology: OMB and Agencies Need to Improve Planning,  Management, and Oversight of Projects Totaling Billions of Dollars, GAO-08-1051T  (Washington, D.C.: July 31, 2008). projects (totaling about $3 billion) were considered both poorly planned  and poorly performing.
OMB took several steps to address our recommendations to improve the  identification and oversight of Management Watch List and High-Risk List  projects; however, further action is needed, including, for example,  identifying the deficiencies (i.e., performance shortfalls) associated with  the high-risk projects. On April 28, 2009, we testified that the future of the  Management Watch List and High-Risk List was uncertain because OMB  officials stated that they had not decided if the agency plans to continue to  use these lists. We noted that OMB needs to decide if it is going to  continue to use the Management Watch List and High-Risk List and, if not,  that OMB should promptly implement other appropriate mechanisms to  help direct and oversee IT investments in the future. In response, the  Federal Chief Information Officer testified that OMB would determine  how to better oversee poorly planned and performing projects by the end  of June 2009.
subsections: section_title: Investment Management Framework Calls for Boards to Select and Oversee IT Investments
paragraphs: Federal agencies face significant challenges in planning for and managing  their IT systems and networks. These challenges can be addressed, in part,  by the use of systematic management processes to select, control, and  evaluate the investments. To further support the implementation of such  processes, we developed an IT investment management (ITIM)  framework for agencies to use. It is based on our research of IT  investment management practices of leading private and public sector  organizations and can be used to determine both the status of an agency’s  current IT investment management capabilities and the additional steps  that are needed to establish more effective processes. The framework  consists of progressive stages of maturity for any given organization  relative to its selection and oversight responsibilities. We have used the  framework in many of our reports, and a number of agencies have  adopted it.
The ITIM maturity framework cites the establishment of “one or more IT  investment management boards” as a fundamental step in establishing a  mature capital planning process. The framework states that a  departmentwide IT investment review board (IRB) composed of senior  executives from both IT and business units should be responsible for  defining and implementing the department’s IT investment governance  process. This department-level IRB is to provide selection and oversight of  department IT projects to ensure that the department’s portfolio of  projects meets mission needs at expected levels of cost and risk. Selecting  projects involves identifying and analyzing projects’ risks and returns  before committing any significant funds to them and selecting those that  will best support the agency’s mission needs; overseeing projects involves  reviewing the progress of projects against expectations and taking  corrective action when these expectations are not being met.
To ensure that agencies’ department-level boards are using a disciplined  selection and oversight process, the ITIM framework also states that,  among other things, the department-level board should: select new  investments and reselect ongoing investments; perform regular reviews of  each project’s performance against stated expectations; and receive data  associated with a project’s actual performance (including cost, schedule,  benefit, and risk performance). Importantly, according to the ITIM  framework, while these functions can be performed by subordinate  boards, the department-level IRBs must maintain ultimate responsibility  for and visibility into the subordinate boards’ activities.
section_title: Prior Reviews Have Identified Weaknesses in Executive-Level Board Involvement in Selection and Oversight
paragraphs: We have previously reported that federal agencies face challenges in  effectively managing their IT investments. Specifically, in January 2004, we  reported that, although most of the major agencies in our review had IRBs  responsible for defining and implementing their investment management  processes, the agencies did not always have the mechanisms in place for  these boards to effectively control their investments. We made  recommendations to the agencies regarding those practices that were not  fully in place. More recently, in 2008, we reported that the Social Security  Administration had not fully developed policies and procedures for  management oversight of its IT projects and systems, such as elevating  problems to the department-level IRB. We also reported that the Social  Security Administration had not tracked corrective actions for  underperforming investments and had not reported the actions to the  department-level IRB. To address these weaknesses, we recommended  that the agency strengthen and expand the board’s oversight  responsibilities for underperforming projects and evaluations of projects  and establish a mechanism for tracking corrective actions for  underperforming investments.
section_title: Major Federal Agencies Have Guidance for Selection and Oversight of IT Investments, but Two Agency Boards Lack Business Unit Representation
paragraphs: The 24 major federal agencies have guidance calling for department-level  IRBs to select and oversee IT investments pursuant to OMB guidance  required by the Clinger-Cohen Act, and specified in practices laid out in  the ITIM framework. However, while all of the agencies had department- level IRBs, the board membership for two agencies did not include  business unit (i.e., mission) representation.
subsections: section_title: Agency Guidance Calls for Department-Level IRBs to Select Projects
paragraphs: Each of the agencies had documented guidance that called for a  department-level IRB to perform selection of the projects to be included in  the agency’s IT investments. For example, according to the Department of  the Treasury’s guidance, its department-level IRB is to consider investment  scoring results and recommendations that are provided to it by the Chief  Information Officer Council (a subordinate board) and select which  investments will be included in Treasury’s IT investment portfolio. The  Department of Transportation’s recently issued IT investment  management policy delegates responsibility for project selection, as well  as project oversight, to its component-level investment review boards, but  requires its components to establish and/or document the existence of  their boards, specifies the roles and responsibilities these boards are to  have, and establishes specific metrics to be used by the department-level  IRB to measure the performance of the component boards.
section_title: Agency Guidance Calls for Department-Level IRBs to Oversee Projects
paragraphs: As with project selection, each of the agencies had documented guidance  that called for the department-level IRB to conduct an oversight reviews of  projects, and the frequency of these reviews varied (see fig. 1 for a  breakdown of the frequency of oversight reviews specified in agencies’  guidance).
For 20 of the 24 agencies, the guidance allowed the delegation of oversight  reviews to other entities. In these cases, the agencies had guidance in  place to help ensure that these other entities were effectively carrying out  their responsibilities. At the remaining four agencies—the National  Science Foundation, Small Business Administration, Department of State,  and the U.S. Agency for International Development —project oversight  was to be primarily performed by the department-level IRB. By having  guidance specifying department-level IRB selection and oversight of  projects, agencies recognize the importance of involving those who have  the ultimate responsibility and accountability for the organization’s  success in key project decisions.
section_title: Two Agencies’ Department- Level Boards Lack Business Unit Representation
paragraphs: It should be noted, however, that while all of the agencies had guidance  requiring department-level IRBs to be responsible for selecting and  overseeing projects, the boards at the Departments of Commerce and  Labor did not include senior executives from business units (e.g., line or  mission units) as called for in the ITIM framework. Specifically, these  boards consisted of executives from IT and other department mission  support units, such as the Chief Financial Officer, Director of Budget, or  Controller, as well as administrative officers, but did not have appropriate  line or mission representation from the organizations’ business units. We  have previously reported that because allocating resources among major  IT investments may require fundamental trade-offs among a multitude of  business objectives, portfolio management decisions are essentially  business decisions and therefore require sufficient business representation  on the department-level IRB.
The two agencies with boards that did not include senior executives from  business units offered the following rationales for this practice.
The Department of Commerce reported that it does not include  nontechnical program representatives on its department-level IRB because  it would be impractical to have fair representation of all 12 of the major  agencies and the dozens of major programs comprising the department. In  addition, Commerce reported that it is run on a federated basis, putting  responsibility on each of the department’s operating units to prioritize its  own investments in determining which should be reviewed by the  department. Finally, Commerce stated that it does not prioritize among  investments from its different operating units; instead, departmental  officials work with each operating unit to ensure that the investment and  investment strategy being recommended is optimum for meeting that  operating unit’s mission. We have previously reported that using this  approach of giving responsibility to subordinate units should include  appropriate department-level involvement, either through review and  approval of their investments that meet certain criteria or through  awareness of the subordinate unit’s investment management activities.  We believe that this corporate visibility should be provided by a board  composed of executives from both business and IT units to ensure that  decisions made are in the best interest of the entire department. In  addition, while Commerce’s practice may not be to prioritize among the  investments at the department level, the department has ultimate  responsibility for the success of its operating units’ investments and the  department-level IRB should therefore include business representation to  ensure that decisions made are in the best interest of the agency.
The Department of Labor reported that the senior IT and administrative  executives who serve on its department-level IRB, have in-depth, detailed,  and expert knowledge of their units’ missions and business objectives and  are capable of representing their units’ interests. However, we have  previously reported that IT and administrative executives responsible for  mission support functions do not constitute sufficient business  representation because, by virtue of their responsibilities, they are not in  the best position to make business decisions.
Until these agencies adjust their board memberships to include  representation from their business units, they will not have assurance that  the department-level IRB includes those executives who are in the best  position to make the full range of decisions needed to enable the agency to  carry out its mission most effectively.
section_title: Many Projects Did Not Receive a Department-Level IRB Selection or Oversight Review
paragraphs: Although all the major agencies had guidance calling for a department- level IRB selection or oversight review, many of the projects we examined  did not receive one of these reviews. Specifically, 12 of the 24 projects  identified by OMB as being poorly planned in 2007 (accounting for about  $4.9 billion) did not receive a selection review, and 13 of 28 poorly  performing projects in 2007 (amounting to about $4.4 billion) did not  receive an oversight review by the department-level IRB. Furthermore, 6 of  the 11 projects identified as being both poorly planned and poorly  performing, with nearly $3.7 billion in funding in the President’s fiscal year  2008 budget request, received neither a selection review nor an oversight  review.
subsections: section_title: Half of the Poorly Planned Projects Did Not Receive a Selection Review by a Department-Level IRB
paragraphs: Of the 24 poorly planned projects in 2007 that we reviewed, 12 projects did  not receive a selection review, while 12 were reviewed by the department- level IRB. The requested funding level for these 24 poorly planned  projects was about $7.3 billion. The 12 projects that were reviewed by a  department-level IRB accounted for approximately $2.4 billion, while the  12 projects not reviewed accounted for about $4.9 billion, about two thirds  out two thirds  of the total requested funding for the 24 projects (see fig. 2 and table 1).  of the total requested funding for the 24 projects (see fig. 2 and table 1).
We assessed five projects as not having received department-level IRB  selection reviews because the agencies did not provide evidence of such  reviews. Agencies offered varying reasons for why selection reviews had  not been performed for the remaining seven. Table 1 shows whether  projects we reviewed received a selection review from the department- level IRB and lists reported reasons why no review was performed, where  applicable.
Following are details on the reasons why the 12 projects did not receive a  department-level IRB review:    A project belonging to Commerce’s USPTO was not reviewed by the  department-level IRB, according to the agency, because the USPTO is a  performance-based organization (PBO), and therefore its projects are not  required to be reviewed by the department-level IRB. According to the  legislation that established the USPTO as a PBO, the office is subject to  the policy direction of the Secretary of Commerce, but it otherwise retains  responsibility for decisions regarding the management and administration  of its operations and exercises independent control of its budget  allocations and expenditures, personnel decisions and processes,  procurements, and other administrative and management functions.
According to the Department of Education, the Common Services for  Borrowers project did not receive a selection review by the department- level board because it is under the oversight of the Federal Student Aid  Executive Leadership Team. In written comments on a draft of this report,  however, the department stated that it plans to bring all of its IT  investments under the department-level board’s oversight.
The Department of Homeland Security did not provide evidence of a  selection review for its two projects but noted that it was reengineering its  investment management process to include department-level IRB reviews  of projects at key milestone decision points.
Although NASA stated that its three projects were governed by oversight  bodies, the documentation provided did not show evidence that reviews  had been performed by the appropriate department-level review board.
At the Nuclear Regulatory Commission, a lower-level board performed the  selection reviews. According to the agency’s guidance, the department- level board should have performed the reviews. It stated that this board  only gets involved when the lower-level board believes issues need to be  elevated. However, NRC’s guidance does not specify when issues need to  be elevated to the department-level IRB. In addition, the agency did not  provide any examples of cases when issues had been elevated to the  department-level IRB.
Officials from the Department of Transportation’s Office of the Chief  Information Officer could not provide a reason why a department-level  board selection review of its projects had not been performed. In  commenting on a draft of this report, the agency stated that it planned to  have this project reviewed in detail by its departmental-level board.
The Department of the Treasury’s projects did not receive a department- level IRB selection review because this board was not active during the  time frame we considered during our review. The department, however,  has since then reestablished its department-level IRB.
section_title: About Half of the Poorly Performing Projects Did Not Receive an Oversight Review by the Department- Level IRB
paragraphs: About half of the poorly performing projects in 2007 we reviewed did not  receive an oversight review by a department-level IRB. Of the 28 projects,  13 did not receive an oversight review by the department-level IRB, while  15 did. The President’s requested fiscal year 2008 funding for the 28  projects totaled approximately $4.7 billion. The 15 projects that received a  review represented approximately $0.3 billion, or 7 percent of the total  $4.7 billion funding request, while the 13 poorly performing projects that  were not reviewed totaled nearly $4.4 billion, or 93 percent of the total  requested funding. (See fig. 3 and table 2.)
Table 2 shows whether projects received oversight reviews, as well as  reported reasons why no review was performed, where applicable.
Agencies provided several reasons why the 13 projects did not receive  oversight reviews, including some which were not consistent with sound  management practices:    One Defense project’s funding was below the financial threshold required  for a review by the department-level IRB, consistent with the agency’s  guidance. However, in May 2007 and May 2009, we reported that DOD’s  guidance and practices did not provide for sufficient oversight and  visibility into component-level investment management activities,  including component reviews of investments such as this project. We  made recommendations to DOD to address these weaknesses, which DOD  has yet to fully implement.
Another Defense project was reportedly being rebaselined (meaning that  its cost, schedule, and performance goals were being modified to reflect a  change in the scope of the work) and therefore had not received a review  by the department-level IRB. This project, however, continues to be  funded and therefore could have benefited from a department-level  oversight review.
According to the Department of Education, the two projects we reviewed  did not receive oversight reviews by the department-level IRB because  they were under the oversight of the Federal Student Aid Executive  Leadership Team. As noted earlier, in written comments on a draft of this  report, the department stated it plans to bring all of its IT investments  under the department-level board’s oversight.
While DHS provided evidence that a lower-level board had agreed to  submit the DHS-Infrastructure project to the department-level IRB for  review, the agency did not provide evidence that this review had been  performed. The department also stated that SBInet and US-VISIT projects  had received an oversight review by the department-level IRB, but did not  provide sufficient evidence to support this, including information  presented to the board for review. In March 2009, however, DHS officials  told us that they had recently made changes to their investment review  process and, as part of these changes, were planning to improve the  documentation associated with department-level IRB reviews.
A Nuclear Regulatory Commission project should have received a review  by the department-level IRB according to the agency’s guidance, but  officials told us that, in practice, this board only gets involved when the  lower-level board elevates issues. However, agency officials were unable  to provide us with any examples where the lower-level board had elevated  issues about the project to the IRB.
The Department of the Treasury’s projects did not receive a department- level IRB oversight review because this board was not active during the  time frame we considered during our review. The department, however,  has since then reestablished its department-level IRB.
According to the U.S. Agency for International Development, its project  did not receive an oversight review because it has not been able to  proceed due to lack of funding. We agree that an oversight review was not  warranted since there was no activity on the project.
A Veterans Affairs project was not reviewed because the IRB is not  required to review projects in the operations and maintenance stage.  Instead, oversight of projects in this stage is the responsibility of the Office  of the Chief Information Officer. However, the IRB does not oversee this  office’s review activities. According to the ITIM framework, boards should  ensure projects are reviewed throughout their life cycle. In addition, they  must maintain ultimate responsibility for and visibility into the activities of  groups that carry out their functions.
section_title: About Half of the Projects That Were Both Poorly Planned and Poorly Performing Received Neither a Selection Review Nor an Oversight Review
paragraphs: Six of the 11 projects that were identified as being both poorly planned  and poorly performing in 2007 did not receive a selection or an oversight  review by the departmental-level IRB. Funding requests for fiscal year  2008 for these 6 projects accounted for about $3.7 billion (see table 3).
Without consistent involvement of department-level IRBs in selecting and  overseeing projects that have been identified as poorly planned or poorly  performing, agencies incur the risk that these projects will not improve, which  could lead to potentially billions of federal taxpayer dollars being wasted.
section_title: Conclusions
paragraphs: Department-level investment review boards’ involvement in selecting and  overseeing their agencies’ IT projects is critical to ensuring that these  projects meet mission needs and that federal funds are not wasted. To  their credit, the 24 major federal agencies have established guidance  calling for department-level boards to perform project selection and  oversight reviews. However, department-level boards for two agencies did  not include representation from their business units and therefore did not  have assurance that the board included all of the executives who are in the  best position to make the full range of decisions needed to enable the  agency to carry out its mission most effectively.
While having selection and oversight guidance is a good step, it is only  worthwhile if effectively implemented. The fact that many poorly-planned or  performing projects were not reviewed by department-level boards is  particularly alarming considering that they represent, in total, about $6 billion in  funding and that the Management Watch List and High-Risk List were  established specifically to draw management attention to such projects. Until  agencies ensure that their department-level review boards are consistently  involved in selecting and overseeing these projects, they will continue to incur  the risk that the projects will not improve and that potentially billions of federal  taxpayer dollars will be wasted.
section_title: Recommendations for Executive Action
paragraphs: To ensure that IT projects are effectively managed, we are making  recommendations to the agencies whose practices were not consistent  with sound management practices. Specifically, we recommend that  the Secretaries of Commerce and Labor ensure their department-level  review boards include business unit (i.e., mission) representation;  the Chairman of the Nuclear Regulatory Commission direct the Executive  Director for Operations to define conditions for elevating issues related to  project selection and oversight to its department-level IRB; and  the Secretary of Veterans Affairs define and implement responsibilities for  the department-level IRB to oversee projects in operations and  maintenance.
In addition, we are recommending that the Secretaries of the Departments  of Defense, Education, Homeland Security, Transportation, Treasury, and  Veterans Affairs, the Administrator for the National Aeronautics and Space  Administration, the Chairman of the Nuclear Regulatory Commission, and  the Administrator for the U.S. Agency for International Development  ensure that the projects that are identified in this report as not having  received departmental-IRB selection or oversight reviews receive these  reviews.
section_title: Agency Comments and Our Evaluation
paragraphs: We sent a draft of this report to the 24 major agencies and received a  response from 20. Of these 20, 15 provided comments, and 5 stated they  did not have any comments (we had not made any recommendations to  these agencies, which were the Department of Health and Human  Services, the Department of State, the Environmental Protection Agency,  the National Science Foundation, and the Office of Personnel  Management). Of the 15 agencies that provided comments, 11 generally  agreed with our recommendations, and 1 (the Department of Justice) did  not. Three agencies (the Department of Housing and Urban Development,  the Department of the Interior, and the Social Security Administration)  provided views on various aspects of our report. Several agencies also  provided technical comments, which we incorporated as appropriate.
The agencies’ comments and our response are summarized below:  In written comments on a draft of the report, the Department of Commerce’s  Chief Information Officer, addressing our recommendation that the  department ensure that its department-level review board include business  unit (i.e. mission) representation, stated that the department had modified the  membership structure of its investment review board to provide operating  unit management with latitude in identifying senior managers most able to  provide effective representation and, as a result had broadened its  membership to include chief financial officers from certain operating units as  well as the Deputy Director of the Bureau of the Census. The Department of  Commerce’s comments are printed in appendix II.
In written comments on a draft of the report, the Department of Defense’s  Deputy Chief Information Officer concurred with our recommendation to  ensure that the Defense Information System for Security receive an  oversight review, stating that, going forward, it will ensure that the project  receives all required IRB reviews. The department partially concurred with  our recommendation to ensure its Integrated Acquisition Environment  Shared Services Provider-Past Performance Information Retrieval System  receive an oversight review, stating, as indicated in the report, that the  project is below the threshold required for department-level IRB oversight.  The department stated, however, that the project will be brought before  the appropriate department-level IRB for compliance review if, and when  it meets the financial threshold. The department also provided technical  comments which we have incorporated as appropriate. The Department of  Defense’s comments are printed in appendix III.
In written comments on a draft of the report, the Department of  Education’s Chief Information Officer, agreed with our recommendation  to ensure that the two projects we identified in the report as not having  received departmental-level IRB selection or oversight reviews receive  such reviews, stating that the IRB will review the investments, render  decisions as appropriate, and incorporate the results in the IT portfolio  currently under review. The department also noted that, while the projects  we reviewed were under the oversight of the Federal Student Aid’s  Executive Leadership Team, they would be brought under the  department’s oversight along with all other investments. The department  disagreed with the statement that the projects reviewed did not receive a  selection or oversight review, stating that they had been selected and  reviewed by the Federal Student Aid’s Executive Leadership Team. In our  report, we have clarified the discussion of these reviews by the Executive  Leadership Team where appropriate. The Department of Education’s  comments are reprinted in appendix IV.
In written comments on a draft of this report, the Department of  Homeland Security’s Director for Departmental GAO/OIG Liaison Office  agreed with the recommendation to conduct department-level reviews of  the three programs we reviewed and provided evidence of department  Acquisition Review Board reviews for these programs during fiscal year  2008. The department disagreed with the assertion that the department- level review boards were not active in overseeing the three projects we  examined during our review and provided decision memoranda—three of  which we had not been provided before—as evidence of reviews by the  boards in place for 2007, the time period we considered. However, in our  report, we do not state that the department-level boards were not active.  Rather, we note that the department did not provide sufficient evidence of  department-level IRB reviews. We did not change our assessments for the  three projects because the additional documentation received still did not  provide sufficient evidence documenting the 2007 reviews.
The documentation we have seen from more recent reviews more  completely documents departmental-level IRB reviews and we have noted  this in our report. The department also provided technical comments. The  department’s comments are reprinted in appendix V.
In written comments on a draft of this report, the Acting Chief Information  Officer of the Department of Housing and Urban Development stated that  the department-level IRB will maintain its disciplined process for program  executives to participate in selecting and overseeing projects. We did not  make any recommendations to the department. The Department of Housing  and Urban Development’s comments are reprinted in appendix VI.
In written comments on a draft of this report, the Department of the  Interior’s Deputy Assistant Secretary for Budget and Business  Management agreed with our conclusions that consistent involvement of  department-level review boards in selecting and overseeing projects,  particularly poorly performing projects, is important in safeguarding  federal taxpayer dollars. The department also asked that the definition of  high-risk projects reflect the fact that some investments designated as  such are performing within acceptable thresholds but require heightened  awareness and oversight by investment review boards because of their  importance. To address this comment, we have added OMB’s criteria for  designating projects as high-risk to our report background. We did not  make any recommendations to the Department of the Interior. The  Department of the Interior’s comments are reprinted in appendix VII.
In written comments on a draft of this report, the Department of Justice’s  Assistant Attorney General for Administration disagreed with our  recommendation that it ensure its department-level review board include  business unit representation and provided clarification on the role and  responsibilities of the Deputy Attorney General who chairs the board and  on the participation of component executives in the board’s  decisionmaking process. Based on this clarification, we agree that the  board provides adequate business unit representation. We have noted this  change in our report and removed the related recommendation. In its  comments, the department also took issue with our use of the term  “poorly performing” to characterize the projects we reviewed. We are not  implying as the department states that these projects are “near failing.” We  have clarified our use of the term in the report and, in the case of the  Sentinel project—which we have reviewed— acknowledged progress  made in managing the project. The Department of Justice’s comments are  reprinted in appendix VIII.
In written comments on a draft of this report, the Department of Labor’s  Assistant Secretary for Administration and Management addressed our  recommendation to ensure that its department-level review board include  business unit representation by acknowledging that the board does not  include senior executives from business units and stating that, while it  believes the executives on the board effectively represented the business  interests of their respective organizations, it will consider appropriate and  efficient steps for including senior executives from business units as part  of the board’s process. The Department of Labor’s comments are reprinted  in appendix IX.
In e-mail comments on a draft of this report, the Department of  Transportation’s Director of Audit Relations addressed our  recommendation to ensure that the projects we identified as not having  received department-level IRB selection or oversight reviews receive these  reviews by stating that actions are underway to schedule a summer IRB  meeting to review the entire budget year 2011 portfolio of IT investments,  and that the Combined IT Infrastructure investment which we reviewed is  expected to be reviewed in detail.
In written comments on a draft of this report, the Department of the  Treasury’s Deputy Assistant Secretary for Information Systems and Chief  Information Officer addressed our recommendation to ensure that the  projects we identified as not having received department-level IRB  selection or oversight reviews receive these reviews by noting recent  efforts to reconstitute a department-level Executive Investment Review  Board, increase the oversight role of its Chief Information Officer Council,  and remediate weaknesses associated with the three projects we  reviewed. The Department of the Treasury’s comments are reprinted in  appendix X.
In written comments on a draft of this report, the Secretary of the  Department of Veterans Affairs concurred with our recommendations to  define and implement responsibilities for the department-level IRB to  oversee projects in operations and maintenance by noting that the  Programming and Long Term Issues Board will include operational  programs/projects in its program reviews for fiscal year 2010. The  department also concurred with our recommendation to ensure that the  project which we identified as not having received department-level IRB  oversight reviews receive these reviews and stated that it will address  actions to ensure this in its plan to address our recommendations. The  Department of Veterans Affairs’ comments are reprinted in appendix XI.
In written comments on a draft of this report, the National Aeronautics  and Space Administration’s Associate Deputy Administrator partially  concurred with our recommendation that projects which are identified in  this report as not having received department-level IRB selection or  oversight reviews receive these reviews stating that the departmental  board will continue to review major IT investments that are not highly  specialized in nature (this includes two of the four projects we reviewed),  while another governing body will maintain responsibility for ensuring the  overall successful performance of NASA’s program portfolio, including the  highly specialized IT investments. We received information about the  second governing body after we sent our report to NASA for comment.  During the comment period, the agency also provided us additional  documentation on the projects we reviewed. After reviewing this  documentation, we have changed the reported reason column in table 1  from “department-level board was not active (i.e., it had not yet been  established)” to “NASA did not provide evidence that a selection review  had been performed by the appropriate department-level IRB” for the  three projects we reviewed for selection. In addition, we changed the  department-level IRB review column in table 2 for the Integrated Financial  Management Improvement program from a “no” to a “yes.” NASA’s  comments are reprinted in appendix XII.
In written comments on a draft of this report, the Nuclear Regulatory  Commission’s Deputy Executive Director for Corporate Management, Office  of the Executive Director for Operations, agreed with our recommendation  to define conditions for elevating issues related to project selection and  oversight to its department-level IRB stating that the commission will  review and enhance the existing guidance for project selection and  oversight to ensure that its process is compliant with the intent of the  Clinger-Cohen Act. This will include updating the Information Technology  Business Council charter for project oversight reviews to include any  necessary changes to the process or criteria for review by the Information  Technology Senior Advisory Council. The commission also agreed with our  recommendation to ensure that the National Source Tracking System which  we identified as not having received a selection or oversight review by the  department-level IRB receive such review. The Nuclear Regulatory  Commission’s comments are reprinted in appendix XIII.
In written comments on a draft of this report, the Commissioner of the  Social Security Administration asked that we remove the Information  Technology Operations Assurance project we reviewed from our report  because it is not a poorly planned or poorly performing project. During the  agency comment period, we informed the agency that we would be  removing the project from our sample, and, based on clarification provided  by the Associate Chief Information Officer that the project reported a  positive cost variance, agreed that it should not be considered poorly  performing. We did not make any recommendations to the agency. The  Social Security Administration’s comments are reprinted in appendix XIV.
In e-mail comments on a draft of this report, the U.S. Agency for  International Development concurred with our recommendation to ensure  that the project which we identified as not having received a department- level IRB oversight review receive this review. The agency noted, however,  that the review might not occur if the project is not funded.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to other interested  congressional committees, the Director of the Office of Management and  Budget, and other interested parties. The report also will be available at no  charge on the GAO Web site at http://www.gao.gov. Should you or your  offices have questions on matters discussed in this report, please contact  me at (202) 512-9286 or at pownerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made key contributions to this  report are listed in   appendix XV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether (1) federal  departments/agencies have guidance on the role of their department-level  investment review boards (IRB) in selecting and overseeing information  technology (IT) projects and (2) these boards are performing selection and  oversight reviews of poorly planned and performing projects.
To address the first objective, we reviewed the investment management  guidance (including policy documents and board charters) of each of 24  agencies listed in the Chief Financial Officers (CFO) Act of 1990 (referred  to in our report as “the 24 major agencies”). In reviewing the guidance, we  determined the role department-level IRBs are expected to play in  selecting and overseeing IT projects, updating the findings from our 2004  governmentwide review of agencies’ use of key investment management  practices. We also reviewed the composition of the boards to determine  whether they included senior executives from both IT and business (i.e.,  mission) units, in accordance with the GAO IT Investment Management  framework which identifies the key practices for creating and maintaining  successful investment management processes.
For the second objective, we selected a sample of 48 IT projects that were  identified as being poorly planned according to the Office of Management  and Budget’s Management Watch List  or reported as poorly performing  on the High-Risk Lists or both. To provide a governmentwide perspective,  we attempted to select one project from the 2007 Management Watch List  and one project from the High-Risk List with performance shortfalls  during 2007 for each of the 24 major agencies. We focused on the high-risk  projects with performance shortfalls in the areas of cost and schedule  since we had reported in September 2007 that these were the most  frequently reported shortfalls. To obtain broader representation of  agencies with high-risk projects, we also selected three High-Risk projects  that had performance shortfalls in 2006. From these lists, we selected  those projects with the highest funding levels according to the fiscal year  2008 President’s budget request. When an agency had a project on only  one of the lists (i.e., only the Management Watch List or High-Risk List),  we selected at least 2 projects from that list. For example, we selected 2  high-risk projects with shortfalls for the Environmental Protection Agency  because the agency did not have any projects on the Management Watch  List for the time frame we considered.
Our selection process resulted in 26 projects from the Management Watch  List, totaling about $7.4 billion in the fiscal year 2008 budget request, and  33 projects from the High-Risk List, totaling about $5.2 billion in the fiscal  year 2008 budget request. Eleven of these projects, totaling about $4  billion, were on both lists. The Department of Energy and the National  Science Foundation did not have any projects on the Management Watch  List or on the High-Risk List with shortfalls and, therefore, we did not  select any projects from these agencies. We removed two Management  Watch List projects and five high-risk projects from our initial sample after  sending the draft report to agency comment because we determined after  further review and discussion with agencies that these projects had not  been on the Management Watch List during 2007 or reported negative cost  or schedule variances exceeding 10 percent between December 2006 and  December 2007. This brought our sample of Management Watch List  projects to 24 projects, totaling about $7.3 billion in the fiscal year 2008  budget request and 28 high-risk projects totaling about $4.7 billion in the  fiscal year 2008 budget request and the number of projects on both lists to  11 projects totaling $4 billion in the fiscal year 2008 budget request.
To determine whether department-level IRBs were performing selection  and oversight reviews of poorly planned and performing projects, we  requested evidence of board reviews for the 48 projects in our sample  during the time they were either on the Management Watch List or High- Risk List. We analyzed the documentation obtained, and, when reviews  had not been performed, we followed up with agencies to determine why  the required reviews were not performed. For the oversight reviews, we  determined whether project cost, benefit, schedule and risk data had been  provided to the board, but we did not assess the reliability of this  information.
We conducted this performance audit from January 2008 to June 2009 in  Washington, D.C., in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Department of Education
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: Comments from the Department of Housing and Urban Development
section_title: Appendix VII: Comments from the Department of the Interior
section_title: Appendix VIII: Comments from the Department of Justice
section_title: Appendix IX: Comments from the Department of Labor
section_title: Appendix X: Comments from the Department of the Treasury
section_title: Appendix XI: Comments from the Department of Veterans Affairs
section_title: Appendix XII: Comments from the National Aeronautics and Space Administration
section_title: Appendix XIII: Comments from the Nuclear Regulatory Commission
section_title: Appendix XIV: Comments from the Social Security Administration
section_title: Appendix XV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Sabine R. Paul, Assistant  Director; William G. Barrick; Neil J. Doherty; Nancy E. Glover; Robert G.  Kershaw; Lee A. McCracken; Tomas Ramirez; and Kevin C. Walsh made  key contributions to this report.