Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories
Published: 2014-10-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO's reports over the past 4 years have found more than 90 areas where opportunities exist for the executive branch or Congress to better manage, reduce, or eliminate fragmentation, overlap, and duplication. GPRAMA calls for the creation of a list (inventory) of all federal programs, along with related budget and performance information, which could make it easier to determine the scope of the federal government's involvement in particular areas and, therefore, where action is needed to address crosscutting issues, including instances of fragmentation, overlap, or duplication.
GPRAMA requires GAO to periodically review its implementation. This report is part of GAO's response to that mandate and examines (1) how OMB and agencies defined programs, (2) the extent to which inventories provide useful information for decision makers, and (3) the extent to which inventories provide a coherent picture of the scope of federal involvement in particular areas. To address these objectives, GAO analyzed the 24 inventories using GPRAMA requirements, OMB guidance, and related leading practices from GAO's past work, and interviewed OMB staff and agency officials. For the first objective, GAO also selected two areas of fragmentation, overlap, and duplication identified in past GAO work—STEM education and nuclear nonproliferation—and compared the lists of programs developed in its past work to those contained in agency inventories. The two areas were selected based on various factors, including the number of agencies involved and whether those agencies published an inventory.
section_title: What GAO Found
paragraphs: To date, the approach used by the Office of Management and Budget (OMB) and agencies has not led to the inventory of all federal programs, along with related budget and performance information, envisioned in the GPRA Modernization Act of 2010 (GPRAMA). In developing the inventory, OMB allowed for significant discretion in several areas—leading to a variety of approaches for defining programs and inconsistencies in the type of information reported. The inconsistent definitions, along with agencies not following an expected consultation process, led to challenges in identifying similar programs in different agencies. As a result of these limitations, the inventory is not a useful tool for decision making. OMB is considering options for enhancing the inventory.
GPRAMA requires OMB to publish a list of all federal programs on a central governmentwide website. It also requires OMB to issue guidance, and agencies to identify and provide to OMB for publication information about each program— including how they defined their programs in line with OMB's guidance. OMB is taking an iterative approach to implement these requirements. Based on experiences from a pilot involving 11 agencies in 2012, OMB issued guidance allowing agencies flexibility to define their programs using different approaches, but within a broad definition of what constitutes a program—a set of related activities directed toward a common purpose or goal. According to OMB staff, this was based on a lesson learned from the pilot effort: a one-size-fits-all approach does not work well; agencies and their stakeholders use the term “program” in different ways because agencies achieve their missions through different programmatic approaches. In May 2013, OMB published the inventories developed by 24 agencies, which used various approaches to define and identify 1,524 programs (see table below).
Because agencies used different approaches, similar programs across agencies may not be identifiable. To illustrate the shortcomings of the inventory, GAO attempted to locate in relevant agencies' inventories the various science, technology, engineering, and mathematics (STEM) education and nuclear nonproliferation programs identified in GAO's past work. GAO was unable to identify in the inventories a large majority of the programs previously identified in its work: 9 of the 179 programs matched exactly and 51 others were identified based on program descriptions.
The lack of comparability may also be the result of agencies not working with each other when developing their inventories. One of OMB's stated purposes for the inventories is to facilitate coordination among programs that contribute to similar outcomes. However, agencies did not work together to consistently define their programs. Increased coordination could help agencies identify where they have programs that contribute to similar goals and thus, opportunities to collaborate in achieving desired outcomes.
The 24 inventories developed by agencies in 2013 did not provide the programs and related budget and performance information required by GPRAMA. This limits the usefulness of the inventories to various decision makers, including Congress and stakeholders. To be useful, the inventories must meet various users' needs for accuracy, completeness, consistency, reliability, and validity, among other factors. Specific steps OMB and agencies could take to ensure the inventories are more useful to decision makers include:
Presenting program-level budget information. Although GPRAMA requires agencies to identify program-level funding, OMB did not direct agencies to include this information in their 2013 inventories—it was to be part of a planned May 2014 update. However, OMB subsequently put the 2014 update on hold to determine how to merge these requirements with implementation of the federal spending information to be reported under the Digital Accountability and Transparency Act of 2014 (DATA Act). Reporting for both laws is web-based, which could more easily enable linkages between the two sites or incorporating information from each other. The Senate version of a bill, the Taxpayers Right-to-Know Act of 2014, which is currently under consideration, would require this linkage in lieu of incorporating budget information on the program inventory site.
Providing complete performance information . GPRAMA and OMB's guidance require agencies to describe each program's contribution to the agency's goals. However, there are instances where agencies omitted that information. For example, agencies did not consistently show how some or all of their programs supported strategic goals (in 7 of 24 inventories) or strategic objectives (in 13 of 24 inventories). Ensuring agencies illustrate this alignment would better explain how programs support the results agencies are achieving.
Consulting with stakeholders . None of the agencies sought input on their inventories from external stakeholders, such as Congress, state and local governments, and third-party service providers, although OMB's 2012 guidance instructed agencies to do so. In several instances, agency officials stated that they thought OMB was soliciting feedback on all inventories. By consulting with stakeholders to understand their needs, agencies would better ensure that the information provided in the inventories is useful for stakeholder decision making.
Other features of OMB's approach further limit the program inventories' ability to present a coherent picture of all federal programs, as required by GPRAMA. First, to date, OMB has only included 24 agencies in this effort. Second, while not specified by GPRAMA, tax expenditures were not included in the 2013 inventory. Tax expenditures, which represent a reduction in a taxpayer's tax liability through credits, deductions, or other means, resulted in $1.1 trillion in forgone revenue in fiscal year 2013, nearly the same amount as discretionary spending that year. By including tax expenditures, OMB could help ensure that agencies are properly identifying their contributions to the achievement of agency goals, as OMB's guidance directs them to do. Finally, OMB's guidance and staff have stated that eventually the inventory will move to a more dynamic, web-based presentation. This could make it easier to tag and sort related or similar programs, for instance, by type of program or contribution to the same or similar goals. Covering additional agencies and tax expenditures in the federal program inventory, along with web-based sorting capabilities, would help decision makers determine the scope of the federal government's involvement in a particular area, and therefore where action is needed to better address fragmentation, overlap, or duplication.
section_title: What GAO Recommends
paragraphs: GAO makes several recommendations to OMB. To present a more coherent picture of all federal programs, GAO recommends OMB revise its guidance to direct agencies to collaborate when defining and identifying programs that contribute to a common outcome, define plans for expanding implementation beyond the current 24 agencies, and include tax expenditures in the federal program inventory. In addition, to improve the usefulness of the information in inventories GAO recommends OMB ensure agencies consistently identify the various goals each program supports, and consult with stakeholders when developing or updating their inventories.
OMB staff generally agreed with these recommendations, although they neither agreed nor disagreed with three of GAO's recommendations related to including tax expenditures and additional performance information. OMB staff stated that until they had firmer plans on how program inventory and DATA Act implementation would be merged, they could not determine if implementing those recommendations would be feasible.

section_title: Letter
paragraphs: Many of the meaningful results that the federal government seeks to  achieve, such as those related to protecting food and agriculture and  providing homeland security, require the coordinated efforts of more than  one federal agency, level of government, or sector. However, agencies  face a range of challenges and barriers when they attempt to work across  organizational boundaries to address these crosscutting issues.  Moreover, our reports over the past 4 years have found more than 90  areas where opportunities exist for executive branch agencies or  Congress to reduce fragmentation, overlap, and duplication.annual reports have highlighted, creating a comprehensive list of federal   As our  programs along with related funding information is critical for identifying  potential fragmentation, overlap, or duplication among federal programs  or activities. The lack of such a list makes it difficult to determine the  scope of the federal government’s involvement in particular areas and,  therefore, where action is needed to eliminate, reduce, or better manage  fragmentation, overlap, or duplication. We also found that budget  information is often not available or not sufficiently reliable to identify the  level of funding provided to programs or activities.
The GPRA Modernization Act of 2010 (GPRAMA) requires the Office of  Management and Budget (OMB) to make publicly available a list of all  federal programs identified by agencies, along with budget and  performance information about each federal program. In May 2013, OMB  began implementing this provision by publishing the initial program  inventories developed by 24 agencies. We are required to review  implementation of GPRAMA at several critical junctures. This report is  part of our response to that mandate. Our specific objectives for this  report were to (1) assess how OMB and agencies defined and identified  the programs contained in the inventories, (2) examine the extent to  which the inventories provide useful information for federal decision  makers, and (3) examine the extent to which the inventories provide a  coherent picture of the scope of federal involvement in particular areas.
To address all three objectives, we assessed the 24 agency program  inventories published by OMB on Performance.gov in May 2013 using  relevant GPRAMA requirements, OMB guidance, and related leading  practices from our past work on managing for results, such as those  related to involving Congress and stakeholders in performance  management activities and ensuring performance information is useful to  decision makers. We also developed a structured set of questions and  conducted interviews with, or received written responses from, officials  involved in developing the inventories at each of the 24 agencies. We  also interviewed staff from OMB’s Office of Performance and Personnel  Management who led governmentwide implementation of these  requirements and developed guidance for the agencies.
In addition, to address our first objective, we selected two areas of  fragmentation, overlap, and duplication from our annual reports—science,  technology, engineering, and mathematics (STEM) education and nuclear  nonproliferation programs—and compared the lists of programs  developed by our work to those contained in agency inventories. We  selected these areas based on the number of agencies involved in the  area as reported in our past work,developed and published an inventory in May 2013, and whether the area  primarily involved domestic or foreign assistance/national security  programs—to ensure diversity among the agencies and programs we  reviewed. Because the two selected areas are a non-generalizable  sample of instances where fragmentation, overlap, and duplication exist,  our results cannot be generalized more broadly. However, the results of  our analysis illustrate how various approaches to defining programs can  lead to differences in how programs are identified. See appendix I for  additional information about the objectives, scope, and methodology for  this report.
We conducted this performance audit from August 2013 to October 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPRAMA requires OMB to make publicly available, on a central  governmentwide website, a list of all federal programs identified by  agencies. For each program, the agency is to provide to OMB for  publication: an identification of how the agency defines the term “program,”  consistent with OMB guidance, including program activities that were  aggregated, disaggregated, or consolidated to be considered a  program by the agency; a description of the purposes of the program and how the program  contributes to the agency’s mission and goals; and an identification of funding for the current fiscal year and the previous  2 fiscal years.
In addition, GPRAMA requires OMB to issue guidance to ensure that the  information provided on the website presents a coherent picture of all  federal programs.
In January 2012, OMB announced that it would conduct a pilot for  implementing the inventory provisions for trade, export, and  competitiveness programs. According to OMB’s memorandum, pilot  agencies were to identify an inventory of programs and map them to  established programmatic and organizational structures as well as the  agency’s strategic plan and performance goals. In addition, agencies  were to provide program purpose and funding information for each  program. The memorandum noted that, based on the pilot, OMB would  issue guidance to all federal agencies detailing the approach for  developing a governmentwide inventory.
In its August 2012 update to its guidance for implementing GPRAMA,  OMB included guidance for broader implementation of the inventory  requirements. Initial implementation was limited to the 24 agencies that  developed agency priority goals (APG) for fiscal years 2012-2013. In  addition, the guidance provided a phased approach for these 24  agencies. For the first phase, for publication in May 2013, agencies were  to describe their program definition approach, identify their programs, and  provide limited funding and performance information. OMB subsequently  published 24 separate inventory documents on the central  governmentwide website required under GPRAMA, which it has  implemented as Performance.gov. For the second phase, originally  planned for publication in May 2014, the 24 agencies were to update their  inventories based on any stakeholder feedback they received and provide  additional program-level funding and performance information. OMB’s  guidance also stated that at that time the inventory information was to be  presented in a more dynamic, web-based approach.
However, agencies did not publish updated inventories in May 2014.  According to OMB staff, plans for updating the inventories are on  indefinite hold as OMB re-evaluates next steps for what type of  information will be presented in the inventories and how it will be  presented based on (1) recent legislative actions that could affect the  information required for the inventories; (2) a lack of stakeholder  feedback; and (3) insufficient funding for,related to, presenting the inventories in a web-based format on   and technological challenges  Performance.gov. Regarding legislative action, OMB staff are considering  how implementation of the expanded reporting requirements for federal  spending information under the recently-enacted Digital Accountability  and Transparency Act of 2014 (DATA Act) could be tied to the program  inventories. The DATA Act is intended to increase accountability and  transparency in federal spending. It requires federal agencies to publicly  report, on at least a quarterly basis, information about any funding made  available to, or expended by, an agency or a component of the agency.  This reporting covers a variety of budget information—budget authority,  obligations, unobligated balances, outlays, and any other budgetary  resources—at different levels of aggregation and disaggregation:  appropriations account, program activities, and object class. Reporting  under the act is required to begin no later than 3 years after enactment  (no later than May 9, 2017). In addition, the DATA Act provides for the  establishment of government-wide financial data standards to, among  other things, produce consistent and comparable data across program  activities. The act also holds agencies more accountable for the quality of  the information disclosed. As we reported in April 2014, such increased  transparency provides opportunities for improving the efficiency and  effectiveness of federal spending and improving oversight to prevent and  detect fraud, waste, and abuse of federal funds.
In addition, the House has passed and the Senate is considering a bill  that would amend the program inventory provisions of GPRAMA. On  February 25, 2014, the House passed H.R. 1423, the Taxpayers Right-to- Know Act, which would require agencies to identify, for each program, an  estimate of the number of clients/beneficiaries served, an estimate of (for  the previous fiscal year) the number of full-time federal employees who  administer the program, and the number of full-time employees—at  organizations that administer or assist in administering the program (e.g.  grantees or contractors)—whose salary is paid in part of full by the federal  government. It would also require agencies to provide additional program- level information in their inventories, including total administrative costs  and expenditures for services. On May 21, 2014, the Senate Committee  on Homeland Security and Governmental Affairs favorably reported to the  full Senate S. 2113, the Taxpayers Right-to-Know Act, which would also  amend the program inventory provisions of GPRAMA to require agencies  to identify additional program-level information in their inventories.
section_title: Agencies Used Various Approaches to Define Their Programs, Which Limits the Comparability of Programs within and across Agencies
subsections: section_title: Agencies Used Different Approaches to Define Their Programs Based on OMB’s Guidance
paragraphs: GPRAMA requires agencies to identify how they define the term  “program,” consistent with guidance provided by the Director of OMB.  As a starting point, OMB’s guidance cites, in part, the definition for  “program” contained in our September 2005 Glossary of Terms Used in  the Federal Budget Process: an organized set of activities directed toward  a common purpose or goal that an agency undertakes or proposes to  carry out its responsibilities.glossary.
section_title: Inconsistent Definitions Limit Comparability of Programs within Agencies and across the Federal Government
paragraphs: The use of inconsistent approaches by agencies to define their programs,  illustrated in Table 2, limits the comparability of programs within agencies  as well as governmentwide. A few agencies did not use the same  program definition approach across their subcomponents or offices, which  limits comparability of their own programs. For example, the Department  of State’s (State) programs are defined using two separate approaches.  State operations programs follow a budget approach while State and U.S.  Agency for International Development (USAID) foreign assistance  programs follow both a budget and outcome-based approach. In addition,  while the U.S. Army Corps of Engineers (Corps) is an agency within the  Department of Defense (DOD), the Corps—Civil Works Program and  DOD developed separate inventories that took different approaches to  define programs.
To illustrate how program definition differences limit comparability across  agencies for similar programs, we selected two areas of potential  fragmentation, overlap, and duplication highlighted in our past annual  reports: science, technology, engineering and mathematics (STEM)  education and nuclear nonproliferation and determined which programs  were recognizable in both. We compared the lists of programs developed  from our past work to those contained in the relevant agencies’  inventories.
As shown in Table 3, of the 158 programs previously listed in our work on  STEM education, we were able to identify 51 programs in relevant agency  program inventories.
Of those 51, 9 were an exact match between how we previously identified  a program in our past work and how the agency identified it in its  inventory. For example, as figure 2 illustrates, both our past work on  STEM education and the Department of Education’s program inventory  identify “Mathematics and Science Partnerships” as a program.
We were able to identify the other 42 programs based on related  information contained in program descriptions in relevant agency  inventories. For four agencies, we were not able to identify any STEM  education programs from our past work in their inventories—covering 27  of the 158 programs. In addition, although our past work identified 3  STEM education programs at the Nuclear Regulatory Commission, it was  not 1 of the 24 agencies directed by OMB to produce a program inventory  for 2013.
In the nuclear nonproliferation area, we were able to identify 9 of the 21  programs identified in our past work in relevant agency program  inventories, as shown in table 4.
None of the nine nuclear nonproliferation programs we were able to  identify in the inventories were exact matches. Rather, we were able to  identify them as activities under larger programs in the inventories based  on the related program descriptions. For example, our past work identified  the Department of Homeland Security’s (DHS) “Global Nuclear Detection  Architecture” as a program within its Domestic Nuclear Detection Office.  DHS’s inventory identifies one program, Domestic Radiological/Nuclear  Detection, Forensics, and Prevention Capability, within the Domestic  Nuclear Detection Office. The inventory describes Global Nuclear  Detection Architecture as an activity within that program, as illustrated in  figure 3.
For both illustrative examples, we were not able to locate a majority of the  programs identified in our prior work in the relevant agency inventories.  The lack of comparability of similar programs across agencies in part  could be due to differences in the way we previously defined “program”  and the way each agency defined its programs, such as different levels of  aggregation of related activities to constitute a program. The flexibility  OMB’s guidance afforded in defining programs allows agencies to align  their definitions with multiple purposes, including carrying out their  mission and goals; serving beneficiaries, customers, and other target  populations; and providing benefits, services, and products. However, it  also can lead to differences in how programs are identified.
In addition, the lack of comparability may also be the result of agencies  not working across organizational boundaries when developing their  inventories. We asked agency officials whether they had worked with any  external parties when determining their program definition approach or  identifying their programs. Officials from all 24 agencies stated that they  had not sought input from any entities outside their own agencies, with  the exception of OMB. One of OMB’s stated purposes for the inventories  is to facilitate coordination among programs that contribute to similar  outcomes. If agencies worked together to more consistently define their  programs, it could also help them identify where they have programs that  contribute to similar outcomes, and therefore opportunities to collaborate.
section_title: In a Few Instances, Several Agencies Did Not Include Activities in Their Inventories Based on OMB’s Criteria for Determining What Constitutes a Program
paragraphs: OMB’s guidance directed agencies to work with OMB to determine the  appropriate primary approach (or mix of approaches) and level of  aggregation/disaggregation to be used to define their programs. It also  identified characteristics agencies should consider when determining   Officials at each of the 24  what constitutes a program (see text box).agencies told us that they worked with OMB—with most stating it was  staff in relevant Resource Management Offices as well as the Office of  Performance and Personnel Management—to determine (1) the  appropriate approach and (2) what constitutes a program. Similarly, staff  within OMB’s Office of Performance and Personnel Management told us  that they worked closely with agencies as they were defining and  identifying their programs.
OMB’s Characteristics of a Program  Externally recognizable. Agencies should use programs that are or relate to  programs or objectives used in Congressional Budget Justifications, statute, are  recognized by Congress and stakeholders, or are already publicly known; agencies  should use program names that are known outside the agency, and generally not  create new names.
Operationally meaningful. Agencies should use programs that are operationally  meaningful to agency senior leadership and components of the agency, and  programs should represent how the agency is managed and delivers on its mission.  Link to an organizational component(s), such as headquarters, bureau or office.  Programs should be operationally meaningful to the agency and agency senior  leadership.  Persistent. Generally, programs that persist over time should be included. However,  agencies have the flexibility to identify short-term efforts as programs, such as  activities related to the Recovery Act.
When we asked officials at each agency if the inventory covered  everything the agency does, most responded that the inventory included  all agency activities. However, officials at a few agencies told us that their  inventories did not always include certain activities. In most cases,  activities were excluded from the inventory because they did not meet  one of the characteristics of a program described in OMB’s guidance. For  instance, the Small Business Administration (SBA) only included  programs that were permanent in nature and excluded pilot programs  from its inventory because, according to SBA officials, they did not  consider pilot programs to meet the “persistent” characteristic. As an  example, they cited SBA’s Boots to Business program, which was in a  pilot phase when the agency’s 2013 inventory was published. SBA  officials told us they expected to add the program in the next inventory  update as the pilot phase had been completed. Because OMB’s guidance  does not clearly define when a short-term activity has persisted long  enough to be considered a program, agencies may be using different  criteria for when to include them as programs in their inventories. As a  result, agencies’ inventories may not be as comprehensive as desired.  Clearer guidance from OMB could lead to a more complete picture of  federal programs in the inventory.
In other instances, Office of Personnel Management (OPM) officials told  us they did not include a voting rights oversight program in their inventory  because it was not operationally meaningful, in that it did not easily align  with OPM’s mission. Social Security Administration (SSA) officials told us  they, in consultation with OMB staff, did not include the Special Benefits  for Certain World War II Veterans program in their inventory because it is  relatively small in size, in terms of its funding ($8 million of the agency’s  total net budget authority of $55.9 billion in fiscal year 2013) and number  of beneficiaries (approximately 1,000, based on information in SSA’s  fiscal year 2015 congressional budget justification) compared to the  agency’s other programs. In response to a question during our interview,  Department of the Interior (Interior) officials realized they did not include  functional management offices, such as the Offices of Policy and the  Chief Financial Officer, as part of a set of “programs” in their inventory  falling under the Office of the Secretary. Agency officials at OPM, SSA  and Interior told us they intend to account for these programs in future  iterations of their program inventories.
section_title: Presenting Program Budget Information, Providing Additional and Consistent Performance Information, and Seeking Stakeholder Feedback Could Help Ensure Inventories Are Useful to Decision Makers
paragraphs: The federal government is one of the world’s largest and most diverse  entities, with about $3.5 trillion in outlays in fiscal year 2013, funding an  extensive array of programs and operations. In responding to the varied  and increasingly complex issues the federal government seeks to  address, it faces a number of significant fiscal, management, and  governance challenges. To operate as effectively and efficiently as  possible and to make difficult decisions to address these challenges,  Congress, the administration, and federal managers must have ready  access to reliable and complete financial and performance information.  However, crucial information on the federal government’s programs is  decentralized, according to OMB’s guidance. The federal program  inventory has the potential to improve public understanding about what  federal programs currently operate and how programs link to budget,  performance, and other information by centralizing this information from  disparate sources—such as the President’s Budget, Congressional  Budget Justifications, USAspending.gov, and the Catalog of Federal  Domestic Assistance—in a single location: Performance.gov. Agencies  should consider the differing needs of various users—agency top  leadership and line managers, OMB, Congress, and stakeholders—to  ensure that this information will be both useful and used in decision   To be useful, this information must meet users’ needs for  making. accuracy, completeness, consistency, reliability, and validity, among other  factors.
subsections: section_title: Agencies Generally Did Not Present Program-level Budget Information in Their Inventories but DATA Act Implementation Could Provide More Detailed and Consistent Data
paragraphs: GPRAMA requires agencies to identify for each program in their  inventories:  the program activities that are aggregated, disaggregated, or  consolidated to be considered a program by the agency; and  funding for the current and two previous fiscal years (for the  inventories published in May 2013, that would have been fiscal years  2013, 2012, and 2011).
OMB’s 2012 and 2013 guidance delayed implementation of these  provisions until the planned May 2014 update.
Although OMB’s guidance delayed this, some agencies provided this  information in their inventories. A few agencies provided information  about how their program activity (PA) lines align with their programs as  defined in their inventory. For example, the National Science  Foundation’s inventory notes that programs presented in it are consistent  with the PA lines presented in the budget. In addition, the Department of  Transportation’s inventory states that its programs generally consist of  one to four PA lines. For example, the Pipeline and Hazardous Materials  Safety Administration’s “Pipeline Safety” program receives its funding  from four PA lines.
In addition, OMB’s guidance directed agencies to include funding (budget  authority) for agency bureaus or subcomponents for fiscal years 2012,  2013, and 2014 (requested). Since some agencies identified a  bureau/subcomponent to constitute a program, the bureau-level funding  information in their 2013 inventories also met GPRAMA’s requirement for  program-level funding information. For example, the Department of  Energy (Energy) defined its Office of Nuclear Energy as a program.
Because the office is also considered a subcomponent, Energy provided  the related funding information, as illustrated in figure 4.
As noted earlier, plans for updating the inventories are on indefinite hold  and agencies did not publish updated inventories, with the planned  program-level budget data, in May 2014—in part due to the enactment of  the DATA Act. According to OMB’s 2014 update to its guidance, it is  working with agencies to merge implementation of the DATA Act and  federal program inventory requirements to the extent possible, but has  not yet determined its implementation strategy. Agency reporting for both  sets of requirements is web-based, which could more easily enable  linkages between the two sites or incorporating information from each  other. The Senate version of the Taxpayers Right-to-Know Act would  require this linkage in lieu of incorporating this budget information on the  program inventory site. As OMB stated, one of its purposes in  implementing the program inventory requirements is to consolidate  information from various sources. By leveraging the DATA Act’s  requirements, OMB could expand the transparency of budget information  made available in the inventories—beyond PA lines and budget authority  and at different levels of disaggregation—and help ensure that  information is consistent across agencies. In addition, the disaggregated  budget information is important since our annual reports on  fragmentation, overlap, and duplication have found that it is not always  available for the programs and activities covered by our work. However,  the DATA Act provides up to 3 years for full implementation of its  reporting requirements. Thus, until OMB and agencies are ready to move  forward with the implementation of the two laws’ requirements, one option  to help ensure the inventories remain relevant and useful would be for  OMB to direct agencies to update the existing information in the 2013  versions for publication.
section_title: Agencies Defined the Purposes of Their Programs but Did Not Consistently Show How Programs Contribute to Agency Missions, Goals, and Results
paragraphs: GPRAMA requires agencies to describe for each program in their  inventories  the purposes of the program and  the program’s contribution to the agency’s mission and goals.
OMB’s guidance for the 2013 inventories directed agencies to provide a  program description (purpose) and identify supported strategic goals and  objectives. In addition, they were to add language directing readers to  Performance.gov to see programs supporting their agency priority goals  (APG) and the cross-agency priority (CAP) goals.
Agencies provided descriptions of their programs’ purposes. However,  some agencies did not consistently show how each program supports  agency missions and goals.
With the exception of one inventory—the joint State/USAID inventory  for foreign assistance programs—all provided a purpose for each  program, although State and USAID provided a link to a separate  document that provided purpose information for their foreign  assistance programs.
Only a few inventories (4 of 24) identified how the agencies’ programs  contribute to their mission. However, because agency strategic goals  and objectives are an outgrowth of the agency’s mission, the linkage  between agency programs and those goals can show the program’s  support to the mission.
A majority of the inventories (17 of 24) identified the agency strategic  goals each program supported, while less than half of the  inventories (11 of 24) identified the agency strategic objectives each  program supported.
Finally, most of the inventories (20 out of 24) referred readers to  Performance.gov for information about how the agencies’ programs  support both CAP goals and APGs.
OMB’s review of the inventories did not always identify instances where  agencies omitted this information. Providing this information allows  agencies to explain how programs contribute to the results the federal  government is achieving. Consistently showing the alignment between  programs and goals could also improve implementation of other  GPRAMA provisions. First, GPRAMA requires OMB and agencies to  identify for publication on Performance.gov the various organizations,  program activities, and other activities that contribute to the CAP goals  and APGs, respectively. In addition, OMB’s guidance directs agencies  to identify these contributors for their strategic objectives. These  provisions are important because they show how agencies are  coordinating efforts toward a common outcome. As we have previously  reported, uncoordinated program efforts can waste scarce funds, confuse  and frustrate program customers, and limit the overall effectiveness of the  federal effort. Our past work has found that OMB and agencies did not  always identify all relevant contributors to their CAP goals and APGs.  Accordingly, in May 2012 and April 2013, we made recommendations to  OMB. Table 5 describes these recommendations and related actions  taken to date.
Second, GPRAMA requires OMB and agencies, on at least a quarterly  basis, to review and report on Performance.gov progress towards the  CAP goals and APGs, respectively. As part of these reviews, OMB and  agencies are to (1) involve relevant stakeholders from contributing  organizations and program activities and (2) assess the contributions of  the various organizations, program activities, and other activities that  support these goals.GPRAMA directed agencies, beginning this year (2014), to conduct  annual reviews of progress towards their strategic objectives—the  outcomes or impacts the agency is intending to achieve. According to  OMB’s guidance, the strategic reviews should inform many of the  agency’s decision-making processes, as well as decision making by the   In addition, OMB’s 2013 guidance for implementing  agency’s stakeholders, such as informing long-term strategy, annual  planning and budget formulation; strengthening collaboration on  crosscutting issues; and improving transparency. If successfully  implemented in a way that is open, inclusive, and transparent—to  Congress, delivery partners (e.g., state and local governments, grantees,  non-profit organizations, associations, contractors), and a full range of  stakeholders—these reviews could help decision makers assess the  relative contributions of various programs that contribute to a given goal.  Successful reviews could also help decision makers identify and assess  the interplay of public policy tools that are being used, to ensure that  those tools are effective and mutually reinforcing, and results are being  efficiently achieved. However, our past work has found that relevant  contributors were not always included in these reviews and accordingly  we made recommendations to OMB, as described in table 6.
OMB’s guidance does not direct agencies to identify in their inventories  the performance goals to which each program contributes. Although not  specifically required for the program inventory, GPRAMA requires  agencies, in their annual performance plans, to identify the various  organizations, program activities, and other activities that contribute to  each performance goal. As noted in the Senate Committee on  Homeland Security and Governmental Affairs report that accompanied  GPRAMA, this provision, among others, was intended to help describe  the strategies to be used to achieve results and the resources to be  applied to those strategies. This information can help Congress  understand and assess the relationship between the agency’s resources  and results. Having a clear description of the strategies and resources an  agency plans to use will allow Congress to assess the likelihood for an  agency to achieve its intended results. The act also requires that agency  performance goals and related information be available in a web-based  format on Performance.gov. According to OMB staff, efforts to expand  the site to include this information have not been possible given the  available appropriations for the E-Government Fund, which supports  Performance.gov and other electronic government requirements and  initiatives.
section_title: Agencies Did Not Take Steps to Ensure Inventories Provide Useful Information for Stakeholders, including Congress
paragraphs: OMB’s 2012 guidance directed agencies to work with key stakeholders,  which can include Congress, state and local governments, third party  service providers, and the public, to validate that their program  inventories would be both internally and externally recognizable.  According to OMB staff, they want to ensure that the inventories are  useful and meaningful to the agencies and their stakeholders.
Although officials we spoke with or received written responses from at  each of the 24 agencies told us that they involved internal stakeholders— officials and staff from various bureaus/component agencies and offices  within their respective department or agency—to validate their programs,  most also shared that they saw no benefit to their agencies from  developing the inventories. Many told us that they viewed it as a  paperwork exercise, repackaging information available elsewhere with the  resulting inventory not useful to their internal agency decision making. In  a few instances, officials offered different views. For example, an official  at DHS told us that her department’s approach to linking programs to  performance information helped the department communicate the results  it was achieving. She noted that DHS has maintained an external-facing,  mission-oriented program structure since it was created. In addition, an  official from SBA told us he thought the program inventories—once fully  implemented as envisioned in OMB’s guidance for the 2014 update— could help his agency identify instances to enhance their coordination and  collaboration with other agencies to achieve common outcomes.
Officials we spoke with or who provided written responses to our  questions also told us that they did not solicit feedback on their  inventories from external stakeholders, including Congress. In addition,  none of them reported receiving any feedback on their inventories beyond  that from OMB staff during development. In some instances, agency  officials told us that they didn’t seek stakeholder input because their  approach for defining their programs resulted in programs with which their  stakeholders were already familiar. For example, officials at several  agencies told us that using a budget approach resulted in programs that  were recognizable to congressional appropriators. In other instances,  agency officials told us they thought OMB staff were collecting feedback  on the inventories, but they had not heard any results at the time of our  interview. When we asked OMB staff about any feedback they had  solicited or received, they told us they had briefed two congressional  committees when the inventories were published in May 2013, but that  they had not received any formal feedback from Congress or any other  stakeholders, with the exception of the preliminary results of our review,  which we published as part of a testimony before the Senate Committee  on Homeland Security and Governmental Affairs in March 2014. In  addition, although Performance.gov has a mechanism for users to  provide feedback on the website, OMB staff stated that they had not  received any feedback on the program inventories from that venue.
See, for example, GAO-13-518; GAO-13-174; GAO-12-621SP; GAO, Performance  Budgeting: PART Focuses Attention on Program Performance, but More Can Be Done to  Engage Congress, GAO-06-28 (Washington, D.C.: Oct. 28, 2005); Managing For Results:  Enhancing the Usefulness of GPRA Consultations Between the Executive Branch and  Congress, GAO/T-GGD-97-56 (Washington, D.C.: Mar. 10, 1997); and Executive Guide:  Effectively Implementing the Government Performance and Results Act, GAO/GGD-96- 118 (Washington, D.C.: June 1996). stakeholder input, and congressional input in particular, on program  inventories and the information presented therein would provide OMB and  agencies another opportunity to ensure they are presenting useful  information for stakeholder decision making.
section_title: Identifying All Federal Programs and Providing Sorting Capabilities Could Help Provide a More Coherent Picture of Federal Involvement in Particular Areas
paragraphs: GPRAMA requires OMB to ensure that the inventory information provided  by agencies and published on Performance.gov presents a coherent   As our annual reports on fragmentation,  picture of all federal programs.overlap, and duplication have stated, the federal program inventory could  be a key tool for addressing crosscutting issues. For example, as  highlighted in OMB’s guidance, the federal program inventory has the  potential to facilitate coordination across programs by making it easier to  find programs that may contribute to a shared goal. Moreover, in its  memorandum regarding the inventory pilot project for trade, export, and  competitiveness programs, OMB noted that duplicative programs make  government less effective, waste taxpayer dollars, and make it harder for  the public to navigate government services. It states that, in order to  continue efforts to reduce duplication and overlap and improve program  outcomes through better coordination across agencies, the executive  branch must achieve greater transparency into all federal programs.
subsections: section_title: OMB Has Not Defined Plans to Expand Inventories to Include All Federal Programs
paragraphs: GPRAMA requires OMB to make publicly available a list of all federal  programs identified by agencies. While the 24 agencies have begun  implementing the inventory requirements, OMB has not defined plans for  when this effort will be expanded beyond these agencies. As noted earlier  in the report, the lack of such a comprehensive list makes it difficult to  determine the scope of the federal government’s involvement in particular  areas and, therefore, where action is needed to address crosscutting  issues.
In addition, OMB has not included all types of federal programs in its  plans for the program inventory. OMB’s guidance identified 12 different  types of federal programs, defined in table 7, for agencies to assign to  their programs.each program in their inventories.
section_title: The Ability to Find and Sort Similar Programs or Programs Contributing to Shared Outcomes Would Enhance the Usefulness of Inventories
paragraphs: As noted earlier, GPRAMA requires OMB to publish the federal program  inventory on Performance.gov. While the inventories published in May  2013 were individual agency documents, OMB’s guidance and staff have  stated that eventually the inventory would move to a more dynamic, web- based approach—originally planned for the May 2014 update and now on  hold. This web-based approach could make it easier to tag and sort  related or similar programs. For instance, OMB’s plans to have agencies  tag each of their programs by one or more program type in a future  iteration of the inventory would provide a sorting capability for identifying  the same type of program. By providing a sorting mechanism by program  type, OMB could help address one of our open recommendations,  described in the text box below, by identifying (1) all programs in a given  type, and (2) of those programs, any that have developed strategies to  effectively overcome measurement challenges.
GAO Recommended OMB and the PIC Develop a Detailed Approach to Examine  and Address Long-standing Performance Measurement Challenges  In our June 2013 report, Managing for Results: Executive Branch Should More Fully  Implement the GPRA Modernization Act to Address Pressing Governance Challenges  (GAO-13-518), we found that agencies continue to face common, long-standing  difficulties in measuring the performance of various types of federal programs and  activities—contracts, direct services, grants, regulations, research and development,  and tax expenditures. We recommended the Director of OMB work with the PIC to  develop a detailed approach to examine these difficulties across agencies, including  identifying and sharing any promising practices from agencies that have overcome  difficulties in measuring the performance of these program types. In commenting on a  draft of the report, OMB staff agreed with this recommendation. As of July 2014, OMB  and the PIC have taken some initial steps to address this recommendation. According to  OMB staff, this includes efforts related to achieving several of the CAP goals. For  example, the “Benchmark and Improve Mission-Support Operations” CAP goal involves  developing common standards and benchmarks to measure the performance and cost  of various agency administrative operations, such as information technology and  acquisition management. In addition, PIC staff told us they have taken initial steps to  address performance measurement issues in a few areas, including a pilot effort  focused on acquisitions (contracts). PIC staff said they plan to expand the model to  focus on other types of programs with performance measurement issues, such as grants  and regulations. We will continue to monitor progress.
As highlighted in OMB’s guidance, the federal program inventory has the  potential to facilitate coordination across programs by making it easier to  find programs that may contribute to a shared goal or a common  outcome. This could also help identify and address instances of  fragmentation, overlap, and duplication. In its guidance for the 2014  update before it was put on hold, OMB intended for agencies to link each  program to the existing web pages on Performance.gov for strategic  goals, strategic objectives, APGs, and CAP goals. According to OMB  staff, once they move forward with the next inventory update and move to  a web-based presentation, this would allow users to sort programs by the  goals to which they contribute. This approach also would allow users to  identify programs that contribute to broader themes on Performance.gov.  The themes generally align with budget functions from the President’s  Budget and include administration of justice; general science, space, and  technology; national defense; and transportation, among others.  Currently, the themes can be used to sort goals on Performance.gov that  contribute to those broad themes.
section_title: Conclusions
paragraphs: The coordinated efforts of multiple federal agencies, different levels of  government, and sectors are generally needed to achieve meaningful  results. This is underscored by our annual reports on fragmentation,  overlap, and duplication, which to date have identified over 90 areas,  each of which involves a patchwork of federal agencies, programs, and  activities that attempt to address the same issue. Executive branch and  congressional efforts to identify, manage, or resolve instances of  fragmentation, overlap, and duplication are hindered by the lack of a  comprehensive list of all federal programs. Such a list, along with related  budget and performance information, could help decision makers  determine the scope of the federal government’s involvement,  investment, and performance in a particular area, which in turn could help  pinpoint where action is needed to better address or avoid fragmentation,  overlap, and duplication. Therefore, effective implementation of  GPRAMA’s federal program inventory requirements could provide  decision makers with critical information that could be used to better  address crosscutting issues, among other purposes.
The executive branch has taken some initial steps to develop program  inventories with related budget and performance information. In 2012,  OMB involved 11 agencies in a pilot program inventory effort for federal  trade, export, and competiveness programs and subsequently developed  guidance—based on lessons learned from the pilot—for broader  implementation at 24 agencies. In May 2013, the 24 agencies published  inventories, providing information about 1,524 programs they collectively  identified. Despite these initial efforts, we have identified a number of  areas where implementation should be improved.
OMB needs to take further actions to effectively meet GPRAMA’s  requirement for the inventories to present a coherent picture of all federal  programs. First, OMB’s guidance provides agencies with flexibility in how  they defined their programs. As a result, agencies used different  approaches to identify their programs, which, in turn, led to a lack of  comparability for similar programs across agencies. This was illustrated  by our analysis of agency inventories for STEM education and nuclear  nonproliferation programs, in which we could only exactly match a small  fraction of the programs covered by our past work on fragmentation,  overlap, and duplication in those areas. According to OMB staff, the  flexibility provided in guidance is derived from lessons learned from the  pilot effort—that agencies are different for valid and legitimate reasons  and a one-size-fits-all approach would not work for all agencies. While  this may be true, OMB could do more to direct agencies to find common  ground on similar programs. One of OMB’s stated purposes for the  inventories is to facilitate coordination across programs that contribute to  similar outcomes. However, as we discovered through our interviews with  agency officials involved with the inventory efforts, none of the agencies  sought input from other agencies on how they defined and identified their  programs. If agencies worked together to more consistently define their  programs, it could also help them identify where they have programs that  contribute to similar outcomes, and therefore opportunities to collaborate.
Second, OMB’s plans to date do not adequately ensure that the  inventories include all federal programs. It has yet to provide firm plans on  when implementation of GPRAMA’s inventory requirements will be  expanded beyond the current 24 agencies. OMB’s guidance does not  provide a clear time frame for how long activities must persist to be  considered programs, which may have resulted in agencies excluding  certain activities from their inventories. In addition, OMB did not include  tax expenditures in the inventory effort. Although GPRAMA does not  specify that tax expenditures should be included in the program inventory,  our work over the past 20 years has shown the need for tax expenditures  to be held to the same scrutiny as spending programs, given the sizeable  federal investment they represent. The omission of these agencies,  programs, and activities severely limits the usefulness of the inventory as  a tool for addressing crosscutting issues.
In addition, other factors limit the usefulness of the inventory as a source  of information for decision makers. To be useful, information must meet  various users’ needs for accuracy, completeness, consistency, reliability,  and validity, among other factors. Agency officials also told us they did  not consult with external stakeholders, including Congress. Although  OMB’s 2012 guidance instructed agencies to seek this input, none of  them did. Subsequently, OMB has removed this direction from its  guidance. By consulting with stakeholders to understand their needs,  agencies would better ensure that the information provided in the  inventories is useful for stakeholder decision making. Involving  congressional stakeholders is of critical importance given Congress’s  power to create and fund programs.
Our work also identified limitations with the performance information  contained within the inventories. Agencies did not consistently identify the  strategic goals, strategic objectives, agency priority goals, and cross- agency priority goals that each program supports, as directed by OMB  guidance. These omissions were not always identified during OMB’s  review of each agency’s inventory. In addition, OMB’s guidance does not  direct agencies to identify in their inventories the performance goals to  which each program contributes. Although not required for the  inventories, GPRAMA requires this type of connection—albeit by  performance goal rather than by program—in agency performance plans.  This information can help Congress and others understand and assess  the relationship between the agency’s resources and results. Without it, it  will be difficult for Congress to assess the likelihood of the agency’s  success in achieving intended results.
section_title: Recommendations for Executive Action
paragraphs: To ensure the effective implementation of federal program inventory  requirements and to make the inventories more useful, we make the  following eight recommendations to the Director of the Office of  Management and Budget.
To better present a more coherent picture of all federal programs, we  recommend the Director of OMB take the following five actions: direct agencies to collaborate with each other in defining and  identifying programs that contribute to common outcomes, and provide a time frame for what constitutes “persistent over time”  that agencies can use as a decision rule for whether to include  short-term efforts as programs; define plans for when additional agencies will be required to develop  include tax expenditures in the federal program inventory effort by designating tax expenditure as a program type in relevant developing, in coordination with the Secretary of the Treasury, a  tax expenditure inventory that identifies each tax expenditure and  provides a description of how the tax expenditure is defined, its  purpose, and related performance and budget information.
To help ensure that the information agencies provide in their inventories  is useful to federal decision makers and key stakeholders, and to provide  greater transparency and ensure consistency in federal program funding  and performance information, we recommend the Director of OMB take  the following three actions:  revise relevant guidance to direct agencies to  consult with relevant congressional committees and stakeholders  on their program definition approach and identified programs  when developing or updating their inventories, and  identify in their inventories the performance goal(s) to which each  program contributes; and ensure, during OMB reviews of inventories, that agencies consistently  identify, as applicable, the strategic goals, strategic objectives, agency  priority goals, and cross-agency priority goals each program supports.
section_title: Agency Comments
paragraphs: We provided a draft of this report for review and comment to the Director  of OMB and the 24 agencies that developed program inventories.
In oral comments provided on October 15, 2014, staff from OMB’s Office  of Performance and Personnel Management stated that they agreed with  five of our eight recommendations and would consider how to implement  those recommendations as they move forward with merging program  inventory implementation with that of the DATA Act. For the other three  recommendations—designating tax expenditures as a type of program,  developing an inventory of tax expenditures, and directing agencies to  identify how their programs contribute to their performance goals—OMB  staff neither agreed nor disagreed. OMB staff told us that until they had  firmer plans on how program inventory and DATA Act implementation  would be merged, they could not determine if implementing these three  recommendations would be feasible.
We also received technical comments from OMB, the Departments of  Energy and State, and the National Aeronautics and Space  Administration, which we incorporated as appropriate.
We are sending copies of this report to the Director of OMB and the  heads of the 24 agencies that developed program inventories as well as  interested congressional committees and other interested parties. This  report will also be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix II.
section_title: Appendix I: Objectives, Scope & Methodology
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) requires us to review  implementation of the act at several critical junctures. This report is part  of our response to that mandate. Our specific objectives for this report  were to (1) assess how OMB and agencies defined and identified the  programs contained in the inventories, (2) examine the extent to which  the inventories provide useful information for federal decision makers,  and (3) examine the extent to which the inventories provide a coherent  picture of the scope of federal involvement in particular areas.
To address all three objectives, we assessed the implementation of  relevant GPRAMA requirements by the Office of Management and  Budget (OMB) and the 24 agencies that developed program inventories,  which were published on Performance.gov in May 2013. requires OMB to make publicly available on a central governmentwide  website, a list of all federal programs identified by agencies. For each  program, the agency is to provide to OMB for publication: an identification of how the agency defines the term “program,”  consistent with OMB guidance, including program activities that were  aggregated, disaggregated, or consolidated to be considered a  program by the agency; a description of the purposes of the program and how the program  contributes to the agency’s mission and goals; and an identification of funding for the current fiscal year and the previous  2 fiscal years.
GPRAMA also requires OMB to issue guidance to ensure that the  information provided on the website presents a coherent picture of all  federal programs.
The 24 agencies directed by OMB to develop program inventories are the Departments  of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services,  Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State,  Transportation, the Treasury, and Veterans Affairs, as well as the U.S. Agency for  International Development, U.S. Army Corps of Engineers—Civil Works Program,  Environmental Protection Agency, General Services Administration, National Aeronautics  and Space Administration, National Science Foundation, Office of Personnel  Management, Small Business Administration, and Social Security Administration.
We also assessed program inventory implementation based on relevant  OMB guidance (Circular No. A-11, Part 6) and related leading practices  from our past work on managing for results, such as those related to  involving Congress and stakeholders in performance management  activities and ensuring performance information is useful to decision  makers. In addition, we developed a structured set of questions and  conducted interviews with, or received written responses from, officials  involved in developing the inventories at the 24 agencies. We also  interviewed staff from OMB’s Office of Performance and Personnel  Management who led governmentwide implementation of these  requirements and developed guidance for agencies.
Additionally, to address our first objective, we selected two areas of  fragmentation, overlap, and duplication from our annual reports—science,  technology, engineering, and mathematics (STEM) education and nuclear  nonproliferation programs—and compared the lists of programs  developed by our work to those contained in agency inventories. To  select these two areas, we first reviewed the existing body of 83 areas of  fragmentation, overlap, and duplication published in our 2011, 2012, and  2013 annual reports,areas that focused on management functions, since they were unlikely to  be captured in the program inventories. For the remaining 60 areas, we  identified the number of agencies involved in the area as reported in our  past worked, whether those agencies had developed and published an  inventory in May 2013, and whether the area primarily involved domestic  or foreign assistance/national security programs—to ensure diversity  among the agencies and programs we reviewed. Finally, we sorted those  lists to select the areas with the most agencies involved in them: STEM  education for domestic programs and nuclear nonproliferation programs  for foreign assistance/national security programs.   and identified and eliminated from selection 23  We compared the lists of programs developed from our past work on  STEM education and nuclear nonproliferation to those contained in the  relevant agencies’ inventories. We identified possible matches in two different manners. First, we identified instances in which program names  exactly matched between the lists developed for our past work and the  agency’s inventory. In addition, we also reviewed the program  descriptions contained in the relevant inventories to determine if they  would lead one to programs/activities related to STEM education or  nuclear nonproliferation. For example, if the program description  contained language such as “science education” or “technology  education,” we included that program as a broader match. In some  instances, we were also able to identify programs from our past work  listed as activities within the program descriptions in relevant inventories.  To ensure consistency in this analysis, the work to identify programs  conducted by one analyst was reviewed and verified by another analyst.
Because the two selected areas are a non-generalizable sample of  instances where fragmentation, overlap, and duplication exist, our results  cannot be generalized more broadly. However, the results of our analysis  illustrate how various approaches to defining programs can lead to  differences in how programs are identified.
We conducted this performance audit from August 2013 to October 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above contact, Lisa M. Pearson (Assistant Director) and  Benjamin T. Licht supervised this review and the development of the  resulting report. Gerard Burke, Amy Bush, Virginia Chanley, A. Nicole  Clowers, Karin Fangman, Heather Krause, and Steven Putansu made  significant contributions to this report. Carole J. Cimitile, Dewi Djunaidy,  Emily Gruenwald, Donna Miller, Leah Q. Nash, and Erinn L. Sauer also  provided key contributions.