Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research
Published: 2011-01-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Amid efforts to improve performance and constrain spending, federal agencies are being asked to expand the use of rigorous program evaluation in decision-making. In addition to performance data, indepth program evaluation studies are often needed for assessing program impact or designing improvements. Agencies can also use their evaluation resources to provide information needed for effective management and legislative oversight. GAO was asked to study federal agencies with mature evaluation capacity to examine (1) the criteria, policies, and procedures they use to determine programs to review, and (2) the influences on their choices. GAO reviewed agency materials and interviewed officials on evaluation planning in four agencies in three departments with extensive evaluation experience: Education, Health and Human Services (HHS), and Housing and Urban Development (HUD). HHS and HUD agreed with the description of how they plan evaluations. HHS noted that the optimal location of evaluation units will vary with the circumstances and purpose of evaluations. HUD felt the draft report did not emphasize enough the influence of the appropriations process. GAO has added text to note its influence on evaluation planning. Education provided technical comments.
section_title: What GAO Found
paragraphs: Although no agency GAO reviewed had a formal policy describing evaluation planning, all followed a generally similar model for developing and selecting evaluation proposals. Agencies usually planned an evaluation agenda over several months in the context of preparing spending plans for the coming fiscal year. Evaluation staff typically began by consulting with a variety of stakeholders to identify policy priorities and program concerns. Then with program office staff they identified the key questions and concerns and developed initial proposals. Generally, the agencies reviewed and selected proposals in two steps: develop ideas to obtain initial feedback from senior officials and develop full-scale evaluation proposals for review and approval. The four general criteria these mature agencies use to plan evaluations were remarkably similar: (1) strategic priorities representing major program or policy area concerns or new initiatives, (2) program-level problems or opportunities, (3) critical unanswered questions or evidence gaps, and (4) the feasibility of conducting a valid study. The agencies' procedures differed on some points. External parties' participation in evaluation planning may reflect these agencies' common reliance on nonfederal program partners. Only the offices GAO reviewed in HHS' Centers for Disease Control and Prevention held formal competitions to rank-order proposals before submitting them for approval; in other agencies, senior officials assessed proposals in a series of discussions. When evaluation authority and funds are tied to the program, evaluators generally choose not which programs to evaluate but which research questions to answer. Sometimes this resulted in a program's never being evaluated. Evaluation units at higher organizational levels conducted a wider range of analytic activities, consulted more formally with program offices, and had less control over approvals. The Congress influences an agency's program evaluation choices through legislating evaluation authority, mandating studies, making appropriations, and conducting oversight. GAO concludes that (1) all four criteria appear key to setting an effective evaluation agenda that provides credible, timely answers to important questions; (2) most agencies could probably apply the general model in which professional evaluators iteratively identify key questions in consultation with stakeholders and then scrutinize and vet research proposals; (3) agencies could adapt the model and decide where to locate evaluation units to meet their own organizational and financial circumstances and authorities; and (4) agencies' reaching out to key program and congressional stakeholders in advance of developing proposals could help ensure that their evaluations will be used effectively in management and legislative oversight.
section_title: What GAO Recommends
paragraphs: GAO makes no recommendations.

section_title: Letter
paragraphs: Amid efforts to improve federal government performance and constrain  federal spending, the Obama Administration has emphasized expanding  the availability and use of rigorous program evaluation. The aim is to  establish an evaluation culture in which agencies regularly use evidence to  inform program design and investment decisions, complementing existing  efforts to strengthen performance measurement and management. Today,  most federal agencies use performance measures to track progress toward  goals. Few appear to conduct in-depth program evaluations regularly to  assess their programs’ impact or inform policymakers on how to improve  results. Increasing demands for performance information prompted your  expressed interest in how agencies, given their limited evaluation  resources, can provide the information that is necessary for effective  management and legislative oversight. To help agencies plan their  evaluation activities strategically, you asked us to study federal agencies  that have mature program evaluation capacity, to learn  1.  What criteria, policies, and procedures do they use to determine which  programs to review?  2.  What conditions influence an agency’s choices?
To find answers that would apply broadly to other federal agencies about  how to prioritize program evaluations, we selected four agencies in three  departments with known evaluation capacity—the ability to systematically  collect, analyze, and use data on program results—in a variety of  substantive areas. The four agencies were the Department of Education  and the Department of Housing and Urban Development (HUD) and two  agencies in the Department of Health and Human Services (HHS)—the  Administration for Children and Families (ACF) and the Centers for  Disease Control and Prevention (CDC).
To identify agencies fulfilling our criteria, we reviewed previous GAO  reports and agency documents for evidence of emphasis on conducting  evaluations. For example, we searched for examples of agencies’  incorporating the results of program evaluations in their annual  performance reports. To obtain a diverse set of cases, we selected  agencies that addressed a variety of content areas and program types— direct services or grants, regulation, and research—and that varied in the  conditions likely to affect their decision process, such as whether they had  a central office responsible for evaluation. To identify evaluation planning  procedures and criteria, we reviewed agency materials and interviewed  agency evaluation officials about their evaluation planning process. We  identified conditions influencing their choices in interviews with agency  officials and in comparisons of what we found across the agencies.
We conducted this performance audit from May 2010 through December  2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. We requested comments  on a draft of this report from the Secretaries of Education, Health and  Human Services, and Housing and Urban Development. HHS and HUD  provided comments that are reprinted in appendixes I and II and described  at the end of this letter. Education provided technical comments.
subsections: section_title: Background
paragraphs: Under the Government Performance and Results Act of 1993 (GPRA)  federal agencies are expected to focus on achieving results and to  demonstrate, in annual performance reports and budget requests, how  their activities help achieve agency or governmentwide goals. In 2002, to  encourage greater use of program performance information in decision  making, the Office of Management and Budget (OMB) created the Program  Assessment Rating Tool (PART). PART was intended to provide a  consistent approach for evaluating federal programs within the executive  budget formulation process. However, because PART conclusions rely on  available program performance and evaluation information, many of the  initial recommendations focused on improving outcome and efficiency  measures. Although GPRA and PART helped improve the availability of  better performance measures, we and OMB have noted that this did not  result in their greater use by the Congress or agencies.
In October 2009, OMB announced a plan to strengthen federal program  evaluation, noting that rigorous independent program evaluations can help  determine whether government programs are achieving their intended  outcomes as well as possible and at the lowest possible cost. Program  evaluations are systematic studies that assess how well a program is  working, and they are individually tailored to address the client’s research  question.
Process (or implementation) evaluations assess the extent to which a  program is operating as intended.
Outcome evaluations assess the extent to which a program is achieving its  outcome-oriented objectives but may also examine program processes to  understand how outcomes are produced.
When external factors such as economic or environmental conditions are  known to influence a program’s outcomes,  Impact evaluations may be used to measure a program’s net effect by  comparing outcomes with an estimate of what would have occurred had  there been no program intervention.
Thus, program evaluation can provide an important complement to agency  performance data that simply track progress toward goals.
In announcing the evaluation initiative, the OMB Director expressed  concern that many important programs had never been evaluated,  evaluations had not sufficiently shaped budget priorities or management  practices, and many agencies lack an evaluation office capable of  supporting an ambitious strategic research agenda. The initiative consisted  of three efforts: posting information online on all agencies’ planned and  ongoing impact evaluations, establishing an interagency group to promote  the sharing of evaluation expertise, and funding some new agency rigorous  impact evaluations and capacity strengthening efforts. As part of the fiscal  year 2011 budget process, OMB allocated approximately $100 million for  the evaluation initiative to support 35 rigorous program evaluations and  evaluation capacity-building proposals.
OMB made a similar evaluation solicitation for the fiscal year 2012 budget  in which nonsecurity agencies were asked to reduce their discretionary  budgets by 5 percent. The budget process evaluation initiative is focused  on impact evaluations and is not intended to cover the full range of an  agency’s evaluation activities. However, to be considered for additional  evaluation funding, agencies must demonstrate that they are both using  existing evaluation resources effectively and beginning to integrate  evaluation into program planning and implementation. With significant  efforts under way to increase agencies’ evaluation resources, it is  especially timely now to learn how agencies with more evaluation  experience prioritize their resources.
A recent GAO review identified three elements that leading national  research organizations consider essential to a sound federal research and  evaluation program: research independence, transparency and  accountability, and policy relevance. These elements align well with  OMB’s new evaluation initiative and expectations for a better integration  of evaluation into program design and management. In this report, we do  not assess the quality of the agencies’ research agendas or their  achievement of these objectives. However, we do describe practices these  agencies took that were designed to achieve those elements.
subsections: section_title: Evaluation at Education
paragraphs: The Department of Education establishes policy for, administers, and  coordinates most federal assistance to elementary, secondary, and  postsecondary education. The department has supported educational  research, evaluation, and dissemination not only since the Congress  created it in 1979 but also earlier, when it was the Office of Education. For  several years, two central offices in the Department of Education have  been responsible for program and policy evaluation. The Policy and  Program Studies Service (PPSS), in the Office of Planning, Evaluation, and  Policy Development (OPEPD), advises the Secretary on policy  development and review, strategic planning, performance measurement,  and evaluation. The Institute of Education Sciences (IES), established in  2002 (replacing the Office of Educational Research and Improvement), is  the research arm of the department. IES is charged with producing  rigorous evidence on which to ground education practice and policy, with  program evaluation housed primarily in the National Center for Education  Evaluation and Regional Assistance (NCEE).
In 2009, the Department of Education launched a review of its evaluation  activities, comparing them to those of other government agencies, seeking  to build analytic capacity, and intending to use available knowledge and  evidence more effectively. This review resulted in a comprehensive,  departmentwide evaluation planning process and clarified the distinct  evaluation responsibilities of the two offices. Starting in 2010, OPEPD was  to lead the planning process, in partnership with IES, to identify the  department’s key priorities for evaluation and related knowledge-building  activities. Starting in fiscal year 2011, NCEE in IES will be responsible for  longer-term (18 months or longer) program implementation and impact  studies, while PPSS in OPEPD will focus on shorter-term evaluation  activities (fewer than 18 months), policy analysis, performance  measurement, and knowledge management activities. Some program  offices also conduct evaluation activities separate from studies conducted  by either of the central offices, such as supporting grantee evaluations or  analyzing grantee performance data for smaller programs where larger- scale evaluations are not practical.
section_title: Evaluation at HUD
paragraphs: The Department of Housing and Urban Development is the principal  federal agency responsible for programs on housing needs, fair housing  opportunities, and community improvement and development. It insures  home mortgages, subsidizes housing for low- and moderate-income  families, promotes and enforces fair housing and equal opportunity  housing, and provides grants to states and communities to aid community  development. At HUD, program evaluation is primarily centralized in one  office—the Office of Policy Development and Research (PD&R)—created  in 1973. It conducts a mix of surveys, independent research,  demonstrations, policy analyses, and short- and long-term evaluations that  inform HUD’s decisions on policies, programs, and budget and legislative  proposals. PD&R provides HUD’s program offices with technical support,  data, and materials relevant to their programs. Although the primary  responsibility for evaluating programs falls to PD&R, some evaluation is  found in program offices, such as the Office of Housing, which routinely  conducts analyses to update its loan performance models for assessing  credit risk and the value of its loan portfolio.
In 2006, the Congress, concerned about the quality of HUD research,  commissioned the National Research Council (NRC) to evaluate the PD&R  office and provide recommendations regarding the course of future HUD  research. A 2008 NRC report noted declining resources for data collection  and research and insufficient external input to its research agenda. On the  heels of the report, the scope of the current economic and housing crisis  led the incoming administration to acknowledge a need both to reform and  transform HUD and to sustain a commitment of flexible budget resources  for these efforts. In 2009, HUD proposed a departmentwide  Transformation Initiative of organizational and program management  improvements to position HUD as a high-performing organization. In fiscal  year 2010, much of PD&R’s research and evaluation activities are funded  through a set-aside created for the initiative, which also supports program  measures, demonstrations, technical assistance, and information  technology projects.
section_title: Evaluation at HHS
paragraphs: Evaluation planning is decentralized at the Department of Health and  Human Services. We reviewed ACF and CDC because they have significant  evaluation experience. HHS’s centrally located Office of the Assistant  Secretary for Planning and Evaluation (ASPE) coordinates agency  evaluation activities, reports to the Congress on the department’s  evaluations, and conducts studies on broad, cross-cutting issues while  relying on agencies to evaluate their own programs. In some cases, ASPE  conducts independent evaluations of programs housed within other HHS  operating and staff divisions (for example, ACF and CDC).
ACF oversees and helps finance programs to improve the economic and  social well-being of families, individuals, and communities—the Head  Start program is an example. It also assists state programs for child  support enforcement as well as Temporary Assistance to Needy Families  (TANF). The Office of Planning, Research, and Evaluation (OPRE) is the  principal office for managing evaluation at ACF. It also provides guidance,  analysis, technical assistance, and oversight related to strategic planning,  performance measurement, research, and evaluation methods. It conducts  statistical, policy, and program analyses and synthesizes and disseminates  research and demonstration findings. OPRE consults with outside groups  on ideas that feed into program and evaluation planning. In each policy  area with substantial evaluation resources, OPRE consults with a group of  researchers, program partners, and other content area experts who share  their knowledge and ideas for research and evaluation.
CDC, as part of the Public Health Service, is charged with protecting the  public health by developing and providing to persons and communities  information and tools for preventing and controlling disease, promoting  health, and preparing for new health threats. It supports some evaluation  activities through the Public Health Service (PHS) evaluation set-aside; in  2010 the Secretary was authorized to use up to 2.5 percent of  appropriations for evaluating programs funded under the PHS Act. The  set-aside is also used to fund databases of the National Center for Health  Statistics and programs that cut across CDC’s divisions. Presently, the  divisions within CDC control most evaluation funding focused on their  respective programs, but evaluation planning across CDC is currently  under review. CDC recently created an Office of the Associate Director for  Program which will have responsibility for supporting performance  measurement and evaluation across CDC, among other duties.
We interviewed staff from evaluation offices in three CDC divisions:  Nutrition, Physical Activity, and Obesity (DNPAO); HIV/AIDS Prevention  (DHAP); and Adolescent and School Health (DASH). These three divisions  oversee cooperative agreements with state and local agencies and plan a  portfolio of evaluations. CDC officials suggested that variation in  evaluation planning in these three offices could provide insight into how  CDC’s centers generally prioritize evaluations to conduct.
DNPAO is charged with leading strategic public health efforts to prevent  and control obesity, chronic disease, and other health conditions through  physical activity and healthy eating. DNPAO supports the First Lady’s Lets  Move! campaign to curb childhood obesity, which is considered an  important public health issue but has a limited body of research on  effective practices. The Nutrition, Physical Activity, and Obesity Program  is a cooperative agreement between CDC and 25 state health departments  to support a range of activities, including process and outcome  evaluations. A consulting group of state evaluators, outside experts, and  divisional representation advises DNPAO on proposing evaluation projects  that would be useful to grantees and the divisions.
DHAP, charged with leadership in helping control the HIV/AIDS epidemic,  has a fairly large program evaluation branch that supports national  performance monitoring and evaluation planning. The evaluation branch is  responsible for monitoring CDC-funded HIV prevention programs,  including 65 health units and 150 community organizations. Within the  branch, the Evaluation Studies Team conducts specific evaluations of  interest and in-depth process evaluations and outcome monitoring studies  of selected HIV prevention interventions delivered by community-based  organizations, state and local health departments, and health-care  providers. In addition to the Division’s strategic plan, the governmentwide  National HIV/AIDS Strategy for the United States, released in July 2010,  informs evaluation planning. DHAP’s work is also shaped by an advisory  committee and findings from an external peer review that provided input  into programs and evaluations through the strategic plan.
DASH is considered somewhat unique among CDC’s divisions because it is  not focused on disease or exposure but has a mission to promote the  health and well-being of a particular population— children and  adolescents. DASH funds tribal governments and state, territorial and local  educational agencies to address child and adolescent health issues,  including nutrition, risky sexual behavior, tobacco prevention, school  infrastructure, and asthma management. DASH typically funds evaluations  in one health risk area each year. Its framework, Coordinated School  Health, involves community, family, teachers, and schools in addressing a  diverse set of health issues. It also partners with nongovernmental and  community-based organizations to reach children who are not in school.  DASH supports rapid evaluations to identify innovative programs and  practices. These evaluations typically last 12 to 24 months and data are  collected within a school calendar year. The evaluation team also has a  small portfolio of evaluation research that includes large longitudinal  randomized controlled trials that assess effectiveness over a 5-to-6-year  period.
section_title: The Agencies’ Generally Similar Informal Evaluation Planning Policies
paragraphs: The agencies we reviewed use a similar but informal evaluation planning  process that involves collaboration between each agency’s evaluation  office and program offices, external groups, and senior officials. Typically,  the evaluation office leads an iterative two-step process to develop ideas  into full-scale proposals by obtaining feedback from senior officials and  considering available resources. The process varies across agencies in the  breadth of the studies and programs considered, the use of ranked  competitions, and the amount of oversight by senior officials. Figure 1  depicts the general process and the agencies’ significant differences.
subsections: section_title: The General Process for Developing and Selecting Evaluation Proposals
paragraphs: In most of the agencies we reviewed, evaluation planning generally starts  and ends in the same fiscal year. General procedures for submitting and  clearing annual spending plans structure the evaluation planning process  at several of these agencies because the approved evaluations may involve  external expenditures. The agencies must approve their evaluation plans  by the start of the next fiscal year, or when appropriated funds become  available, so that they can issue requests for proposals from the  contractors that conduct the evaluations. Planning evaluations can include  reviews by policy officials, such as deputy and assistant secretaries, and  budget officials, such as an agency’s chief financial officer. For example,  ACF’s evaluation staff develop evaluation proposals in the fall and early  winter, before sending them to the agency’s assistant secretary for  approval in the late winter to allow the assistant secretary to make  approval decisions in time to meet the deadlines for awarding contracts in  that fiscal year. Although most of the agencies finish their planning by the  start of the next fiscal year, the process can start as late as July at CDC’s  DNPAO or in the fall of the current fiscal year at ACF.
Planning begins at each agency with internal coordination to define the  goals and procedures for developing evaluation proposals. At ACF, this  process begins informally, with evaluation and program staff meeting to  discuss their priorities for the coming year. The other agencies we  reviewed (including Education beginning in 2010) issue memorandums  describing the planning process to the staff members involved. They may  describe the staff members who will lead proposal-development teams, the  substantive focus of the year’s process, the evaluation plan’s connection to  spending plans, and the role of senior officials. They may also give a  schedule of key deadlines. CDC’s DASH distributes a broader call for  project nominations to agency staff members and researchers, state and  local education agencies, and other program partners. In recent years, the  call has specified the type of interventions the division seeks to evaluate,  stated deadlines for submitting nominations, and solicited information  from nominators about particular interventions. CDC’s DNPAO issues a  call for proposals that addresses the process and broad criteria for project  selections that can involve many people and proposals. The calls at each  agency are informal planning documents, however, as no agency we  reviewed has an official policy that specifies the process for developing  and selecting evaluations. Having developed informal processes over time,  senior officials, and evaluation and program office staff have a common  understanding of how they will develop, review, and select evaluations.
After the agencies identify their planning goals and steps, the evaluation  and program staff begin to develop evaluation proposals. At some  agencies, the program staff may develop the initial proposals  independently of the evaluation staff, in response to the same call for  proposals. The program staff may later consult with the evaluation staff to  improve the proposals before they are reviewed further. This process is  common in the CDC divisions we reviewed, where the evaluation staff are  located inside program offices dedicated to particular health issues, so  both program and evaluation staff may individually or jointly submit  proposals for consideration.
At other agencies, the evaluation staff meet with the program staff  specifically to discuss ideas for evaluation and then develop initial  proposals from the input they receive. The evaluation staff at one of these  agencies said they incorporate the priorities, questions, and concerns the  program staff conveyed from their day-to-day experience into evaluation  planning and that collaboration helps ensure later support for the  approved evaluations. Alternatively, HUD’s evaluation unit includes  program staff on the teams that develop proposals in specific policy areas,  such as fair housing and homelessness. The program offices also  contribute to the initial proposals by providing comments to senior  officials. At all the agencies, the evaluation staff use their expertise in  designing social research and assessing the reliability of data, among other  skills, to ensure the quality and feasibility of proposals.
In addition to consulting internal program staff, most of the agencies we  examined consult external groups to obtain ideas for evaluation proposals.  Evaluation staff members cited a number of reasons for consulting  external groups in developing proposals: the ability to identify unanswered  research questions, coordinate evaluations across federal agencies,  uncover promising programs or practices to evaluate, and inform strategic  goals and priorities. Some evaluation staff reported consulting external  groups as they develop program priorities and strategic plans, which they  cited as criteria for planning evaluations. Over the past 2 years, PD&R has  participated in a philanthropic foundation-funded partnership with  research organizations that conducted several research projects to help  inform the Department’s development of an evidence-based housing and  urban policy agenda. Other staff said that they consult with state and local  program partners, such as state welfare offices, to identify potentially  useful projects.
External groups have formal roles in developing proposals at two  agencies. CDC’s DASH directly consults with external researchers, state  and local education officials, and school health professionals for  nominations of promising interventions to evaluate. In planning for fiscal  year 2011, HUD asked the public to submit ideas for evaluation on its  “HUD User” Web site. At most of the agencies, however, external groups  do not explicitly develop evaluation proposals. For example, ACF staff  said they informally consult with researchers about possible evaluation  topics, partly in regular research conferences, but they do not ask their  advisory panels or individual researchers to review specific evaluation  proposals. In recent years, PD&R also contacted the office of HUD’s  Inspector General for evaluation ideas that build on that office’s work.
Generally, the agencies review and approve evaluation proposals in two  steps. First, evaluation or program staff members develop ideas or brief  concept papers for initial feedback from senior officials. The feedback can  involve a series of proposal development meetings, as at Education and  ACF, where senior officials give staff members strategic direction to  develop and refine their proposals. Alternatively, senior officials may  review all draft proposals that the evaluation and program staff have  developed independently, as at HUD and CDC’s DNPAO and DHAP. Initial  feedback helps prevent staff from investing large amounts of time in  proposals that would have a small chance of being approved. The  feedback expands proposal development beyond the evaluation and  program offices and helps ensure that the proposals support the agency’s  broader strategic goals.
Second, once the initial proposals are sufficiently well developed, senior  officials review and select from a group of revised, full-scale proposals.  These may contain detailed information about cost and design for later use  in the contracting process. Evaluation officials at ACF and HUD select  from the pool of revised proposals those they wish to present to agency  leaders, such as the secretary or assistant secretary, for final approval.  Branch leaders at CDC’s DNPAO and DHAP choose a group of proposals  to compete for division resources against proposals from other branches  within their divisions. Review panels rank-order all proposals (discussed  below), and then division leaders decide, based on the rankings and  available resources, which proposals the division will fund. In fiscal year  2011, Education staff plan to present to senior officials the entire proposed  evaluation portfolio, identifying how evaluation studies address key  questions and agency priorities.
section_title: The Significant Differences in Agency Processes
paragraphs: Some of the agencies we reviewed focus specifically on planning program  evaluations, while others use the same annual process to plan a variety of  analytic studies. The central evaluation offices at ACF, Education, and  HUD perform a continuum of research and evaluation studies that may  include collecting national survey data, conducting policy analyses, and  describing local program activities, among other activities. These agencies  use the same process to make funding decisions across these various  analysis proposals, which allows them to weigh the pros and cons of  evaluation against other information needs when sufficient funds are  available. Consequently, program evaluations may compete with other  types of studies that require specific funding each year.
Although the evaluation branch of CDC’s DNPAO provides a narrower  range of services, the division uses a similar, unified process to decide  how to develop proposals for all evaluations and research activities. In  contrast, DASH plans their different types of studies separately. It uses  one annual process to develop evaluation proposals for promising  practices and interventions often implemented by grantees. It uses a  different process to develop “evaluation research” proposals, which  evaluation staff defined as national-level evaluations or long-term studies  of program impact, often involving randomized controlled trials. By  considering these types of study separately, DASH does not require longer- term evaluations to compete with shorter-term studies for the same funds.
Programs compete against one another for evaluation resources at some  but not most of the agencies we reviewed. The scope of evaluation  planning at one group of agencies is limited to the same programs or  policy areas each year. These agencies have designed their planning  processes to select not programs to evaluate but evaluation questions to  answer in a program area. For example, the ACF evaluation staff indicated  that they identify important questions for each program with evaluation  funding and then allocate funds to the most important questions for each  program. Consequently, the agency typically conducts evaluations in  programs with evaluation funds (such as TANF) every year but has not  evaluated programs which do not have evaluation funding (such as the  Community Services and Social Services Block Grants).
HUD and, to a certain extent, two CDC divisions seek to identify which  programs are important to evaluate as well as what questions are  important to answer about those programs. Agency staff have the  flexibility to direct resources to the programs that they believe most need  evaluation. HUD evaluation staff said that this broad scope allows them to  build a portfolio of evaluations across policy areas and serve the agency’s  most pressing evaluation needs. Senior officials consider the value of  proposals from all policy areas combined but make some effort to achieve  balance across policy areas.
Only CDC’s divisions hold formal, ranked competitions to review and  select proposals. In each division, staff members or external panels rate  and rank all evaluations the evaluation and program offices propose, once  they have been fully developed. Senior leaders at CDC’s DHAP and  DNPAO select proposals by rank and available funds. In addition, senior  leaders at DNPAO rank and select proposals within each of its three policy  areas: nutrition, physical activity, and obesity. Senior leaders at DASH  consider information collected from site visits and interviews for a small  group of semi-finalists that were selected based on the input of the  external panel. CDC staff reported that CDC often uses ranked  competitions to award grants and contracts across the agency. At the  other agencies we reviewed, evaluation staff said that proposals are  reviewed and selected in a series of discussions between the agency’s  policy officials, such as assistant or deputy secretaries, and the senior  leaders of its evaluation and program offices. None of these agencies  reported formally ranking their proposed evaluations but, instead,  qualitatively consider the relative strengths and weaknesses of each  evaluation.
Proposal review and selection in the CDC divisions involves less  department-level input than at ACF, Education, and HUD. CDC’s  evaluation staff reported that division leadership makes the final decision  on evaluation projects and does not need the approval of the Office of the  CDC Director or HHS officials, although a key criterion in project ranking  and selection is often alignment with larger CDC, HHS, and national  priorities. CDC is studying evaluation planning across the agency,  however, and may increase central oversight in the future. The assistant  secretary at ACF, not departmental officials, makes final approval  decisions, but the evaluation staff reported consulting informally with staff  of the Office of the Assistant Secretary for Planning and Evaluation  (ASPE) when proposals are developed.
The processes at Education and HUD are more centralized than at CDC or  ACF. At these agencies, senior department officials—such as the  secretary, deputy secretary, or assistant secretary—make the final  selection decisions, after the evaluation and program staff have developed  and reviewed the initial proposals. Beginning in fiscal year 2010, HUD staff  indicated that the agency funded many of its evaluations from a  departmental Transformation Initiative fund, whose board must approve  proposed evaluations and other projects. Board members include the  assistant secretaries of PD&R and Community Planning and Development,  the Chief Information Officer, as well as the Director of Strategic Planning  and Management.
One agency does not strictly plan evaluations for the next fiscal year.  CDC’s DASH staff plan evaluations that are funded during the current  fiscal year rather than evaluations that will be funded in the next fiscal  year. Local education agencies typically partner with the agency to  conduct evaluations during the school year, when parents, students, and  teachers are available to researchers. As a result, the agency cannot wait  until funds are scheduled for appropriation in October or later, because  their data collection plans and site selections must be final before the  school year begins, typically in late August or early September.
Education adjusted its evaluation planning guidance in 2010 to explicitly  plan evaluations to be conducted in fiscal year 2011 as well as to inform its  budget request for fiscal year 2012. The agency links its evaluation  planning to the budget, partly to ensure that funding or authority will be  available and that evaluations are aligned with program goals and  objectives, congressional mandates, and the agency’s strategic priorities.  In addition, Education has proposed, for reauthorization of the Elementary  and Secondary Education Act, to submit a biennial evaluation plan to the  Congress and establish an independent advisory panel to advise the  department on these evaluations. These plans align well with the  American Evaluation Association’s (AEA) recommendation, made in a  recent policy paper on federal government evaluation, that federal  agencies prepare annual and multiyear evaluation plans to guide program  decision-making and consult with the Congress and nonfederal  stakeholders in defining program and policy objectives, critical operations,  and definitions of success.
section_title: Common Types of Criteria Agencies Use to Prioritize Evaluation Proposals
paragraphs: We found these mature agencies remarkably similar in the four general  criteria they used for selecting evaluations to conduct during the next  fiscal year: strategic priorities, program concerns, critical unanswered  questions, and the feasibility of conducting a valid evaluation study.  Another important consideration, in situations in which several program  offices draw on the same funding source, was establishing balance across  program areas. Most agencies indicated no hierarchy among these criteria.  Rather, they considered them simultaneously to create the best possible  portfolio of studies.
The first criterion, strategic priorities, represents major program or policy  areas identified as a focus of concern and reflected in a new initiative or  increased level of effort. Strategic priorities might be expressed by a  department or the White House as strategic goals or secretarial initiatives  or by the Congress in program authorizations or study mandates. Under  GPRA, agencies are expected to revise their strategic plans at least every 3  years, providing an opportunity to align their plans with current  conditions, goals, and concerns. The plans can chart expectations for both  program and evaluation planning. CDC’s DHAP officials described waiting  for the White House release of the National AIDS Strategy in July to  finalize their strategic plan and objectives and to prioritize evaluation  activities that would address them. In addition to national priorities,  division priorities are informed by their own research, surveillance, and  program evaluation, identifying the subpopulations and geographic areas  most affected by the disease. HUD’s PD&R conducts the national Housing  Discrimination Study every 10 years, which provides a unique benchmark  and input to the department’s long-term planning.
Strategic priorities may also arise from congressional mandates.  Education officials noted that the Congress generally mandates  evaluations when it reauthorizes large formula grant programs, such as the  national assessments of title I of the Elementary and Secondary Education  Act, and that it has also mandated the evaluation of major new programs  that might have great public interest or promise. They said that they  schedule evaluations so that they will produce useful information for  reauthorization, usually every 6 to 8 years.
The second criterion, program-level concerns, represents more narrowly  focused issues concerning an identified problem or opportunity.  Evaluation staff reported that valuable ideas for evaluations often  reflected the questions and concerns that arise in daily program  operations. ACF noted that Head Start teachers’ reports of disruptive  children who prevented other children from learning led to a large-scale  evaluation of several potentially effective practices to enhance children’s  socio-emotional development and teachers’ classroom management  practices.
Accountability concerns that OMB, GAO, and Inspector General reports  raise may lead to follow-up studies to assess the effectiveness of  corrective actions. For example, PD&R staff stated that after a GAO report  criticized the Section 202 demonstration grant program for not building  housing projects in a timely fashion, the Congress introduced a  competition for grants to speed up development. A follow-up evaluation  will assess whether timeliness has improved. Other evaluation questions  may address crosscutting issues that influence program success, such as a  provider’s ability to leverage resources or promote partnerships with other  stakeholders. CDC’s DNPAO places a priority on proposals that develop  collaborations with external partners and among operational units within  the division.
The third criterion, critical unanswered questions, reflects the state of  knowledge and evidence supporting government policies and programs.  For example, agency staff talk with advisory groups, academics and other  researchers in their field to identify useful research and evaluation  questions that could advance understanding of the field and improve  practice. CDC staff indicated that filling knowledge gaps is a particularly  important criterion for project selection, because some public health areas  lack an extensive research base or evidence on effective practices. An  OPRE senior official described OPRE staff as looking for compelling,  essential questions of enduring interest. ACF programs attempt to solve  persistent social problems, such as testing diverse strategies to promote  employment retention and advancement for low-wage workers and  current or former TANF recipients. Because formal impact evaluations of  these efforts may take 5 or 6 years to complete, OPRE staff look for  questions that are persistent and studies that are likely to advance  knowledge. Gathering information on emerging, promising practices was a  consideration, particularly where evidence of effective practice has not yet  been demonstrated. This was particularly important to the CDC divisions,  DNPAO and DASH, where the public health research base was limited and  effectiveness evaluations of promising practices were needed to expand  the pool of evidence-based interventions to offer grantees.
The fourth criterion, evaluation feasibility, encompassed a range of  pragmatic issues such as whether data were available and at what cost,  whether the proposed evaluation could answer questions persuasively,  and whether grantees had the interest and capacity to participate in  evaluation. Naturally, agencies weigh their evaluation priorities in the  context of their fiscal and budget constraints. Evaluators described  determining whether the most important questions could be answered and  the resources that would be needed to answer them. When “hard” data are  lacking, some evaluators find that in-house exploratory work and  investment in data gathering may be needed before scaling up to a  contracted evaluation. Like the other evaluation units, PD&R compares  the feasibility and cost of a study to alternative proposals. The evaluation  staff noted that cost cannot be the sole criterion, however, because studies  of some programs, such as the large block grants, are more resource  intensive than the approaches available for studying other programs, such  as housing voucher programs.
When working with community-based organizations, agencies find grantee  evaluation capacity can be very important. To ensure that the selected  grantee is implementing the program faithfully, is ready for evaluation,   and can collect valid and reliable data, CDC’s DASH staff conduct site  visits to assess candidate projects on such issues as appropriate logical  links between program components and expected outcomes, staff  turnover, political conflicts, fiscal sustainability, and staff interest in and  capacity to conduct the evaluation. ACF evaluators were pleased to note  that many state and local TANF officials participate in OPRE’s annual  welfare research conference, show interest in conducting evaluations, and  have been willing to randomly assign recipients to new programs for  research purposes.
section_title: Prioritization Depends on Funding Sources and Agency Circumstances
paragraphs: Although the agencies generally followed a similar process in developing  their evaluation agendas, some agency characteristics or conditions  appeared to influence their choices and may be important for other  agencies to consider as they develop their own evaluation agendas. The  four conditions we identified as strongly influencing the evaluation  planning process were  1.  the location of evaluation funding, whether with the program or  2.  the scope of the evaluation unit’s responsibility within the agency;  3.  how much the evaluators rely on program partners; and  4.  the extent and form of congressional oversight over agency program  evaluations.
subsections: section_title: Evaluation Funding’s Location
paragraphs: Where evaluation funds come from largely controls the selection of  programs to evaluate. In ACF, CDC, and Education, authority and funds  for evaluation are primarily attached to the programs, not to the  evaluation office. This has implications for both how evaluation offices  conduct their planning and for whether a program is likely to be evaluated.
Where evaluation funds and authority are tied to the program, and funds  are available, evaluation staff generally choose not which programs to  evaluate but which research questions to answer. Thus, evaluators in ACF  and Education work separately with each program office that has  evaluation funds to develop proposals.
In contrast, at HUD, when the evaluation office has uncommitted  evaluation funds, selecting proposals can involve deciding between  programs. Therefore, besides considering policy priorities and feasibility  issues, HUD senior managers try to balance available evaluation funding  across programs or policy areas after proposals are developed within  program areas. This involves soliciting input from program office leaders  on the preliminary agenda and discussing competing needs in the final  selection process. CDC’s DNPAO, with its three distinct program areas— nutrition, physical activity, and obesity—made similar efforts to obtain a  balanced portfolio by forming teams to rank order proposals separately  and having senior division leaders consider program balance in selecting  proposals.
One consequence of tying evaluation funds and authority to programs is  that programs that do not have their own evaluation authority may not get  evaluated at all. Staff at ACF and Education told us that because their  evaluation offices did not have significant discretionary funds for external  contracts, they had not conducted any evaluations of several programs,  even though they believed that some of those programs should be  evaluated. Not discussing the pros and cons of evaluating a particular  program can lead to inappropriately excluding some from evaluation. HUD  officials noted that it was important to attempt to balance evaluation  spending across program areas because, otherwise, some programs might  be avoided as too difficult or expensive to evaluate. Education officials  said they plan to address this issue by developing a departmental portfolio  of strong evaluation proposals based on policy and management needs,  without regard to existing evaluation authority, and then request funds for  them. Then, in future legislative proposals, they plan to ask the Congress  for more flexibility in evaluation funds to better meet the field’s needs.
section_title: Scope of Responsibility
paragraphs: The agency evaluation offices we examined were located at different  organizational levels, affecting the scope of their program and analytic  responsibilities as well as the range of issues they considered.
At CDC, the evaluation offices are generally within program offices, so  they do not need a separate step for consulting with program staff to  identify their priorities. Instead, the divisions solicit evaluation proposals  from staff throughout the division. In the other agencies we examined,  evaluation offices are either parallel to program offices (ACF) or at the  departmental level (Education and HUD), which leads them to consult  more formally with the program offices during both development and  selection.
Location and scope of responsibilities also influenced evaluation approval.  CDC’s divisions, with the narrowest scope among the units we examined,  exerted considerable control over their evaluation funds and did not  require approval of their evaluation agendas by either the director or the  department. DASH did, however, report coordinating evaluation planning  with other agencies and HHS offices on specific cross-cutting programs,  and DHAP reported delaying its selection of evaluation proposals this past  spring to coordinate with the new National AIDS Strategy. In contrast, at  Education and HUD, where evaluation offices have departmental scope,  final approval decisions are made at the department level. In the middle,  OPRE selections are approved by the ACF assistant secretary and do not  require departmental approval.
Being responsible for a wide range of analytic activities also influenced an  evaluation office’s choice of evaluations. Evaluators in the more  centralized offices in ACF and HUD described having the flexibility to  address the most interesting questions feasible. For example, if it is too  early to obtain hard data on an issue, PD&R staff said that they might turn  to in-house exploratory research on that issue. ACF staff noted that they  often conducted small descriptive studies of the operations of state TANF  programs because of the decentralized nature of that program. This  flexibility can mean, however, that they must also consider the range of  the program office’s information needs when developing their portfolio of  studies. PD&R staff noted that they try to ensure that some studies are  conducted in-house to meet program staff interest in obtaining quick  turnaround on results. DNPAO aims to achieve a balanced portfolio of  studies by ranking cross-cutting proposals within categories of purpose,  such as monitoring or program evaluation. Education officials propose to  create a comprehensive departmental evaluation plan that identifies the  department’s priorities for evaluation and other knowledge-building  activities, is aligned with their strategic plan, and will support resource  allocation.
Several of the evaluation offices we examined also provide technical  assistance in performance monitoring and evaluation. While this may help  strengthen relationships with program staff and understanding of program  issues, the responsibility can also reduce the resources available for  evaluation studies. All three CDC divisions require evaluations or  performance monitoring from their grantees; therefore, providing grantees  with technical assistance is a major activity for these evaluation offices. In  DHAP and DNPAO, staff workload, including providing technical  assistance, was cited among the resource constraints in developing  evaluation proposals. ACF staff noted that if program offices prioritize  their available funds on technical assistance and monitoring, there may  not be enough to conduct an evaluation.
In our cases, placing the evaluation office inside the program office (as in  the CDC divisions we examined) was associated with conducting more  formal proposal ranking. We considered several possible explanations for  this: (1) staff adopted the competitive approach they generally take in  assessing proposals for project and research funds, (2) a large volume of  proposals required a more systematic process for making comparisons, or  (3) the visibility of the selections created pressure for a transparent  rationale. The first point may be true but does not explain why the other  agencies are also deliberative in assessing and comparing evaluation  proposals but do not rate them numerically.
The two other explanations appear to be more relevant and may be related  to the fact that evaluations are being selected within the program office  and thus cover a relatively narrow range of options. CDC staff said that  they did not need to formally rate and rank the three or four proposals  they submitted for OMB’s Evaluation Initiative but might have done so had  the number of proposals to consider been greater. DASH and DHAP issue  broad calls each year for nominations of promising practices to evaluate  and, thus, gather a large number of proposals to assess. Staff in DASH,  which also solicits project nominations from the public, indicated that  over time their process has become more formal, accountable, and  transparent so that selections appear to the public to be more systematic  and less idiosyncratic. Although information is limited, we believe that  systematically rating and ranking proposals may be a useful procedure to  consider case by case.
section_title: Reliance on Program Partners
paragraphs: The influence of nonfederal program partners on developing and selecting  evaluation proposals was observed in most of the agencies we examined,  although it did not vary much among them. The importance of program  stakeholders to planning should be expected because these particular  agencies generally rely on external partners—state and local agencies and  community–based organizations—to implement their programs. However,  the extent of coordination with external parties on evaluation planning  seen here may not be necessary in agencies that are not so reliant on third  parties. ACF’s evaluation staff pointed out that they cannot evaluate  practices that a state or local agency is not willing to use. Efforts to engage  the academic and policy communities in discussing ideas for future  research at ACF and Education also reflect these agencies’ decades-long  history of sponsoring research and evaluation. CDC’s DHAP and DNPAO  also employ advisory groups, including CDC staff and external experts, to  advise them on strategic planning and topics that will help meet the needs  of their grantees, but only DASH involved external experts directly in  assessing evaluation proposals. DASH evaluators assemble panels to  assess nominations of sites implementing a promising practice; depending  on the topic and stage of the process, these panels might include external  experts and experts from across CDC or other agencies serving children  and families.
Program partners’ evaluation capacity is especially important to evaluation  planning in the CDC divisions we examined because their evaluations tend  to focus on the effectiveness of innovative programs or practices. Each  year, DASH publicly solicits nominations of promising projects of a  designated type of intervention and uses review panels and site visits to  rank dozens of sites on an intervention’s strength and promise, as well as  the feasibility of conducting an evaluation. Staff said that it was important  to ensure that the grantee organization was stable and able to cooperate  fully with an evaluation and noted that evaluation is sometimes difficult  for grantees.
section_title: The Form of Congressional Oversight
paragraphs: Congress influences agencies’ evaluation choices in a variety of ways.  Congress provides agencies with the authority and the funds with which to  conduct evaluations and may mandate specific studies. The evaluation  offices in ACF, Education, and HUD all noted their responsibility to  conduct congressionally mandated evaluation studies in describing the  criteria they used in evaluation planning. The CDC offices indicated that  they did not have specific study mandates but, rather, authority to conduct  studies with wide discretion over the particular evaluation topics or  questions. Of course, in addition to legislatively mandating studies, the  Congress expresses interest in evaluation topics through other avenues,  such as oversight and appropriations hearings. DHAP evaluators noted  that they receive a lot of public scrutiny and input from the Congress and  the public health community that works its way into project selection  through the division’s setting of priorities.
Agency evaluators described a continuum of evaluation mandates, from a  general request for a study or report to a list of specific questions to  address. Education officials noted that the Congress generally mandates  evaluations of the largest programs when they are reauthorized or new  programs or initiatives for which public interest or promise might be great.  Some evaluators noted that sometimes the Congress and agency leaders  want answers to policy questions that research cannot provide. They  indicated that, where legislative language was vague or confusing, they did  their best to interpret it and create a feasible evaluation. In a previous  study of agency studies’ not meeting congressional information needs, we  suggested that expanding communication between congressional staff and  agency program and evaluation staff would help ensure that information  needs are understood and that requests and reports are suitably framed  and adapted as needs evolve.
Evaluators told us that whether and how much funding was attached to an  evaluation mandate also influenced how the mandate was implemented.  They said that when appropriate funding was available, they always  conducted congressionally mandated evaluations. However, sometimes  the amounts available do not reflect the size of the evaluation needs in a  program. This was particularly a problem for small programs where a  fixed set-aside of program funds for evaluation might yield funds  inadequate for rigorous evaluation.
Evaluators described a related challenge when evaluation authorities are  attached to single programs which preclude pooling funds across  programs. Such limitations on using evaluation funds could lead to missed  opportunities to address cross-cutting issues. In cases where no additional  funding was provided for legislatively mandated studies, agencies had to  decide how and whether to fund them. Some agency evaluators told us  that they generally conducted what they saw as “unfunded mandates” but  would interpret the question and select an approach to match the funds  they had available. This might mean that without funds to collect new  data, a required report might be limited to simply analyzing or reporting  existing data.
HUD receives considerable congressional oversight of its research and  evaluation agenda, reflecting congressional concern about its past  research priorities and greater decision-making flexibility under the new  Transformation Initiative. In 2008, a congressionally requested National  Research Council review of HUD’s research and evaluation lauded most of  PD&R’s work as “high quality, relevant, timely, and useful” but noted that  its resources had declined over the previous decade, its capacity to  perform effectively was deteriorating, and its research agenda was  developed with limited input from outside the department.
NRC recommended that, among other things, HUD actively engage  external stakeholders in framing its research agenda. In response, PD&R  solicited public suggestions online for research topics for fiscal year 2011  and beyond. In addition, HUD proposed a Transformation Initiative of  organizational and program management improvement projects in 2009  and asked that up to 1 percent of its program budget be set aside in a  proposed Transformation Initiative fund to support research and  evaluation, information technology, and other projects. The House and  Senate Appropriations Committees approved the fund (at somewhat less  than the requested amount) with a proviso that HUD submit a plan for  appropriations committee approval, detailing the projects and activities  the funds would be used for.
section_title: Concluding Observations
paragraphs: An effective evaluation agenda aims to provide credible, timely answers to  important policy and program management questions. In setting such  agendas, agencies may want to simultaneously consider the four general  criteria we identified: strategic priorities, program concerns, critical  unanswered questions, and the feasibility of conducting a valid study. In  the short run, because agency evaluation resources are limited, ensuring  balance in evaluations across programs may not be as important as  addressing strategic priorities. However, developing a multiyear evaluation  plan could help ensure that all an agency’s programs are examined over  time.
To produce an effective evaluation agenda, agencies may want to follow  the general model we identified at the agencies we reviewed: professional  evaluators lead an iterative process of identifying important policy and  program management questions, vetting initial ideas with the evaluations’  intended users, and scrutinizing the proposed portfolio of studies for  relevance and feasibility within available resources. Since professional  evaluators have the knowledge and experience to identify researchable  questions and the strengths and limitations of available data sources, they  are well suited to leading a consultative process to ensure that decision  makers’ information needs can be met.
However, agencies may need to adapt the general model’s steps to match  their own organizational and financial circumstances. For example, they  may not need to formally rank proposals unless they have many more  high-quality proposals than they can fund. They may find advantages to  placing evaluation offices within program offices (for focusing on program  needs, for example) and at higher levels (for addressing broader policy  questions). Where analytic demands are significant and resources permit,  they may find a combined approach best-suited to their needs.
To ensure that their evaluations provide the information necessary for  effective management and legislative oversight, evaluation offices are  likely to need to seek out in advance the interests and concerns of key  program and congressional stakeholders, especially program partners, and  discuss preliminary proposals with the intended users.
section_title: Agency Comments
paragraphs: The Departments of Health and Human Services and Housing and Urban  Development provided comments on a draft of this report, which are  reprinted in appendixes I and II.
HHS appreciated the attention that this report gives to the importance of  strong prioritization processes for selecting evaluation studies and  allocating resources to complete them, and was pleased that the practices  of ACF and CDC in this area are models for emulation by others. It also  noted that, given the diversity of purposes for evaluations, the optimal  location and organization of evaluation activities will vary with the  circumstances. This is consistent with our concluding observation that  agencies may need to adapt the general model—including where to locate  evaluation offices—to match their own organizational and financial  circumstances.
HUD agreed with our description of how it plans evaluations but was  concerned that the report did not place enough emphasis on the  appropriations process as a major influence on what projects it funds and  when it can begin the contracting process. We have added text to note that  the Congress influences the agencies’ evaluation processes through  providing them with both the authority and funds with which to conduct  evaluations, as well as mandating specific studies.
Education, HHS, and HUD also provided technical comments that were  incorporated where appropriate throughout the text.
We are sending copies of this report to the Secretaries of Education,  Health and Human Services, and Housing and Urban Development; the  Director of the Office of Management and Budget; and appropriate  congressional committees. The report is also available at no charge on  GAO’s Web site at www.gao.gov.
If you have questions about this report, please contact me at (202) 512- 2700 or kingsburyn@gao.gov. Contacts for our Office of Congressional  Relations and Office of Public Affairs are on the last page. Key  contributors are listed in appendix III.
Nancy Kingsbury Managing Director  Applied Research and Me , Ph.D.
section_title: Appendix I: Comments from the Department of Health and Human Services
section_title: Appendix II: Comments from the Department of Housing and Urban Development
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Stephanie Shipman, Assistant  Director; Valerie Caracelli; and Jeff Tessin made significant contributions  to this report.
section_title: Bibliography
paragraphs: American Evaluation Association. An Evaluation Roadmap for a More  Effective Government. September 2010. www.eval.org/EPTF.asp  Leviton, Laura C., Laura Kettel Khan, and Nicola Dawkins, eds. “The  Systematic Screening and Assessment Method: Finding Innovations Worth  Evaluating.” New Directions for Evaluation no. 125, 2010.
National Research Council, Committee to Evaluate the Research Plan of  the Department of Housing and Urban Development, Center for Economic,  Governance, and International Studies, Division of Behavioral and Social  Sciences and Education. Rebuilding the Research Capacity at HUD.  Washington, D.C.: National Academies Press, 2008.
Office of Management and Budget. Analytical Perspectives—Budget of the  United States Government, Fiscal Year 2011. Washington, D.C.:  Executive Office of the President, Feb. 1, 2010.
Office of Management and Budget. Evaluating Programs for Efficacy and  Cost-Efficiency. M-10-32 Memorandum for the Heads of Executive  Departments and Agencies. Washington, D.C.: Executive Office of the  President, July 29, 2010.  www.whitehouse.gov/sites/default/files/omb/memoranda/2010/m10-32.pdf  Office of Management and Budget. Increased Emphasis on Program  Evaluations. M-10-01 Memorandum for the Heads of Executive  Departments and Agencies. Washington, D.C.: Executive Office of the  President, Oct. 7, 2009.  www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10- 01.pdf  U.S. Department of Education, Office of Planning, Evaluation, and Policy  Development. A Blueprint for Reform: The Reauthorization of the  Elementary and Secondary Education Act. Washington, D.C.: March 2010.
U.S. Department of Health and Human Services. Evaluation: Performance  Improvement 2009. Washington, D.C.: 2010.
section_title: Related GAO Products
paragraphs: Employment and Training Administration: Increased Authority and  Accountability Could Improve Research Program. GAO-10-243.  Washington, D.C.: January 29, 2010.
Program Evaluation: A Variety of Rigorous Methods Can Help Identify  Effective Interventions. GAO-10-30. Washington, D.C.: November 23, 2009.
Continuing Resolutions: Uncertainty Limited Management Options and  Increased Workload in Selected Agencies. GAO-09-879. Washington, D.C.:  September 24, 2009.
Results-Oriented Management: Strengthening Key Practices at FEMA  and Interior Could Promote Greater Use of Performance Information.  GAO-09-676. Washington, D.C.: August 17, 2009.
Performance Budgeting: PART Focuses Attention on Program  Performance, but More Can Be Done to Engage Congress. GAO-06-28.  Washington, D.C.: October 28, 2005.
Program Evaluation: OMB’s PART Reviews Increased Agencies’  Attention to Improving Evidence of Program Results. GAO-06-67.  Washington, D.C.: October 28, 2005.
Managing for Results: Enhancing Agency Use of Performance  Information for Management Decision Making. GAO-05-927. Washington,  D.C.: September 9, 2005.
Performance Measurement and Evaluation: Definitions and  Relationships. GAO-05-739SP. Washington, D.C.: May 2005.
Program Evaluation: An Evaluation Culture and Collaborative  Partnerships Help Build Agency Capacity. GAO-03-454. Washington,  D.C.: May 2, 2003.
Program Evaluation: Improving the Flow of Information to the  Congress. GAO/PEMD-95-1. Washington, D.C.: January 30, 1995.