Program Evaluation: Annual Agency-Wide Plans Could Enhance Leadership Support for Program Evaluations
Published: 2017-09-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GPRAMA aims to ensure that agencies use performance information in decision making to achieve results and improve government performance. GPRAMA requires GAO to evaluate the act's implementation; this report is one in a series on its implementation. GAO examined the extent of agencies' use of program evaluations—a particular form of performance information—and factors that may hinder or facilitate their use in program management and policy making.
GAO surveyed a stratified random sample of 4,395 federal civilian managers and supervisors to obtain their perspectives on several results-oriented management topics, including the extent of and factors influencing evaluation use. GAO compared the results to those of a similar GAO survey of federal managers in 2013 and a GAO survey of Performance Improvement Officers in 2014. GAO also interviewed OMB staff and reviewed guidance on using evaluation in decision making.
section_title: What GAO Found
paragraphs: In a 2017 government-wide survey, GAO found that most federal managers lack recent evaluations of their programs. Forty percent reported that an evaluation had been completed within the past 5 years of any program, operation, or project they were involved in. Another 39 percent of managers reported that they did not know if an evaluation had been completed, and 18 percent reported having none. Managers who reported having evaluations also reported that those evaluations contributed to a great or very great extent to improving program management or performance (54 percent) and to assessing program effectiveness or value (48 percent). These figures are not statistically different from the results of GAO's 2013 survey.
Of the 40 percent of managers who reported having evaluations, the factor most often rated as having hindered use to a great or very great extent, as in 2013, was lack of resources to implement the evaluation findings (29 percent). Managers reported limited knowledge of congressional support for using their results; 35 percent were not able to judge whether lack of support was a barrier.
Federal managers who reported having evaluations most frequently reported that agency leadership support for evaluation, staff involvement, and an evaluation's relevance to decision makers facilitated evaluation use. GAO previously reported that involving agency staff in planning and conducting evaluations helps to ensure they are relevant, credible, and used in decision making. The Office of Management and Budget (OMB) encouraged agencies to use the annual strategic reviews the GPRA Modernization Act of 2010 (GPRAMA) requires to assess evidence gaps and inform their strategic decisions and budget making.
GAO and OMB have noted the importance of developing an evaluation plan or agenda to ensure that an agency's scarce research and evaluation resources are targeted to its most important issues. While 28 percent of managers with evaluations rated consultation with stakeholders high for facilitating use, another 22 percent reported having no basis to judge. GAO previously noted limited knowledge of agency consultation with the Congress. While 23 percent of managers with evaluations reported congressional requests or mandates facilitated evaluation use, more (31 percent) reported having no basis to judge.
GAO concludes that
Agencies' continued lack of evaluations may be the greatest barrier to their informing managers and policy makers and constitutes a lost opportunity to improve the efficiency and effectiveness of limited government resources.
Although only some agencies have developed agency-wide evaluation plans, evaluators who have them found that obtaining stakeholder input helped ensure evaluation relevance and facilitate use of their results.
Congressional consultation on agency evaluation plans could increase the studies' credibility with those whose support is needed to implement program reforms.
An agency's annual strategic review provides a good opportunity to help target its evaluation agenda to its management, budget, and policy priorities.
section_title: What GAO Recommends
paragraphs: To help ensure that agencies obtain the evidence needed to address important questions to improve program implementation and performance, GAO recommends that the Director of OMB direct federal agencies to prepare an annual agency-wide evaluation plan that describes the congressional and other stakeholders that were consulted.
OMB staff stated that agencies should be encouraged, rather than directed, to create an annual evaluation plan. Because OMB has already been encouraging evaluation, GAO believes a more directive approach is needed.

section_title: Letter
paragraphs: The federal government faces significant financial and performance  management challenges in its efforts to meet current and future needs  while constraining spending. Performance measures and program  evaluations can be key in program planning, management, and oversight  by providing feedback on program design and execution. The reporting  requirements of the Government Performance and Results Act of 1993  (GPRA) were intended to provide both congressional and executive  decision makers with objective information on the relative effectiveness  and efficiency of federal programs and spending.
Although GPRA helped improve the availability of agency performance  information, federal managers reported that the use of performance data  for decision making was limited. The GPRA Modernization Act of 2010  (GPRAMA) makes additional changes to agency planning and reporting  requirements to ensure that executive branch agencies use performance  information in decision making and holds them accountable for achieving  results and improving government performance. The act also established  agency leadership positions, including the role of Performance  Improvement Officer (PIO) to promote the use of evidence to improve  program performance. However, our 2013 survey found few significant  changes in federal managers’ reported use of performance information.
In the same period, the Office of Management and Budget (OMB)  encouraged agencies to strengthen their program evaluations— systematic studies of program performance—and expand their use of  evidence and evaluation in budget, management, and policy decisions  with the goal of improving government effectiveness. However, in our  2013 survey of federal managers government-wide, only an estimated 37  percent reported having access to recent evaluations of their programs.
GPRAMA requires GAO to periodically review how its implementation is  affecting agency performance management. This report is one of a  series of reports responding to that mandate. Here, we explore how  agency evaluation use has or has not changed since our prior surveys.
Our objectives here were to identify, since 2013, the extent to which  1.  agency managers’ reported access to and use of program evaluations  2.  agency managers’ views of the factors that facilitate or hinder the use  of program evaluation have changed.
To address these objectives, we surveyed a stratified random sample of  4,395 persons from a population of approximately 153,779 civilian  managers and supervisors working in the 24 executive branch agencies  covered by the Chief Financial Officers Act of 1990 (CFO Act agencies).  The questionnaire was designed to obtain the observations and  perceptions of respondents on various performance management issues.  The web-based survey was administered from November 2016 through  March 2017. About 67 percent of the eligible sample responded with  usable questionnaires. The sample allowed us to generalize our results to  the government-wide population of federal managers.
The government-wide percentage estimates based on our sample from  2017 presented in this report have 95 percent confidence intervals within  plus or minus 4 percentage points of the estimate itself for the initial  question about whether an evaluation had been completed, and within  plus or minus 7 percentage points for subsequent questions about the  use of those evaluations. (Appendix I has more information on the  survey.)
This report first analyzes responses to a subset of survey questions  concerning managers’ access to and use of evaluation and their views on  the factors that influence evaluation use. It then compares these  responses to similar questions in our 2013 survey of federal managers.  We then compare the managers’ responses in 2017 to questions raised in  our 2014 survey of the PIOs of the 24 CFO Act agencies about their  agencies’ investments in evaluation capacity and their views on the  usefulness of various resources and activities for building the capacity to  conduct and use evaluation in decision making. We surveyed the PIOs  because of the central role GPRAMA and OMB assigned these senior  officials to promote agency performance assessment and improvement  efforts and help agencies secure evaluations and other research, as  needed. In the 2014 survey, the senior executive PIOs reported uneven  levels of evaluation capacity across government and some efforts to  increase their agencies’ evaluation capacity.
We further discuss the 2017 survey’s results in a September 2017 report  summarizing our body of work on the implementation of GPRAMA and in  a supplement showing the responses to all survey items at the  government-wide and individual agency levels. We also interviewed  OMB staff about their efforts to encourage agencies to strengthen their  conduct and use of evaluations in decision making, and we reviewed  OMB’s and others’ guidance on using evaluation in decision making.
We conducted this performance audit from April 2016 to September 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Program evaluations are systematic studies that use research methods to  address specific questions about program performance. Evaluation is  closely related to performance measurement and reporting. Whereas  performance measurement entails the ongoing monitoring and reporting  of program progress toward preestablished goals, program evaluation  typically assesses the achievement of a program’s objectives and other  aspects of performance in the context in which the program operates. In  particular, evaluations can be designed to better isolate the causal impact  of programs from other external economic or environmental conditions in  order to assess a program’s effectiveness. Thus, an evaluation study can  provide a valuable supplement to ongoing performance reporting by  measuring results that are too difficult or expensive to assess annually,  explaining the reasons why performance goals were not met, or  assessing whether one approach is more effective than another.
Evaluation can be key in program planning, management, and oversight  by providing feedback on both program design and execution to program  managers, legislative and executive branch policy officials, and the public.  In our 2013 survey of a stratified random sample of federal managers, we  found that most federal managers reported lacking recent evaluations of  their programs. Although only about a third had recent evaluations of their  programs or projects, the majority of those who had evaluations reported  that they contributed to understanding program performance, assessing  program effectiveness or value, making changes to improve program  management or performance, and sharing what works with others.
Those who had evaluations cited most often a lack of resources as a  barrier to implementing evaluation findings. Agency evaluators noted that  it takes a number of studies rather than just one study to influence  change in programs or policies. Experienced evaluators identified three  strategies to facilitate evaluation influence: leadership support of  evaluation, building a strong body of evidence, and engaging  stakeholders throughout the evaluation process.
Our previous literature review found that the key elements of national or  organizational capacity to conduct and use evaluation in decision making  include an enabling environment that has leadership support for using  evidence in decision making; organizational resources to support the  supply and use of credible evaluations; and the robust, transparent  availability of evaluation results. Our 2014 survey of PIOs on the  presence of these elements in their agencies found uneven levels of  evaluation expertise, organizational support within and outside the  organization, and use across the government. About half the 24 agencies  reported committing resources to obtain credible evaluation by  establishing a central office responsible for evaluation, yet those agencies  with centralized leadership reported greater evaluation coverage and use  of the results in decision making. Only six of these agencies reported  having stable funding or agency-wide evaluation plans.
subsections: section_title: A Greater Role for Evidence in Federal Decision Making
paragraphs: GPRAMA established an expectation that evidence would have a greater  role in agency decision making. The act changed agency performance  management roles, planning and review processes, and reporting to  ensure that agencies use performance information in decision making  and are held accountable for achieving results and improving government  performance. The act required the 24 CFO Act agencies and OMB to  establish agency priority goals and government-wide cross-agency  priority goals, review progress on the goals quarterly, and report publicly  on their progress and strategies to improve performance on a government  performance website.
In addition, GPRAMA, along with OMB guidance, established and defined  performance management responsibilities for agency officials in key  management roles. In particular, the PIO was given a central role in  promoting the agencies’ use of evaluation and other evidence to improve  program performance. The act charged the Performance Improvement  Council, which includes PIOs from all 24 CFO Act agencies, to facilitate  agencies’ exchange of successful practices and the development of tips  and tools to strengthen agency performance management.
OMB’s guidance implementing GPRAMA also directed agencies to  conduct strategic reviews of annual progress toward each strategic  objective in their strategic plans to inform agency strategic decision  making, budget formulation, and preparation of annual performance plans  and reports. Guided by the PIO, agencies are to consider a wide range of  evidence (including research, evaluation, and performance indicators) in  these reviews and identify areas where additional evaluations or analyses  of performance data are needed.
Further, GPRAMA is part of a government-wide focus on the crucial role  of evidence for improving the effectiveness of federal programs. Since  2009, OMB has issued several memorandums urging efforts to  strengthen the use of rigorous impact evaluation, designate a high-level  official responsible for evaluation to develop and manage a research  agenda, and demonstrate the use of evidence and evaluation in budget  submissions, strategic plans, and performance plans. A 2013 OMB  memorandum urged agencies to develop an evidence and innovation  agenda to exploit existing administrative data to conduct low-cost  experiments and implement outcome-focused grant designs and research  clearinghouses to catalyze innovation and learning.
OMB staff have also established several interagency workgroups to  promote sharing evaluation expertise and have organized a series of  workshops and interagency collaborations. For example, in 2016 we  recommended that OMB establish a formal means for agencies to  collaborate on tiered evidence grants, a new grant design in which  funding is based on the level of evidence available on the effectiveness of  the grantee’s service delivery model. OMB’s Evidence Team convened  an interagency working group on tiered evidence grants that meets  quarterly and established a website for the group to share resources. This  team also co-chairs the Interagency Council on Evaluation Policy, a group  of 10 agency evaluation offices that have collaborated on developing  common policies and conducting workshops. The Trump Administration’s  2018 Budget proposal endorses a continued commitment to agencies  building a portfolio of evidence on what works and how to improve results,  investing in evidence infrastructure and capacity, and acting on a strong  body of evidence to obtain results.
In 2016, the Congress enacted and the President signed two pieces of  legislation encouraging federal agency evaluation. The Evidence-Based  Policymaking Commission Act of 2016 created the Commission and  charged it with conducting a comprehensive study of the data inventory,  data infrastructure, database security, and statistical protocols related to  federal policy making and the agencies responsible for maintaining that  data. This study was to include a determination of the optimal  arrangement for which administrative data on federal programs and tax  expenditures, survey data, and related statistical data series may be  integrated and made available to facilitate program evaluation, continuous  improvement, policy-relevant research, and cost-benefit analyses, while  considering the privacy of personally identifiable information. In its  September 2017 report, the Commission made 22 recommendations to  improve secure, private, and confidential access by researchers to  government data; modernize data privacy protections; implement a  National Secure Data Service to manage secure record linkage and data  access for evidence building; and strengthen federal agency evidence- building capacity. In particular, the Commission recommended that each  federal department should identify a Chief Evaluation Officer and develop  a multi-year learning agenda of high priority research and policy  questions to address.
The Foreign Aid Transparency and Accountability Act of 2016 (FATAA)  requires the President to set guidelines for monitoring and evaluating  federal foreign assistance by January 2018. The guidelines are to  provide direction to the several federal agencies that administer foreign  assistance on how to, for example, establish annual monitoring and  evaluation plans, quality assurance procedures, and public dissemination  of findings and lessons learned.
section_title: Although Most Federal Managers Lack Current Program Evaluations, They Find That Their Availability Helps Improve Programs
paragraphs: In 2017, we surveyed federal managers asking the same questions as  those we asked in 2013 about managers’ access to evaluation and their  use in decision making. Our 2017 survey found no change government- wide in managers’ access to evaluations since 2013. We estimate that 40  percent of federal managers reported having access to recent evaluations  of their programs, while another 39 percent reported that they did not  know if an evaluation had been conducted. About half the managers who  had evaluations once again reported that they contributed to a great or  very great extent to improving program management or performance and  assessing program effectiveness (54 and 48 percent, respectively), while  fewer reported that they contributed to allocating program resources or  informing the public (35 and 22 percent, respectively).
subsections: section_title: Managers Report Access to Evaluation Unchanged Since 2013
paragraphs: In 2017, an estimated 40 percent of federal managers reported that an  evaluation had been completed within the past 5 years for any of the  programs, operations, or projects they were involved in—statistically  unchanged from the 2013 survey (37 percent). As in 2013, Senior  Executive Service (SES) managers reported having evaluations  statistically significantly more often than non-SES managers did (56  percent versus 39 percent in 2017; 54 percent versus 36 percent in  2013).This should be expected, since SES managers are likely to  oversee a range of programs broader than that of non-SES managers,  any one of whose programs might have been evaluated.
An estimated 18 percent of managers reported not having any  evaluations, while twice as many managers (an estimated 39 percent)  reported that they did not know if an evaluation had been conducted. We  believe this may reflect midlevel managers’ lack of familiarity with  activities outside their programs. As in 2013, non-SES managers reported  twice as often as SES managers that they did not know whether an  evaluation had been performed (40 percent versus 19 percent in 2017; 41  percent versus 24 percent in 2013). And in other questions in our survey  about GPRAMA provisions, non-SES managers reported significantly  more often than SES managers that they were not familiar with cross- agency priority goals (42 versus 22 percent), one or more of their  agency’s priority goals (21 versus 9 percent), or their agency’s quarterly  performance reviews (61 versus 44 percent). Because these goals and  their related reviews apply only to a subset of an agency’s goals, midlevel  managers are less likely to be directly involved in them.
Of the estimated 40 percent of managers who reported having  evaluations, most (86 percent) reported that the agency itself primarily  conducted or contracted for these evaluations. Many of these managers  also reported that studies were completed by their Inspector General (49  percent), GAO (38 percent), or others such as the National Academy of  Sciences and independent boards (17 percent). Because of variation in  the responsibilities of managers, we cannot deduce from these results  how many programs have been evaluated. However, even if additional  evaluations had been conducted by others within or outside the agency, if  managers were unaware of them, their results would not have been  available for use.
section_title: Most Managers Who Had Evaluations Continue to Report That They Helped Them Assess and Improve Programs
paragraphs: Because evaluations are designed to meet decision makers’ information  needs, our survey asked federal managers who had recent evaluations to  what extent those evaluations contributed to 11 different activities. For the  40 percent of managers who reported having evaluations, the results are  very similar to the results of our 2013 survey: federal managers with  evaluations credited them with contributing to a great or very great extent  to assessing program effectiveness or implementing changes to improve  program management or performance (48 and 54 percent, respectively),  with no statistically significant changes since 2013. Managers reported  less frequently that evaluations contributed greatly to allocating program  resources or informing the public (figure 1).
Consistent with the 2013 survey results, many managers who reported  having evaluations reported that they contributed to a great or very great  extent to direct efforts to improve programs such as:  implementing changes to improve program management or  performance (an estimated 54 percent in 2017), developing or revising performance goals (45 percent),  sharing what works or other lessons learned with others (44 percent),  and designing or supporting program reforms (39 percent).
Evaluations vary in their scope and complexity and may address  questions about program implementation as well as program  effectiveness, so any resulting recommendations may point to simple  corrections or broad re-thinking of a policy’s relevance or effectiveness. In  a previous study, evaluators told us that it usually takes a number of  studies, rather than just one, to influence change in programs or policies.  As one evaluator put it, “the process by which evaluation influences  change is iterative, messy, and complex. Policy changes do not occur as  a direct result of an answer to an evaluation question; rather, a body of  evaluation results, research, and other evidence influences policy and  practice over time.” Moreover, designing and approving major program  reforms typically involves a number of stakeholders outside the agency.
Sharing what works with others is often the most direct action federal  managers can take in decentralized programs in which they do not have  direct control of program activities conducted by others at the state and  local levels. To address this, federal agencies use a variety of methods to  disseminate evaluation findings to local decision makers, such as  establishing searchable evaluation clearinghouses online or  disseminating findings through electronic listservs, through webinars, or  at research and evaluation conferences.
Fewer managers reported that evaluations contributed to streamlining  programs to reduce duplicative activities to a great or very great extent  (an estimated 27 percent). We have issued several reports outlining  numerous areas of potential duplication, overlap, and fragmentation in  federal programs. In these reviews, we identified the need for improved  coordination and collaboration as well as better evaluation of these  programs’ performance and results to help inform decisions about how to  better manage these programs. Evaluation studies, if carefully designed,  can address specific questions about the extent of fragmentation, overlap,  and duplication as well as the individual and joint effectiveness of related  programs. A broad review of evidence on related programs and the  relationships among them can clarify the extent of and reveal  opportunities for reducing or better managing fragmentation, overlap, and  duplication.
subsections: section_title: Increasing Understanding of Program Performance
paragraphs: Managers who reported having access to evaluations reported that  evaluations contributed to a great or very great extent to improving their  understanding of program performance, such as by assessing program effectiveness, value, or worth (an estimated 48  increasing understanding about the program or topic (48 percent); and  supplementing or explaining performance results (44 percent).
The primary purpose of program and policy evaluations is to provide  systematic evidence on how well a program is working, whether it is  operating as intended or achieving its intended results. They can be  especially useful for helping improve program performance when they  help identify for whom or under what conditions a program or approach is  effective or ineffective or the reasons for change (or lack of change) in  program performance. We have also reported that evaluations can help  measure more complex or costly forms of performance than can be  obtained routinely, such as by following up on high school students’  success in college.
section_title: Allocating Program Resources
paragraphs: Similar to the 2013 survey results, fewer managers found that evaluations  contributed to a great or very great extent to allocating resources within  the program (35 percent), or supporting program budget requests (33  percent), than to improving program management or understanding (54  and 48 percent, respectively). This result is not surprising because many  factors and priorities influence the budget process and need to be  considered when deciding how to allocate limited resources among  competing needs. Evaluators told us that high-stakes decisions such as  funding are taken rarely on the basis of a single study but, rather, on the  basis of a body of evidence.
Our 2014 survey of the PIOs at the 24 CFO Act agencies provided a  mixed picture of evaluation use in allocating resources. Almost half (10)  reported that their agencies had increased their use of evaluation in  supporting budget requests and allocating resources within programs  since 2010; while 5 PIOs either provided no opinion or reported little or no  agency use of evaluation evidence to support budget or policy changes  as part of their agency’s annual budget process.
section_title: Informing the Public
paragraphs: Similar to the 2013 survey results, less than half the federal managers  who reported having evaluations also reported that evaluations  contributed to informing the public about how programs are performing to  a great or very great extent (an estimated 22 percent). In fact, similar to  2013, 20 percent of these managers reported no basis to judge whether  these evaluations informed the public. As we noted in our 2013 report,  federal managers’ use of evaluation appears to be oriented more  internally than externally, and they may think that they are not in a  position to know whether the public reads their reports. This does not  mean that agencies do not make their evaluation reports public. In our  2014 survey of the 24 PIOs, half reported that their agencies posted  evaluation reports in a searchable database on their websites, and a third  reported disseminating evaluation reports by electronic mailing lists.
section_title: Lack of Resources Hinders Evaluation Use, but Leadership Support and Evaluation Relevance Facilitate It
paragraphs: Simply having program evaluations does not ensure that managers will  use their results in management or policy making. As we noted above,  our reviews of the research and policy literature have found that  organizational and national capacity to conduct and use evaluation in  decision making relies on leadership support for using evidence in  decision making, organizational resources, and the availability of  evaluation results.
In addition, the nature of study results can influence evaluation use;  mixed or inconclusive results may not suggest a clear path of action. To  help understand the relative importance of these factors for evaluation  use, our survey asked federal managers who had recent evaluations of  any of their programs, operations, or projects to what extent specific  factors regarding leadership support, policy context, staff capabilities, or  evaluation characteristics hindered or facilitated using evaluations in their  agencies.
Managers’ views of which factors facilitate or hinder evaluation use have  changed little since our 2013 survey. Managers who reported having  evaluations once again most often reported that lack of resources to  implement results was a barrier to evaluation use (an estimated 29  percent). They most often identified leadership support for evaluation (38  percent), and the evaluation’s relevance to decision makers (36 percent)  as facilitators of evaluation use. While 19 percent perceived lack of staff  knowledgeable in evaluation as a barrier, 35 percent reported that staff  involvement facilitated use. As in our 2013 survey, many agency  managers (35 percent) reported they had no basis to judge the influence  of the presence or absence of congressional support for evaluation. In our  2014 survey, the PIOs generally identified the same factors facilitating  evaluation use.
subsections: section_title: Lack of Resources Was Reported as a Hindrance More Often than Agency Capacity or Study Limitations
paragraphs: Managers who reported having access to recent evaluations of their  programs rated lack of resources to implement evaluation findings more  often than any other potential barrier (see figure 2). They also reported  modest concerns related to program context and agency capacity or  support for evaluation as barriers to evaluation use more often than  potential problems with study quality.
For the estimated 40 percent of managers who reported having  evaluations, the factor that they most often reported hindering the use of  program evaluations to a great or very great extent was a lack of  resources to implement evaluation findings (29 percent), which was also  the most commonly reported factor in 2013 (33 percent, difference not  statistically significant). This is not surprising given today’s constrained  federal budget resources. In a climate of budget reductions, agencies are  hard-pressed to argue for expanding or creating new programs. But  agencies may also lack resources to undertake corrective action within  existing programs, such as providing additional staff training or increasing  oversight or enforcement efforts.
subsections: section_title: Barriers Related to Program or Policy Context
paragraphs: Few federal managers who reported having evaluations cited factors  related to agency and policy context as barriers that hinder the use of  evaluations to a great or very great extent, such as: difficulty resolving differences of opinion among internal or external  stakeholders (an estimated 18 percent), difficulty distinguishing between the results produced by the program  and results caused by other factors (17 percent), and  concern that the evaluation did not address issues of relevance to  decision makers (15 percent).
The wide range of stakeholders for federal programs can include the  Congress, executive branch officials, nonfederal program partners (state  and local agencies and community-based organizations), program  beneficiaries, regulated entities, and the policy research community. Their  perspectives on evaluation results may differ because of differences in  their policy opinions or the complexity of evaluation findings. For  programs with broad goals, stakeholders may differ in their perception of  a program’s purpose and how program “success” should be defined.  Disagreements about what to do next can occur when evaluation findings  are not wholly positive or negative.
Some federal managers who reported having evaluations also reported  that difficulty distinguishing between results produced by the program and  results caused by other factors was a great or very great barrier to  evaluation use (18 percent). Across the federal government, programs  aim to achieve outcomes that they do not control, that are influenced by  other programs or external social, economic, or environmental factors,  complicating the task of assessing program effectiveness. Typically, this  challenge is met by conducting a net impact evaluation that compares  what occurred with an estimate of what would have occurred in the  absence of the program. However, these studies can be difficult to  conduct, may have unexpected or contradictory findings, and need to be  considered in the context of the larger body of evidence.
Some managers (an estimated 15 percent) rated concern about the  relevance of an evaluation’s issues to decision makers as hindering use  to a great or very great extent, but three times as many managers (47  percent) reported that this was a small or insignificant barrier. Our  previous literature review found that collaboration with program  stakeholders in evaluation planning is a widely recognized element of  evaluation capacity. We also described in a previous report how  experienced agency evaluation offices reach out to key program  stakeholders to identify important policy and program management  questions, vet initial ideas with the evaluations’ intended users, and then  scrutinize the proposed portfolio of studies for relevance and feasibility  within available resources. The resulting evaluation agenda aims to  provide timely, credible answers to important policy and program  management questions. This can help ensure that their evaluations will  be used effectively in management and legislative oversight.
More recently, OMB, in the President’s proposed budget for fiscal year  2018, encouraged agencies to expand on this practice by adopting a  “learning agenda” in which they collaboratively identify the critical  questions that, when answered, will help their programs be more  effective. A learning agenda would then identify the most appropriate  tools and methods (for example, research, evaluation, analytics, or  performance measures) to answer each question. OMB noted that the  selected questions should reflect the priorities and needs of a wide array  of stakeholders involved in program and policy decision making:  Administration and agency officials, program offices and program  partners, researchers, and the Congress. As we noted above, in 2017,  the Commission on Evidence-Based Policymaking also recommended  that departments create learning agendas.
section_title: Lack of Agency Capacity or Support for Evaluation
paragraphs: Two infrequently reported barriers related to agency evaluation resources  at both the staff and executive levels, at about the same levels as in  2013, are:  lack of staff knowledgeable about interpreting or analyzing program  evaluation results (an estimated 19 percent rated great or very great  extent), and  lack of ongoing top executive commitment or support for using  program evaluation to make program or funding decisions (17  percent).
In contrast, almost half of agency managers who reported having  evaluations reported that these two issues hindered evaluation use to a  small extent or not at all (an estimated 46 to 47 percent, respectively).  The research literature has clearly established leadership support for  using evidence in decision making as important for evaluation use.  However, it is likely that most managers who have evaluations also have  at least some leadership support for evaluation. Our 2014 survey of 24  PIOs found that the 9 agencies who reported having independent,  centralized evaluation authority reported greater evaluation use in  management and policy making.
Program evaluations–-especially net impact evaluations that attempt to  isolate a program’s effects from the effects of other factors-–typically  employ more complex analytic techniques than performance monitoring,  so their results may be unfamiliar to staff without training in research and  statistics. Evaluation expertise is needed to plan, conduct, or procure  evaluation studies, but program staff also need sufficient knowledge to  understand and translate evaluation results into steps toward program  improvement. Our 2014 survey of 24 PIOs found that about half the  agencies reported increases in hiring staff with research and evaluation  expertise and in training staff in research and analysis skills since 2011,  but 7 acknowledged additional training was needed to a great or very  great extent in data management and statistical analysis, performance  measurement and monitoring, and translating evaluation results into  actionable recommendations.
section_title: Potential Barriers Associated with Study Limitations
paragraphs: In both the 2013 and 2017 surveys, the agency managers with  evaluations agreed that factors related to study limitations were not  serious barriers; approximately half reported that they hindered evaluation  use to a small extent or not at all: difficulty determining how to use evaluation findings to improve the  program (an estimated 50 percent rated a small extent or not at all), difficulty obtaining study results in time to be useful (51 percent),  concern about the credibility (validity or reliability) of study results (55  percent), difficulty generalizing the results to other persons or localities (56 difficulty accepting findings that do not conform to expectations (58  percent).
We have reported that an effective evaluation design aims to provide  credible, timely answers to the intended users’ questions. Even with the  best planning, however, an evaluation might not meet decision makers’  needs. First, the pace of policy making is much quicker than the time it  takes to conduct an evaluation. Second, there is no guarantee that study  results will point to a clear path of action. We previously reported that, to  manage these uncertainties, experienced evaluators recommended  building a strong body of evidence and engaging stakeholders throughout  the process.
A body of evidence—including various forms of evidence—is considered  more valuable than a single study because having multiple studies with  similar results strengthens confidence in the conclusions, and a body of  information can yield answers to a variety of different questions,  whenever stakeholders pose them. Comparing results obtained under  different conditions can help explain what might be driving seemingly  contradictory results. Evaluators pointed out that they rarely based  decisions on a single study.
Individual evaluation studies typically do not simply identify whether a  program works but, rather, they assess the effects of an individual  program or intervention on specific domains for the specific populations or  conditions studied. Developing a body of evidence is also a strategy for  ensuring that information is available for input to fast-breaking policy  discussions.
Engaging stakeholders throughout the evaluation process permits  targeting the evaluation’s questions and timing to decision makers’ needs,  gaining their buy-in to the study’s credibility and relevance, and providing  stakeholders with interim results or lessons learned about program  changes that they can implement right away.
section_title: Managers’ Limited Knowledge of Congressional Support for Evaluation
paragraphs: Few agency managers who reported having evaluations viewed lack of  ongoing congressional commitment or support for using program  evaluation to make program or funding decisions as a barrier to use to a  great or very great extent (an estimated 16 percent). However, twice as  many managers (35 percent) reported they had no basis for determining  whether congressional commitment was a barrier. We found this same  phenomenon in 2013 as well (18 percent and 39 percent, respectively),  most likely reflecting midlevel managers’ lack of direct contact with  congressional members and staff. This is also consistent with responses  to a parallel question included in our survey of federal managers about  congressional commitment or support for using performance information  to make program or funding decisions. About a third of the full sample of  federal managers reported that they had no basis to judge whether lack of  congressional support for using performance information hindered its use.
Congressional committees have a number of opportunities to  communicate their support for evaluation, such as: consulting with  agencies as they revise their strategic plans and agency priority goals  (APG); requesting agency evaluations to address specific questions  about policy or program implementation or results; conducting oversight  hearings on agency performance; and reviewing agency evaluation plans  to ensure that they address issues of congressional interest. While the  Congress holds numerous oversight hearings and requests studies from  GAO, it is not clear whether it regularly requests agencies to conduct  evaluations. In our 2014 survey, fewer than half the PIOs (10) reported  having congressional mandates to evaluate specific programs.
Despite GPRAMA’s requirement that agencies consult with the Congress  in developing their strategic plans and priority goals, we found their  communication to be one-directional, resembling reporting more than  dialogue. In our 2013 interviews with evaluators, one evaluator explained  that, for the most part, they conduct formal briefings for the Congress in a  tense, high-stakes environment; they lack the opportunity for informal  discussion of their results. In 2013 we recommended that OMB ensure  that agencies adhere to OMB’s guidance for website updates to provide a  description of how congressional consultations were incorporated in each  APG.
Our analysis of the sections on the 2016—2017 APGs on  Performance.gov in October 2016 generally found that agencies either  did not include information about congressional input or had not updated  Performance.gov to reflect the most recent round of stakeholder  engagement. As of June 2017, Performance.gov has been archived as  agencies develop updated goals and objectives for release in February  2018 with the President’s next Budget submission to the Congress.
section_title: Leadership Support for Evaluation and Policy Relevance of Evaluation Facilitate Use
paragraphs: To learn what factors facilitate evaluations’ use in decision making, we  added a new question to our survey of federal managers with evaluations  on the extent to which 12 factors facilitate their use (see figure 3). We  selected these factors to parallel factors found in our 2013 survey to  hinder use as well as others that were found to facilitate use in our  previous interviews with evaluators and in our 2014 survey of the PIOs.
In 2017, federal managers who reported having evaluations most  frequently reported that agency leadership support for evaluation, staff  involvement, and evaluation relevance to decision makers facilitated  evaluation use. Although neither the survey respondents nor the survey  questions are directly comparable, the PIOs we surveyed in 2014  reported similar factors as facilitating evaluation use.
These groups differed in their views on the importance of quarterly  performance reviews, possibly reflecting their different responsibilities and  levels of involvement. Both the federal managers and the senior agency  officials reported limited knowledge of congressional requests for or  interest in evaluation.
subsections: section_title: Top Executive Commitment and Policy Relevance
paragraphs: Consistent with the literature on factors supporting evaluation use, about  one-third of agency managers who reported having evaluations rated top  executive commitment or support for using program evaluation to make  program or funding decisions the most often of the factors presented (an  estimated 38 percent to a great or very great extent). About twice as  many managers reported this factor as facilitating evaluation use as those  who rated its absence as hindering evaluation use to a great or very great  extent (17 percent). This may be because, as we noted above, these  respondents have evaluations and thus probably already have some  leadership support for evaluation; lack of leadership support was not  much of a problem for them.
While our 2014 survey did not ask the PIOs to what extent top leadership  support for using evaluations in decision making facilitated its use, many  reported that their agencies’ senior leadership demonstrated commitment  to using evidence (of various types) in management and policy making  through guidance (17 of 22) or internal agency memorandums (12 of 22).  Some PIOs also rated holding goal leaders accountable for progress on  APGs—another form of leadership support—very useful for improving  their agencies’ capacity to use evaluations in decision making (8 of 23  PIOs).
GAO and others have commented that for evaluation results to be acted  on, not only must decision makers generally support using evidence to  inform decisions but also the studies themselves must be seen as  relevant and credible. About one-third of agency managers with  evaluations in 2017 rated importance of an evaluation’s issues to agency  decision makers as facilitating use to a great or very great extent (an  estimated 36 percent). This is about twice as many as the managers who  said the absence of relevance hindered evaluation use to a great or very  great extent (15 percent). We interpret this to mean that the managers  perceived their evaluations as generally addressing relevant issues and  that the evaluations’ relevance contributed to their use in agency decision  making.
Despite managers’ high regard for top management’s support for  evaluation, it is notable that few managers reported that consideration of  evaluation findings in agency quarterly performance reviews facilitated  their use in decision making. GPRAMA introduced these reviews to  encourage the use of performance information in agency decision making  by requiring agencies to review progress on their APGs quarterly and to  report publicly on their progress and strategies to improve performance,  as needed. Although about a quarter of the PIOs reported in 2014 (6 of  23) that these reviews were very useful in improving agencies’ capacity to  use evaluations, the managers surveyed in 2017 were not as sanguine.  About a third of the managers with evaluations reported that they had no  basis to judge whether these reviews facilitated use (35 percent), and few  (14 percent) rated them as facilitating use to a great or very great extent.  It may be that few middle managers participated in these reviews; they  are only required for APGs, a small subset of an agency’s performance  goals (generally 2—8 goals at each agency). Sixty-one percent of the  total sample of managers reported that they were not at all familiar with  these reviews.
Alternatively, evaluations might contribute more effectively to the annual  strategic reviews, which aim for a comprehensive assessment of progress  on the results the agency aims to achieve. OMB’s guidance for these  reviews directs agencies to consider a broad array of evidence and  external influences on their objectives, identify any gaps in their evidence  and areas where additional evaluations or other analyses are needed,  and thus focus their limited evaluation resources to inform the strategic  decisions facing the agency. Our 2017 survey did not ask federal  managers about these strategic reviews; thus, we do not know whether  midlevel managers were aware of or involved in these reviews.
section_title: Staff Involvement
paragraphs: Experienced evaluators have told us that engaging staff throughout the  evaluation process can gain their buy-in on the relevance and credibility  of evaluation findings. In addition, providing program staff with interim  results or lessons learned from early program implementation can ensure  timely data for program decisions. In 2017, one-third of agency managers  with evaluations rated program staff involvement in planning or  conducting evaluation studies as greatly or very greatly facilitating use (an  estimated 35 percent). This is consistent with our 2014 survey, in which  about half the PIOs also rated staff involvement in planning and  conducting evaluation studies as very useful for improving agency  capacity to use evaluations in decision making (11 of 23).
Evaluations may use complex analytic techniques with which program  staff are unfamiliar, thus inhibiting staff’s involvement and their ability to  interpret the findings. However, only an estimated 19 percent of  managers rated lack of staff who are knowledgeable about interpreting or  analyzing program evaluation results as greatly or very greatly hindering  use. A quarter of managers (25 percent) reported that one possible  response—providing program staff and grantees with technical  assistance on evaluation and its use–-facilitated evaluation use to a great  or very great extent. In 2014, about half the surveyed PIOs agreed; 11 of  23 rated this strategy as very useful for improving agency capacity to use  evaluations.
section_title: Ability to Make Recommended Changes
paragraphs: Other factors that managers in the 2017 survey rated often as facilitating  use were parallel to factors that they rated often as barriers. About a  quarter of managers (an estimated 29 percent) reported that agency staff  ability to make recommended program changes facilitated use to a great  to very great extent. This factor is parallel to the most frequently rated  factor to hinder use—lack of resources to implement the evaluation  findings—that a similar number identified (29 percent great to very great  extent). As we noted above, midlevel managers may not have the  authority or resources to implement a study’s recommendations.
In addition, the positive characteristics of a study may influence its use.  About a third of agency managers who reported having evaluations  reported that clear implications of results for improving program design or  management (31 percent) facilitated use to a great to very great extent.  The absence of such clarity is one of the factors that an evaluator  previously told us could lead to disagreements, and such disagreements  may lead to inaction. Mixed results or the absence of a clear explanation  for disappointing program results can impede consensus on an  evaluation’s lessons for program improvement. A strong evaluation  design can help prevent message muddling by testing alternative  explanations, but it cannot ensure that an evaluation will provide clear  implications because the results of an evaluation, like a research study,  are inherently uncertain.
section_title: Agency Evaluation Policies
paragraphs: Written evaluation policies and standards help provide benchmarks for  ensuring the quality of an organization’s processes and products. The  American Evaluation Association (AEA) publishes a guide for developing  and implementing U.S. government evaluation programs that  recommends that agencies, among other things, develop written  evaluation policies and quality standards, consult with program  stakeholders, and prepare annual and long-term evaluation plans to  support future decision making. In our 2014 survey of PIOs, about a  quarter of the 24 PIOs surveyed reported that their agencies had written  agency-wide policies or guidance for key issues contained in that guide:  selecting and prioritizing evaluation topics,  consulting program staff and subject matter experts, ensuring internal and external evaluator independence and objectivity,  selecting evaluation approaches and methods, ensuring completeness and transparency of evaluation reports,  timely public dissemination of evaluation findings and  recommendations, or  tracking implementation of evaluation findings.
A few more PIOs (10 of 24) reported having agency-wide policies on  ensuring the quality of data collection and analysis.
In our 2017 survey, we estimate that 28 percent of managers who  reported having evaluations reported that agency policies and procedures  to ensure evaluation quality facilitated use to a great or very great extent.  Our survey did not ask which types of policies they had, so we do not  know whether they included all of the topics listed above.
Only a small number of managers—13 percent—reported having no basis  to judge their policies’ influence, suggesting that most agencies have  evaluation policies, although those policies may not apply agency-wide.  The reported positive influence of such policies on evaluation quality is  also consistent with the fact that about half the managers with evaluations  reported that various factors regarding study limitations did not  significantly hinder evaluation use in decision making, as discussed  above.
Experienced evaluators consult with stakeholders in developing their  evaluation or learning agenda to help ensure their evaluations’ credibility  and relevance to current management and policy issues. In the 2017  survey, managers with evaluations rated consultation with stakeholders  on the agency’s evaluation agenda high for facilitating evaluation use (28  percent to a great or very great extent), although 22 percent responded  they had no basis to judge. In our 2014 survey of PIOs, only 7 reported  having an agency-wide evaluation agenda.
section_title: Limited Congressional Requests for Evaluation
paragraphs: The Congress is a prominent member of federal program stakeholders  but congressional interest in and requests for evaluation were not widely  reported by the PIOs we surveyed in 2014. Congressional mandates are  requirements in statute for an agency (including GAO) to conduct a study,  usually specifying the topic and a reporting date. GAO is often requested  to report on the progress and success of new programs or program  provisions. In our 2014 survey, fewer than half the PIOs (10 of 23)  reported that they had any congressional mandate to evaluate a specific  program in their agency.
Consistent with this low reporting of congressional requests for  evaluation, about one-third of managers who reported having evaluations  in our 2017 survey reported that they had no basis to judge whether  congressional requests or mandates facilitated evaluation use (31  percent). However, 23 percent reported that such requests facilitated use  to a great or very great extent. Thus, while congressional evaluation  requests are not widely reported among PIOs, they appear to be  influential among some federal managers.
section_title: Conclusions
paragraphs: For several years, OMB has encouraged agencies to use program  evaluations and other forms of evidence to learn what works and what  does not, and how to improve results. Yet, agencies appear not to have  expanded their capacity to conduct or use evaluation in decision making  since 2013. Because the majority of agency managers who reported  having evaluations also reported that they contributed to improving  program performance (54 percent), this lack of evaluation capacity  constitutes a lost opportunity to improve the efficiency and effectiveness  of limited government resources.
The survey results reinforce lessons from our previous reports: involving  agency staff and executives in planning and conducting evaluations helps  ensure that those evaluations are relevant, credible, and used in agency  decision making. Agency managers who reported having evaluations also  reported top executive support for using evaluations to make decisions,  the importance of the evaluation’s issues to decision makers, and  involving agency staff in planning or conducting evaluation studies, most  often among factors facilitating evaluation use.
GAO, as well as OMB, AEA, and the Commission on Evidence-Based  Policymaking, has noted that it is important to develop an evaluation plan  or agenda to ensure that even an agency’s scarce research and  evaluation resources are targeted to its most important issues and can  shape budget and policy priorities and management practices. Although  only some agencies have developed agency-wide evaluation agendas,  evaluators who have them have found that consulting with stakeholders  on their evaluation agendas helps ensure evaluation credibility and  relevance, and facilitates the use of evaluation results.
Congressional support—through either authorization or appropriation of  funds—is often needed for agencies to implement desired program  reforms. Although 28 percent of federal managers with evaluations  reported that consulting with external stakeholders on their evaluation  agendas greatly contributes to their use, we saw limited knowledge of  congressional consultation. Congressional consultation on agency  evaluation plans could increase the studies’ credibility and relevance for  those audiences.
Although evaluations were generally not reported as contributing greatly  to quarterly performance reviews of progress on agency priority goals,  they might contribute more effectively to an agency’s annual strategic  review. OMB’s guidance envisions strategic reviews as a more  comprehensive assessment of a broad range of evidence on and factors  influencing progress on an agency’s desired results. Agencies are also  directed to identify any gaps in their evidence and take steps to address  them in these reviews; thus, the strategic review could produce an  evaluation agenda that is targeted to the agency’s management, budget,  and policy priorities.
section_title: Recommendation
paragraphs: To help ensure that federal agencies obtain the evidence needed to  address the most important questions to improve program implementation  and performance, we recommend that the Director of the Office of  Management and Budget direct each of the 24 Chief Financial Officer Act  agencies to prepare an annual agency-wide evaluation plan that  describes the  key questions for each significant evaluation study that the agency  plans to begin in the next fiscal year, and  congressional committees; federal, state and local program partners;  researchers; and other stakeholders that were consulted in preparing  their plan. (Recommendation 1)
section_title: Agency Comments
paragraphs: We requested comments on a draft of this report from the Director of the  Office of Management and Budget. In an email response, an OMB staff  member commented that it would be more appropriate and effective to  encourage agencies to create an annual evaluation plan, rather than  require or direct them to do so. Because OMB has encouraged agencies  to conduct and use evaluations in decision making for several years with  mixed success, we believe that a more directive approach is needed.
We are sending copies of this report to the Director of the Office of  Management and Budget, and to appropriate congressional committees.  This report is also available at no cost on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2700 or kingsburyn@gao.gov. Contact points for our  Office of Congressional Relations and Office of Public Affairs may be  found on the last page of the report. Staff who made key contributions to  the report are listed in appendix II.
section_title: Appendix I: Methodology for Federal Managers Survey
paragraphs: We administered a web-based questionnaire on organizational  performance and management issues to a stratified random sample of  4,395 from a population of approximately 153,779 mid-level and upper- level civilian managers and supervisors working in the 24 executive  branch agencies covered by the Chief Financial Officers Act of 1990  (CFO Act), as amended. The sample was drawn from the Office of  Personnel Management’s (OPM) Enterprise Human Resources  Integration database as of September 2015, using file designators for  performance of managerial and supervisory functions.
The sample was stratified by agency and by whether the manager or  supervisor was a member of the Senior Executive Service (SES). The  management levels covered general schedule (GS) or equivalent  schedules in other pay plans at levels comparable to GS-13 through GS- 15 and career SES or equivalent. In reporting the questionnaire data, we  use “government-wide” or “across the federal government” to refer to  these 24 CFO Act executive branch agencies, and “federal managers”  and “managers” to refer to both managers and supervisors.
We designed the questionnaire to obtain the observations and  perceptions of respondents on various aspects of such results-oriented  management topics as the presence and use of performance information,  agency climate, and program evaluation use. In addition, to address the  implementation of GPRA Modernization Act of 2010 (GPRAMA), the  questionnaire included a section requesting respondents’ views on its  various provisions including cross-agency priority goals, agency priority  goals, and quarterly performance reviews.
This survey is similar to surveys we have conducted five times previously  at the 24 CFO Act agencies—in 1997, 2000, 2003, 2007, and 2013. The  questions on GPRAMA provisions and program evaluation use were new  in 2013. The 2017 questionnaire includes new questions on the use of  performance information and factors that facilitate the use of program  evaluation. Several components of the new evaluation question were  drawn from our 2014 survey of Performance Improvement Officers (PIOs)  on their agencies’ evaluation capacity resources and activities, discussed  below, and interviews with agency officials. Before administering the  survey, GAO subject matter experts, survey specialists, and a research  methodologist reviewed new questions. We also conducted pretests of  the new questions with federal managers in several of the 24 CFO Act  agencies and based revisions on the feedback we received.
The objectives of this report address whether agency managers reported  change in their access to and use of program evaluations since 2013 and  their views about factors that facilitate or hinder the use of program  evaluation. Therefore, this report analyzes results on a subset of survey  questions concerning those topics. It then compares these results, when  appropriate, to results previously obtained in the 2013 survey of federal  managers, as well as the results of our 2014 PIO survey.
For the 2014 PIO survey, we administered a web-based questionnaire to  the PIOs or their deputies at the 24 CFO Act agencies about agencies’  evaluation resources, policies, and activities and the activities and  resources they found useful in building their evaluation capacity. GAO  subject matter experts, a survey specialist, and research methodologist  also reviewed this survey’s questions. In addition we pretested the  questionnaire in person with PIOs at three federal agencies.
Because this was not a sample survey, it has no sampling errors but may  be subject to nonsampling errors that stem from differences in how a  question is interpreted. The survey of PIOs is not directly comparable to  the survey of federal managers because the questions about factors  influencing evaluation use are not exactly the same, and the PIOs, as  senior officials typically reporting to the agency Chief Operating Officer,  have very different responsibilities from the population of midlevel and  upper-level managers and supervisors responding to the Federal  Managers Survey.
Most of the items on the 2017 Federal Managers Survey were closed- ended, meaning that depending on the particular item, respondents could  choose one or more response categories or rate the strength of their  perception on a 5-point “extent” scale ranging from “to no extent” at the  low end of the scale to “to a very great extent” at the high end. On most  items, respondents also had an option of choosing the response category  “no basis to judge/not applicable.” A few items gave respondents “yes,”  “no,” or “do not know” options.
To administer the survey, we sent an e-mail to managers in the sample  that notified them of the survey’s availability on the GAO website and  included instructions on how to access and complete the survey.  Managers in the sample who did not respond to the initial notice received  multiple e-mail reminders and follow-up phone calls asking them to  participate in the survey. We administered the survey to all 24 CFO Act  agencies from November 2016 through March 2017. For additional details  on the survey methodology, see our report summarizing our body of work  on GPRAMA’s implementation.
From the 4,395 managers selected for the 2017 survey, we found that  388 of the sampled managers had left the agency, were on detail, or had  some other reason that excluded them from the population of interest. We  received usable questionnaires from 2,726 sample respondents. The  response rate across the 24 CFO Act agencies ranged from 36 percent to  82 percent, with a weighted response rate of 67 percent for the entire  sample. An estimated 40 percent of respondents reported that an  evaluation had been completed within the past 5 years for any of the  programs, operations, or projects with which they had been involved.
The overall survey results can be generalized government-wide to the  population of managers as described above at each of the 24 CFO Act  agencies. The responses of each eligible sample member who provided a  useable questionnaire were weighted in the analysis to account  statistically for all members of the population. All results are subject to  some uncertainty or sampling error as well as nonsampling error. The  government-wide percentage estimates based on our sample from 2017  presented in this report have 95 percent confidence intervals within plus  or minus 4 percentage points of the estimate itself for the initial question  about whether an evaluation had been completed and within plus or  minus 7 percentage points for subsequent questions posed to those who  reported having evaluations. Online supplemental materials show all the  questions asked on the survey along with the percentage estimates and  associated 95 percent confidence intervals for each question for each  agency and government-wide.
section_title: Appendix II: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Stephanie Shipman (Assistant  Director), Valerie Caracelli (Analyst in Charge), Pille Anvelt, Timothy  Guinane, Jill Lacey, Benjamin Licht, Krista Loose, Anna Maria Ortiz,  Penny Pickett, and Steven Putansu made key contributions to this report.
section_title: Related GAO Products
paragraphs: Managing for Results: Further Progress Made in Implementing the GPRA  Modernization Act, but Additional Actions Needed to Address Pressing  Governance Challenges. GAO-17-775. Washington, D.C.: September 29,  2017.
Supplemental Material for GAO-17-775: 2017 Survey of Federal  Managers on Organizational Performance and Management Issues.  GAO-17-776SP. Washington, D.C.: September 29, 2017.  2017 Annual Report: Additional Opportunities to Reduce Fragmentation,  Overlap, and Duplication and Achieve Other Financial Benefits.  GAO-17-491SP. Washington, D.C.: April 26, 2017.
Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early  Implementation and Inform Future Federal Efforts. GAO-16-818.  Washington, D.C.: September 21, 2016.
Fragmentation, Overlap, and Duplication: An Evaluation and  Management Guide. GAO-15-49SP. Washington, D.C.: April 14, 2015.
Program Evaluation: Some Agencies Reported that Networking, Hiring,  and Involving Program Staff Help Build Capacity. GAO-15-25.  Washington, D.C.: November 13, 2014.
Managing for Results: Executive Branch Should More Fully Implement  the GPRA Modernization Act to Address Pressing Governance  Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.
Managing for Results: 2013 Federal Managers Survey on Organizational  Performance and Management Issues. GAO-13-519SP. Washington,  D.C.: June 2013.
Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation  in Program Management and Policy Making. GAO-13-570. Washington,  D.C.: June 26, 2013.
Managing for Results: Agencies Should More Fully Develop Priority Goals  under the GPRA Modernization Act. GAO-13-174. Washington, D.C.:  April 19, 2013.
Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.:  January 2012.
Program Evaluation: Experienced Agencies Follow a Similar Model for  Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.
Government Performance: Lessons Learned for the Next Administration  on Using Performance Information to Improve Results. GAO-08-1026T.  Washington, D.C.: July 24, 2008.
Program Evaluation: Studies Helped Agencies Measure or Explain  Program Performance. GAO/GGD-00-204. Washington, D.C.: September  29, 2000.