Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity
Released: 2014-11-13T00:00:00

Letter

Background

Program evaluations are systematic studies that use research methods to address specific questions about program performance. Evaluation is closely related to performance measurement and reporting. Whereas performance measurement entails the ongoing monitoring and reporting of program progress toward preestablished goals; program evaluation typically assesses the achievement of a program’s objectives and other aspects of performance in the context in which the program operates. In particular, evaluations can be designed to isolate the causal impacts of programs from other external economic or environmental conditions in order to assess a program’s effectiveness. Thus, an evaluation study can provide a valuable supplement to ongoing performance reporting by measuring results that are too difficult or expensive to assess annually, explaining the reasons why performance goals were not met, or assessing whether one approach is more effective than another.

Evaluation can play a key role in program planning, management, and oversight by providing feedback on both program design and execution to program managers, legislative and executive branch policy officials, and the public. In our 2013 survey of federal managers, we found that while only about a third had recent evaluations of their programs or projects, the majority of those who had evaluations reported that they contributed to understanding program performance, sharing what works with others, and making changes to improve program management or performance.

GPRAMA Established an Expectation That Evidence Would Have a Greater Role in Agency Decision Making

GPRAMA made changes to agency performance management roles, planning and review processes, and reporting intended to ensure that agencies used performance information in decision making and were held accountable for achieving results and improving government performance. The act required the 24 CFO Act agencies and OMB to establish agency and governmentwide cross-agency priority goals, review progress on those goals quarterly, and report publicly on their progress and strategies to improve performance, as needed, on a governmentwide performance website. It also encouraged a more detailed and comprehensive understanding of those strategies by requiring agencies to identify and coordinate the program activities, organizations, regulations, policies, and other activities—both internal and external—that contribute to each agency priority goal. GPRAMA, along with related OMB guidance, established and defined performance management responsibilities for agency officials in key management roles: the Chief Operating Officer (COO), the PIO, and a goal leader responsible for coordinating efforts to achieve the cross-agency and agency priority goals.

The PIO role was created in 2007 by executive order. GPRAMA established the role in law and specified that it be given to a “senior executive” at each agency who reports directly to the agency’s COO or to its deputy agency head. The PIO is to advise the head of the agency and the COO on goal setting, measurement, and reviewing progress on the agency priority goals. OMB guidance gave PIOs a central role in promoting agency use of evaluation and other evidence to improve program performance, describing their roles as “. . . driving performance improvement efforts across the organization, by using goal-setting, measurement, analysis, evaluation and other research, data-driven performance reviews on progress, cross-agency collaboration, and personnel performance appraisals aligned with organizational priorities.” “Help components, program office leaders and goal leaders to identify and promote adoption of effective practices to improve outcomes, responsiveness and efficiency, by supporting them in . . . securing evaluations and other research as needed . . . and creating a network for learning and knowledge sharing about successful outcome- focused, data-driven performance improvement methods across all levels of the organization and with delivery partners.”

The act also charged the Performance Improvement Council (PIC), the Office of Personnel Management (OPM), and OMB with responsibilities to improve agency performance management capacity. The PIC is an interagency council that was created by executive order, but GPRAMA established it in law and specified that it would be chaired by the OMB Deputy Director for Management and that membership would include the PIOs from all 24 CFO Act agencies, as well as any others. The PIC’s duties include facilitating agencies’ exchange of successful practices and the development of tips and tools to strengthen agency performance management, and assisting OMB in implementing certain GPRAMA requirements. The PIC holds “principals only” and broader meetings open to other agency staff, has formed several working groups that focus on issues relating to implementing GPRAMA and related guidance, and provides a networking forum for staff from different agencies who are working on similar issues. In 2012 through 2014, OMB and the PIC supported several interagency forums on evaluation and evidence that were open to all federal agency staff.

The act charged OPM with (1) identifying key skills and competencies needed by federal employees for developing goals, evaluating programs, and analyzing and using performance information for improving governmental efficiency and effectiveness; (2) incorporating those skills and competencies into relevant position classifications; and (3) working with agencies to incorporate these skills and competencies into agency training. OPM identified core competencies for performance management staff, PIOs, and goal leaders and published them in a January 2012 memorandum. OPM identified relevant existing position classifications that are related to the competencies and worked with the PIC Capacity Building working group to develop related guidance and tools for agencies. In December 2012, the PIC released a draft Performance Analyst position design, recruitment, and selection toolkit. OPM worked with the Chief Learning Officers Council and the PIC Capacity Building working group to develop a website—the Training and Development Policy wiki—that lists some resources for personnel performance management and implementing GPRAMA. OPM is currently conducting pilot studies through 2015, in collaboration with the Chief Human Capital Officers Council, of how to build staff capacity in several competencies identified as mission critical across government, including data analysis. OPM officials also noted that they make databases, such as the Federal Employee Viewpoint Survey, available to agencies for their staff to use in program evaluations.

OMB Efforts to Improve Agency Evaluation Capacity

OMB has taken several steps to help agencies develop evaluation capacity by issuing guidance, promoting the exchange of evaluation expertise through the PIC, and working selectively with certain agencies. Since 2009, OMB has issued several memorandums urging efforts to strengthen the use of rigorous impact evaluation, and demonstrate the use of evidence and evaluation in budget submissions, strategic plans, and performance plans. In May 2012, OMB encouraged agencies to designate a high-level official responsible for evaluation who could develop and manage the agency’s research agenda and provide independent input to agency policymakers on resource allocation and to program leaders on program management. In July 2013, the Directors of OMB, the Domestic Policy Council, the Office of Science and Technology Policy, and the Chairman of the Council of Economic Advisers, jointly issued a memorandum encouraging agencies to adopt an “evidence and innovation agenda”: applying existing evidence on what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery. In particular, the memorandum encouraged agencies to exploit existing administrative data to conduct low-cost experiments, and implement outcome-focused grant designs and research clearinghouses to catalyze innovation and learning.

OMB staff established an interagency group to promote sharing of evaluation expertise, and organized a series of workshops and interagency collaborations. The workshops addressed issues such as potential procedural barriers to evaluation (e.g., the Paperwork Reduction Act information collection reviews) and promising practices for collecting evidence (e.g., developing a common evidence framework). OMB staff facilitated the collaboration of staff from the Department of Education and the National Science Foundation in developing common standards of evidence for reviewing research proposals, and another group of agencies in developing a common framework of standards for reviewing completed evaluations.

Federal Agencies’ Capacity to Conduct and Use Program Evaluation Is Uneven

Studies of organization or government evaluation capacity have found that it requires analytic expertise and access to credible data as well as organizational support both within and outside the organization to ensure that credible, relevant evaluations are produced and used. Our survey found levels of evaluation expertise, support, and use uneven across the government. For example, 7 of the 24 agencies have central leaders responsible for evaluation; in contrast, 7 agencies reported having no recent evaluations for any of their performance goals.

An Agency’s Evaluation Capacity Depends on Both Policy Makers’ Requests for Information and the Agency’s Ability to Produce Credible, Relevant Information

To address our first objective and guide our assessment of agency evaluation capacity, we reviewed the research and policy literature on evaluation capacity, including assessments of agencies in Canada and the United Kingdom, and guidance from the American Evaluation Association (AEA) and the United Nations Evaluation Group. While the details vary, these frameworks commonly emphasize three general categories of elements of organizational, especially national, evaluation capacity:

An enabling environment supporting the use of evidence in management and policymaking: credible information and statistical systems, legislation or policies to institutionalize monitoring and evaluation, public interest in evidence of government performance, and senior leadership commitment to transparency, accountability, and managing for results.

Organizational resources to support the supply and use of credible evaluations: a senior evaluation leader; an evaluation office with clearly defined roles and responsibilities, a stable source of funding, and independence; an evaluation agenda, policies and tools to ensure study credibility and utility; staff expertise and access to experts; and collaboration with program managers and stakeholders.

Evaluation results and use: evaluation quality and credibility; coverage of the agency’s key programs or goals; transparent reporting and public dissemination of reports; recommendation follow-up; and the use of evaluation results in program management, policy making, and budgeting.

Our Survey Respondents

To learn about federal agencies’ evaluation capacity, we surveyed the PIOs or their deputies at the 24 CFO Act agencies because of the central role GPRAMA and OMB assigned them to promote agency performance assessment and improvement efforts. Our 2012 survey of PIOs found that they held senior leadership positions and that most of them were involved in the central aspects of agency performance management to a large extent. Although the PIO position was created in 2007, only one of the initial PIOs continued to hold this position at the time of our 2014 survey. Half had started serving in this position within the past 2 years. Many of our survey respondents held key senior leadership positions in their agencies: 8 PIOs served as the agency’s Chief Financial Officer, another 4 as Assistant Secretary or Deputy for Administration or Management. Seventeen reported to their agency’s COO, 2 to the agency’s administrator or commissioner, and 3 to the agency’s CFO. In order to report on the policies and practices of offices throughout these agencies, we encouraged the PIOs to consult with others when completing the survey and several indicated that they did so.

Most Federal Agencies Reported Access to and Commitment to Using Evidence but Fewer Reported Congressional Requests for Program Evaluation

GPRA represents a central component of the enabling environment for U.S. government evaluation capacity by providing, for over 20 years, a statutory framework for performance management and accountability across the government. Accordingly, most PIOs reported that their senior leadership demonstrated a commitment to using evidence in management and policy making through agency guidance (17), internal agency memorandums (12), congressional hearings (9), and speeches (8). Other avenues offered in comments included budget justifications (10) and town hall meetings or videos for agency managers and staff (2).

Moreover, as we have noted previously, GPRA has produced a solid foundation of generating and reporting performance information. Three- quarters of the agencies (18) said that reliable performance data are available on outcomes for all their priority goals, 3 more said data are available for more than half their priority goals. (One of the independent agencies was exempt from developing priority goals.)

However, our survey respondents indicated that congressional interest in and requests for program evaluation are not widespread. Although the federal government has long invested in evaluation, about half the agencies (13) reported having explicit agency-wide authority to use appropriated funds for evaluation. Some pointed to specific legislative authorities, while one PIO commented, “Evaluation is considered inherent to responsible management and programs use appropriated fund for this purpose.” Less than half the agencies (10) indicated that they had congressional mandates to evaluate specific programs. However, one- third (7) indicated that they had neither explicit agency-wide authority nor a program-specific requirement to conduct evaluations. The importance of this is that in a prior study agency evaluators told us that not having explicit evaluation authority represented a barrier to the use of program funds for evaluation.

Half of Federal Agencies Report Committing Resources to Obtain Credible, Relevant Program Evaluations

Our survey asked the PIOs about the agency resources and policies committed to obtaining credible, relevant evaluations. Their responses indicated uneven levels of development across the agencies. About half the agencies (11) reported committing resources to obtain evaluations by establishing a central office responsible for evaluating agency programs, operations, or projects. However, less than a third of agencies have an evaluation plan or agency-wide policies or guidance for ensuring study credibility.

Central Evaluation Leadership and Resources

About one-third of the agencies (7) reported having assigned responsibility to a single high-level official to oversee their evaluation studies. Although agencies do not need a central evaluation leader in order to conduct credible evaluations, establishing such a position with clear responsibilities sends a message about the importance of evaluation to agency managers. Almost all these individuals (6) were responsible for setting these agencies’ evaluation agendas but only half (3) were responsible for following-up evaluation recommendations. Similar numbers of departments and independent agencies reported having such officials with titles such as Chief Evaluation Officer, Chief Strategic Officer, and Assistant Secretary.

According to AEA guidance, a central evaluation office can promote an agency’s evaluation capacity and provide a stable organizational framework for planning, conducting, or procuring evaluation studies. All the agencies with a single official responsible for overseeing evaluations also reported having a central office responsible for evaluating agency programs, operations, or projects, but only about half the agencies in total (11) had a central office. The central offices could have other responsibilities as well, such as strategic planning. Most of these offices were said to be independent of program offices in making decisions about evaluation design, conduct, and reporting and to have access to analytic expertise through external experts or contractors, but about half were reported to have a stable source of funding (6). Funding generally came through regular appropriations, although two agencies reported having evaluation set-asides—that is, the ability to tap a percentage of operating divisions’ appropriations for evaluation. A larger proportion of independent agencies (5 of 9) than departments (6 of 15) reported having central offices.

As discussed earlier, having analytic expertise is a critical element of evaluation capacity. Most agencies with a central office responsible for evaluations (7—8 of 11) reported that the evaluation staff had training and experience to a great or very great extent in each of the following areas: research design and methods, data management and statistical analysis, performance measurement and monitoring, and translating evaluation results into actionable recommendations. Slightly fewer reported that central evaluation office staff had great or very great subject matter expertise (5). Three survey respondents also volunteered that their staff had additional expertise, including economic analysis, geographical information systems and Lean cost reduction analysis.

Few Agencies Have Policies to Ensure Credible and Relevant Evaluations

Organizations, whether government agencies or professional societies, develop written policies or standards in order to provide benchmarks for ensuring the quality of their processes and products. AEA has published guides for the individual evaluator’s practice and for developing and implementing U.S. government evaluation programs. About one-quarter of agencies reported having agency-wide written policies or guidance for key issues addressed in those guides: ensuring internal or external evaluator independence and objectivity; ensuring completeness and transparency of evaluation reports; selecting and prioritizing evaluation topics; consulting program staff and subject matter experts; selecting evaluation approaches and methods; timely, public dissemination of evaluation findings and recommendations; or tracking implementation of evaluation findings.

A few more agencies, but less than half, reported having policies on ensuring quality of data collection and analysis, which could apply to research as well as program evaluation. Central evaluation leadership was not required to adopt evaluation policies; as only about half of the agencies with agency-wide evaluation policies had a central evaluation office. Agencies provided us with examples of guidance on information quality or scientific integrity as well as program evaluations specifically.

We, along with OMB and AEA, have all noted that developing an evaluation agenda is important for ensuring that an agency’s often scarce research and evaluation resources are targeted to the most important issues and can shape budget and policy priorities and management practices. Less than a third of the agencies (7) reported having an agency-wide evaluation plan. Most such plans were reported to cover multiple years and programs across all major agency components. Senior agency officials and program managers were said to have been consulted in developing all these plans, but few agencies reported consulting congressional stakeholders or researchers. All but 1 of the 7 agencies that had a plan also had a central evaluation office.

Because we found in a previous report that stakeholder involvement facilitates the use of evaluation studies, we asked whether stakeholders were consulted in designing and conducting evaluation studies, either formally or informally. Almost all the PIOs reported consulting senior agency officials (20) and program managers (21) and three-quarters consulted researchers, but few (5) reported consulting congressional staff, less than local program providers or regulated entities.

Agency Components’ Resources and Policies

Agency evaluation offices are located at different organizational levels, which we have previously found affects the scope of their program and analytic responsibilities as well as the range of issues they consider. In a previous study, we found that evaluators in central research and evaluation offices described having a broader and more flexible choice of topics than did evaluators in program offices. In our 2014 survey, half the federal agencies (12) reported that some agency components (such as an administration or bureau) had a central office responsible for evaluation and that the number of such components ranged from 1 to 12 within a department or independent agency. These offices generally existed in addition to, rather than instead of an agency-wide office responsible for evaluation; as a result, 10 agencies had neither type of office. As might be expected, component offices were less likely than central offices to be considered independent of program offices (6 of 12 agencies reported that all or many of their offices had independence in decision making), but 10 of 12 reported that all or many of these offices had access to external experts, and, like the central offices, few reported having a stable source of funds.

About half the agencies with component central offices for evaluation reported that the evaluation staff had training and experience to a great or very great extent in research design and methods, data management and statistical analysis, performance measurement and monitoring, and translating evaluation results into actionable recommendations. These were slightly lower than the ratings for the central office staff’s training. As one might expect, staff were characterized as having great to very great subject matter expertise more often in component offices (9 of 12) than in central evaluation offices (5 of 11).

Only a few PIOs (2 to 4) reported that many or all component central offices for evaluation had written evaluation policies or guidance for any of the issues we listed. More often PIOs (2 to 6) reported not knowing if they had those specific policies.

Agencies Report Moderate Use of Evaluations in Managing Programs, Setting Policy, or Allocating Resources

To assess the results or outcomes of agency evaluation activity, our survey asked the PIOs about the characteristics of the evaluations they produced and their use in decision making. In line with the level of resources they committed to evaluation, the availability and use of program evaluations were uneven across the 24 federal agencies. Even though agencies may not have many evaluations, more than a third report using them from a moderate to a very great extent to support several aspects of program management and policy making.

Because agencies use the term “program” in different ways, we chose to assess agencies’ evaluation coverage of key programs and missions by the proportion of performance goals for which evaluations had been completed in the past 5 years or were in progress. The number of performance goals may vary across agencies but, per OMB guidance, they are supposed to be specific, near-term, realistic targets that an agency seeks to influence to advance its mission, and publicly reports. Only four agencies reported full evaluation coverage of their performance goals. Two-thirds of the agencies reported evaluation coverage of less than half their performance goals; including 7 that reported having evaluations for none of their performance goals. Evaluation coverage was greater in agencies that established centralized authority for evaluation. Three of the 4 agencies with full coverage of their performance goals had both a central evaluation leader and central evaluation office, while all 7 agencies with no coverage had neither. Interestingly, 2 of the 7 agencies that reported having no evaluations of their performance goals did report having component evaluation offices, so they might have had some evaluations that simply did not address topics considered key to advancing their mission.

GAO guidance notes that strong evaluations rely on sufficient and appropriate evidence; document their assumptions, procedures, and modes of analysis; and rule out competing explanations. Thus, transparent reporting of data sources and analyses are critical for ensuring that evaluations are considered credible and trustworthy. About half the PIOs (10) reported that their evaluation reports are transparent to a great or very great extent in describing the data sources used and the analyses performed forming the basis of conclusions; another 7 indicated that they did not know or did not respond to the question.

According to the evaluation capacity literature, timely, public dissemination of evaluation findings is important to support government accountability for results to the legislature and the public and to ensure that findings are available to inform decision making. Half the agencies (11) reported publicly disseminating their evaluation results by posting reports to a searchable database on their websites; fewer reported presenting findings at professional conferences (9), sending a notice and link to the report through electronic mailing lists (7), or conducting webinars on findings for the policy community (6). A couple of the PIOs commented that they post some, but not all, reports on the agency website. Of the 11 agencies posting evaluation reports to a website, half reported that they did so within 3 months of completion, although 1 indicated it can take from 6 months to a year. In addition, a few agencies sponsor research clearinghouses that review evaluations of social interventions and provide the results in searchable databases on their websites to help managers and policy makers identify and adopt effective practices.

Responding to or Using Evaluation Findings

If program evaluations or any form of performance information are to lead to performance improvement, they must be acted on. Seven agencies reported that they had procedures for obtaining management’s response to evaluation recommendations, 8 for obtaining follow-up action on those recommendations. In their comments, a few PIOs noted that they had policies for responding to reports or recommendations from GAO or the Inspector General. Another PIO reported that a number of internal briefings are held to ensure management awareness of evaluation findings as well as a cross-agency research utilization committee composed of staff from program, public affairs, and congressional and intergovernmental relations offices that decides on the appropriate level of publicity effort for the report.

Over a third of the agencies (9 to 10) reported that evaluations were used to a moderate or greater extent to support policy changes, budget changes, or internal proposals for change in resource allocation or management, or to award competitive grants (figure 1). Five agencies reported using evaluation to support all these activities to a moderate or greater extent on average. In comments, PIOs described a variety of ways in which evaluation evidence could be used in awarding competitive grants: reviewing the merit of research proposals, evaluating grantee prior performance and outcomes, assessing credit worthiness, and allocating tiered evidence-based funding, which varies the level of funding based on the extent and quality of the evaluation evidence supporting a program’s effectiveness.

Agencies with centralized evaluation authority, independence, and expertise reported greater evaluation use in management and policy making, demonstrating its importance. More than half of the 7 agencies that reported great use of evaluation had a senior evaluation leader or a central evaluation office. Moreover, the agencies whose central offices were independent of the program office, those with access to external experts or contractors, and those whose staff were rated as having great or better expertise in research methods and subject matter reported greater use of evaluation in decision making.

Some Agencies Improved Their Evaluation Capacity after GPRAMA Was Enacted

GPRAMA was enacted in January 2011, revising existing GPRA provisions and adding new reporting requirements. Around the same time, OMB increased its outreach to agencies to encourage them to conduct program evaluations. We assessed change in agency evaluation capacity in this period through survey questions about when an office started conducting evaluations and whether the frequency of certain activities had changed. While organizational changes in evaluation capacity were few during this period, half the agencies reported a greater use of evaluation in decision making since 2010.

A Few Agencies Centralized Their Evaluation Responsibilities after GPRAMA Was Enacted

Organizational evaluation capacity has grown some since 2010. One-third of the agencies have a high-level official responsible for oversight of the agency’s evaluation studies, and 2 of those 7 positions were created after 2010, both in 2013. In fact, in its May 2012 memorandum, OMB encouraged agencies to designate a high-level official responsible for evaluation who can

Conduct or oversee rigorous and objective studies;

Provide independent input to agency policymakers on resource “Develop and manage the agency’s research agenda; allocation and to program leaders on program management;

Attract and retain talented staff and researchers, including through flexible hiring authorities such as the Intergovernmental Personnel Act; and

Refine program performance measures, in collaboration with program managers and the Performance Improvement Officer.”

In addition, 4 of 11 agencies with a central office responsible for evaluation reported that this office started conducting evaluations after 2010. One agency added both a central leader and a central office in 2013; 3 others just added a central office. Of the 12 agencies that reported having evaluation offices in their major components, most existed before GPRAMA was enacted, but 5 agencies have established new component evaluation offices since then.

Half the Agencies Reported Increasing Their Capacity Building Activities after GPRAMA Was Enacted

Presumably in response to greater administration attention to program evaluation, half the agencies reported that efforts to improve their capacity to conduct credible evaluations had increased at least somewhat since GPRAMA was enacted in January 2011. About half the PIOs reported increases in staff participation in evaluation conferences and knowledge sharing forums, hiring staff with research and analysis expertise, training staff in research and evaluation skills, and consultation with external research and evaluation specialists. Nine agencies reported increases in all these activities. Most of the remaining agencies reported no change in training or consultation with specialists (4 to 5), or decreases in hiring or participating in conferences (4 to 5) in this period. These decreases may reflect federal budget constraints and the general decline in federal hiring in recent years.

Half the Agencies Reported Increasing Their Use of Evaluations as Evidence in Decision Making after 2010

In line with the increases reported in capacity building activities and organizational resources, about half the agencies reported that their use of evaluation as supportive evidence had increased at least somewhat since 2010 (only a few reported great increases). About half the PIOs reported that the use of evaluation had increased for implementing changes in program management or performance, designing or supporting program reforms, sharing what works or other lessons learned with others, allocating resources within a program, or supporting program budget requests. The rest reported that their use of evaluation evidence remained about the same in this period, with none reporting a decline in use of evaluation as evidence. Eight agencies reported increased use in all these activities, and an equal number reported that their use remained the same on all. Since, in a separate question, 5 agencies either provided no opinion or reported little or no current use of evaluation evidence to support budget, policy, or program management, we conclude that this group has continued to make little or no use of evaluations since 2010.

Some Agencies Reported that Professional Networking, Hiring, Engaging Program Staff, and Some GPRAMA Provisions Were Useful for Building Evaluation Capacity

Our survey asked the PIOs how useful various activities or resources were for improving their agency’s capacity to conduct credible evaluations. Several PIOs did not answer these questions, in part because they were not familiar with such activities. Many of those who did respond found that hiring, professional networking, consulting with experts, and training as well as some of the GPRAMA accountability provisions were very useful for improving capacity to conduct evaluations. Our survey also asked about the usefulness of various activities or resources for improving an agency’s capacity to use evaluations in decision making. Again, several agencies did not respond, but most of those that did reported that engaging program staff, conducting quarterly progress reviews, and holding goal leaders accountable for progress on agency priority goals were very useful in improving agency capacity to make use of evaluation information. Some other GPRAMA-related activities were not found as useful for enhancing evaluation use. In addition, agencies had not taken full advantage of available technology to disseminate evaluation results, thus potentially limiting their influence on decision making.

PIOs Reported Hiring, Professional Networking, and Consulting with Experts Were Very Useful in Improving the Capacity to Conduct Evaluations

Our survey asked the PIOs about the usefulness of 14 different actions or resources for improving their capacity to conduct evaluations, drawn from the literature and some GPRAMA provisions related to building agency capacity. About a third of the respondents indicated either that they had no opinion or did not respond to these questions, similar to the number not responding or reporting no change in the use of capacity-building activities since 2010. About two-thirds of agencies (15) reported hiring staff with research and analysis expertise, and 11—nearly half of the PIOs—thought it was very useful for improving agency capacity to conduct credible evaluations. Almost half the agencies used special hiring authorities, such as the Presidential Management Fellows, Intergovernmental Personnel Act, or American Association for the Advancement of Science (AAAS) fellows program, and generally found them useful for improving agency evaluation capacity. Other agency- specific means of obtaining staff were mentioned in comments—for example, an Evaluation Fellowship Program at the Centers for Disease Control and Prevention. Figure 4 summarizes agencies’ reports on the usefulness of the full range of activities and resources posed for building capacity to conduct evaluations.

The PIO survey respondents also gave high marks to professional networking for building staff capacity. Two-thirds of the PIOs reported that staff participation in professional conferences or evaluation interest groups for knowledge sharing was useful, with 9 PIOs citing these activities as very useful in improving agency capacity to conduct credible evaluations. Examples mentioned included the Association for Public Policy Analysis and Management research conference and an Evaluation Day conference sponsored by the U.S. Department of Health and Human Services (HHS) Office of the Assistant Secretary for Planning and Evaluation.

The exchange of evaluation tips and leading practices through the PIC or other network was considered moderately useful for capacity building by a third of the PIOs. PIOs provided examples of information-sharing networks besides the PIC, such as OMB’s Evaluation Working Group, which holds governmentwide meetings on government performance topics; Federal Evaluators, an informal association of evaluation officials across government; Washington Evaluators, a local affiliate of the American Evaluation Association; and the National Academy of Public Administration. Some agencies have established informal networks to share information internally, such as HHS and the U.S. Department of Labor. Also mentioned were communities of practice that engage both public and private sectors but are focused on a specific domain—for example, the Organisation for Economic Co-operation and Development’s EvalNet, which focuses on international development, and the Environmental Evaluators Network.

Consultation with external experts for conceptual or technical support was rated as very useful for improving the capacity to conduct evaluations by most using it (9 of 15). However, this did not apply to other forms of external consultation. Seven agencies reported having an annual or multi- year evaluation agenda, and 3 of them reported consulting with congressional or other external stakeholders on their plan. These 3 found consultation useful to varying degrees for building their agency’s evaluation capacity to conduct evaluation.

PIOs Reported Training Is Needed to Build Skills

Training in specific skills and knowledge—for example, types of evidence, assessing evidence quality, report writing, and communication—is frequently cited in the evaluation literature as a way to build organizational or individual evaluation capacity. Besides asking about participating in professional conferences and networks, our survey asked about the usefulness of training in evaluation skills—for example, describing program logic models, choosing appropriate evaluation designs, and collecting and analyzing data. Half the agencies reported engaging in internal or external training—whether delivered in a classroom, online, or in webinars. Half the agencies using internal training reported that it was very useful for improving capacity to conduct credible evaluations. PIOs who reported on agency experience with external evaluation training were less enthusiastic, but still considered the training useful for developing evaluation skills overall.

OMB, in addition to encouraging agencies to conduct evaluations through guidance, sponsored a number of governmentwide open forums on performance issues. About half the PIOs reported a range of opinions on the usefulness of the OMB forums on the Paperwork Reduction Act, procurement, data sharing, and related rules and procedures to help improve agency capacity for conducting credible evaluations.

Nevertheless, 7 or more of the agencies identified training or guidance in several skills as still needed to a great or very great extent to improve their agencies’ capacity to conduct credible evaluations. These skills included: translating evaluation results into actionable recommendations—a requirement for getting evaluation results used— data management and statistical analysis, and performance measurement and monitoring. Few reported that more training in research design and methods or subject matter expertise was greatly needed. Our survey asked what other types of training or guidance might be needed to improve agency capacity. A few PIOs commented that training is needed in preparing statements of work for evaluation contracts, data analytics and visualization of information, and learning how to effectively use evidence and evaluation information.

PIOs Reported Some GPRAMA–Related Activities Useful for Building the Capacity to Conduct Evaluation

Our survey asked about several activities and resources related to GPRAMA provisions linked to creating an enabling environment for agency evaluation capacity. Majorities of PIOs stated that conducting quarterly progress reviews on their priority goals, and holding goal leaders accountable for progress on those goals, were moderately to very useful in improving their agency’s ability to conduct credible evaluations. In response to GPRAMA provisions to improve agency performance management capacity, the PIC and OPM developed a Performance Analyst position design, recruitment, and selection toolkit to assist agencies’ hiring. Seven PIOs reported that their agencies used the toolkit, and 3 did not find it useful for building agency evaluation capacity.

About a third of the PIOs reported that their agencies made an effort to incorporate the core competencies that OPM identified for performance management staff into internal agency training. However, 2 of the 7 agencies did not find the effort useful for improving staff evaluation capacity. The competencies primarily address general management skills and define planning and evaluating fairly simply—as setting and monitoring progress on performance goals—so they do not address some of the specific analytic skills PIOs reported were still needed for conducting evaluations. GAO previously recommended that OPM, in coordination with the PIC and the Chief Learning Officer Council, identify performance management competency areas needing improvement and work with agencies to share information about available agency training in those areas. OPM agreed with those recommendations and has embarked on a 2-year pilot program to test how to build capacity in several mission critical competencies identified across government, such as strategic thinking, problem solving, and data analysis, to ensure that both program staff and management can use evaluation and analysis of program performance.

OMB senior officials also engaged with agency officials on the Performance Improvement Council to collaborate on improving program performance. Eight of the 14 agencies that responded considered the exchange of evaluation tips and leading practices through the PIC or other networks as at least moderately useful for improving their evaluation capacity. For example, the PIC developed a guide to best practices for setting milestones and a guide and evaluation tool to help agencies set their agency priority goals.

PIOs Reported Engaging Program Staff and Holding Goal Leaders Accountable Were Very Useful for Building Capacity to Use Evaluations in Decision Making

Previously, we found that experienced evaluators emphasized three basic strategies to facilitate evaluation’s influence on program management and policy: demonstrate leadership support of evaluation for accountability and program improvement, build a strong body of evidence, and engage stakeholders throughout the evaluation process. Accordingly, our survey asked the PIOs how useful various activities or resources were for improving their agency’s capacity to use evaluations in decision making. Several did not answer these questions because they did not use the particular activity or resource or had no opinion. The PIOs who responded mainly cited engaging program staff, conducting quarterly progress reviews, and holding goal leaders accountable for progress on agency priority goals as very useful for improving agency capacity to make use of evaluation information in decision-making.

Over two-thirds of the PIOs responded that involving program staff in planning and conducting evaluation studies was useful for improving agency use of evaluation; 11 saw it as very useful. Engaging staff throughout the process can gain their buy-in on the relevance and credibility of evaluation findings; providing program staff with interim results or lessons learned from early program implementation can help ensure timely data for program decisions. Majorities of PIOs affirmed that other forms of program staff engagement were also very useful: providing program staff and grantees with technical assistance on evaluation and its use and agency peer-to-peer presentations of evaluation studies to discuss methods and findings As mentioned earlier, majorities of PIOs viewed the new GPRAMA activities of conducting quarterly reviews and holding goal leaders accountable as moderately to very useful for improving agency capacity to conduct credible evaluations. Majorities of the responding PIOs also viewed those same activities as moderately to very useful for improving agency capacity to use evaluations in decision making. However, another GPRAMA provision—coordinating with OMB and other agencies to review progress on cross-agency priority (CAP) goals—met with a range of opinions. Equal numbers reported that it was moderately to very useful, somewhat useful, or not useful at all for improving an agency’s use of evaluation. Because the 14 CAP goals for this period cover 5 general management improvement and 9 cross-cutting but specific policy areas, some of the 24 PIOs may have been more involved than others in those reviews.

Other activities potentially useful for improving the capacity to use information from evaluations rely on leveraging resources. A third of the PIOs reported that exchanging leading practices, tips, and tools for using evidence to improve program or agency performance through the PIC or other network was moderately or very useful in improving agency capacity to use evaluation results in decision making. Many of the same networks named as helping to improve their capacity to conduct credible evaluations were also named with regard to improving capacity to use evaluations in decision making. These included the Environmental Evaluators Network, Federal Evaluators, the National Academy of Public Administration, and the OMB Evaluation Working Group.

Seven agencies reported having an agency-wide annual or multi-year evaluation plan or agenda of planned studies, and 6 PIOs reported consulting with congressional and other external stakeholders on that plan. However, these consultations were not viewed as useful for improving their agency’s capacity to use evaluations in decision making. The absence of consultation may miss an opportunity to ensure that evaluations will address the questions of greatest interest to congressional decision makers and will be perceived as credible support for proposed policy or budget changes. In previous work, we found that dialog between congressional committees and executive branch agencies was necessary to achieve a mutual understanding that would allow agencies to provide useful information for oversight.

Agencies May Not Be Taking Full Advantage of Technology to Disseminate Evaluation Results

Previously we found that a key strategy for promoting the use of evaluation findings was to make them digestible and usable and to proactively disseminate them. Our survey posed various options that agencies could take to publicly disseminate their evaluation findings. Half the respondents reported posting evaluation reports in a searchable database on their websites, and half of them viewed this practice as moderately to very useful for improving their agency’s capacity to use evaluations in decision making. However, 3 did not find the practice useful. Electronic mailing lists are more proactive than posting a report to a website and permit tailoring the message to different audiences. A third of all respondents disseminated evaluation reports by electronic mailing lists, which most saw as somewhat to very useful for facilitating the use of evaluations in decision making. Tailoring messages for particular audiences—for example, federal policy makers, state and local agencies, and local program affiliates—may, however, increase the applicability and use of evaluation findings by these other audiences.

GPRAMA requires OMB to provide quarterly updates on agency and cross-agency priority goals on a central, government-wide website, Performance.gov, to make federal program and performance information more accessible to the Congress and the public. In our survey, PIO reviews were mixed about the utility of this website to improve agency capacity to use evaluations in decision making. Almost half the agencies found the practice somewhat to moderately useful for improving the agencies’ use of evaluation findings in decision making, but one-fourth of the agencies did not. In 2013, GAO reviewed Performance.gov and recommended that OMB work with the General Services Administration and the PIC to clarify specific ways that intended audiences could use the website and specify changes to support these uses. OMB staff agreed with our recommendations, and Performance.gov continues to evolve. Currently each agency has a home page that provides links to the agency’s strategic plan, annual performance plans and reports, and other progress reviews.

Data.gov is a federal government website that provides descriptions of datasets generated or held by the federal government in order to increase the ability of the public to locate, download, and use those datasets. A third of the PIOs reported that sharing databases in public repositories such as Data.gov for researchers and the public to use helped in improving agency capacity to use evaluations in decision making, but 1 thought it was not useful. However, a third of PIOs stated that the agency did not use this vehicle. Vehicles such as Data.gov and Performance.gov are primarily intended to improve government transparency and expand information’s use by the Congress and the public, but they can also help support agency requests for budget and policy changes to improve government performance.

Concluding Observations

Although OMB and several agencies have taken steps since 2010 to expand federal evaluation efforts, most agencies demonstrate rather modest evaluation capacity. Those with centralized evaluation authority reported greater evaluation coverage and use in decision making, but additional effort will be required to expand agencies’ evaluation capacity beyond those that already possess evaluation expertise. In addition to hiring and training staff and consulting experts, promoting information sharing through informal and formal evaluation professionals’ networks offers promise for building agencies’ capacity to conduct evaluation in a constrained budget environment. Engaging program staff, regularly reviewing progress on agency priority goals, and holding goal leaders accountable can help build agency use of evaluation in decision making, as our survey results show. While timely, public dissemination of performance and evaluation results may not directly influence agency decision making, it is important to support government transparency and accountability for results to the Congress and the public.

Directly engaging intended users (for example, involving program staff in planning and conducting evaluations and holding regular progress reviews) was strongly associated with increasing evaluation use in internal agency decision making. In contrast, few agencies reported consulting congressional and other external stakeholders in conducting their evaluation studies or developing their evaluation agendas. However, some program reforms require program partners and legislators to take action. Engaging congressional and other stakeholders in evaluation planning might increase their interest in evaluation as well as their adoption of evaluation findings and recommendations.

In the absence of explicit authority or congressional request, agencies may be reluctant to spend increasingly scarce funds on evaluation studies that are perceived as resource intensive. A stable source of evaluation funding could help maintain a viable evaluation program that produced a steady stream of information to guide program management and policy making. Even so, only a quarter of the agencies in our survey reported that their evaluation offices had a stable source of funding. Congressional appropriators could direct the use of program or agency funds for evaluating federal programs and policies. As we have noted before, congressional committees can also communicate their interest in evaluation in a variety of ways to encourage agencies to produce credible, relevant studies that inform decision making: consult with agencies on proposed revisions to their strategic plans and priority goals, as GPRAMA requires them to do every 2 years, to ensure that agency missions are focused, goals are specific and results-oriented, and strategies and funding expectations are appropriate and reasonable; request agency evaluations to address specific questions about the implementation and results of major program or policy reforms, in time to consider their results in program reauthorization; and review agencies’ annual evaluation plans or agendas to ensure that they address issues that will inform budgeting, reauthorization, and ongoing program management.

Agency Comments

We requested comments on a draft of this report from the Director of the Office of Management and Budget, whose staff provided technical comments that we incorporated as appropriate, and from the Director of the Office of Personnel Management, who provided none.

We are sending copies of this report to other interested congressional committees, and the Director of the Office of Management and Budget and the Director of the Office of Personnel Management. In addition, the report will be available on our web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2700 or by e-mail at kingsburyn@gao.gov. Contacts for our Office of Congressional Relations and Office of Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Methodology for the Survey of Performance Improvement Officers in CFO Act Agencies

We administered a web-based questionnaire from May 2, 2014, to June 19, 2014, on federal agency evaluation capacity resources and activities to the Performance Improvement Officers (PIO) or their deputies at the 24 agencies covered by the Chief Financial Officers Act of 1990 (CFO act). We received responses from all 24 agencies (listed at the end of this appendix.) The survey gave us information about agencies’ evaluation resources, policies, and activities, and the activities and resources they have found useful in building their evaluation capacity. (The survey questions and summarized results are in appendix II.) We sent respondents an e-mail invitation to complete the survey on a secure GAO web server. Each e-mail contained a unique username and password. During the data collection period, we sent follow-up e-mails and, if necessary, called nonresponding agencies on the telephone.

Because this was not a sample survey, it has no sampling errors. In practice, however, any survey may introduce nonsampling errors that stem from differences in how a particular question is interpreted, the availability of sources of information, or how the survey data are analyzed. All can introduce unwanted variability into the survey results. We took a number of steps to minimize these nonsampling errors. A social science survey specialist designed the questionnaire, in collaboration with our staff who had subject matter expertise. In addition, we pretested the questionnaire in person with PIOs at three federal agencies to make sure that the questions were relevant, clearly stated, easy to comprehend, and unbiased. We also affirmed that data and information the PIOs would need to answer the survey were readily obtainable and that answering the questionnaire did not place an undue burden on them.

Additionally, a senior methodologist within our agency independently reviewed a draft of the questionnaire before we administered it. We made appropriate revisions to its contents and format after the pretests and independent review. When we analyzed data from the completed survey, an independent analyst reviewed all computer programs used in our analysis. Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire; thus, we did not key the data into a database, avoiding data entry errors.

Additionally, in reviewing the agencies’ answers, we confirmed that the PIOs had correctly bypassed inapplicable questions (such as questions we expected them to skip). We concluded from our review that the survey data were sufficiently reliable for the purposes of this report.

The 24 agencies subject to the CFO Act include

Agency for International Development

Department of Agriculture

Department of Commerce

Department of Defense

Department of Education

Department of Energy

Department of Health and Human Services

Department of Homeland Security

Department of Housing and Urban Development

Department of the Interior

Department of Justice

Department of Labor

Department of State

Department of Transportation

Department of the Treasury

Department of Veterans Affairs

Environmental Protection Agency

General Services Administration

National Aeronautics and Space Administration

National Science Foundation

Nuclear Regulatory Commission

Office of Personnel Management

Small Business Administration

Social Security Administration.

Appendix II: Results from 2014 Survey of Performance Improvement Officers

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Stephanie Shipman (Assistant Director), Thomas Beall, Valerie Caracelli, Timothy Carr, Joanna Chan, Stuart Kaufman, and Penny Pickett made key contributions to this report.

References

Administration for Children and Families. Evaluation Policy. Washington, D.C.: Department of Health and Human Services, November 2012. Accessed September 24, 2014. http://www.acf.hhs.gov/programs/opre/resource/acf-evaluation-policy.

America Achieves. “Investing in What Works Index: Better Results for Young People, Their Families, and Communities.” Results for America, Washington, D.C., May 2014. Accessed September 11, 2014. http://www.InvestInWhatWorks.org/policy-hub.

American Evaluation Association. An Evaluation Roadmap for a More Effective Government. N.p.: Revised October 2013. Accessed September 22, 2014. http://www.eval.org/d/do/472.

Auditor General of Canada. 2013 Spring Report of the Auditor General of Canada. Ch. 1. “Status Report on Evaluating the Effectiveness of Programs.” Ottawa: 2013. Accessed September 15, 2014. http://www.oag- bvg.gc.ca/internet/English/parl_oag_201304_01_e_38186.html.

Bourgeois, Isabelle, and J. Bradley Cousins. “Understanding Dimensions of Organizational Evaluation Capacity,” American Journal of Evaluation, 34:3 (2013): 299—319.

Chapel, Thomas. “Building and Sustaining Evaluation Capacity in a Diverse Federal Agency.” Paper presented at Federal Evaluators Conference, Washington, D.C., November 1, 2012. Accessed September 11, 2014. http://www.fedeval.net/presen.htm.

Clapp-Wincek, Cindy. “The Complexity of Building Capacity at USAID.” Paper presented at Federal Evaluators Conference, Washington, D.C., November 1, 2012. Accessed September 11, 2014. http://www.fedeval.net/presen.htm.

Cousins, J. Bradley, Swee C. Goh, Catherine J. Elliott, and Isabelle Bourgeois. “Framing the Capacity to Do and Use Evaluation,” New Directions for Evaluation, 133 (Spring 2014): 7—24.

Dawes, Katherine. “Program Evaluation at EPA.” Paper presented at Federal Evaluators Conference, Washington, D.C., November 1, 2012. Accessed September 11, 2014. http://www.fedeval.net/presen.htm.

Goldman, Ian. “Developing a National Evaluation System in South Africa,” eVALUatiOn Matters: A quarterly knowledge publication of the African Development Bank, 2(3) (September 2013): 42—49.

Labin, Susan N., Jennifer L. Duffy, Duncan C. Meyers, Abraham Wandersman, and Catherine A. Lesesne. “A Research Synthesis of the Evaluation Capacity Building Literature,” American Journal of Evaluation, 33:307 (2012).

National Audit Office. Cross-Government: Evaluation in Government. Report by the National Audit Office. London, Eng. December 2013. Accessed September 24, 2014. www.nao.org.uk.

Partnership for Public Service and Grant Thornton. A Critical Role at a Critical Time: A Survey of Performance Improvement Officers. Washington, D.C.: April 2011. Accessed September 16, 2014. http://ourpublicservice.org/OPS/publications/viewcontentdetails.php?id=1 60.

Partnership for Public Service and Grant Thornton. Taking Measure: Moving from Process to Practice in Performance Management. Washington, D.C.: September 2013. Accessed September 16, 2014. http://ourpublicservice.org/OPS/publications/viewcontentdetails.php?id=2 32.

Partnership for Public Service and IBM Center for the Business of Government. From Data to Decisions III: Lessons from Early Analytics Programs. Washington, D.C.: November 2013. Accessed September 16, 2014. http://ourpublicservice.org/OPS/publications/viewcontentdetails.php?id=2 33.

Pew Charitable Trusts and MacArthur Foundation. States’ Use of Cost- Benefit Analysis: Improving Results for Taxpayers. Philadelphia: Pew- MacArthur Results First Initiative, July 29, 2013. Accessed October 31, 2014. http://www.pewtrusts.org/en/research-and- analysis/reports/2013/07/29/states-use-of-costbenefit-analysis.

Rist, Ray C., Marie-Helene Boily, and Frederic Martin. Influencing Change: Building Evaluation Capacity to Strengthen Governance. Washington, D.C.: The World Bank, 2011. Accessed September 24, 2014. https://openknowledge.worldbank.org/ Segone, Marco, Caroline Heider, Riitta Oksanen, Soma de Silva, and Belen Sanz. “Towards a Shared Framework for National Evaluation Capacity Development,” eVALUatiOn Matters: A Quarterly Knowledge Publication of the African Development Bank, 2(3) (September 2013): 7— 25.

Segone, Marco, and Jim Rugh (eds.). Evaluation and Civil Society: Stakeholders’ Perspectives on National Evaluation Capacity Development. New York: UNICEF, EvalPartners, IOCE, 2013. Accessed September 24, 2014. http://www.mymande.org/Evaluation_and_Civil_Society.

Treasury Board of Canada. 2011 Annual Report on the Health of the Evaluation Function. Ottawa: 2012. Accessed September 24, 2014. http://www.tbs-sct.gc.ca/report/orp/2012/arhef-raefetb-eng.asp.

United Nations Evaluation Group. National Evaluation Capacity Development: Practical Tips on How to Strengthen National Evaluation Systems. A report for the United Nations Evaluation Group Task Force on National Evaluation Capacity Development. New York: 2012. Accessed September 24, 2014. www.uneval.org/document/detail/1205.

U.S. Agency for International Development. Evaluation: Learning from Experience. USAID Evaluation Policy. Washington, D.C.: January 2011. Accessed September 25, 2014. http://www.usaid.gov/evaluation.

U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. Improving the Use of Program Evaluation for Maximum Health Impact: Guidelines and Recommendations. Atlanta: November 2012. Accessed September 24, 2014. http://www.cdc.gov/eval.

U.S. Department of Labor. U.S. Department of Labor Evaluation Policy. Washington, D.C.: November 2013. Accessed September 25, 2014. http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm.

U.S. Department of State. Department of State Program Evaluation Policy. Washington, D.C.: February 23, 2012. Accessed September 24, 2014. http://www.state.gov/s/d/rm/rls/evaluation/.

Related GAO Products

Managing for Results: Agencies’ Trends in the Use of Performance Information to Make Decisions. GAO-14-747. Washington, D.C.: September 26, 2014.

Managing for Results: Enhanced Goal Leader Accountability and Collaboration Could Further Improve Agency Performance. GAO-14-639. Washington, D.C.: July 22, 2014.

Managing for Results: OMB Should Strengthen Reviews of Cross-Agency Goals. GAO-14-526. Washington, D.C.: June 10, 2014.

Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts. GAO-14-8. Washington, D.C.: December 5, 2013.

Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. GAO-13-570. Washington, D.C.: June 26, 2013.

Managing for Results: Leading Practices Should Guide the Continued Development of Performance.gov. GAO-13-517. Washington, D.C.: June 6, 2013.

Managing for Results: Agencies Have Elevated Performance Management Leadership Roles, but Additional Training Is Needed. GAO-13-356. Washington, D.C.: April 16, 2013.

Managing for Results: Data-Driven Performance Reviews Show Promise but Agencies Should Explore How to Involve Other Relevant Agencies. GAO-13-228. Washington, D.C.: February 27, 2013.

Managing for Results: A Guide for Using the GPRA Modernization Act to Help Inform Congressional Decision Making. GAO-12-621SP. Washington, D.C.: June 15, 2012.

President’s Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning and Dissemination. GAO-12-673. Washington, D.C.: May 31, 2012.

Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.: January 2012.

Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program. GAO-11-285. Washington, D.C.: March 15, 2011.

Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.

Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program. GAO-10-243. Washington, D.C.: January 29, 2010.

Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions. GAO-10-30. Washington, D.C.: November 23, 2009.

Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.: May 2, 2003.

Program Evaluation: Improving the Flow of Information to Congress. GAO/PEMD-95-1. Washington, D.C.: January 30, 1995.