Program Evaluation: Agencies Challenged by New Demand for Information on Program Results
Published: 1998-04-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed federal agencies' efforts to provide information on federal program results, focusing on: (1) the current resources and roles for program evaluation in federal agencies; (2) the anticipated effects of governmentwide reforms and other initiatives on evaluation of federal programs; and (3) potential strategies for agencies to respond to the anticipated effects and provide information on program results.
section_title: What GAO Found
paragraphs: GAO noted that: (1) existing federal evaluation resources--at least as currently configured and deployed--are likely to be challenged to meet increasing demands for program results information; (2) agencies reported devoting variable but relatively small amounts of resources to evaluating program results; (3) morever, agencies reported that the primary role of program evaluation was internally focused on program improvement, rather than direct congressional or other external oversight; (4) interest in the program by high-level officials was most often cited as a criterion for initiating evaluation work; a small portion of studies were said to be conducted for a congressional committee or in response to a legislative mandate; (5) some of the evaluation officials and experts that GAO interviewed anticipated not only increased interest in learning the results of federal programs and policies but also additional complications in obtaining that information; (6) some evaluation officials from states with performance measurement experience noted that effectiveness evaluations would continue to be needed to assess policy impact and address problems of special interest or larger policy issues, such as the need for any government intervention at all in an area; (7) to meet the anticipated increase in demand for program results information as well as the associated technical challenges, some evaluation officials GAO interviewed described efforts to leverage both federal and nonfederal resources; (8) however, some agencies anticipated that major investments in their data systems would be required to produce reliable data on program outcomes; and, in a prior study, program officials were concerned that reliance on less rigorous methods would not provide an accurate picture of program effectiveness; (9) moreover, while some federal evaluation officials envisioned providing increased technical assistance to state and local evaluators, a few state evaluation officials suggested an alternative strategy for the federal government; (10) GAO drew several conclusions from its comparison of current federal evaluation resources with the anticipated challenges to meeting increased demand for information on program results; (11) federal evaluation resources have important roles to play in responding to increased demand for information on program results, but--at least as currently configured and deployed--they are likely to be challenged to meet that demand; and (12) in the future, carefully targeting federal agencies' evaluation resources shows promise for addressing key questions about program results.

section_title: Letter
paragraphs: Congressional and agency decisionmakers need evaluative information about whether federal programs are working well or poorly, both to manage programs effectively and to help decide how to allocate limited federal resources. Increased interest in learning the results of federal programs and activities is reflected in government reforms, such as the Government Performance and Results Act of 1993 (GPRA or the Results Act), which institutes a governmentwide requirement for agencies to, among other things, report on their results in achieving their agency and program goals. However, other recent reforms, such as reducing the size and authority of the federal government while maintaining a level of services, has the potential to hinder agencies’ ability to obtain this information. Because data on program results are typically more difficult and resource intensive to obtain than data on program activities, limited budget dollars mean that investing in obtaining information on results may compete with spending on program activities. The proper balance between the two spending priorities is essential, since information on program results can contribute to deciding how to allocate resources to activities to maximize program benefits.
Federal agencies are the primary source of evaluation information about their programs. In past surveys of federal agencies, we found limited (and diminishing) resources spent on formal studies of program results, that is, program evaluation. Because evaluation can be vitally important in improving program results, we asked how in a context of limited federal resources and responsibility, can agencies support additional requests for program results information? This report, which we prepared under our basic legislative responsibilities, responds to that question by discussing the current status of and future needs for program evaluation in federal agencies. Because of your interest in improving the quality of information on federal programs, we are addressing this report to you. Our objectives were to identify (1) the current resources and roles for program evaluation in federal agencies, (2) the anticipated effects of governmentwide reforms and other initiatives on evaluation of federal programs, and (3) potential strategies for agencies to respond to the anticipated effects and provide information on program results.
subsections: section_title: Background
paragraphs: Program and policy decisions require a wide array of information that answers various questions. For example, descriptive information tells how a program operates—what activities are performed, who performs them, and who is reached. In contrast, evaluative information speaks to how well a program is working—such as whether activities are managed efficiently and effectively, whether they are carried out as intended, and to what extent the program is achieving its intended objectives or results. There are a variety of methods for obtaining information on program results, such as performance measurement and program evaluation, which reflect differences in how readily one can observe program results.
Performance measurement, as defined by the Results Act, is the ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals. It tends to focus on regularly collected data on the type and level of program activities (process), the direct products and services delivered by the program (outputs), and the results of those activities (outcomes). While performance may be defined more broadly as program process, inputs, outputs, or outcomes, results usually refer only to the outcomes of program activities. For programs that have readily observable results, performance measurement may provide sufficient information to demonstrate program results.
In some programs, however, results are not so readily defined nor measured. In such cases, program evaluations may be needed, in addition to performance measurement, to examine the extent to which a program is achieving its objectives. Program evaluations are systematic studies conducted periodically to assess how well a program is working. While they may vary in their focus, these evaluations typically examine a broader range of information on program performance and its context than is feasible in ongoing performance measurement. Where programs aim to produce changes, as a result of program activities, outcome (or effectiveness) evaluations assess the extent to which those outcomes or results were achieved, such as whether students increased their understanding of or skill in the material of instruction. In cases where the program’s outcomes are influenced by complex systems or events outside the program’s control, impact evaluations use scientific research methods to establish the causal connection between outcomes and program activities, estimate what would have happened in the absence of the program, and thus isolate the program’s contribution to those changes. For example, although outcome measures might show a decline in a welfare program’s caseload after the introduction of job placement activities, a systematic impact evaluation would be needed to assess how much of the observed change was due to an improved economy rather than the new program.
In addition, a program evaluation that also systematically examines how a program was implemented can provide important information about why a program did or did not succeed and suggest ways to improve it. For the purposes of this report, we used the definition of program evaluation that is used in the Results Act, “an assessment, through objective measurement and systematic analysis, of the manner and extent to which federal programs achieve intended objectives.” We asked about assessments of program results, which could include both the analysis of outcome-oriented program performance measures as well as specially conducted outcome or impact evaluations.
Two government initiatives could influence the demand for and the availability and use of program evaluation information. The Results Act seeks to promote a focus on program results, by requiring agencies to set program and agency performance goals and to report annually on their progress in achieving them (beginning with fiscal year 1999). In addition to encouraging the development of information on program results for activities across the government, the Results Act recognizes the complementary nature of program evaluation and performance measurement. It requires agencies to include a schedule for future program evaluations in their strategic plans, the first of which was to be submitted to Congress by September 30, 1997. The Results Act also requires agencies to review their success in achieving their annual performance goals (which are set forth in their annual performance plans) and to summarize the findings of program evaluations in their annual program performance reports (the first of which is due by March 31, 2000). The National Performance Review (NPR) led by the Vice President’s office has asked agencies to reexamine their policies, programs, and operations to find and implement ways to improve performance and service to their customers. Both of these initiatives—because of their focus on program results—could be expected to increase the demand for and the availability and use of program evaluation information.
Other recent governmentwide initiatives could have potentially conflicting effects. In several program areas, devolution of program responsibility from the federal level and consolidation of individual federal programs into more comprehensive, multipurpose grant programs has shifted both program management and accountability responsibilities toward the states. These initiatives may thus make it more difficult for federal agencies to evaluate the results of those programs. In addition, efforts to reduce the growth of the federal budget have resulted in reductions in both federal staff and program resources in many agencies. The combination of these initiatives raises a question: In an environment of limited federal resources and responsibility, how can agencies meet the additional needs for program results information?
To identify the roles and resources available for federal program evaluation, in 1996, we conducted a mail survey of offices identified by federal agency officials that were conducting studies of program results or effectiveness in 13 cabinet-level departments and 10 independent executive agencies. Detailed information on program evaluation studies refers to those conducted during fiscal year 1995 (regardless of when they began or ended). To identify how recent reforms were expected to affect federal evaluation activities and what strategies were available for responding to those changes, we interviewed external evaluation experts and evaluation and other officials at selected federal and state agencies. In this report, we use the term “agency” to include both cabinet-level departments and independent agencies.
section_title: Scope and Methodology
paragraphs: We distributed surveys in 1996 regarding federal evaluation activities within 13 cabinet level departments and 10 independent executive agencies in the federal government. We excluded the Department of Defense from our survey of evaluation offices because of the prohibitively large number of offices it identified as conducting assessments of effectiveness. Although we asked agency officials to be inclusive in their initial nominations of offices that conducted evaluations of program results, some offices that conducted evaluations may have been overlooked and excluded from our survey. However, many offices that were initially identified as having conducted evaluations later reported that they had not done so. In our survey, we asked each office about the range of its analytic and evaluation activities and about the length, cost, purpose, and other characteristics of the program evaluation studies they conducted during fiscal year 1995. (See appendix I for more details on the scope and methodology of the survey.)
Between 1996 and 1997, we conducted interviews of program evaluation practitioners selected to represent divergent perspectives. We asked what had been or were expected to be the effects of various government changes and reforms on federally supported and related program evaluation activities and strategies for responding to those effects. We identified individuals with evaluation knowledge and expertise from a review of the survey responses, the evaluation literature, and our prior work; they were from an array of federal and state agencies and the academic and consulting communities. We then judgmentally selected 18 people to interview to reflect (1) a mix of different program types and diverse amounts of experience with program evaluation and (2) experience with some of the reforms at the state or federal level. Those selected included nine evaluation officials (six from offices in federal agencies and three from state legislative audit agencies) and seven external evaluation experts (four from private research organizations and three from universities). In addition, we interviewed an OMB official and one official from a state executive branch agency, and we also asked the officials from the state legislative audit agencies about their experiences with state performance reporting requirements.
We conducted our review between May 1996 and July 1997 in accordance with generally accepted government auditing standards. However, we did not independently verify the types of studies conducted, other information reported by our respondents, nor information gained from interviewees.
section_title: Federal Agencies Devote Variable but Small Amounts of Resources to Evaluating Program Results, Primarily for Internal Use
paragraphs: The resources allocated to conducting systematic assessments of program results (or evaluation studies) were small and unevenly distributed across the 23 agencies (departments and independent agencies) we surveyed. We found 81 offices that reported expending resources—funds and staff time—on conducting program effectiveness studies in fiscal year 1995. Over half of those offices had 18 or fewer full-time equivalent staff FTEs, while only a few offices had as many as 300 to 400 FTEs. (See figure 1.)
Moreover, about one-third of the offices reported spending 50 percent or more of their time on evaluation activities (including development of performance measures and assessments of program effectiveness, compliance, or efficiency), since program evaluation was only one of these offices’ many responsibilities. (See survey question 9 in appendix I.) Two of the 3 largest offices (over 300 FTEs) spent about 10 percent of their staff time on program evaluation activities. Thus, the estimated staff and budget resources that the 81 offices actually devoted to evaluation activities totaled 669 FTEs and at least $194 million across the 23 agencies surveyed.
In addition, most (61 of 81) offices reported also conducting management analysis activities; the most frequent activities were conducting management studies, developing strategic plans, and describing program implementation. Of those offices that could estimate their staff time, about half reported spending less than 25 percent of their time on management analysis. Similarly, many offices reported conducting policy planning and analysis, but most of them reported spending less than 25 percent of their time on it. Thus, a majority of the offices (45 of the 81 identified) conducted few evaluation studies (5 or less in fiscal year 1995), while 16 offices—representing 7 agencies—accounted for two-thirds of the 928 studies conducted. (See table 1.)
Finally, 6 of the 23 agencies we surveyed did not report any offices conducting evaluation studies in fiscal year 1995. A few of these agencies indicated that they analyzed program accomplishments or outputs or conducted management reviews to assess their programs’ performance but did not conduct an evaluation study per se. Some of the 6 agencies also reported conducting other forms of program reviews that focused on assessing program compliance or efficiency rather than program results.
Offices conducting program evaluations were located at various levels of federal agencies. A few of the 81 offices were located in the central policy or administrative office at the highest level of the organization (5 percent) or with the Inspector Generals (5 percent); many more were located in administrative offices at a major subdivision level (43 percent) or in program offices or technical or analytic offices supporting program offices (30 and 16 percent, respectively). (See table 2.)
Four of the 23 agencies surveyed had offices at all 3 levels (agency, division, and program), and over half the agencies (14 of 23) had conducted evaluations at the program level. The 16 offices conducting 20 or more studies were more likely to be centralized at the agency or division level than at the program level.
A diverse array of evaluation studies were described in the surveys. Just over half of the studies for which we have such information were conducted in-house (51 percent), and 27 percent lasted under 6 months. But the studies that were contracted out tended to be larger investments—almost two-thirds of them took over a year to complete, and over half cost between $100,000 and $500,000. Moreover, almost a third of all the studies lasted more than 2 years, reflecting some long-term evaluations. (See table 3.)
For example, a study of the impact of a medical treatment program, which used an experimental design with a complex set of Medicare program and clinical data from thousands of patients on numerous outcomes (for both patients and program costs), took over 2 years and cost over $1 million to complete.
Many of the 1995 studies reportedly used relatively simple designs or research methods, and many relied on existing program data. The two most commonly reported study designs were judgmental assessments (18 percent) as well as experimental designs employing random assignment (14 percent). (See table 4 for a list of designs ranging from the most to least amount of control over the study conditions.)
Many of the studies (over 70 of the 129) that used experimental designs were evaluations of state demonstration programs, which were required by law to use such methods, and were conducted out of one office. Experimental designs and designs using statistical controls are used to identify a program’s net impact on its objectives where external factors are also known to affect its outcome. However, without knowing the circumstances of many of the programs being evaluated, it is impossible for us to determine the adequacy of the designs used to assess program effectiveness.
At least 40 percent of the studies employed existing program records in their evaluations, while about one-quarter employed special surveys or other ad hoc data-collection methods specially designed for the studies. Just under half (40 percent) of the studies used data from program administrative records that were produced and reported at the federal level; almost a third (28 percent) used data from routinely produced, but not typically reported, program records; 5 percent of the studies used data from administrative records of other federal agencies; and 14 percent used administrative records from state programs. Some studies may have used many types of data sources, which would suggest a heavy reliance on administrative and other program-related data. (See table 5.)
subsections: section_title: Studies Primarily Served Internal Program Improvement Purposes, Not Congressional Oversight
paragraphs: The primary reported purpose of the studies was to evaluate ongoing programs, either on an office’s own initiative or at the request of top agency officials. In the survey, most officials conducting evaluations reported having a formal and an ad hoc planning process for deciding what evaluation work they would do. Many criteria were indicated as being used to select which studies to do (such as a program office request, congressional interest, or continuation or follow-up of past work), but the criterion most often cited was the interest of high-level agency officials in the program or subject area. Moreover, about one-fourth of the studies were requested by top agency officials. About one-fourth of the studies were indicated to be self-initiated. Most offices were not conducting studies for the Congress or as the result of legislative mandates; only 17 percent of the studies were reported to have been requested in those ways. (See table 6.)
For those offices reporting that they conducted studies, about half of the 570 studies for which we have information evaluated ongoing programs.Ongoing programs of all sizes were evaluated, ranging in funding from less than $10 million to over $1 billion. About one-third of these studies evaluated demonstration programs and many of them cost less than $10 million. In contrast, few reported evaluations of new programs and many of these new programs reportedly were small (with funding under $10 million).
Program evaluation was reported to be used more often for general program improvement than for direct congressional oversight. Their primary uses most often were said to be to improve program performance (88 percent), assess program effectiveness (86 percent), increase general knowledge about the program (62 percent), and guide resource allocation decisions within the program (56 percent). (See table 7.) Accordingly, these offices overwhelmingly (over three-fourths of respondents) reported program managers and higher-level agency officials as the primary audience of their studies. (See table 8.)
About one-third of the offices reported support for budget requests as a primary use and one-third reported congressional audiences were primary users for their studies. Fewer respondents (20 percent) reported program reauthorization as a primary use of the study results. (See tables 7 and 8.)
Program evaluation was not the primary responsibility for most of these offices and the offices often reported “seldom, if ever” performing the program evaluation roles we asked about. The role most likely to be characterized as ‘most often performed’ was conducting studies of programs administered elsewhere in their agency. (See table 9.)
About one-half of those who responded reported “sometimes” providing technical or design assistance to others or conducting joint studies, while a few offices saw their role as training others in research or evaluation methods. One office dealing with an evaluation mandate conducted work sessions with state and local program managers and evaluators as well as provided training to enhance state evaluation capabilities. Two-thirds of the offices seldom, if ever, designed evaluations conducted by other offices or agencies, trained others in research or evaluation methods, or approved plans for studies by others.
section_title: Reforms May Increase Interest In, But Complicate The Process of, Obtaining Results
paragraphs: Some of our interviewees thought that recent governmentwide reforms would increase interest in learning the results of federal programs and policies but would also complicate the task of obtaining that information. Devolution of federal program responsibility in the welfare and health care systems has increased interest in evaluation because the reforms are major and largely untested. However, in part because programs devolved to the states are expected to operate quite diversely across the country, some evaluation officials noted that evaluating the effects of these reforms was expected to be more difficult. In addition, federal budget reductions over the past few years were said by some not only to have reduced the level of federal evaluation activity but also to have diminished agency technical capacity through the loss of some of their most experienced staff.
Because implementation of the Results Act’s performance reporting requirements is not very far along (the first annual reports on program performance are not due until March 2000), several of our interviewees thought it was too early to estimate the effect of the Results Act. Some hoped the Act would increase the demand for results information and expand the role of data and analysis in decisionmaking. One interviewee thought it would improve the focus of the evaluations they now conduct. A few evaluation officials were concerned that a large investment would be required to produce valid and reliable outcome (rather than process) data. A few also noted that resources for obtaining data on a greatly expanded number of program areas would compete for funds used for more in-depth evaluations of program impact. Other evaluators noted that changes in the unit of analysis for performance reporting from the program level to budget account or organization might make classic program evaluation models obsolete.
As we previously reported, the federal program officials who have already begun implementing performance measurement appeared to have an unusual degree of program evaluation support and found it quite helpful in addressing the analytic challenges of identifying program goals, developing measures, and collecting data. Many of these program officials said they could have used more of such assistance; but, when asked why they were not able to get the help they needed, the most common response was that it was hard to know in advance that evaluation expertise would be needed. In addition to using program evaluation techniques to clarify program goals and develop reliable measures, several of these program officials saw the need for impact evaluations to supplement their performance data. Their programs typically consisted of efforts to influence highly complex systems or events outside government control, where it is difficult to attribute a causal connection between their program and its desired outcomes. Thus, without an impact evaluation or similar effort to separate the effects of their programs from those of other external events or factors, program officials from the previous study recognized that simple examination of outcome measures may not accurately reflect their programs’ performance.
Some states’ experiences with performance measurement suggested that performance measurement will take time to implement, and the federal experience suggests that it will not supplant the need for effectiveness evaluations. Two state officials described a multiyear process to develop valid and reliable measures of program performance across the state government. While performance measures were seen as useful for program management, some state agency and legislative staff also saw a continuing need for evaluations to assess policy impact or address problems of special interest or “big-picture” concerns, such as whether a government program should be continued or privatized.
NPR was seen by several of those we interviewed as not having much of an effect on efforts to evaluate the results of their programs beyond increasing the use of customer surveys. This may have been because it was seen as primarily concerned with internal government operations, or because, as one agency official reported, its effect was most noticeable in only a few areas: regulatory programs and other intergovernmental partnerships. However, one agency official said that NPR had a big impact on reorienting their work toward facilitating program improvement, while two others felt that it reaffirmed changes they had already begun.
section_title: Agencies Suggested Varied Strategies for Obtaining Program Results Information
subsections: section_title: Leveraging Limited Resources
paragraphs: Given constraints on federal budgets, some officials we interviewed in general did not expect federal evaluation resources to rise to meet demand, so they described efforts to leverage and prioritize available resources. While an evaluation official reported supplementing his evaluation teams with consultants, concern was also expressed that staff reductions in their unit had left the technical expertise too weakened to competently oversee consultants’ work. Another evaluation official explained that they responded to the increasing demand for information by narrowing the focus and scope of their studies to include only issues with major budget implications or direct implications for agency action. Both a state official and two external evaluation experts felt that states grappling with new program responsibilities would have difficulty evaluating them as well, so that continued federal investment would be needed. A federal official, however, noted that private foundations could fund the complex rigorous studies needed to answer causal questions about program results.
Some of the evaluators we interviewed expected that fewer impact studies would be done. Some expected that the range of their work may broaden to rely on less rigorous methods and include alternatives such as monitoring program performance and customer satisfaction. From our interviews, we learned that a few agencies have devolved responsibility for evaluations to the program offices, which may have more interest in program improvement. Another agency reported that it had built evaluation into its routine program review system, which provides continuous information on the success of the program and its outcomes, noting that it thereby reduced the need for special evaluation studies. One evaluation official reported that by having redefined evaluation as part of program management, program evaluation became more acceptable in his agency because it no longer appeared to be overhead.
A few agencies reported that they were adapting the elements of their existing program information systems to yield information on program results. But in other agencies, evaluation officials and external experts thought that their systems were primarily focused on program process, rather than results. The evaluation official said that structural changes to, and a major investment in, their data systems will be required to provide valid and meaningful data on results.
section_title: Creating New Evaluation Designs and Partnerships
paragraphs: As program responsibility shifts to state and local entities, evaluation officials and others we interviewed described the need for study designs that can handle greater contextual complexity, new ways to measure outcomes, and the need to build partnerships with the programs’ stakeholders. One of the officials saw classical experimental research designs as no longer feasible in programs, which, due to increased state flexibility in how to deliver services, no longer represented a discrete national program or were unlikely to employ rigorous evaluation techniques that entailed random assignment of particular program services to individuals. Others noted the need to develop evaluation designs that could reflect the multiple levels on which programs operate and the organizational partnerships involved. To address some of these complexities, federal offices with related program interests have formed task groups to attempt to integrate their research agendas on the effects of major changes in the health and welfare systems. Similarly, a few federal evaluation officials reported an interest in consulting with their colleagues in other federal offices to share approaches for tackling the common analytic problems they faced.
In other strategies, federal evaluation officials described existing or planned efforts to change the roles they and other program stakeholders played in conducting evaluations. One agency has arranged for the National Academy of Sciences to work with state program officials and the professional communities involved to help build a prototype performance measurement system for federal assistance to state programs. One evaluation office expects to shift its role toward providing more technical assistance to local evaluators and synthesizing their studies’ results. Another federal office has delegated some evaluation responsibility to the field while it synthesizes the results to answer higher level policy questions, such as which types of approaches work best.
The Results Act recognizes and encourages the complementary nature of program evaluations and performance measures by asking agencies to provide a summary of program evaluation findings along with performance measurement results in their annual performance reports. One federal evaluation official said his agency had efforts under way to “align” program evaluation and performance measurement through, for example, planning evaluations so that they will provide the performance data needed. But, the official also expressed concern about how to integrate the two types of information. Officials from states that had already begun performance measurement and monitoring said they would like to see the federal government provide more leadership by (1) providing a catalog of performance measures available for use in various program areas and (2) funding and designing impact evaluations to supplement their performance information.
section_title: Conclusions
paragraphs: Seeking to improve government performance and public confidence in government, the Results Act has instituted new requirements for federal agencies to report on their results at the same time that other management reforms may complicate the task of obtaining such information. Comparison of current federal program evaluation resources with the anticipated challenges leads us to several conclusions.
First, federal agencies’ evaluation resources have important roles to play in responding to increased demand for information on program results, but—as currently configured and deployed—they are likely to be challenged to meet these future roles. It is implausible to expect that, by simply conducting more program evaluation studies themselves, these offices can produce data on results across all activities of the federal government. Moreover, some agencies reported that they had reduced their evaluation resources to the point that the remaining staff feel unable to meet their current responsibilities. Lastly, the devolution of some program responsibilities to state and local governments has increased the complexity of the programs they are being asked to evaluate, creating new challenges.
Second, in the future, carefully targeting and reshaping the use of federal evaluation resources and leveraging federal and nonfederal resources show promise for addressing the most important questions about program results. In particular, federal evaluators could assist program managers to develop valid and reliable performance reporting by sharing their expertise through consultation and training. Early agency efforts to meet the Results Act’s requirements found program evaluation expertise helpful in managing the numerous analytical challenges involved, such as clarifying program goals and objectives, developing measures of program outcomes, and collecting and analyzing data. In addition, because performance measures will likely leave some gaps in needed information, strategic planning for future evaluations might strive to fill those gaps by focusing on those questions judged to have the most policy importance. In many programs, performance measures alone are not sufficient to establish program impact or the reasons for observed performance. Program evaluations can also serve as valuable supplements to program performance reporting by addressing policy questions that extend beyond or across program borders, such as the comparative advantage of one policy alternative to another.
Finally, without coordination, it is unlikely that the increasingly diverse activities involved in evaluating an agency’s programs will efficiently supplement each other to meet both program improvement and policymaking information needs. As some agencies devolve some of the evaluations they conducted in the past to program staff or state and local evaluators, they run the risk that, due to differences in evaluation resources and questions, data from several studies conducted independently may not likely be readily aggregated. Thus, in order for such devolution of evaluation responsibility to better provide an overall picture of a national program, those evaluations would have to be coordinated in advance. Similarly, as federal agencies increasingly face common analytic problems, they could probably benefit from cross-agency discussion and collaboration on approaches to those problems.
section_title: Agency Comments
paragraphs: The Director of OMB commented on a draft of this report and generally agreed with our conclusions. OMB noted that other countries are experiencing public sector reforms that include a focus on results and increasing interest in program evaluation. OMB also provided technical comments that we have incorporated as appropriate throughout the text. OMB’s comments are reprinted in appendix II.
We are sending copies of this report to the Chair and Ranking Minority Member of the House Committee on Government Reform and Oversight, the Director of OMB, and other interested parties. We will also make copies available to others on request.
Please contact me or Stephanie Shipman, Assistant Director at (202) 512-7997 if you or your staff have any questions. Major contributors to this report are listed in appendix III.
section_title: Scope and Methodology of Survey
subsections: section_title: Departments and Independent Agencies Surveyed
paragraphs: The 23 federal executive agencies (13 cabinet-level departments and 10 independent agencies) that we surveyed are listed as follows.
These represent 23 of the 24 executive agencies (we excluded the Department of Defense) covered by the Chief Financial Officer’s Act. The 24 represent about 97 percent of the executive branch’s full-time staff and cover over 99 percent of the federal government’s outlay for fiscal year 1996.
subsections: section_title: Identifying Offices Conducting Evaluation Studies
paragraphs: To identify the roles and resources expended on federal program evaluation, we surveyed all offices (or units) in the 23 executive branch departments and independent agencies that we identified as conducting evaluation in fiscal year 1995. We defined evaluation as systematic analysis using objective measures to assess the results or the effects of federal programs, policies, or activities. To identify these evaluation offices, we (1) began with the list of evaluation offices that we surveyed in 1984 (2) added offices based on a review of office titles implying analytical responsibilities and discussions with experts knowledgeable about evaluation studies, and (3) talked with our liaison staff and other officials in the federal departments and agencies to ensure broad yet appropriate survey coverage.
In some instances, the survey was distributed to offices throughout an agency by agency officials, while in other instances we sent the survey directly to named evaluation officials. We attempted to survey as many evaluation offices as possible; however, in some cases, we may not have been told about or directed to all such offices. Therefore, we cannot assume that we have identified all offices that conducted program evaluation studies in fiscal year 1995. Overall, we received about 160 responses, of which 81 were from offices that conducted such studies.
The survey was directed toward results-oriented evaluation studies, such as formal impact studies, assessments of program results, and syntheses or reviews of evaluation studies. We sought to exclude studies that focused solely on assessing client needs, describing program operations or implementation, or assessing fraud, compliance, or efficiency. However, we allowed the individual offices to (1) define “program” since a federal program could be tied to a single budget account, represent a combination of several programs, or involve several state programs and (2) determine whether or not they did this type of study and, if not, they could exempt themselves from completing the survey. We did not verify the accuracy of the responses provided by evaluation units. We also had some information on fiscal year 1996 activities but did not report those results since they were comparable to the fiscal year 1995 results.
section_title: Respondent Reporting
paragraphs: Some respondents were unable to complete different parts of the survey. About one-third of the respondents did not report either the office’s budget, its number of full-time equivalent staff (FTE), cost information about studies, or the sources of data used in the studies. For some questions, respondents were asked to answer in terms of the number of studies conducted, and we used the total number of studies indicated by all respondents to the question as the denominator when computing percents. However, when the level of nonresponse to individual survey questions was above 20 percent or was unclear due to incomplete information on how many studies had been reported on, we used the full complement of 928 studies to provide a conservative estimate.
section_title: Selected Survey Questions
paragraphs: The questions for which we reported results are reproduced on the following pages.
section_title: Comments From the Office of Management and Budget
section_title: Major Contributors
subsections: section_title: General Government Division
section_title: Bibliography
paragraphs: Committee on Governmental Affairs, United States Senate. “Government Performance and Results Act of 1993.” Report No. 103-58, June 16, 1993.
Evaluation Practice. “Past, Present, Future Assessments of the Field of Evaluation.” Entire Issue. M.F. Smith, ed., Vol. 15, #3, Oct. 1994.
Martin, Margaret E., and Miron L. Straf (eds.). Principles and Practices for a Federal Statistical Agency. Washington, D.C.: National Academy Press, 1992.
National Performance Review. “Mission-Driven, Results-Oriented Budgeting.” Accompanying Report of the National Performance Review of the Office of the Vice President, Sept. 1993.
New Directions for Program Evaluation. “Evaluation in the Federal Government: Changes, Trends, and Opportunities.” Entire issue. C.G. Wye and R. Sonnichsen, eds. #55, Fall 1992.
New Directions for Program Evaluation. “Progress and Future Directions in Evaluation: Perspectives on Theory, Practice, and Methods.” Entire issue. Debra Rog and Deborah Fournier, eds. #76, Winter 1997.
Office of Evaluation and Inspections. Practical Evaluation for Public Managers: Getting the Information You Need. Washington, D.C.: Office of Inspector General, Department of Health and Human Services, 1994.
Public Law 103-62, Aug. 3, 1993, “Government Performance and Results Act of 1993.”
Wargo, Michael J. “The Impact of Federal Government Reinvention on Federal Evaluation Activity.” Evaluation Practice, 16(3) (1995), pp. 227-237.
section_title: Related GAO Products
paragraphs: The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans (GAO/GGD-10.1.19, Mar. 1998).
Balancing Flexibility and Accountability: Grant Program Design in Education and Other Areas (GAO/T-GGD/HEHS-98-94, Feb. 11, 1998).
The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).
Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997).
Block Grants: Issues in Designing Accountability Provisions (GAO/AIMD-95-226, Sept. 1995).
Program Evaluation: Improving the Flow of Information to the Congress (GAO/PEMD-95-1, Jan. 30, 1995).
Management Reform: Implementation of the National Performance Review’s Recommendations (GAO/OGC-95-1, Dec. 5, 1994).
Public Health Service: Evaluation Set-Aside Has Not Realized Its Potential to Inform the Congress (GAO/PEMD-93-13, Apr. 1993).
Program Evaluation Issues (GAO/OCG-93-6TR, Dec. 1992). “Improving Program Evaluation in the Executive Branch.” A Discussion Paper by the Program Evaluation and Methodology Division (GAO/PEMD-90-19, May 1990).
Program Evaluation Issues (GAO/OCG-89-8TR, Nov. 1988).
Federal Evaluation: Fewer Units, Reduced Resources, Different Studies from 1980 (GAO/PEMD-87-9, Jan. 23, 1987). (966704/973810)
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.