Congress and Program Evaluation: An Overview of Randomized Controlled Trials (RCTs) and Related Issues
Released: 2006-03-07T00:00:00

Summary:
Program evaluations can play an important role in public policy debates and in oversight of government programs, potentially affecting decisions about program design, operation, and funding. One technique that has received significant recent attention is the randomized controlled trial (RCT). There are also many other types of evaluation, including observational and qualitative designs.
An RCT attempts to estimate a program's impact upon an outcome of interest (e.g., crime rate). An RCT randomly assigns subjects to treatment and control groups, administers an intervention to the treatment group, and afterward measures the average difference between the groups. The quality of an RCT is typically assessed by its internal, external, and construct validity. At the federal level, RCTs have been a subject of interest and some controversy in education policy and the George W. Bush Administration's effort to integrate budgeting and performance using the Program Assessment Rating Tool (PART). In addition, in the 109th Congress, pending legislation provides for RCTs (e.g., Sections 3 and 15 of S. 1934; Section 114 of S. 667 (Senate committee-reported bill); and Section 5 of S. 1129).
Views about the practical capabilities and limitations of RCTs, compared to other evaluation designs, have sometimes been contentious. There is wide consensus that, under certain conditions, well-designed and implemented RCTs provide the most valid estimate of an intervention's impact, and can therefore provide useful information on whether, and the extent to which, an intervention causes favorable impacts for a large group of subjects, on average. However, RCTs are also seen as difficult to design and implement well. There also appears to be less consensus about what proportion of evaluations that are intended to estimate impacts should be RCTs and about the conditions under which RCTs are appropriate. Many observers argue that other types of evaluations are necessary complements to RCTs, or sometimes necessary substitutes for them, and can be used to establish causation, help bolster or undermine an RCT's findings, or in some situations validly estimate impacts. There is increasing consensus that a single study of any type is rarely sufficient to reliably support decision making. Many researchers have therefore embraced systematic reviews, which synthesize many similar or disparate studies.
A number of issues regarding RCTs might arise when Congress considers making program evaluation policy or when actors in the policy process present program evaluations to influence Congress. Should Congress focus on RCTs in these situations, a number of issues might be considered, including an RCT's parameters, capabilities, and limitations. In addition, Congress might examine the types of program evaluations that are necessary, question an evaluation's definitions or assumptions, consider how to appropriately use evaluation information in its learning and decision making, evaluate how much confidence to have in a study, and investigate whether agencies have capacity to properly conduct, interpret, and objectively present evaluations. This report will be updated in the 110th Congress.

subsections: section_title: Introduction
paragraphs: Program evaluations can play an important role in public policy debates and in oversight of government programs, potentially affecting decisions about program design, operation, and funding. Many different techniques of program evaluation can be used and presented with an intention to inform and influence policy makers. One technique that has received significant recent attention in the federal government is the randomized controlled trial (RCT). This report discusses what RCTs are and identifies a number of issues regarding RCTs that might arise when Congress considers making program evaluation policy. For example, in the 109 th Congress, Section 3 of S. 1934 (as introduced) would establish a priority for RCTs when evaluating offender reentry demonstration projects; Section 114 of S. 667 (Senate Finance Committee-reported bill) would require RCTs for demonstration projects for low-income families; and Section 5 of S. 1129 (as introduced) would call for RCTs for projects and policies of multilateral development banks. Issues regarding RCTs could also arise when actors in the policy process present specific program evaluations to Congress (e.g., in the President's budget proposals) to influence Congress's views and decision making. For many reasons, evaluations often merit scrutiny and care in interpretation.
Before discussing RCTs in detail, the report places them in context by discussing (1) questions that program evaluations are typically intended to address, (2) how RCTs relate to other program evaluation methods, and (3) two major roles that Congress often takes with regard to program evaluation. The report next describes the basic attributes of an RCT, major ways to judge an RCT's quality, and diverse views about the practical capabilities and limitations of RCTs as a form of program evaluation. In light of concerns about the reliability of individual studies to support decision making, the report also discusses how RCTs can fit into systematic reviews of many evaluations. The report next highlights two areas where RCTs have garnered recent attention—in education policy and the President's annual budget proposal to Congress. Finally, the report identifies potential issues for Congress that could apply to the highlighted cases, oversight of other policy areas, and pending legislation. Because the vocabulary of program evaluation can be confusing, an appendix provides a glossary with definitions of selected terms.
subsections: section_title: Congress, Program Evaluation, and Policy Making
subsections: section_title: Key Questions about Government Programs and Policies
paragraphs: Citizens, elected officials, civil servants, interest groups, and many other participants in governance of the United States have an interest in the performance and results of government programs and policies. To that end, stakeholders might want answers to many questions about programs and policies. For example, how should public policy problem(s) be defined? Is a program addressing some or all of the problem(s)? How well are federal programs and policies managed? What are they achieving? How can they improve? How are stakeholders affected? What unintended consequences might result? In the future, what activities and policies should the federal government pursue in order to best serve the public? What resources should be devoted to a program or policy?
In addition, stakeholders might want answers to questions about the quality of evaluations that are brought to policy discussions, given that participants in the policy process will not always advertise weaknesses in studies that also happen to support their policy positions. What might those weaknesses be? Stakeholders might also ask how well federal agencies evaluate the programs they lead and administer. For example, what methods are appropriate to assess a given type of program or policy? Given the available quantity and quality of research, what degree of confidence should be placed in findings, to date? Do agencies have sufficient capacity to evaluate their programs? Are they performing the necessary types of evaluation? Do agencies have sufficient independence to credibly evaluate their programs and policies? What role should agencies play in evaluating programs?
At times, many or all of these questions might be of interest to Congress and program stakeholders. All of them will typically be of interest to agency program managers and leaders. Therefore, any of these questions might be potential subjects of congressional oversight or law making.
section_title: Program Evaluation and Informed Policy Making
paragraphs: In response to questions like those posed above, program evaluations can be introduced into policy discussions by actors in the policy-making process. These actors—who include organizations and individuals both inside and outside of government—might be interest groups, think tanks, academics, legislators, state or local governments, the President, federal agencies, or nonpartisan institutions. Many actors bring evaluations to policy discussions on their own initiative, oftentimes to emphasize the results or findings that they interpret to support their positions. Some actors (e.g., federal agencies) might bring evaluations in response to legislative or executive branch requirements. Depending on many circumstances, the evaluations that agencies bring might, or might not, support the policy views of the agency's head or the President.
When actors bring program evaluations into policy discussions, the studies will oftentimes use different approaches, because there are many possible ways to help answer the questions cited previously. The term program evaluation , therefore, has in practice been interpreted in several ways. For example, there is no consensus definition for the term program . In practice, the term has been used to refer to a government policy, activity, project, initiative, law, tax provision, function, or set thereof. Accordingly, this report uses the term program to refer to any of these things, as appropriate, that someone might wish to evaluate. The term evaluation can seem similarly ambiguous. A recent reference work in the program evaluation literature defined evaluation as "an applied inquiry process for collecting and synthesizing evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance, or quality of a program, product, person, policy, proposal, or plan." Perhaps with many of these considerations in mind, Congress defined program evaluation, for purposes of the Government Performance and Results Act of 1993 (GPRA), as "an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives." GPRA requires most executive branch agencies to develop five-year strategic plans, annual performance plans (including goals and performance indicators, among other things), and annual program performance reports. When reporting GPRA to the Senate, the Senate Committee on Governmental Affairs contemplated that not all forms of evaluation and measurement would necessarily be quantifiable, because of the diversity of federal government activities. In sum, program evaluation has been considered in practice, in the scholarly literature, and under GPRA as concerned with investigating both a program's operations and its results. Furthermore, program evaluation has been seen as (1) informing conclusions at particular points in time, and also (2) a cumulative process over time of forming conclusions, as more evaluation information is collected and interpreted.
Program evaluations might help inform policy makers, including Members and committees of Congress in their authorizations, appropriations, and oversight work. However, viewpoints about program evaluations can be contentious, both in policy debates and among expert evaluators. In their interactions with Congress, many actors cite program evaluations as part of the rationale for policy changes. In addition, observers and practitioners sometimes disagree about the practical capabilities and limitations of various program evaluation methods, the quality of an individual evaluation, or how a study's findings should be interpreted and used. Therefore, many observers believe it is important for policy makers, including Members and committees of Congress, to be informed consumers of evaluation information when weighing these considerations and making policy decisions.
section_title: Types of Program Evaluation
paragraphs: Practitioners and theorists categorize different types of program evaluation (sometimes referred to as different designs or methods) in several ways. Unfortunately, these categorizations are not always consistent with each other, and practitioners and theorists do not always use consistent terminology to describe program evaluations. They sometimes use different definitions for the same term, or use different terms as synonyms for the same definition. This report discusses some of these methods, but does not attempt to provide an overall taxonomy for program evaluation types. The sections below provide basic descriptions of RCTs and other methods. A more detailed description and discussion of RCTs is located later in the report.
subsections: section_title: Randomized Controlled Trials (RCTs)
paragraphs: One program evaluation method that has been a subject of recent interest at the federal level, as well as some controversy, is the RCT. As discussed later in this report, an RCT attempts to estimate a program's impact on an outcome of interest . An outcome of interest is something, oftentimes a public policy goal, that one or more stakeholders care about (e.g., unemployment rate, which many actors might like to be lower). An impact is an estimated measurement of how an intervention affected the outcome of interest, compared to what would have happened without the intervention. A simple RCT randomly assigns some subjects to one or more treatment groups (also sometimes called experimental or intervention groups ) and others to a control group . The treatment group participates in the program being evaluated and the control group does not. After the treatment group experiences the intervention, an RCT compares what happens to the two groups by measuring the difference between the two groups on the outcome of interest. This difference is considered an estimate of the program's impact. The terms randomized field trial (RFT) , random assignment design , experimental design , random experiment , and social experiment are sometimes used as synonyms for RCT, and vice versa. However, use of the word field in this context is often intended to imply that an evaluation is being conducted in a more naturalistic setting instead of a laboratory or other artificial environment.
section_title: Many Other Methods
paragraphs: Many other types of program evaluation that are not RCTs can also be conducted in order to address one or more of the questions posed at the beginning of this report. Some of these "other" methods have been called, as a group, observational designs. The term observational design has been used in different ways, but often refers to empirical and qualitative studies of many types that are intended to help explain cause-and-effect relationships, but that do not attempt to approximate an experimental design. Quasi-experimental designs refer to studies that attempt to estimate a treatment's impact on a group of subjects, but, in contrast with RCTs, do not have random assignment to treatment and control groups. Some quasi-experiments are controlled studies (i.e, with a control group and at least one treatment group), but others lack a control group. Some quasi-experiments do not measure the outcome of interest before the treatment takes place. Many observers and practitioners consider quasi-experiments to be a form of observational design, but others put them in their own category. Methods that attempt to estimate an impact are sometimes called impact analysis designs. Qualitative evaluation often refers to judging the effectiveness of a program (e.g., whether it accomplishes its goals) by conducting open-ended interviews, directly observing program implementation and outcomes, reviewing documents, and constructing case studies. As used by different researchers, the term nonexperiment has been used at times to refer specifically to quasi-experiments and other times to anything that is not an RCT. Systematic reviews synthesize the results of many studies, as discussed later in this report. Many other program evaluation methods, including surveys and cost-benefit analyses, are also used to assess programs. Because this report focuses on RCTs and related issues, only some of these "other" methods are discussed further in this report.
section_title: Possible Congressional Roles Concerning Program Evaluation
paragraphs: Congress can assume at least two major roles regarding program evaluation. These roles might be called (1) "making program evaluation policy" and, (2) when presented with one or more program evaluations, "scrutinizing and learning from program evaluations." Each of these broad roles can raise a number of issues for Congress regarding program evaluations generally, as well as RCTs specifically.
subsections: section_title: Making Program Evaluation Policy
paragraphs: First, Congress might make policy regarding how, when, and the extent to which agencies are to conduct, fund, or use program evaluations. For example, Congress might, among other things, establish agencies or offices that have missions to evaluate programs, require that a percentage of a program's funding be devoted to program evaluation activities, appropriate funds for specific evaluations, articulate what questions should be studied, or specify what methods should or must be used. When policy makers consider and make these decisions, two considerations that many observers would likely consider important are for policy makers to be aware of both the practical capabilities and limitations of various program evaluation methods, and also how those capabilities and limitations might be balanced in light of multiple evaluation objectives.
section_title: Scrutinizing and Learning From Program Evaluations
paragraphs: Second, Members of Congress might use specific program evaluations to help inform their thinking, policy making, and oversight of federal policies. In the course of Congress's lawmaking and oversight work, actors inside and outside of government frequently cite program evaluations to justify their policy proposals and recommendations. In these situations, consumers of evaluation information, including Congress, can face challenges of assessing (1) quality and depth of evaluation information, which can be uneven, and (2) the relevance of evaluation information to a policy problem, which can vary. Therefore, should Congress wish to critically assess or scrutinize program evaluations, having insight into how to assess the quality, depth, and relevance of evaluation methods might be helpful.
Program evaluations themselves can help inform policy in several ways. Among other things, they can provide deeper understanding of a policy problem, suggest possible ways to modify or improve a program or policy, provide perspectives on whether goals are being accomplished, reveal consequences that might not have been intended, and inform deliberations regarding the allocation of scarce resources. Nonetheless, observers and stakeholders frequently disagree on the appropriate goals of government activities, which can make evaluations controversial. Furthermore, because "[p]rogram evaluation is a site for the resolution of ethical and democratic dilemmas," any assessment of a program's merit or worth is arguably always made in part through the lens of an observer's priorities, beliefs, values, and ethics. Merit and worth are program evaluation terms that often are defined as the overall intrinsic and extrinsic value, respectively, of a program to individuals and society. Even when there is some consensus on goals, it has sometimes been difficult or impossible to specify with a single number, program evaluation, performance measure, or even group of evaluations and performance measures, how to comprehensively judge an organization's or program's success in accomplishing its mission. Thus, as experience has shown, the concepts of merit and worth are often in the eye of the beholder. Still, evaluations might help clarify what programs are accomplishing, and, when evaluations are done well, help policy makers make informed judgments when reconciling diverse views about policy problems and values.
section_title: Randomized Controlled Trials (RCTs)
subsections: section_title: What are RCTs?
subsections: section_title: RCT Defined
paragraphs: As briefly noted earlier, an RCT is a type of program evaluation that seeks to assess whether a program had an impact for one or more outcomes of interest (e.g., number of weeks a person remains unemployed) compared to what would have happened without the program. An impact is usually calculated for a large sample of subjects as the difference between (1) a measurement of the outcome of interest after an intervention takes place, averaged across subjects who received the treatment, and (2) a measurement of the outcome of interest after the intervention, averaged across subjects who did not receive the treatment. The study randomly assigns the subjects (also called units of analysis ) to one or more treatment groups and also a control group. A treatment group experiences the intervention, and the control group does not. After the intervention, measurement of the outcome of interest for the treatment group provides information on how the intervention might have affected these subjects. The control group, by contrast, is intended to simulate what would have happened to the treatment group subjects if they had not received the intervention. Depending on the policy area studied, an intervention could be, for example, a training regimen for unemployed workers or a new policy to reduce crime. Because the estimated impact is an average across subjects, the impact reflects the weighted average of the subjects who experienced favorable impacts, subjects who did not experience a change, and others who experienced unfavorable impacts.
In theory, random assignment helps ensure that all of the groups in the study are made statistically equivalent at the beginning of the study. If the only important difference in the subsequent experience of each group is the intervention, then differences in the outcome of interest that are observed at the end of the trial can be attributed with greater confidence to the intervention, rather than to initial differences between the groups. Various statistical tools can be used to estimate whether observed differences are likely due to the intervention (i.e., the difference is found to be statistically significant with a small chance of error) or to chance.
The quality of an RCT is often assessed by two criteria: internal validity and external validity . A third criterion, construct validity , is not always discussed, but is also considered important for judging an evaluation's quality.
section_title: Internal Validity
paragraphs: Internal validity is typically defined as the confidence with which one can state that the impact found or implied by a study was caused by the intervention being studied. For example, if an RCT of an aftercare program for juveniles shows that the juveniles who attended a program re-commit crimes (recidivate) at a lower rate than the juveniles who did not attend the program (the control group), an assessment of the study's internal validity would suggest whether this result was due to the aftercare program or whether it might have been due to some other factor. Internal validity is predicated on the methodological rigor of the study and an absence of other factors, unrelated to the program, affecting the outcome of interest for either the treatment or control group differently from the other group. The term also reflects that the better designed and implemented a study is, the more reliable its conclusions about causation will tend to be. From the perspective of internal validity, the methodological rigor of an RCT study can depend on a number of factors, including, but not limited to the following:
how effectively the random assignment of units creates statistically equivalent groups; whether the group of subjects is sufficiently numerous to ensure that an impact large enough to be of interest to stakeholders, if it occurs, will be found statistically significant (the more numerous the units in the group being studied, the better chance there is of detecting a program's potential impact); whether attrition in the treatment and control groups (e.g., subjects dropping out of the study) is comparable with respect to the attributes of subjects; whether factors other than the treatment "contaminate" one group but not the other (e.g., due to problems with delivering the treatment or incomplete environmental controls); whether the behavior of researchers or subjects is affected because they know who is receiving a treatment or not receiving a treatment (ideally, RCTs are double-blind studies, in which neither the subjects nor the researchers know which group gets the treatment, but double-blind studies in social science are uncommon); whether the units being studied comply with the intervention being provided (e.g., did the patient take the medicine being studied?); whether the presence of a randomized evaluation influences the treatment group's experience (e.g., randomization altering the process of selection into the program); whether subjects in the control group can obtain close substitutes for the treatment outside the program; and whether data collection and analysis procedures are reliable.
section_title: External Validity
paragraphs: External validity is typically defined as the extent to which an intervention being studied can be applied to other settings, times, or groups of subjects and be expected to deliver a similar impact on an outcome of interest. Thus, external validity relates to both (1) whether the intervention itself can be replicated with high confidence and (2) whether an intervention will most likely result in a similar impact in other situations or environments, or with other subjects. In practice, some users of the term emphasize the second aspect noted here. The terms generalizability , replicability , and repeatability are sometimes used as synonyms for external validity.
The external validity of a study can depend on a variety of factors. As noted above, one factor is the confidence a person has that an intervention itself can be replicated. For example, if a new curriculum is introduced in a school, it is possible that the school might deviate from the prescribed curriculum in order to accommodate events or student needs unanticipated by the designers of the new curriculum or the study researchers. Unless the deviation were clearly documented, it might be difficult or impossible to replicate the same intervention in other sites. Furthermore, if the deviation affects the study's outcome of interest either positively or negatively compared to what it would have been without the deviation, the study's findings might not be generalizable.
Another major factor relating to external validity, and one of the most frequently cited, is the way in which a study's subjects were selected. For example, the results of a study measuring the impact of a certain curriculum in schools in Boston, New York City, and Philadelphia might not be generalizable to classrooms in small towns in the Midwest due to potential differences among the underlying populations and environments. The results also might not be generalizable to classrooms in other large cities. Among RCTs, the most generalizable are often those that estimate the same intervention's impact in different settings (also known as multi-site RCTs), as well as those which feature samples of subjects with diverse socioeconomic and demographic characteristics. RCTs can also be more generalizable if subjects are randomly selected from a certain population to participate in the RCT, in order to make the subjects more representative of that population. This is not always possible, however, because sometimes subjects can be selected only in a nonrandom way (e.g., if subjects volunteer for the program). Attrition can also affect external validity. Even if attrition among the treatment and control groups is equivalent, if too many people with certain characteristics drop out of a study, a treatment group that was diverse enough to provide some generalizability at the beginning of the study may no longer have as much at the end, even if the sample remains large enough to produce statistically significant results concerning the intervention's impact.
In response to many of these considerations, researchers sometimes carefully describe the intervention, a study's subjects, and the local environment so that other researchers and stakeholders can attempt to assess a study's external validity.
section_title: Construct Validity
paragraphs: A third type of validity that is considered to be important in any type of evaluation, but is not always explicitly discussed, is construct validity . As one reference work explains,
[s]implistically, construct validity is about naming something (a program, an attribute) accurately. For example, when an evaluator measures student achievement, the issue of construct validity entails a judgment about whether the measures or the operationalization of achievement are really measuring student achievement, as opposed to, for example, social capital.
The "construct" in this example is a specific way of measuring student achievement. Thus, one definition of construct validity concerns the extent to which a study actually evaluates the question it is being represented as evaluating. Perhaps unsurprisingly, actors in the policy process will sometimes have different views on appropriate ways to measure student achievement, or more generally, appropriate ways to measure "success" in achieving a program's mission and goals.
Alternatively, when the term construct validity is used in relation to a program rather than a measurement method, it often refers to the extent to which an actual program reflects one's ideas and theories of (1) how the program is supposed to operate, and (2) the causal mechanism through which it is supposed to achieve outcomes. This view of construct validity can be important when attempting to improve a program (e.g., modifying it to better achieve goals) or to understand the circumstances that are necessary for the program to achieve similar results at another time or place, or with different subjects. With insight into the mechanism of causation, it might also be possible to mitigate any unintended consequences.
section_title: Evaluation Quality
paragraphs: Each type of validity is considered important to the overall quality of an RCT. High internal validity helps to ensure that estimated impacts were due to the intervention being studied and not to other factors such as contamination of the experiment (e.g., improper treatment delivery or incomplete or improper environmental controls, when the treatment and control groups experience different events aside from the treatment). High external validity helps to ensure that an intervention could achieve similar results for other subjects, at another time, or in a different setting. High construct validity helps to increase confidence that (1) the outcome of interest actually measures what it is being represented as measuring and (2) the program actually caused an impact in the way that was theorized or intended.
However, it is not necessarily always possible to enjoy the best of all of these worlds in an evaluation. For example, many scholars and practitioners have viewed RCTs as an evaluation design that, although potentially having high internal validity, in certain cases can lack external validity (e.g., if random assignment makes it more difficult to use typical subjects and natural or representative settings). Some have seen RCTs as trading off external validity in order to achieve high internal validity, but others disagree that there is an implied tradeoff. RCTs with low internal validity cannot be used to confidently state that an intervention caused an observed impact, because the evidence they provide may be dubious. RCTs with high internal validity but low external validity may indicate that an intervention somehow made an impact for one population, but not whether the intervention can be replicated and would make an impact in a different population or setting. With regard to construct validity, there is not always consensus that a particular outcome of interest represents the "best" way to evaluate a program. In addition, establishing the mechanisms of causation (e.g., to ensure the intervention caused an impact in the theorized way) can be difficult. Complementary evaluation methods, in addition to an RCT, might be required to do so. Commentators have also suggested other criteria for assessing quality. For example, some have concluded that "[e]ven within [RCTs], quality is an elusive metric," and that in addition to internal validity, "a complete definition of quality also should take into account the trial's external validity and its statistical analysis, as well as, perhaps, its ethical aspects."
section_title: Practical Capabilities and Limitations of RCTs
paragraphs: Claims about RCTs' practical capabilities and limitations, both in comparison with other research designs and in isolation, have at times been controversial. Although there is much regarding RCTs about which many observers agree, certain issues have at times sparked controversy. Sorting out these arguments can be challenging, however, because the terms that observers use to describe RCTs can be difficult to interpret. For example, some observers and practitioners view RCTs as the best way to determine a program's "effectiveness." However, it is not always clear whether these observers are using the term effectiveness as a synonym for impact, merit and worth, or accomplishment of specific, intended goals, as illustrated later in this report in connection with education policy and recent uses of program evaluation during the federal budget process.
Apart from complications stemming from terminology, some observers appear to have advocated that government social and economic activities should be funded only if they can be "proven effective" by RCTs and certain quasi-experiments. Others might dispute such emphasis on RCTs or have a different threshold for what "proven" means.
In addition, some observers appear to see RCTs and "non-RCTs" (i.e., any evaluation design other than an RCT) primarily as competitors or substitutes for each other when judging "effectiveness." For example, they might see value in RCTs and quasi-experiments for their ability to estimate an impact, but less in other evaluation methods that are not designed to estimate an impact. Observers might also see less value in other methods that do attempt to estimate an impact but that are judged to be so unreliable as to have little or no value. As noted in the next section, other observers argue observational and qualitative methods can be appropriate for estimating impacts in various circumstances. At the same time, many observers have seen RCT and non-RCT designs as complements rather than substitutes in some situations. For example, many observers have argued that non-RCT studies, such as in-depth case studies and other observational or qualitative methods, are, among other things, (1) capable of casting doubt on an RCT's findings or causal claims by showing the RCT was contaminated or had poor design or implementation (i.e., revealing poor internal validity); (2) capable of showing a study's inferences are flawed or questionable (e.g., if the measured outcome of interest is judged to not fully reflect the program's goal(s), raising questions of construct validity); (3) essential for establishing the theory and conditions under which an intervention would be expected to make a favorable impact (increasing external validity); and (4) capable of establishing or strongly suggesting causation in certain circumstances (increasing internal validity), even if a study was not intended to estimate an impact. Nonetheless, different observers have at times seen either quantitative or qualitative methods as misguided, and the other methods as preferable or more legitimate.
In light of the issues noted above, among others, several considerations about the practical capabilities and limitations of RCTs are summarized below.
subsections: section_title: RCT Capabilities
paragraphs: There is wide consensus that, under certain conditions, well-designed and implemented RCTs provide the most valid estimate of an intervention's average impact for a large sample of subjects, as measured on an outcome of interest. This is the reason for the often stated claim by some observers, particularly in the medical field, that well-designed and implemented RCTs that are also double-blind are the "gold standard" for making a causal inference about an intervention's impact on an outcome of interest. (Usage of the term "gold standard" in fields other than medicine to describe the value of RCTs, however, has been considerably more contentious.) RCTs have been extensively used for decades in the medical arena as, usually, the third of four phases of the process for helping the Food and Drug Administration (FDA) evaluate drugs, devices, and biological products for approval, and for helping the medical community identify procedures that yield the most favorable health outcomes on selected outcomes of interest. Rigorously estimating an impact is highly valued, because it provides a measurement of the magnitude of a program's impact on one or more outcomes that a stakeholder values and permits comparison among alternative treatments.
In contrast, studies that rely on non-random assignment between treatment and control groups (e.g., quasi-experiments), or no control group at all, can be subject to certain threats to internal validity that undermine one's ability to make a causal inference when estimating an average impact for a large number of subjects. For example, if subjects are not randomly assigned to treatment and control groups, there is greater risk that a prior existing difference between the two groups might be responsible for observed differences on an outcome of interest after an intervention. This threat to validity is often called selection bias . In view of these strengths and advantages, among others, there seems to be some consensus among program evaluation theorists and practitioners in a variety of disciplines (public policy analysis, evaluation, economics, statistics, etc.) that, generally speaking, more RCTs should be performed in social science-related areas, when appropriate as part of a broad portfolio of evaluation strategies and methods.
Nonetheless, there appears to be less consensus, in a variety of disciplines, about what proportion of evaluations intended to estimate impacts should be RCTs (e.g., as opposed to quasi-experiments and other designs); what proportion of evaluations overall should be RCTs, in light of diverse evaluation needs; and the conditions under which RCTs would be most valuable, appropriate, and likely to result in valid findings. For example, many economists and other observers argue that RCTs have an important and often preferred role to play in estimating impacts, compared to quasi-experiments, due to their high potential internal validity. However, they have also argued that certain types of quasi-experimental methods (i.e., those often called econometric methods) are, under certain conditions for each method, capable of validly estimating impacts in ways that come reasonably close to the estimates from experimental methods. They furthermore have argued that quasi-experimental methods can provide useful information if an RCT is judged inappropriate due to factors like research needs, program circumstances, expense, timing requirements, ethical considerations, or other factors. It should also be noted, however, that some academics and practitioners have perceived arguments for RCTs and other quantitatively oriented evaluation methods as attempts to exclude some kinds of qualitative research from being considered "scientific" or being funded or considered in policy-oriented research or debates.
section_title: RCT Limitations
paragraphs: Scholars and practitioners have also qualified what they view as practical capabilities of RCTs, particularly in areas of public policy that closely relate to the social sciences. RCTs are seen as very strong in making cause-effect inferences about impacts for large samples of subjects, if they are designed and implemented well. However, RCTs are often seen as difficult to design and implement well. A number of observers argue that "[t]here is a sizable divergence between the theoretical capabilities of evaluations based on random assignment and the practical results of such evaluations." Some policy areas have been seen as more difficult compared to others for successfully implementing RCTs, leading some observers to disavow the "gold standard" title for RCTs, while still supporting increased use of RCTs as the most credible way to estimate an impact. Some researchers in the medical field have argued that certain types of well-designed and implemented observational studies can yield similar results to RCTs. As with all forms of study, poorly designed or implemented RCTs can yield inaccurate results. In addition, an RCT's capability to support causal inferences does not necessarily hold for determining causality in small numbers of subjects or individual cases, for which other methods are often judged more appropriate. Nor do RCTs necessarily provide an advantage, compared to other evaluation research designs, in generalizing a specific intervention's ability to make an impact to a broader or different population. RCTs have therefore been seen by some observers as often having external validity limitations. Frequently, RCTs rely on support from other evaluation methods for making inferences about external validity.
As discussed in more detail later in this report, RCTs can sometimes be seen as impractical, unethical, requiring too much time, or being too costly compared to other designs that also seek to assess whether a program causes favorable impacts and outcomes. There is wide consensus that RCTs are particularly well suited for answering certain types of questions, but not necessarily other questions, compared to other evaluation research designs. For example, RCTs typically do not assess how and why impacts occur, how a program might be modified to improve program results, or a program's cost-effectiveness. RCTs also typically do not provide a full picture of whether unintended consequences may have resulted from a program or indicate whether a study is using valid measures or concepts for judging a program's success (e.g., assessing a study's or a measure's construct validity). Many of these kinds of questions have been considered to be more appropriately addressed with observational or qualitative designs.
section_title: RCTs in Context: Program Evaluation and Systematic Review
subsections: section_title: Concerns About Single Studies and Study Quality
paragraphs: In a variety of research fields, there appears to be consensus that a single study, no matter how well designed or implemented, is rarely sufficient to reliably support decision making. For example, in health care, where RCTs are often seen as the "gold standard" (for making causal inferences about impacts) and are more widely used than in any other field of study, there is strong reluctance to rely on a single RCT study. According to the Cochrane Collaboration, an international non-profit group founded in 1993 that supports the production and dissemination of RCT information about health care interventions, most single RCTs are seen as "not sufficiently robust against the effects of chance" and often having limited external validity. Moreover, there have been widespread concerns about the quality of individual studies of any design. The Cochrane Collaboration has said that the amount of information about health care, including from individual RCTs, is overwhelming, but that "much of what [information] is available is of poor quality."
In seeking to address questions of how to use single studies and how to judge study quality, two major strategies that researchers have used include (1) classifying study types within "hierarchies of evidence" and (2) conducting systematic reviews.
section_title: Study Quality: A Hierarchy of Evidence?
paragraphs: In recent years, there has been considerable debate on how to define what "quality" should mean when describing evaluations. Some observers in medicine and the social sciences hold the view that RCTs should be placed at the top of a hierarchy, in view of an RCT's potential for high internal validity in estimating an impact. Due to concerns about the varying quality of individual studies, however, some participants in health care evaluation have reoriented their approach. The U.S. Preventive Services Task Force (USPSTF), an independent panel of experts in primary care and prevention that systematically reviews evidence regarding clinical preventive services, offered this assessment:
For some years, the standard approach to evaluating the quality of individual studies was based on a hierarchical grading system of research design in which RCTs received the highest score.... The maturation of critical appraisal techniques has drawn attention to the limitations of this approach, which gives inadequate consideration to how well the study was conducted, a dimension known as internal validity. A well-designed cohort study may be more compelling than an inadequately powered or poorly conducted RCT.
In response to these conclusions, the USPSTF listed a hierarchy of research designs, but as only one among several inputs to evaluating the quality of individual studies. That hierarchy placed designs in the following rankings, largely on the basis of a design's potential internal validity:
RCTs; controlled trials without randomization (also called quasi-experiments, with a treatment group and a control group whose subjects were not assigned randomly); cohort or case-control analytic studies (observational studies in which similar groups serve as control and treatment groups); multiple time series with or without the intervention (uncontrolled experiments which look at the effects of an intervention on units over a significant amount of time); and opinions of respected authorities (based on clinical experience, descriptive studies and case reports, or reports of expert committees).
To determine a study's overall quality, however, the task force also included realized internal validity (including design and implementation aspects) and, in addition, external validity, which the task force considered "on par" in importance with internal validity. Thus, an RCT might rank high in terms of potential internal validity, but it might have experienced implementation problems leading to poorly realized internal validity, or it might have limited external validity. In either case, or to provide assurance against chance, researchers and decision makers often wish to consider other studies, including studies other than RCTs, to inform their thinking and potential decision making. Two additional USPSTF criteria for judging quality included assessments of internal and external validity of all relevant studies for a given research question, and also the extent to which relevant studies or groups of studies linked interventions directly or indirectly to outcomes of interest. These latter efforts relate closely to systematic review.
section_title: Systematic Review in Health Care
paragraphs: In response to concerns about reliance on single studies and study quality, the health care field has broadly embraced systematic review as a method for identifying gaps in knowledge, drawing whatever conclusions are possible about interventions based on available evidence (including impact analyses), and thereby helping to inform decision making about research priorities and provision of health care to patients. Systematic review has been defined as "a form of structure[d] literature review that addresses a question that is formulated to be answered by analysis of evidence, and involves objective means of searching the literature, applying predetermined inclusion and exclusion criteria to this literature, critically appraising the relevant literature, and extraction and synthesis of data from evidence base to formulate findings." This leads some researchers to place a systematic review as an additional category above RCTs at the top of an evidence hierarchy focused solely on potential internal validity. Some researchers also place a meta-analysis (of RCTs or perhaps other intervention studies) at the top of an evidence hierarchy. A meta-analysis is a type of systematic review that uses statistical methods to derive quantitative results from the analysis of multiple sources of quantitative evidence.
However, for various reasons, both conducting and interpreting a systematic review can be challenging and can require caution. A systematic review, like an RCT or other evaluation, might not be comprehensive for all stakeholders, or even necessarily for a single stakeholder, in assessing their evaluation needs. Systematic reviews typically focus on a specific question by looking at a specific outcome of interest, as described in relevant studies or chosen by an evaluator (e.g., in health care, outcomes like mortality, quality of life, clinical events). However, they would not necessarily focus on all important outcomes of interest. This might raise issues in the context of evaluating public policies, because a program's mission might encompass a variety of potential outcomes of interest, or be difficult to represent with one or more outcome measures. Furthermore, different stakeholders might not agree about the relative importance of varying outcome measures or might be interested in different ones. Finally, although systematic reviews typically focus much attention on concerns about internal validity of various studies, judgments about external validity, or generalizability of findings, are often left to readers to assess, based on their implicit or explicit decision, "how applicable the [systematic review's] evidence is to their particular circumstances." Unfortunately, these judgments are often hindered, because many studies provide little information that might assist in assessing external validity.
section_title: Systematic Review in Social Science-Related Areas
paragraphs: Usage of systematic review in health care raises the question of how systematic review might be used in other contexts (e.g., when evaluating government programs of various types, which are often assessed with social science research methods). In program evaluation, systematic reviews have been performed under various names (e.g., evaluation synthesis, integrative review, research synthesis) and in different ways. However, they have been much less common in social science-related areas than in health care. This might be the case, in part, because RCTs and other, non-RCT evaluations have been relatively more scarce in policy areas related to the social sciences, as compared to medicine. For example, over 250,000 RCT studies had reportedly been published in the medical literature as of 2002, but about 11,000 were known in all of the social sciences combined. Other possible historical reasons for a relative lack of systematic review in the social sciences, compared with health care, might include the following: comparatively less funding devoted to evaluation; more technically challenging research settings and problems (e.g., absence of laboratory controls that can make experimental evaluations more difficult to successfully design and implement, increasing the risk that studies might result in evaluation funding being wasted); resistance to using RCTs; disagreements about appropriate ways to evaluate programs; and less interest from policy makers and institutions.
In response to such comparisons, some efforts have been undertaken to increase production of systematic reviews in social science-related areas. For example, a group of social science researchers created the Campbell Collaboration, a non-profit organization that promotes the use of systematic reviews in the social sciences. In defining "evidence," the Campbell Collaboration has focused primarily on RCTs and secondarily on quasi-experiments in order to determine impacts. However, the organization's guidelines also allow implementation studies and qualitative research to be included in a systematic review.
section_title: Recent Attention to Using RCTs in Program Evaluation
paragraphs: The following two subsections briefly illustrate how RCTs have been a subject of attention in two contexts: (1) setting program evaluation policy in one specific policy area (education) and (2) the citation and use of individual studies, or claimed lack thereof, to justify policy and budget proposals to Congress (in this case, as a component of the George W. Bush Administration's Program Assessment Rating Tool). Because this report's purpose is limited to providing an overview of RCTs and related issues, these cases are not analyzed in detail in the report. However, many of the issues identified in this report could be applied to these and other cases.
subsections: section_title: Controversy in Education Policy: A Priority for RCTs?
paragraphs: In January 2005, the U.S. Department of Education (ED) published a "notice of final priority" in the Federal Register . The notice established a department-wide "priority" for the use of specific types of program evaluation, and especially RCTs, when evaluating certain education programs. Under the priority, ED asserted that RCTs were "best for determining project effectiveness," and with some exceptions, would be preferred for funding compared to other evaluation methods. If ED determined an RCT to be infeasible, a quasi-experimental design would receive priority over other designs. The ED priority has provoked controversy in the education policy area and evaluation field generally.
subsections: section_title: Authority Cited for the ED Priority: NCLB and "Scientifically Based Research"
paragraphs: The ED priority was established at the discretion of the Secretary of Education and was not required by law. However, the priority appeared in a broader context of program evaluation-related statutory provisions enacted by Congress. Specifically, ED cited the Elementary and Secondary Education Act of 1965 (ESEA), as reauthorized by the No Child Left Behind Act of 2001 (NCLB; 115 Stat. 1425; P.L. 107-110 ), as the statutory authority for establishing the priority. In its Federal Register notice, ED asserted
[t]he ESEA as reauthorized by the NCLB uses the term scientifically based research more than 100 times in the context of evaluating programs to determine what works in education or ensuring that Federal funds are used to support activities and services that work. This final priority is intended to ensure that appropriate federally funded projects are evaluated using scientifically based research.
Under ESEA as reauthorized by NCLB, scientifically based research is defined as "research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs." The statutory definition also enumerates several kinds of research that are included within the term. The first enumerated item explicitly includes research that employs either observational or experimental methods. In the fourth enumerated item, the definition also includes research that is evaluated using experimental or quasi-experimental designs. Among experimental and quasi-experimental designs, the definition expresses "a preference for random-assignment experiments ..." (Section 9101(37)(B)(iv)). Thus, the statutory definition of scientifically based research does not appear to give higher priority to experimental designs above designs that draw on observation, except when contrasting experimental and quasi-experimental designs versus one another. Apparently in light of these definitions, ED's notice of priority also said
[t]he definition of scientifically based research in section 9201(37) [sic] of NCLB includes other research designs in addition to the random assignment and quasi-experimental designs that are the subject of this priority. However, the Secretary considers random assignment and quasi-experimental designs to be the most rigorous methods to address the question of project effectiveness.
Additional statutory provisions related to program evaluation in education, located within the Education Sciences Reform Act of 2002 (ESRA; 116 Stat. 1940; P.L. 107-279 ), were enacted after NCLB and a year before the ED priority was proposed. The ED priority did not cite ESRA, but ESRA's provisions privilege RCTs in some ways that appear to be related to ED's subsequent actions. ESRA established ED's Institute of Education Sciences (IES) and set forth its functions. ESRA's definition for scientifically based research standards holds that when IES-funded research is intended to "mak[e] claims of causal relationships," the IES research should include only "random assignment experiments" and "other designs (to the extent such designs substantially eliminate plausible competing explanations for the obtained results)." ESRA does not specify what these "other" designs are or what it means to "make claims of causal relationships." Therefore, it appears that a claim of causal relationship need not be restricted to evaluations that seek to estimate an impact. ESRA's definition for scientifically valid education evaluation holds that when IES's National Center for Education Evaluation and Regional Assistance conducts evaluations to estimate the impact of programs, it should "employ experimental designs using random assignment, when feasible, and other research methodologies that allow for the strongest possible causal inferences when random assignment is not feasible." These ESRA definitions hold for funding controlled by IES, not for the entire Education Department.
section_title: Reactions to the Priority
paragraphs: Originally proposed in November 2003, the ED priority generated considerable debate in the evaluation field. The priority did not explicitly define the word effectiveness . As noted earlier in this report, effectiveness is a program evaluation term that has been used in multiple ways (i.e., synonym for impact, goal achievement, or merit and worth). Upon close reading, the priority's usage of the term appeared to indicate the term was probably used in most cases as a synonym for impact . However, the priority's use of the term effectiveness and the phrase what works appeared to be interpreted by many observers to go beyond the definition of impact . In addition, some text in the priority appeared to be interpreted to claim RCTs were best for demonstrating causation, even apart from estimating an impact.
During the proposed priority's month-long comment period, nearly 300 parties sent comments to ED. These comments were summarized and analyzed in the ED January 25, 2005, notice of final priority, which also included statements from then-Secretary of Education Rod Paige regarding where he agreed or disagreed with comments that were submitted. From ED's summary and categorization of the comments, it appears many more comments were critical of the priority than supportive. ED determined that while the comments it received were substantive, the comments did not warrant changes in the priority.
Both prior to and after publication of the ED priority, many observers who supported the priority (as well as some who opposed the ED priority) agreed that more RCTs were needed in education policy, in certain circumstances, in order to estimate impacts of different educational interventions. Many also agreed that RCTs had been unjustifiably de-emphasized in the past compared to other evaluation methods, to greater or lesser extents. Furthermore, many supporters of the ED priority argued the priority was an appropriate change in the education field, because they believed more information about the impacts of educational interventions was needed to help inform practitioners and policy makers, and because they believed ED's research agenda had been previously influenced by hostility to RCTs and similar types of studies. However, many critics of the ED priority argued that RCTs have been oversold in terms of their practical capabilities; that the priority unjustifiably de-emphasized other evaluation designs in terms of their practical capabilities to contribute to understanding of causes, effects, impacts, "effectiveness," and in some cases to making claims of causal relationships (even if some of the designs are not intended to calculate impacts); and that the ED priority would detrimentally affect overall priorities for evaluation.
section_title: Implications and Related Developments
paragraphs: To the extent that evaluations help frame future choices, it appears the way in which the ED priority will be implemented could affect the future course of education programs and policy. At a minimum, the priority has influenced the use of hundreds of millions of research dollars controlled by ED and arguably education policy, as implemented by ED. For example, ED's department-wide strategic planning documents and activities appear to reflect the priority. The ED strategic plan established a goal that 75% of "new research and evaluation projects funded by the Department that address causal questions ... employ randomized experimental designs." The question of what types of research can make causal claims, or make causal claims while substantially eliminating plausible competing explanations, has often been contentious in the evaluation field and the philosophy of science. The ED "performance budget" accompanying the department's FY2007 budget proposal contains a department-wide objective to "encourage the use of scientifically based methods within federal education programs." One performance measure is listed for that objective: "[t]he proportion of school-adopted approaches that have strong evidence of effectiveness compared to programs and interventions without such evidence."
The ED strategic plan also states that "[t]he Department will seek funding for programs that work, and will seek to reform or eliminate programs that do not." The strategic plan does not define what is meant by the word "work," but the word might mean "increases student achievement," in some cases. However, if ED determined that some programs "increase student achievement," but at undesirable cost or with unintended side effects, then presumably ED would not seek funding for those programs. Thus, the word "work" might instead be intended to indicate merit and worth , in some cases.
A month after the priority was originally proposed, ED issued a prominent guidance document "to provide educational practitioners with user-friendly tools to distinguish practices supported by rigorous evidence from those that are not." The ED guidance asserted that evaluation methods other than RCTs and certain quasi-experiments (1) have "no meaningful evidence" to contribute to establishing whether an intervention was "effective" and (2) cannot be considered "scientifically-rigorous evidence" or "rigorous evidence" to support using an educational practice to "improve educational and life outcomes for [children]." The document appears to define "evidence" that would support these decisions to include only estimations of impact. The document also cites NCLB as calling on "educational practitioners to use 'scientifically-based research' to guide their decisions about which interventions to implement," but does not discuss roles many observers argue that other evaluation methods can play in complementing, bolstering, or undermining an RCT's findings.
In addition, the department established a What Works Clearinghouse (WWC) to "evaluate the strength of the evidence of effectiveness of educational interventions ... to help educators and education policymakers incorporate scientifically based research into their educational decisions." In this formulation, it appears the WWC might be using the word effectiveness and the phrase what works as synonyms for "showing favorable impact on an outcome of interest." According to the WWC website, the WWC "was established in 2002 by the U.S. Department of Education's Institute of Education Sciences (IES) to provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in education." WWC study reports state that "neither the What Works Clearinghouse (WWC) nor the U.S. Department of Education endorses any interventions."
The WWC states it includes only certain evaluation designs in its databases—which happen to be only the designs that were listed in the ED priority —because they "provide the strongest evidence of effects." It appears the term effects is used here as a synonym for impacts . The WWC excludes other types of evaluations, because the WWC asserts they are not "outcome evaluations." However, in the latest Government Accountability Office (GAO) pamphlet on definitions of major types of program evaluation, GAO defined outcome evaluation to also include evaluations that focus on unintended effects and that "assess program process to understand how outcomes are produced." These other types of evaluations would appear to be excluded from the WWC and, perhaps, from ED determinations of "what works." In addition, the ED strategic plan states under the department's Objective 1.4 that every ED program will develop a "'what works' guide," to be distributed to program grantees, that "whenever possible ... will be informed by the What Works Clearinghouse." Therefore, it appears that ED and IES determinations of "what works" might be drawn primarily, or only, from studies that include RCTs, certain quasi-experiments, and the exceptions allowed for in the ED priority, and not other kinds of program evaluation.
At the time of this report's writing, the WWC website presents a definition for the term scientifically based research that is at variance with the term's definition under NCLB. As noted previously, the NCLB definition for scientifically based research includes observational studies and experimental studies, expressing a preference for RCTs over quasi-experiments, but not expressing a preference for RCTs over other evaluation research designs, even when questions of causation are being examined. However, the WWC website presents a different definition for the term scientifically based research . Specifically, the WWC website instead presents what appears to be the ESRA definition for scientifically based research standards , which applies only to IES-funded research. The ESRA term's definition for scientifically based research standards holds, as noted earlier, that when IES-funded research is intended to "mak[e] claims of causal relationships," the IES research should include only "random assignment experiments" and "other designs (to the extent such designs substantially eliminate plausible competing explanations for the obtained results)." Although ESRA does not indicate what these "other designs" could be, the WWC appears to include only the designs mentioned in the ED priority in that category. Nevertheless, the ED priority was established for the entire department. At the same time, the WWC website appears to tie WWC to NCLB's term scientifically based research .
In effect, as demonstrated through ED policy and strategic planning documents, the ED priority appears in some respects to be extending ED's interpretation of ESRA definitions and preferences for certain types of evaluations (especially RCTs) beyond IES to the entire Education Department, notwithstanding the apparently differing definitions and preferences expressed in NCLB.
section_title: Assessing Programs in the Budget Process: The PART
subsections: section_title: The Bush Administration's Program Assessment Rating Tool (PART)
paragraphs: In 2004, the Bush Administration elevated RCTs as the preferred way to evaluate federal executive branch "programs" under portions of its Program Assessment Rating Tool (PART) initiative. The PART is a set of questionnaires that the Office of Management and Budget (OMB) developed in 2002 and annually revised thereafter to determine the "overall effectiveness" of programs included in the President's annual budget proposal. Although OMB did not provide an explicit definition for the term overall effectiveness , OMB and the Bush Administration appeared to use the term as at least a partial synonym for merit and worth . The Administration has used the PART, with some controversy, to justify budget proposals and the proposed elimination or reduction of many programs.
OMB first presented the PART to Congress for the FY2004 budget cycle, assessing programs that represented approximately 20% of the federal budget. For succeeding budget cycles, OMB said that cumulative 20% increments of federal programs would be assessed with the PART, in addition to some reassessments of programs previously "PARTed." The Administration subsequently released PART ratings for selected programs along with the President's FY2004, FY2005, FY2006, and FY2007 budget proposals. An additional round of ratings is planned to be released with the President's FY2008 budget proposal, with the final year assessing all remaining executive branch spending and programs. Thereafter, all programs would presumably be assessed or reassessed each year.
Depending on how a PART questionnaire is filled in and evaluated for a program, it will produce a single numerical score in percentage terms between 0% and 100%. This figure determines the program's overall effectiveness rating. Four ratings are possible based on the score: "effective" (a score of 85% to 100%), "moderately effective" (70% to 84%), "adequate" (50% to 69%), and "ineffective" (0% to 49%). OMB characterizes these ratings as qualitative rather than quantitative. A different designation was created, regardless of PART score, for programs that OMB decided "do not have acceptable performance measures or have not yet collected performance data." The designation was called "results not demonstrated." GAO has said "[i]t is important for users of the PART information to interpret the 'results not demonstrated' designation as 'unknown effectiveness' rather than as meaning the program is 'ineffective.'" GAO also found that disagreements between OMB and agencies on appropriate performance measures for certain programs helped lead to the 'results not demonstrated' designation being given to these programs for purposes of the PART.
section_title: Use of the PART
paragraphs: While the Administration has called the PART a management tool, it has also said the PART's overall purpose is to "lay the groundwork" for funding decisions. Furthermore, the Administration also provided guidance to agencies that a program's PART questionnaire should include "good" performance goals that "provide information that helps make budget decisions," but need not include "performance goals to improve the management of the program." In the context of the President's FY2006 budget proposal, which called for terminating or substantially reducing 154 discretionary programs, an OMB official reportedly said "we have to focus more resources on what works, and the PART is the primary tool to make that judgment."
Of 99 discretionary programs proposed by the Administration for termination for FY2006, 48, or nearly half of all proposed terminations, were in the Department of Education. In the Administration's justification document, RCT evaluations were explicitly cited as supporting termination of the ED Even Start program, and the Administration further cited lack of performance information or lack of "rigorous evaluations" to support termination of many of the other ED programs. On December 23, 2005, the White House reportedly sent to "reporters and surrogates" a listing of "terminations accepted in whole or in part" by Congress, which some media posted on their websites. Neither the White House nor OMB posted the document on their websites. According to the OMB document, 17 of the 49 proposed ED terminations were accepted by Congress in whole or in part, including a 56% cut in Even Start. For the President's FY2007 budget proposal, the Administration proposed "141 programs that should be terminated or significantly reduced in size," government-wide. Of these, the Administration proposed to terminate 42 programs within the Department of Education (nearly 30% of the government-wide total proposed for termination or major reduction), "including many that the PART has shown to be ineffective or unable to demonstrate results." The department said in a press release the 42 programs were "proven ineffective." ED also stated in budget justification documents that a requested termination was "consistent with the Department's goal to eliminate support for programs that show limited or no evidence of effectiveness."
section_title: RCTs and the PART
paragraphs: For purposes of the PART, in 2004, OMB elevated RCTs as the preferred way to evaluate a program's "effectiveness" with release of a document entitled What Constitutes Strong Evidence of a Program ' s Effectiveness? This document, intended to provide guidance to agencies on appropriate evaluations, particularly highlighted RCTs as "best" for evaluating "effectiveness." The document was apparently written with the assistance of the Coalition for Evidence-Based Policy (CEBP), an organization sponsored by the Council for Excellence in Government. According to CEBP's website, the PART and OMB's annual guidance to agencies for the PART (in this case, for the FY2006 budget) were "revised," as a result of collaboration between CEBP and OMB, "to endorse randomized controlled trials as the preferred method for measuring program effectiveness, and well-matched quasi-experimental studies as a possible alternative when randomized trials are not feasible." Nevertheless, the OMB document also stated that
RCTs are not suitable for every program and generally can be employed under very specific circumstances. Therefore, agencies often will need to consider alternative evaluation methodologies. In addition, even where it is not possible to demonstrate impact, use of evaluation to assist in the management of programs is extremely important.
In OMB's What Constitutes document, OMB used the term effectiveness in at least two senses: (1) arguably as a partial synonym for merit or worth (consistent with the PART's usage) or (2) referring to demonstrating impact , which OMB defined as "the outcome of a program, which otherwise would not have occurred without the program intervention." These concepts do not necessarily represent the same thing, because merit or worth can be judged by many other factors in addition to impact on a specific outcome of interest. In some cases, the What Constitutes document is clear which definition of effectiveness is being used (e.g., when using the term impact ). However, other instances are less clear or could potentially be interpreted by agencies as treating the definitions impact and merit and worth equivalently. A sampling is reprinted below.
"The [PART] was developed to assess the effectiveness of federal programs and help inform management actions, budget requests, and legislative proposals directed at achieving results" (p. 1). "The revised PART guidance this year underscores the need for agencies to think about the most appropriate type of evaluation to demonstrate the effectiveness of their programs. As such, the guidance points to the [RCT] as an example of the best type of evaluation to demonstrate actual program impact" (p. 1). "Few evaluation methods can be used to measure a program's effectiveness, where effectiveness is understood to mean the impact of the program" (p. 1). "The most significant aspect of program effectiveness is impact —the outcome of the program, which otherwise would not have occurred without the program intervention" (p. 2, italics in original). "[Non-experimental direct analysis studies] often lack rigor and may lead to false conclusions if used to measure program effectiveness, and therefore, should be used in limited situations and only when necessary. Such methods may have use for examining how or why a program is effective, or for providing information that is useful for program management" (p. 3, italics in original). "Well-designed and implemented RCTs are considered the gold standard for evaluating an intervention's effectiveness across many diverse fields of human inquiry, such as medicine, welfare and employment, psychology, and education" (p. 4).
OMB's annual guidance to agencies for the FY2006 budget's PART similarly elevated RCTs as the preferred way to assess effectiveness and impact . The guidance called impact "the most significant aspect of program effectiveness," called RCTs "generally the highest quality, unbiased evaluation to demonstrate the actual impact of the program," and further asserted that "[t]he most definitive data supporting a program's overall effectiveness would be from [an RCT], when appropriate and feasible." The guidance also stated that RCTs are not suitable or feasible for every program, because "[f]ederal programs vary so dramatically." In such situations, the guidance suggested well-designed quasi-experimental studies as another way to assess impact, and other types of evaluations to "help address how or why a program is effective (or ineffective)" (italics in original).
For the FY2007 PART, however, the tenor of OMB's statements in its guidance about RCTs may have changed to some degree. OMB's discussion in the revised guidance largely mirrored that of the FY2006 guidance, but eliminated the description of RCTs as "the highest quality, unbiased evaluation to demonstrate the actual impact" and replaced it with language calling RCTs "particularly well suited to measuring impacts." When RCTs are judged by OMB and agencies to not be feasible or suitable, the guidance exhorts agencies and OMB to consult with in-house or external evaluation experts, and directs them to "supplemental guidance" in the What Constitutes document.
section_title: Judging "Success"
paragraphs: In practice, it appears that OMB's judgments regarding quality and suitability of evaluation designs have sometimes trumped agency judgments and therefore determined what evaluation methods are to be used for the PART, in spite of disagreements between OMB and agencies. Disputes about the proper ways to judge program success have also emerged. For example, in the context of controversy over the Administration's "ineffective" PART rating of the Community Development Block Grant (CDBG) program and FY2006 budget proposal to significantly cut and consolidate 18 community and economic development programs, an article quoting OMB's Deputy Director for Management Clay Johnson III suggested that political views, or at least different views about program goals, might play a role:
Johnson acknowledges that CDBG fails the [PART] test in part because the administration is applying a new definition of success. "We believe the goal of housing programs is not just to build houses, but the economic development that comes with them. So those are the results we want to focus on," Johnson said. "You can say we are imposing our political views on people, or our favored views of the housing world or the CDBG world on people. Well, guilty as charged. It's important to focus on outcomes, not outputs."
Another example might be what OMB has called for purposes of the PART the Vocational Education State Grants program, within ED, which the Administration proposed for termination for the FY2007 budget. This program is the largest budgetary component relating to the Carl D. Perkins Vocational and Technical Education Act of 1998, commonly called Perkins III. For the FY2007 budget's PART, the Administration deemed the program "ineffective," the lowest PART rating. The Administration justified termination by citing a particular study, the National Assessment of Vocational Education (NAVE), stating that the NAVE "found no evidence that high school vocational courses themselves contribute to academic achievement or college enrollment." This perspective appears to consider academic achievement and college enrollment to be the goals of federally supported vocational education. However, the June 2004 NAVE also found that "[t]he short- and medium-term benefits of vocational education are most clear when it comes to its longstanding measure of success—earnings," citing research findings that "students earned almost 2 percent more for each extra high school vocational course they took," extending to varying degrees "to the large group of high school graduates who enroll in postsecondary education and training, to both economically and educationally disadvantaged students, to those with disabilities, and to both men and women." The study authors further observed:
Perkins III and its legislative predecessors have largely focused on improving the prospects for students who take vocational education in high school, a group that has historically been considered low achieving and noncollege-bound. However, students who participate most intensively in vocational progams ... are actually quite diverse... . ... The vocational courses most high school students take improve their later earnings but have no effect on other outcomes that have become central to the mission of secondary education—such as improving academic achievement or college transitions... . Whether the program as currently supported by federal legislation is judged successful depends on which outcomes are most important to policymakers.
The PART assessment of the program released with the FY2007 budget was originally released with the FY2004 budget in February 2003, reflecting data available in 2002, and had not been updated to reflect the June 2004 NAVE. The PART assessment's worksheet provided only brief reference to evidence regarding the impact of vocational education on earnings. Neither the Administration's justification document for terminating the program nor ED's FY2007 Budget Summary mentioned earnings benefits of federally supported vocational education.
Disputes about an existing program's proper goals can raise questions about the construct validity of studies that purport to evaluate the program, regardless of whether the study is an RCT, the PART, or another type of evaluation. How should one measure "success"? What outcome of interest—or outcomes—are the most important ones? Proponents of the PART have viewed favorably the initiative's effort to raise program performance to a more salient place in budget deliberations. Many observers have also seen favorably the PART's transparency, with detailed justifications for the Administration's views available on the Web for consideration by Congress and the public. However, critics and other observers have said the Administration's criteria for evaluating programs sometimes deviated from the programs' purposes, as determined by Congress, or that the Administration and OMB substituted their views about appropriate program goals and measures over those developed by agencies under the statutory framework established by GPRA, which explicitly provides for stakeholder views, including those of Congress.
section_title: Potential Issues for Congress
paragraphs: The previous section of this report illustrated how RCTs have been subjects of prominent attention in two contexts: (1) setting program evaluation policy and (2) citation and use of individual studies, or lack thereof, to justify policy and budget proposals to Congress. In these and potentially other cases, a focus on RCTs might raise multiple issues for Congress. Some relate specifically to RCT studies, including an RCT's structural requirements and constraints. Other issues relate to program evaluation generally, and therefore to RCTs. A number of these issues are identified and analyzed below.
subsections: section_title: Issues When Directing or Scrutinizing RCTs
paragraphs: If Congress wants to focus on RCTs in the context of program evaluation policy (e.g., developing legislation for, or conducting oversight over, a program, agency, or the entire government; or prospectively deciding whether to fund specific evaluations), Congress might consider a number of issues related to the parameters of these studies and prospective risks to their internal and external validity. In addition, issues could arise if Congress wants to interpret or scrutinize individual RCT studies (e.g., when they are presented to Congress in the budget or authorization processes).
subsections: section_title: Considering Study Parameters
paragraphs: When making program evaluation policy, Congress might opt to focus on some key parameters of studies, including random assignment, the cost of an RCT, the length of time that an RCT would take before producing findings, and privacy or ethical considerations.
subsections: section_title: Random Assignment
paragraphs: As discussed earlier, the central attribute of an RCT is the random assignment of subjects to treatment and control groups, which helps a researcher to make inferences that a particular intervention was responsible for an impact and to estimate that impact with reliable statistical tools. In some programs, however, it may not be feasible or cost-effective to randomly assign units to an intervention group and a control group. For example, it is not possible to conduct an RCT on whether a policy regulating the release of chlorofluorocarbons into the environment contributes to overall global warming, because there is only one planet earth to study. Thus, if Congress is considering whether to require certain types of evaluation for a program, or if Congress is asked by an actor in the policy process to change funding for a program due to lack of experimental evidence of program impact, it might be important to question whether random assignment is possible or practical. On the other hand, if a program would appear to allow for an evaluation using random assignment but none is planned, or an alternative is planned (e.g., a quasi-experiment), it might be important to consider whether an RCT evaluation would be more appropriate. For example, with respect to the cases discussed in this report, what programs assessed by the PART are practically suitable for evaluation by an RCT? Under the ED priority, why should quasi-experiments also receive a priority for funding, in addition to RCTs?
section_title: Cost of RCTs
paragraphs: If Congress considers setting evaluation policy or directing specific studies for an agency or program, it might take the likely costs of an evaluation method or study into consideration, to weigh against the study's potential benefits. Large scale multi-site RCTs are typically expensive, especially when the units being studied are not individual persons but rather organizations such as schools or jails. Large multi-site RCTs of K-12 education funded by ED have reportedly cost $10 million to $50 million. In many cases, this level of funding is not available for the study of a federal program, due in part to tight budgetary constraints and the fact that program evaluations frequently must be paid for out of a program's budget. Smaller scale RCTs featuring the random assignment of 100-200 individuals might be relatively inexpensive, reportedly costing from $300,000 to $700,000. Even this cost, however, can often be as much as a studied program's funding level. Quasi-experiments are frequently, but not always, less expensive, but also might bring a different set of potential benefits and risks compared to an RCT.
Even if considerable funding is available for evaluations, pursuing a particular evaluation will leave fewer resources for other evaluation needs that might be judged important. The opportunity cost of pursuing certain evaluations rather than others (i.e., cost associated with opportunities that are foregone by not putting resources to their highest value use) can be high. Nevertheless, because the potential benefits and costs of evaluations can vary widely depending on an agency's or program's portfolio of evaluation needs, weighing these considerations can be difficult. In addition, priorities might change depending on developments in an agency or policy environment. In light of these and other considerations, Congress might weigh the possible benefits of a study against the likely cost in view of the broader portfolio of evaluation needs, mindful of the risk that a study might be poorly designed or implemented or suffer from contamination that reduces confidence in the study's findings. Evaluation designs that do not offer the theoretical advantages of RCTs regarding internal validity might, or might not, be worthwhile alternatives to RCTs, depending on Congress's evaluation objectives in specific situations.
section_title: Length of Time to Yield Findings
paragraphs: RCTs might take a long period of time to yield findings that can inform thinking and policy decisions. For example, an RCT that aims to estimate whether a certain aftercare program reduces the recidivism of juvenile offenders must follow the program's graduates, and the control group, over a multi-year time span. This elongated time window might be problematic if policy decisions need to be made expeditiously. However, the length of time necessary to conduct an RCT (and perhaps other complementary evaluations) might justify waiting to make a policy decision until more evidence becomes available. Furthermore, programs or the external environment might have changed by the time the "old" program was evaluated. How should the possible benefits of evaluations be weighed against risks of evaluation information becoming dated or obsolete? What are the implications for the types and extent of evaluation research to be conducted? Congress could be called upon to consider these issues if it chose to establish program evaluation policy or directing and funding specific evaluations.
section_title: Privacy, Ethics, and Study Oversight
paragraphs: When considering whether to direct the use of RCTs or other evaluations of public programs and policies, Congress might consider whether the agencies conducting them are charged, or should be charged, with ethical duties to protect RCT study participants' privacy, access to programs, and opportunity to give informed consent. In addition, if Congress determined that an agency should be charged with the ethical duties, Congress might consider requiring oversight—by law, regulation, or institutional action—to help ensure that the duties were fulfilled.
Privacy issues may arise in an RCT if, for example, a program evaluation captures information about individual citizens or clients that could be used inside the government for a purpose other than for which it was collected, or released to the public in some form. What, if any, safeguards should be required of those collecting the information? Several privacy protections are currently legally required for agency-conducted program evaluations and RCTs by the Privacy Act (5 U.S.C. § 552a). Among other things, the act sets conditions concerning the disclosure of personally identifiable information, prescribes requirements for the accounting of certain disclosures of the information, requires agencies to specify their authority and purposes for collecting personally identifiable information from an individual, and provides civil and criminal enforcement arrangements. If a program evaluation is funded or directed by an agency but conducted by a non-federal entity (e.g., if a non-federal entity creates and maintains records about program evaluation participants), the Privacy Act's coverage is often stated in contracts.
The issue of access to government programs might arise in an RCT if, for example, the RCT were designed with a control group that was to be denied access to a program as a part of the evaluation. Although access in some cases might not be required by law, its denial raises the question of whether the benefits of testing a program outweigh the burden of denying access to certain prospective subjects. Would it be appropriate to design RCTs for entitlement programs, which guarantee services to clients, with such control groups?
The issue of informed consent might arise in an RCT if it were deemed appropriate to enroll only willing participants. Although informed consent would not always be required by law for many RCTs and other types of program evaluation, its use would guarantee that persons who participate in RCTs fully understand and agree to their participation. Would the benefits of this outweigh burdens of the time and money that must be spent to achieve such a goal?
If Congress chose to ensure that some or all of the above individual protections were implemented in RCTs and other forms of program evaluation, it might consider requiring some form of protection-specific oversight. Although probably not required by law for RCTs, the institutional review of a proposed research trial's protections for human participants is a common requirement for a great deal of research. These reviews generally require that researchers obtain the approval of an institutional board prior to beginning the research. The board checks to make certain that each proposal includes adequate protections for participants' privacy and ensures that the plan to obtain participants' informed consent is sufficient, among other things. Though this type of review may be time consuming and add additional costs to a research project, it can prove beneficial not only by protecting the study's participants, but also by incidentally improving a study's design. Congress and agencies have also instituted other oversight mechanisms for program evaluations, including competitions for grants and peer review of grant applications.
section_title: Scrutinizing, or Prospectively Assessing, Studies' Internal and External Validity
paragraphs: Whether making program evaluation policy or scrutinizing studies, Congress might also focus on issues of study interpretation and implementation (e.g., deciding whether to direct or fund evaluations in light of potential contamination risks and projected external validity, or judging how much confidence to put in the internal and external validity of a study presented during the budget or reauthorization processes).
subsections: section_title: Contamination and Internal Validity
paragraphs: Actors in the policy process will not necessarily advertise any defects in the studies they present to influence Congress. With that in mind, when actors in the policy process present Members or committees of Congress with program evaluations intended to influence a policy decision, Congress might consider the evaluation's realized, and not merely theoretical, internal validity. In addition, when Congress sets program evaluation policy for a given policy area, it might be possible to prospectively consider the probability of a study's successful design, implementation, and corresponding internal validity. A major threat to the internal validity of RCTs and quasi-experiments has been called contamination . It should be noted that other designs that attempt to estimate impacts are subject to additional threats, including selection bias, as noted previously.
To avoid contamination, well-designed and implemented RCTs ideally insulate the treatment and control groups from events that might affect one group during the study differently from the other group in a way that will affect outcomes. Doing so is intended to ensure the only systematic difference in the experience of the two groups is whether or not they received the intended treatment. RCTs also ideally ensure that the intended treatment was administered properly. Because social science research usually does not occur under tightly controlled laboratory conditions, however, it is often difficult to insulate a study from, or control for, unforeseen variables that might systematically affect the treatment and control groups differently.
Three examples might help illustrate the threat of contamination. First, if an experiment is not double-blinded, subjects in the treatment group might be aware of their inclusion in a special program. If this awareness results in psychological effects that are not considered part of the treatment and subjects behave differently, the study's results might be contaminated. Similarly, some subjects in a control group might learn they are not in the treatment group and either decide to avail themselves on their own initiative of alternative treatments that are not associated with the intervention, or resent or undermine the treatment being given to the other group. Second, if a program to curb crime in a city were evaluated with an RCT, certain districts might be the units of analysis. Perpetrators of crimes might move from experimental districts to control districts, contaminating findings for both groups. Third, with regard to treatment delivery, it might be difficult in some studies to ensure that the intended treatment is delivered properly for all subjects or all sites. If some subjects in the control group get the treatment, for example, or if the intended treatment is not delivered properly, inferences about the intended intervention's impact might be contaminated.
In light of these considerations, several questions might be of concern. When Congress is presented with program evaluation findings, for example, what confidence should Congress have that contamination did not degrade a study's internal validity? Does the study adequately address these risks? Also, when setting program evaluation policy, how much confidence should Congress have that studies in certain policy areas or contexts will be able to avoid contamination? What is the track record in a given area? What are the implications for how Congress and agencies should allocate scarce evaluation resources and structure an agency's portfolio of evaluations?
section_title: Generalizability (External Validity)
paragraphs: When Congress is presented with a program evaluation to justify a policy position, the program that was evaluated presumably operated at a specific time and place, and under specific conditions. Without further analysis to gauge an evaluation's external validity, however, it will not always be clear whether the intervention itself or the study's findings can be generalized to other circumstances (e.g., future conditions, other subjects), as noted earlier in this report.
If generalizability were of concern, Congress might in the first place consider whether the intervention itself (i.e., as it was actually implemented) is replicable at a different time or place. If the intervention was highly customized to a particular time or locale, for example, an evaluation's findings might not be generalizable elsewhere unless the intervention were fully replicated in all important respects. Alternatively, if an actual intervention were not clearly documented (e.g., how it operated, whom it served), it might be unclear how to replicate the intervention. In that case, if the intervention were evaluated, it might not be reasonable to expect the evaluation's findings would be repeated elsewhere.
With regard to generalizability of findings (as opposed to the intervention itself), if a single-site RCT finds that an intervention had an impact for a group of subjects in one instance, it might not necessarily follow that it will have a similar impact for other subjects, times, and circumstances. Congress might therefore look for multiple impact analysis studies on the subject. Large scale, multi-site RCTs are often considered more generalizable than single-site RCTs, because they estimate an intervention's impact in a potentially wider array of geographic locations and populations. However, multi-site RCTs are also typically more expensive and difficult to successfully implement compared to single-site RCTs. In addition, complementary studies are often considered necessary to make assessments of generalizability. Complementary studies might include observational or qualitative evaluations, which can potentially be used to better understand an intervention's mechanism of causation, potential unintended consequences, and conditionality (i.e., the conditions that are required for the intervention to work as intended).
If these considerations were a source of concern, Congress might scrutinize an evaluation's findings for external validity to other times, conditions, and subjects. Alternatively, if Congress is setting evaluation policy for an agency or program, or is directing that specific studies occur, Congress might provide direction or guidance regarding the evaluation methods that might be necessary for establishing a program's generalizability to other circumstances.
section_title: Issues When Directing or Scrutinizing Program Evaluations
paragraphs: Congress might also consider issues that apply to program evaluation generally. Because an RCT is one of many types of program evaluation, these issues might be important when Congress (1) considers making program evaluation policy for specific agencies or programs (e.g., what types of evaluations to direct or fund) or (2) scrutinizes individual evaluations, including RCTs, when making policy decisions and conducting oversight. For example, if Congress considers legislation that provides for program evaluation in one or more policy areas, to what extent should RCTs be the focus of these evaluation policies? To what extent should other evaluation methods be the focus? Should multiple methods fit into a broader evaluation framework? The potential issues discussed below raise these and further questions.
subsections: section_title: What Types of Evaluations are Necessary?
paragraphs: Given the nature of a policy area and the diverse needs of stakeholders—including agency program managers, the President, citizens, and notably Congress—what different types of evaluations should be directed, funded, and considered? This report began with a subsection titled " Key Questions about Government Programs and Policies ," which outlined a number of questions that stakeholders often want to be informed about. In response, a wide array of program evaluation techniques have been developed.
Certain evaluation types often address, or help address, multiple kinds of questions. Indeed, many evaluation types are considered complementary to each other. At the same time, the different types of evaluations bring their own sets of practical capabilities and limitations, and experts and practitioners sometimes disagree on the nature and importance of these capabilities and limitations. Because only finite resources are available for evaluation activities and staff, it can be challenging and controversial to determine the appropriate methods to be used to help answer certain stakeholder questions. Furthermore, when scrutinizing evaluations, it can be challenging to discern potential gaps in the perspectives provided by an actor in the policy process. For example, it can be challenging to discern clues that might suggest other complementary evaluation types are needed.
Given these considerations, many issues might be of congressional concern. For example, in Congress's view, how should the executive branch be pursuing evaluations under the PART initiative, which particularly highlighted RCTs? How should Congress oversee and respond to the Administration's efforts to achieve "budget and performance integration" through use of the PART? In the education policy arena, to what extent is ED appropriately implementing the program evaluation aspects of NCLB and ESRA? To what extent is the ED priority, which elevated RCTs and quasi-experiments above other evaluation types, consistent with congressional intent? How is ED implementing the priority? As Congress considers legislation in other policy areas, should Congress provide direction or guidance regarding program evaluation policy or methods? If so, what kinds of studies might agencies, Congress, and outside stakeholders need? When actors in the policy process present evaluations to influence Congress, are the actors presenting the full story, or are there gaps in the presentation? Are the evaluations they present capturing the key questions that need to be answered? Any of these multiple questions might be ripe for attention.
section_title: What Definitions and Assumptions are Being Used?
paragraphs: As noted earlier, many actors in the policy-making process use program evaluations to help justify their policy recommendations and to attempt to persuade Congress to make decisions consistent with their policy objectives. Therefore, many observers have considered it important that policy makers, including Members and committees of Congress, be informed consumers of evaluation information. Unfortunately, however, the vocabulary of program evaluation can sometimes be confusing. For example, many observers and practitioners use the same terms, but with differing definitions for those terms. When someone says a program is "effective," in what sense is the term being used? As noted previously, the term effectiveness might refer to (1) a program's overall merit or worth, (2) the extent to which a program is accomplishing its goals, (3) the program's impact on a particular outcome of interest, or (4) an ambiguous mix of all three prior definitions. These are very different concepts. In any of these senses, determining whether a program is "effective" can hinge upon an observer's views and assumptions about the program's mission, objectives, appropriate outcome(s) of interest, and progress. Thus, should these definitional aspects of program evaluations be of concern, Congress might scrutinize them closely.
Furthermore, assumptions that are implicit in an evaluation might go unstated. Some stakeholders or evaluators might implicitly argue that their preferred way of evaluating a program is "best" and that other methods are comparatively inappropriate or less appropriate. However, there is not always consensus among well-respected experts regarding when certain methods are best or most appropriate. Should these matters be of concern to Congress when they occur, Congress might investigate a number of questions. For example, if one method is claimed to be "best" compared to others, what are the stated and unstated reasons for that opinion? What would other stakeholders and evaluators say? To what extent, if any, might opinions of appropriateness be due to an underlying agenda (e.g., to support policy views) or self-interest (e.g., to get funding for a program or a type of evaluation)? These questions might also be applied to the cases discussed in this report. For example, in justifying the ED priority for RCTs, which claimed RCTs are "best" for determining "effectiveness," which definition for effectiveness is ED using? Is the ED definition consistent with how ED intends to use RCTs? Should ED employ additional or alternative types of evaluation for these intended uses of RCTs? In its strategic planning, budgeting, and operations, is ED using multiple definitions? What definition is being used for overall effectiveness , for purposes of the PART?
section_title: How Should Congress Use Evaluation Information When Considering and Making Policy?
paragraphs: When Congress is presented with evaluation information in the policy process by various actors (e.g., lobbyists, experts, think tanks, academics, or agencies, among others), how should Congress use the information and findings? There is widespread consensus that program evaluations can help policy makers gain insights into policy problems and make better-informed decisions regarding ways to improve government performance, transparency, accountability, and efficiency. Nonetheless, the use of evaluation in the policy-making process can be controversial.
For example, some advocates of performance-based budgeting and evidence-based policy have argued that a program's future funding or existence should be "based" on its "performance" or on "evidence" of its "effectiveness." The terms performance-based budgeting and evidence-based policy , however, do not have consensus definitions, because different actors typically have, among other things, different definitions for what constitutes "performance" and "evidence," conceptions of what it means to "base" decision making on performance or evidence, views about whether a decision should be "based" on past performance or evidence, and views about what other factors should legitimately help drive decision making.
Another fundamental question might be of concern to policy makers and stakeholders. What role should evaluation of past events play in forming future strategies and plans? A prominent argument in favor of using evaluations to shape future strategies is that past performance (by a person, program, agency, etc.) is usually the best predictor of future performance. However, a prominent counter-argument is that focusing primarily on the past can be compared to "driving a car using only the rearview mirror." Although evaluation of past performance is widely considered helpful for informing thinking and decisions, the process of strategic decision making has been found in the social science and management literatures to be legitimately driven by many more factors. These have included, among others, basic and applied research, forecasting and scenario planning, risk assessment, professional judgment from individual and group experience, theoretical extrapolation, intuition (especially when information is incomplete, consensus interpretations of information are lacking, the future is uncertain, or synthesis is necessary), and values. In addition, in spite of efforts to make decisions as rational as possible in the face of uncertainty and limited information, a wide body of social science has found that there are practical limits to rationality in decision making. Should these considerations be of concern to Congress when considering policy questions or conducting oversight, several questions might be asked. For example, in an RCT context, when considering or scrutinizing RCT studies, how should these studies be used by Congress to inform thinking and decision making? How should they be used by agencies and OMB? Are agencies and OMB using them in appropriate ways? What other factors can or should be considered?
section_title: How Much Confidence Should One Have in a Study in Order to Inform One's Thinking and Decisions?
paragraphs: If Congress is presented in a decision-making situation with a single program evaluation, is the evaluation enough to have high confidence in the findings? Program evaluations can provide helpful insight into policy problems and the manner and extent to which federal policies address those problems. Unfortunately, however, one or more program evaluations do not always produce information that is comprehensive, accurate, credible, or unbiased.
For example, a program evaluation can be designed to answer a certain question, but the way someone frames the original evaluation question can influence how a program is ultimately portrayed. Most prominently, this can be the case when setting criteria for "success," such as a program's goals or the preferred outcomes of interest. However, actors in the policy process often have varying views on how to judge a program's or policy's success. It is not always clear, therefore, that a study's research question will be viewed by most observers as covering what should have been covered to validly or comprehensively evaluate a program.
In addition, a program evaluation will not always necessarily be well designed or implemented. In such cases, a study might produce results that are flawed or inaccurate. Even in the best case—if an evaluation is appropriate for the research question being studied, is well designed and implemented, and there is widespread consensus on how to judge "success"—it is still possible that random chance or unforeseen events might result in an evaluation that produces information that is inaccurate or flawed. For example, it is possible that an evaluation might provide a "false positive" or "false negative" result (e.g., the study finds the program successful when actually it was not, or finds a program unsuccessful when it actually was successful). If this situation were the case, the study findings might not reveal it. The subject of data or information quality is also oftentimes a subject of concern when conducting or interpreting program evaluations.
How might Congress cope with these possibilities? How confident must a Member or committee be in evaluation information, including from RCTs, in order to use the information to inform thinking and conclusions about a policy? In response, social science researchers have recommended that consumers of evaluation information be aware of the practical capabilities and limitations of various program evaluation methods and also scrutinize a study's claims of internal, external, and construct validity. They have also suggested looking for multiple studies and, if available, systematic reviews. Other observers have suggested using the resources of GAO or other congressional support agencies to help interpret or validate conclusions and scrutinizing these matters through hearings and oversight. Finally, Congress might consider whether federal agencies have sufficient capacity and independence to conduct, interpret, and objectively present program evaluations to Congress.
section_title: Do Agencies Have Capacity and Independence to Properly Conduct, Interpret, and Objectively Present Program Evaluations?
paragraphs: At times, Congress and other actors have expressed concern over the capacity of agencies to adequately perform certain tasks, including management functions that range from procurement to financial management. One management function that has been frequently cited as a topic of concern is program evaluation. Over a long period of time, GAO has "found limited (and diminishing) resources spent on ... program evaluation" and "reason to be concerned about the capacity of federal agencies to produce evaluations of their programs' effectiveness." Even with recent emphasis on program evaluation under GPRA and the Bush Administration's PART, it is unclear the extent to which agencies have capacity to properly conduct, interpret, or use program evaluations. Many, if not all, of the issues discussed in this report could apply equally to organizations and decision makers within federal agencies.
Should program evaluation capacity in federal agencies be seen as a topic of concern, several questions might be considered by Congress. Given the complex issues and debates involved in the production, interpretation, and use of program evaluations—as well as complex debates about the appropriateness of different evaluation types in certain circumstances—do agencies have capacity to use evaluation information to soundly inform strategic and operational decisions? In addition, do agencies have the capacity to make objective, methodologically sound presentations and interpretations of evaluation information to Congress, including information from RCTs?
Furthermore, do agency program evaluation offices and personnel have the necessary independence from politics (e.g., partisan or institutional) and self-interest to, without undue hindrance, raise potentially uncomfortable issues and surface objective, valid, and reliable findings for consideration by policy makers, including Congress? Should they have this kind of independence?
Finally, and more broadly, Congress might consider issues of evaluation capacity that go beyond federal programs. In the past, Congress has established agencies that focus on evaluation issues in entire policy areas. For example, in 1989, Congress established a new agency within the Department of Health and Human Services to serve as a focal point in health care research. Congress reauthorized the agency, now called the Agency for Healthcare Research and Quality (AHRQ), in 1999. Rather than focus only on evaluating federal programs, AHRQ's statutory mission is to conduct and support research in all aspects of health care, synthesis and dissemination of available scientific evidence for use by multiple stakeholders, and initiatives to advance health care quality. As noted previously in this report, Congress also established IES within ED in 2002. IES has a multi-part mission "to provide national leadership in expanding fundamental knowledge and understanding in education from early childhood though post-secondary study" for many stakeholders, providing them with reliable information about "the condition and progress of education in the United States...," "educational practices that support learning and improve academic achievement and access to educational opportunities for all students," and "the effectiveness of Federal and other education programs" (116 Stat. 1944).
Although there are differences in the missions of these agencies, one aspect that arguably makes them similar is the scope of their research and evaluation work. Specifically, their focus goes beyond federal programs to instead encompass research and evaluations throughout an entire policy area such as "education" or "health care," whether interventions are delivered by the federal government or another entity. Should Congress view program evaluation capacity as an issue for an entire policy area, Congress could move to consider whether establishment of a policy research and evaluation agency might be warranted.
subsections: section_title: Appendix. Glossary of Selected Terms and Concepts
paragraphs: The Vocabulary of Program Evaluation
Unfortunately for consumers of evaluation information, the vocabulary of program evaluation can sometimes be technical and difficult. The field is multi-disciplinary and some concepts are complex. Sometimes it is not always clear in what sense a term is being used and whether the term is being used appropriately. Technical experts and actors in the policy process sometimes use the same terms for different concepts or use different terms for the same concept. Nonetheless, in program evaluations, understanding the meanings of and distinctions between key terms can make a significant difference in how to interpret study findings and limitations and in how to scrutinize evaluations to see if they are being represented objectively and forthrightly.
This appendix draws on the report to briefly define and, if necessary, explain several recurring terms and to briefly identify several definitions for the same term, as appropriate. However, the definitions provided below are illustrative only and do not necessarily indicate what an evaluation's author or what an actor in the policy process intends to communicate. More definitive assessments typically must be made on a case-by-case basis. The footnotes in this report provide written resources that can help with understanding evaluations and the terms they employ, and CRS analysts can provide additional assistance or refer readers to other resources. In each entry, a term that is included elsewhere in the glossary is written in italics the first time it is used.
Selected Terms and Concepts
Construct validity
In practice, there are several definitions of this term: (1) in measuring outcomes, the extent to which a study actually evaluates what it is being represented as evaluating (e.g., does the study's outcome of interest actually measure "student achievement"?); and (2) in relation to a program , the extent to which the actual program reflects one's ideas and theories of how the program is supposed to operate, and the causal mechanism through which it is supposed to achieve outcomes.
Contamination
In an RCT , something aside from the intended treatment that might affect the treatment group or control group differently from the other group in a way that will affect observed outcomes. For example, RCTs should ideally insulate the treatment and control groups from contaminating events in order to ensure that the only difference in the experience of the two groups is whether or not they received the intended treatment. In addition, RCTs should ideally ensure that the treatment was administered properly; otherwise, the treatment might be considered contaminated.
Control group
In an RCT , a group of subjects chosen by random assignment that is comparable to the treatment group but that does not experience the program being studied.
Effect
Depending on usage, something that inevitably follows an antecedent (as a cause or agent); for example, the act of dropping a pen—a cause—is closely followed by a noise—an effect—when the pen strikes the floor. Also used sometimes as a synonym for impact .
Effective (effectiveness)
A term with multiple possible definitions. In practice, it is used as a synonym for impact , merit and worth , or accomplishment of specific, intended goals.
Evaluation
An applied inquiry process for collecting and synthesizing evidence that culminates in conclusions about the state of affairs, value, merit , worth , significance, or quality of a program, product, person, policy, proposal, or plan.
External validity
The extent to which an intervention being studied can be (1) applied to and replicated in other settings, times, or groups of subjects and (2) expected to deliver a similar impact on an outcome of interest . The terms generalizability, replicability, and repeatability are sometimes used as synonyms for external validity. Some usage of this term refers only to one of the two aspects noted here.
Government Performance and Results Act (GPRA) of 1993
A federal law that requires most executive branch agencies to develop five-year strategic plans, annual performance plans (including goals and performance indicators, among other things), and annual program performance reports. In GPRA's legislative history, it was contemplated that not all forms of program evaluation and measurement would necessarily be quantifiable because of the large diversity of federal government activities.
Impact
An estimated measurement of how a program intervention affected the outcome of interest for a large group of subjects, on average, compared to what would have happened without the intervention. For example, if the unemployment rate in a geographic area would have been 6% without an intervention, but was estimated to be 5% because of the intervention, the impact would be a 1% reduction in the unemployment rate (i.e., 6% minus 5% equals an impact of 1%), or, alternatively, a 16.7% reduction in the unemployment rate, if one characterizes the impact as a proportion of the prior unemployment rate. Depending on the chosen outcome of interest, the average impact across all subjects usually reflects the weighted average of the subjects who experienced favorable impacts, subjects who did not experience a change, and others who experienced unfavorable impacts. Some theorists and practitioners use the term effect as a synonym for impact.
Internal validity
In an RCT , the confidence with which one can state that the impact found or inferred by a study was caused by the intervention being studied.
Merit
The overall intrinsic value of a program to individuals. This term is usually paired with the term worth .
Meta-analysis
A type of systematic review that uses statistical methods to derive quantitative results from the analysis of multiple sources of quantitative evidence.
Observational design
A term that has been used in different ways, but that often refers to empirical and qualitative evaluations of many types that are intended to help explain cause-and-effect relationships but do not attempt to approximate an RCT .
Outcome of interest
Something, oftentimes a public policy goal, that one or more stakeholders care about (e.g., unemployment rate, which many actors might like to be lower). There can be many potential outcomes of interest related to a program . Actors in the policy process will not necessarily agree which outcomes are important. Outcomes of government programs need not always be quantitative (e.g., sending humans safely to the moon and back to earth).
Performance measurement (performance measure)
A term that can mean many things but is usually considered to be different from program evaluation . Typically, the term refers to ongoing and periodic monitoring and reporting of program operations or accomplishments (e.g., progress toward quantitative goals) and sometimes also statistical information related to, but not necessarily influenceable by, a program. Occasional synonyms for "measure" are indicator, metric, and target. Sometimes the word "performance" is dropped, especially when stakeholders believe the measure does not necessarily indicate whether the program itself caused changes in favorable or unfavorable directions.
Program
A government policy, activity, project, initiative, law, tax provision, function, or set thereof, that someone might wish to evaluate. In program evaluation , synonyms for program include treatment and intervention.
Program evaluation
Under the Government Performance and Results Act (GPRA) of 1993 , "an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives." Program evaluation has been seen as (1) informing conclusions at particular points in time and also (2) a cumulative process over time of forming conclusions, as more evaluation information is collected and interpreted. Practitioners and theorists categorize different types of program evaluation in several ways. The varying types are sometimes referred to generically as designs or methods. Typical synonyms for this term include evaluation and study.
Qualitative evaluation
A wide variety of evaluation types that judge the effectiveness of a program (e.g., whether it accomplishes its goals) by, among other things, conducting open-ended interviews, directly observing program implementation and outcomes, reviewing documents, and constructing case studies.
Quasi-experimental design
A type of evaluation that attempts to estimate a treatment's impact on an outcome of interest for a group of subjects but, in contrast with RCTs , does not have random assignment to treatment and control groups . Some quasi-experimental designs are controlled studies (i.e, with a control group and at least one treatment group), but others lack a control group. Some quasi-experiments do not measure the outcome of interest before the treatment takes place. Some observers and practitioners consider quasi-experiments to be a form of observational design , but others put them in their own category.
Random assignment
The process of assigning subjects into a control group and one or more treatment groups by random chance.
Random selection
The process of drawing a sample by random chance from a larger population (e.g., to undertake a survey that is intended to be representative of a broader population). This term is different from, and sometimes confused with, random assignment .
Randomized controlled trial (RCT)
In its basic form, an evaluation design that uses random assignment to assign some subjects to a treatment group and also to a control group . The treatment group participates in the program being evaluated and the control group does not. After the treatment group experiences the intervention, an RCT attempts to compare what happens to the two groups, as measured by the resulting difference between the two groups on the outcome of interest , in order to estimate the program's impact . The terms randomized field trial (RFT), random assignment design, experimental design, random experiment, and social experiment are sometimes used as synonyms for RCT, and vice versa. Use of the word "field" in this context is often intended to imply that an evaluation is being conducted in a more naturalistic setting instead of a laboratory or other artificial environment. Double-blind studies are those in which neither the subjects nor the researchers know which group gets the treatment. Single-blind studies are those in which the subjects do not know they are getting the treatment being investigated.
Statistical significance
In the context of an RCT , a finding of statistical significance is typically interpreted as a level of confidence (usually expressed as a probability, e.g., 95%, which is also referred to as "significance at the .05 level") that an estimated impact is not merely the result of random variation. Assuming the RCT suffered from no defects, this finding would indicate that at least some of the measured impact may with substantial confidence (e.g., 95% confidence) be attributed to the treatment as a cause. Stated another way, significance at the .05 level indicates that there is a 1 in 20 chance that the observed difference could have occurred by chance, if the program actually had no impact. However, simply because an estimated impact is found to be statistically significant does not necessarily mean the impact is large or important.
Systematic review
A form of structured literature review that addresses a question that is formulated to be answered by analysis of evidence, and involves objective means of searching the literature, applying predetermined inclusion and exclusion criteria to this literature, critically appraising the relevant literature, and extraction and synthesis of data from the evidence base to formulate findings. Although systematic reviews typically focus much attention on concerns about internal validity of various studies, judgments about external validity , or generalizability of findings, are often left to readers to assess, based on their implicit or explicit decision how applicable the systematic review's evidence is to their particular circumstances. In program evaluation , systematic reviews have been performed under various names (e.g., evaluation synthesis, integrative review, research synthesis), in different ways, and usually in decentralized fashion. Some systematic reviews focus on RCTs (and might include quasi-experiments ), and others include disparate types of studies.
Treatment group
In an RCT , the group of subjects chosen by random assignment that experiences or participates in a program ; also sometimes called an experimental or intervention group.
Unit of analysis
In an RCT , the subjects of the study who are randomly assigned to one or more treatment groups and also a control group . Subjects are typically individual persons but sometimes might be things or organizations like schools, hospitals, or police stations.
Validity
See entries for internal validity , external validity , and construct validity .
Worth
The overall extrinsic value of a program to society. This term is usually paired with the term merit .