Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse
Released: 2010-07-23T00:00:00

Letter

Background

The mission of the WWC is to be a central source of scientific evidence for what works in education. To accomplish this, the WWC reviews existing education research and posts information based on its reviews on the WWC Web site, http://ies.ed.gov/ncee/wwc/.

The types of products currently available on the WWC Web site are described in table 1.

In addition to the Clearinghouse, Education provides other technical assistance and research-related resources to assist states, districts, and schools. Examples of research-related resources include the Regional Educational Laboratories (REL) and the Doing What Works (DWW) Web site (http://dww.ed.gov): Regional Educational Laboratories. IES’s Regional Educational Laboratory Program is a network of 10 laboratories that conduct research and provide policymakers and practitioners with expert advice, training, and technical assistance on how to interpret findings from scientifically valid research.

Doing What Works. Led by Education’s Office of Planning, Evaluation and Policy Development, DWW is a Web-based resource intended to help teachers, schools, districts, states and technical assistance providers implement research-based instructional practice.

Initial Years of the What Works Clearinghouse

In 2002, IES awarded a $27 million 5-year contract to the initial contractors to operate the Clearinghouse. The WWC contractors developed the Clearinghouse’s research review standards with IES and reviewed research related to topic areas considered to be pressing issues in education. One of the goals of the Clearinghouse was to promote informed education decision making through a Web-based dissemination system that featured rigorous reviews of studies on the effectiveness of educational interventions.

The WWC experienced a slow start due in part to the amount of work involved in developing a research review and reporting process that was valid, transparent, and replicable, according to the initial contractors. In developing the research review process, the contractors and IES addressed over 60 technical issues, such as determining what constitutes an acceptable level of participant loss (attrition) from a study and what methods should be in place to accommodate common education research techniques. In addition, initial plans for topic areas and reporting formats were modified. For example, IES decided to drop one planned topic area because IES officials determined it to be too broad. The WWC and IES also spent a substantial amount of time developing and refining a reporting format to communicate research results to a lay audience. As a result, the WWC began releasing reports in 2006. By September 2007, the WWC had released 89 intervention reports, six topic reports, and three practice guides.

WWC Research Review Process for Intervention Reports

The WWC uses a three-step review process to assess the quality of studies and report on what the research indicates about the effectiveness of interventions. The WWC definition of interventions includes programs (such as whole school reform), products (such as a textbook or curriculum), practices (such as mixed-age grouping), or policies (such as class size reduction). The process begins with an initial screening of published and unpublished studies relevant to the intervention being reviewed. Studies are collected from electronic databases, journals, conference proceedings, and nominations solicited from the general public. The studies that pass initial screens are reviewed to determine whether they provide valid evidence of an intervention’s effectiveness. Using these studies, the WWC then synthesizes the evidence about the effectiveness of an intervention and publishes a report describing its findings. The Clearinghouse categorizes interventions as either having positive effects, potentially positive effects, mixed effects, no discernable effects, potentially negative effects, or negative effects (see fig. 1).

The WWC uses evidence standards to assess the strengths and weaknesses of a study’s methodology, such as the type of design it uses, the quality of the study’s data, and the appropriateness of the study’s statistical procedures. Until recently, the WWC accepted two types of study designs—randomized experiments and quasi-experimental studies. Only randomized controlled trials (or randomized experiments) that WWC has determined to be well-designed and well-implemented are considered strong evidence and can receive the highest rating of “meets evidence standards without reservations.” The WWC also considers evidence from quasi-experiments it has determined to be well-designed and well- implemented. The highest rating a study using quasi-experimental design can receive is “meets evidence standards with reservations.” This rating category is intended to inform educators to interpret the study results with caution, as the results may reflect other factors, in addition to the impact of the intervention (see table 2).

IES Oversight and Support of the WWC

The WWC is administered by IES through a contract with a private research organization. IES monitors implementation of the specific tasks detailed in the WWC contract by reviewing an annual work plan and monthly performance and expenditure reports submitted by the contractor. IES tracks implementation of the tasks, completion of performance goals, and adherence to the budget outlined in the contractor work plan. The contractor monitors the work of any subcontractors that it uses to perform services such as research reviews, technological support, and communications support.

IES is also involved in the development and dissemination of WWC products. IES reviews and approves proposed topics for WWC products, product formats, and the research review procedures. It also coordinates a group of independent researchers to peer review WWC products and reviews and approves all WWC products prior to public release. IES required the contractor to develop a communications plan to inform WWC customers about features of the Web site.

WWC Reviews Research in Accordance with Accepted Standards and Has Responded to Recommendations and Criticisms

WWC Follows Accepted Review Standards and Is Improving Its Review Process in Response to a Congressionally Mandated Expert Panel Report

We found that the WWC review process follows generally accepted practices for reviewing research. Specifically, GAO’s November 2009 report reviewing federally supported efforts to identify effective interventions found that the WWC determines whether a study provides credible evidence on effectiveness based on several dimensions, including the quality of the research design, how the study was implemented, and other technical considerations. Our 2009 report also noted that WWC follows a common approach to conducting its reviews, and provides information to help educators understand the body of existing research on specific interventions.

Additionally, a congressionally mandated panel of experts found in October 2008 that WWC’s research review process was based on appropriate methods for judging the strength of the evidence regarding the effectiveness of interventions. For example, the panel agreed that the minimum qualifications a study must meet in order to be reviewed by the WWC are appropriate. The panel also found that WWC’s reporting process is reasonable and that the WWC provides succinct and relevant evidence on the effectiveness of education interventions. While the panel concluded that the WWC’s processes are generally appropriate, the panel made several recommendations to the WWC for continued improvement. The recommendations primarily related to establishing or clarifying procedures, reviewing statistical methods, and documenting the screening process.

The WWC implemented or is considering implementing 14 of the panel’s 17 recommendations. The WWC implemented nine recommendations, in part by modifying some procedures and creating a procedures and standards handbook. For example, in response to the panel’s recommendation that the WWC include a table of study dispositions (e.g., whether studies meet WWC evidence standards) at the front of intervention reports, the WWC is modifying the report template to include a summary table along with the existing listing of dispositions in the reference section. The WWC also addressed panel concerns about technical issues in its review process by making its treatment of study attrition—the rate at which subjects drop out of a study—more consistent across topic areas. The panel noted that the WWC’s practice of determining acceptable attrition levels by topic area led to arbitrary inconsistencies across the topic areas. In response to the panel’s recommendation that the WWC reconsider this practice, the WWC took steps to increase its consistency by developing attrition guidance that applies to all topic areas. (See app. III for a table detailing the recommendations, WWC and IES’s response, and the status of any changes made in response to recommendations.)

In addition, the WWC is considering implementing five other panel recommendations. For example, the panel raised concerns that the WWC does not document some potential conflicts of interest for the studies it reviews. In response to this concern, the WWC is considering tracking and publishing whether studies of a program are funded or conducted by the program’s developers. Further, in response to the panel’s concern that the WWC’s screening process may exclude some eligible studies, the WWC is undertaking an evaluation of the reliability of its screening process. According to IES officials, they will postpone decisions about the recommendations until the newly appointed Commissioner for the WWC is on board and actively involved in the decision making.

WWC Also Responded to Criticism That It Produces Limited and Potentially Misleading Information

Some researchers claim that the WWC presents potentially misleading information by including brief experiments involving small numbers of students when evaluating interventions. As a result, according to critics, educators may accept the WWC’s rating of the intervention’s effectiveness, even though the evidence behind the rating is limited. One researcher suggested the WWC emphasize larger studies that span significant periods of time and set a minimum sample size requirement. According to WWC staff, such changes would exclude valuable research and prevent the WWC from providing educators with research-based information about some interventions. Instead of changing its treatment of sample size and study duration, the WWC began publishing information on the extent of the evidence supporting its findings in 2007. The WWC’s “extent of evidence” rating alerts educators when the WWC effectiveness ratings are based on a small amount of evidence. As figure 2 shows, 76 percent of interventions with positive or potentially positive ratings of effectiveness are based on a small amount of evidence (see fig. 2).

Further, researchers suggested that the WWC presents misleading information by rating interventions based on studies in which measures of student performance closely match the content taught to the intervention group, but not the control group. In such studies, higher test scores among the intervention group may not accurately represent the effectiveness of the intervention more generally. The researchers suggested that the WWC exclude such measures, or at least report on them separately. However, the WWC includes these measures because, according to IES officials, they answer questions about whether different interventions lead to different content knowledge. The WWC agrees that there is a concern regarding the reliability of outcome measures that are overly similar to the intervention, but maintains that WWC procedures attempt to exclude such measures. In addition, in response to researcher concerns that tests created by intervention developers may be biased, the WWC added information to the intervention reports noting whether outcome measures are based on tests created by the developer.

Some researchers and education professionals we interviewed suggested that the WWC produces limited information because its screening criteria are too restrictive—currently screening out about 90 percent of studies initially identified as potentially relevant (see fig. 3). Until recently, the WWC reviewed only two types of study designs—randomized experiments and quasi-experimental studies—and according to critics, this limited the amount and type of information available to educators. For example, staff from one REL noted that educators may not be able to find reviews of the interventions they are using or considering because so few studies meet WWC standards. Staff from another REL told us that if educators cannot find relevant and useful information, they may be discouraged from using evidence-based practices. Staff from a third lab noted that the narrow focus prevents educators from learning from less rigorous but nonetheless useful research, such as case studies describing an intervention’s costs and implementa tion requirements.

The WWC maintains that its screening criteria and study inclusion standards focus on studies that provide strong evidence of an intervention’s effectiveness, and lowering these standards could undermine the validity of the findings reported by the WWC. Although the Clearinghouse screens out most studies, many of its reports have identified interventions with positive effects. Data from the contractor indicate that 58 percent of WWC’s intervention reports identify positive or potentially positive effects of interventions. While the WWC plans to continue using its methodological standards for reviewing randomized and quasi-experimental studies, the Clearinghouse acknowledges that the emphasis on randomized experiments and quasi-experiments can exclude useful information on interventions in certain topic areas, such as special education, that do not lend themselves to these study designs. The WWC created new standards to include additional study designs.

The WWC also introduced practice guides in 2007 in response to criticisms that its intervention reviews exclude too much research and consequently provide limited information to educators. Written by a panel of experts, practice guides include recommendations for educators on various topics, such as reducing high school drop-out rates and reducing behavioral problems in the classroom. Whereas WWC’s intervention reviews are based entirely on studies that meet WWC evidence standards, practice guides also incorporate studies that do not have designs that are eligible for WWC review, or in some cases, are reviewed and do not meet WWC evidence standards, and include the views of experts. To develop recommendations, the practice guide panel reviews available literature about the particular topic and then meets several times to discuss the topic. Through consensus, the panel identifies effective practices based on the evidence. Once the practice guide is developed, it undergoes a quality assurance review by WWC and IES staff and external peer review. The following text box provides an example of practice guide recommendations and the level of evidence supporting them.

Example of Practice Guide Recommendations and Evidence Levels In 2009, the WWC published a practice guide to help educators assist students struggling with reading in the primary grades. The practice guide authors used an early detection and prevention framework known as Response to Intervention. The panel that authored the practice guide consisted of six researchers and one expert in implementation of the Response to Intervention model. Two WWC staff also assisted in the practice guide development.

The panel’s recommendations follow.

WWC’s Output and Costs Increased; However, IES Has Not Developed Adequate Performance Measures Related to Cost or Product Usefulness

WWC Increased Output and Introduced New Products

WWC’s report output increased under the current contract, and its scope expanded to include new products and processes to support production. Under the current contract, the WWC increased its total number of publications from the first contract year to the second contract year and generally kept pace with its increased scope, as specified in the Clearinghouse’s annual plans. For example, the current contract calls for the WWC to increase the number of topic areas and intervention reports. Under the current contract, the WWC added three new topic areas and released 60 intervention reports, including 5 in the new topic areas as of June 2010. In addition, the WWC produces practice guides and quick reviews and increased its production of both of these products between the first and second year of the current contract. Figure 4 shows the production of all three WWC products as of June 30, 2010, the end of the third contract year.

The WWC’s scope of work increased under the current contract with the addition of new products and work processes, as well as responsibilities related to the American Recovery and Reinvestment Act of 2009 (Recovery Act), which provided additional innovation and improvement funding. The WWC is developing three new types of publications and conducts an annual review of Education-sponsored studies for IES’s internal use. Specifically, the WWC is developing research briefs, research perspectives, and practice briefs, which will focus on Education policy priorities. Like practice guides, the new publications will incorporate expert opinion and a broad range of research. Table 3 provides more information on these new initiatives.

IES’s Reviews Have Delayed the Release of Some Reports

While the WWC contractor increased its report production, IES’s review process did not keep pace with output. IES is responsible for administering independent peer reviews of all products and conducting final reviews and approvals before products are released, and has internal time frame estimates used in scheduling and completing such reviews. For example, according to IES planning documents, IES estimates 15 business days for the completion of peer reviews for intervention reports and 6 business days for WWC quick reviews. However, throughout 2009, IES took increasingly more time to schedule and coordinate the completion of peer reviews for some intervention reports and quick reviews. As a result, the release of 20 reports—11 intervention reports and nine quick reviews—was delayed by more than 6 months. For example, in the first quarter of the current contract year (third contract year, 2009 to 2010), IES took an average of over 50 business days to have intervention reports and quick reviews peer reviewed, compared to an average of 7 business days during the first quarter of second contract year (see fig. 5).

These delays in the IES-administered peer review process resulted in significant backlogs of intervention reports and quick reviews awaiting release. For example, as shown in figure 6, reports that entered the peer review process in the first quarter of the second contract year (CY2-Q1) were completed within that quarter. However, the majority of reports entering review the first quarter of the third contract year (CY3-Q1) remained in process for subsequent quarters. While the backlog persisted through the third quarter of the third contract year (CY3-Q3), the number of reports that completed peer review in the third and fourth quarters increased from prior quarters. Figure 6 shows that 11 intervention reports completed peer review in CY3-Q3 and an additional 27 completed peer review in CY3-Q4, compared with 4, 5, and 8 intervention reports in the prior three quarters.

IES attributed these delays to several factors and recently took steps to eliminate the backlog. IES officials told us that delays were, in part, attributable to difficulty in identifying and scheduling independent peer reviewers, vacancies in WWC-related positions at IES, and an increasing amount of research that met WWC standards. For example, IES officials told us identifying and scheduling a sufficient number of qualified, independent peer reviewers had become increasingly difficult because several former peer reviewers were now associated in some way with th WWC and therefore were no longer independent. To reduce the delays and eliminate the backlog, IES recently implemented a new database to help staff track and manage the work of peer reviewers and other WWC-relate tasks. IES officials also told us that they began identifying additional potential peer reviewers using the WWC online registry of researchers. In addition, IES increased a staff member’s responsibilities related to scheduling and coordinating peer reviews. These efforts reduced the amount of time reports remain in the IES peer review process and eliminated the backlog as of June 2010.

In addition to delays in the peer review process, WWC contractors told us that many of their daily decisions need IES approval, and slow responses delayed contractor processes. For example, the contractor needs IES approval on the format and content of the products in development, hindering further work when responses are delayed. IES officials acknowledged that some delays in the approval process occurred during contract year three and told us that this was largely due to staff vacancies that they anticipate filling.

The Cost of the Current WWC Contract Has Increased from the Previous One

WWC’s contracted costs have doubled from about $5.3 million per year under the previous 5-year contract to the current level of about $10.7 million per year. The increase in contracted costs reflects the expand scope—more publications and new products and processes—of the second contract compared to the first. IES’s contract for the WWC includes a variety of tasks that the contractor is responsible for, inc tasks related to report production and product development. Table 4 provides a description of six broad task categories and how they chan between contracts.

IES Has Not Developed Performance Measures Related to Production Costs or Product Usefulness

IES established performance goals, which the WWC met or exceeded; however, these goals do not address production costs or the usefulness of WWC products. IES established WWC-related performance goals in its annual organizational assessment, but Education discontinued the use of these performance measures for fiscal year 2010. In addition, IES established performance goals for its WWC contractor in the contractor award fee plan, which IES uses to determine the amount of performance- based funds awarded to the contractor.

IES measured WWC program performance from fiscal year 2003 to fiscal year 2009, as part of Education’s Organizational Assessment—its departmentwide performance management system. The WWC-related performance goals included in Education’s Organizational Assessment focused on WWC Web site visits and the quantity of publications, both of which were areas of concern as the WWC was getting established. Specifically, these performance goals included increased WWC Web site visits, sustained productivity in the release of intervention reports and quick reviews, and increased practice guide production. The WWC met or exceeded these performance goals annually; however, according to IES officials, these performance goals will not be included in Education’s fiscal year 2010 Organizational Assessment, in part because IES is now satisfied with WWC activity in these areas.

IES has not developed performance measures related to the cost of specific WWC products. IES officials noted that the costs per WWC publication vary greatly depending on the amount of available research relevant to the specifications of a product. For example, intervention reports based on a large number of studies meeting WWC standards take longer and cost more to produce than do reports for which few studies qualify for review. IES has tasked the current WWC contractor to develop ways to streamline production processes and to conduct a cost study, the results of which would improve budget estimates and strengthen IES’s monitoring of the contract. While the contractor has begun this work, IES officials told us that they do not know when cost-related performance measures, such as acceptable cost ranges for each type of product, will be established.

WWC does not currently have a performance measure related to product usefulness. While Web site visits were tracked as a measure of WWC utilization in IES’s Organizational Assessment through fiscal year 2009, this metric did not assess the degree to which WWC products were reaching their target audience and did not provided any information on the extent to which educators find WWC products to be useful. IES’s 2010 budget justification calls for a representative survey of WWC use among education decision makers to be conducted by 2013. However, IES officials told us that they were unsure whether the survey would take place, and IES does not currently have a plan in place to implement this survey.

Education Has Three Primary Ways to Disseminate Information about WWC Products, but Awareness and Use Vary among Target Audiences

Education Has Various Ways to Disseminate Information about WWC Products, but Awareness of the Clearinghouse Is Generally Limited

Education uses the WWC contractor, RELs, and DWW to disseminate information about the Clearinghouse to its target audience, which includes state and school district officials, as well as teachers and principals. In accordance with its contract, the WWC contractor disseminates information about its products electronically and through various events, such as formal presentations at conferences. The Clearinghouse’s electronic dissemination methods include an e-mail listserv, Web-based seminars (webinars), and newsletters. For example, the WWC sends out notices to its e-mail listserv, alerting subscribers of the availability of new products, including intervention reports, practice guides, and quick reviews. WWC staff told us that the webinars cover the same topics as their reports and are a relatively cost-effective way to disseminate information about products and methodology. In addition, WWC staff disseminate information about WWC products at education conferences, such as teacher, principal, and researcher conferences. At these conferences, WWC staff may conduct formal presentations, have an exhibit featuring their products, or both. At conference exhibits, Clearinghouse staff answer questions about their products and provide literature to conference attendees. From July 2009 through June 2010, WWC staff were scheduled to present or have an exhibit at 14 conferences. WWC staff also told us that they work with other groups, such as education, research, and legislative organizations, in order to further disseminate information about WWC products to their members.

In addition, Education disseminates information about WWC products through IES’s 10 RELs, which hold events that may feature information based on practice guides and refer educators to Clearinghouse products. Officials at all 10 RELs told us that they spent time disseminating information about WWC, in part by holding events that bridge research and practice. According to REL officials, these bridge events are attended primarily by school-, district-, and state-level education professionals and provide an opportunity for educators to discuss ways to implement research-based practices. Officials at all 10 RELs told us that bridge events focused on practice guides to some extent, and 7 indicated that WWC practice guides were the primary focus of these events. According to REL officials and WWC staff, these events sometimes included a WWC staff member to discuss methodology and panelists who helped develop the practice guides. RELs also disseminate research from WWC when responding to educator questions or concerns. Officials from 7 of the 10 RELs told us their respective RELs generally use relevant WWC products (practice guides and others) when searching for research-based information to address educator questions.

In addition, Education’s Office of Planning, Evaluation, and Policy Development disseminates information about WWC practice guides on its DWW Web site, which provides an online library of resources designed to help educators implement research-based instructional practice. This Web site uses different formats to present content based primarily on WWC Practice Guides and provides examples of possible ways educators might apply WWC research findings. For instance, to help educators implement the recommendations from the practice guide on dropout prevention, the DWW Web site features slideshows with examples of supportive academic environments and interviews with educators and experts on dropout prevention. In addition, the Web site includes sample materials, such as lesson plan templates, that provide an example of how to implement recommendations. The DWW also includes information on the research behind the recommendations and a link to the WWC Web site and the individual practice guides. According to IES officials, a recent analysis of the DWW Web site traffic showed that 49 state Web sites have links to the DWW Web site, which helps disseminate WWC products further to the education community.

We found that 33 of the 38 states that responded to our survey reported that they had heard of the WWC. Based on our survey results, we estimate that 42 percent of school districts have heard of the WWC and that the percentage is greater for school districts that rely to a very large extent on external sources for information on research-based practices.

School districts identified several sources of information about the Clearinghouse, including conferences and Education (see fig. 8).

While the majority of states have accessed the WWC Web site, we estimate that only 34 percent of school districts have done so. Specifically, among the states that responded to our survey, 33 of 38 states reported that they had accessed the WWC Web site at least once. In addition, 19 of states reported visiting the Web site at least seven times per year. In contrast, an estimated 34 percent of school districts accessed the WWC Web site at least once. Further, we estimate that only 11 percent of school districts visited the Web site at least seven times per year. States and school districts that visited the WWC Web site less than seven times per year most often cited time constraints as the primary reason for their infrequent use.

In addition to the WWC, states and school districts use a variety of other sources of information to identify effective education practices. Most states and school districts use several broad sources of information, such as academic journals, education periodicals, and associations of educators. For example, 37 states reported using academic journals to identify such practices, and we estimate that about 97 percent of school districts used academic journals. Overall, more school districts and states that responded to our survey used the WWC than used other research synthesis organizations.

While the WWC also includes teachers and principals in its target audience, we found that relatively few of the teachers and principals we contacted at education conferences had heard of the WWC. While not a generalizeable sample, we found that out of a total of 391 teachers who completed our questionnaire at four education conferences, only 18 had accessed the WWC Web site. In addition, 341 teachers who had not accessed the WWC Web site told us they had not heard of the Web site. Similarly, among the 208 principals and other school administrators who completed the questionnaire, only 32 had accessed the WWC Web site. Further, 135 principals and other school administrators told us they had not heard of the WWC.

States and School Districts Generally Used the Clearinghouse to a Small or Moderate Extent to Inform Decisions and Used Specific WWC Products to Varying Extents

Based on our survey, most states and school districts that reported accessing the WWC Web site used it to inform decisions on effective education practices—a stated purpose of the WWC—to a small or moderate extent. Specifically, 25 of the 33 states that use the Clearinghouse indicated that they use it to a small or moderate extent to inform their decisions, while 6 reported using it to a large or very large extent. We estimate that 72 percent of school districts that have accessed the Clearinghouse use the WWC to inform education decisions to a small or moderate extent, while only 18 percent use it to a large or very large extent.

States that used the WWC to inform decisions reported that they used the Clearinghouse for various purposes, including informing professional development and curriculum decisions. For example, 25 states reported using the Clearinghouse to inform professional development programs for teachers, and 22 reported using it to inform curriculum decisions. Fewer states used the Clearinghouse to advise districts that were not making adequate yearly progress (AYP) in meeting academic targets or to develop improvement plans for such districts. (Fig. 9 provides a breakdown of the extent to which these states use the Clearinghouse for various purposes.)

In addition, we estimate that among school districts that use the WWC to inform decisions on effective education practices, about 90 percent used it to inform curriculum decisions at least to a small extent, similar to the percentage that used the WWC to inform professional development decisions. However, fewer school districts used it to advise schools that did not meet academic goals or to develop school-level plans to help such schools improve. Figure 10 provides a breakdown of the extent to which nt to which these school districts use the Clearinghouse for various purposes. these school districts use the Clearinghouse for various purposes.

States reported using specific WWC products—intervention reports and practice guides—more than quick reviews. Specifically, of the states that had used the Clearinghouse, 21 reported that they used intervention reports and 20 reported using practice guides, while only 12 reported using quick reviews. States used intervention reports and practice guides to a similar extent to inform education decisions. For example, for each product, six states reported using them to large or very large extent to inform such decisions (see fig. 11).

However, the relative use of specific WWC products was different among school districts. We estimate that among school districts that use the Clearinghouse to inform decisions on effective education practices, more school districts use intervention reports relative to practice guides or quick reviews. Specifically, we estimate that 74 percent of those school districts that use the WWC have used its intervention reports to inform education decisions, while practice guides and quick reviews were each used by about half of such districts. Based on our survey, an estimated 21 percent of school districts that use the WWC have used intervention reports to a large or very large extent, while about 10 percent use t 10 percent use practice guides to a large or very large extent (see fig. 12). practice guides to a large or very large extent (see fig. 12).

The 95 percent confidence interval for this estimate is (5.3, 17).

States and School Districts Would Likely Increase Their Use of the Clearinghouse If the WWC Made Certain Changes

Many states and school districts that had accessed the Clearinghouse reported that they would likely increase their use of the WWC if the Clearinghouse provided a broader array of information. For example, many states and school districts would be likely to increase their use of the Clearinghouse if it reviewed more studies, covered additional topics, or provided more relevant or timely reports. For example, 21 of the 33 states that had used the Clearinghouse reported that they would be somewhat or very likely to use the Clearinghouse more often if it had reviews that were more timely (see fig. 13).

In addition, based on our survey, we estimate that about two thirds of school districts that had accessed the Clearinghouse would likely increase their use if it included reviews of programs or interventions being used or considered in their school district. An estimated 50 percent of school districts would likely increase their use of the Clearinghouse if it had reviews that were more timely (see fig. 14).

Conclusions

In 2007, Education substantially increased its financial investment in the WWC, and the Clearinghouse is significantly expanding its scope in an effort to better serve its target audience. Some of the new products aim to be more responsive to educators and education decision makers by providing timely information about evidence-based practices relevant to pressing needs. Such information could help states and districts identify strategies as they implement educational reform efforts—such as reforming low-performing schools or improving professional development—under ESEA and the Recovery Act. For example, WWC research perspectives, still in development, are intended to help education decision makers as they address challenges related to spending Recovery Act funds. However, the development of these products and the release of other products were delayed, in part, by a substantial backlog in IES’s review and approval processes. These delays hindered the timely release of several publications, and some products were released months after they were completed by the contractor. While IES recently eliminated the backlog, educators need to be able to rely on the Clearinghouse for timely and relevant information. According to our survey, many states and school districts reported that they would likely increase their use of the Clearinghouse if it released information more quickly.

While IES has increased annual report production, IES has not established reasonable production cost ranges or specific cost-related performance measures related to each product type. Without acceptable per product cost ranges, it is difficult for IES to assess the reasonableness of costs associated with certain products, even as IES takes steps to streamline production. IES’s current study on costs may help IES establish acceptable cost ranges that could inform IES’s performance measurements related to the WWC. In addition, such information could inform cost comparisons between the WWC and other research evaluation organizations or provide baselines for future contractor work.

In addition, IES has not established meaningful performance measures related to product usefulness. Until fiscal year 2010, IES tracked visits to its Web site and annual report production as a way to measure the productivity of the Clearinghouse. While these measures were important to accurately track the WWC’s initial growth, they did not evaluate the degree to which the products were meeting the needs of educators. Specifically, IES currently does not have a way to gauge user satisfaction with WWC products, which is a common practice when developing and providing new products. Further, while IES currently incorporates some feedback from the WWC Web site users, to inform future topic areas, it does not systematically gauge its target audience’s major areas of interest or concern—such as gathering information on interventions currently being used or considered in specific school districts or states. IES decides how to spread its limited resources across the various product types without directly measuring the extent to which educators use the WWC or how useful they find the various products to be. Measuring the use and usefulness of its products could help IES continue to improve content, develop products, and respond to the needs of educators and policymakers.

While some educators and policymakers find WWC products useful, many other educators are not familiar with the Clearinghouse. IES has spent a substantial amount of money, time, and effort producing various summaries of evidence-based practices, which cover both specific education interventions and general practices. This investment in the WWC was made in order to inform education professionals at all levels— from classroom teachers to policymakers—as they make decisions on how best to educate the nation’s children. Improved dissemination of WWC products could increase awareness and use of the WWC. Increased use of the Clearinghouse could help education professionals identify and implement effective educational interventions and practices, and potentially lead to increased student achievement.

Recommendations for Executive Action

We are making the following four recommendations based on our review.

To consistently release WWC products in a timely manner, we recommend the Secretary of Education direct IES to develop and implement strategies that help avoid future backlogs and ensure that IES’s review and approval processes keep pace with increased contractor production. Strategies could include shifting IES resources to ensure sufficient staff time for managing the peer review process and streamlining its approval processes.

To better track the costs and usefulness of the WWC, we recommend that the Secretary of Education direct IES to incorporate findings from its cost studies to develop performance measures related to costs, such as identifying a range of acceptable costs per product and using that information to monitor contractor spending; and develop performance measures related to product usefulness and periodically assess whether WWC products are meeting the needs of target audiences by gathering information on product usefulness in the proposed survey or through other means.

To reach more members of the target audience, we recommend the Secretary of Education direct IES to assess and improve its dissemination efforts to promote greater awareness and use of the WWC, for example, by developing a way to inform school districts of new products or encouraging educator professional development programs to focus on research-based practices such as those discussed in practice guides.

Agency Comments and Our Evaluation

We provided a draft of this report to the U.S. Department of Education for review and comment. Education officials provided written comments on a draft of this report, which are reproduced in appendix IV. Education also provided technical comments, which we incorporated into the report as appropriate.

Education generally agreed with our recommendations. Specifically, Education agreed to our recommendations on consistently releasing WWC products in a timely manner and assessing and improving its dissemination efforts. In its response to our recommendation on tracking the cost and usefulness of the WWC and its products, Education noted that IES has taken some steps that address the recommendation. With regard to costs, Education stated that it intends to incorporate the results of current cost studies into future work plans and monitoring efforts. We continue to recommend that these results be used to inform performance measures related to costs for future operations. With regard to tracking the usefulness of the WWC, Education noted that it uses a variety of tools to gather consumer input, such as a Help Desk and online voting for future report topics. While such feedback provides some information to the WWC, it relies on existing users and reflects the views of those users who provide feedback, rather than those of the broader population. However, as shown in our survey, only an estimated 34 percent of school districts have accessed the WWC Web site at least once—and fewer have used the Web site frequently. Education also noted that it would include a customer satisfaction survey in IES’s review of its own performance, but whether the survey would be directed at current Clearinghouse customers or a broader audience, or whether the survey would identify how useful various WWC products are and how the WWC can be improved is unclear. More nationally representative information could help IES prioritize topics for intervention reports and practice guides and inform budget priorities.

We are sending copies of this report to the appropriate congressional committees, the Secretary of the U.S. Department of Education, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions regarding this report, please contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

To address all three objectives, we interviewed officials from the Institute of Education Sciences (IES), What Works Clearinghouse (WWC or Clearinghouse) contractors, and representatives from various educational organizations. To assess the research review process used by the IES’s WWC, we reviewed WWC standards and procedures, reviewed an expert panel report that assessed the validity of the WWC review process, and collected information about the extent to which the WWC has implemented the panel’s recommendations. To determine how performance and costs changed over time, we analyzed the costs and productivity of the two WWC contractors. To obtain information about the usefulness of WWC products, we conducted a Web-based survey of all state education agencies and a nationally representative sample of school districts. We also collected information about the usefulness of the WWC from teachers and principals at four education conferences. We conducted our work from September 2009 through July 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Assessment of WWC Research Review Process

GAO previously assessed the procedures and criteria used by the WWC by reviewing documents and interviewing IES officials and WWC contractors. We reviewed WWC standards and procedures and examined the degree of consistency of these standards and procedures across education topic areas. We also reviewed the findings and recommendations from an expert panel report that assessed the validity of the WWC review process. We obtained information from IES officials and WWC contractors on the extent to which the WWC has implemented the panel’s recommendations. Further, we identified other concerns about the WWC review process through a literature review and interviews with researchers, and we interviewed IES officials and WWC contractors to assess the extent to which the Clearinghouse has addressed these concerns. We also examined the degree to which the WWC’s review process is similar to that used by other entities engaged in systematic research review efforts.

Performance and Cost Data Analyses

To determine how performance and costs changed over time, we analyzed the costs and productivity of the two WWC contractors. We reviewed budget data and product release dates to analyze cost and productivity trends of the WWC. To examine performance, we interviewed the two contractors, as well as IES officials, and compared IES’s performance measures and goals to actual outcomes.

We assessed the reliability of the WWC performance and cost data by (1) reviewing existing information about the data and the system that produced them and (2) interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report.

Survey of States and School Districts

To determine how WWC products are disseminated, we interviewed officials from IES and all 10 RELs, as well as WWC contractors. To determine how useful education professionals find WWC products to be, we designed and administered a Web-based survey of state education agencies in the 50 states and the District of Columbia and a nationally representative sample of local educational agencies (LEA). Specifically, the survey asked officials about (1) their general sources of information on effective educational practices, (2) the extent to which they use WWC products to inform curriculum decisions (including questions on specific intervention reports and practice guides), (3) how useful the officials find the information in the WWC, (4) the likelihood they would increase their usage if certain changes were made to the WWC Web site, and (5) the extent to which the officials use the Doing What Works and Best Evidence Encyclopedia Web sites to inform curriculum decisions and how useful the officials find these other information sources to be. We reproduce the questions we used in our analysis in figure 15. The survey was administered from February 18, 2010 to April 14, 2010.

To determine how the WWC was being used at the state level, we surveyed the state Secretary, Commissioner, or Superintendent of Education in the 50 states and the District of Columbia. Out of the 51 state officials surveyed, 38 responded to the survey.

To determine how the WWC was being used at the school district level, we surveyed a nationally representative sample of school districts across the country. We selected a stratified random sample of 625 LEAs from the population of 17,620 LEAs included in our sample frame of data obtained from the Common Core of Data for the 2007-08 school year. A total of 454 LEAs responded, resulting in a final response rate of 74 percent. Because we surveyed a sample of LEAs, survey results for the district are estimates of a population of LEAs and thus are subject to sampling errors that are associated with samples of this size and type. Our sample is only one of a large number of samples that we might have drawn. As each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., plus or minus 10 percentage points). We excluded 12 of the sampled LEAs for various reasons—6 were closed, 3 did not administer any schools, 2 managed schools in a correctional facility, and 1 was a private school—and therefore were considered out of scope. All estimates produced from the sample and presented in this report are representative of the in-scope population.

The practical difficulties of conducting any survey may introduce nonsampling errors, such as difficulties interpreting a particular question, which can introduce unwanted variability into the survey results. We took steps to minimize nonsampling errors by pretesting the questionnaire over the phone with officials from two school districts and one state department of education in November and December 2009. We conducted pretests to verify that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on officials, and (4) the questionnaire was comprehensive and unbiased. An independent reviewer within GAO also reviewed a draft of the questionnaire prior to its administration. We made revisions to the questionnaire based on feedback from the pretests and independent review before administering the survey.

The survey-related data used in this report is based on the state and school district responses to the survey questions.

The following tables contain the estimates and associated confidence intervals for the data displayed in figures 8, 10, 12, and 14.

Information from Teachers, Principals, and Researchers

In addition to interviews with teacher, principal, and research organizations, we obtained information about the usefulness of the WWC by administering a questionnaire at four conferences of teachers and principals. Table 9 provides more information about the conferences we attended.

We selected these conferences because they were relevant to segments of the WWC’s target population that we were not reaching through our survey and they were held at times that coincided with our report time frames. At each of these conferences, conference organizers agreed to have GAO have a table either inside the exhibit hall or just outside it. The questionnaires included questions on awareness and use of WWC— including use of specific products and use of other information sources to identify effective educational practices. For those who had not used the WWC, we also asked them to specify the reason they had not used it. The information gathered through the questionnaires is not generalizable and does not represent the views of teachers and principals nationwide.

Appendix II: Other Sources of Information Districts Use To Identify Effective Education Practices

Estimated percent of district officials that find this source Very Useful

Estimated percent of district officials find this source not at all useful 80.6 (76.46, 84.79) 77.4 (72.85, 81.84)

16.6 (12.73, 20.53) 14.9 (11.30, 19.13)

2.3 (0.90, 4.67) 6.2 (3.79, 9.58)

0.5 (0.06, 1.67) 1.5 (0.48, 3.55)

77.3 (72.93, 81.74) 74.5 (69.97, 79.08) 70.9 (66.00, 75.72) 65 (59.94, 70.04)

16.7 (12.88, 20.58) 21 (16.74, 25.17) 23 (18.50, 27.50) 23.1 (18.59, 27.57)

4.2 (2.28, 7.14) 3.2 (1.53, 5.84) 3.2 (1.48, 6.02) 6.3 (3.83, 9.66)

1.7 (0.60, 3.73) 1.3 (0.42, 3.05) 2.9 (1.34, 5.49) 5.6 (3.39, 8.72)

64.3 (59.22, 69.41) 54.4 (49.10, 59.63)

27.7 (22.90, 32.45) 30.6 (25.62, 35.53)

4.7 (2.59, 7.81) 7.9 (5.12, 11.59)

3.3 (1.64, 5.85) 7.1 (4.58, 10.52)

51.7 (46.48, 56.97) 50.3 (45.11, 55.49)

39.9 (34.67, 45.06) 35.4 (30.25, 40.45)

6.2 (3.82, 9.53) 9.3 (6.30, 13.16)

2.2 (0.95, 4.19) 5 (2.94, 7.98)

45.9 (40.66, 51.03)

31.3 (26.32, 36.18)

11.5 (8.18, 15.59)

11.4 (8.16, 15.36)

39.8 (34.68, 45.01)

37.8 (32.64, 42.88)

15.79 (11.87, 20.10)

6.7 (4.33, 9.91)

17.9 (13.99, 21.83) 24.4 (20.07 28.76 22.8 (18.40, 27.10) 21.7 (17.63, 25.83) 10.1 (7.34, 13.36) 9.7 (6.74, 13.29) (29.65, 39.77) 14.8 (11.20, 19.09) 13.8 (10.42, 17.68) 43.9 (38.58, 49.11) 14 (10.48, 18.09) 14.5 (10.93, 18.77) (19.41, 28.67) 16.1 (12.25 20.57) 14.2 (10.52, 18.50) 17 (13.06, 21.60) 18.9 (14.54, 23.25) 16.73 (12.84, 21.25) (18.82, 27.86) 44.7 (39.37 49.96) 49.3 (44.03, 54.62) 17.4 (13.22, 21.59) 57.1 (51.77, 62.40) 59.1 (53.84, 64.33)

Appendix III: IES and WWC Response to Expert Panel Recommendations

lES is considering an appropriate mechanism and time for conducting a complete review of the WWC. lES believes that the first 2 years of the current contract necessitated a tremendous development effort to transfer the infrastructure of the Clearinghouse in year one from one contractor to another, and in year two, to complete reviews in a consistent manner that began under the original contract. Now that the Clearinghouse is more clearly in the production phase, this may be the appropriate time to plan for a complete review.

lES is currently considering having the WWC develop a standard for assessing crossover compliance, following the process recently used to revise its attrition standard.

Currently, the WWC documents crossover reported in studies. Principal investigators have discretion to use this information to determine whether a study represents a reasonable test of the intervention. Evidence of crossover and assignment noncompliance is documented in the intervention report and its appendix table A.1. Readers can use that information to assess the findings. lES agrees there is value in adjusting intent-to-treat effects for compliance, but believes this adjustment is inconsistent with its goal of having the WWC be transparent in how it reports findings. Making its own estimates to account for compliance will lead to differences between what the WWC reports and what is found in publicly available literature.

Currently the WWC does adjust for clustering when authors report their findings incorrectly. However, the purpose of the clustering adjustment is to correct for an analytic problem in the methods authors use to estimate variances, which generally causes them to overstate the precision of their findings. In contrast, adjusting for compliance will yield an alternate estimate of effects that may differ from the one reported by the study.

Though not based on a standard, WWC practice is for reviewers to document the counterfactual in study review guides and in intervention reports (the information is reported in appendix table A.1). Reviewers routinely send author queries for this information, if it is not provided in the study.

Considered but not planning to implement lES has asked the WWC to assess how other review organizations report counterfactual information and the utility of incorporating this information into its reports. IES officials are also considering an alternative approach that would code information about the counterfactual in a study into the study database, which then would generate summary tables that would report results for studies that have similar counterfactuals. This approach has downsides as well, since the set of counterfactuals could be quite varied and many assumptions would have to be made to group counterfactuals together. We are therefore proceeding cautiously in making any changes to current WWC practice.

WWC staff consulted with Dr. Schochet to investigate the possibility of revising the multiple comparison standards.

Dr. Schochet indicated that his report focused on issues related to multiple comparisons within single studies. It did not tackle issues related to multiple comparisons issues that may arise when synthesizing evidence for a set of studies. WWC procedures are consistent with his report for handling multiple comparisons within a study.

At the time of the National Board of Education Sciences Expert Panel’s data collection, the WWC was already reviewing its attrition standards. The WWC released new attrition standards in December 2008 in the Procedures and Standards Handbook. The new standards requires a principal investigator in a topic area to choose one of two well-specified attrition boundaries, and the standards include guidance on how to choose between the boundaries based on the nature of research in the topic area.

The attrition discussion is in the WWC Procedures and Standards Handbook (Version 2.0) posted on the Clearinghouse’s Web site.

IES is considering options for collecting and documenting potential conflicts of interest.

Sources of funding are rarely included in published documents beyond government and foundation support. An alternate source of information for tracking potential conflicts of interest would be for the WWC to request that study authors identify their source of funding, which would provide the WWC with a basis for flagging a potential conflict of interest. Any effort would depend on cooperation from authors because the WWC has no leverage to formally require authors to declare potential conflicts (which some academic journals require as a condition for publication). WWC’s experience to date is that study authors frequently fail to respond to requests for additional information, and IES officials expect that many study authors likewise will not respond to requests for information about funding sources, or may judge that it is not in their proprietary interest to provide the information. Currently the WWC only queries authors in cases where the Clearinghouse needs additional information. Querying all authors and tracking their responses would increase costs for intervention reports.

Another potential option is to ask developers, when they are reviewing the list of studies WWC found during the literature search for comprehensiveness, to note any studies that they funded.

The current version of the handbook gives guidance on standards for random assignment in simple cases. The next version of the handbook (forthcoming in 2010) will provide guidance and examples for multilevel settings, with explicit guidance on acceptable practice and potential issues with random assignment in a multilevel setting. lES asked the WWC to review the search process. The WWC now takes steps to ensure that search records are maintained. Each team and the library maintain a record of conducted searches. More documentation on the process will be included in the forthcoming revision of the handbook.

The WWC is undertaking a pilot using five recent evidence reports in different topic areas. Because WWC screeners are encouraged to pass to the next stage any study for which they are uncertain about eligibility, the proportion of eligible studies that are excluded is the salient error rate (the other source of error is when screeners include an ineligible study in a review, but this error is then offset by the review).

IES officials are not aware of any established standards for acceptable error rates (there are tradeoffs between making Type I vs. Type II errors relating to cost), but will examine this issue further. If the screening error rate is larger than the lES and the WWC believes is acceptable, IES officials will assess whether additional training or two screeners is appropriate given the different costs and benefits of each approach.

Currently, reference lists for WWC intervention reports include all studies, both eligible and ineligible, located in the search process. Ineligible studies are flagged with the primary reason for not qualifying for further WWC review. Intervention reports do not list materials such as product descriptions or reviews of products that are deemed not relevant to the intervention being reviewed.

To make the number of studies (both eligible and ineligible studies) more apparent to readers, the WWC will add a text box to intervention reports located in front of the listing of reports. The text box will summarize the number of studies that met different conditions (this approach currently is used for reports in which none of the studies meet standards). The box will serve the same purpose as a flow chart but the codes used to describe the final status for reports will be the same ones currently used in the citation appendix. The WWC plans to begin including the text box in reports released in 2010 and thereafter.

Ideally, the primary source for reanalyses of data would be study authors. However, as noted above in response to recommendation 2(v), it is common for authors not to respond to the WWC’s requests for additional information. Reanalyzing the data also would require additional effort by the authors and would run into difficulties when studies are dated or are based on data that has been destroyed to comply with confidentiality or privacy restrictions.

The WWC recently undertook a survey of published clustering estimates. It found that the WWC’s current default clustering correction is consistent with published estimates for achievement and behavioral outcomes. The WWC will continue to monitor research developments on this topic.

There are, of course, many possible ways to summarize evidence. Given its intended broad and primarily nontechnical audience, the WWC’s current approach is designed to be transparent and easily explained. lES believes that having the WWC conduct its own analyses to estimate intervention effects, as statistical meta- analyses do, would be inconsistent with these goals.

However, as an alternative to modifying the WWC’s main approach for reporting findings, lES is considering having the WWC conduct supplemental meta-analyses related to specific questions of interest, and releasing these findings as a separate report that would complement intervention reports. For example, a report could analyze whether computer mediated approaches to teaching reading are more or less effective than approaches that rely solely on teachers, based on already-released interventions reports. Having a separate report enables the WWC to continue using its current transparent approach, while also using statistical techniques that combine evidence in other ways.

Topic area protocols are available on the topic area home pages.

Just after the expert panel’s report, the WWC released its Procedures and Standards Handbook in December 2008. A revision currently is under way that will include more detail on the screening process. See the response to (3) above related to results of the screening process and the response to (5) regarding the results of the screening process. lES agrees that these products need to remain distinct. Practice guides are on a separate Web site tab that separates them from intervention reports. The next revision of the handbook (forthcoming in 2010) will include a chapter describing the practice guide development process and how it is different from that of the evidence reports.

The recently released guide on What Works for Practitioners also provides more information on reports and practice guides, and the WWC is preparing a video tutorial that will explain the differences to users.

The WWC tries to keep abreast of developments in the field, for example, by routinely checking materials from the Cochrane Collaborative when developing new standards or approaches.

Most recently, the WWC has undertaken the following outreach efforts to connect with other organizations conducting systematic reviews: The WWC sponsored a forum on research methods in December 2008 that featured speakers from the National Academy of Sciences, the National Cancer Institute, and the Cochrane Collaboration.

In June 2009 WWC staff attended the Cochrane conference on practice guides in June to learn about state of the art methods in research synthesis and practice guides.

The WWC is presenting a workshop on WWC standards at the upcoming annual conference of the Association of Public Policy and Management.

The WWC has also met with six international contacts (from Sweden, Denmark, Hungary, England, Interamerican Development Bank, and Trinidad/Tobago) in response to inquiries about how governments or organizations could implement their own clearinghouse operations.

Recently the WWC began a webinar series to disseminate its new practice guides. The webinar includes researchers and practitioners in its audience.

The WWC convened two groups of researchers to develop its forthcoming standards on single-subject designs and regression discontinuity designs. It will continue to bring together researchers as needs for new standards are identified. This approach will continue to be used for developing new standards.

Appendix IV: Comments from the Department of Education

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, individuals making key contributions to this report include Elizabeth Morrison (Assistant Director), Nagla’a El-Hodiri (analyst-in-charge), James Ashley, Carl Barden, James Bennett, Valerie Caracelli, Laura Henry, Geoffrey King, Jill Lacey, Luann Moy, Robert Owens, Cathy Roark, Stephanie Shipman, Kate Van Gelder, and Craig Winslow.

Related GAO Products

Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions. GAO-10-30. Washington, D.C.: November 23, 2009.

Teacher Quality: Sustained Coordination among Key Federal Education Programs Could Enhance State Efforts to Improve Teacher Quality. GAO-09-593. Washington, D.C.: July 6, 2009.

Teacher Preparation: Multiple Federal Education Offices Support Teacher Preparation for Instructing Students with Disabilities and English Language Learners, but Systematic Departmentwide Coordination Could Enhance This Assistance. GAO-09-573. Washington, D.C.: July 20, 2009.

No Child Left Behind Act: Education Actions Could Improve the Targeting of School Improvement Funds to Schools Most in Need of Assistance. GAO-08-380. Washington, D.C.: February 29, 2008.

Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals. GAO-02-923. Washington, D.C.: September 30, 2002.

The Evaluation Synthesis. PEMD-10.1.2. Washington, D.C.: March 1992.