Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse
Published: 2010-07-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In connection with the Omnibus Appropriations Act, 2009, GAO was required to study the What Works Clearinghouse (WWC), a federal source of evidence about effective education practices. Operating through a 5-year contract awarded by the U.S. Department of Education's Institute of Education Sciences (IES), the WWC reviews education research and disseminates its findings. GAO examined: (1) the extent to which the WWC review process meets accepted standards for research evaluation and how the WWC has responded to recommendations and criticism, (2) how WWC output and costs have changed over time and how its performance is measured, and (3) how WWC products are disseminated and how useful educators find them to be. To conduct its work, GAO reviewed WWC-related documents, analyzed cost and performance data, surveyed all states and a nationally representative sample of school districts, and interviewed IES officials, WWC contractors, researchers, and others.
section_title: What GAO Found
paragraphs: GAO as well as a congressionallymandated panel of experts, found that the WWC's review process, which includes screening studies to determine if they meet WWC criteria, follows accepted standards for evaluating research on the effectiveness of education interventions. WWC is responding to recommendations made by the expert panel to further improve its review and reporting processes. For example, the panel recommended improvements in the way the WWC presents information to readers on the reasons why studies do not qualify for review. The WWC is revising a report template to include a table summarizing which studies met or did not meet WWC criteria for evaluating research. The WWC has also responded to researchers who have criticized the WWC for presenting limited information because its screening criteria exclude some rigorous research designs that may be appropriate for evaluating certain education programs, such as special education. The WWC responded to this criticism by creating new standards that include two additional study designs and by creating a new product, called a practice guide, which includes a wider range of research. WWC's report output and scope increased under the current contract. For example, the WWC increased its production of various reports, introduced new products, and developed new processes for evaluating research. However, IES had a substantial backlog in its product review process from January 2009 to May 2010. The backlog generally decreased the timeliness of WWC reports, with 20 reports being delayed by up to 6 months. To support the increases in output and scope, WWC's costs doubled from the previous contract to the current one. Both contracts designated about 60 percent of costs to production, while the other 40 percent of costs support other tasks, such as communications, dissemination, and process development. IES' performance goals for the WWC primarily relate to the number of reports produced. However, IES has not developed performance measures related to the cost or usefulness of WWC products. Education uses WWC contractors, Regional Educational Laboratories (RELS) and the Doing What Works (DWW) Web site to disseminate information about WWC products; however, awareness and use of the WWC varies among states, districts, teachers, and principals. WWC contractors disseminate product information in various ways including email alerts and presentations. The RELs host events featuring WWC products for state, district, and school officials and DWW provides resources to educators based on WWC products. Based on our survey, officials from 33 of 38 state education agencies that responded to our survey and an estimated 42 percent of school districts have heard of the WWC. Those states and school districts generally used the WWC to a small or moderate extent to inform decisions on effective practices. Based on our survey, states and school districts reported that they would likely increase their use of the WWC if it included a broader array of information or more timely information.
section_title: What GAO Recommends
paragraphs: GAO recommends that IES: develop and implement strategies to avoid backlogs in WWC product reviews; establish performance measures related to costs and usefulness; and improve dissemination efforts to promote awareness and use of the WWC. Education generally agreed with GAO's recommendations.

section_title: Letter
paragraphs: The U.S. Department of Education’s What Works Clearinghouse (WWC or  Clearinghouse) was established as a federal source of scientific evidence  about “what works” in education. The Institute of Education Sciences  (IES), a division of the U.S. Department of Education (Education), created  the WWC in 2002, in part to help educators identify and use scientifically- based practices as specified in the Elementary and Secondary Education  Act of 1965 (ESEA). The WWC, which is operated by an independent  contractor, conducts systematic reviews of education research and  disseminates information on its Web site about the effectiveness of the  practices reported in these research studies. Currently operating under a  $50 million 5-year contract, the Clearinghouse has generated criticism in  the education research evaluation field on the timeliness of its reviews, its  standards for study inclusion, and the methodological soundness of its  research review process.
An explanatory statement submitted in lieu of a conference report for the  Omnibus Appropriations Act, 2009, directed GAO to examine how the  WWC reviews education research and to address concerns about the  operation, cost, and usefulness of the WWC. Specifically, GAO was  required to determine whether the WWC review process met current  standards for evaluating research and to examine the output and cost for  completing reviews, the degree of consistency of review procedures  across the various topics addressed, and the usefulness of the  Clearinghouse for practicing educators. To conduct this work, we  examined (1) the extent to which the WWC review process meets  accepted standards for research evaluation and how the WWC has  responded to recommendations and criticisms of its processes, (2) how  the WWC’s output and costs have changed over time and how IES  measures WWC performance, and (3) how WWC products are  disseminated and how useful education professionals find them to be.
To address all of our objectives, we interviewed and obtained information  from IES officials and the current and former WWC contractors, as well as  representatives from various educational organizations. In addition, to  address objective 1, we reviewed a prior GAO report that examined WWC  procedures and standards, an expert panel report that previously assessed  the validity of the WWC review process, literature, and procedures used by  other organizations that conduct systematic reviews of research. We also  reviewed the Clearinghouse’s response to the expert panel and to specific  criticisms in education research literature. To determine how performance  and costs changed over time (objective 2), we analyzed the costs and  productivity of the WWC contractors by reviewing budget, expenditure,  and performance data. For objective 3, we administered a Web-based  survey to state education agencies in all 50 states and the District of  Columbia and a nationally representative sample of school districts;  interviewed IES’s 10 Regional Educational Laboratories; and gathered  nongeneralizeable information from teachers and principals at four  conferences. Appendix I explains our scope and methodology in more  detail. We performed our work from September 2009 to July 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: The mission of the WWC is to be a central source of scientific evidence for  what works in education. To accomplish this, the WWC reviews existing  education research and posts information based on its reviews on the  WWC Web site, http://ies.ed.gov/ncee/wwc/.
The types of products currently available on the WWC Web site are  described in table 1.
In addition to the Clearinghouse, Education provides other technical  assistance and research-related resources to assist states, districts, and  schools. Examples of research-related resources include the Regional  Educational Laboratories (REL) and the Doing What Works (DWW) Web  site (http://dww.ed.gov):    Regional Educational Laboratories. IES’s Regional Educational  Laboratory Program is a network of 10 laboratories that conduct research  and provide policymakers and practitioners with expert advice, training,  and technical assistance on how to interpret findings from scientifically  valid research.
Doing What Works. Led by Education’s Office of Planning, Evaluation and  Policy Development, DWW is a Web-based resource intended to help  teachers, schools, districts, states and technical assistance providers  implement research-based instructional practice.
subsections: section_title: Initial Years of the What Works Clearinghouse
paragraphs: In 2002, IES awarded a $27 million 5-year contract to the initial contractors  to operate the Clearinghouse. The WWC contractors developed the  Clearinghouse’s research review standards with IES and reviewed  research related to topic areas considered to be pressing issues in  education. One of the goals of the Clearinghouse was to promote  informed education decision making through a Web-based dissemination  system that featured rigorous reviews of studies on the effectiveness of  educational interventions.
The WWC experienced a slow start due in part to the amount of work  involved in developing a research review and reporting process that was  valid, transparent, and replicable, according to the initial contractors. In  developing the research review process, the contractors and IES  addressed over 60 technical issues, such as determining what constitutes  an acceptable level of participant loss (attrition) from a study and what  methods should be in place to accommodate common education research  techniques. In addition, initial plans for topic areas and reporting formats  were modified. For example, IES decided to drop one planned topic area  because IES officials determined it to be too broad. The WWC and IES  also spent a substantial amount of time developing and refining a reporting  format to communicate research results to a lay audience. As a result, the  WWC began releasing reports in 2006. By September 2007, the WWC had  released 89 intervention reports, six topic reports, and three practice  guides.
section_title: WWC Research Review Process for Intervention Reports
paragraphs: The WWC uses a three-step review process to assess the quality of studies  and report on what the research indicates about the effectiveness of  interventions. The WWC definition of interventions includes programs  (such as whole school reform), products (such as a textbook or  curriculum), practices (such as mixed-age grouping), or policies (such as  class size reduction). The process begins with an initial screening of  published and unpublished studies relevant to the intervention being  reviewed. Studies are collected from electronic databases, journals,  conference proceedings, and nominations solicited from the general  public. The studies that pass initial screens are reviewed to determine  whether they provide valid evidence of an intervention’s effectiveness.  Using these studies, the WWC then synthesizes the evidence about the  effectiveness of an intervention and publishes a report describing its  findings. The Clearinghouse categorizes interventions as either having  positive effects, potentially positive effects, mixed effects, no discernable  effects, potentially negative effects, or negative effects (see fig. 1).
The WWC uses evidence standards to assess the strengths and weaknesses  of a study’s methodology, such as the type of design it uses, the quality of  the study’s data, and the appropriateness of the study’s statistical  procedures. Until recently, the WWC accepted two types of study  designs—randomized experiments and quasi-experimental studies. Only  randomized controlled trials (or randomized experiments) that WWC has  determined to be well-designed and well-implemented are considered  strong evidence and can receive the highest rating of “meets evidence  standards without reservations.” The WWC also considers evidence from  quasi-experiments it has determined to be well-designed and well- implemented. The highest rating a study using quasi-experimental design  can receive is “meets evidence standards with reservations.” This rating  category is intended to inform educators to interpret the study results with  caution, as the results may reflect other factors, in addition to the impact  of the intervention (see table 2).
section_title: IES Oversight and Support of the WWC
paragraphs: The WWC is administered by IES through a contract with a private  research organization. IES monitors implementation of the specific tasks  detailed in the WWC contract by reviewing an annual work plan and  monthly performance and expenditure reports submitted by the  contractor. IES tracks implementation of the tasks, completion of  performance goals, and adherence to the budget outlined in the contractor  work plan. The contractor monitors the work of any subcontractors that  it uses to perform services such as research reviews, technological  support, and communications support.
IES is also involved in the development and dissemination of WWC  products. IES reviews and approves proposed topics for WWC products,  product formats, and the research review procedures. It also coordinates a  group of independent researchers to peer review WWC products and  reviews and approves all WWC products prior to public release. IES  required the contractor to develop a communications plan to inform WWC  customers about features of the Web site.
section_title: WWC Reviews Research in Accordance with Accepted Standards and Has Responded to Recommendations and Criticisms
subsections: section_title: WWC Follows Accepted Review Standards and Is Improving Its Review Process in Response to a Congressionally Mandated Expert Panel Report
paragraphs: We found that the WWC review process follows generally accepted  practices for reviewing research. Specifically, GAO’s November 2009  report reviewing federally supported efforts to identify effective  interventions found that the WWC determines whether a study provides  credible evidence on effectiveness based on several dimensions, including  the quality of the research design, how the study was implemented, and  other technical considerations. Our 2009 report also noted that WWC  follows a common approach to conducting its reviews, and provides  information to help educators understand the body of existing research on  specific interventions.
Additionally, a congressionally mandated panel of experts found in  October 2008 that WWC’s research review process was based on  appropriate methods for judging the strength of the evidence regarding the  effectiveness of interventions. For example, the panel agreed that the  minimum qualifications a study must meet in order to be reviewed by the  WWC are appropriate. The panel also found that WWC’s reporting process  is reasonable and that the WWC provides succinct and relevant evidence  on the effectiveness of education interventions. While the panel concluded  that the WWC’s processes are generally appropriate, the panel made  several recommendations to the WWC for continued improvement. The  recommendations primarily related to establishing or clarifying  procedures, reviewing statistical methods, and documenting the screening  process.
The WWC implemented or is considering implementing 14 of the panel’s 17  recommendations. The WWC implemented nine recommendations, in  part by modifying some procedures and creating a procedures and  standards handbook. For example, in response to the panel’s  recommendation that the WWC include a table of study dispositions (e.g.,  whether studies meet WWC evidence standards) at the front of  intervention reports, the WWC is modifying the report template to include  a summary table along with the existing listing of dispositions in the  reference section. The WWC also addressed panel concerns about  technical issues in its review process by making its treatment of study  attrition—the rate at which subjects drop out of a study—more consistent  across topic areas. The panel noted that the WWC’s practice of  determining acceptable attrition levels by topic area led to arbitrary  inconsistencies across the topic areas. In response to the panel’s  recommendation that the WWC reconsider this practice, the WWC took  steps to increase its consistency by developing attrition guidance that  applies to all topic areas. (See app. III for a table detailing the  recommendations, WWC and IES’s response, and the status of any changes  made in response to recommendations.)
In addition, the WWC is considering implementing five other panel  recommendations. For example, the panel raised concerns that the WWC  does not document some potential conflicts of interest for the studies it  reviews. In response to this concern, the WWC is considering tracking and  publishing whether studies of a program are funded or conducted by the  program’s developers. Further, in response to the panel’s concern that  the WWC’s screening process may exclude some eligible studies, the WWC  is undertaking an evaluation of the reliability of its screening process.  According to IES officials, they will postpone decisions about the  recommendations until the newly appointed Commissioner for the WWC is  on board and actively involved in the decision making.
section_title: WWC Also Responded to Criticism That It Produces Limited and Potentially Misleading Information
paragraphs: Some researchers claim that the WWC presents potentially misleading  information by including brief experiments involving small numbers of  students when evaluating interventions. As a result, according to critics,  educators may accept the WWC’s rating of the intervention’s effectiveness,  even though the evidence behind the rating is limited. One researcher  suggested the WWC emphasize larger studies that span significant periods  of time and set a minimum sample size requirement. According to WWC  staff, such changes would exclude valuable research and prevent the WWC  from providing educators with research-based information about some  interventions. Instead of changing its treatment of sample size and study  duration, the WWC began publishing information on the extent of the  evidence supporting its findings in 2007. The WWC’s “extent of evidence”  rating alerts educators when the WWC effectiveness ratings are based on a  small amount of evidence. As figure 2 shows, 76 percent of interventions  with positive or potentially positive ratings of effectiveness are based on a  small amount of evidence (see fig. 2).
Further, researchers suggested that the WWC presents misleading  information by rating interventions based on studies in which measures of  student performance closely match the content taught to the intervention  group, but not the control group. In such studies, higher test scores  among the intervention group may not accurately represent the  effectiveness of the intervention more generally. The researchers  suggested that the WWC exclude such measures, or at least report on them  separately. However, the WWC includes these measures because,  according to IES officials, they answer questions about whether different  interventions lead to different content knowledge. The WWC agrees that  there is a concern regarding the reliability of outcome measures that are  overly similar to the intervention, but maintains that WWC procedures  attempt to exclude such measures. In addition, in response to researcher  concerns that tests created by intervention developers may be biased, the  WWC added information to the intervention reports noting whether  outcome measures are based on tests created by the developer.
Some researchers and education professionals we interviewed suggested  that the WWC produces limited information because its screening criteria  are too restrictive—currently screening out about 90 percent of studies  initially identified as potentially relevant (see fig. 3). Until recently, the  WWC reviewed only two types of study designs—randomized experiments  and quasi-experimental studies—and according to critics, this limited the  amount and type of information available to educators. For example,  staff from one REL noted that educators may not be able to find reviews of the interventions they are using or considering because so few studies  meet WWC standards. Staff from another REL told us that if educators  cannot find relevant and useful information, they may be discouraged from using evidence-based practices. Staff from a third lab noted that the  narrow focus prevents educators from learning from less rigorous but  nonetheless useful research, such as case studies describing an  intervention’s costs and implementa tion requirements.
The WWC maintains that its screening criteria and study inclusion  standards focus on studies that provide strong evidence of an  intervention’s effectiveness, and lowering these standards could  undermine the validity of the findings reported by the WWC. Although the  Clearinghouse screens out most studies, many of its reports have  identified interventions with positive effects. Data from the contractor  indicate that 58 percent of WWC’s intervention reports identify positive or  potentially positive effects of interventions. While the WWC plans to  continue using its methodological standards for reviewing randomized and  quasi-experimental studies, the Clearinghouse acknowledges that the  emphasis on randomized experiments and quasi-experiments can exclude  useful information on interventions in certain topic areas, such as special  education, that do not lend themselves to these study designs. The WWC  created new standards to include additional study designs.
The WWC also introduced practice guides in 2007 in response to criticisms  that its intervention reviews exclude too much research and consequently  provide limited information to educators. Written by a panel of experts,  practice guides include recommendations for educators on various topics,  such as reducing high school drop-out rates and reducing behavioral  problems in the classroom. Whereas WWC’s intervention reviews are  based entirely on studies that meet WWC evidence standards, practice  guides also incorporate studies that do not have designs that are eligible  for WWC review, or in some cases, are reviewed and do not meet WWC  evidence standards, and include the views of experts. To develop  recommendations, the practice guide panel reviews available literature  about the particular topic and then meets several times to discuss the  topic. Through consensus, the panel identifies effective practices based on  the evidence. Once the practice guide is developed, it undergoes a quality  assurance review by WWC and IES staff and external peer review. The  following text box provides an example of practice guide  recommendations and the level of evidence supporting them.
Example of Practice Guide Recommendations and Evidence Levels  In 2009, the WWC published a practice guide to help educators assist students  struggling with reading in the primary grades. The practice guide authors used an early  detection and prevention framework known as Response to Intervention. The panel that  authored the practice guide consisted of six researchers and one expert in  implementation of the Response to Intervention model. Two WWC staff also assisted in  the practice guide development.
The panel’s recommendations follow.
section_title: WWC’s Output and Costs Increased; However, IES Has Not Developed Adequate Performance Measures Related to Cost or Product Usefulness
subsections: section_title: WWC Increased Output and Introduced New Products
paragraphs: WWC’s report output increased under the current contract, and its scope  expanded to include new products and processes to support production.  Under the current contract, the WWC increased its total number of  publications from the first contract year to the second contract year and  generally kept pace with its increased scope, as specified in the  Clearinghouse’s annual plans. For example, the current contract calls for  the WWC to increase the number of topic areas and intervention reports.  Under the current contract, the WWC added three new topic areas and  released 60 intervention reports, including 5 in the new topic areas as of  June 2010. In addition, the WWC produces practice guides and quick  reviews and increased its production of both of these products between  the first and second year of the current contract. Figure 4 shows the  production of all three WWC products as of June 30, 2010, the end of the  third contract year.
The WWC’s scope of work increased under the current contract with the  addition of new products and work processes, as well as responsibilities  related to the American Recovery and Reinvestment Act of 2009 (Recovery  Act), which provided additional innovation and improvement funding.  The WWC is developing three new types of publications and conducts an  annual review of Education-sponsored studies for IES’s internal use.  Specifically, the WWC is developing research briefs, research perspectives,  and practice briefs, which will focus on Education policy priorities. Like  practice guides, the new publications will incorporate expert opinion and  a broad range of research. Table 3 provides more information on these  new initiatives.
section_title: IES’s Reviews Have Delayed the Release of Some Reports
paragraphs: While the WWC contractor increased its report production, IES’s review  process did not keep pace with output. IES is responsible for  administering independent peer reviews of all products and conducting  final reviews and approvals before products are released, and has internal  time frame estimates used in scheduling and completing such reviews. For  example, according to IES planning documents, IES estimates 15 business  days for the completion of peer reviews for intervention reports and 6  business days for WWC quick reviews. However, throughout 2009, IES  took increasingly more time to schedule and coordinate the completion of  peer reviews for some intervention reports and quick reviews. As a result,  the release of 20 reports—11 intervention reports and nine quick  reviews—was delayed by more than 6 months. For example, in the first  quarter of the current contract year (third contract year, 2009 to 2010), IES  took an average of over 50 business days to have intervention reports and  quick reviews peer reviewed, compared to an average of 7 business days  during the first quarter of second contract year (see fig. 5).
These delays in the IES-administered peer review process resulted in  significant backlogs of intervention reports and quick reviews awaiting  release. For example, as shown in figure 6, reports that entered the peer  review process in the first quarter of the second contract year (CY2-Q1)  were completed within that quarter. However, the majority of reports  entering review the first quarter of the third contract year (CY3-Q1)  remained in process for subsequent quarters. While the backlog persisted  through the third quarter of the third contract year (CY3-Q3), the number  of reports that completed peer review in the third and fourth quarters  increased from prior quarters. Figure 6 shows that 11 intervention reports  completed peer review in CY3-Q3 and an additional 27 completed peer  review in CY3-Q4, compared with 4, 5, and 8 intervention reports in the  prior three quarters.
IES attributed these delays to several factors and recently took steps to  eliminate the backlog. IES officials told us that delays were, in part,  attributable to difficulty in identifying and scheduling independent peer  reviewers, vacancies in WWC-related positions at IES, and an increasing  amount of research that met WWC standards. For example, IES officials  told us identifying and scheduling a sufficient number of qualified,  independent peer reviewers had become increasingly difficult because  several former peer reviewers were now associated in some way with th WWC and therefore were no longer independent. To reduce the delays and eliminate the backlog, IES recently implemented a new database to help  staff track and manage the work of peer reviewers and other WWC-relate tasks. IES officials also told us that they began identifying additional  potential peer reviewers using the WWC online registry of researchers. In  addition, IES increased a staff member’s responsibilities related to  scheduling and coordinating peer reviews. These efforts reduced the  amount of time reports remain in the IES peer review process and  eliminated the backlog as of June 2010.
In addition to delays in the peer review process, WWC contractors told us  that many of their daily decisions need IES approval, and slow responses  delayed contractor processes. For example, the contractor needs IES  approval on the format and content of the products in development,  hindering further work when responses are delayed. IES officials  acknowledged that some delays in the approval process occurred during  contract year three and told us that this was largely due to staff vacancies  that they anticipate filling.
section_title: The Cost of the Current WWC Contract Has Increased from the Previous One
paragraphs: WWC’s contracted costs have doubled from about $5.3 million per year  under the previous 5-year contract to the current level of about $10.7  million per year. The increase in contracted costs reflects the expand scope—more publications and new products and processes—of the  second contract compared to the first. IES’s contract for the WWC  includes a variety of tasks that the contractor is responsible for, inc tasks related to report production and product development. Table 4  provides a description of six broad task categories and how they chan between contracts.
section_title: IES Has Not Developed Performance Measures Related to Production Costs or Product Usefulness
paragraphs: IES established performance goals, which the WWC met or exceeded;  however, these goals do not address production costs or the usefulness of  WWC products. IES established WWC-related performance goals in its  annual organizational assessment, but Education discontinued the use of  these performance measures for fiscal year 2010. In addition, IES  established performance goals for its WWC contractor in the contractor  award fee plan, which IES uses to determine the amount of performance- based funds awarded to the contractor.
IES measured WWC program performance from fiscal year 2003 to fiscal  year 2009, as part of Education’s Organizational Assessment—its  departmentwide performance management system. The WWC-related  performance goals included in Education’s Organizational Assessment  focused on WWC Web site visits and the quantity of publications, both of  which were areas of concern as the WWC was getting established.  Specifically, these performance goals included increased WWC Web site  visits, sustained productivity in the release of intervention reports and  quick reviews, and increased practice guide production. The WWC met or  exceeded these performance goals annually; however, according to IES  officials, these performance goals will not be included in Education’s fiscal  year 2010 Organizational Assessment, in part because IES is now satisfied  with WWC activity in these areas.
IES has not developed performance measures related to the cost of  specific WWC products. IES officials noted that the costs per WWC  publication vary greatly depending on the amount of available research  relevant to the specifications of a product. For example, intervention  reports based on a large number of studies meeting WWC standards take  longer and cost more to produce than do reports for which few studies  qualify for review. IES has tasked the current WWC contractor to develop  ways to streamline production processes and to conduct a cost study, the  results of which would improve budget estimates and strengthen IES’s  monitoring of the contract. While the contractor has begun this work, IES  officials told us that they do not know when cost-related performance  measures, such as acceptable cost ranges for each type of product, will be  established.
WWC does not currently have a performance measure related to product  usefulness. While Web site visits were tracked as a measure of WWC  utilization in IES’s Organizational Assessment through fiscal year 2009,  this metric did not assess the degree to which WWC products were  reaching their target audience and did not provided any information on the  extent to which educators find WWC products to be useful. IES’s 2010  budget justification calls for a representative survey of WWC use among  education decision makers to be conducted by 2013. However, IES  officials told us that they were unsure whether the survey would take  place, and IES does not currently have a plan in place to implement this  survey.
section_title: Education Has Three Primary Ways to Disseminate Information about WWC Products, but Awareness and Use Vary among Target Audiences
subsections: section_title: Education Has Various Ways to Disseminate Information about WWC Products, but Awareness of the Clearinghouse Is Generally Limited
paragraphs: Education uses the WWC contractor, RELs, and DWW to disseminate  information about the Clearinghouse to its target audience, which includes  state and school district officials, as well as teachers and principals. In  accordance with its contract, the WWC contractor disseminates  information about its products electronically and through various events,  such as formal presentations at conferences. The Clearinghouse’s  electronic dissemination methods include an e-mail listserv, Web-based  seminars (webinars), and newsletters. For example, the WWC sends out  notices to its e-mail listserv, alerting subscribers of the availability of new  products, including intervention reports, practice guides, and quick  reviews. WWC staff told us that the webinars cover the same topics as  their reports and are a relatively cost-effective way to disseminate  information about products and methodology. In addition, WWC staff  disseminate information about WWC products at education conferences,  such as teacher, principal, and researcher conferences. At these  conferences, WWC staff may conduct formal presentations, have an  exhibit featuring their products, or both. At conference exhibits,  Clearinghouse staff answer questions about their products and provide  literature to conference attendees. From July 2009 through June 2010,  WWC staff were scheduled to present or have an exhibit at 14 conferences.  WWC staff also told us that they work with other groups, such as  education, research, and legislative organizations, in order to further  disseminate information about WWC products to their members.
In addition, Education disseminates information about WWC products  through IES’s 10 RELs, which hold events that may feature information  based on practice guides and refer educators to Clearinghouse products.  Officials at all 10 RELs told us that they spent time disseminating  information about WWC, in part by holding events that bridge research  and practice. According to REL officials, these bridge events are attended  primarily by school-, district-, and state-level education professionals and  provide an opportunity for educators to discuss ways to implement  research-based practices. Officials at all 10 RELs told us that bridge events  focused on practice guides to some extent, and 7 indicated that WWC  practice guides were the primary focus of these events. According to REL  officials and WWC staff, these events sometimes included a WWC staff  member to discuss methodology and panelists who helped develop the  practice guides. RELs also disseminate research from WWC when  responding to educator questions or concerns. Officials from 7 of the 10  RELs told us their respective RELs generally use relevant WWC products  (practice guides and others) when searching for research-based  information to address educator questions.
In addition, Education’s Office of Planning, Evaluation, and Policy  Development disseminates information about WWC practice guides on its  DWW Web site, which provides an online library of resources designed to  help educators implement research-based instructional practice. This Web  site uses different formats to present content based primarily on WWC  Practice Guides and provides examples of possible ways educators might  apply WWC research findings. For instance, to help educators implement  the recommendations from the practice guide on dropout prevention, the  DWW Web site features slideshows with examples of supportive academic  environments and interviews with educators and experts on dropout  prevention. In addition, the Web site includes sample materials, such as  lesson plan templates, that provide an example of how to implement  recommendations. The DWW also includes information on the research  behind the recommendations and a link to the WWC Web site and the  individual practice guides. According to IES officials, a recent analysis of  the DWW Web site traffic showed that 49 state Web sites have links to the  DWW Web site, which helps disseminate WWC products further to the  education community.
We found that 33 of the 38 states that responded to our survey reported  that they had heard of the WWC. Based on our survey results, we estimate  that 42 percent of school districts have heard of the WWC and that the  percentage is greater for school districts that rely to a very large extent on  external sources for information on research-based practices.
School districts identified several sources of information about the  Clearinghouse, including conferences and Education (see fig. 8).
While the majority of states have accessed the WWC Web site, we estimate  that only 34 percent of school districts have done so. Specifically, among  the states that responded to our survey, 33 of 38 states reported that they  had accessed the WWC Web site at least once. In addition, 19 of states  reported visiting the Web site at least seven times per year. In contrast,  an estimated 34 percent of school districts accessed the WWC Web site at  least once. Further, we estimate that only 11 percent of school districts  visited the Web site at least seven times per year. States and school  districts that visited the WWC Web site less than seven times per year  most often cited time constraints as the primary reason for their  infrequent use.
In addition to the WWC, states and school districts use a variety of other  sources of information to identify effective education practices. Most  states and school districts use several broad sources of information, such  as academic journals, education periodicals, and associations of  educators. For example, 37 states reported using academic journals to  identify such practices, and we estimate that about 97 percent of school  districts used academic journals. Overall, more school districts and  states that responded to our survey used the WWC than used other  research synthesis organizations.
While the WWC also includes teachers and principals in its target  audience, we found that relatively few of the teachers and principals we  contacted at education conferences had heard of the WWC. While not a  generalizeable sample, we found that out of a total of 391 teachers who  completed our questionnaire at four education conferences, only 18 had  accessed the WWC Web site. In addition, 341 teachers who had not  accessed the WWC Web site told us they had not heard of the Web site.  Similarly, among the 208 principals and other school administrators who  completed the questionnaire, only 32 had accessed the WWC Web site.  Further, 135 principals and other school administrators told us they had  not heard of the WWC.
section_title: States and School Districts Generally Used the Clearinghouse to a Small or Moderate Extent to Inform Decisions and Used Specific WWC Products to Varying Extents
paragraphs: Based on our survey, most states and school districts that reported  accessing the WWC Web site used it to inform decisions on effective  education practices—a stated purpose of the WWC—to a small or  moderate extent. Specifically, 25 of the 33 states that use the  Clearinghouse indicated that they use it to a small or moderate extent to  inform their decisions, while 6 reported using it to a large or very large  extent. We estimate that 72 percent of school districts that have accessed  the Clearinghouse use the WWC to inform education decisions to a small  or moderate extent, while only 18 percent use it to a large or very large  extent.
States that used the WWC to inform decisions reported that they used the  Clearinghouse for various purposes, including informing professional  development and curriculum decisions. For example, 25 states reported  using the Clearinghouse to inform professional development programs for  teachers, and 22 reported using it to inform curriculum decisions. Fewer  states used the Clearinghouse to advise districts that were not making  adequate yearly progress (AYP) in meeting academic targets or to develop  improvement plans for such districts. (Fig. 9 provides a breakdown of the  extent to which these states use the Clearinghouse for various purposes.)
In addition, we estimate that among school districts that use the WWC to  inform decisions on effective education practices, about 90 percent used it  to inform curriculum decisions at least to a small extent, similar to the  percentage that used the WWC to inform professional development  decisions. However, fewer school districts used it to advise schools that  did not meet academic goals or to develop school-level plans to help such  schools improve. Figure 10 provides a breakdown of the extent to which  nt to which  these school districts use the Clearinghouse for various purposes.  these school districts use the Clearinghouse for various purposes.
States reported using specific WWC products—intervention reports and  practice guides—more than quick reviews. Specifically, of the states that  had used the Clearinghouse, 21 reported that they used intervention  reports and 20 reported using practice guides, while only 12 reported using  quick reviews. States used intervention reports and practice guides to a  similar extent to inform education decisions. For example, for each  product, six states reported using them to large or very large extent to  inform such decisions (see fig. 11).
However, the relative use of specific WWC products was different among  school districts. We estimate that among school districts that use the  Clearinghouse to inform decisions on effective education practices, more  school districts use intervention reports relative to practice guides or  quick reviews. Specifically, we estimate that 74 percent of those school  districts that use the WWC have used its intervention reports to inform  education decisions, while practice guides and quick reviews were each  used by about half of such districts. Based on our survey, an estimated 21  percent of school districts that use the WWC have used intervention  reports to a large or very large extent, while about 10 percent use  t 10 percent use  practice guides to a large or very large extent (see fig. 12).  practice guides to a large or very large extent (see fig. 12).
The 95 percent confidence interval for this estimate is (5.3, 17).
section_title: States and School Districts Would Likely Increase Their Use of the Clearinghouse If the WWC Made Certain Changes
paragraphs: Many states and school districts that had accessed the Clearinghouse  reported that they would likely increase their use of the WWC if the  Clearinghouse provided a broader array of information. For example,  many states and school districts would be likely to increase their use of  the Clearinghouse if it reviewed more studies, covered additional topics,  or provided more relevant or timely reports. For example, 21 of the 33  states that had used the Clearinghouse reported that they would be  somewhat or very likely to use the Clearinghouse more often if it had  reviews that were more timely (see fig. 13).
In addition, based on our survey, we estimate that about two thirds of  school districts that had accessed the Clearinghouse would likely increase  their use if it included reviews of programs or interventions being used or  considered in their school district. An estimated 50 percent of school  districts would likely increase their use of the Clearinghouse if it had  reviews that were more timely (see fig. 14).
section_title: Conclusions
paragraphs: In 2007, Education substantially increased its financial investment in the  WWC, and the Clearinghouse is significantly expanding its scope in an  effort to better serve its target audience. Some of the new products aim to  be more responsive to educators and education decision makers by  providing timely information about evidence-based practices relevant to  pressing needs. Such information could help states and districts identify  strategies as they implement educational reform efforts—such as  reforming low-performing schools or improving professional  development—under ESEA and the Recovery Act. For example, WWC  research perspectives, still in development, are intended to help education  decision makers as they address challenges related to spending Recovery  Act funds. However, the development of these products and the release of  other products were delayed, in part, by a substantial backlog in IES’s  review and approval processes. These delays hindered the timely release  of several publications, and some products were released months after  they were completed by the contractor. While IES recently eliminated the  backlog, educators need to be able to rely on the Clearinghouse for timely  and relevant information. According to our survey, many states and school  districts reported that they would likely increase their use of the  Clearinghouse if it released information more quickly.
While IES has increased annual report production, IES has not established  reasonable production cost ranges or specific cost-related performance  measures related to each product type. Without acceptable per product  cost ranges, it is difficult for IES to assess the reasonableness of costs  associated with certain products, even as IES takes steps to streamline  production. IES’s current study on costs may help IES establish acceptable  cost ranges that could inform IES’s performance measurements related to  the WWC. In addition, such information could inform cost comparisons  between the WWC and other research evaluation organizations or provide  baselines for future contractor work.
In addition, IES has not established meaningful performance measures  related to product usefulness. Until fiscal year 2010, IES tracked visits to  its Web site and annual report production as a way to measure the  productivity of the Clearinghouse. While these measures were important  to accurately track the WWC’s initial growth, they did not evaluate the  degree to which the products were meeting the needs of educators.  Specifically, IES currently does not have a way to gauge user satisfaction  with WWC products, which is a common practice when developing and  providing new products. Further, while IES currently incorporates some  feedback from the WWC Web site users, to inform future topic areas, it  does not systematically gauge its target audience’s major areas of interest  or concern—such as gathering information on interventions currently  being used or considered in specific school districts or states. IES decides  how to spread its limited resources across the various product types  without directly measuring the extent to which educators use the WWC or  how useful they find the various products to be. Measuring the use and  usefulness of its products could help IES continue to improve content,  develop products, and respond to the needs of educators and  policymakers.
While some educators and policymakers find WWC products useful, many  other educators are not familiar with the Clearinghouse. IES has spent a  substantial amount of money, time, and effort producing various  summaries of evidence-based practices, which cover both specific  education interventions and general practices. This investment in the  WWC was made in order to inform education professionals at all levels— from classroom teachers to policymakers—as they make decisions on how  best to educate the nation’s children. Improved dissemination of WWC  products could increase awareness and use of the WWC. Increased use of  the Clearinghouse could help education professionals identify and  implement effective educational interventions and practices, and  potentially lead to increased student achievement.
section_title: Recommendations for Executive Action
paragraphs: We are making the following four recommendations based on our review.
To consistently release WWC products in a timely manner, we recommend  the Secretary of Education direct IES to develop and implement strategies  that help avoid future backlogs and ensure that IES’s review and approval  processes keep pace with increased contractor production. Strategies  could include shifting IES resources to ensure sufficient staff time for  managing the peer review process and streamlining its approval processes.
To better track the costs and usefulness of the WWC, we recommend that  the Secretary of Education direct IES to  incorporate findings from its cost studies to develop performance  measures related to costs, such as identifying a range of acceptable costs  per product and using that information to monitor contractor spending;  and    develop performance measures related to product usefulness and  periodically assess whether WWC products are meeting the needs of target  audiences by gathering information on product usefulness in the proposed  survey or through other means.
To reach more members of the target audience, we recommend the  Secretary of Education direct IES to assess and improve its dissemination  efforts to promote greater awareness and use of the WWC, for example, by  developing a way to inform school districts of new products or  encouraging educator professional development programs to focus on  research-based practices such as those discussed in practice guides.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the U.S. Department of Education for  review and comment. Education officials provided written comments on a  draft of this report, which are reproduced in appendix IV. Education also  provided technical comments, which we incorporated into the report as  appropriate.
Education generally agreed with our recommendations. Specifically,  Education agreed to our recommendations on consistently releasing WWC  products in a timely manner and assessing and improving its dissemination  efforts. In its response to our recommendation on tracking the cost and  usefulness of the WWC and its products, Education noted that IES has  taken some steps that address the recommendation. With regard to costs,  Education stated that it intends to incorporate the results of current cost  studies into future work plans and monitoring efforts. We continue to  recommend that these results be used to inform performance measures  related to costs for future operations. With regard to tracking the  usefulness of the WWC, Education noted that it uses a variety of tools to  gather consumer input, such as a Help Desk and online voting for future  report topics. While such feedback provides some information to the  WWC, it relies on existing users and reflects the views of those users who  provide feedback, rather than those of the broader population. However,  as shown in our survey, only an estimated 34 percent of school districts  have accessed the WWC Web site at least once—and fewer have used the  Web site frequently. Education also noted that it would include a customer  satisfaction survey in IES’s review of its own performance, but whether  the survey would be directed at current Clearinghouse customers or a  broader audience, or whether the survey would identify how useful  various WWC products are and how the WWC can be improved is unclear.  More nationally representative information could help IES prioritize topics  for intervention reports and practice guides and inform budget priorities.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of the U.S. Department of Education, and other  interested parties. In addition, the report will be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions regarding this report, please  contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made major contributors to this  report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To address all three objectives, we interviewed officials from the Institute  of Education Sciences (IES), What Works Clearinghouse (WWC or  Clearinghouse) contractors, and representatives from various educational  organizations. To assess the research review process used by the IES’s  WWC, we reviewed WWC standards and procedures, reviewed an expert  panel report that assessed the validity of the WWC review process, and  collected information about the extent to which the WWC has  implemented the panel’s recommendations. To determine how  performance and costs changed over time, we analyzed the costs and  productivity of the two WWC contractors. To obtain information about the  usefulness of WWC products, we conducted a Web-based survey of all  state education agencies and a nationally representative sample of school  districts. We also collected information about the usefulness of the WWC  from teachers and principals at four education conferences. We conducted  our work from September 2009 through July 2010 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Assessment of WWC Research Review Process
paragraphs: GAO previously assessed the procedures and criteria used by the WWC by  reviewing documents and interviewing IES officials and WWC  contractors. We reviewed WWC standards and procedures and examined  the degree of consistency of these standards and procedures across  education topic areas. We also reviewed the findings and  recommendations from an expert panel report that assessed the validity of  the WWC review process. We obtained information from IES officials and  WWC contractors on the extent to which the WWC has implemented the  panel’s recommendations. Further, we identified other concerns about the  WWC review process through a literature review and interviews with  researchers, and we interviewed IES officials and WWC contractors to  assess the extent to which the Clearinghouse has addressed these  concerns. We also examined the degree to which the WWC’s review  process is similar to that used by other entities engaged in systematic  research review efforts.
section_title: Performance and Cost Data Analyses
paragraphs: To determine how performance and costs changed over time, we analyzed  the costs and productivity of the two WWC contractors. We reviewed  budget data and product release dates to analyze cost and productivity  trends of the WWC. To examine performance, we interviewed the two  contractors, as well as IES officials, and compared IES’s performance  measures and goals to actual outcomes.
We assessed the reliability of the WWC performance and cost data by (1)  reviewing existing information about the data and the system that  produced them and (2) interviewing agency officials knowledgeable about  the data. We determined that the data were sufficiently reliable for the  purposes of this report.
section_title: Survey of States and School Districts
paragraphs: To determine how WWC products are disseminated, we interviewed  officials from IES and all 10 RELs, as well as WWC contractors. To  determine how useful education professionals find WWC products to be,  we designed and administered a Web-based survey of state education  agencies in the 50 states and the District of Columbia and a nationally  representative sample of local educational agencies (LEA). Specifically,  the survey asked officials about (1) their general sources of information on  effective educational practices, (2) the extent to which they use WWC  products to inform curriculum decisions (including questions on specific  intervention reports and practice guides), (3) how useful the officials find  the information in the WWC, (4) the likelihood they would increase their  usage if certain changes were made to the WWC Web site, and (5) the  extent to which the officials use the Doing What Works and Best Evidence  Encyclopedia Web sites to inform curriculum decisions and how useful  the officials find these other information sources to be. We reproduce the  questions we used in our analysis in figure 15. The survey was  administered from February 18, 2010 to April 14, 2010.
To determine how the WWC was being used at the state level, we surveyed  the state Secretary, Commissioner, or Superintendent of Education in the  50 states and the District of Columbia. Out of the 51 state officials  surveyed, 38 responded to the survey.
To determine how the WWC was being used at the school district level, we  surveyed a nationally representative sample of school districts across the  country. We selected a stratified random sample of 625 LEAs from the  population of 17,620 LEAs included in our sample frame of data obtained  from the Common Core of Data for the 2007-08 school year. A total of 454  LEAs responded, resulting in a final response rate of 74 percent. Because  we surveyed a sample of LEAs, survey results for the district are estimates  of a population of LEAs and thus are subject to sampling errors that are  associated with samples of this size and type. Our sample is only one of a  large number of samples that we might have drawn. As each sample could  have provided different estimates, we express our confidence in the  precision of our particular sample’s results as a 95 percent confidence  interval (e.g., plus or minus 10 percentage points). We excluded 12 of the  sampled LEAs for various reasons—6 were closed, 3 did not administer  any schools, 2 managed schools in a correctional facility, and 1 was a  private school—and therefore were considered out of scope. All estimates  produced from the sample and presented in this report are representative  of the in-scope population.
The practical difficulties of conducting any survey may introduce  nonsampling errors, such as difficulties interpreting a particular question,  which can introduce unwanted variability into the survey results. We took  steps to minimize nonsampling errors by pretesting the questionnaire over  the phone with officials from two school districts and one state  department of education in November and December 2009. We conducted  pretests to verify that (1) the questions were clear and unambiguous, (2)  terminology was used correctly, (3) the questionnaire did not place an  undue burden on officials, and (4) the questionnaire was comprehensive  and unbiased. An independent reviewer within GAO also reviewed a draft  of the questionnaire prior to its administration. We made revisions to the  questionnaire based on feedback from the pretests and independent  review before administering the survey.
The survey-related data used in this report is based on the state and school  district responses to the survey questions.
The following tables contain the estimates and associated confidence  intervals for the data displayed in figures 8, 10, 12, and 14.
section_title: Information from Teachers, Principals, and Researchers
paragraphs: In addition to interviews with teacher, principal, and research organizations,  we obtained information about the usefulness of the WWC by administering  a questionnaire at four conferences of teachers and principals. Table 9  provides more information about the conferences we attended.
We selected these conferences because they were relevant to segments of  the WWC’s target population that we were not reaching through our  survey and they were held at times that coincided with our report time  frames. At each of these conferences, conference organizers agreed to  have GAO have a table either inside the exhibit hall or just outside it. The  questionnaires included questions on awareness and use of WWC—  including use of specific products and use of other information sources to  identify effective educational practices. For those who had not used the  WWC, we also asked them to specify the reason they had not used it. The  information gathered through the questionnaires is not generalizable and  does not represent the views of teachers and principals nationwide.
section_title: Appendix II: Other Sources of Information Districts Use To Identify Effective Education Practices
subsections: section_title: Estimated percent of district officials that find this source Very Useful
paragraphs: Estimated percent of  district officials find this  source not at all useful  80.6  (76.46, 84.79) 77.4  (72.85, 81.84)
16.6  (12.73, 20.53) 14.9  (11.30, 19.13)
2.3  (0.90, 4.67) 6.2  (3.79, 9.58)
0.5  (0.06, 1.67) 1.5  (0.48, 3.55)
77.3 (72.93, 81.74) 74.5  (69.97, 79.08) 70.9  (66.00, 75.72) 65  (59.94, 70.04)
16.7  (12.88, 20.58) 21 (16.74, 25.17) 23  (18.50, 27.50) 23.1  (18.59, 27.57)
4.2  (2.28, 7.14) 3.2  (1.53, 5.84) 3.2  (1.48, 6.02) 6.3  (3.83, 9.66)
1.7  (0.60, 3.73) 1.3  (0.42, 3.05) 2.9  (1.34, 5.49) 5.6  (3.39, 8.72)
64.3  (59.22, 69.41) 54.4  (49.10, 59.63)
27.7  (22.90, 32.45) 30.6  (25.62, 35.53)
4.7  (2.59, 7.81) 7.9  (5.12, 11.59)
3.3  (1.64, 5.85) 7.1  (4.58, 10.52)
51.7  (46.48, 56.97) 50.3  (45.11, 55.49)
39.9  (34.67, 45.06) 35.4  (30.25, 40.45)
6.2 (3.82, 9.53) 9.3  (6.30, 13.16)
2.2  (0.95, 4.19) 5  (2.94, 7.98)
45.9  (40.66, 51.03)
31.3  (26.32, 36.18)
11.5  (8.18, 15.59)
11.4  (8.16, 15.36)
39.8  (34.68, 45.01)
37.8  (32.64, 42.88)
15.79  (11.87, 20.10)
6.7  (4.33, 9.91)
17.9 (13.99, 21.83) 24.4  (20.07 28.76 22.8  (18.40, 27.10) 21.7 (17.63, 25.83) 10.1  (7.34, 13.36) 9.7  (6.74, 13.29) (29.65, 39.77) 14.8  (11.20, 19.09) 13.8  (10.42, 17.68) 43.9  (38.58, 49.11) 14  (10.48, 18.09) 14.5  (10.93, 18.77) (19.41, 28.67) 16.1  (12.25 20.57) 14.2  (10.52, 18.50) 17  (13.06, 21.60) 18.9  (14.54, 23.25) 16.73  (12.84, 21.25) (18.82, 27.86) 44.7  (39.37 49.96) 49.3  (44.03, 54.62) 17.4  (13.22, 21.59) 57.1  (51.77, 62.40) 59.1  (53.84, 64.33)
section_title: Appendix III: IES and WWC Response to Expert Panel Recommendations
subsections: section_title: lES is considering an appropriate mechanism and time for conducting a complete review of the WWC. lES believes that the first 2 years of the current contract necessitated a tremendous development effort to transfer the infrastructure of the Clearinghouse in year one from one contractor to another, and in year two, to complete reviews in a consistent manner that began under the original contract. Now that the Clearinghouse is more clearly in the production phase, this may be the appropriate time to plan for a complete review.
paragraphs: lES is currently considering having the WWC develop a standard  for assessing crossover compliance, following the process recently  used to revise its attrition standard.
Currently, the WWC documents crossover reported in studies.  Principal investigators have discretion to use this information to  determine whether a study represents a reasonable test of the  intervention. Evidence of crossover and assignment  noncompliance is documented in the intervention report and its  appendix table A.1. Readers can use that information to assess  the findings. lES agrees there is value in adjusting intent-to-treat  effects for compliance, but believes this adjustment is inconsistent  with its goal of having the WWC be transparent in how it reports  findings. Making its own estimates to account for compliance will  lead to differences between what the WWC reports and what is  found in publicly available literature.
Currently the WWC does adjust for clustering when authors report  their findings incorrectly. However, the purpose of the clustering  adjustment is to correct for an analytic problem in the methods  authors use to estimate variances, which generally causes them to  overstate the precision of their findings. In contrast, adjusting for  compliance will yield an alternate estimate of effects that may differ  from the one reported by the study.
Though not based on a standard, WWC practice is for reviewers to  document the counterfactual in study review guides and in  intervention reports (the information is reported in appendix table  A.1). Reviewers routinely send author queries for this information,  if it is not provided in the study.
Considered but not  planning to implement  lES has asked the WWC to assess how other review organizations  report counterfactual information and the utility of incorporating this  information into its reports. IES officials are also considering an  alternative approach that would code information about the  counterfactual in a study into the study database, which then  would generate summary tables that would report results for  studies that have similar counterfactuals. This approach has  downsides as well, since the set of counterfactuals could be quite  varied and many assumptions would have to be made to group  counterfactuals together. We are therefore proceeding cautiously  in making any changes to current WWC practice.
WWC staff consulted with Dr. Schochet to investigate the  possibility of revising the multiple comparison standards.
Dr. Schochet indicated that his report focused on issues related to  multiple comparisons within single studies. It did not tackle issues  related to multiple comparisons issues that may arise when  synthesizing evidence for a set of studies. WWC procedures are  consistent with his report for handling multiple comparisons within  a study.
At the time of the National Board of Education Sciences Expert  Panel’s data collection, the WWC was already reviewing its  attrition standards. The WWC released new attrition standards in  December 2008 in the Procedures and Standards Handbook. The  new standards requires a principal investigator in a topic area to  choose one of two well-specified attrition boundaries, and the  standards include guidance on how to choose between the  boundaries based on the nature of research in the topic area.
The attrition discussion is in the WWC Procedures and Standards  Handbook (Version 2.0) posted on the Clearinghouse’s Web site.
IES is considering options for collecting and documenting potential  conflicts of interest.
Sources of funding are rarely included in published documents  beyond government and foundation support. An alternate source  of information for tracking potential conflicts of interest would be for  the WWC to request that study authors identify their source of  funding, which would provide the WWC with a basis for flagging a  potential conflict of interest. Any effort would depend on  cooperation from authors because the WWC has no leverage to  formally require authors to declare potential conflicts (which some  academic journals require as a condition for publication). WWC’s  experience to date is that study authors frequently fail to respond  to requests for additional information, and IES officials expect that  many study authors likewise will not respond to requests for  information about funding sources, or may judge that it is not in  their proprietary interest to provide the information. Currently the  WWC only queries authors in cases where the Clearinghouse  needs additional information. Querying all authors and tracking  their responses would increase costs for intervention reports.
Another potential option is to ask developers, when they are  reviewing the list of studies WWC found during the literature  search for comprehensiveness, to note any studies that they  funded.
The current version of the handbook gives guidance on standards  for random assignment in simple cases. The next version of the  handbook (forthcoming in 2010) will provide guidance and  examples for multilevel settings, with explicit guidance on  acceptable practice and potential issues with random assignment  in a multilevel setting.  lES asked the WWC to review the search process. The  WWC now takes steps to ensure that search records are  maintained. Each team and the library maintain a record of  conducted searches. More documentation on the process will be  included in the forthcoming revision of the handbook.
The WWC is undertaking a pilot using five recent evidence reports  in different topic areas. Because WWC screeners are encouraged  to pass to the next stage any study for which they are uncertain  about eligibility, the proportion of eligible studies that are excluded  is the salient error rate (the other source of error is when screeners  include an ineligible study in a review, but this error is then offset  by the review).
IES officials are not aware of any established standards for  acceptable error rates (there are tradeoffs between making Type I  vs. Type II errors relating to cost), but will examine this issue  further. If the screening error rate is larger than the lES and the  WWC believes is acceptable, IES officials will assess whether  additional training or two screeners is appropriate given the  different costs and benefits of each approach.
Currently, reference lists for WWC intervention reports include all  studies, both eligible and ineligible, located in the search process.  Ineligible studies are flagged with the primary reason for not  qualifying for further WWC review. Intervention reports do not list  materials such as product descriptions or reviews of products that  are deemed not relevant to the intervention being reviewed.
To make the number of studies (both eligible and ineligible studies)  more apparent to readers, the WWC will add a text box to  intervention reports located in front of the listing of reports. The  text box will summarize the number of studies that met different  conditions (this approach currently is used for reports in which  none of the studies meet standards). The box will serve the same  purpose as a flow chart but the codes used to describe the final  status for reports will be the same ones currently used in the  citation appendix. The WWC plans to begin including the text box  in reports released in 2010 and thereafter.
Ideally, the primary source for reanalyses of data would be study  authors. However, as noted above in response to recommendation  2(v), it is common for authors not to respond to the WWC’s  requests for additional information. Reanalyzing the data also  would require additional effort by the authors and would run into  difficulties when studies are dated or are based on data that has  been destroyed to comply with confidentiality or privacy  restrictions.
The WWC recently undertook a survey of published clustering  estimates. It found that the WWC’s current default clustering  correction is consistent with published estimates for achievement  and behavioral outcomes. The WWC will continue to monitor  research developments on this topic.
There are, of course, many possible ways to summarize evidence.  Given its intended broad and primarily nontechnical audience, the  WWC’s current approach is designed to be transparent and easily  explained. lES believes that having the WWC conduct its own  analyses to estimate intervention effects, as statistical meta- analyses do, would be inconsistent with these goals.
However, as an alternative to modifying the WWC’s main approach  for reporting findings, lES is considering having the WWC conduct  supplemental meta-analyses related to specific questions of  interest, and releasing these findings as a separate report that  would complement intervention reports. For example, a report  could analyze whether computer mediated approaches to teaching  reading are more or less effective than approaches that rely solely  on teachers, based on already-released interventions reports.  Having a separate report enables the WWC to continue using its  current transparent approach, while also using statistical  techniques that combine evidence in other ways.
Topic area protocols are available on the topic area home pages.
Just after the expert panel’s report, the WWC released its  Procedures and Standards Handbook in December 2008. A  revision currently is under way that will include more detail on the  screening process. See the response to (3) above related to  results of the screening process and the response to (5) regarding  the results of the screening process.  lES agrees that these products need to remain distinct. Practice  guides are on a separate Web site tab that separates them from  intervention reports. The next revision of the handbook  (forthcoming in 2010) will include a chapter describing the practice  guide development process and how it is different from that of the  evidence reports.
The recently released guide on What Works for Practitioners also  provides more information on reports and practice guides, and the  WWC is preparing a video tutorial that will explain the differences  to users.
The WWC tries to keep abreast of developments in the field, for  example, by routinely checking materials from the Cochrane  Collaborative when developing new standards or approaches.
Most recently, the WWC has undertaken the following outreach  efforts to connect with other organizations conducting systematic  reviews:    The WWC sponsored a forum on research methods in  December 2008 that featured speakers from the National  Academy of Sciences, the National Cancer Institute, and the  Cochrane Collaboration.
In June 2009 WWC staff attended the Cochrane conference on  practice guides in June to learn about state of the art methods in  research synthesis and practice guides.
The WWC is presenting a workshop on WWC standards at the  upcoming annual conference of the Association of Public Policy  and Management.
The WWC has also met with six international contacts (from  Sweden, Denmark, Hungary, England, Interamerican Development  Bank, and Trinidad/Tobago) in response to inquiries about how  governments or organizations could implement their own  clearinghouse operations.
Recently the WWC began a webinar series to disseminate its new  practice guides. The webinar includes researchers and  practitioners in its audience.
The WWC convened two groups of researchers to develop its  forthcoming standards on single-subject designs and regression  discontinuity designs. It will continue to bring together researchers  as needs for new standards are identified. This approach will  continue to be used for developing new standards.
section_title: Appendix IV: Comments from the Department of Education
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making key  contributions to this report include Elizabeth Morrison (Assistant  Director), Nagla’a El-Hodiri (analyst-in-charge), James Ashley, Carl  Barden, James Bennett, Valerie Caracelli, Laura Henry, Geoffrey King, Jill  Lacey, Luann Moy, Robert Owens, Cathy Roark, Stephanie Shipman, Kate  Van Gelder, and Craig Winslow.
section_title: Related GAO Products
paragraphs: Program Evaluation: A Variety of Rigorous Methods Can Help Identify  Effective Interventions. GAO-10-30. Washington, D.C.: November 23, 2009.
Teacher Quality: Sustained Coordination among Key Federal Education  Programs Could Enhance State Efforts to Improve Teacher Quality.  GAO-09-593. Washington, D.C.: July 6, 2009.
Teacher Preparation: Multiple Federal Education Offices Support  Teacher Preparation for Instructing Students with Disabilities and  English Language Learners, but Systematic Departmentwide  Coordination Could Enhance This Assistance. GAO-09-573. Washington,  D.C.: July 20, 2009.
No Child Left Behind Act: Education Actions Could Improve the  Targeting of School Improvement Funds to Schools Most in Need of  Assistance. GAO-08-380. Washington, D.C.: February 29, 2008.
Program Evaluation: Strategies for Assessing How Information  Dissemination Contributes to Agency Goals. GAO-02-923. Washington,  D.C.: September 30, 2002.
The Evaluation Synthesis. PEMD-10.1.2. Washington, D.C.: March 1992.