Draft

Exploring a Cross-Agency Platform 

for Judging Evidence:

Resources for Federal Agencies

Informal Federal Interagency Workgroup on

A Common Evidence Framework

February 2013
DRAFT Common Evidence Framework 

Workgroup

∗ Agencies involved in the initial draft

∗ U.S. Department of Education

∗ Institute for Education Sciences

∗ U.S. Department of Health and Human Services

∗ Office of the Assistant Secretary for Planning and 

Evaluation

∗ U.S. Department of Labor

∗ Chief Evaluation Office

∗ Will expand to include other agencies
DRAFT

Workgroup Purpose

∗ Establish common evidence framework for Federal 

agencies

∗ First stage: Develop a framework for evidence 

standards
DRAFT

Guiding Principles

∗ Build on existing agency work

∗ Establish standards that can be used for both reviewing 

and conducting evaluations and related research

∗ Establish standards that are applicable to all types of 

evaluation and research designs, and all types of results

∗ Establish standards that are useful and relevant for all 

Federal agencies and assess areas where agencies may 

need to augment or adapt

∗ Establish an approach to efficiently share evaluations and 

research that have been reviewed
DRAFT Framework for Evidence and Evaluation Reviews

 

 

 

Types of Research Assessment of Single Studies

Studies 

Exploratory Analysis 

(Early Stage Design, 

Problem Identification, 

Descriptive Analysis , 

Hypothesis Generation)

 

Developmental Designs 

(Concept Proofs, 

Pilots/Pre-Pilots)

Implementation & 

Operational Studies 

(Process Analysis, 

Implementation 

Science, Institutional 

Analysis, Performance 

Analysis)

Formal Impact 

Evaluations of 

Interventions, 

Programs, Models, & 

Systems (Efficacy Tests, 

Scale-up Tests, 

Replications )

Cost, Cost-Benefit & 

Cost Effectiveness 

Analysis

Standards for 

Research Quality 

and Results 

Infrastructure 

for Creating 

Reporting & 

Review 

Guidelines 

(including 

template) for 

Coding 

Individual 

Studies

Central 

Collected 

Library of 

Reviewed/ 

Coded Studies 

(linking 

clearinghouses, 

sharing reviews 

and guidelines 

across agencies) 

Analytic Products 

From 

Assessments

 

Intervention 

Effectiveness 

Summaries/ 

Evidence Reviews 

(e.g., Teen 

Pregnancy 

Prevention, 

CrimeSolutions, 

What Works 

Clearinghouse)

 

Cost-Effectiveness 

Analyses 

 

Implementation, 

Performance & 

Operational 

Analyses

 

Technical 

Assistance 

Products (e.g., 

Promising 

Practice Guides)

 

Policy Analyses & 

Syntheses 

Actions

Decision-Making 

for Replication 

Includes assessing:

•Evidence of 

Impacts

• Implementation 

Readiness

• Site Readiness 

• Fit 

Decision-Making 

for Research

Includes assessing: 

•Need for Refining, 

Improving, 

Enhancing 

Evaluations

• Implications/ 

Recommendations 

for New Research 

Studies

Decision-Making 

for Innovation

Includes assessing:

• Key Research 

Gaps

• Promising 

Approaches from 

Exploratory Work 

that Require 

Demonstration
DRAFT

General Categories of Standards

∗ Research Design:

∗ Study questions

∗ Conceptual framework or logic model

∗ Evidence/theoretical base

∗ Evaluation design

∗ Analytic Approach:

∗ Outcome/impact measures

∗ Sampling (e.g., sample size, sample selection method, baseline equivalency, 

stratification, attrition)

∗ Data collection methods (e.g., instrumentation, follow-up)

∗ Analytic methods (e.g., internal/external validity, modeling, subgroup analysis)

∗ Operational setting/context (e.g., implementation fidelity)

∗ External Review:

∗ Peer review

∗ Funder monitoring

∗ Advisory/technical group review

∗ Reports/Products:

∗ Presentation quality and clarity
DRAFT

Reporting and Review Guidelines

∗ Examples of agencies/programs using and updating 

specific guidelines to review studies:

∗ Crime Solutions, DoJ

∗ Prevention Research Synthesis, HHS/CDC

∗ What Works Clearinghouse, ED

∗ Teen Pregnancy Prevention (TPP) Evidence Review, HHS

∗ FindYouthInfo.gov – provides youth programs from Crime Solutions and 

the TPP Evidence Review

∗ CLEAR - Clearinghouse of Labor Evaluation and Research, DoL

∗ Workforce System Strategies, DoL

∗ Self Sufficiency Research Clearing House, DoL

∗ Home Visiting Evidence of Effectiveness, HHS
DRAFT Ratings, Scores, Levels of 

Evidence....

∗ We have agency/program-specific coding of studies but there are many underlying 

common dimensions

∗ “Study Design Ratings” (HHS-TPP): High (high quality RCTs), Moderate (RCTs with high attrition 

and QEDs with baseline equivalence), Low Study rating; (these ratings are based on study 

design, attrition, baseline equivalence, reassignment, confounding factors)

∗ “Evidence Standards” (ED-WWC): Meets standards without reservation (high quality RCTs), 

meets standards with reservations (RCTs with attrition and QEDs with equivalence), does not 

meet standards (insufficient causal validity)
DRAFT

Library of Coded Studies

∗ Definition:

∗ A central collection of all of the individual studies that have been 

reviewed and coded by any clearinghouse

∗ Purpose: 

∗ If studies in the library have been coded for similar elements, 

coded versions of individual studies can be pulled out from the 

library for different analytic products, and different 

clearinghouses can capitalize on existing work

∗ Contents:

∗ A searchable list of each study reviewed, including links to:

∗ The protocol that was followed for conducting the review (which will identify population of interest, age level, eligible outcomes, etc.)

∗ Outcomes that were considered

∗ Study rating for each outcome: meets standards, doesn’t meet 

standards, etc.

∗ Study details that have been recorded on the coding template (for example, the Study Review Guide for the WWC)
DRAFT

Ideas for Next Steps

∗ Share the draft framework and gather feedback

∗ Develop some core codes and checklists for the general 

categories of standards that can be adopted in new work 

across agencies for:

∗ Clearinghouses and resource libraries

∗ Proposals and study designs

∗ Assessing additional types of studies beyond causal impact

∗ Reporting of results for studies that do not meet standards

∗ Develop specifics for others areas of the framework