Common Standards for 

Education Research and 

Development Proposals: NSF and 

the U.S. Department of Education 

 2/28/13 
A Framework for Producing and Sharing 

Knowledge and Evidence Resources 

2 
A Joint Committee of ED and NSF Membership 

Guided the Development of these Standards 

 Ex Officio: Joan Ferrini-Mundy Assistant Director, EHR and John Easton, Director, Institute of Education Sciences

Co-Chairs: Janice Earle, EHR and Rebecca Maynard, National Center for Evaluation and Regional Assistance, ED

Members: 

 ED: Elizabeth Albro, IES; Karen Cator, Office of Technology; Michael Lach, Office of the Secretary; Loy Lesnick, Center for Evaluation and 

Regional Assistance; Ruth Curran Nield, National Center for Evaluation 

and Regional Assistance; Lynn Okagaki, IES; Jefferson Pestronk, Office of 

Innovation and Improvement; Anne Ricciuti, IES; Allen Ruby, IES; Deborah 

Speece, National Center for Special Education.

 NSF: Jinfa Cai, EHR-DRL; Edith Gummer, EHR-DRL; Jim Hamos, EHR-DUE; 

Gavin Fulmer, EHR-DRL;Janet Kolodner CISE and EHR-DRL; Susan Winter- SBE

3 
How did Idea for Common Standards 

Originate? 

 Recommendations from OSTP and guidance 

from OMB to establish agreed-upon 

benchmarks across the research and 

development continuum that will allow 

“apples to apples” comparisons of research 

design, rationale, measurement instruments, 

outcomes and evaluation among other 

characteristics of R and D projects aimed at 

improving STEM education

4 
Goals of the Common Standards Project 

 Improve the quality and pace of 

findings from education research and 

development proposals 

 Develop an education infrastructure 

that supports more rapid and 

efficient knowledge development

 Aid NSF and ED in making informed 

choices about where to invest scarce 

research and development dollars

 Provide clarity for the field

5 
What’s Included in Education Research 

and Development? 

1. Proposals that investigate some application 

of practice (e.g., a new curriculum, a new 

technology)

2. Proposals that study the impact of an 

education practice or policy (e.g., what 

effects are the new common core state 

standards in mathematics having on 

instruction?)

6 
What’s Included, cont. 

3. Proposals that design and test new 

approaches to teaching and learning

4. Studies that seek to test, develop, or refine 

theories of education and leaning that 

advance knowledge acquisition about the 

factors that facilitate or impede learning.

7 
What’s NOT included 

1. Education outreach activities (e.g., 

conducting sessions with schools/ 

classrooms on some aspect of science)

2. Scholarship programs such as Noyce or GRF

3. Projects that aim to build capacity that are 

based on evidence but where there is no 

empirical question under investigation

8 
Joint Committee’s Assumptions 

 Knowledge development is not linear, but multidirectional

 Scientific contributions by multiple researchers, 

research teams and practitioners are needed

 An efficient research and development program 

may require fewer studies as one moves from 

foundational to impact evaluation studies

 Implementation research is important for design 

and development projects and for impact 

evaluations

 Outcomes may be education and/or learning in 

multiple settings

9 
Types of Studies 

 Foundational research and development

◦ Fundamental knowledge that may contribute to 

teaching and/or learning

 Early stage/exploratory

◦ Examines relationships among constructs to 

establish logical connections

 Design and development

◦ Design and iteratively develop particular 

interventions (programs, policies, practices or 

technologies); can also pilot test fully developed 

intervention to see if it achieves its intended 

outcomes 

10
Types of Studies 

 Efficacy Studies

◦ Estimate the impacts of strategies under optimal 

conditions of implementation

 Effectiveness Studies

◦ Examine implementation and estimate impacts 

similar to conditions of routine practice but still on 

a limited scale

 Scale-up Studies

◦ Explores implementation and estimates impacts 

under conditions that prevail under wide-scale 

adoption 

11
Document Organization 

 Introductory material

 NSF and ED missions- distinct but 

complementary

 Standards for six types of research and 

development proposals

◦ Foundational 

◦ Early Stage and Exploratory 

◦ Design and Development

◦ Impact Studies

 Efficacy 

 Effectiveness 

 Scale-up 

 Conclusions and recommendations for use

12
General Structure 

Section Heading Content & Example Subheadings 

Purpose • Gives an overview of the Study Type.

• No subheadings

Justification Standards • Significance for policy and/or practice

• Basis in prior theory & evidence

Evidence Standards • Likely project outcomes

• Plans for the research & development

• Plan for external feedback

13
Example: Design and Development-1 

Purpose

1. Design and Development Projects are intended to develop and 

test the promise of new or improved strategies for achieving 

particular educational objectives. Typically they occur in four 

stages: (1) development of an intervention (e.g., instructional 

approaches, curricula, professional development, technology, 

school-wide programs, or education policies) based on a wellspecified theory of action appropriate to a well-defined end user, 

(2) creation of measures to assess the implementation of the 

intervention, (3) collection of data on the feasibility of 

implementing the intervention in typical delivery settings by 

intended users, and (4) conducting of a pilot study to examine 

the promise of the project for generating the intended beneficial 

outcomes. 

2. In some cases funders will expect that all four stages will be 

completed within a given project; in other cases, design and 

development projects may entail sequential projects

14
Example: Design and Development-2 

Justification Standards

Theoretical & 

Empirical Basis

The proposal should include a strong theoretical and 

empirical justification for the proposed intervention 

(i.e., program, policy, technology or practice). If the 

theoretical basis rests on evidence related to 

individual features or components, the project should 

provide a compelling rationale for how combining 

these features or components into a new intervention 

is expected to achieve the intended outcomes.

15
Example: Design & Development-3 

Evidence Standards

Project 

Outcomes

Project outcomes should include:

1. a fully developed version of the proposed design-research 

(including all materials necessary for its implementation);

2. a well-specified theory of action, including evidence 

supporting or refuting key assumptions of the original 

theoretical basis for the intervention;

3. descriptions of the major design iterations and the resulting 

evidence to support or question key assumptions about the 

theory of action;

4. description of the adjustments to the theory of action and 

intervention design that resulted from design testing;

5. measures for assessing the feasibility of implementing the 

intervention in an authentic education delivery setting and 

data demonstrating the project’s success in such 

implementation; and 

6. pilot data on the intervention’s promise for generating the 

intended beneficial student outcomes. 

16
Example: Design & Development- 4 

Evidence Standards

Research Plan

The research plan should describe the method for developing 

the intervention to the point where it can be used by the 

intended end-users (iterative development process), the method 

for collecting evidence on the feasibility that end users can 

implement the intervention in an authentic education delivery 

setting (evidence of feasibility of implementation), and the 

method for obtaining pilot data on the promise of the 

intervention for achieving the expected outcomes (pilot study). 

The discussion of the development process should include how 

the initial version of the intervention will be developed, and how 

the intervention will be improved through iterative design and 

testing cycles that entail refining and retesting the usability of 

the intervention (or components of it). 

 ∶

17
Example: Impact Evaluations—Efficacy 1 

18

Purpose

1. Intended to generate reliable estimates of the ability of a fullydeveloped intervention to achieve intended outcomes. The three 

types of impact evaluations vary with regard to the conditions 

under which the intervention is implemented and the populations 

to which the findings generalize.

2. Efficacy studies are intended to determine if an intervention can 

work to improve outcomes rather than to determine what the 

expected impact of the intervention is if implemented under 

conditions of routine practice. 

3. To this end, Efficacy Studies test interventions under what is 

sometimes called “ideal” conditions (e.g., conditions that include 

more implementation support or more highly trained personnel 

than would be expected under routine practice or implemented 

among a more homogeneous sample of students, teachers, 

schools, and/or districts than is typical).
Example: Impact Evaluations—Efficacy 2 

Justification Standards

Theoretical & 

Empirical Basis

Should demonstrate one or more of the following:

1. Empirical evidence of the promise of the intervention 

from a well-designed and implemented pilot study (e.g., 

a study conducted as part of a design and development 

project)

2. Evidence of promise based on one or more well-designed 

and implemented early-stage research demonstrating 

empirical support for critical links in the theory of action 

3. Evidence that the intervention is widely used 

4. Evidence of favorable impacts from a previous efficacy 

study if in a different setting or with a substantively 

different population group.

19
Example: Impact Evaluations—Efficacy 3 

Evidence Standards

Project 

Outcomes

Detailed descriptions of:

1. Study goals

2. Design and implementation

3. Data collection and quality

4. Analysis and reporting plans

Study designs should:

1. Meet WWC design standards

2. Provide for rich descriptions of the intervention and 

counterfactual conditions

3. Implementation analysis and descriptive relating of impacts 

and implementation findings

20
Example: Impact Evaluations—Efficacy 4 

Evidence Standards

Research Plan

The research plan should identify and justify the following: 

1. Study design that will be used to estimate causal impact of 

the intervention

2. Key outcomes of interest

3. Minimum relevant effect (MRE) size that has policy or 

practical relevance

4. Setting(s) and target population(s) for the study

5. Sample design and power to detect impacts as small as the 

MRE

6. Data collection plan (procedures, measures, and strategies 

for ensuring reliability and validity)

7. Plans for documenting implementation, comparison group 

practices, and study context

8. Analysis and reporting plan

21
Outreach Activities and Next Steps 

 At NSF:

◦ Brown Bag for all NSF POs April 31, 2012

◦ Briefings for all EHR Divisions: November 27, 2012 (DRL); October 4 (HRD), 2012; 

October 10, 2012(DGE); October 11, 2012 (DUE)

◦ Briefing planned for SBE, January 2013

◦ Briefings planned for other directorates, January-February 2013

◦ Incorporated in the FY 12 GPRA goal on K-12

◦ SMaRT update, December 2012

◦ Internal clearance process, December – January 2013

 At ED

◦ Clearance through IES

 Outside NSF and ED

◦ Proposed sessions at professional meetings in 2013 -- American Education Research 

Association, National Association for Research on Science Teaching, National Council 

of Teachers of Mathematics, Society for Research on Educational Effectiveness

 Plans for External Review of Common Standards by outside experts, 

January 2013

◦ To be organized by IES at ED

22
Other Connections 

 Relates to the “design principles” for CoSTEM

Strategic Plan

 Mentioned in MOU with US Department of 

Education

 Under discussion at OMB

23
Possible uses of Common Standards 

 Cited in program solicitations as applicable 

for research and development proposals and 

posted on web sites of both agencies for 

reference

 Some language from Common Standards 

included in relevant program solicitations

 Cited by other governmental organizations 

such as OSTP, OMB

24
Suggestions? 

 Develop additional documents such as FAQ 

sheets? 

 Identify programs in the Directorate to which 

these standards apply?

 Conduct additional briefings?

 Other??

 NSF Contact: Janice Earle (jearle@nsf.gov)

25