# Presentation: 139608

## Evaluation of State and Local Education Programs: A New Grant Program at the Institute of Education Sciences

- Allen Ruby
- Associate Commissioner
- Policy and Systems Division
- National Center for Education Research

## Agenda

- Institute of Education Sciences and the NCER/NCSER grant programs
- Research priorities
- Purpose of new grant program
- Type of Programs and Policies to be evaluated
- The Research Narrative
- Grant and review information

## IES Organizational Structure

- Office of the Director

- National Board for Education Sciences

- National Center for Education Statistics

- National Center for Special Ed Research

- National Center for Education Research

- National Center for Education Evaluation

## Long-Term Research Goals

- Develop or identify programs, practices, policies, & approaches that enhance academic achievement
- Identify what does _not_ work and thereby encourage innovation and further research
- Explain variations in effectiveness of  education programs, practices, policies, & approaches

## Outcomes of Interest(Dependent Variables)

*Prekindergarten*
- Readiness for schooling
- Developmental outcomes for infants and toddlers with disabilities

## Outcomes of Interest(Dependent Variables)

*Kindergarten through Grade 12*
- Reading and writing
- Mathematics and science
- Behaviors and social skills that support learning in school and successful transitions to post-school opportunities
- Functional skills that support independent living for students with disabilities

## Outcomes of Interest(Dependent Variables)

*Postsecondary Education*
- Enrollment in postsecondary education
- Completion of postsecondary education
*Adult Education*
- Basic reading, writing, and mathematics

## Conditions of Education(Independent Variables)

- Curriculum
- Instruction
- Assessment
- Quality of the education workforce
- Systems-level programs and policies

## Research and Research Training Grant Programs

- Education and Special Education Research Grant Programs
- Predoctoral and Postdoctoral Research Training Grant Programs
- National Research and Development Centers
- Statistical and Research Methodology in Education
- Evaluation of State and Local Education Programs and Policies

## Evaluation of State and Local Education Programs and Policies Grant Program

- Purpose
- Evaluate programs/policies selected by states and districts and implemented by SEAs and LEAs
- Foster rigorous evaluation by states and districts and use of the results in decision-making
- Promote research community’s interest in state and district actions
- Foster partnerships between states/districts and research community in support of these rigorous evaluations
- Provide useful information to other states and districts

## State and Local Programs and Policies	

- Implemented by SEA or LEA
- Not by other agencies that may work in schools or with school-age populations
- To improve student achievement
- Directly: Outcomes of Interest discussed earlier
- Indirectly: intermediate outcomes that are expected or known to affect Outcomes of Interest

## State and Local Programs and Policies

- Fully developed
- All supports in place
- All materials available for distribution
- Evidence that it is already developed or that it will be fully developed by start date of grant

## State and Local Programs and Policies

- Widely implemented
- State(s) or district(s)-wide
- Evidence of current or assured future implementation
- On a sufficient scale to make generalizations
- On a sufficient scale and across a variety of conditions to allow subgroup and moderator analysis
- Under typical implementation conditions

## State and Local Programs and Policies

- Substantial modification of existing practice
- Of state or district existing practice
- Of practice by other states or districts
- Not yet rigorously evaluated
- Adoptable by other states or districts

## Research Narrative	

- 4 Sections
- Significance
- Research Plan
- Personnel
- Resources
- 25 pages, single spaced

## Significance 

- Describe the program/policy in detail
- Assume peer reviewers know nothing about it
- Detail all components
- Describe who is responsible for implementing each component
- Describe outcomes to be affected

## Significance

- Provide theory of change
- How is the program/policy expected to improve student achievement
- What is it to directly and indirectly change
- Graphics (logic models, flow charts) often help
- Describe outcomes to be affected
- Student achievement
- Intervening outcomes

## Significance

- Describe how the treatment is different from current program/policy
- Describe what the comparison group will receive
- Show that treatment is substantially different
- Describe why treatment will lead to major improvements and increase student achievement

## Significance

- Show wide implementation: current or expected
- Sufficient scale
- Variety of conditions
- Typical implementation conditions
- Show feasibility of implementation
- Fully developed
- Supports in place
- Can use evidence from other districts/states
- Describe the source and provide evidence of availability of funding for implementation

## Research Plan

- State research questions/hypotheses
- Describe sample
- Define sampling universe
- Selection procedures
- Exclusion and inclusion rules
- Reducing attrition

## Research Plan

- Randomized Control Trial Design Preferred
- Note unit of randomization and reason for it
- Describe process for random assignment
- Staggered rollout or variation in treatment design may offset concerns of no intervention for the control group

## Research Plan

- If cannot use RCT, justify why
- Alternatives to RCT
- Regression discontinuity designs
- Well designed quasi-experimental designs
- Power
- Provide and justify details of power analysis and method used to calculate power
- Peer reviewers should be able to check power calculations

## Research Plan

- Outcome measures (outcomes of interest)
- Relevant to states, districts, and schools
- Provide reliability, validity, and appropriateness
- Link to theory of change
- Fidelity of implementation
- Measures used (reliability and validity)
- Describe how to be implemented
- Fidelity in both treatment and control groups

## Research Plan

- Comparison group
- Who makes up comparison group: how will they be similar/different from treatment group?
- What do they receive in place of the treatment?
- How will contamination be avoided?

## Research Plan

- Mediating and moderating variables
- Identified in theory of change
- Describe how they will be measured in both treatment and control
- Detail data analysis procedures
- Provide model
- Show how based on design
- Note software to be used
- Also address mediators, moderators, and fidelity

## Research Plan

- Cost Analysis
- Document financial costs
- A cost-effectiveness or cost-benefit analysis not required

## Personnel

- Show that have personnel with expertise in: 1) content, 2) intervention, 3) research design, 4) implementation of design, and 5) analysis.
- If research organization and district/state working together, show that district/state personnel have large role

## Resources

- Show the institutions involved have the capacity to carry out the evaluation
- Show that the key decision-makers in the state/district fully understand and support the evaluation

## Grant Information	

- $500,000 - $1.2 million a year (Direct + Indirect)
- Can ask for less or more: key point is to justify budget
- Up to 5 years
- Funds only for evaluation (not for implementation)
- Application Due: 10/2/08 by 4:30 PM EST
- Earliest Expected Start Date: 7/1/09

## Grants.gov: http://www.grants.gov/

- Register ahead of time
- Application package posted 8/4/08
- Use correct application package
- CFDA 84-305E2009-1
- Help Line: 1-800-518-4726
- Start/complete before deadline

## Peer Review of Application

- Application is reviewed for compliance.
- Compliant proposals are assigned to a review panel.
- Two or three panel members conduct primary review of each application.
- Competitive applications are reviewed by full panel at the panel meeting.

## Notification

- All applicants will receive email notification of the status of their application.
- All applicants receive copies of reviewer comments.
- If you do not receive an award the first time, consider resubmitting, and talk to program officer.

## Institute of Education Sciences
http://ies.ed.gov

FY 2009 Request for Applications available at:
http://ies.ed.gov/ncer/funding

Application forms will be available at:
http://www.grants.gov

Program Officer
Allen.Ruby@ed.gov
202-219-1591

- _[http://ies.ed.gov](http://ies.ed.gov/)_
**FY 2009 Request for Applications available at:**
- _[http://ies.ed.gov/ncer/funding](http://ies.ed.gov/ncer/funding)_
**Application forms will be available at:**
- _[http://www.grants.](http://www.grants.gov/)__[gov](http://www.grants.gov/)_
**Program Officer**
- _[Allen.Ruby@ed.](mailto:Allen.Ruby@ed.gov)__[gov](mailto:Allen.Ruby@ed.gov)_
- 202-219-1591