# Presentation: 226787

## Case Study:

**Case Study****:**
** **
**National Evaluation of **
**Early Reading First**
**Peter Z. Schochet**

**Decision Information Resources, Inc.**

**Notes:**
Based on report prepared for ASPE, coauthored with Anu and Laura

## Overview

**ERF program**
**Evaluation research questions**
**Regression discontinuity design**
**Conclusions **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## ERF Program

**Part of the No Child Left Behind Act**
**3-year ERF grants provided to a collaboration of preschools**
**Funding focus is on low-income children**
***Goal:***** Enhance the language, cognitive, and early reading skills of preschool children**
** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## ERF Funds Are Intended to: 

**Provide professional development for teachers**
**Create high-quality and print-rich environments **
**Promote the use of scientifically proven literacy methods and instructional materials**
**Identify preschool children at risk for         reading failure**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Study Research Questions

**What are the impacts of ERF on****:**
**Children’s language and literacy?**
**Quality of language and literacy instruction, practice, and materials?**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## KEY DESIGN FEATURES

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Study Focus Is on FY 2003 ERF Grant Applicants

**700 sites submitted pre-applications**
**126 invited to submit full applications  **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Random Assignment Was Not Possible

**ED required that funds be awarded based on *****rankings***** ****of applications**
**Applications were “scored”**
**30 sites were funded with scores ******** 74 **
**96 unfunded sites with scores < 74**
**Scoring criteria were set **_***a priori***_**   **
**Favorable conditions for a RD design  **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## The Ideal ERF RD Design: Compare     “73s” to “75s”: Almost Experimental

_**Cutoff**_

_**Mean for 75s**__** **_

_**Mean for 73s**_

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## But There Are Not Enough 73s and 75s: Need to Include Other Sites

- _Score_                  _Number of Sites_
**42 to 53****	****	****	****22**
**54 to 63****	****	****	****21****	**
_**64 to 73**__**	**__**	**__**	**__**21   **_**Cutoff Value=74**
**74 to 83****	****	****	****18**
**84 to 95****	****	****	****12**

**UNFUNDED**

**FUNDED**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Sampling

**All ****funded**** sites agreed to participate**
**Sorted ****unfunded**** sites by their scores**
** **
**Sites with largest scores contacted first**
**64 sites contacted**
**37 agreed to participate**
**Obtained lists of classrooms in sites**
**Sampled 3 classrooms per site **
**Selected 9 four-year olds per classroom**
** **
**94 percent parental consent rates**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## The RD Method Visually

**Estimated Regression Lines**

**Unfunded**

**Funded**

_**Impact**_

_**Cutoff**_

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Key Identifying Assumption

**There must be a **_***continuous relationship***_** between the outcome measure and the application score  **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Differences-in-Means Estimates Could Be Biased 

**Funded**

_**Spurious Impact**_

** ****= Simple Means**

**Unfunded**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## RD Designs Require Larger Samples Than Experimental Designs

**Controlling for the application score reduces power**
**Design effects are 3 to 4**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## The Correct Functional Form Specification Is Crucial for Obtaining Unbiased Estimates

_**Assume Linearity When the True **_
_**Relationship is Nonlinear**_

**Spurious**
**Impact**

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## There Also Has to Be a Clear Functional Form Relationship

**% of Classrooms in Site That Engage in an Activity  **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Basic Regression (HLM) Model

***Y *****= *****β******0****** *****+ *****β******1*******T *****+ *****β******2*******f(SCORE-74) + u***** **
***Y = *****Outcome measure (****child or teacher level)**
***T***** = 1 if funded site, 0 if unfunded**
***f(SCORE)***** = Function of application score**
***β******1 *****= Impact estimate**
***u***** = Error term accounting for site and**
**         ****classroom-level **_***clustering***_** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Selecting f(SCORE)

**Graph Y on SCORE****  **
**Add SCORE****2****, SCORE****3****, and T*SCORE interaction terms and test for significance**
**Use nonparametric methods**
**Specification tests**
**Impacts = 0 using *****baseline***** data  **
**Impacts = 0 at “artificial” cutoff values**
**Adding covariates should not change impacts  **
** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Interpretation of Impact Estimates

**In the “73-75” model, results pertain **_***only***_** to sites around the cutoff value****  **
**Results “generalize” more broadly using the **_***parametric***_** approach**
**Does this stretch the results too far? **
**But not using the **_***nonparametric***_** approach**
** **
** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Conclusions

**RD designs can produce rigorous impact estimates under the right conditions:**
**Need exogenous “scores” **
**Scores and outcomes must have a smooth relationship that can be credibly modeled**
**But there are limitations to the RD approach:**
**Need larger samples than an experimental design  **
**Generalizability **
**Nonresponse a problem in unfunded sites**
** **
** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## EXTRA SLIDES

** **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Data: Fall 2004 and Spring 2005 (Spring Response Rates)

**Child assessments (97%)**
**Teacher behavioral ratings (96%)**
**Teacher/classroom observations (79%)**
**Parent surveys (69%)**
**Teacher surveys (91%)**
**Center director interviews (88%) **

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Child Assessment Instruments

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?

## Observations and Surveys

**Notes:**
Three main questions about married-parent families’ use of TANF and FSP and how it compares to that of single-parent families
Can we explain differences?  For instance, we might suspect that, even among the eligible population, married-parent families might tend to have higher incomes on average than single-parent families. MP families might tend to live in states with more restrictive work requirements. Do differences in observable characteristics and state policies fully explain differences in eligibility and participation rates?