# Presentation: 247221

## Penny Wise and Effect Size Foolish

- Greg J. Duncan
- Northwestern University

## “Penny Wise and Effect Size Foolish”

- Greg Duncan and Katherine Magnuson
*Child Development Perspectives*, forthcoming

## Consider the Tennessee Star class size experiment

- Assigned children to classes that averaged 15 vs. 22 students for a mean of 2.3 years
- Schanzenbach (forthcoming) estimated a **.15** standard deviation achievement impact on ACT test scores at the end of high school.
-  According to Cohen, .15 is a decidedly small effect size
- Are small classes bad policy?

## Costs and benefits of Tennessee Star

- The smaller class translated into additional per pupil expenditures of **$11,500**
- The economics literature suggests that a one standard deviation increase in test scores produces a ~20 percent increase in lifetime earnings
-  .15 standard deviation achievement impacts translate into lifetime earnings gains of **$17,000**, some 1.5 times the intervention’s cost
- Other possible benefits (e.g., crime reduction) might add to benefit total

## In general, there is no “in general”

- Inexpensive programs with small effects may generate more benefits than costs and thus be worthwhile public investments
- Expensive programs with big effects may cost more than they are “worth”

## Tricks of the trade

- Consider **total** rather than just **taxpayer** costs and benefits
- Total social cost and benefits = participant + taxpayer costs and benefits
- Consider a wide array of potential benefits
- => measure more than intended program targets
- E.g., an early behavioral intervention might reduce grade failure or placement in special education, and boost test scores

## Tricks of the trade - II

- Measure outcomes that can be linked to important social costs
- E.g., grade failure, reduced crime, higher productivity (earnings)
- Try to measure “spillover” benefits and costs
- E.g., Does improving one child’s behavior enable classmates to learn more?
- Do immunizations prevent epidemics?

## Tricks of the trade - III

- Measure outcomes that cannot be monetized
- E.g., tolerance, citizenship
- After monetary benefits and costs have been tallied up, ask whether these other benefits or cost might change conclusions

## Tricks of the trade - III

- Measure outcomes that cannot be monetized
- E.g., tolerance, citizenship
- After monetary benefits and costs have been tallied up, ask whether these other benefits or cost might change conclusions
- Succumb to the cruel truth of discounting
- Distant benefit dollars are worth much less than today’s cost dollars

## The utility of order of magnitude estimates

- On cost side, staff costs usually dominate
- Hours of profession time per subject – 10, 100, 1000?
- Order of magnitude per subject costs?
- On benefit side, can any conceivable valuation of benefits exceed costs?

## Example: Perry Preschool

- 1-2 year center-based learning-focused curriculum for 3-4 year olds
- 2.5 hours per day
- Four teachers for 20-25 students
- Weekly visits to parents

## Perry Preschool at age 27

| Major benefits | For Participant | For Public | Total |
| --- | --- | --- | --- |
| K-12 education, e.g, grade retention | 0 | 8,434 | 8,434 |
| Earnings | 40,001 | 11,063 | 51,634 |
| Decrease in crime | 0 | 154,762 | 154,762 |
| ... | ... | ... | ... |
| Total Benefits | 38,924 | 176,276 | 215,200 |
|  |  |  |  |
| Cost of Program | 0 | -15,166 | -15,166 |

## Cautions

- Avoid the “Perry Preschool Shuffle”:
- Few if any early childhood investments are this profitable and many fail to generate benefits that exceed costs (Aos et al., 2004)
- Quality matters – less intensive programs do not appear to be as profitable
- Model programs may not scale up (TN Star and the CA class size initiative)

## Some best bet programs based on evaluation evidence

- VERY speculative
- Many guesses about long-run benefits

## developingchild.net

## Slide 16

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |

## Slide 17

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
|  |  |  |  |  |  |  |
| Access to health care |  |  |  |  |  |  |
| Nurse-Family Partnership for high-risk families |  |  |  |  |  |  |
|  |  | High quality Pre-K | K-5 aligned to Pre-K | Proven programs and curricula |  |  |
|  | Healthy and safe child care |  |  |  |  |  |

## Access to health care

- Brain architecture can be damaged by:
- Alcohol
- Cocaine
- Environmental toxins
- “toxic” stress
- Pediatric visits can identify:
- Early hearing, vision and other problems

## Slide 19

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
| Work supports for low-wage work |  |  |  |  |  |  |
| Access to health care |  |  |  |  |  |  |
| Nurse-Family Partnership for high-risk families |  |  |  |  |  |  |
|  |  | High quality Pre-K | K-5 aligned to Pre-K | Proven programs and curricula |  |  |
|  | Healthy and safe child care |  |  |  |  |  |

## Nurse-family partnerships 

- Targeted to very high risk pregnancies
- Weekly visits by a trained nurse beginning in 2nd trimester through neo-natal period; less frequent after that
- Most effective for low SES first-time mothers
- Less intensive versions are often ineffective

## Slide 21

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
| Work supports for low-wage work |  |  |  |  |  |  |
| Access to health care |  |  |  |  |  |  |
| Nurse-Family Partnership for high-risk families |  |  |  |  |  |  |
|  |  | High quality Pre-K | K-5 aligned to Pre-K | Proven programs and curricula |  |  |
|  | Healthy and safe child care |  |  |  |  |  |

## High quality pre-K 

- Perry Preschool evidence is well known, but how relevant today?
- Study of pre-K programs in five states:
- .25 sd impacts for receptive vocabulary and math
- .65 sd impact on print awareness
- Tulsa pre-K program:
- .80 sd impacts on pre-reading skills
- .40 sd impacts for math

## Slide 23

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
| Work supports for low-wage work |  |  |  |  |  |  |
| Access to health care |  |  |  |  |  |  |
| Nurse-Family Partnership for high-risk families |  |  |  |  |  |  |
|  |  | High quality Pre-K | K-5 aligned to Pre-K | Proven programs and curricula |  |  |
|  | Healthy and safe child care |  |  |  |  |  |

## Slide 24

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Prenatal | 0-2 | 3-5 | K-5 | 6-8 |  | 9-12 |
| Work supports for low-wage work |  |  |  |  |  |  |
| Access to health care |  |  |  |  |  |  |
| Nurse-Family Partnership for high-risk families |  |  |  |  |  |  |
|  |  | High quality Pre-K | K-5 aligned to Pre-K | Proven programs and curricula |  |  |
|  | Healthy and safe child care |  |  |  |  |  |

## Impacts of Welfare Programs on Young Children’s Achievement

**Human Capital Development**

**Work First**

**No Impacts on Family Income**

**Earnings Supplements**

**All with Impacts on Income**

## Policy should be guided by:

- Convincing evaluation designs
- Focus on benefits relative to costs, not just effect sizes
- Wide-ranging look at policy options

## greg-duncan@northwestern.edu