# Presentation: 587710

## Effectiveness of Reading and Math Software ProductsFindings From the First Student CohortMark DynarskiMay 2007

## Study Synopsis 

**Study Synopsis **

**Design**
**Nine reading and six math software products in 132 volunteer schools**
**Treatment teachers could use products, control teachers could not**
**Implementation**
**Companies provided training and other types of support**
**Study purchased various upgrades and hardware components**
**Key Findings**
**Test scores at the end of the school year were not statistically different**
**Few relationships between effects and implementation factors**

## Study Size 

**Study Size **

## Implementation Framework

**Implementation Framework**

**Teacher training ****[O, R]**
**Amount of use ****[I, R]**
**Technical difficulties and teacher support ****[I]**

**Student and teacher roles ****[O]**
**Teaching approach ****[O]**
**On-task behavior ****[O]**
**Consistency with suggested use of  products ****[I]**

- Key: O indicates observations, R records, I interviews

## General Implementation Findings

**General Implementation Findings**

**Nearly all teachers received training and believed it prepared them to use products**
**Difficulties using hardware mostly were minor**
**Total use of software products higher in treatment classrooms**
**When products are used**
**teachers more likely to be “facilitators” and students more likely to work on their own**
**more students on task in math classrooms**

## Difference in Technology Use in Treatment and Control Classrooms: First Grade

**Difference in Technology Use in Treatment and Control Classrooms: First Grade**

## Effects on Classroom Practices 

**Effects on Classroom Practices **

- Percent Difference: Teacher as Facilitator

- 0%

- 25%

- 50%

- 75%

- 100%

**First **
**Grade**

**Fourth**
**Grade**

**Sixth **
**Grade**

**Algebra**

- Note:  * Significantly different from zero at the 0.05 level

- *

- *

- *

- *

## Effects on Classroom Practices 

**Effects on Classroom Practices **

- Percent Difference: Students On Task

- Note:  * Significantly different from zero at the 0.05 level

- 0%

- 10%

- 20%

- 30%

*****

- X

**First **
**Grade**

**Fourth**
**Grade**

**Sixth **
**Grade**

**Algebra**

## Estimating Effects

**Estimating Effects**

**Outcome: spring test score**
**3-level model (students, classrooms, schools)**
**3-level model extended to estimate effects of conditions and practices (implementation)**
**interactions of treatment effect and classroom and school characteristics**

## Effect Sizes By School: First Grade 

**Effect Sizes By School: First Grade **

## Test Scores: First Grade 

**Test Scores: First Grade **

- SAT-9 Reading Score

- -0.05

- 0.05

- 0.15

- 0.25

**Overall****Score**

- EffectSize

- Note:  * Significantly different from zero at the 0.05 level

**Sounds****and Letters**

**Sentence****Reading**

**Word****Reading**

## Test Scores: First Grade 

**Test Scores: First Grade **

- Test of Word Reading Efficiency Score

- -0.10

- 0.00

- 0.10

- 0.20

- 0.30

**Overall****Score**

**Phonemic****Decoding **
**Efficiency**

**Sight Word****Efficiency**

- EffectSize

- Note:  None of the effect sizes is significantly different from zero at the 0.05 level

## Interactions: First Grade

**Larger effects**
**More experienced teachers**
**Smaller student-teacher ratio**

**No relationship**
**Product usage**
**Problems getting access**
**Technical difficulties**
**Computer specialist in school**
**Professional development last year on using technology**
**Poverty, urban area, African-American students, Hispanic,  special education students**

## Test Scores: Fourth Grade 

**Test Scores: Fourth Grade **

- SAT-10 Reading Score

- -0.10

- 0.00

- 0.10

- 0.20

- 0.30

**Overall ****Score**

**Vocabulary**

**Word Study ****Skills**

**Comprehension**

- Note:  None of the effect sizes is significantly different from zero at the 0.05 level

- X

- EffectSize

## Interactions: Fourth Grade

**Larger effects**
**Product usage**
**	**

**No relationship**
**Problems getting access**
**Technical difficulties**
**Computer specialist in school**
**Professional development last year on using technology**
**Poverty, urban area**

## Test Scores: Sixth Grade 

**Test Scores: Sixth Grade **

- SAT-10 Math Score

- -0.10

- 0.00

- 0.10

- 0.20

- 0.30

**Overall**** Score**

**Procedures**

**Problem**** Solving **

- Note:  None of the effect sizes is significantly different from zero at the 0.05 level

- EffectSize

## Interactions: Sixth Grade

**No statistically significant relationships**

## Test Scores: Algebra 

**Test Scores: Algebra **

- ETS Algebra Exam

- -0.30

- -0.10

- 0.10

- 0.30

**Overall Score**

**Concepts**

**Processes**

**Skills**** **

- Note:  None of the effect sizes is significantly different from zero at the 0.05 level

- EffectSize

## Interactions: Algebra

**Smaller effects when teachers had technical difficulties**** **

## Study Tradeoffs

**Study Tradeoffs**

**15 reading and math products studied**
**Many products and types of technology not in the study **
**Precision to detect small effect sizes**
**Average effect reported**
**Experimental design **
**Teachers have not used these products in**
**current classrooms**

## Second Year Study

**Second Year Study**

**10 products, data from 77 schools, treatment and control teachers with new cohort of students **
**Will study relationship between teacher experience using products and effects**
**Effects reported for products**