# Presentation: 670351

## Developing a Severity-of-Disability Scale and Modeling Early Reading and Math Performance in a Longitudinal Study of Preschoolers with Disabilities

- Elaine Carlson
- Tamara Daley
- Frank Jenkins
- Westat
- The Pre-Elementary Education Longitudinal Study is funded by the U.S. Department of Education.

## PEELS design

- 6-year national longitudinal study of preschoolers with disabilities, currently in its fourth year.
- Designed to describe
- the children,
- the services they receive,
- their transitions from early intervention to preschool and preschool to elementary school, and
- their early academic and functional skills.
- Sample includes 3,104 children who were ages 3-5 and were receiving special education services in 2003-04.
- Produces statistical estimates that generalize to the national population of children with disabilities ages 3-5.

## PEELS design, cont’d

- Data collection
- a one-on-one assessment of early academic skills,
- a parent telephone interview,
- mail questionnaires to children’s teachers and school-, district-, and state-level administrators, and
- indirect assessments of students’ social skills, problem behaviors, academic skills, and motor skills

## Sample design

- Nationally representative sample of 223 LEAs (208 in Wave 1, plus 15 added in Wave 2)
- LEAs stratified by
- enrollment size
- geographic region
- wealth

## LEA participation

- Main sample
- 709 released for recruiting
- 245 agreed in 2001
- 189 participated in 2003-04 (Wave 1)
- Nonresponse sample
- 32 of 464 non-recruited LEAs selected
- 19 participated in 2003-04 (Wave 1)
- Supplemental sample to add 1 previously nonparticipating SEA
- 15 of 24 participated in 2004-05 (Wave 2)
- Total Sample = 223 LEAs

## Family participation

- Wave 1 main and nonresponse samples
- (ages 3-4 in 2003-04)
- 52,871 on sampling frame
- 5,330 selected
- 4,072 eligibility determined
- 2,906 agreed (81% of those eligible)

## Family participation

- Wave 2 supplemental sample
- (ages 4-6 in 2004-05)
- 7,727 on sampling frame
- 542 selected
- 433 eligibility determined
- 198 agreed (63% of those eligible)
- Total sample = 3,104 families

## PEELS schedule

## Unweighted response rates, by wave

|  |  | Wave |  |
| --- | --- | --- | --- |
|  | 1 | 2 | 3 |
| SEA questionnaire | 100% | - | - |
| LEA questionnaire | 84% | - | - |
| Principal/program director questionnaire* | 73% | 77% | 56% |
| Teacher questionnaire | 79% | 84% | 84% |
| Parent interview | 96% | 93% | 88% |
| Child assessment | 96% | 94% | 93% |
|      English direct assessment | 84% | 87% | 88% |
|      Alternate assessment | 11% | 7% | 5% |

- -Not applicable
- * QED data were used to supplement these data, bringing percentage of children with some school data in waves 1, 2, & 3 to 94%, 95%, & 94%, respectively

## Demographic characteristics

| Gender |  |
| --- | --- |
|   Male | 70.7 |
|    Female | 29.3 |
| Race/ethnicity |  |
|    American Indian/Alaska Native | 3.3 |
|    Asian | 3.2 |
|    Black or African American | 14.2 |
|    White | 84.6 |
|    Pacific Islander | 0.8 |
|    Hispanic, any race | 21.0 |

- NOTE Demographic and household characteristics are for the unweighted sample.
- SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), “Parent Interview (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

## Household characteristics

| Mother’s education |  |
| --- | --- |
|    <H.S diploma | 18.9 |
|    H.S. diploma or GED | 31.0 |
|    Some college | 29.4 |
|    4-year degree or higher | 21.0 |
| Household Income |  |
|    $20,000 or less | 26.3 |
|    $20,000-$40,000 | 28.3 |
|    >$40,000 | 45.3 |

- SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), “Parent Interview (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

## Primary disabilities

| Speech or language impairment | 49.8 |
| --- | --- |
| Developmental delay | 26.4 |
| Autism | 6.7 |
| Mental retardation | 4.0 |
| Learning disability | 2.5 |
| Other health impairment | 2.4 |
| Orthopedic impairment | 1.6 |
| Emotional disturbance | 1.1 |
| Low incidence disability | 5.5 |

- NOTE Low incidence includes visual impairment, hearing impairment, deaf-blindness, traumatic brain injury, multiple disabilities.
- SOURCE: U.S. Department of Education, National Center for Special Education Research, Pre-Elementary Education Longitudinal Study (PEELS), ‘Early Childhood and Kindergarten Teacher Questionnaires (Wave 1 for main and nonresponse samples, Wave 2 for supplemental sample),” previously unpublished tabulation.

## Developing a Severity-of-Disability Scale

## Severity

- Under IDEA, children classified according to one of 13 disability categories
- Children in each category have a range of abilities
- Heterogeneity within categories often overlooked
- Categorical labels especially problematic for preschoolers

## Approaches to measurement

- Pediatric health conditions (PEDI, WeeFim)
- Self-care, mobility, and social cognition
- Require a certain threshold of impairment
- Single disability conditions in children
- Hearing, vision, language, depression
- ICIDH (WHO, 1980)
- Published as a research tool
- Not widely adopted for use

## Approaches to measurement

- ABILITIES Index (Simeonsson & Bailey, 1991):
-     Severity in 9 domains
**A**udition
**B**ehavior and social skills
**I**ntellectual function
**L**imbs
**I**ntentional communication
**T**onicity
**I**ntegrity of physical health
**E**yes
**S**tructural status

## Approaches to measurement (cont’d)

- Since its original publication, 6 new domains added (Simeonsson, 2006):
- Regulation of attention,
- Regulation of activity level, and
- Regulation feeling/emotions,
- Academic skills,
- Motivation, and
- Impulse control

## ABILITIES index

- In Special Education Expenditures Project (SEEP) (Chambers et al., 2004) :
- Federal disability category able to explain only 10% of the variation in total expenditures
- In contrast, the ABILITIES Index alone was able to account for 40% of the variation in total educational expenditures for special education students
- Including the federal disability categories increased the variance accounted for by 2%

## Question of current study

- 	To what extent do functional markers of severity of childhood disability predict measurement of child outcomes?

## Overall method

- Identify items from parent interview similar to ABILITIES domains
- Create composite domains where necessary
- Examine distribution of severity within the 15 domains
- Examine regressions of the 15 domains on multiple outcomes
- Create short version of index using 6 domains
- Compare 6 domain and 15 domain versions of index to outcomes to select better version
- Compare 6 domain version of index with other indicators of severity

## Outcome measures

- Cognitive
- PPVT-III
- Woodcock-Johnson III: Letter-Word Identification
- Woodcock-Johnson III: Applied Problems
- Social/Behavioral
- PKBS Social Competence
- PKBS Problem Behaviors
- Alternate Assessment
- ABAS Conceptual Domain
- ABAS Practical Domain
- ABAS Social Domain

**Wave 1 data**

## Creating the severity measure

- Used parent report from Wave 1 CATI
- Created 4-point scale for each domain (normal/typical, mild, moderate, severe)
- 11 domains already with 4 point scales:
- 	4 levels of severity
- 4 domains with 3 point scales:
- 	1  1,  2  2,   3  4 to preserve range

## Creating the severity measure

- Some domains matched to a single question
- Use of arms (within limbs domain): “How well does {child} use {his/her} arms and hands for things like throwing, lifting, or carrying?”
- Motivation: “Some children try to finish things, even if it takes a long time. How much does this sound like {child}... ”
- For domains with multiple questions, worked with ABILITIES Index author to best reflect original intention of domain

## Example: Combining Items to Create ‘Communication with Others’

- When {CHILD} talks to people {he/she} doesn’t know well, is {he/she}
- 1:  Very easy to understand
- 2:  Fairly easy to understand
- 3:  Somewhat hard to understand
- 4:  Very hard to understand
- 5:  DOES NOT OR WILL NOT TALK AT ALL

- Compared with other children about the same age, how well does {CHILD} make {his/her} needs known to you and others? Communication can be any form, for example crying, pointing or talking. Would you say {he/she}
- 1:  Communicates just as well as other children
- 2:  Has a little trouble communicating
- 3:  Has a lot of trouble communicating
- 4:  Does not communicate at all?

- 1:  Communicates just as well as other children and very easy to understand
- 2:  Some difficulties communicating or being understood
- 3:  Moderate difficulties communicating or being understood
- 4:  Does not communicate at all or very hard to understand

## ResultsPopulation weighted percentages (n = 2,986)

- Audition, vision, use of arms, legs, regulation of emotions
- 80-95% in normal/typical category
- Inappropriate or unusual behavior, overall health, social skills, use of hands, understanding
- 50-60% in normal/typical, 20-30% mild

## ResultsPopulation weighted percentages (n = 2,986)

- Motivation, regulation of attention, regulation of activity level,
- 20-30% in normal/typical, 33-42% mild, 16-30% severe
- Communicating with others
- 29% in normal/typical, 9% mild, 45% moderate, 17% severe
- Intellectual function
- 10% in normal/typical, 43% mild, 32% moderate, 14% severe

## Regressions on outcome variables

- All predictors entered simultaneously; examined beta weights and bivariate correlations between predictors and each outcome
- Significant predictors of at least three outcomes:*cognition*, *communicating with others*, *understanding*, *overall health*, and *regulation of activity level*
- Significant predictor of two outcomes:*regulation of attention*
- The remaining severity domains generated a mixture of significant and nonsignificant associations
- No significant loading on any outcome variables: *use of hands and arms* and *use of legs*

## Creation of indices

- Index A, 15 items:  Sum of all domains
- Index B, 6 items: *cognition*, *communicating with others,* *understanding, overall health,* *regulation of activity, regulation of attention** *
- No significant differences in correlations between Index A and outcomes and Index B and outcomes

## Index A and B correlations with outcomes

|  | Index A
15 var* | Index B
6 var* |
| --- | --- | --- |
| PPVT | -.32 | -.36 |
| Letter-Word Identification | -.22 | -.26 |
| Applied Problems | -.40 | -.45 |
| PKBS Social Skills Composite | -.47 | -.43 |
| PKBS Problem Behavior Composite | .35 | -.35 |
| ABAS Conceptual Domain | -.53 | -.46 |
| ABAS Practical Domain | -.53 | -.43 |
| ABAS Social Domain | -.40 | -.35 |

- *The associated p-values are less than .0001 for all coefficients.

## Final severity measure items

- Cognition
- Communicating with others
- Understanding
- Overall health
- Regulation of activity
- Regulation of attention

## Distribution of final severity measure

## Validation: correlations between Index B and other indicators

- Wave 1 Parent report: Age at which children began receiving special education or therapy service*r (2,802)* = -.22, *p* _<_ .0001
- Wave 1 Teacher report: Amount of modification needed to curriculum materials*r (248)* =  .42, *p* _< _.0001
- Wave 1 Teacher report: Number of services the child receives in school *r *(2,014)= .37, *p* _< _.0001

## Validation: comparison of mean scores on severity measure

- From teacher/parent declassification measure, Wave 1: Children remaining in special education*M* = 13.2Children no longer receiving special education 	*M* = 10.7 , *p *< .0001
- From assessment measures, Wave 1:Children taking the alternate assessment *M* = 16.3Children completing the direct assessment*M* = 12.4, *p *< .0001

## Summary

*To what extent do functional markers of severity of childhood disability predict measurement of child outcomes?** *
- Of the 15 domains examined, most were significant predictors of at least two outcomes
- An index of only six variables was as effective as the longer version
- Severity was significantly correlated with intervention variables
- Severity differentiated children in two groups

## Modeling Early Reading and Math Performance

## Hierarchical analysis

- PEELS has children in naturally-occurring hierarchies:
- time points within children
- children within districts
- districts

- What we want to know
- What factors relate to children’s cognitive growth over 3 years?
- What we have
- observations of PEELS children over 3 years.
- Yearly information about child’s SES, health, severity, and services received
- Yearly measures of 3 academic outcomes
- Adapted Peabody Picture Vocabulary Test (PPVT)
- Woodcock-Johnson III: Letter-Word Identification
- Woodcock-Johnson III: Applied Problems

## Problem: How to make sense out of longitudinal data 

- Hierarchical data is clustered: i.e., repeated measures are not independent observations. Standard regression assumes independent observations.
- Ordinary repeated measures analyses do not allow for missing time points or clustering.
- Repeated cross-sectional analyses ignore the growth of individual children. Mean growth is not the same as growth of individuals.

## Solution: Hierarchical linear modeling

- Data is modeled at 3 levels of hierarchy at the same time.
- Most of the clustering in the sample is accounted for, leading to correct statistical tests.
- Focus is on individual growth profiles. Modeling seeks to explain differences in growth between children.

## Hierarchical structure of data

## HLM model

- Level 1: Repeated observations within child
-   	is the outcome for individual *i* measured at wave *t* .
- 	is the age of the child at time at time *t*.
- 	is the growth intercept: average achievement for the individual
- 	is the growth curve slope: How much the outcome changes over years.
- 	is a random error term.

## HLM modelLevel 2: Children nested within districts.

- is an intercept for child *ji*,

- is the individual growth curve for child *ji*

-  to

- are child factors that predict growth.

-    is the deviation from average achievement for each child.

-   is the deviation from average growth for each child.

## HLM modelLevel 3: Districts

- Where,
-         is the achievement status of district j ,
-         is the  grand mean of achievement, and
-         is the district effect on achievement.

## Hypothetical growth curve HLM

- Individual slopes estimated for high & low severity children
-                                                                                         <- Low severity
-                                                                                                             <- High severity

- Individual slopes estimated for high & low severity children
-                                                                                         <- Low severity
-                                                                                                             <- High severity

## Four Sets of Predictors

- Predictors are added in four sets of similar variables.  These sets include
-  SES: e.g., mother’s education, SES scale
-  Severity: e.g., severity scale, age services
-     started
-  Health: e.g., child’s general health, health scale
-  Services: e.g., % time in a regular classroom, parent
-     involvement scale

## Predictors of PPVT growth

**Accounted for 22 % of PPVT growth**

**Note: Controlled for age and cohort**

| Predictor | Factor | Effect | Prob |
| --- | --- | --- | --- |
| Intercept |  | 229.55 | .00 |
| Slope |  | 54.24 | .00 |
| # Years in special education | Severity | -1.37 | .01 |
| Severity of disability scale | Severity | -.78 | .00 |
| Ease of transition | Service | -1.85 | .01 |
| % Time in a regular classroom | Service | .03 | .03 |

## Predictors of Letter-Word growth

**Accounted for 57 % of Letter-Word growth**

**Note: Controlled for age and cohort**

| Predictor | Factor | Effect | Prob |
| --- | --- | --- | --- |
| Intercept |  | 318.18 | .00 |
| Slope |  | 37.76 | .00 |
| Years in high poverty school | SES | 2.10 | .00 |
| Mother had some education after HS | SES | 2.92 | .00 |
| # Years in special education | Severity | -1.62 | .00 |
| Age services started | Severity | .09 | .02 |
| Severity of disability scale | Severity | -.69 | .00 |
| Ease of transition | Service | -2.61 | .00 |
| % Time in regular classroom | Service | .08 | .00 |
| Parent involvement scale | Service | 1.49 | .03 |

## Predictors of Applied Problems growth

| Predictor | Factor | Effect | Prob |
| --- | --- | --- | --- |
| Intercept |  | 393.26 | .00 |
| Slope |  | 24.00 | .00 |
| Household Income | SES | -1.26 | .03 |
| Child is Hispanic | SES | 1.53 | .03 |
| Mother had some education after HS | SES | 1.42 | .02 |
| Parent SES scale | SES | 1.38 | .01 |
| Problems with health | Health | 2.50 | .00 |
| Health scale | Health | 1.18 | .05 |
| Child’s general health | Health | 2.49 | .00 |

## Predictors of Applied Problems growth (continued)

**Accounted for 12 % of Applied Problems growth**

**Note: Controlled for age and cohort**

| Predictor | Factor | Effect | Prob |
| --- | --- | --- | --- |
| # Years in special education | Severity | -1.02 | .00 |
| Age services started | Severity | .09 | .00 |
| Severity of disability scale | Severity | -.24 | .02 |
| % Time in regular classroom | Service | .03 | .00 |
| Parent involvement scale | Service | .75 | .06 |

## Summary 

- Scores increased from 11⁄2 to 2 standard deviations over 3 years
- SES, severity, health, & service predictors accounted for 12% to 57% of growth

| Percent of growth accounted for |  |
| --- | --- |
|  |  |
| PPVT | 22% |
| Letter-Word | 57% |
| Applied Problems | 12% |

## Summary (cont.)

- Service-related predictors of growth
- % Time in a regular classroom
- 	Positively related to growth for all outcomes
- Parent involvement
- 	Positively related for 2 of 3 outcomes
-     (Letter-Word & Applied Problems)
- Ease of transition
- 	Positively related for 2 of 3 outcomes
-     (PPVT & Letter-Word)

## Summary (cont.)

- Other predictors of growth
- # Years child was in special education
- 	Predicted lower growth for all outcomes
- Parent’s rating of severity
- 	Predicted lower growth for all outcomes
- Mother’s education
- 	Positively related to growth for 2 of 3 outcomes
-     (Letter-Word & Applied Problems)

## WEBSITE:  WWW.PEELS.ORG