# Presentation: 059484

## Child Outcome Measuresin the Study ofChild Care Quality

- Martha Zaslow, Tamara Halle, Laurie Martin, Natasha Cabrera, Julia Calkins, Lindsay Pitzer, &
- Nancy Geyelin Margie

## Two Key Questions	

- Are we choosing the outcomes that are important to study with respect to child care quality?
- Are there problems with the characteristics of the child outcome measures used that might hinder finding associations between quality and outcomes?

## Review Approach

- Reviewed studies in Vandell & Wolfe (2000) and updated with studies through December 2005
- Total of 65 studies
- Within each outcome area, looked at domains of child outcomes studied, as well as at more specific constructs within domains and measures
- Reviewed for specific psychometric issues

**Notes:**
Mention that we used NEGP dimensions regardless of how the authors defined what their outcomes were.  

## Review Approach

- For Domains, Used Five School Readiness Dimensions:
- Physical Well-being and Motor Development
- Social and Emotional Development
- Language Development
- Approaches to Learning
- Cognition and General Knowledge

## Review Approach

- For Constructs Within Domains:
- Discrete child outcomes within domains that were actually included in analyses
- For Measures:
- Tool actually implemented. Often used to cover more than one construct
- Example:
- Within domain of social and emotional development, constructs of internalizing and externalizing behavior problems, measured using the Child Behavior Checklist

## Review Approach

- Psychometric Issues Identified at Level of Measure
- 0*	*No methodological concerns
- A	Ambiguity (aspect of quality or child outcome?)
- B	No reliability information provided (not standardized or established measure)
- C	No validity information provided (not standardized or established measure)
- D	No reliability or validity data for culture measure used in
- E	No reliability or validity data for modified measure
- F	Adequacy of reliability can be questioned
- G	Adequacy of validity can be questioned
- H	Detail or specificity is lacking about content of the measure
- I	Composite score used without empirical basis
- J	Part or all of measure’s content not clearly related to underlying construct being measured

## Number of Studies Addressing Each Domain of School Readiness

-        _Domain_
- Physical Well-being &
-    Motor Development
- Social & Emotional Development
- Language Development
- Approaches to Learning
- Cognition &	 General Knowledge

- _Number of Studies_
- _(out of 65)_
- 3
- 52
- 33
- 16
- 35

**Notes:**
80% focused on social and emotional development

54% on lanugage and early literacy

51% on cognition and general knowledge

Together/ nearly 2/3 of studies focused on either language and early literacy or cognition and general knowledge

Less focus onapproaches towards learning: 25%
Minimal focus on physical well-being and motor development

## Issues of Alignment

- Very limited focus on physical well-being and motor development unexpected given emphasis on health in measures of quality
- For example, in ECERS-R
- Rating of meals and snacks with focus on nutritional value, sanitary conditions, and accommodations for food allergies
- Rating of space for gross motor play with focus on safety of space
- Rating of adequacy of supervision for gross motor activities
- Health and safety practices emphasized

**Notes:**
Peg Burchinal says that health measures may be hard to get because they are low frequency.  

Should these gaps be studied?  Are we expecting that if we improve the quality of care that we’ll get kids with better task orientation?  This gets back to the question of what is our logic model for this area.

## Issues of Alignment

- Lack of a logic model for linkages between child care quality and child outcomes?
- We appear to have a broad logic model: structural features of quality are linked with process quality which in turn predicts child outcomes (NICHD Study)
- We seem to be lacking models of how _specific_ aspects of quality are linked with _specific_ child outcomes

## Issues of Alignment

- Difficulty relating findings _across_ studies if child outcome constructs and measures differ widely
- Found that specific constructs studied and measures used to study them recur in areas of language as well as cognition and general knowledge
- Not the case for measures of socioemotional development, especially positive aspects

## Examples of Positive Socialand Emotional Constructs

- Attachment to mother and the provider
- Emotional expressiveness
*Easy* vs. *Difficult* temperament
- Attractiveness
- Playfulness
- Popularity
- Leadership
- Assertiveness
- Efficacy

- Peer vs. Adult orientation
- Positive separation and reunion behavior
- Social bids
- Social cognition
- Self esteem
- Considerateness
- Empathy
- Acceptance by peers
- Conflict negotiation
- Emotional well-being

**Notes:**
Long list of constructs as well as long list of measures (57 used across 52 studies). Newly developed observational systems

Not an issue for problematic aspects of social and emotional development. Internalizing, externalizing and total behavior problems. Specific measures recur.

Much more overlap across studies in constructs and measures used in language and cogntion. E.g., constructs of receptive and expressive language. PPVT and WJR Letter-Word Identification and other subtests.

## Additional Issues with Constructs in Social and Emotional Domain

- Are these measures of the environment or of an outcome specific to a child?
- Are these measures malleable? (e.g., shyness, temperament, hesitancy)
- Is there a positive end of a continuum? (e.g., peer orientation)
- Self-regulation is highlighted in research, but not often studied.

**Notes:**
Secure attachment to caregiver an outcome or a reflection of reciprocal interactrion?  Does playfulness reflect an outcome or what environment affords?

Should malleability be a prerequisite?

Should it be clear that there is a positive end of continuum?

Self-regulation now receiving greater focus

Caveat: Experimentation is positive!  However, reaching consensus on what is important to study would permit examination of replication of measures across studies.  
Which leads to questions of pyschometrics...

## Presence of Methodological Concerns

-  At least one methodological concern regarding measures in studies focusing on:
- Social & Emotional Development
- Language Development
- Cognition &	 General Knowledge

- _Percent of Studies_
- 71%
- 36%
- 43%

## Most Frequently Occurring Problems

- Measures of Social and Emotional Development
- Absence of validity information
- Modification of a measure without providing reliability or validity information regarding modified measure
- Language Development
- Absence of validity information
- Lack of information on content of measure
- Cognition and General Knowledge
- Lack of information on content of measure

## Most Frequently Occurring Problems

- Lack of information presented more than evidence of limitations with measures
- No indication of pervasive methodological problems
- But issues most prevalent for measures of social and emotional development

## Next Steps

- Need for better coverage of health, approaches to learning
- Need for focus on alignment: explicit logic model, consistency across studies, level of analysis
- Need for work on measures of social & emotional development, especially positive aspects

**Notes:**

Logic model -- there is a need to go beyond existing school readiness dimensions to consider additional important child outcomes, such as cultural identification.  

## New Efforts

**Roundtable on Measuring Quality in Early Childhood and School-Age Settings: at the Junction of Research, Policy and Practice**
- December 4-6 2006
- Sponsored by Child Care Research Team within Office of Planning, Research and Evaluation, US DHHS

## New Efforts

- At Quality Roundtable:
- State representatives in states implementing or developing Quality Rating Systems were open to the idea of including more specific measures of child care quality (instead of, or in addition, to broad measures)
- At same time, state representatives felt strongly that measures focusing on health aspects of child care quality were needed.

## New Efforts

- Follow-up steps to Quality Roundtable:
- Meeting summary to be posted on Research Connections as well as Child Trends websites (currently under review)
- Compendium of Measures of Quality in Early Care and Education Settings, being developed by Child Trends

## New Efforts

- New analyses will be carried out to identify aspects of quality that are the strongest predictors of specific child outcomes
- Papers will be prepared looking at aspects of quality that have received limited focus, such as health and culture

## New Efforts

- New work on developing child outcome measures funded by a consortium led by NICHD
- Committee on Developmental Outcomes and Assessments for Young Children recently launched at National Research Council