# Presentation: 116803

## What Use Are International Assessments for States?

- 30 May 2008
- Jack Buckley
- Deputy Commissioner
- National Center for Education Statistics
- Institute of Education Sciences

***This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.***

## Benchmarking states to their international competition
Linking: Gary Phillips’s approach
An Experimental Small Area Estimation Model for PISA

Beyond benchmarking: Policy analysis with international comparative data
Problems for macro policy evaluation
Measurement issues with policy variables
Difficulty of cross-national attitude comparisons

- Linking: Gary Phillips’s approach
- An Experimental Small Area Estimation Model for PISA
**Beyond benchmarking: Policy analysis with international comparative data**
- Problems for macro policy evaluation
- Measurement issues with policy variables
- Difficulty of cross-national attitude comparisons

## “Benchmarking” is the primary use of this type of data

- How can the states get onto the league table?
**Direct estimation**: Test a large sample of students in each participating state
**Statistical linking**: Use information from a similar test to map scores to states
**Small area estimation**: Use a more complex statistical model to predict state scores

## Phillips proposes linking using NAEP

- Advantages:
- Large state-level NAEP program already exists
- NAEP scale and achievement levels already familiar to education policy makers
- Linkages to TIMSS/PIRLS inexpensive and relatively simple given data in hand
- Disadvantages
- NAEP testing grades (and possibly content) preclude linking to PISA
- Not a “real” state test result (c.f. Indiana’s TIMSS results)

## PISA small area estimation as an alternative

- NCES and AIR feasibility study in progress using PISA 2006 science
- Uses the observed data (40 states had a least one school sampled in 2006) plus auxiliary information (from NAEP and CCD) to predict school-level PISA scores
- Weighting by school numbers of 15 year-olds to “roll up” results to the state level

## Pros and cons of the small area approach

- Advantages:
- State-level benchmarking on PISA possible without large state samples
- Could be improved through larger samples without needing the full recommended 50 schools/1500 students
- Might be modified to use information from state tests
- Disadvantages
- Also not a “real” state test result
- Uncertainty of estimates is quite large—retrospective modeling likely to be inadequate
- Depends more on statistical model

## Preliminary small area estimates--large standard errors

- Next step is to search for a design that allows for acceptable standard errors
- Initial analysis suggests that 50 schools/1500 students per state may be larger than necessary, particularly for small states
- Use simulations to examine what happens if only some states have a larger sample

## What else can we do with these data?

- International assessments filled with “policy relevant” variables
- Student, family, teacher, principal, school characteristics
- Student attitudes
- Curriculum information
- School governance measures

## Hedges: Risk of misusing these data is high

- The international assessments are cross-sectional--not well-suited for hypothesis testing or even hypothesis generation
- PISA: assessment content is not tied to curriculum, so particularly hard to examine school policies
- Some researchers have attempted policy evaluation anyway; results are not particularly credible
- Differencing two years of data, country-level
- Instrumental variables approaches
- Fixed-effects (school level) models and related

## Several problematic measurement issues: 

- Unit of analysis: Due to the BiB spiral sampling used in these studies, they may not provide reliable scores at the student, classroom (IEA), or even school or district level
- Timing: PISA is a cross-sectional yield study of 15 y.o.’s in the Fall. So what level of education governance gets the credit or blame for student performance?
- Cross-national: PISA asks who has “considerable responsibility” in school decision making, but the response options do not include a close match to what the U.S. calls the school district

## Cross-national attitude comparisons particularly questionable 

- Analysts have observed apparently counterintuitive results in some cases
- Nations where students generally have better attitudes towards math and more self-efficacy generally score lower on average (e.g. Loveless 2006)
- Within nations, however, most relationships are as expected, suggesting “aggregation bias” and/or differential item functioning cross-nationally
- Other measures (e.g. principals’ perception of resource inadequacy in PISA) are also vulnerable

## Solutions for the cross-national attitude issue?

-  2006 PISA report recognizes the problem: “Comparison across countries should be made with caution as students may not be answering these questions the same way in different countries.”
-  The WHO experience with international self-reported health suggests that adding anchoring questions (vignettes) may allow for correction, but costly in terms of additional data