# Presentation: 037092

## Comparing International Assessments to NAEP

- May 30, 2008
- Eugene Owen
- Senior Advisor, NCES International Activities Program

***This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.***

## Goal of this Presentation

- Present results from comparison studies to elaborate on international assessments:
- How each is related to NAEP and
- What each can contribute to the knowledge base on student performance

## Recent Int’l and Nat’l Assessments

|  | 4th grade | 8th grade | 12th grade |
| --- | --- | --- | --- |
| Reading | NAEP 2007
PIRLS 2006 | NAEP 2007 | NAEP 2005 |
| Mathematics | NAEP 2007
TIMSS 2007 | NAEP 2007
TIMSS 2007 | NAEP 2005 |
| Science | NAEP 2005
TIMSS 2007 | NAEP 2005
TIMSS 2007 | NAEP 2005 |

- 15 year olds

- PISA 2006*

- PISA 2006

- PISA 2006

## Overall Differences

- Assessments have subtle but important distinctions with regard to U.S. curricula
- International assessments provide benchmarks with different groups of countries
- The students being studied may represent different groups of students
- Different assessments are designed to measure at different levels of precision and disaggregation

## Content Differences: General

- Sources of difference
- Framework Organization
- Distribution of items across content areas and topics
- Distribution of items across cognitive skills/ processes
- General consistency with grade-specific content/topics in NAEP framework
- Likelihood of being in U.S. curricula or on NAEP

## Mathematics Frameworks

| NAEP 2005/7
Content Strands
Number Prop./Operations
Measurement
Geometry
Data Analysis and Probability
Algebra
Mathematical Complexity
Low
Moderate
High | TIMSS 2007
Content Domains
Number
Geometric Shapes, Measures (4)/Geometry (8)
Data Display (4)/Data and Chance (8)
Algebra (8)
Cognitive Domains
Knowing
Applying
Reasoning
 | PISA 2006
Overarching Ideas
Quantity
Space/shape
Change and relationships
Uncertainty

Competency Clusters
Reproductions
Connections
Reflections
Situations/Contexts |
| --- | --- | --- |

## Notes and comments on TIMSS-NAEP 2007 math comparisons

- Some differences in distribution across cognitive domains (smaller between NAEP and TIMSS than PISA)
- PISA most consistent with 8th grade topics in NAEP
- Vast majority of TIMSS items are “likely” in U.S. curricula
- Difficulty in placing some TIMSS items on NAEP framework reflects “holes” in NAEP framework
- TIMSS has notable number of pattern problems, and problems requiring drawing symmetrical figures

## Science Frameworks

| NAEP 2005/7
Fields of Science
Life science
Physical science

Earth science
“Knowing and Doing”
Conceptual understanding
Scientific investigation
Practical reasoning
Nature of science
Themes
                     | TIMSS 2007
Content Domains
Life science / biology
Physics
Chemistry
Earth science
Cognitive Domains
Knowing
Applying
Reasoning

Scientific Inquiry | PISA 2006
Knowledge
Of natural world: living systems, physical systems, earth/space systems, technology systems
About science itself: scientific inquiry and explanations
Competencies
Identifying scientific issues
Explaining phenomena scientifically
Drawing conclusions based on evidence
Contexts and Attitudes |
| --- | --- | --- |

## Notes and comments on NAEP-TIMSS science comparisons

- Vast majority of TIMSS items “likely in U.S. curricula”
- Difficulty in placing some TIMSS items on NAEP framework reflects “holes” in NAEP framework
- TIMSS has notable number of “science as context” items

## Notes and comments on PISA-NAEP science comparisons

- PISA items are equally consistent with the 8th and 12th grade NAEP frameworks (about 40% each)
- “Rate” as a concept does not appear in the NAEP framework, whereas it is in numerous PISA items
- The closer PISA comes to its objective of assessing scientific literacy, the harder it is to map items to NAEP
- Over half rated as “not likely to be a NAEP item”

## Thank you