# Presentation: 146255

## What do international assessments measure: PISA

- Raymond J. Adams

- Washington DC, May 30 2008

***This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.***

## What is PISA?

- Programme for International Student Assessment
- Funded by the Organisation for Cooperation and Development (OECD)
- International data collection has been managed and led by ACER since its inception

## Developing PISA

- 1995: OECD countries sought
- comparative assessment of yield of education systems
- a basis for defining educational standards cross-nationally
- 1997: OECD countries adopted strategy and financial framework to...
- monitor a broad range of curricular and cross-curricular outcomes...
- ...	within a comparative framework established collaboratively by countries
- 1998: Contract let

## Participation

- 2000
- OECD 28, non-OECD 4
- PISA+, 11 non-OECD
-  170,000 students
- 2003
- OECD 30, non-OECD 12
-  250,000 students
- 2006
- OECD 30, non-OECD 27
- 14365 schools
- 398,750 students (representing a cohort of over 22 million)
- 2009
- OECD 30, non-OECD 35 (*plus Scotland*)

## Plus: Dubai, Kazakhstan, Moldova

## PISA survey cycle

## PISA survey cycle

- eg: self-regulation, metacognitive

- eg: Reading Activities, motivation, interest, engagement

## PISA survey cycle

## PISA survey cycle

## PISA survey cycle

## Separating policy and science

- Two consortia commissioned to run PISA 2009
- Core A

- Australian Council for Educational Research

- Unit for the Analysis of Educational Systems and Practices. University of Liege, Belgium

- Westat, USA

- National Institute for Educational Research, Japan

- The German Institute for International Educational Research

- cApStAn Linguistic Quality Control

## Separating policy and science

- Core B
- Netherlands National Institute for Educational Measurement (CITO)
- University of Jyvaskyla
- University of Twente
- French Ministry of Education

## Sets policy objectives and priorities

- Sets budget
- Monitors adherence to policy objectives
- Guides analysis and reporting of results
- Establishes quality standards
- Adjudicates on breaches of quality standards

## Consortia are commissioned to run PISA

## Innovations in PISA

- International, intergovernmental ownership
- primary focus on public policy issues
- sustained commitment
- collaborative development
- Age not grade-based target population
- Substance
- not common denominator of national curricula
- not whether students have learned
- whether they can use what they have learned
- criterion-based reporting

## PISA’s Literacy Orientation

- Projective (rather than reflective) approach to instrument development
- Not specifically related to curriculum
- It is about authentic application of what is learned
- In contrast to assessments that are like instructional exercises
- A broadening, not narrowing notion

## Literacy Orientation

- Domain definitions include two components:
- each domain defined, in terms of knowledge and skills needed in adult life, not merely in terms of mastery of the school curriculum.
- emphasis is placed on the mastery of processes, the understanding of concepts and the ability to function in various situations related to real life.

## Basic Methodology

- Target population: 15-year-olds in school
- Students are spread over classes and grades
- Target sample sizes of 5250
- 150 schools of 35 students
- 50 schools required for national sub-entities
- Rotated two-hour written tests
- 13 booklets
- 3 domains (two minor and one major)
- About 200 items in total
- About 11 responses per item in each school
- Students respond to an average of 13 minor domain items and 30 major domain items
- Some students are not assessed in minor domains

## Basic Methodology

- Student questionnaires
- School questionnaires
- Parent questionnaires
- No teacher questionnaires
- Students from multiple classrooms
- Focus is not limited to *taught* subjects

## Focus of the Results

- Yield for well-defined populations and domains
- Reading, Mathematics, and Science every three years
- Sub-domains every nine years
- National sub-group performance
- Profiles encompassing broad domain definitions
- Literacy orientation, habits, behaviours, attitudes
- Trends
- Focus on comparisons of equity
- Eg Gender, minority status, socio-economic status
- Relations between national structures outcomes
- Eg Tracking, Assessment, funding and Decision making policies

## Shall I slice the pizza into four or eight pieces?

- Make it four ! I could never eat eight !