Problem Solving

U.S. Department of EducationInstitute of Education SciencesNCES 2005-003

PISA 2003 Results From theU.S. PerspectiveHighlights

onal Outcomes of Learning in Mathematics Literacy and Problem Solving: PISA 2003 Results From the U.S. Perspective 2004
International

Outcomes of Learning

in Mathematics

Literacy and Problem

Solving: 

PISA 2003 Results From the

U.S. Perspective

Highlights U.S. Department of Education

Institute of Education Sciences

NCES 2005–003

Mariann Lemke

National Center for Education Statistics

Anindita Sen

Erin Pahlke

Lisette Partelow

David Miller

Education Statistics Services Institute

Trevor Williams

David Kastberg

Leslie Jocelyn

Westat

December 2004
U.S. Department of Education

Rod Paige

Secretary

Institute of Education Sciences

Grover J. Whitehurst

Director

National Center for Education Statistics

Robert Lerner

Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting,

analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics

on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent,

reliable, complete, and accurate indicators of education status and trends; and report timely,

useful, and high quality data to the U.S. Department of Education, the Congress, the states,

other education policymakers, practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in

communicating information effectively. If you have any comments or suggestions about this or

any other NCES product or report, we would like to hear from you. Please direct your comments to:

National Center for Education Statistics

Institute of Education Sciences

U.S. Department of Education

1990 K Street NW

Washington, DC 20006-5651

December 2004

The NCES World Wide Web Home Page is http://nces.ed.gov

The NCES World Wide Electronic Catalog is http://nces.ed.gov/pubsearch

Suggested Citation

Lemke, M., Sen, A., Pahlke, E., Partelow, L., Miller, D., Williams, T., Kastberg, D., Jocelyn, L. (2004).

International Outcomes of Learning in Mathematics Literacy and Problem Solving: PISA 2003

Results From the U.S. Perspective. (NCES 2005–003). Washington, DC: U.S. Department of

Education, National Center for Education Statistics.

For ordering information on this report, write:

U.S. Department of Education

ED Pubs

P.O. Box 1398

Jessup, MD 20794-1398

Call toll free 1-877-4ED-PUBS or order online at http://www.edpubs.org

Content Contact:

Mariann Lemke, (202) 502-7314

E-mail: Mariann.Lemke@ed.gov

International Outcomes of Learning in Mathematics Literacy and Problem Solving

ii
Summary

The Program for International Student Assessment (PISA) is a system of international

assessments that measures 15-year-olds’ capabilities in reading literacy, mathematics literacy, and science literacy every 3 years. PISA was first implemented in 2000 and is carried

out by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries. Each PISA data-collection effort

assesses one subject area in depth, even as all three are assessed in each cycle so that participating countries have an ongoing source of achievement data in every subject area (figure 1). In addition to the major subject areas of reading literacy, mathematics literacy, and

science literacy, PISA also measures general or cross-curricular competencies such as

learning strategies. In this second cycle, PISA 2003, mathematics literacy was the subject

area assessed in depth, along with the new cross-curricular area of problem solving. Major

findings for 2003 in mathematics literacy and problem solving are provided here, as well as

brief discussions of student performance in reading literacy and science literacy and

changes in performance between 2000 and 2003.

U.S. Performance in Mathematics Literacy and Problem Solving

In 2003, U.S. performance in mathematics literacy and problem solving was lower than the

average performance for most OECD countries (tables 2 and 3). The United States also

performed below the OECD average on each mathematics literacy subscale representing a

specific content area (space and shape, change and relationships, quantity, and uncertainty).

This is somewhat different from the PISA 2000 results, when reading literacy was the

major subject area, which showed the United States performing at the OECD average

(Lemke et al. 2001).

Along with scale scores, PISA 2003 also uses six proficiency levels (levels 1 through 6, with

level 6 being the highest level of proficiency) to describe student performance in mathematics literacy (exhibit 5) and three proficiency levels (levels 1 through 3, with level 3 being the

highest level of proficiency) to describe student performance in problem solving (exhibit 9).

In mathematics literacy, the United States had greater percentages of students below level 1

and at levels 1 and 2 than the OECD average percentages (figure 5, table B-6). The United

States also had a lower percentage of students at levels 4, 5, and 6 than the OECD average

percentages. Results for each of the four mathematics content areas followed a similar pattern. In problem solving, the United States also had greater percentages of students below

level 1 and at level 1 than the OECD average percentages, and a lower percentage of students

at levels 2 and 3 than the OECD average percentages (figure 8, table B-15). 

This is also somewhat different from the PISA 2000 reading literacy results, which showed

that while the percentages of U.S. students performing at level 1 and below were not measurably different from the OECD averages, the United States had a greater percentage of

students performing at the highest level (level 5) compared to the OECD average (Lemke et

al. 2001). In mathematics literacy and problem solving in 2003, even the highest U.S. achievers (those in the top 10 percent in the United States) were outperformed on average by their

OECD counterparts (figures 4 and 7, tables B-4 and B-13).

There were no measurable changes in the U.S. scores from 2000 to 2003 on either the space

and shape subscale or the change and relationships subscale, the only content areas for

which trend data from 2000 to 2003 are available (table B-11). In both 2000 and 2003, about

two-thirds of the other participating OECD countries outperformed the United States in

these content areas. iii

PISA 2003 Results From the U.S. Perspective
U.S. Performance in Reading Literacy and Science Literacy

The U.S. average score in reading literacy was not measurably different from the OECD

average in 2000 or 2003 (figure 9, table B-16), nor was there any measurable change in the

U.S. reading literacy score from 2000 to 2003. 

The U.S. score was below the OECD average science literacy score in 2003 (figure 9, table B17). There was no measurable change in the U.S. science literacy score from 2000 to 2003. 

Differences in Performance by Selected Student Characteristics

Sex

Males outperformed females in mathematics literacy in the United States and in two-thirds

of the other countries (figure 10, table B-18). Within the United States, greater percentages

of male students performed at level 6 (the highest level) than female students in mathematics literacy, but larger percentages of females were not seen at lower levels (below level 1

and levels 1 through 5; table B-19). In other words, differences in the overall scores between

males and females in the United States were due at least in part to the fact that a higher

percentage of males were found among the highest performers, not to a higher percentage of

females found among the lowest performers. 

In the majority of the PISA 2003 countries (32 out of 39 countries), including the United

States, there were no measurable differences in problem-solving scores by sex (figure 10,

table B-21). However, females outscored their male peers in problem solving in six of the

seven remaining participating countries, as well as at the OECD average. Males outscored

females in problem solving in Macao-China. 

Socioeconomic Background

In 2003, a few countries showed stronger relationships between socioeconomic background

(as measured by parental occupational status) and student performance than the United

States, while more showed weaker relationships. In 2003, the relationship between socioeconomic background and student performance in mathematics literacy was stronger in 5

countries (Belgium, Czech Republic, Germany, Hungary, and Poland) than in the United

States, while 11 countries had weaker relationships (table B-25). Three of the same five

countries (Belgium, Germany, and Hungary) had stronger relationships between socioeconomic background and problem-solving performance than the United States, while 12 had

weaker relationships.

Race/Ethnicity

In the United States in PISA 2003, Blacks and Hispanics scored lower on average than

Whites, Asians, and students of more than one race in mathematics literacy and problem

solving (figure 11, table B-26). Hispanic students, in turn, outscored Black students. In 

both mathematics literacy and problem solving, the average scores for Blacks and Hispanics

were below the OECD average scores, while scores for Whites were above the OECD 

average scores. 

For further results from PISA 2003, see the Organization for Economic Cooperation and

Development (OECD) publication Learning for Tomorrow’s World — First Results From PISA

2003, available at http://www.pisa.oecd.org (OECD 2004). A technical report for PISA 2003—

which describes in detail all the procedures used in the design, data collection, quality control, and analysis for the study, as well as the PISA 2003 data itself—will also be made available at that site.

International Outcomes of Learning in Mathematics Literacy and Problem Solving

iv
Acknowledgments

The authors of this report cannot take full credit for its production. Many people contributed

to making this report possible, and the authors wish to thank all those who have assisted

with various aspects of the report, including data analysis, reviews, and design.

Members of the Trends in International Mathematics and Science Study-Program for

International Student Assessment (TIMSS-PISA) expert panel provided valuable input on

issues related to communication and dissemination. Members are listed in appendix C.

We also wish to thank the technical reviewers, Marilyn Seastrom, Devon Carlson, and Zeyu

Xu, and other reviewers (Mary Lindquist, Jeremy Kilpatrick, Joan Ferrini-Mundy, and Dana

Kelly) for their comments. Thanks are also owed to Brian Henigin, Michael Stock, and Tracey

Summerall of Westat for their design work.

Members of the PISA international management team, including the Australian Council for

Educational Research (ACER) and the Organization for Economic Cooperation and

Development (OECD), graciously assisted with data analysis and documentation questions.

In particular, we would like to gratefully acknowledge the assistance of Claudia Tamassia,

Andreas Schleicher, and Claire Shewbridge at the OECD.

Finally, we wish to especially thank the students, schools, and principals who participated in

PISA 2003. Their time and effort provide us with data to look beyond our borders and gain

valuable insight into our own educational practices.

v

PISA 2003 Results From the U.S. Perspective
International Outcomes of Learning in Mathematics Literacy and Problem Solving

vi
Table of Contents

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iii

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix

List of Reference Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xiii

List of Exhibits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

PISA in Brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

The Unique Contribution of PISA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

How PISA 2003 Was Conducted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

U.S. Performance in Mathematics Literacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Changes in Mathematics Literacy Performance From 2000 to 2003 . . . . . . . . . . .21

U.S. Performance in Problem Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

U.S. Performance in Reading Literacy and Science Literacy . . . . . . . . . . . . .33

Differences in Performance by Selected Student Characteristics . . . . . . . .35

Sex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35

Socioeconomic Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37

Race/Ethnicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

For Further Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

Appendix A: Technical Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44

Sampling, Data Collection, and Response Rate Requirements . . . . . . . . . . . . . .45

Sampling, Data Collection, and Response Rates in the United States . . . . . . .47

Test Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Translation and the Metric-Imperial Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54

Test Administration and Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54

Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

Data Entry and Cleaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55

vii

PISA 2003 Results From the U.S. Perspective
Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56

Scaling and Plausible Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56

Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58

Data Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Descriptions of Background Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62

Confidentiality and Disclosure Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

Statistical Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64

Appendix B: Reference Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68

Appendix C: TIMSS-PISA 2003 Expert Panelists . . . . . . . . . . . . . . . . . . . . . . .112

Appendix D: PISA Online Resources and Publications . . . . . . . . . . . . . . . . .114

Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115

NCES Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115

International Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115

International Outcomes of Learning in Mathematics Literacy and Problem Solving

viii
ix

PISA 2003 Results From the U.S. Perspective

List of Tables

Table 1. Participation in the Program for International Student Assessment 

(PISA), by country: 2000 and 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

Table 2. Average combined mathematics literacy scores and subscale scores of 

15-year-old students, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

Table 3. Average scores of 15-year-old students on the problem-solving scale,

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

Table A-1. Coverage of target population, student and school samples, and 

participation rates in the Program for International Student Assessment (PISA), 

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52

Table A-2. Response rates of 15-year-old students for selected background 

variables, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
International Outcomes of Learning in Mathematics Literacy and Problem Solving

x
List of Reference Tables

Table B-1. Percentage distribution of 15-year-old students, by grade and 

country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69

Table B-2. Percentage distribution and average combined mathematics literacy 

scores of U.S. 15-year-old students, by type of mathematics class: 2003 . . . . . . . . . . . . .71

Table B-3. Average mathematics literacy scores and subscale scores of 

15-year-old students, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

Table B-4. Combined mathematics literacy scores of 15-year-old students, 

by percentiles and country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73

Table B-5. Standard deviations of 15-year-old students’ combined mathematics 

literacy scores, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Table B-6. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the combined mathematics literacy scale, by country: 2003 . . . . . . .76

Table B-7. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the mathematics literacy quantity subscale, by country: 2003 . . . . . .78

Table B-8. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the mathematics literacy space and shape subscale, 

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80

Table B-9. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the mathematics literacy change and relationships subscale, 

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Table B-10. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the mathematics literacy uncertainty subscale, by country: 2003 . . . .84

Table B-11. Average selected mathematics literacy subscale scores of 

15-year-old students, by country: 2000 and 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

Table B-12. Average problem-solving scores of 15-year-old students, 

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87

Table B-13. Problem-solving scores of 15-year-old students, by percentiles and 

country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88

Table B-14. Standard deviations of 15-year-old students’ problem-solving scores, 

by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90

Table B-15. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the problem-solving scale, by country: 2003 . . . . . . . . . . . . . . . . . . . .91

Table B-16. Average reading literacy scores of 15-year-old students, by country: 

2000 and 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92

Table B-17. Average science literacy scores of 15-year-old students, by country: 

2000 and 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

xi

PISA 2003 Results From the U.S. Perspective
Table B-18. Average combined mathematics literacy scores of 15-year-old 

students, by sex and country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94

Table B-19. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the combined mathematics literacy scale, by sex and 

country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95

Table B-20. Average mathematics literacy subscale scores of 15-year-old 

students, by sex and country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99

Table B-21. Average problem-solving scores of 15-year-old students, by sex 

and country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103

Table B-22. Average combined reading literacy and science literacy scores of 

15-year-old students, by sex and country: 2000 and 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . .104

Table B-23. Mean International Socioeconomic Index (ISEI) score of 15-year-old 

students, by quarters of the ISEI index and country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . .108

Table B-24. Average combined mathematics literacy scores of 15-year-old 

students, by quarters of the International Socioeconomic Index (ISEI) and 

country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109

Table B-25. Change in the combined mathematics literacy and problem-solving 

scores per one standard deviation change in the International Socioeconomic 

Index (ISEI) score, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .110

Table B-26. Average combined mathematics literacy, problem-solving, reading 

literacy, and science literacy scores of U.S. 15-year-old students, by 

race/ethnicity: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111

International Outcomes of Learning in Mathematics Literacy and Problem Solving

xii
List of Figures

Figure 1. Program for International Student Assessment (PISA) cycle . . . . . . . . . . . . . .1

Figure 2. Percentage distribution of U.S. 15-year-old students, by grade: 2003 . . . . . . . . .4

Figure 3. Percentage distribution of U.S. 15-year-old students, by type of 

mathematics class: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Figure 4. Distribution of combined mathematics literacy scores of 15-year-old 

students, by country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Figure 5. Percentage distribution of 15-year-old students in the OECD countries 

and the United States on the combined mathematics literacy scale, by proficiency 

level: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 6. Percentage distribution of 15-year-old students on the combined 

mathematics literacy scale, by proficiency level and country: 2003 . . . . . . . . . . . . . . . . . .20

Figure 7. Distribution of problem-solving scores of 15-year-old students, by 

country: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

Figure 8. Percentage distribution of 15-year-old students in the OECD countries 

and the United States on the problem-solving scale, by proficiency level: 2003 . . . . . . .32

Figure 9. Average reading literacy and science literacy scores of 15-year-old 

students in the OECD countries and the United States: 2003 . . . . . . . . . . . . . . . . . . . . . . .33

Figure 10. Differences in average scores of 15-year-old students on the combined 

mathematics literacy scale and in problem solving, by sex and country: 2003 . . . . . . . .36

Figure 11. Average scores of U.S. 15-year-old students on the combined 

mathematics literacy scale and in problem solving, by race/ethnicity: 2003 . . . . . . . . . .38

Figure A-1. School response rate requirements for PISA 2003 . . . . . . . . . . . . . . . . . . . . .46

xiii

PISA 2003 Results From the U.S. Perspective
International Outcomes of Learning in Mathematics Literacy and Problem Solving

xiv
List of Exhibits

Exhibit 1. Space and shape sample item: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Exhibit 2. Change and relationships sample item: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Exhibit 3. Quantity sample item: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Exhibit 4. Uncertainty sample item: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

Exhibit 5. Description of proficiency levels for combined mathematics 

literacy: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Exhibit 6. Problem-solving sample item 1: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

Exhibit 7. Problem-solving sample item 2: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Exhibit 8. Problem-solving sample item 3: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Exhibit 9. Description of proficiency levels for problem solving: 2003 . . . . . . . . . . . . . . .31

Exhibit A-1. Description of proficiency levels for mathematics literacy 

subscales: 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59

xv

PISA 2003 Results From the U.S. Perspective
International Outcomes of Learning in Mathematics Literacy and Problem Solving

xvi
Introduction

PISA in Brief

The Program for International Student

Assessment (PISA) is a system of international assessments that measures 15-yearolds’ capabilities in reading literacy, mathematics literacy, and science literacy every 3

years. PISA was first implemented in 2000

(figure 1). 

PISA is sponsored by the Organization for

Economic Cooperation and Development

(OECD), an intergovernmental organization

of 30 industrialized nations. In 2003, 41 countries participated in PISA, including 30

OECD countries and 11 non-OECD countries (table 1). Of those 41 countries, comparisons for 39 countries (29 OECD countries and 10 non-OECD countries) are provided in this report. Data for one country,

Brazil, were not available at the time of

report production, and data for one other, the

United Kingdom, are not discussed due to

low response rates.

Figure 1. Program for International Student

Assessment (PISA) cycle

NOTE: The subject in all capital letters in each assessment

cycle is the major domain for that cycle.

SOURCE: Organization for Economic Cooperation and

Development (OECD), Program for International Student

Assessment (PISA), 2003.

R

EADING

M

athematics

S

cience

SCIENCE

Mathematics

Reading

Readin

g

MATHEMATIC

S

Scien

c

e

2000

(2009...)

2003

(2012...)

2006

(2015...)

1

PISA 2003 Results From the U.S. Perspective

Table 1. Participation in the Program for

International Student Assessment

(PISA), by country: 2000 and 2003

Country 2000 2003

Organization for Economic

Cooperation and Development

(OECD) countries

Australia • •

Austria • •

Belgium • •

Canada • •

Czech Republic • •

Denmark • •

Finland • •

France • •

Germany • •

Greece • •

Hungary • •

Iceland • •

Ireland • •

Italy • •

Japan • •

Korea, Republic of • •

Luxembourg • •

Mexico • •

Netherlands1 • •

New Zealand • •

Norway • •

Poland • •

Portugal • •

Slovak Republic •

Spain • •

Sweden • •

Switzerland • •

Turkey •

United Kingdom2 • •

United States • •

Non-OECD countries

Brazil3 • •

Hong Kong-China •

Indonesia •

Latvia • •

Liechtenstein • •

Macao-China •

Russian Federation • •

Serbia and Montenegro •

Thailand •

Tunisia •

Uruguay • 1

Due to low response rates, PISA 2000 data for the

Netherlands are not discussed in this report. For information

on the results for the Netherlands, see OECD (2001).

2

Due to low response rates, PISA 2003 data for the United

Kingdom are not discussed in this report.

3

Although Brazil participated in PISA 2003, its data were not

available in time for production of this report. 

NOTE: A "•" indicates that the country participated in PISA

in the specific year. Because PISA is principally an OECD

study, non-OECD countries are displayed separately from the

OECD countries. 

SOURCE: Organization for Economic Cooperation and

Development (OECD), Program for International Student

Assessment (PISA), 2000 and 2003.
In order to provide a critical external perspective on the achievement of U.S. students through comparisons to other nations,

the United States participates at the international level in PISA, the Progress in

International Reading Literacy Study

(PIRLS), and the Trends in International

Mathematics and Science Study (TIMSS).1

TIMSS and PIRLS seek to measure students’ mastery of specific knowledge, skills,

and concepts, and are designed to reflect

curriculum frameworks in the United States

and other participating countries.

PISA provides a unique and complementary

perspective to these studies by not focusing

explicitly on curricular outcomes, but on the

application of knowledge in reading, mathematics, and science to problems with a reallife context (OECD 1999). The framework for

each assessment area is based upon content, processes, and situations or contexts.

For example, for mathematics literacy, the

content is made up of major mathematical

ideas, such as space and shape and uncertainty. The processes describe what strategies students use to solve mathematics

problems, such as making connections or

performing simple calculations. The situations or contexts refer to the kinds of places

in which students might encounter mathematical problems, such as personal or educational. Assessment items are then developed based on these descriptions.

PISA uses the terminology of “literacy” in

each subject area to denote its broad focus

on application of knowledge and skills; that

is, PISA seeks to ask if 15-year-olds are

mathematically literate, or to what extent

they can apply mathematical knowledge and

skills to a range of different situations they

may encounter in their lives. Literacy itself

refers to a continuum of skills—it is not a

condition that one has or does not have (i.e.,

literacy or illiteracy), but rather each person’s skills place them in a particular place

on the literacy continuum.

Each PISA data-collection effort assesses

one subject area in depth, even as all three

are assessed in each cycle so that participating countries have an ongoing source of

achievement data in every subject area. In

addition to the reading literacy, mathematics

literacy, and science literacy, PISA also

measures general or cross-curricular competencies such as learning strategies. In

this second cycle, PISA 2003, mathematics

literacy was the subject area assessed in

depth, along with the new cross-curricular

area of problem solving. In 2006, PISA will

focus on science literacy. Results from

PISA 2000, which focused on reading literacy, are described in Lemke et al. (2001) and

Organization for Economic Cooperation and

Development (OECD) (2001). In addition, a

series of thematic reports exploring topics

related to reading literacy in greater depth

are available through http://www.pisa.oecd.org

(see also the PISA resources and publications section of this report for information

about PISA publications).

This report focuses on the performance of

U.S. students in the two major areas

assessed in 2003, mathematics literacy and

problem solving. Achievement in the minor

domains of reading literacy and science literacy in 2003 is also presented, and differences in achievement by selected student

characteristics are covered in the final section.

The Unique Contribution of PISA

The United States has conducted surveys of

student achievement at a variety of grade

levels and in a variety of subject areas

through the National Assessment of

Educational Progress (NAEP) for many

years. NAEP provides a regular benchmark

for states and the nation and a means to

monitor progress in achievement over time. 

International Outcomes of Learning in Mathematics Literacy and Problem Solving

1

The United States has also participated in international comparative assessments of civics knowledge and skills (CivEd 1999)

and adult literacy (International Adult Literacy Survey [IALS 1994] and Adult Literacy and Lifeskills Survey [ALL 2003]).

2
The target age of 15 allows countries to

compare outcomes of learning as students

near the end of compulsory schooling.

PISA’s goal is to answer the question “what

knowledge and skills do students have at

age 15?” taking into account schooling and

other factors that may influence their performance. In this way, PISA’s achievement

scores represent a “yield” of learning at age

15, rather than a direct measure of attained

curriculum knowledge at a particular grade

level, since 15-year-olds in the United

States and elsewhere come from several

grade levels and are enrolled in a variety

of classes (figures 2 and 3, tables B-1 and

B-2).

How PISA 2003 Was Conducted

PISA 2003 was sponsored by the OECD and

carried out at the international level through

a contract with the PISA Consortium, led by

the Australian Council for Educational

Research (ACER).2 The National Center for

Education Statistics (NCES) of the Institute

of Education Sciences at the U.S.

Department of Education was responsible

for the implementation of PISA in the

United States. Data collection in the United

States was carried out through a contract

with Westat. A review panel (see appendix

C for a list of members) provides input on

the development and dissemination of PISA

(and TIMSS) in the United States.

PISA 2003 was a 2-hour paper-and-pencil

assessment of 15-year-olds collected from

nationally representative samples in participating countries. Like other large-scale

assessments, PISA was not designed to

provide individual student scores, but rather

national and sub-national estimates of performance. Every student in PISA 2003 was

assessed in mathematics literacy; reading,

problem solving, and science questions were

spread among students (for more information on PISA 2003’s design, see the technical notes in appendix A).

3

PISA 2003 Results From the U.S. Perspective

2

The PISA Project Consortium consists of the Australian Council for Educational Research (ACER), the Netherlands National

Institute for Educational Measurement (CITO), Educational Testing Service (ETS, USA), National Institute for Educational Policy

Research (NIER, Japan), and Westat (USA). 

3

The sample frame data for the United States for public schools were from the Common Core of Data (CCD), and the data for

private schools were from the Private School Survey (PSS). Any school containing at least one 7th- through 12th-grade class as

of the school year 2000–01 was included on the school sampling frame.

PISA 2003 was administered between March

and May 2003. The U.S. sample included

both public and private schools, randomly

selected and weighted to be representative

of the nation.3 In the United States, to

improve response rates (a response rate of

approximately 50 percent was projected for

the end of the data collection period) and

better accommodate school schedules, a

second testing window was opened from

September through November 2003. In total,

262 schools and 5,456 students participated

in PISA 2003 in the United States. An overall weighted school response rate of 65 percent before the use of replacement schools

and a weighted student response rate of 83

percent was achieved after testing in the

second window was complete (see technical

notes in appendix A for additional details on

sampling, administration, response rates,

and other issues).

For further results from PISA 2003, see the

Organization for Economic Cooperation and

Development (OECD) publication Learning

for Tomorrow’s World — First Results From

PISA 2003, available at http://www.pisa.oecd.org

(OECD 2004). A technical report for PISA

2003—which describes in detail all the procedures used in the design, data collection,

quality control, and analysis for the study, as

well as the PISA 2003 data itself—is also

available at that site.

This report provides results for the United

States in relation to the other countries participating in PISA 2003, distinguishing

OECD countries and non-OECD countries.

All differences described in this report have

been tested for statistical significance at

the .05 level. Additional information on statistical procedures used in this report is provided in the technical notes.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

4

Grades 7 

and 8

Grade 9 Grade 10 Grade 11 

and above

0

10

20

30

40

50

60

70

80

90

100

Percent

Grade

2

30

61

7

Figure 2. Percentage distribution of U.S. 15-year-old students, by grade: 2003

9

Algebra I

29

Algebra II 

21

Geometry 

31

3

Pre-algebra or 

general math

Precalculus or 

calculus

Other 

8

Figure 3. Percentage distribution of U.S. 15-year-old students, by type of mathematics class:

2003

NOTE: Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.

NOTE: Type of class refers to the mathematics class in which the student was enrolled at the time of assessment. Detail may

not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.
U.S. Performance in

Mathematics Literacy

PISA’s major focus in 2003 was mathematics

literacy. Mathematics literacy is defined as: 

...an individual’s capacity to identify and

understand the role that mathematics plays

in the world, to make well-founded judgements and to use and engage with mathematics in ways that meet the needs of that

individual’s life as a constructive, concerned, and reflective citizen. (OECD 2003,

p.24)

PISA’s emphasis is on the ability to apply a

range of knowledge and skills to a variety of

problems with real-life contexts. In the

PISA 2003 mathematics literacy assessment, students completed exercises

designed to assess their capabilities in

using a range of mathematical competencies, grouped and described as “competency clusters.” These clusters—reproduction,

connections, and reflection—describe sets

of skills students may use to solve problems. The reproduction cluster involves the

reproduction of the practiced material and

performing routine operations. The connections cluster calls for integration and connection of material, and the modest extension of practiced material. The reflection

cluster relates to students’ abilities in

advanced reasoning, argumentation,

abstraction, generalization, and modeling

applied to new contexts. 

The problems themselves were designed to

come from the variety of situations (personal, educational/occupational, public, or scientific) that students encounter, and to have

a real-life context. The mathematical content of the problems was drawn from four

overarching ideas: space and shape, change

and relationships, quantity, and uncertainty.

5

PISA 2003 Results From the U.S. Perspective

These overarching ideas represent a way to

organize mathematical content broadly and

encompass many traditional curricular areas

such as algebra or geometry (see also

Steen 1990).

• Space and shape includes recognizing

shapes and patterns, describing, encoding, and decoding visual information,

understanding dynamic changes to

shapes, understanding similarities and

differences and relative positions, and

understanding the relationship between

visual representations and real shapes

and images.

• Change and relationships covers the representation of change, including mathematical functions such as linear, exponential, or logistic, as well as data analysis needed to specify relationships or

translate between representations.

• Quantity focuses on quantitative reasoning (including number sense, estimating,

mental arithmetic, understanding meaning of operations, having a feel for the

magnitude of numbers, and computations) and understanding of numerical

patterns, counts, and measures. 

• Uncertainty includes the two related topics

of data and chance, or statistics and probability, including data analysis and graphic

and numeric representations of data.

A comparative analysis of the NAEP, PISA,

and TIMSS mathematics assessments sponsored by NCES found that the 2003 PISA

mathematics literacy assessment used far

fewer multiple-choice items than NAEP or

TIMSS. PISA also had a much stronger content focus on the “data” area (which often

deals with using charts and graphs), which

fits with PISA’s emphasis on using materials with a real-world context (see technical

notes for more information on the results of

the assessment comparisons).4

4

See Neidorf, T.S., Binkley, M., Gattis, K., and Nohara, D. (forthcoming) and the technical notes in appendix A for more information. Other comparative analyses focus on assessments of science and reading in PISA, NAEP, TIMSS, and PIRLS. See

Neidorf, T.S., Binkley, M., and Stephens, M. (forthcoming); Binkley, M., and Kelly, D. (2003); Binkley, M., Afflerbach, P., and Kelly, D.

(forthcoming); and Nohara, D. (2001).
Sample mathematics literacy items for each

of these areas and student responses are

shown here. For more information about the

mathematics literacy domain, refer to The

PISA 2003 Assessment Framework:

Mathematics, Reading, Science, and Problem

Solving Knowledge and Skills (OECD 2003).

Additional mathematics literacy sample

items can be found at http://nces.ed.gov/

surveys/pisa, in the PISA 2003 framework

document referenced above, in Measuring

Student Knowledge and Skills: The PISA 2000

Assessment of Reading, Mathematical and

Scientific Literacy (OECD 2000) and in

Sample Tasks from the PISA 2000

Assessment: Reading, Mathematical and

Scientific Literacy (OECD 2002). 

International Outcomes of Learning in Mathematics Literacy and Problem Solving

6
7

PISA 2003 Results From the U.S. Perspective

Exhibit 1. Space and shape sample item: 2003 

Cubes

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.

(a)

(b)

(c)

(d)

(e)

(f)
International Outcomes of Learning in Mathematics Literacy and Problem Solving

8

Exhibit 2. Change and relationships sample item: 2003 

The Best Car

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.
9

PISA 2003 Results From the U.S. Perspective

Exhibit 2. Change and relationships sample item: 2003—Continued

The Best Car

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

10

Exhibit 3. Quantity sample item: 2003

Exchange Rate

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.
11

PISA 2003 Results From the U.S. Perspective

Exhibit 3. Quantity sample item: 2003—Continued

Exchange Rate

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

12

Exhibit 4. Uncertainty sample item: 2003

Test Scores

SOURCE: Organization for Economic

Cooperation and Development

(OECD), Program for International

Student Assessment (PISA), 2003.
Combined mathematics literacy scores are

reported on a scale with a mean of 500 and

standard deviation of 100.5 Fifteen-year-old

students in the United States had an average score of 483 on the combined mathematics literacy scale, lower than the OECD

average score of 500 (tables 2 and B-3). U.S.

students were less mathematically literate

than their peers in 20 of the other 28 OECD

countries and 3 of the 10 non-OECD countries. Eleven countries (5 OECD countries

and 6 non-OECD countries) reported lower

scores compared to the United States in

mathematics literacy. 

U.S. students also had lower scores than 

the OECD average scores for each of the

four content area subscales (space and

shape, change and relationships, quantity,

and uncertainty). Twenty-four countries (20

OECD and 4 non-OECD countries) outperformed the United States on the space and

shape subscale, 21 countries (18 OECD and 

3 non-OECD countries) outperformed the

United States on the change and relationships subscale, 26 countries (23 OECD and 3

non-OECD countries) outscored the United

States on the quantity subscale, and 19

countries (16 OECD and 3 non-OECD countries) outscored the United States on the

uncertainty subscale.

13

PISA 2003 Results From the U.S. Perspective

5

Because the average was set for the combined mathematics literacy scale, average scores for the mathematics literacy subscales

differ slightly from 500. PISA 2000 mathematics literacy scores were re-scaled using the greater detail in PISA 2003 data in order to

provide a more complete measure of achievement than that available in 2000. See technical notes in appendix A for more information on scaling. PISA’s intent for each subject area is to draw baseline information for describing changes and trends in achievement from the cycle in which that subject area is the major domain. The use of minor domains allows PISA to provide indicative

information about changes in performance over time; however, changes in a subject area are best measured from the cycle in which

it is the major domain. Thus, changes in reading literacy achievement are based upon PISA 2000 data, when reading literacy was the

major domain, and changes in mathematics literacy scores, in turn, are based upon this 2003 cycle. Science literacy scores from 2000

and 2003 may be re-scaled based up on the much greater detail for science literacy which will be available in 2006.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

14

Table 2. Average combined mathematics literacy scores and subscale scores of 15-year-old

students, by country: 2003

Combined mathematics literacy

Mathematics subscales

Space and shape Change and relationships

Country Score Country Score Country Score

OECD average 500 OECD average 496 OECD average 499

OECD countries OECD countries OECD countries

Finland 544 Japan 553 Netherlands 551

Korea 542 Korea 552 Korea 548

Netherlands 538 Switzerland 540 Finland 543

Japan 534 Finland 539 Canada 537

Canada 532 Belgium 530 Japan 536

Belgium 529 Czech Republic 527 Belgium 535

Switzerland 527 Netherlands 526 New Zealand 526

Australia 524 New Zealand 525 Australia 525

New Zealand 523 Australia 521 Switzerland 523

Czech Republic 516 Canada 518 France 520

Iceland 515 Austria 515 Czech Republic 515

Denmark 514 Denmark 512 Iceland 509

France 511 France 508 Denmark 509

Sweden 509 Slovak Republic 505 Germany 507

Austria 506 Iceland 504 Ireland 506

Germany 503 Germany 500 Sweden 505

Ireland 503 Sweden 498 Austria 500

Slovak Republic 498 Poland 490 Hungary 495

Norway 495 Luxembourg 488 Slovak Republic 494

Luxembourg 493 Norway 483 Norway 488

Poland 490 Hungary 479 Luxembourg 487

Hungary 490 Spain 476 United States 486

Spain 485 Ireland 476 Poland 484

United States 483 United States 472 Spain 481

Portugal 466 Italy 470 Portugal 468

Italy 466 Portugal 450 Italy 452

Greece 445 Greece 437 Greece 436

Turkey 423 Turkey 417 Turkey 423

Mexico 385 Mexico 382 Mexico 364

Non-OECD countries Non-OECD countries Non-OECD countries

Hong Kong-China 550 Hong Kong-China 558 Hong Kong-China 540

Liechtenstein 536 Liechtenstein 538 Liechtenstein 540

Macao-China 527 Macao-China 528 Macao-China 519

Latvia 483 Latvia 486 Latvia 487

Russian Federation 468 Russian Federation 474 Russian Federation 477

Serbia and Montenegro 437 Serbia and Montenegro 432 Serbia and Montenegro 419

Uruguay 422 Thailand 424 Uruguay 417

Thailand 417 Uruguay 412 Thailand 405

Indonesia 360 Indonesia 361 Tunisia 337

Tunisia 359 Tunisia 359 Indonesia 334

See notes at end of table. 
15

PISA 2003 Results From the U.S. Perspective

Table 2. Average combined mathematics literacy scores and

subscale scores of 15-year-old students, by country:

2003—Continued

Mathematics subscales

Quantity Uncertainty

Country Score Country Score

OECD average 501 OECD average 502

OECD countries OECD countries

Finland 549 Netherlands 549

Korea 537 Finland 545

Switzerland 533 Canada 542

Belgium 530 Korea 538

Netherlands 528 New Zealand 532

Canada 528 Australia 531

Czech Republic 528 Japan 528

Japan 527 Iceland 528

Australia 517 Belgium 526

Denmark 516 Ireland 517

Germany 514 Switzerland 517

Sweden 514 Denmark 516

Iceland 513 Norway 513

Austria 513 Sweden 511

Slovak Republic 513 France 506

New Zealand 511 Czech Republic 500

France 507 Austria 494

Ireland 502 Poland 494

Luxembourg 501 Germany 493

Hungary 496 Luxembourg 492

Norway 494 United States 491

Spain 492 Hungary 489

Poland 492 Spain 489

United States 476 Slovak Republic 476

Italy 475 Portugal 471

Portugal 465 Italy 463

Greece 446 Greece 458

Turkey 413 Turkey 443

Mexico 394 Mexico 390

Non-OECD countries Non-OECD countries

Hong Kong-China 545 Hong Kong-China 558

Liechtenstein 534 Macao-China 532

Macao-China 533 Liechtenstein 523

Latvia 482 Latvia 474

Russian Federation 472 Russian Federation 436

Serbia and Montenegro 456 Serbia and Montenegro 428

Uruguay 430 Thailand 423

Thailand 415 Uruguay 419

Tunisia 364 Indonesia 385

Indonesia 357 Tunisia 363

Average is significantly higher than the U.S. average

Average is not significantly different than the U.S. average

Average is significantly lower than the U.S. average

NOTE: Statistical comparisons between the U.S. average and the Organization for

Economic Cooperation and Development (OECD) average take into account the contribution of the U.S. average toward the OECD average. The OECD average is the average

of the national averages of the OECD member countries with data available. Because the

Program for International Student Assessment (PISA) is principally an OECD study, the

results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD average. Due to low response rates, data for the

United Kingdom are not discussed in this report.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program

for International Student Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

16

Along with scale scores, PISA 2003 also uses

six proficiency levels (levels 1 through 6, with

level 6 being the highest level of proficiency)

to describe student performance in mathematics literacy (exhibit 5). An additional level

(below level 1) encompasses students whose

skills cannot be described using these proficiency levels. The proficiency levels describe

what students at each level can do and allow

comparisons of the percentages of students

in each country who perform at different levels of mathematics literacy (see technical

notes in appendix A for more information

about how levels were set).

The U.S. average score of 483 on the combined mathematics literacy scale was just

above the bottom cut point for level 3; the

OECD average score of 500 was near the

midpoint of level 3 (table 2, exhibit 5). The

cutoff score of 607 for U.S. high performers

(those in the top 10 percent in the United

States) placed it just into level 5; the OECD

score for high performers was near the midpoint of level 5. The cutoff U.S. score of 356

for low performers (those in the bottom 10

percent) was below level 1, while the OECD

cutoff score of 369 for the bottom 10 percent

was a level 1 score (figure 4, exhibit 5).

On average, the highest U.S. achievers

(those in the top 10 percent of U.S. students)

were outperformed by their OECD counterparts (figure 4, table B-4). To be in the top 10

percent in the United States, students had

to score 607 or higher, while on average

across the OECD countries, students would

have had to score 628 or higher to be in the

top 10 percent. Scores for the top 10 percent

of students within countries ranged from 466

or better in Indonesia and Tunisia to 672 or

better in Hong Kong-China. Low performers

in the United States (those in the bottom 10

percent) had a cutoff score of 356 or lower,

which was lower than the cutoff score of 369

or lower for the OECD average. There was

approximately a 251 point score difference,

or about two and a half standard deviations,

between the cutoff scores for the top 10 percent and the bottom 10 percent of 15-year-old

students for mathematics literacy in the

United States, compared to about a 259 point

difference using the OECD average scores.

The standard deviation (which measures the

spread of scores around the average) for the

United States (95), in fact, was lower than

the OECD average standard deviation of 100

(table B-5). Sixteen countries (10 OECD and

6 non-OECD countries) showed less variation in performance than the United States,

while three countries (Belgium, Germany,

and Uruguay) had larger standard deviations.
17

PISA 2003 Results From the U.S. Perspective

0 100 200 300 400 500 600 700 800 900 10

Tunisia

Indonesia

Thailand

Uruguay

Serbia and Montenegro

Russian Federation

Latvia

Macao-China

Liechtenstein

Hong Kong-China

Mexico

Turkey

Greece

Italy

Portugal

United States

Spain

Hungary

Poland

Luxembourg

Norway

Slovak Republic

Ireland

Germany

Austria

Sweden

France

Denmark

Iceland

Czech Republic

New Zealand

Australia

Switzerland

Belgium

Canada

Japan

Netherlands

Korea, Republic of

Finland

OECD average

Average score

NOTE: The Organization for Economic Cooperation and Development (OECD) average is the average of the national averages 

of the OECD member countries with data available. Because the Program for International Student Assessment (PISA) is 

principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries and 

are not included in the OECD average. Due to low response rates, data for the United Kingdom are not discussed in this report.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment 

(PISA), 2003.

10 75 90 

Percentiles of performance

25 th th th th

Non-OECD countries

OECD countries

Country

Mean and 95% Confidence 

interval (+/- 2 standard error)

Figure 4. Distribution of combined mathematics literacy scores of 15-year-old students, by

country: 2003
International Outcomes of Learning in Mathematics Literacy and Problem Solving

18

Exhibit 5. Description of proficiency levels for combined mathematics literacy: 2003

Proficiency level Task descriptions

Level 1 At Level 1 students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to

identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow

immediately from the given stimuli. 

Level 2 At Level 2 students can interpret and recognize situations in contexts that require

no more than direct inference. They can extract relevant information from a single

source and make use of a single representational mode. Students at this level can

employ basic algorithms, formula, procedures, or conventions. They are capable of

direct reasoning and making literal interpretations of the results.

Level 3 At Level 3 students can execute clearly described procedures, including those that

require sequential decisions. They can select and apply simple problem solving

strategies. Students at this level can interpret and use representations based on

different information sources and reason directly from them. They can develop short

communications reporting their interpretations, results, and reasoning.

Level 4 At Level 4 students can work effectively with explicit models for complex concrete

situations that may involve constraints or call for making assumptions. They can

select and integrate different representations, including symbolic, linking them

directly to aspects of real-world situations. Students at this level can utilize welldeveloped skills and reason flexibly, with some insight, in these contexts. They can

construct and communicate explanations and arguments based on their interpretations, arguments, and actions.

Level 5 At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare, and evaluate appropriate problem solving strategies for dealing with complex problems

related to these models. Students at this level can work strategically using broad,

well-developed thinking and reasoning skills, appropriate linked representations,

symbolic and formal characterizations, and insight pertaining to these situations.

They can reflect on their actions and formulate and communicate their interpretations and reasoning.

Level 6 At Level 6 students can conceptualize, generalize, and utilize information based on

their investigations and modeling of complex problem situations. They can link different information sources and representations and flexibly translate among them.

Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understandings along with a mastery

of symbolic and formal mathematical operations and relationships to develop new

approaches and strategies for attacking novel situations. Students at this level can

formulate and precisely communicate their actions and reflections regarding their

findings, interpretations, arguments, and the appropriateness of these to the original situations.

NOTE: In order to reach a particular level, a student must have been able to correctly answer a majority of items at that

level. Students were classified into mathematics literacy levels according to their scores. Exact cut point scores are

as follows: below level 1 (a score less than or equal to 357.77); level 1 (a score greater than 357.77 and less than or equal to

420.07); level 2 (a score greater than 420.07 and less than or equal to 482.38); level 3 (a score greater than 482.38 and less

than or equal to 544.68); level 4 (a score greater than 544.68 and less than or equal to 606.99); level 5 (a score greater than

606.99 and less than or equal to 669.3); level 6 (a score greater than 669.3).

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
19

PISA 2003 Results From the U.S. Perspective

States (four of these nine—Greece, Italy,

Mexico, and Turkey—were OECD countries).

These same nine countries, as well as the

Russian Federation and Portugal, had more

students at level 1 than the United States.

The United States had a lower percentage of

students at level 6 than the OECD average for

each of the four content area subscales

(space and shape, change and relationships,

quantity, and uncertainty) and a smaller percentage than the OECD average at level 4 and

level 5 on three of the four subscales (exceptions were for uncertainty at level 5 and change

and relationships at level 4) (tables B-7

through B-10). 

The United States also had a higher percentage of students at level 1 than the OECD

average on each of the four subscales and

more at level 2 for all subscales except uncertainty. On the quantity and uncertainty subscales, the United States also had greater

percentages of students than the OECD average percentages below level 1.

The United States had greater percentages of

students below level 1 and at levels 1 and 2

than the OECD average percentages (figure 5,

table B-6). The United States also had a lower

percentage of students at levels 4, 5, and 6,

than the OECD average percentages. This is

somewhat different from the 2000 results,

when reading literacy was the major domain.

PISA 2000 results showed that while the percentages of U.S. students performing at level 1

and below were not measurably different from

the OECD averages, the United States had a

greater percentage of students performing at

the highest level (level 5) compared to the

OECD average (Lemke et al. 2001). 

In mathematics literacy in 2003, half (19) of the

other 38 countries had a higher percentage of

students at level 6 than the United States,

including 16 OECD countries and 3 non-OECD

countries (Hong Kong-China, Liechtenstein,

and Macao-China) (figure 6, table B-6). In 

contrast, nine countries had a higher percentage of students below level 1 than the United

United States

 OECD average

0 20 40 60 80 100

Country

Percent

10 16 24 24 17 8

11

2

8 13 21 24 19 11 4

Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6

Figure 5. Percentage distribution of 15-year-old students in the OECD countries and the

United States on the combined mathematics literacy scale, by proficiency level: 2003

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a majority of items at

that level. Students were classified into mathematics literacy levels according to their scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a score greater than 357.77 and less than or equal to 420.07); level

2 (a score greater than 420.07 and less than or equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68);

level 4 (a score greater than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The Organization for Economic Cooperation and Development (OECD) average is the

average of the national averages of the OECD member countries with data available. Detail may not sum to totals because of

rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

20

Figure 6. Percentage distribution of 15-year-old students on the combined mathematics 

literacy scale, by proficiency level and country: 2003

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a majority of items at

that level. Students were classified into mathematics literacy levels according to their scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a score greater than 357.77 and less than or equal to 420.07); level

2 (a score greater than 420.07 and less than or equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68);

level 4 (a score greater than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The Organization for Economic Cooperation and Development (OECD) average is the

average of the national averages of the OECD member countries with data available. Because PISA is principally an OECD

study, the results for non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. Due to low response rates, data for the United Kingdom are not discussed in this report. Detail may not

sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.

Below level 1 Level 1 Levels 2, 3, 4, and 5 Level 6

Tunisia

Indonesia

Uruguay

Thailand

Serbia and Montenegro

Russian Federation

Latvia

Liechtenstein

Hong Kong-China

Macao-China

Non-OECD countries

Mexico

Turkey

Greece

Italy

Portugal

United States

Germany

Spain

Hungary

Luxembourg

Belgium

Norway

Poland

Slovak Republic

Austria

France

Sweden

Czech Republic

Switzerland

New Zealand

Japan

Denmark

Ireland

Iceland

Australia

Netherlands

Korea

Canada

Finland

OECD average

OECD countries

0 20 40 60 80 100

Country

Percent

51 27 22

50 28 22

26 22 51

24 30 46

18 24 58

11 19 68 2

8 16 75 2

5 7 80 7

4 7 79 11

2 9 84 5

38 28 34

28 25 45 2

18 21 60 1

13 19 67 2

11 19 69 1

10 16 72 2

9 12 74 4

8 15 76 1

8 15 75 2

7 14 76 2

7 9 75 9

7 14 76 3

7 15 76 2

7 13 77 3

6 13 77 4

6 11 80 3

6 12 79 4

5 12 78 5

5 10 78 7

5 10 78 7

5 9 78 8

5 11 80 4

5 12 81 2

5 10 81 4

4 10 80 6

3 8 82 7

2 7 82 8

2 8 84 5

1 5 87 7

8 13 75 4
21

PISA 2003 Results From the U.S. Perspective

6

For more information on scaling, see the technical notes in appendix A.

Of the five countries that showed increases

on the space and shape subscale, Belgium

and the Czech Republic already outperformed the United States in 2000 and also

improved their scores in 2003. Italy, despite

its improvement in score, was not measurably different from the United States in either

year. Poland, which was not measurably different from the United States in 2000,

outscored the United States in 2003, and

Luxembourg, which scored below the United

States in 2000, also outscored the United

States in 2003.

Two countries (Mexico and Iceland) showed

decreased scores from 2000 to 2003 on the

space and shape scale. Despite these

decreases in performance, there was no

change in the relative position of either

country compared to the United States: that

is, Iceland outperformed the United States

in 2000 and 2003 on the space and shape subscale, and Mexico performed worse than the

United States in 2000 and 2003.

Of the other 25 OECD countries, 11 had their

scores improve from 2000 to 2003 on the

change and relationships subscale, while no

country had a decrease. Of the 11 countries

that improved from 2000 to 2003, several

already outperformed the United States in

2000: Belgium, Canada, Denmark, Finland,

and Korea all scored higher than the United

States in 2000 on the change and relationships subscale. Several other countries

were not measurably different from the

United States in 2000, but outperformed the

United States in 2003 (Czech Republic,

Germany, Hungary). Three countries

(Luxembourg, Poland, and Spain) had lower

scores than the United States in 2000 on the

change and relationships subscale, but were

not measurably different from the United

States in 2003. Portugal, despite its

improvement in score, still scored lower than

the United States in 2000 and 2003.

Changes in Mathematics Literacy

Performance From 2000 to 2003

Because mathematics literacy was a minor

domain in 2000, items from only two content

areas (space and shape and change and relationships) were administered in that assessment cycle. As a result, it is not possible to

describe changes since 2000 for the combined

mathematics literacy scale or for the other

two content areas (quantity or uncertainty).

Rather, changes can only be discussed for

the two content areas represented in 2000 and

2003 (space and shape and change and relationships). Data from 2000 were re-scaled

using 2003 mathematics literacy data in order

to make these comparisons.6 Comparisons

were available only for OECD countries common to both the 2000 and 2003 cycle (28 countries) but results for the United Kingdom and

the Netherlands are not discussed here due

to low response rates for the United Kingdom

in 2003 and the Netherlands in 2000. In total,

results for 26 OECD countries were available

for comparisons and are discussed here.

There were no measurable changes in the

U.S. scores from 2000 to 2003 on either the

space and shape subscale or the change and

relationships subscale (table B-11). In both

2000 and 2003, about two-thirds of the other

countries outperformed the United States

on these scales. Eighteen of the other 25

OECD countries outscored the United

States on the space and shape scale in 2003

(compared to 19 in 2000); 17 OECD countries

outscored the United States on the change

and relationships scale in 2003 (compared to

14 in 2000).

Five countries had their scores improve on

the space and shape subscale. Four of the

five countries with improved scores on the

space and shape subscale also showed

improvements on the change and relationships scale (Belgium, Czech Republic,

Luxembourg, and Poland; Italy improved its

score on the space and shape scale but not

on the change and relationships scale).
International Outcomes of Learning in Mathematics Literacy and Problem Solving

22

U.S. Performance in

Problem Solving

As noted, one of PISA’s major goals is to

assess skills that cut across traditional curricular areas. In 2003, PISA assessed students’ abilities in problem solving.7

Problem solving is defined as: 

...an individual’s capacity to use cognitive

processes to confront and resolve real,

cross-disciplinary situations where the

solution is not immediately obvious, and

where the literacy domains or curricular

areas that might be applicable are not within a single domain of mathematics, science, or reading. (OECD 2003, p. 156).

Students completed exercises that assessed

their capabilities in using reasoning

processes not only to draw conclusions but

to make decisions, to troubleshoot (i.e., to

understand the reasons for malfunctioning

of a system or device), or to analyze 

the procedures and structures of a

complex system (such as a simple kind of programming language). Problem-solving

items required students to

apply various reasoning

processes, such as inductive and deductive reasoning, 

reasoning about cause and

effects, or combinatorial reasoning (i.e., systematically comparing all the possible variations which can

occur in a well-described situation).

Students were also assessed in their skills

in working toward a solution and communicating the solution to others through appropriate representations. Sample problemsolving items and student responses are

shown here.

7

PISA 2003’s problem-solving assessment focused explicitly on problem-solving skills, using a variety of contexts, disciplines, and

problem types. The items used to measure problem solving in PISA 2003 were different from other items, such as those measuring

mathematics literacy. Problem solving can also be embedded within measures of content areas such as mathematics or science,

however. TIMSS 2003, for example, incorporated an explicit aspect of problem solving and inquiry into the description of desired

outcomes for mathematics and science. A review of mathematics and science items in PISA and TIMSS showed that 38 percent of

eighth-grade TIMSS 2003 mathematics items and 48 percent of PISA 2003 mathematics literacy items measured some aspect of

problem solving; additionally, 26 percent of eighth-grade TIMSS 2003 science items and 49 percent of PISA science literacy items

measured problem-solving skills (Dossey, O’Sullivan, and McCrone forthcoming).

For more information about the problemsolving framework, please refer to The PISA

2003 Assessment Framework: Mathematics,

Reading, Science, and Problem Solving

Knowledge and Skills (OECD 2003).

Additional released problem-solving items

can be found at

http://nces.ed.gov/surveys/pisa.
23

PISA 2003 Results From the U.S. Perspective

Exhibit 6. Problem-solving sample item 1: 2003

Design By Numbers©1

1

Design by Numbers was developed by the Aesthetics and Computation Group at the MIT Media Laboratory.

Copyright 1999, Massachusetts Institute of Technology. The program can be downloaded from

http://dbn.media.mit.edu.

SOURCE: Organization for Economic

Cooperation and Development (OECD),

Program for International Student

Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

24

Exhibit 6. Problem-solving sample item 1: 2003—Continued

Design By Numbers

SOURCE: Organization for Economic

Cooperation and Development (OECD),

Program for International Student

Assessment (PISA), 2003.
25

PISA 2003 Results From the U.S. Perspective

Exhibit 7. Problem-solving sample item 2: 2003

Irrigation

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

26

Exhibit 7. Problem-solving sample item 2: 2003—Continued

Irrigation

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
27

PISA 2003 Results From the U.S. Perspective

Exhibit 8. Problem-solving sample item 3: 2003

Library System

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

28

Exhibit 8. Problem-solving sample item 3: 2003—Continued

Library System

SOURCE: Organization for

Economic Cooperation and

Development (OECD), Program for

International Student Assessment

(PISA), 2003.
29

PISA 2003 Results From the U.S. Perspective

Problem-solving scores are reported

on a scale with a mean of 500 and

standard deviation of 100. Fifteenyear-old students in the United

States had an average score of 477

on the problem-solving scale, lower

than the OECD average score of 500

(table 3, table B-12). U.S. students

scored lower in problem solving

than their peers in 25 of the other 38

countries (22 OECD and 3 nonOECD countries). Eight countries (3

OECD—Greece, Mexico, and

Turkey—and 5 non-OECD countries)

reported lower scores compared to

the United States in problem solving. Three OECD country scores

(and two non-OECD country scores)

were not measurably different from

the U.S. average score in problem

solving. 

On average, U.S. high achievers for

problem solving (those scoring in

the top 10 percent in the United

States) were outperformed by their

OECD counterparts (figure 7, table

B-13). To be in the top 10 percent of

students in the United States, students needed at least a score of 604,

while they needed a score of 446 or

better in Tunisia but 675 or better in

Japan. Low performers in the United

States (those in the bottom 10 percent) scored 347 or lower, which was

lower than the cutoff score of 368 or

lower for the OECD average. There

was approximately a 256 point score

difference, or two and a half standard deviations, between the cutoff

scores for the top 10 percent (604)

and the bottom 10 percent (347) of

15-year-old students for problem

solving in the United States. 

Table 3. Average scores of 15-year-old students on

the problem-solving scale, by country: 2003

Country

OECD average 500

OECD countries

Korea 550

Finland 548

Japan 547

New Zealand 533

Australia 530

Canada 529

Belgium 525

Switzerland 521

Netherlands 520

France 519

Denmark 517

Czech Republic 516

Germany 513

Sweden 509

Austria 506

Iceland 505

Hungary 501

Ireland 498

Luxembourg 494

Slovak Republic 492

Norway 490

Poland 487

Spain 482

United States 477

Portugal 470

Italy 470

Greece 449

Turkey 408

Mexico 384

Non-OECD countries

Hong Kong-China 548

Macao-China 532

Liechtenstein 529

Latvia 483

Russian Federation 479

Thailand 425

Serbia and Montenegro 420

Uruguay 411

Indonesia 361

Tunisia 345

Average is significantly higher than the U.S. average

Average is not significantly different than the U.S. average

Average is significantly lower than the U.S. average

NOTE: Statistical comparisons between the U.S. average and the

Organization for Economic Cooperation and Development (OECD) average take into account the contribution of the U.S. average toward the

OECD average. The OECD average is the average of the national averages of the OECD member countries with data available. Because the

Program for International Student Assessment (PISA) is principally an

OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD

average. Due to low response rates, data for the United Kingdom are not

discussed in this report.

SOURCE: Organization for Economic Cooperation and Development

(OECD), Program for International Student Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

30

0 100 200 300 400 500 600 700

Tunisia

Indonesia

Uruguay

Serbia and Montenegro

Thailand

Russian Federation

Latvia

Liechtenstein

Macao-China

Hong Kong-China

Mexico

Turkey

Greece

Italy

Portugal

United States

Spain

Poland

Norway

Slovak Republic

Luxembourg

Ireland

Hungary

Iceland

Austria

Sweden

Germany

Czech Republic

Denmark

France

Netherlands

Switzerland

Belgium

Canada

Australia

New Zealand

Japan

Finland

Korea

Non-OECD countries

Country

OECD average

OECD countries

Average score

NOTE: The Organization for Economic Cooperation and Development (OECD) average is the average of the national averages of the OECD 

member countries with data available. Because the Program for International Student Assessment (PISA) is principally an OECD study, the 

results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD average. Due to 

low response rates, data for the United Kingdom are not discussed in this report.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2003.

10 75 90 

Percentiles of performance

Mean and 95% Confidence 

interval (+/- 2 standard error)

25 th th th th

Figure 7. Distribution of problem-solving scores of 15-year-old students, by country: 2003
31

PISA 2003 Results From the U.S. Perspective

Along with scale scores, PISA 2003 also

uses three proficiency levels (levels 1

through 3, with level 3 being the highest level

of proficiency) to describe student performance in problem solving. An additional level

(below level 1) encompasses students

whose skills cannot be described using

these proficiency levels (exhibit 9). The proficiency levels describe what students at

each level can do and allow comparisons of

the percentages of students in each country

who performed at different levels in problem

solving (see appendix A for more information about how levels were set).

Of the 38 other participating countries, 22

countries (including 16 OECD countries) had

less variation (as measured by standard

deviation) in performance in problem solving

than the United States, while 3 countries

(Belgium, Japan, and Uruguay) showed

greater variation in performance (table B14). The U.S. variation in performance was

not measurably different from the OECD

average variation. 

Exhibit 9. Description of proficiency levels for problem solving: 2003

Proficiency level Task descriptions

Level 1 At Level 1 students can solve problems where they have to deal with a single data

source containing discrete, well-defined information. They understand the nature of a

problem and consistently locate and retrieve information related to the major features of the problem. Level 1 students may be able to transform the information in the

problem to present the problem differently (e.g., take information from a table to create a drawing or graph). Also, students may be able to apply information to check a

limited number of well-defined conditions within the problem. However, Level 1 students are generally incapable of dealing with multi-faceted problems involving more

than one data source or requiring the student to reason with the information provided. 

Level 2 At Level 2 students use reasoning and analytic processes and solve problems

requiring decision-making skills. Level 2 students apply various types of reasoning (inductive and deductive reasoning, reasoning about causes and effects, or

combinatorial reasoning, that is, systematically comparing all possible variations

in well-described situations) to analyze situations and to solve problems that

require students to make a decision among well-defined alternatives. To analyze a

system or make decisions, Level 2 students combine and synthesize information

from a variety of sources. Students may need to combine various forms of representations (e.g., a formalized language, numerical information, and graphical

information), handle unfamiliar representations (e.g., statements in a proto-programming language or flow diagrams related to a mechanical or structural

arrangement of components), or draw inferences based on two or more sources of

information.

Level 3 At Level 3 students do not only analyze a system and make decisions, they also

represent the underlying relationships in a problem and relate these to the solution. Level 3 students approach problems systematically, construct their own representations and verify that their solution satisfies all requirements of the problem.

These students communicate their solutions to others using written statements

and other representations.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a majority

of items at that level. Students were classified into problem-solving levels according to their scores. Exact cut point

scores are as follows: below level 1 (a score less than or equal to 404.06); level 1 (a score greater than 404.06 and less

than or equal to 498.08); level 2 (a score greater than 498.08 and less than or equal to 592.10); level 3 (a score greater

than 592.10).

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

32

The U.S. average score of 477 on the problem-solving scale placed it at level 1, while

the OECD average score was at level 2

(table B-12, exhibit 9). The cutoff score of

604 for U.S. high performers (those in the top

10 percent in the United States) equated to 

a level 3 score, while the U.S. cutoff score of

347 for low performers (those in the bottom

10 percent) was below level 1 (table B-13,

exhibit 9).

Twenty-four percent of U.S. students scored

below level 1, 34 percent at level 1, 30 percent

at level 2, and 12 percent at level 3 (figure 8,

table B-15). The United States had greater

percentages of students below level 1 and at

level 1 than the OECD average percentages.

The United States also had a lower percentage of students at levels 2 and 3 than the

OECD average percentages. Four countries

(Finland, Hong Kong-China, Japan, and

Korea) had 30 percent or more of their students performing at level 3 in problem solving, compared with 12 percent for the United

States and 18 percent for the OECD average. 

United States

 OECD average

0 20 40 60 80 100

Country

Percent

24 34 30 12

17 30 34 18

Below level 1 Level 1 Level 2 Level 3

Figure 8. Percentage distribution of 15-year-old students in the OECD countries and the

United States on the problem-solving scale, by proficiency level: 2003

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a majority of

items at that level. Students were classified into problem-solving levels according to their scores. Exact cut point scores

are as follows: below level 1 (a score less than or equal to 404.06); level 1 (a score greater than 404.06 and less than or equal

to 498.08); level 2 (a score greater than 498.08 and less than or equal to 592.10); level 3 (a score greater than 592.10). The

Organization for Economic Cooperation and Development (OECD) average is the average of the national averages of the

OECD member countries with data available. Detail may not sum to totals because of rounding. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003.
33

PISA 2003 Results From the U.S. Perspective

8

Due to low response rates, data for the Netherlands were not discussed for PISA 2000; data for PISA 2003 for the United Kingdom

are also not discussed due to low response rates; data for Brazil were not available at the time of production for this report.

9

Large standard errors for the United States in 2000 may account at least in part for the fact that U.S. reading literacy and science literacy scores were not measurably different from 2000 to 2003 and that the scores were not different from the OECD averages in 2000.

There was no measurable change in either

the U.S. reading literacy score from 2000 to

2003 or the U.S. position compared to the

OECD average, although scores in 12 other

countries did change (table B-16).9 Four

countries saw their average reading literacy

scores increase (two non-OECD countries,

Latvia and Liechtenstein, and two OECD

countries, Luxembourg and Poland). The

United States outperformed all four of these

countries in 2000; in 2003, scores for Latvia

and Poland were not measurably different

from the U.S. scores in reading literacy,

while Liechtenstein outscored the United

States in 2003. Despite an increase in

Luxembourg’s average reading literacy

score, the United States outperformed it in

2000 and 2003.

U.S. Performance in

Reading Literacy and

Science Literacy

Of the 41 countries that participated in PISA

2003, 32 also participated in PISA 2000.

Changes in reading literacy and science literacy are reported for 29 of these 32 countries.8

In 2003, the average U.S. score in reading

literacy was 495, not measurably different

from the OECD average of 494 (figure 9,

table B-16). Eleven countries (including 

9 OECD countries) among the other 38

countries outperformed the United States

in reading literacy in 2003. 

Figure 9. Average reading literacy and science literacy scores of 15-year-old students in the

OECD countries and the United States: 2003 

494 500 495 491*

Reading literacy Science literacy

0

100

200

300

400

500

600

700

Average 

score

OECD average United States

* Average is significantly different from OECD average.

NOTE: The Organization for Economic Cooperation and Development (OECD) average is the average of the national averages

of the OECD member countries with data available. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

34

Eight countries’ scores (including seven

OECD countries) were lower in 2003 than

2000 in reading literacy. Decreases in two of

these eight countries’ scores resulted in a

change relative to the United States. Japan,

which outperformed the United States in

reading literacy in 2000, was not measurably

different in 2003, while Spain, which did not

perform measurably differently in 2000, performed worse than the United States in 2003.

In 2003, the U.S. average score in science 

literacy was 491, lower than the OECD average of 500 (figure 9, table B-17). Eighteen

countries (including 15 OECD countries)

outscored the United States in science in

2003.

There was no measurable difference

between the U.S. average science literacy

score of 499 in 2000 and 491 in 2003, although

the relative position of the United States

compared to the OECD average did change

(the U.S. science literacy score in 2000 was

not measurably different from the OECD

average, while in 2003 the U.S. score was

below the OECD average). Seventeen 

countries showed changes in their

scores from 2000 to 2003—5 countries (all OECD countries) had

lower scores in 2003 than in

2000 and 12 countries (including 9 OECD countries) had

higher scores (table B-17).

The OECD average score in

science literacy was 500 in

2000 and 2003. 

Of the 12 countries whose science literacy scores improved between 2000 and 2003, 

8 also improved their performance relative to

the United States. Belgium, the Czech

Republic, France, Germany, Liechtenstein,

and Switzerland did not perform differently

from the United States in 2000 but outscored

the United States in 2003. Latvia and the

Russian Federation scored below the U.S.

average in 2000 but were not measurably different in 2003. Of the five countries whose

science literacy scores decreased between

2000 and 2003, two (Canada and Korea) continued to outperform the United States, one

(Norway) was not measurably different in

either year, one (Mexico) performed

below the U.S. average in both

years, and one (Austria) went

from outscoring the United

States to not being measurably different from the

United States.
35

PISA 2003 Results From the U.S. Perspective

mathematics literacy, but larger percentages

of females were not seen at lower levels

(below level 1 and levels 1 through 5, table B19). In other words, differences in the overall

scores between males and females in the

United States were due at least in part to

the fact that a greater percentage of males

were found among the highest performers,

not to a greater percentage of females found

among the lowest performers. 

On average across the OECD countries,

males outperformed females on each of the

four mathematics literacy subscales (table

B-20). In the United States, differences

between males and females were evident

only on the space and shape subscale.

In the majority of the PISA 2003 countries

(32 out of 39 countries), including the United

States, there were no measurable differences in problem-solving scores by sex 

(figure 10, table B-21). However, females

outscored their male peers in problem solving in six of the remaining seven participating countries (including four OECD countries), as well as at the OECD average.

Males outscored females in problem solving

in Macao-China.

As in 2000, females in the United States and

nearly every other participating country

outscored males in reading literacy in 2003

(table B-22). Only Liechtenstein showed no

statistical difference between males and

females in 2003, although there was a difference in favor of females in 2000. 

There was no measurable difference

between the performance of U.S. males and

females in science literacy in PISA 2000 or

PISA 2003, and scores for neither group

changed between 2000 and 2003. Thirteen

countries showed differences between

males and females in 2003 (12 OECD countries and the Russian Federation). Eleven of

the 13 countries showed differences in favor

of males, but in Finland and Iceland females

outperformed males. 

Differences in

Performance by

Selected Student

Characteristics

This section provides information about how

students with various characteristics (males

and females, students of different races and

from different socioeconomic backgrounds)

performed on PISA 2003. Because PISA

2003’s emphasis was on mathematics literacy and problem solving, the focus in this

section is on performance in these areas.10

This report does not address possible

changes in performance for these groups

from 2000 to 2003.

When considering these results, it is important to bear in mind that there need not be a

cause-and-effect relationship between being

a member of a group and achievement in

PISA 2003. Student performance can be

affected by a complex mix of educational

and other factors that are not examined

here.

Sex

Fifteen-year-old females in the United

States scored 480 on the combined mathematics literacy scale, which was lower than

the average male score of 486 (figure 10,

table B-18). Males also outperformed

females in 25 other countries (20 OECD

countries and 5 non-OECD countries), a pattern evident in the OECD average scores of

494 for females and 506 for males. Iceland

was the only country in which females

scored higher in mathematics literacy than

males.

Within the United States, greater percentages of male students performed at level 6

(the highest level) than female students in

10Information on performance in reading literacy and science literacy by sex and race/ethnicity is provided, however.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

36

NOTE: Each bar above represents the average score difference between males and females on combined mathematics literacy

and problem solving. The Organization for Economic Cooperation and Development (OECD) average is the average of the

national averages of the OECD member countries with data available. Because the Program for International Student

Assessment (PISA) is principally an OECD study, the results for non-OECD countries are displayed separately from those of

the OECD countries and are not included in the OECD average. Due to low response rates, data for the United Kingdom are not

discussed in this report.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.

Uruguay

Tunisia

Thailand

Serbia and Montenegro

Russian Federation

Macao-China

Liechtenstein

Latvia

Indonesia

Hong Kong-China

Non-OECD countries

United States

Turkey

Switzerland

Sweden

Spain

Slovak Republic

Portugal

Poland

Norway

New Zealand

Netherlands

Mexico

Luxembourg

Korea

Japan

Italy

Ireland

Iceland

Hungary

Greece

Germany

France

Finland

Denmark

Czech Republic

Canada

Belgium

Austria

Australia

OECD countries

OECD average

60 40 20 0 20 40 60

Country

Combined mathematics literacy

Females Males

Problem solving

Females Males

Average score difference is statistically significant

Average score difference

Average score difference is not statistically significant

60 40 20 0 20 40 60

Average score difference

Figure 10. Differences in average scores of 15-year-old students on the combined 

mathematics literacy scale and in problem solving, by sex and country: 2003 
37

PISA 2003 Results From the U.S. Perspective

Socioeconomic Status

The measure of student socioeconomic status

(SES) used in PISA 2003 is based on the

occupational status of the student’s father or

mother (whichever was higher) as reported by

the student. Parental occupation was coded

based on the International Standard

Classification of Occupations (ISCO)

(International Labor Organization 1990).

Occupational codes were in turn mapped onto

an internationally comparable index of occupational status, the International Socioeconomic

Index (ISEI), developed by Ganzeboom, De

Graaf, and Treiman (1992). Using the index,

students were assigned numbers ranging from

about 16 to 90 based on their parents’ occupations, so that they were arrayed on a continuum from low to high socioeconomic status,

rather than placed into discrete categories.

Typical occupations among parents of 15-yearolds with between 16 and 35 points on the ISEI

scale include small-scale farmer, metalworker,

mechanic, taxi or truck driver, and waiter/waitress. Between 35 and 53 index points, the most

common occupations are bookkeeping, sales,

small business management, and nursing. As

the required skills increase, so does the status

of the occupation. Between 54 and 70 points,

typical occupations are marketing management, teaching, civil engineering, and accountant. Finally, between 71 and 90 points, the top

international quarter of the index, occupations

include medicine, university teaching, and law

(OECD 2001). 

The average ISEI index score for the United

States in 2003 was 55, higher than that of all

but two countries (Norway and Iceland) (table

B-23). Low ISEI students in the United States

were also comparatively better off in terms of

socioeconomic status than most of their

OECD peers. U.S. students with low ISEI

(those in the bottom 25 percent in the United

States) had an average index value of 33,

which was higher than the index values for low

ISEI students in 35 of the other 38 PISA 2003

countries (including 25 OECD countries). Two

countries (Japan and Norway) reported higher

average index values for low ISEI students

compared to the United States. 

Within the United States, students with low

ISEI values were outperformed in mathematics literacy by their peers with higher ISEI

values (table B-24). Moreover, U.S. students

with low ISEI values were outperformed by

their peers with low ISEI values in 22 of the

39 PISA 2003 countries (including 18 OECD

countries) for mathematics literacy.

Students with the highest ISEI background

in the United States (those in the top quarter) were outperformed by high ISEI students from 20 other countries (including 19

OECD countries) in mathematics literacy.

The overall linkage of ISEI to mathematics

literacy and problem solving can be examined by the specific change in score on the

combined mathematics literacy scale in

response to a one standard deviation change

in the ISEI index score for each country. A

greater increase in the average achievement

score in a country implies a stronger relationship between socioeconomic status and

performance in that country. 

For example, in the United States, a one

standard deviation change in the ISEI index

was associated with an average difference of

30 points on the combined mathematics literacy and 31 points on the problem-solving

scale (table B-25). In Macao-China, socioeconomic background differences in achievement were at a minimum—one standard deviation’s difference on the ISEI index was

associated with a 10 point difference on the

combined mathematics literacy scale and a

12 point difference on the problem-solving

scale. By contrast, among students in

Hungary, a one standard deviation change in

ISEI score was associated with about a 41

point difference in both mathematics literacy

and problem-solving achievement scores.

Twelve countries (including six OECD countries) had a weaker relationship between

ISEI and problem-solving performance than

the United States, while three countries

(Belgium, Germany, and Hungary) had a

stronger one. Belgium, Germany, and

Hungary also had stronger relationships

between ISEI and mathematical literacy than
International Outcomes of Learning in Mathematics Literacy and Problem Solving

38

the United States, as did the Czech Republic

and Poland. Eleven countries (including 6

OECD countries) had weaker relationships. 

Race/Ethnicity

Racial and ethnic groups vary between countries, so it is not possible to compare their performance across countries on international

assessments. Thus, this section refers only to

2003 findings for the United States. Throughout

this section, “White” refers to White, nonHispanic students, “Black” to Black, nonHispanic students, “Asian” to Asian, nonHispanic students, and “Hispanic” to Hispanic

students of any race. Results for two groups

(American Indian or Alaska Native and

Hawaiian or Other Pacific Islander) are not

shown separately because small sample sizes

did not allow for accurate estimates.

In both mathematics literacy and problem

solving, Blacks and Hispanics scored lower,

on average, than Whites, Asians, and students of more than one race (figure 11, table

B-26). Hispanic students, in turn, outscored

Black students. This pattern of performance

on PISA 2003 by race/ethnicity is similar to

that found in PISA 2000 and on the National

Assessment of Educational Progress

(NAEP) (Braswell, Daane, and Grigg 2003;

Lemke et al. 2001). 

In both mathematics literacy and problem

solving, the average scores for Blacks and

Hispanics were below the respective OECD

average scores, while scores for Whites

were above the OECD average scores.

Students who were White, Asian, and of

more than one race scored at level 3 in

mathematics literacy, compared to level 2 for

Hispanic students and level 1 for Black students (figure 11, exhibit 5). In problem solving, average scores for Whites and Asians

placed them in level 2, while Black, Hispanic,

and students of more than one race scored at

level 1 (figure 11, exhibit 9).

* Average is significantly different from OECD average.

NOTE: Reporting standards not met for American Indian/Alaska Native and Native Hawaiian/Other Pacific Islander. Black

includes African American and Hispanic includes Latino. Racial categories exclude Hispanic origin.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.

OECD 

average

White Asian More 

than

one 

race

More 

than

one 

race

Hispanic Black OECD 

average

White Asian Hispanic Black

0

100

200

300

400

500

600

Average 

score

Race/ethnicity

Mathematics literacy

500 512* 506 502

443*

417*

500 506* 505 498

436* 413*

Problem solving

700

Figure 11. Average scores of U.S. 15-year-old students on the combined mathematics literacy

scale and in problem solving, by race/ethnicity: 2003
39

PISA 2003 Results From the U.S. Perspective

For Further Information

This report provides selected findings from

PISA 2003 from a U.S. perspective. Readers

may be interested in exploring other aspects

of PISA’s results. Additional findings are

presented in the OECD report on PISA 2003

and further results will be published in a

series of OECD thematic reports on PISA

2003. Data with which researchers can conduct their own analyses are also available at

http://www.pisa.oecd.org.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

40

References
41

PISA 2003 Results From the U.S. Perspective

Ganzeboom, H.B.G., De Graaf, P., Treiman,

D.J. (with De Leeuw, J.). (1992). A

Standard International Socio-Economic

Index of Occupational Status. Social

Science Research, 21(1): 1-56.

Ganzeboom, H.B.G., and Treiman, D.J. (1996).

Internationally Comparable Measures of

Occupational Status for the 1988

International Standard Classification of

Occupations. Social Science Research,

25(3): 201-239.

International Labor Organization. (1990).

ISCO-88: International Standard

Classification of Occupations. Geneva:

Author.

Lemke, M., Calsyn, C., Lippman, L., Jocelyn,

L., Kastberg, D., Liu, Y.Y., Roey, S.,

Williams, T., Kruger, T., Bairu, G. (2001).

Outcomes of Learning: Results from the

2000 Program for International Student

Assessment of 15-Year-Olds in Reading,

Mathematics, and Science Literacy (NCES

2002–115). U.S. Department of Education,

National Center for Education Statistics.

Washington, DC: U.S. Government

Printing Office.

Neidorf, T.S., Binkley, M., Gattis, K., and

Nohara, D. (forthcoming). A Content

Comparison of the National Assessment of

Educational Progress (NAEP), Trends in

International Mathematics and Science

Study (TIMSS), and Program for

International Student Assessment (PISA)

2003 Mathematics Assessments. (NCES

2005–112). U.S. Department of Education.

Washington, DC: National Center for

Education Statistics.

Adams, R. (Ed.). (forthcoming). PISA 2003

Technical Report. Paris: Organization for

Economic Cooperation and Development.

Adams, R. (Ed.). (2002). PISA 2000 Technical

Report. Paris: Organization for Economic

Cooperation and Development.

Binkley, M., Afflerbach, P., and Kelly, D.

(forthcoming). A Content Comparison of the

NAEP and PISA Reading Assessments

(NCES 2005–110). U.S. Department of

Education. Washington, DC: National

Center for Education Statistics.

Binkley, M., and Kelly, D. (2003). A Content

Comparison of the NAEP and PIRLS FourthGrade Reading Assessments (NCES

2003–10). U.S. Department of Education.

Washington, DC: National Center for

Education Statistics Working Paper.

Braswell, J.S., Daane, M.C., and Grigg, W.S.

(2003). The Nation’s Report Card:

Mathematics Highlights 2003 (NCES

2004–451). U.S. Department of Education.

National Center for Education Statistics.

Washington, DC: U.S. Government

Printing Office. 

Dossey, J., O’Sullivan, C., and McCrone, S.

(forthcoming). Problem Solving in

International Comparative Assessments

(NCES 2005–107). U.S. Department of

Education. Washington, DC: National

Center for Education Statistics.

Ferraro, D., Czuprynski, J., and Williams, T.

(forthcoming). U.S. 2003 PISA Nonresponse

Bias Analysis. (2005–102). U.S. Department

of Education. Washington, DC: National

Center for Education Statistics.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

42

Neidorf, T.S., Binkley, M., and Stephens, M.

(forthcoming). A Content Comparison of

the National Assessment of Educational

Progress (NAEP) 2000 and Trends in

International Mathematics and Science

Study (TIMSS) 2003 Science Assessments

(NCES 2005–106). U.S. Department of

Education. Washington, DC: National

Center for Education Statistics.

Nohara, D. (2001). A Comparison of the

National Assessment of Educational

Progress (NAEP), the Third International

Mathematics and Science Study-Repeat

(TIMSS-R), and the Programme for

International Student Assessment (PISA)

(NCES 2001–07). U.S. Department of

Education. Washington, DC: National

Center for Education Statistics Working

Paper.

Organization for Economic Cooperation and

Development (OECD). (1999). Measuring

Student Knowledge and Skills: A New

Framework for Assessment. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (2000). Measuring

Student Knowledge and Skills: The PISA

2000 Assessment of Reading, Mathematical

and Scientific Literacy. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (2001). Knowledge

and Skills for Life: First Results from the

OECD Programme for International Student

Assessment. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (2002). Sample Tasks

for the PISA 2000 Assessment: Reading,

Mathematical and Scientific Literacy. Paris:

Author.

Organization for Economic Cooperation and

Development (OECD). (2003). The PISA

2003 Assessment Framework: Mathematics,

Reading, Science and Problem Solving

Knowledge and Skills. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (2004). Learning for

Tomorrow’s World — First Results From

PISA 2003. Paris: Author. 

Organization for Economic Cooperation and

Development (OECD). (2004). Problem

Solving for Tomorrow’s World — First

Measures of Cross-Curricular Skills From

PISA 2003. Paris: Author.

Raudenbush, S. W. and Bryk, A. S. (2002).

Hierarchical Linear Models: Applications

and Data Analysis Methods (2nd ed.).

Newbury Park, CA: Sage Publications.

Steen, L. A. (Ed.). (1990). On the Shoulders of

Giants: New Approaches to Numeracy.

Washington, DC: National Academy Press.

Westat. (2000). WesVar 4.0 User’s Guide.

Rockville, MD: Author.

Wilson, M. and Xie, Y. (2004). Report on the

Imperial Versus Metric Study. BEAR

Research Report, Berkeley, CA: University

of California. Available at

http://bear.berkeley.edu/pub.html.
43

PISA 2003 Results From the U.S. Perspective
International Outcomes of Learning in Mathematics Literacy and Problem Solving

44

Appendix A:

Technical Notes
45

PISA 2003 Results From the U.S. Perspective

The Program for International Student

Assessment (PISA) is a system of international assessments that measures 15-yearolds’ capabilities in reading literacy, mathematics literacy, and science literacy every

three years. PISA was first implemented in

2000 and is carried out by the Organization

for Economic Cooperation and Development

(OECD). In addition to the major subject

areas, PISA also measures general or

cross-curricular competencies such as

learning strategies. In this second cycle,

PISA 2003, mathematics literacy was the

major focus, along with the new cross-curricular cognitive domain of problem solving.

This appendix describes features of the

PISA 2003 survey methodology, including

sample design, test design, scoring, data

reliability, and analysis variables. For further

details about the assessment and any of the

topics discussed here, see the OECD’s PISA

2003 Technical Report (Adams forthcoming)

and the PISA 2000 Technical Report (Adams

2002).

Sampling, Data Collection, and Response

Rate Requirements

To provide valid estimates of student

achievement and characteristics, the sample

of PISA students had to be selected in a

way that represented the full population of

15-year-old students in each country. The

international desired population in each

country consisted of 15-year-olds attending

both publicly and privately controlled educational institutions in grades 7 and higher. A

minimum of 4,500 students from a minimum

of 150 schools was required. Within schools,

a sample of 35 students was to be selected

in an equal probability sample unless fewer

than 35 students aged 15 were available (in

which case all students were selected).

International standards required that students be sampled based on an age definition

of 15 years and 3 months to 16 years and 2

months at the beginning of the testing period.

The testing period was required not to

exceed 42 days between March 1, 2003, and

August 31, 2003. Each country collected its

own data, following international guidelines

and specifications. 

A minimum response rate target of 85 percent was required for initially selected educational institutions. In instances in which

the initial response rate of educational institutions was between 65 and 85 percent, an

acceptable school response rate could still

be achieved through the use of replacement

schools. Replacement schools were to be

selected at the time of sample selection. 

Three school response rate zones—acceptable, intermediate, and not acceptable—

were defined (figure A-1). “Acceptable”

meant that the country’s data would be

included in all international comparisons.

“Not Acceptable” meant that the country’s

data would be a candidate for not being

reported in international comparisons

unless considerable evidence was presented

that nonresponse bias was minor. “Intermediate”

meant that a decision on whether or not to

include the country’s data in comparisons

would be made while taking into account a

variety of factors, such as student response

rates, quality control, closeness of the

response rates to the acceptable level, etc.

For the purposes of calculating response

rates, schools with less than 50 percent of

students responding were considered nonresponding and their students were excluded

from the student response rates. If the student response rates within such schools

were at least 25 percent, these schools and

students were included in the PISA 2003

database. Schools with student response

rates below 25 percent were not used in any

type of analysis nor are the data for these

students or schools available in the PISA

2003 database. Note that schools with student response rates above 25 percent were

included in the nonresponse bias analyses

described in this report.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

46

six countries for schools that would otherwise have been excluded. Special booklets

were used in Austria, Belgium, the Czech

Republic, Hungary, the Netherlands, and the

Slovak Republic. Within schools, exclusion

decisions were made by staff members who

were knowledgeable about students with

Individualized Education Programs (IEPs) or

students who were limited English proficient, using the following international

guidelines on possible student exclusions:

• Functionally disabled students.

These were students who were permanently physically disabled in such a way

that they could not perform in the testing

situation. Functionally disabled students who could respond were to be

included in the testing. Any sampled

PISA 2003 also required a minimum participation rate of 80 percent of sampled students from original and replacement schools

within each country. A student was considered to be a participant if he or she participated in the first testing session or a followup or makeup testing session.

Exclusion guidelines allowed for 0.5 percent

at the school level for approved reasons (for

example, remote regions or very small

schools), and 2 percent for special education

schools. Overall estimated student exclusions to be under 5 percent. PISA’s intent

was to be as inclusive as possible. No

accommodations were offered in the United

States for PISA. A special one-hour booklet

with lower difficulty items, which was scaled

with the regular PISA booklets, was used in

0

60

55

65

70

75

80

85

90

95

100

Percent after 

replacement

Percent before replacement

55 60 65 70 75 80 85 90 95 100

Not acceptable Intermediate

Acceptable

NOTE: A minimum response target of 85 percent was required for initially selected educational institutions. In instances in

which the initial response rate of educational institutions was between 65 and 85 percent, an acceptable school response rate

could still be achieved through the use of replacement schools.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.

Figure A-1. School response rate requirements for PISA 2003
47

PISA 2003 Results From the U.S. Perspective

11The Northeast region consists of Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New

Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont. The Central region consists of Illinois, Indiana,

Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, Wisconsin, and South Dakota. The West region

consists of Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah,

Washington, and Wyoming. The Southeast region consists of Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana,

Mississippi, North Carolina, South Carolina, Tennessee, Virginia, West Virginia.

student who was temporarily disabled

such that s/he could not participate in

the assessment was considered absent

from the assessment.

• Students with mental or emotional

disabilities. These were students who

were considered in the professional

opinion of the school principal or by

other qualified staff members to be intellectually disabled or who had been psychologically tested as such. This included students who were emotionally or

mentally unable to follow even the general instructions of the test. Students were

not to be excluded solely because of

poor academic performance or normal

disciplinary problems.

• Students with limited proficiency in

the test language. These were students

who had received less than one year of

instruction in the language of the test.

Generally, these were students who were

unable to read or speak the language of

the test (English in the United States)

and would be unable to overcome the

language barrier in the test situation. 

Quality monitors from the PISA Consortium

visited schools in every country to ensure

testing procedures were carried out in a 

consistent manner across countries.

Sampling, Data Collection, and Response

Rates in the United States

The 2003 PISA school sample was drawn for

the United States in November 2002. The

sample design for this school sample was

developed to retain some of the properties

of the 2000 PISA U.S. school sample, and to

follow international requirements as given in

the PISA sampling manual. Unlike the 2000

PISA sample, which had a three-stage

design, the U.S. sample for 2003 was a twostage sampling process with the first stage

a sample of schools, and the second stage 

a sample of students within schools. For

PISA in 2000, the U.S. school sample had

the selection of a sample of geographic

Primary Sampling Units (PSUs) as the first

stage of selection. The sample was not clustered at the geographic level for PISA 2003.

This change was made in an effort to reduce

the design effects observed in the 2000 data

and to spread the respondent burden across

school districts as much as possible.

The sample design for PISA was a stratified

systematic sample, with sampling probabilities proportional to measures of size. The

PISA sample had no explicit stratification

and no oversampling of subgroups. The

frame was implicitly stratified (i.e., sorted

for sampling) by five categorical stratification variables: grade span of the school (five

levels), type of school (public or private),

region of the country11 (Northeast, Central,

West, Southeast), type of location relative to

populous areas (eight levels), minority status (above or below 15 percent). The last sort

key within the implicit stratification was by

estimated enrollment of 15-year-olds based

on grade enrollments.

At the same time that the PISA sample was

selected, replacement schools were identified following the PISA guidelines by

assigning the two schools neighboring the

sampled school on the frame as replacements. There were several constraints on

the assignment of substitutes. One sampled

school was not allowed to substitute for

another, and a given school could not be

assigned to substitute for more than one

sampled school. Furthermore, substitutes

were required to be in the same implicit stratum as the sampled school. If the sampled

school was the first or last school in the

stratum, then the second school following or

preceding the sampled school was identified

as the substitute. One was designated a first

replacement and the other a second replace-
International Outcomes of Learning in Mathematics Literacy and Problem Solving

48

data collection, the school sample included

only original schools from the sample that

had refused to participate in the spring but

indicated a willingness to participate in a

fall assessment. Substitute schools were

not included in the fall sample because their

participation would have had little effect on

raising the final response rate. In order to

achieve a comparable sample of students in

spring and fall, the age definition for students tested in the fall was adjusted such

that all students tested were the same age. 

Of the 420 sampled schools, 382 were eligible

(some did not have any 15-year-olds enrolled)

and 179 agreed to participate in the spring of

2003. An additional 70 original schools participated in the fall assessment for a total of

249 participating original schools. The school

response rate (including spring and fall

assessments) before replacement was 65

percent (weighted and unweighted), placing

the United States in the “intermediate”

response rate category. The weighted

school response rate before replacement is

given by the formula:

where Y denotes the set of responding original sample schools with age-eligible students, N denotes the set of eligible nonresponding original sample schools, Wi

denotes the base weight for school i, Wi =

1/Pi , where Pi denotes the school selection

probability for school i, and Ei denotes the

enrollment size of age-eligible students, as

indicated on the sampling frame. 

In addition to the 249 participating original

schools, 13 replacement schools also participated in the spring for a total of 262 participating schools. 

WiEi

WiEi

= 

iY

i( )

’

Y N

weighted school response 

rate before replacement

ment. If an original school refused to participate, the first replacement was then contacted. If that school also refused to participate,

the second school was then contacted. 

The U.S. PISA school sample consisted of

420 schools. This number was increased from

the international minimum requirement of

150 to offset school nonresponse, reduce

design effects, and include additional students in a metric-imperial experiment

(described below).

The schools were selected with probability

proportionate to the school’s estimated

enrollment of 15-year-olds from the 2003

NAEP school frame with 2000-2001 school

data. The data for public schools were from

the Common Core of Data (CCD), and the

data for private schools were from the

Private School Survey (PSS). Any school

containing at least one 7th- through 12thgrade class as of the school year 2000–01

was included on the school sampling frame.

Participating schools provided lists of 15-

year-old students, and a sample of 35 students was selected within each school in an

equal probability sample. The overall sample

design for the United States was intended to

approximate a self-weighting sample of students as much as possible, with each 15-

year-old student having an equal probability

of being selected.

In the United States, for a variety of reasons

reported by school administrators (such as

increased testing requirements at the

national, state, and local levels, concerns

about timing of the PISA assessment and

loss of learning time), many schools in the

original sample declined to participate. As

it was clear that the United States would not

meet the minimum response rate standards,

in order to improve response rates and better accommodate school schedules, a second testing window was opened from

September to November 2003 with the agreement of the PISA Consortium. For the fall
49

PISA 2003 Results From the U.S. Perspective

A total of 7,598 students were sampled for

the assessment. Of these students, 261 were

deemed ineligible because of their enrolled

grades, birthdays, or other reasons, and

were removed from the sample. Of the eligible 7,337 sampled students, an additional 534

students were excluded using the criteria

described above, for a weighted exclusion

rate of 7 percent. 

Of the 6,803 remaining sampled students, a

total of 5,456 students participated in the

assessment in the United States, but 114 of

these came from schools which had less

than 50 percent student participation.

Schools which had less than 50 percent student participation were classified as school

nonrespondents, and these students (114

participating students and 187 nonparticipating students) were therefore excluded for the

purposes of calculating student response

rates. Thus, although data for 5,456 students

are included in the database, student

response rates were calculated by subtracting the 114 students from the 5,456 for a

total of 5,342 participating students. The

denominator for the student response rate is

6,502, which consists of 7,598 sampled students minus the following students: 261 ineligible, 534 excluded, 114 responding students

from nonresponding schools, and 187 nonresponding students from nonresponding

schools. An overall weighted student

response of 83 percent was achieved (82 percent unweighted).

Two separate bias analyses were conducted

in the United States to address potential

problems in the data due to school nonresponse and possible achievement differences between students in spring and fall

testing windows.

The analysis of school nonresponse was

conducted in two parts, examining first the

original sample of schools (spring and fall

participants) and then the final sample of

schools (including replacements), treating

as nonrespondents those schools from

whom a final response was not received

(Ferraro, Czuprynski and Williams forthcoming). Schools with 25 to 49 percent student

response rates were treated as respondents

in the nonresponse bias analysis, since their

data are included in the PISA database.

Schools with student response rates less

than 25 percent were treated as nonrespondents in the analysis and were not included

in the PISA database.

In order to compare PISA respondents and

nonrespondents, it was necessary to match

the sample of schools back to the sample

frame to detect as many characteristics as

possible that might provide information

about the presence of nonresponse bias.

Comparing frame characteristics for respondents and nonrespondents is not always a

good measure of nonresponse bias if the

characteristics are unrelated or weakly

related to more substantive items in the survey; however, this was the only approach

available given that no comparable school or

student level achievement data were available. Frame characteristics were taken from

the 2000–01 Common Core of Data (CCD) for

public schools and from the 2000–01 Private

School Survey (PSS) for private schools.

For categorical variables, response rates by

characteristics were calculated. The hypothesis of independence between the characteristics and response status was tested using

a Rao-Scott modified Chi-square statistic.

For continuous variables, summary means

were calculated. The 95 percent confidence

interval for the difference between the mean

for respondents and the overall mean was

tested to see whether or not it included zero.

In addition to these tests, logistic regression

models were set up to identify whether any
International Outcomes of Learning in Mathematics Literacy and Problem Solving

50

While the implications of these analyses for

the direction of any resulting bias achievement are not entirely clear, an attempt was

made to minimize any bias by incorporating

the variables in question into the adjustment

for school nonresponse that was a component of the sampling weights. 

One other country, the United Kingdom, also

fell below the acceptable range for school

response rates, although response rate

problems were largely limited to England

(Scotland and Northern Ireland also participated). In that case, however, the PISA

Consortium was unable to make adjustments for any potential bias, and data for the

United Kingdom are therefore annotated and

are not included in the main text or figures.

Data for one additional participating PISA

2003 country, Brazil, were not available in

time for production of this report. 

The other U.S. bias analysis aimed to

address the question of whether there was a

“session” effect between students tested in

the spring and fall, in order to provide evidence for the acceptability of combining

data from both sessions for the United

States. Despite PISA’s focus on an age

sample, concern remained that students

tested at the beginning of the school year

might perform worse than their peers tested

at the end of the previous school year. 

The approach taken was to investigate session effects in a multilevel model, since

these were school-level effects—all students within a school were in either the

spring or the fall sessions. Two similar twolevel models were estimated. In each, student achievement in PISA was modeled as a

function of various school characteristics (in

particular those on the sample frame known

to be related to willingness to participate in

the original testing window, including public/private status, number of age-eligible

students, region, and location) and time of

testing (spring/fall) and, in one model, the

of the frame characteristics were significant

in predicting response status. All analyses

were performed using WesVar and replicate

weights to properly account for the complex

sample design. The JK2 method was used to

create the weights. The school base weights

used in these analyses did not include a

nonresponse adjustment factor. The base

weight for each original school was the

reciprocal of its selection probability. The

base weight for each replacement school

was equal to the base weight of the original

school it replaced.

Characteristics available for public and private schools included: public/private affiliation, community type, region, number of ageeligible students enrolled, total number of

students, and percentage of various

racial/ethnic groups (percentage Asian or

Pacific Islander; Black, non-Hispanic;

Hispanic; American Indian or Alaska Native;

White, non-Hispanic). Percentage of students eligible for free or reduced-price lunch

was also available for public schools only

(however, this variable was missing for 50 of

the 359 public schools). For the original sample of schools, two of these variables

showed a relationship to response status in

tests of independence and in the multivariate logistic regression model: region

(specifically, schools in the Central region

were less likely to respond) and percentage

of Asian or Pacific Islander students

(responding schools had fewer of these students than the original sample schools).

Using the same analytic procedure for the

final sample (including replacement

schools), tests of independence again

showed that responding schools were more

likely to be in the West. Responding schools

were also more likely to have fewer Asian or

Pacific Islander students and more Black,

non-Hispanic students. However, the only

variable found to be significant in the logistic

regression model predicting response was

the percentage of Asian or Pacific Islander

students (again, responding schools were

likely to have fewer Asian or Pacific Islander

students).
51

PISA 2003 Results From the U.S. Perspective

student characteristic grade level. In the

simpler of the student-level models no predictors of achievement were included. In the

second model, student grade level was

included as a predictor of achievement to

allow for the possibility that the school

means predicted in the school-level model

were affected by differences in the

spring/fall distribution of students across

grades. That is, the school-level model was

predicting mean school achievement adjusted for grade-level differences. The two models proposed were estimated with HLM

(Raudenbush and Bryk 2002). Neither model

showed evidence of a statistically significant

session effect. On this basis, and on the

basis of the adjustments made to the sampling weights based on the nonresponse

bias analysis, the PISA Consortium concluded that the data for the United States

was adequate to generalize to the U.S. 15-

year-old population and should be included

in the international report and database. 

Table A-1 provides summary information on

the samples of all countries. A more

detailed presentation can be found in the

OECD’s PISA 2003 Technical Report (Adams

forthcoming).

Test Development

The development of the PISA 2003 assessment instruments was an interactive

process among the PISA Consortium, various expert committees, and OECD members. The assessment was developed by

international experts and PISA Consortium

test developers, and items were reviewed by

representatives of each country for possible

bias and relevance to PISA’s goals. The

intention was to reflect the national, cultural, and linguistic variety among OECD countries. The assessments included material

selected from among items submitted by

participating countries as well as items that

were developed by the Consortium’s test

developers. 

The final assessment consisted of 85 mathematics items, 35 science items, 19 problem

solving items, and 32 reading items allocated

to 13 test booklets. In the United States, an

additional 4 test booklets were included in

PISA 2003 in order to investigate the possible effects of the use of metric units on U.S.

student performance, for a total of 17 booklets (see description that follows). Each

booklet was made up of 4 test clusters.

There were 7 mathematics clusters (M1 -

M7), 2 science clusters (S1 - S2), 2 problem

solving clusters (P1 - P2) and 2 reading clusters (R1 - R2). The clusters were allocated in

a rotated design to the 13 booklets. Each

cluster contained approximately 12 test

items, equivalent to 30 minutes of test material. Each student took one booklet, with

about 2 hours worth of testing material.

Approximately one-third of the mathematics

literacy items were multiple choice and complex multiple choice, one-third were closed

or short response types in which students

wrote an answer which was simply either

correct or incorrect, and about one-third

were open constructed responses for which

students wrote answers which were marked

by trained scorers based upon an international scoring guide. In PISA 2003, every

student answered mathematics items.

Problem solving, science, and reading items

were spread throughout other booklets. For

more information on assessment design, see

the OECD’s PISA 2003 Technical Report

(Adams forthcoming). 

In order to examine similarities and differences between national and international

assessments, NCES has sponsored a number of comparative studies of assessment

frameworks and items. In October 2003 a

study of the NAEP, TIMSS, and PISA 2003

mathematics assessments was undertaken.

The aim of the study was to provide information that would be useful in interpreting and

comparing the results from the three assessments, based on an in-depth look at the con-
International Outcomes of Learning in Mathematics Literacy and Problem Solving

52

Table A-1. Coverage of target population, student and school samples,

and participation rates in the Program for International

Student Assessment (PISA), by country: 2003

Country

Total 

population of

15-year-olds

(number)

Percent

Coverage 

of 

15-year-old 

population

Coverage of

national

desired 

population

Overall 

student

exclusion

rate

OECD countries

Australia 268,164 87.9 97.8 2.2

Austria 94,515 90.9 98.4 1.6

Belgium 120,802 92.6 98.5 1.5

Canada 398,865 82.8 93.2 6.8

Czech Republic 130,679 92.7 98.8 1.2

Denmark 59,156 87.5 94.7 5.3

Finland 61,107 94.7 96.6 3.4

France 809,053 90.8 96.6 3.4

Germany 951,800 92.9 98.1 1.9

Greece1 111,286 94.5 96.8 3.2

Hungary 129,138 82.9 96.1 3.9

Iceland 4,168 94.2 97.4 2.6

Ireland 61,535 89.1 95.7 4.3

Italy 561,304 85.8 98.1 1.9

Japan 1,365,471 90.8 99.0 1.0

Korea, Republic of 606,722 87.9 99.1 0.9

Luxembourg 4,204 97.1 98.4 1.6

Mexico 2,192,452 48.9 95.7 4.3

Netherlands 194,216 95.2 98.1 1.9

New Zealand 55,440 87.7 94.9 5.1

Norway 56,060 94.2 96.6 3.4

Poland 589,506 90.7 96.1 3.9

Portugal 109,149 88.7 97.7 2.3

Slovak Republic 84,242 91.5 97.0 3.0

Spain 454,064 75.8 92.7 7.3

Sweden 109,482 97.8 95.8 4.2

Switzerland 83,247 103.9 95.6 4.4

Turkey 1,351,492 35.6 99.3 0.7

United States 3,979,116 79.1 92.7 7.3

Non-OECD countries

Hong Kong-China 75,000 96.6 99.0 1.0

Indonesia2 4,281,895 46.0 99.7 0.3

Latvia 37,544 89.6 95.1 4.9

Liechtenstein 402 84.1 98.5 1.5

Macao-China 8,318 78.7 99.8 0.2

Russian Federation 2,496,216 86.3 98.3 1.7

Serbia and Montenegro3 98,729 69.5 94.3 5.7

Thailand 927,070 68.7 98.9 1.1

Tunisia4 164,758 91.6 99.6 0.4

Uruguay 53,948 62.6 99.6 0.4

United Kingdom5 768,180 90.9 94.6 5.4

See notes at end of table. 
53

PISA 2003 Results From the U.S. Perspective

Table A-1. Coverage of target population, student and school samples, and participation 

rates in the Program for International Student Assessment (PISA), by country:

2003—Continued

Country

Percent

Number of

participating

schools after

replacement

Number of

participating

students

Weighted

school 

participation

rate before

replacement

Weighted

school 

participation

rate after

replacement

Weighted 

student 

participation

rate after

replacement

OECD countries

Australia 86.3 90.4 83.3 314 12,425

Austria 99.3 99.3 83.6 192 4,597

Belgium 83.4 95.6 92.5 282 8,796

Canada 80.0 84.4 83.9 1,066 27,953

Czech Republic 91.4 99.1 89.0 259 6,320

Denmark 84.6 98.3 89.9 205 4,218

Finland 97.4 100.0 92.8 197 5,796

France 88.7 89.2 88.1 163 4,300

Germany 98.1 98.8 92.2 213 4,600

Greece1 80.6 95.8 95.4 171 4,627

Hungary 97.3 99.4 92.9 252 4,765

Iceland 99.9 99.9 85.4 129 3,350

Ireland 90.2 92.8 82.6 143 3,880

Italy 97.5 100.0 92.5 406 11,639

Japan 87.1 95.9 95.1 144 4,707

Korea, Republic of 95.9 100.0 98.8 149 5,444

Luxembourg 99.9 99.9 96.2 29 3,923

Mexico 94.0 95.5 92.3 1,102 29,983

Netherlands 82.6 87.9 88.3 153 3,992

New Zealand 91.1 97.6 85.7 171 4,511

Norway 87.9 90.4 87.9 180 4,064

Poland 95.1 98.1 82.0 163 4,383

Portugal 99.3 99.3 87.9 152 4,608

Slovak Republic 78.9 99.1 91.9 281 7,346

Spain 98.4 100.0 90.6 383 10,791

Sweden 99.1 99.1 92.6 185 4,624

Switzerland 97.3 98.5 94.7 444 8,420

Turkey 93.3 100.0 96.9 159 4,855

United States 64.9 68.1 82.7 262 5,456

Non-OECD countries

Hong Kong-China 81.9 95.9 90.2 145 4,478

Indonesia2 100.0 100.0 98.1 344 10,761

Latvia 95.3 95.3 93.9 157 4,627

Liechtenstein 100.0 100.0 98.2 12 332

Macao-China 100.0 100.0 98.0 39 1,250

Russian Federation 99.5 100.0 95.7 211 5,974

Serbia and Montenegro3 100.0 100.0 91.4 149 4,405

Thailand 91.5 100.0 97.8 179 5,236

Tunisia4 100.0 100.0 96.3 149 4,721

Uruguay 93.2 97.1 90.8 239 5,835

United Kingdom5 64.3 77.4 77.9 361 9,535

1

Fifteen-year-olds in primary school in Greece were originally excluded from the assessment. Changes in the target population

definition to 15-year-olds in grades 7 and above required Greece to adjust its data to reflect the fact that 15-year-olds in primary

school would no longer be considered part of the target population.

2

Indonesia excluded 4 provinces and close to 5 percent of its eligible population for security reasons. There were 4,137,103 15-yearolds in the total population, but the 4 provinces were already excluded. Therefore, the 144,792 noted as being excluded in these

provinces was added to this number to get 4,281,895 15-year-olds. The number of enrolled 15-year-olds was noted as 2,968,756 so

144,792 was also added to this.

3

Serbia and Montenegro excluded Kosovo; however, there were no estimates for the number of 15-year-olds, so this does not

appear as an exclusion.

4

Tunisia noted late in the process that one French school needed to be excluded because of French (rather than Arabic) language.

The school had 33 eligible students.

5

Due to low response rates, data for the United Kingdom are not discussed in this report.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

54

adapt the instrument for cultural purposes,

even in nations such as the United States

that use English as the primary language of

instruction. For example, words such as

“lift” might be adapted to “elevator” for the

United States. The PISA Consortium verified the national translation and adaptation

of all instrumentation. Copies of printed

materials were sent to the PISA

Consortium for a final optical check prior to

data collection.

As noted, in the United States, an additional

4 test booklets were included in PISA 2003

that used adapted versions of 27 mathematics items. These items in their original format used metric units of measurement, such

as meters, liters, etc. To investigate the possible effects of the use of metric units on

U.S. student performance, the items were

adapted to use “imperial” forms with familiar units such as feet, gallons, and degrees

Fahrenheit. Differential item analysis

showed that U.S. students were not disadvantaged by the use of metric units in PISA

2003. The few discrepancies that were

observed are possibly due to (1) differences

in the nature of the two systems (e.g., decimal vs. duodecimal, or no equivalent wording

of the units), and (2) difficulties in the modification process (e.g., no comparable scoring guides for some incorrect approaches to

an item). For more information, see Wilson

and Xie (2004).

Test Administration and Quality Assurance

PISA 2003 emphasized the use of standardized procedures in all countries. Each country collected its own data, based on comprehensive manuals and trainings provided by

the PISA Consortium to explain the survey’s

implementation, including precise instructions for the work of school coordinators and

scripts for test administrators for use in

testing sessions. Test administration in the

United States was carried out by professional staff trained according to the internatent of the respective frameworks and items.

The results showed that PISA used far

fewer multiple choice items and had a much

stronger content focus on the “data” area

(which often deals with using charts and

graphs), which fits with PISA’s emphasis on

using materials with a real-world context.

For more results from the study, see A

Content Comparison of the NAEP, TIMSS,

and PISA 2003 Mathematics Assessments

(Nohara forthcoming). An earlier study

compared NAEP 2000, PISA 2000, and

TIMSS 1999 mathematics and science items.

That study found that PISA items required

multistep reasoning more often than TIMSS

or NAEP and that PISA mathematics and

science literacy items more often involved

the interpretation of charts and graphs or

other “real life” material (Nohara 2001).

In addition to the cognitive assessment, 

students also received a 30-minute questionnaire designed to provide information about

their backgrounds, attitudes, and experiences

in school. Principals in schools were PISA

was administered also received a 20-30

minute questionnaire about their schools.

Results from the school survey are not discussed in this report but are available at

http://www.pisa.oecd.org.

Translation and the Metric-Imperial Study

Source versions of all instruments (assessment booklets, questionnaires, and manuals) were prepared in English and French

and translated into the primary language or

languages of instruction in each nation.

PISA recommended that countries prepare

and consolidate independent translations

from both source versions, and provided

precise translation guidelines that included

a description of the features each item was

measuring and statistical analysis from the

field trial. In cases where one source language was used, independent translations

were required and discrepancies reconciled.

In addition, it was sometimes necessary to
55

PISA 2003 Results From the U.S. Perspective

tional guidelines. School staff were asked

only to assist with listings of students, identifying space for testing in the school, and

specifying any parental consent procedures

needed for sampled students. Use of calculators was at the discretion of participating

countries; in the United States, this choice

was left to schools based on school, district,

or state policy. Students were asked at the

end of their test booklets if they had used a

calculator and if so, what type. Approximately

12 percent of U.S. students did not respond.

Of the responding students, 91 percent of

U.S. students reported using a calculator.

Students who reported using a calculator

had a mean score of 498 on the combined

mathematics literacy scale compared to 461

for those who reported not using a calculator. 

Members of the PISA Consortium visited all

national centers to review data collection

procedures, and members of the PISA

Consortium also visited a randomly selected

subsample of approximately 10 percent of

the educational institutions to ensure that

procedures were being carried out in accordance with international guidelines. For a

detailed description of the quality assurance

procedures, see the OECD’s PISA 2003

Technical Report (Adams forthcoming).

Scoring

At least one-third of the PISA assessment

was devoted to items requiring constructed

responses. The process of scoring these

items was an important step in ensuring the

quality and comparability of the PISA data.

Detailed guidelines were developed for the

scoring guides themselves, training materials to recruit scorers, and workshop materials used for the training of national scorers.

Prior to the national training, the PISA

Consortium organized training sessions to

present the material and train the scoring

coordinators from the participating countries, who trained the national scorers.

For each test item, the scoring guide

described the intent of the question and how

to code the students’ responses to each

item. This description included the credit

labels—full credit, partial credit, or no credit—attached to the possible categories of

response. Also included was a system of

double-digit coding for some mathematics

and science items where the first digit represented the score, and the second digit represented different strategies or approaches

that students used to solve the problem. The

second digit generated national profiles of

student strategies and misconceptions. In

addition, the scoring guides included real

examples of students’ responses accompanied by a rationale for their classification for

purposes of clarity and illustration.

To examine the consistency of this marking

process in more detail within each country

and to estimate the magnitude of the variance components associated with the use of

markers, the PISA Consortium conducted

an interscorer reliability study on a subsample of assessment booklets. Homogeneity

analysis was applied to the national sets of

multiple scoring and compared with the

results of the field trial. A full description of

this process and the results can be found in

the PISA 2003 Technical Report published by

the OECD (Adams forthcoming).

Data Entry and Cleaning

Responsibility for data entry was taken by the

national project manager from each nation.

The data collected for PISA 2003 were

entered into data files with a common international format, as specified in the 

PISA 2003 Data Entry Manual. Data entry was

facilitated by the use of a common software

available to all participating nations

(KeyQuest). The software facilitated the

checking and correction of data by providing

various data consistency checks. The data

were then sent to the Australian Council for
International Outcomes of Learning in Mathematics Literacy and Problem Solving

56

weight for each replacement school was

equal to the base weight of the original

school it replaced.

Scaling and Plausible Values

PISA used Item Response Theory (IRT)

methods to produce scale scores that summarized the achievement results. PISA 2003 utilized a mixed coefficients multinomial logit

IRT model. This model is similar in principle

to the more familiar two-parameter IRT

model. With this method, the performance of

a sample of students in a subject area or sub

area can be summarized on a simple scale or

a series of scales, even when different students are administered different items.

Because of the reporting requirements for

PISA and the large number of background

variables associated with the assessment,

PISA used these IRT procedures to produce

accurate results for groups of students while

limiting the testing burden on individual students. Furthermore, these procedures provided data that could be readily used in secondary analyses. IRT scaling provides estimates

of item parameters (e.g., difficulty, discrimination) that define the relationship between the

item and the underlying variable measured by

the test. Parameters of the IRT model are

estimated for each test question, with an

overall scale being established as well as

scales for each predefined content area specified in the assessment framework. For example, PISA 2003 had five scales describing

mathematics (a combined score and subscale

scores in four domains) and one each for

reading, problem solving, and science.

The reading literacy and science literacy

reporting scales used for PISA 2000 and

PISA 2003 are directly comparable. The

value of 500, for example, has the same

meaning as it did in PISA 2000—that is, the

mean score in 2000 of the sampled students

in the 27 OECD countries that participated

in PISA 2000.

Educational Research (ACER) for cleaning.

ACER’s role in this instance was to check that

the international data structure was followed,

check the identification system within and

between files, correct single case problems

manually, and apply standard cleaning procedures to questionnaire files. Results of the

data cleaning process were documented and

shared with the national project managers

and included specific questions when

required. The national project manager then

provided ACER with revisions to coding or

solutions for anomalies. ACER then compiled

background univariate statistics and preliminary classical and Rasch Item Analysis.

Detailed information on the entire data entry

and cleaning process can be found in the

forthcoming PISA 2003 technical report.

Weighting

Students included in the final PISA sample

for a given country were not all equally representative of the full student population,

even though random samplings of schools

and students were used to select the sample. The use of sampling weights is necessary for the computation of statistically

sound, nationally representative estimates.

Survey weights help adjust for intentional

over- or under-sampling of certain sectors of

the population, school or student nonresponse, or errors in estimating size of a

school at the time of sampling. Survey

weighting for PISA 2003 was carried out by

Westat, as part of the PISA Consortium.

The internationally defined weighting specifications for PISA required that each

assessed student’s sampling weight be the

product of the inverse of the school’s probability of selection, an adjustment for schoollevel nonresponse, the inverse of the student’s probability of selection, and an

adjustment for student-level nonresponse.

All PISA analyses were conducted using

these adjusted sampling weights. The base
57

PISA 2003 Results From the U.S. Perspective

This is not the case, however, for mathematics literacy. Mathematics literacy, as the

major domain, was the subject of major

development work for PISA 2003, and the

PISA 2003 mathematics literacy assessment

was much more comprehensive than the

PISA 2000 mathematics assessment—the

PISA 2000 assessment covered just two

(space and shape, and change and relationships) of the four areas that are covered in

PISA 2003. Because of this broadening in

the assessment it was deemed inappropriate

to report the PISA 2003 mathematics literacy scores on the same scale as the PISA

2000 mathematics scores.

The PISA 2000 and PISA 2003 assessments

of mathematics, reading and science literacy are linked assessments. That is, the sets

of items used to assess each of mathematics, reading and science literacy in PISA

2000 and the sets of items used to assess

each of mathematics, reading and science

literacy in PISA 2003 include a subset of

items common to both sets. For mathematics there were 20 items that were used in

both assessments, in reading there were 28

items used in both assessments and for science 25 items were used in both assessments. These common items are referred to

as link items.

To establish common reporting metrics for

PISA 2000 and PISA 2003 the difficulty of

link items (items used in 2000 and 2003) was

compared. Items were calibrated using 2003

data only, and then 2000 items were re-calibrated using the 2003 parameters. Adjustments

were then made to ability estimate to

account for booklet effects seen in 2000. The

comparison of the item difficulties on the

two occasions was used to determine a

score transformation that allows the reporting of the data from the two assessments on

a common scale. The change in the difficulty

of each of the individual link items is used in

determining the transformation and as a

consequence the sample of link items that

has been chosen will influence the choice of

transformation. This means that if an alternative set of link items had been chosen the

resulting transformation would be slightly

different. The consequence is an uncertainty

in the transformation due to the sampling of

the link items, just as there is an uncertainty

in values such as country means due to the

use of a sample of students. The section on

statistical testing below describes how this

uncertainty has been accounted for in making comparisons over time.

Plausible Values

During the scaling phase, plausible values

were used to characterize scale scores for

students participating in the assessment. To

keep student burden to a minimum, PISA

administered few assessment items to each

student—too few to produce accurate content-related scale scores for each student.

To account for this, PISA generated five

possible scale scores for each student that

represented selections from the distribution

of scale scores of students with similar

backgrounds who answered the assessment

items the same way. The plausible values

technology is one way to ensure that the

estimates of the average performance of

student populations and the estimates of

variability in those estimates are more accurate than those determined through traditional procedures, which estimate a single

score for each student. During the construction of plausible values, careful quality control steps ensured that the subpopulation

estimates based on these plausible values

were accurate.

It is important to recognize that plausible

values are not test scores for individuals and

they should not be treated as such.

Plausible values are randomly drawn from

the distribution of scores that could be reasonably assigned to each individual. As

such, the plausible values contain random
International Outcomes of Learning in Mathematics Literacy and Problem Solving

58

error variance components and are not optimal as scores for individuals. The PISA student file contains many plausible values, five

for each of the PISA 2003 cognitive scales

(combined mathematics literacy scale, four

mathematics literacy subscales, reading literacy, science literacy, and problem solving).

If an analysis is to be undertaken with one of

these cognitive scales, then (ideally) the

analysis should be undertaken five times,

once with each of the five relevant plausible

value variables. The results of these five

analyses are averaged and then significance

tests that adjust for variation between the

five sets of results are computed.

PISA uses the plausible value methodology

to represent what the true performance of an

individual might have been, had it been

observed, using a small number of random

draws from an empirically derived distribution of score values based on the student’s

observed responses to assessment items

and on background variables. Each random

draw from the distribution is considered a

representative value from the distribution of

potential scale scores for all students in the

sample who have similar characteristics and

identical patterns of item responses. The

draws from the distribution are different

from one another to quantify the degree of

precision (the width of the spread) in the

underlying distribution of possible scale

scores that could have caused the observed

performance. The PISA plausible values

function like point estimates of scale scores

for many purposes, but they are unlike true

point estimates in several respects. They

differ from one another for any particular

student, and the amount of difference quantifies the spread in the underlying distribution of possible scale scores for that student. Because of the plausible values

approach, secondary researchers can use

the PISA data to carry out a wide range of

analyses.

Levels

While the basic form of measurement in

PISA describes student literacy in each

country in terms of a range of scale scores,

PISA also treats proficiency in mathematics

literacy in terms of six described levels, and

proficiency in problem solving in three

described levels. In both cases, increasing

levels represent tasks of increasing complexity. As a result, the findings are reported

in terms of percentages of the population

proficient at handling tasks of different levels of difficulty.

Each of the five mathematics literacy

scales—the combined score and the four

subscale scores—is divided into six levels

based on the type of knowledge and skills

students need to demonstrate at a particular

level. A seventh level (below level 1) is made

up of students whose abilities could not be

accurately described based on their

responses. Exact cut point scores are as follows: below level 1 (a score less than or

equal to 357.77); level 1 (a score greater than

357.77 and less than or equal to 420.07); level

2 (a score greater than 420.07 and less than

or equal to 482.38); level 3 (a score greater

than 482.38 and less than or equal to 544.68);

level 4 (a score greater than 544.68 and less

than or equal to 606.99); level 5 (a score

greater than 606.99 and less than or equal to

669.30); level 6 (a score greater than 669.30.

The tasks that represent each level of performance for the specific mathematics

processes on the combined mathematics literacy scale are described in exhibit 5. Exhibit

A-1 describes the kind of tasks that represent each level of performance on the mathematics subscales.

The problem-solving scale is divided into

three levels based on the type of knowledge

and skills students must demonstrate at a

particular level. A fourth level (below level 1)

is made up of students whose abilities could

not be accurately described based on their

responses. In order to reach a particular
59

PISA 2003 Results From the U.S. Perspective

Exhibit A-1. Description of proficiency levels for mathematics literacy 

subscales: 2003

Proficiency level 

Task descriptions

Space and shape Quantity

Level 1 Students at Level 1 or 2 can

work with a single mathematical representation where the

mathematical content is direct

and clearly presented, use

mathematical thinking in familiar contexts, identify geometric

patterns, and apply basic geometric concepts.

Students at Level 1 or 2 can

interpret simple tables, carry

out basic arithmetic calculations, work with simple quantitative models, interpret a simple quantitative model (e.g., a

proportional relationship), and

apply the model using basic

arithmetic calculations.

Level 2 

Level 3 Students at Level 3 can begin

to use visual and spatial reasoning, begin linking different

representations, use elementary problem solving (devising

simple strategies), apply simple algorithms, and interpret

textual descriptions of unfamiliar geometric situations.

Students at Level 3 can use simple problem solving strategies,

interpret tables to locate information, carry out well-described

calculations, interpret a text

description of a sequential calculation process, correctly implement the process, and use basic

problem-solving procedures.

Level 4 Students at Level 4 can use

more advanced and flexible

reasoning, link and integrate

different representations, use

multi-step processes, use

well-developed spatial visualization and interpretation, and

use reasoning about numeric

relationships in geometric

problems. 

Students at Level 4 can work

effectively with simple models

of complex situations, use reasoning skills, insight and interpretation with different representations, use a variety of calculation skills to solve problems, and accurately apply a

given numeric algorithm involving a number.

Level 5 Students at Level 5 have the

ability to make or work with

assumptions, use insight, interpretation and linking of different representations, and can

carry out multiple and sequential processes. They can also

use well-developed spatial reasoning.

Students at Level 5 have the

ability to work effectively with

increasingly complex situations and models and have

well-developed reasoning

skills. They can also use insight

and interpretation of different

representations and carry out

multiple sequential problems.

Level 6 Students at Level 6 can manipulate complex and multiple representations, link different information, use significant insight and

reflection, make generalizations,

communicate the solution and

explanation of a problem in

unstructured form, and interpret

complex textual descriptions and

relate these to other problems.

Students at Level 6 can conceptualize and work with complex mathematical processes

and relationships, use

advanced thinking and reasoning skills to link multiple contexts, use sequential calculation processes, and conceptualize complex mathematical

processes.

See notes at end of exhibit.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

60

Exhibit A-1. Description of proficiency levels for mathematics literacy 

subscales: 2003—Continued

Proficiency level 

Task descriptions

Change and relationships Uncertainty

Level 1 Students at Level 1 or 2 can

work with simple algorithms,

formula, and procedures, link

text with single representations, begin to interpret and

use elementary reasoning, and

interpret text to produce a simple mathematical model in an

applied context.

Students at Level 1 or 2 can

understand and use basic

probabilistic ideas in familiar

experimental contexts, locate

statistical information presented in familiar graphical form,

and understand basic probability concepts in the context of a

simple and familiar experiment.

Level 2 

Level 3 Students at Level 3 can work

with related representations

(text, graph, table and simple

algebra) including some interpretation and reasoning, interpret unfamiliar graphical representations of real-world situations, and link and connect multiple related representations.

Students at Level 3 can interpret information and data, link

different information sources,

use basic reasoning with simple probability concepts, interpret tabular information, use

insight into aspects of data

presentation, and link data to

suitable chart types.

Level 4 Students at Level 4 can understand and work with multiple

representations, including

explicitly mathematical models of real-world situations,

carry out a sequence of calculations involving percentage or

proportion, and show insight

into three-dimensional geometric problems.

Students at Level 4 can use

basic statistical and probability

concepts combined with logical

reasoning in less familiar contexts, use argumentation based

on interpretation of data, interpret text, including in an unfamiliar (scientific) context, and

translate text description into

mathematics problems.

Level 5 Students at Level 5 have quite

advanced use of algebraic and

other formal mathematical

expressions and models and

have the ability to link formal

mathematical representations

to complex real-world situations. They can also solve complex and multi-step problems.

Students at Level 5 can apply

statistical knowledge in situations that are somewhat structured and where the mathematical representation is partially

apparent and use reasoning

and insight to interpret given

information.

Level 6 Students at Level 6 can use significant insight, well-developed

reasoning skills and explicit

technical knowledge to solve

problems and to begin to generalize mathematical solutions to

complex real-world problems

and can interpret complex

mathematical information in

the context of a problem.

Students at Level 6 can use

high-level thinking and reasoning skills in statistical or probabilistic contexts to create

mathematical representations

of real-world situations, use

insight, reflection and argumentation to communicate

arguments and explanations,

and interpret and reflect.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly

answer a majority of items at that level. Students were classified into mathematics literacy levels according to their scores. Exact cut point scores are as follows: below level 1 (a score less

than or equal to 357.77); level 1 (a score greater than 357.77 and less than or equal to 420.07); level 2

(a score greater than 420.07 and less than or equal to 482.38); level 3 (a score greater than 482.38 and

less than or equal to 544.68); level 4 (a score greater than 544.68 and less than or equal to 606.99);

level 5 (a score greater than 606.99 and less than or equal to 669.3); level 6 (a score greater than

669.3).

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for

International Student Assessment (PISA), 2003.
61

PISA 2003 Results From the U.S. Perspective

Nonsampling Errors

Nonsampling error is a term used to

describe variations in the estimates that

may be caused by population coverage limitations, nonresponse bias, and measurement

error, as well as data collection, processing,

and reporting procedures. For example, the

sampling frame was limited to regular public

and private schools in the 50 states and the

District of Columbia. The sources of nonsampling errors are typically problems like

unit and item nonresponse, the differences

in respondents’ interpretations of the meaning of the questions, response differences

related to the particular time the survey was

conducted, and mistakes in data preparation. Some of these issues (particularly unit

nonresponse) are discussed above in the

section on U.S. sampling and data collection.

Missing Data

There are four kinds of missing data.

“Nonresponse” data occurs when a respondent was expected to answer an item but no

response is given. Responses that are

“missing or invalid” occur in multiple-choice

items where an invalid response is given.

The code is not used for open-ended questions. An item is “not applicable” when it is

not possible for the respondent to answer

the question. Finally, items that are “not

reached” are consecutive missing values

starting from the end of each test session.

All four kinds of missing data are coded 

differently in the PISA 2003 database.

Missing background data are not included in

the analyses for this report and are not

imputed. In general, item response rates for

variables discussed in this report were over

the NCES standard of 85 percent to report

without notation (table A-2). The one case

in which more than 15 percent of the student

responses were missing (for New Zealand

for student report of parent occupation, with

an item response rate of 84 percent) is

flagged in the supporting statistical data

tables in appendix B. 

proficiency level, a student must have been

able to correctly answer a majority of items

at that level. Students were classified into

problem-solving levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or

equal to 404.06); level 1 (a score greater than

404.06 and less than or equal to 498.08); level

2 (a score greater than 498.08 and less than

or equal to 592.10); level 3 (a score greater

than 592.10).

All students within a level are expected to

answer at least half of the items from that

level correctly. Students at the bottom of a

level have a 62 percent chance of success on

the easiest items from that level and a 42

percent chance of success on the hardest

items from that level (overall response probability was 62). Students at the top of a level

are able to provide the correct answers to

about 70 percent of all items from that level,

have a 62 percent chance of success on the

hardest items from that level, and have a 78

percent chance of success on the easiest

items from that level. Students just below

the top of a level would score less than 50

percent on an assessment of the next higher

level. Students at a particular level not only

demonstrate the knowledge and skills associated with that level but also the proficiencies defined by lower levels. Thus, all students proficient at level 3 are also proficient

at levels 1 and 2. Patterns of responses for

students below level 1 suggest they are

unable to answer at least half of the items in

level 1 correctly.

Data Limitations

As with any study, there are limitations to

PISA 2003 that researchers should take into

consideration. Estimates produced using

data from PISA 2003 are subject to two

types of error, nonsampling and sampling

errors. Nonsampling errors can be due to

errors made in the collection and processing

of data. Sampling errors can occur because

the data were collected from a sample rather

than a complete census of the population.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

62

In general, it is difficult to identify and estimate either the amount of nonsampling error

or the bias caused by this error. In PISA

2003, efforts were made to prevent such

errors from occurring and to compensate for

them when possible. For example, the design

phase entailed a field test that evaluated

items as well as the implementation procedures for the survey. It should also be recognized that most background information

was obtained from students’ self-reports,

which are subject to respondent bias. One

potential source of respondent bias in this

survey was social desirability bias, for example, if students reported that they were good

at mathematics. 

Sampling Errors

Sampling errors occur when the discrepancy

between a population characteristic and the

sample estimate arises because not all

members of the reference population are

sampled for the survey. The size of the sample relative to the population and the variability of the population characteristics both

influence the magnitude of sampling error.

The particular sample of 15-year-old students from the 2002–03 school year was just

one of many possible samples that could

have been selected. Therefore, estimates

produced from the PISA 2003 sample may

differ from estimates that would have been

produced had another sample of 15-year-old

students been drawn. This type of variability

was called sampling error because it arises

from using a sample of 15-year-old students

in 2002, rather than all 15-year-old students

in that year.

The standard error is a measure of the variability due to sampling when estimating a

statistic. The approach used for calculating

sampling variances in PISA was the

Balanced Repeated Replication (BRR), or

Balanced Half-Samples (Fay’s method).

Standard errors can be used as a measure

for the precision expected from a particular

sample.

Standard errors for all of the estimates are

included in appendix B to this report. These

standard errors can be used to produce confidence intervals. There is a 95 percent

chance that the true average lies within the

range of 1.96 times the standard errors above

or below the estimated score. For example, it

was estimated that 15.5 percent of U.S. students scored at level 1 on the combined

mathematics literacy scale, and this statistic had a standard error of 0.81. Therefore, it

can be stated with 95 percent confidence

that the actual percentage of U.S. students

at level 1 for the total population in 2003 was

between 13.9 and 17.1 percent (1.96 x 0.81 =

1.59; confidence interval = 15.5 +/- 1.59).

Descriptions of Background Variables

Full PISA 2003 student and school questionnaires are available at

http://nces.ed.gov/surveys/pisa or

http://www.pisa.oecd.org.

Socioeconomic Status

The measure of student socioeconomic status

used in PISA 2003 is based on the occupational status of the student’s father and/or

mother (whichever is higher) as reported by

the student. Parental occupation was coded

to 4 digits based on the International

Standard Classification of Occupations

(ISCO). Occupational codes were in turn

mapped onto an internationally comparable

index of occupational status, the International

Socioeconomic Index (ISEI), developed by

Ganzeboom, De Graaf, and Treiman (1992).

Using the index, students were assigned numbers ranging from about 16 to 90 based on

their parents’ occupations, so that they were

arrayed on a continuum from low to high

socioeconomic status, rather than placed into

discrete categories. The range of ISEI scores

given for the 1988 ISCO occupations listed in

Ganzeboom and Treiman (1996) goes from 16,

the lowest (agricultural laborer), to 90, the

highest (judge). Typical occupations among
63

PISA 2003 Results From the U.S. Perspective

Table A-2. Response rates of 15-year-old students for selected background

variables, by country: 2003

Country

Selected background variable

Sex

Socioeconomic

status (ISEI)

Race/

ethnicity

Type of mathematics class

OECD countries

Australia 100 95 † †

Austria 100 96 † †

Belgium 100 94 † †

Canada 97 93 † †

Czech Republic 100 96 † †

Denmark 100 97 † †

Finland 100 99 † †

France 100 96 † †

Germany 99 91 † †

Greece 100 94 † †

Hungary 100 94 † †

Iceland 100 98 † †

Ireland 100 96 † †

Italy 100 98 † †

Japan 100 89 † †

Korea, Republic of 100 98 † †

Luxembourg 100 96 † †

Mexico 100 95 † †

Netherlands 100 93 † †

New Zealand 100 84 † †

Norway 100 97 † †

Poland 100 98 † †

Portugal 100 97 † †

Slovak Republic 100 96 † †

Spain 100 96 † †

Sweden 100 97 † †

Switzerland 100 97 † †

Turkey 100 88 † †

United States 100 94 97 95

Non-OECD countries

Hong Kong-China 100 96 † †

Indonesia 100 91 † †

Latvia 100 97 † †

Liechtenstein 100 97 † †

Macao-China 100 97 † †

Russian Federation 100 98 † †

Serbia and Montenegro 100 94 † †

Thailand 100 95 † †

Tunisia 100 95 † †

Uruguay 100 90 † †

United Kingdom1 100 94 † †

† Not applicable.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: Cases where more than 15 percent of the student responses are missing are flagged in the supporting statistical data tables in appendix B. For more information about the variables, see the Description of

Variables section in appendix B. The overall percentage refers to the sample estimate for the overall 15-

year-old student population. The International Socioeconomic Index (ISEI) is an internationally comparable index of occupational status, with a range of approximately 16 to 90, developed by Ganzeboom, De

Graaf, and Treiman (1992).

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

64

PISA schools and adding an additional

measure of uncertainty of school and student identification through random swapping

of data elements within the student and

school files. 

Statistical Procedures

Tests of Significance

Comparisons made in the text of this report

have been tested for statistical significance.

For example, in the commonly made comparison of country averages against the average

of the United States, tests of statistical significance were used to establish whether or

not the observed differences from the U.S.

average were statistically significant.

The estimation of the standard errors that

are required in order to undertake the tests

of significance is complicated by the complex sample and assessment designs which

both generate error variance. Together they

mandate a set of statistically complex procedures in order to estimate the correct standard errors. As a consequence, the estimated standard errors contain a sampling variance component estimated by Balanced

Repeated Replication (BRR)—the Fay

method of BRR; and, where the assessments

are concerned, there is an additional imputation variance component arising from the

assessment design. Details on the BRR

procedures used can be found in the WesVar

4.0 User’s Guide (Westat 2000).

In almost all instances, the tests for significance used were standard t tests. These fell

into two categories according to the nature

of the comparison being made: comparisons

of independent and non-independent samples. In PISA, country samples are independent. To determine whether the average

scores for two countries are different we

test the null hypothesis:

H 0 : μˆ (country1) – μˆ (country2)= 0

parents of 15-year-olds with between 16 and

35 points on the ISEI scale include smallscale farmer, metalworker, mechanic, taxi or

truck driver, and waiter/waitress. Between 35

and 53 index points, the most common occupations are bookkeeping, sales, small business management, and nursing. As the

required skills increase, so does the status of

the occupation. Between 54 and 70 points,

typical occupations are marketing management, teaching, civil engineering, and accountant. Finally, between 71 and 90 points, the top

international quarter of the index, occupations

include medicine, university teaching, and law

(OECD 2001). 

Race/Ethnicity

In the United States, students’ race/ethnicity was obtained through student responses

to a two-part question. Students were asked

first whether they were Hispanic or Latino,

and then asked whether they were members

of the following racial groups: American

Indian or Alaska Native, Asian, Black or

African American, Native Hawaiian or other

Pacific Islander, or White. Multiple responses to the race classification question were

allowed. Results are shown separately for

Asians, Blacks, Hispanics, Whites, and students who selected more than one race.

Students identifying themselves as Hispanic

and also other races were included in the

Hispanic group, rather than in a racial group.

Confidentiality and Disclosure Limitations

The PISA 2003 data are hierarchical and

include school data and student data from

the participating schools. Confidentiality

analyses for the United States were

designed to provide reasonable assurance

that public use data files issued by the PISA

Consortium would not allow identification of

individual U.S. schools or students when

compared against public data collections.

Disclosure limitation included the identification and masking of potential disclosure-risk
65

PISA 2003 Results From the U.S. Perspective

To test this hypothesis, the two observed

values and their respective standard errors

are needed to perform a t test. The standard

error on the estimate for some statistic  is:

Thus, in simple comparisons of independent

averages, such as the average score of

country 1 with that of country 2, the following

formula was used to compute the t-statistic:

t = est1 - est2 / SQRT[(se1)2 + (se2)2

]

where est1 and est2 are the estimates being

compared (e.g., averages of country 1 and

country 2) and se1 and se2 are the corresponding standard errors of these averages.

This test may also be used for comparisons

within a particular country if the categorical

variable used to define the groups being

compared was used as an explicit stratification variable; however, there was no explicit

stratification used in the United States

sample. 

The second type of comparison used in this

report occurred when comparing differences

of non-subset, non-independent groups.

When this occurs, the correlation and related covariance between the groups must be

taken into account, such as when comparing

a country mean with the OECD mean which

includes that particular country, or when

comparing the average scores of males versus females within the United States.

How are scores like those for μˆ (boys) and μˆ (girls)

correlated? Suppose that in the school sample, a coeducational school attended by low

achievers is replaced by a coeducational

school attended by high achievers. The country mean will increase slightly, as well as 

the males’ and the females’ means. If such 

a school replacement process is continued, 

μˆ (boys) and μˆ (girls) will likely increase in a similar

2

) ˆ(

2

) ˆ ) ( ˆ ˆ ( i j i j θ θ θ θ σ = σ +σ −

pattern. Indeed, a coeducational school

attended by high achieving males is usually

also attended by high achieving females.

Therefore, the covariance between μˆ (boys) and

μˆ (girls) will be positive.

What does the covariance between the two

variables, i.e., μˆ (boys), μˆ (girls) , tell us? A positive

covariance means that if μˆ (boys) increases

then μˆ (girls) will also increase. A covariance

equal or close to 0 means that μˆ (boys) can

increase or decrease with μˆ (girls) remaining

unchanged. Finally, a negative covariance

means that if μˆ (boys) increases, then μˆ (girls) will

decrease, and inversely.

Next, to determine whether the females’ 

performance differs from the males’ performance, for example, as for all statistical

analyses, a null hypothesis has to be tested.

In this particular example, it will consist of

computing the difference between the

males’ performance mean and the females’

performance mean (or the inverse). The null

hypothesis will be:

H 0 : μˆ (boys) – μˆ (girls)= 0.

The variance of the observed difference is

needed to test this null hypothesis. The variance of a difference is equal to the sum of

the variances of the two initial variables

minus two times the covariance between the

two initial variables. A sampling distribution

has the same characteristics as any distribution, except that units consist of sample

estimates and not observations. Therefore,

the sampling variance of a difference is

equal to the sum of the two initial sampling

variances minus two times the covariance

between the two sampling distributions on

the estimates.

2

(μˆ x - μˆ y)= 2(μˆ X ) + 2(μˆ Y ) - 2cov(μˆ X,μˆ Y)

The estimation of the covariance between,

for instance, μˆ (boys) and μˆ (girls) requires the

selection of several samples and then the
International Outcomes of Learning in Mathematics Literacy and Problem Solving

66

In PISA, in each of the three subject matter

areas, a common transformation was estimated from the link items, and this transformation was applied to all participating countries. It follows that any uncertainty that

was introduced through the linking is common to all students and all countries. Thus,

for example, suppose the unknown linking

error (between PISA 2000 and PISA 2003) in

reading literacy resulted in an over-estimation of student scores by two points on the

PISA 2000 scale. It follows that every student’s score will be over-estimated by two

score points. This over-estimation will have

effects on certain, but not all, summary statistics computed from the PISA 2003 data.

For example, consider the following:

• each country’s mean will be over-estimated by an amount equal to the link

error, (in our example this is two score

points); 

• the mean performance of any subgroup

will be over-estimated by an amount

equal to the link error (in our example

this is two score points);

• the standard deviation of student scores

will not be affected because the overestimation of each student by a common

error does not change the standard deviation;

• the difference between the mean scores

of two countries in PISA 2003 will not be

influenced because the over-estimation

of each student by a common error will

have distorted each country’s mean by

the same amount;

• the difference between the mean scores

of two groups (e.g., males and females)

in PISA 2003 will not be influenced,

because the over-estimation of each student by a common error will have distorted each group’s mean by the same

amount;

analysis of the variation of μˆ (boys) in conjunction with μˆ (girls). Such a procedure is, of

course, unrealistic. Therefore, as for any computation of a standard error in PISA, replication methods using the supplied replicate

weights are used to estimate the standard

error on a difference. Use of the replicate

weights implicitly incorporates the covariance between the two estimates into the estimate of the standard error on the difference.

To test such comparisons, the following formula was used to compute the t statistic:

t = estgrp1 – estgrp2 /se(estgrp1 – estgrp2)

Estgrp1 and estgrp2 are the non-independent

groups estimates being compared; 

se (estgrp1 – estgrp2) is the standard error of

the difference calculated using Balanced

Repeated Replication (BRR) to account for

any covariance between the estimates for

the two non-independent groups.

A third type of comparison (addition of a standard error term to the standard t test shown

above for simple comparisons of independent

averages) was also used when analyzing

change in performance over time. The uncertainty that results from the link item sampling

(described in the scaling section above) is

referred to as linking error and this error must

be taken into account when making certain

comparisons between PISA 2000 and PISA

2003 results. Just as with the error that is

introduced through the process of sampling

students, the exact magnitude of this linking

error cannot be determined. We can, however,

estimate the likely range of magnitudes for

this error and take this error into account when

interpreting PISA results. As with sampling

errors, the likely range of magnitude for the

errors is represented as a standard error. The

standard error of linking for reading is 3.74, the

standard error of linking for science is 3.02,

and the standard error for mathematics (space

and shape scale) is 6.01 and mathematics

(change and relationships scale) is 4.84.
67

PISA 2003 Results From the U.S. Perspective

• the difference between the performance

of a group of students (e.g., a country)

between PISA 2000 and PISA 2003 will

be influenced because each student’s

score in PISA 2003 will be influenced by

the error; and

• a change in the difference in performance between two groups from PISA

2000 to PISA 2003 will not be influenced.

This is because neither of the components of this comparison, which are differences in scores in 2000 and 2003

respectively, is influenced by a common

error that is added to all student scores

in PISA 2003.

In general terms, the linking error need only

be considered when comparisons are being

made between PISA 2000 and PISA 2003

results, and then usually only when group

means are being compared. Because the

linking error need only be used in a limited

range of situations we have chosen not to

report the linking error in the tables included

in this report. The general formula is given by: 

t= est1 – est2 / SQRT[(se1)2+ (se2)2+(selinking)2

]

The most obvious example of a situation

where there is a need to use linking error is

in the comparison of the mean performance

for a country between PISA 2000 and PISA

2003. For example, let us consider a comparison between 2000 and 2003 of the performance of Italy in reading. The mean performance of Italy in 2000 was 487 with a standard

error of 2.9, while in 2003 the mean was 476

with a standard error of 3.0. The standardized difference in the Italian mean is 1.97,

which is computed as follows:

and is statistically significant. 

1.97 = (487 - 476) 2.9 + 2 3.0 + 2 3.7 2

In the U.S. report on PISA 2000, a Bonferroni

adjustment was used in all multiple comparisons of countries. This was not the case in

2003, which may result in some differences

in how 2000 results are reported in 2003. This

may also result in some differences between

the PISA 2003 U.S. and OECD reports

(which uses a Bonferroni adjustment for

multiple comparisons of country averages).

The discontinuation of the use of the

Bonferroni adjustment for multiple comparisons was made in order to avoid the possibility that comparisons of achievement

between countries could be interpreted differently depending on the numbers of countries compared.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

68

Appendix B: 

Reference Tables
69

PISA 2003 Results From the U.S. Perspective

Table B-1. Percentage distribution of 15-year-old students, by grade and country: 2003 

Country 7th 8th 9th 10th

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 0.5 0.04 4.7 0.11 35.9 0.25 52.7 0.26

OECD countries

Australia # † ‡ † 8.3 0.40 72.3 0.70

Austria ‡ † 5.1 0.91 43.1 1.47 51.3 1.55

Belgium ‡ † 3.7 0.34 29.5 0.67 65.2 0.64

Canada 0.6 0.15 2.5 0.26 13.7 0.46 82.0 0.58

Czech Republic ‡ † 2.8 0.34 44.7 1.14 52.4 1.14

Denmark ‡ † 9.1 0.58 86.9 0.85 3.8 0.65

Finland ‡ † 12.4 0.51 87.3 0.51 # †

France ‡ † 5.4 0.57 34.9 1.17 57.3 1.11

Germany 1.7 0.27 15.0 0.79 59.8 0.74 23.2 0.57

Greece ‡ † 2.1 0.37 6.6 0.99 76.1 1.41

Hungary 1.1 0.21 5.0 0.47 65.1 0.71 28.8 0.62

Iceland # † # † # † 100.0 †

Ireland # † 2.8 0.32 60.9 1.31 16.7 1.35

Italy ‡ † 1.4 0.36 14.2 0.60 80.0 0.82

Japan # † # † # † 100.0 †

Korea, Republic of # † # † 1.6 0.23 98.3 0.23

Luxembourg # † 14.9 0.22 55.8 0.25 29.3 0.18

Mexico 3.6 0.49 11.0 0.96 40.8 2.38 43.7 2.81

Netherlands ‡ † 4.4 0.56 45.6 1.14 49.3 1.31

New Zealand # † ‡ † 6.8 0.48 89.2 0.51

Norway # † # † ‡ † 98.7 0.25

Poland 0.7 0.16 3.1 0.32 95.7 0.42 ‡ †

Portugal 4.2 0.64 10.6 0.90 20.3 1.62 64.3 2.39

Slovak Republic 0.6 0.22 0.9 0.23 37.1 1.56 60.9 1.54

Spain # † 3.2 0.36 27.0 0.85 69.7 1.01

Sweden # † 2.4 0.21 93.0 0.97 4.6 0.93

Switzerland 0.7 0.16 16.9 1.01 62.7 2.06 19.3 2.50

Turkey ‡ † 4.4 1.60 3.2 0.67 52.1 2.18

United States ‡ † 2.4 0.61 29.7 1.42 60.6 1.35

Non-OECD countries

Hong Kong-China 5.1 0.37 10.7 0.80 25.7 0.95 58.4 0.98

Indonesia 2.4 0.35 12.7 0.80 48.8 1.69 34.5 1.91

Latvia 1.0 0.16 16.1 0.86 73.1 2.08 5.9 0.56

Liechtenstein ‡ † 20.4 1.00 71.3 0.94 ‡ †

Macao-China 12.3 0.50 25.9 0.72 36.8 0.74 24.7 0.60

Russian Federation ‡ † 2.6 0.34 28.7 1.92 67.2 2.18

Serbia and Montenegro # † # † 97.6 0.35 2.4 0.35

Thailand ‡ † 1.1 0.39 44.1 1.20 53.3 1.24

Tunisia 15.4 0.55 22.0 0.70 25.1 0.86 34.5 1.38

Uruguay 5.7 0.62 9.7 0.84 18.2 1.23 59.4 1.74

United Kingdom1 # † # † # † 33.8 1.10

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

70

Table B-1. Percentage distribution of 15-year-old students, by grade and

country: 2003—Continued

Country 11th 12th Grade not reported

Percent s.e. Percent s.e. Percent s.e.

OECD average 6.0 0.13 0.1 0.00 # †

OECD countries

Australia 19.2 0.67 ‡ † # †

Austria # † # † ‡ †

Belgium 0.8 0.09 # † ‡ †

Canada 1.2 0.12 # † # †

Czech Republic # † # † # †

Denmark # † # † # †

Finland # † # † # †

France 2.2 0.30 # † # †

Germany ‡ † # † ‡ †

Greece 15.0 0.90 # † # †

Hungary # † # † # †

Iceland # † # † # †

Ireland 19.6 1.36 # † # †

Italy 4.3 0.48 # † # †

Japan # † # † # †

Korea, Republic of 0.1 † # † # †

Luxembourg ‡ † # † # †

Mexico 0.9 0.49 # † # †

Netherlands ‡ † # † # †

New Zealand 3.7 0.24 # † ‡ †

Norway ‡ † # † # †

Poland # † # † # †

Portugal 0.6 † # † # †

Slovak Republic 0.5 0.22 # † # †

Spain # † # † # †

Sweden # † # † # †

Switzerland 0.2 † # † ‡ †

Turkey 39.2 2.37 ‡ † # †

United States 7.0 0.89 # † # †

Non-OECD countries

Hong Kong-China ‡ † # † # †

Indonesia 1.6 0.21 ‡ † # †

Latvia ‡ † # † 3.7 2.49

Liechtenstein # † # † # †

Macao-China ‡ † # † # †

Russian Federation 1.1 0.18 # † # †

Se
71

PISA 2003 Results From the U.S. Perspective

Table B-2. Percentage distribution and average 

combined mathematics literacy scores 

of U.S. 15-year-old students, by type of

mathematics class: 2003

Type of class Percent s.e. Average s.e

Pre-algebra or 

general mathematics 8.7 0.80 419.5 4.97

Algebra I 28.6 1.01 442.1 3.28

Geometry 31.1 1.18 498.4 3.47

Algebra II 20.7 1.01 537.2 3.67

Precalculus or calculus 3.1 0.39 595.6 7.53

Other 7.7 0.68 482.8 8.57

NOTE: Type of class refers to the mathematics class in which the student

was enrolled at time of the assessment. Detail may not sum to totals

because of rounding. s.e. means standard error.

SOURCE: Organization for Economic Cooperation and Development

(OECD), Program for International Student Assessment (PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

72

Table B-3. Average mathematics literacy scores and subscale scores of 15-year-old students, by country: 2003

Country

Combined 

mathematics

literacy

Mathematics subscales

Space and shape

Change and 

relationships Quantity Uncertainty

Average s.e. Average s.e. Average s.e. Average s.e. Average s.e.

OECD average 500.0 0.63 496.3 0.65 498.8 0.70 500.7 0.63 502.0 0.61

OECD countries

Australia 524.3 2.15 520.6 2.33 525.3 2.30 516.9 2.06 530.9 2.21

Austria 505.6 3.27 515.2 3.48 499.8 3.60 513.2 3.00 493.8 3.13

Belgium 529.3 2.29 529.6 2.26 535.3 2.45 529.6 2.31 525.7 2.21

Canada 532.5 1.82 517.8 1.81 536.7 1.93 528.1 1.85 541.6 1.83

Czech Republic 516.5 3.55 527.4 4.12 514.8 3.50 528.0 3.54 500.3 3.11

Denmark 514.3 2.74 512.4 2.76 509.3 2.99 515.6 2.64 515.6 2.78

Finland 544.3 1.87 539.0 2.04 543.1 2.19 548.5 1.83 544.8 2.09

France 510.8 2.50 507.6 2.98 519.7 2.62 506.9 2.49 506.1 2.39

Germany 503.0 3.32 499.6 3.28 507.2 3.73 513.8 3.37 492.5 3.29

Greece 444.9 3.90 437.1 3.80 435.6 4.31 445.9 3.97 458.4 3.53

Hungary 490.0 2.84 479.1 3.34 494.6 3.10 496.3 2.72 489.0 2.63

Iceland 515.1 1.42 503.5 1.46 509.5 1.43 513.3 1.50 527.8 1.50

Ireland 502.8 2.45 476.2 2.43 506.0 2.45 501.7 2.48 517.2 2.65

Italy 465.6 3.08 470.3 3.14 452.1 3.21 474.8 3.38 462.6 3.03

Japan 534.1 4.02 553.2 4.31 536.1 4.33 526.6 3.79 527.9 3.88

Korea, Republic of 542.2 3.24 551.7 3.80 547.6 3.52 537.2 2.97 538.3 3.03

Luxembourg 493.2 0.97 488.2 1.35 487.0 1.15 501.5 1.06 492.1 1.06

Mexico 385.2 3.64 381.7 3.20 364.1 4.14 393.8 3.94 389.8 3.26

Netherlands 537.8 3.13 526.1 2.87 551.4 3.12 528.3 3.09 549.3 2.99

New Zealand 523.5 2.26 524.9 2.34 525.7 2.37 511.1 2.22 532.2 2.30

Norway 495.2 2.38 482.7 2.54 487.7 2.64 494.2 2.22 512.8 2.59

Poland 490.2 2.50 490.3 2.66 484.3 2.70 491.8 2.47 493.5 2.35

Portugal 466.0 3.40 450.2 3.43 467.9 3.95 465.4 3.51 470.6 3.41

Slovak Republic 498.2 3.35 505.4 4.01 494.4 3.48 512.5 3.43 475.8 3.21

Spain 485.1 2.41 476.5 2.59 480.7 2.80 492.3 2.53 489.0 2.42

Sweden 509.0 2.56 498.3 2.56 505.1 2.94 513.6 2.49 510.8 2.72

Switzerland 526.6 3.38 539.5 3.50 522.7 3.65 532.6 3.08 516.5 3.28

Turkey 423.4 6.74 417.4 6.35 422.8 7.57 413.2 6.78 442.6 6.21

United States 482.9 2.95 472.0 2.78 485.5 3.03 476.4 3.18 491.5 2.97

Non-OECD countries

Hong Kong-China 550.4 4.54 558.4 4.85 539.7 4.68 545.2 4.19 558.3 4.56

Indonesia 360.2 3.91 360.8 3.70 333.9 4.58 357.5 4.28 384.5 2.86

Latvia 483.4 3.69 486.4 4.04 487.2 4.36 481.7 3.60 473.8 3.28

Liechtenstein 535.8 4.12 538.2 4.58 539.5 3.67 533.5 4.12 523.4 3.68

Macao-China 527.3 2.89 527.9 3.29 518.8 3.53 533.0 2.99 531.6 3.21

Russian Federation 468.4 4.20 474.3 4.69 476.8 4.64 472.4 4.04 436.5 4.02

Serbia and Montenegro 436.9 3.75 432.5 3.91 419.0 3.97 456.3 3.79 427.9 3.49

Thailand 417.0 3.00 423.9 3.35 405.0 3.39 414.8 3.15 422.7 2.53

Tunisia 358.7 2.54 358.9 2.56 336.6 2.78 364.4 2.79 363.3 2.30

Uruguay 422.2 3.29 412.0 2.98 417.0 3.60 429.7 3.22 418.6 3.11

United Kingdom1 508.3 2.43 496.0 2.50 512.9 2.54 498.5 2.52 520.1 2.41

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data available. Because PISA is principally an

OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD average.

s.e. means standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2003. 
73

PISA 2003 Results From the U.S. Perspective

Table B-4. Combined mathematics literacy scores of 15-year-old students, by percentiles

and country: 2003

Country 5th percentile 10th percentile 25th percentile 50th percentile

Score s.e. Score s.e. Score s.e. Score s.e.

OECD average 331.7 1.30 369.0 1.11 432.4 0.93 501.1 0.26

OECD countries

Australia 364.3 4.44 398.6 3.43 459.8 2.75 527.5 0.68

Austria 353.4 6.64 384.4 4.44 439.4 4.02 506.2 0.91

Belgium 333.8 6.53 380.7 4.61 456.2 3.44 537.1 1.48

Canada 386.2 3.05 419.3 2.54 473.9 2.19 534.0 0.62

Czech Republic 358.0 6.25 391.7 5.72 449.4 4.55 516.8 1.46

Denmark 360.7 4.39 395.8 4.53 453.2 3.67 516.1 1.54

Finland 406.4 3.83 438.0 2.77 488.2 2.21 543.7 0.88

France 352.3 5.96 388.7 5.56 449.1 3.74 514.2 0.86

Germany 324.0 6.08 363.0 5.65 432.2 4.66 509.2 0.97

Greece 287.6 5.39 323.5 5.13 382.4 4.57 446.1 21.6

Hungary 335.3 5.62 369.6 4.23 426.1 3.01 490.0 0.94

Iceland 362.4 4.05 396.1 2.74 454.2 2.81 518.2 1.30

Ireland 360.4 4.68 393.1 3.21 445.0 3.38 503.5 0.72

Italy 307.2 6.39 342.4 5.86 400.5 4.34 466.1 1.11

Japan 360.9 8.24 401.7 6.26 467.2 5.37 539.5 1.66

Korea, Republic of 387.8 4.61 422.8 4.46 479.3 3.74 544.1 1.69

Luxembourg 338.5 3.87 372.7 2.69 430.2 2.15 495.2 0.52

Mexico 247.1 5.39 276.1 4.70 326.6 4.32 384.7 1.14

Netherlands 385.2 6.86 415.4 5.84 470.9 5.44 539.6 1.51

New Zealand 358.5 4.07 394.3 3.89 455.2 2.91 525.3 1.38

Norway 343.5 3.96 375.9 3.42 432.9 2.87 495.3 1.30

Poland 343.4 5.78 376.0 3.62 428.2 3.13 489.7 1.18

Portugal 320.9 6.26 351.9 5.25 406.0 4.96 467.2 2.40

Slovak Republic 342.4 6.91 378.5 5.81 435.6 4.57 498.3 1.28

Spain 335.0 5.13 368.6 3.54 426.2 2.98 487.1 1.12

Sweden 352.7 5.29 387.1 4.38 446.1 3.02 509.6 1.69

Switzerland 358.7 4.80 395.7 4.16 460.8 3.57 529.7 0.97

Turkey 269.7 5.76 300.2 5.01 350.8 5.26 414.5 1.75

United States 323.0 4.88 356.5 4.55 418.0 3.69 483.5 1.36

Non-OECD countries

Hong Kong-China 373.8 11.05 417.0 8.02 484.8 6.91 559.5 1.49

Indonesia 233.2 5.22 260.5 4.81 306.0 3.49 356.9 2.06

Latvia 339.2 5.90 370.8 5.14 423.5 3.90 484.1 1.19

Liechtenstein 361.9 19.68 408.0 9.77 469.9 7.58 539.2 6.71

Macao-China 382.3 8.76 414.5 5.97 467.2 4.41 529.0 1.66

Russian Federation 318.5 5.46 350.8 4.96 405.8 4.83 467.1 1.29

Serbia and Montenegro 298.9 4.37 328.6 4.47 378.6 3.96 436.6 2.56

Thailand 289.9 3.95 316.3 3.10 360.5 2.92 411.9 1.96

Tunisia 228.6 3.80 256.3 3.51 303.0 2.55 355.5 0.77

Uruguay 255.3 4.30 291.3 3.80 353.3 4.07 425.0 1.71

United Kingdom1 356.1 4.92 387.8 4.05 444.1 3.18 509.8 0.99

See notes at end of table. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

74

Table B-4. Combined mathematics literacy scores of 15-year-old students,

by percentiles and country: 2003—Continued

Country 75th percentile 90th percentile 95th percentile

Score s.e. Score s.e. Score s.e.

OECD average 570.5 0.70 628.3 0.74 660.2 0.95

OECD countries

Australia 591.6 2.50 644.7 3.04 675.7 3.53

Austria 571.4 4.18 626.2 3.96 658.2 4.96

Belgium 611.2 2.49 664.4 2.35 693.4 2.38

Canada 593.3 2.13 644.2 2.58 672.7 3.39

Czech Republic 584.4 3.98 641.0 4.35 671.9 4.89

Denmark 578.2 3.14 631.5 3.65 662.0 4.73

Finland 602.6 2.32 651.7 2.83 680.2 3.13

France 575.2 3.01 627.7 3.58 656.2 3.46

Germany 578.3 3.48 632.3 3.50 661.7 3.64

Greece 507.9 4.28 565.9 5.25 597.8 5.10

Hungary 555.9 3.90 610.7 4.71 643.8 4.59

Iceland 578.4 1.95 629.2 3.02 657.9 3.77

Ireland 561.9 3.01 613.9 3.59 641.0 3.30

Italy 530.2 3.01 589.1 3.63 623.2 3.74

Japan 605.1 4.36 659.6 6.14 690.2 6.58

Korea, Republic of 606.1 4.22 659.2 5.37 690.2 6.83

Luxembourg 557.2 1.91 611.4 3.20 641.4 2.72

Mexico 443.6 4.46 497.1 4.69 526.9 5.65

Netherlands 608.3 3.84 656.5 3.21 683.5 3.43

New Zealand 593.0 2.21 650.0 3.20 682.3 2.91

Norway 560.0 3.32 613.6 3.56 644.7 3.92

Poland 552.8 2.87 607.4 3.34 639.9 3.50

Portugal 526.1 3.52 579.9 3.29 609.9 3.72

Slovak Republic 564.6 3.78 619.1 3.49 648.4 4.07

Spain 546.4 3.12 597.4 3.50 626.0 3.70

Sweden 575.6 3.19 630.5 3.82 661.9 4.80

Switzerland 595.0 4.89 652.1 5.23 684.0 6.84

Turkey 484.9 8.53 559.7 14.23 613.6 22.75

United States 549.7 3.36 607.4 3.87 638.0 5.14

Non-OECD countries

Hong Kong-China 621.8 3.74 671.8 4.10 699.5 3.97

Indonesia 411.5 4.77 465.8 6.50 498.8 7.69

Latvia 543.5 4.72 596.4 4.43 626.3 4.97

Liechtenstein 608.6 7.91 655.3 9.53 686.4 16.38

Macao-China 587.3 4.01 639.1 5.48 668.4 8.28

Russian Federation 530.1 4.95 588.1 5.28 622.4 6.10

Serbia and Montenegro 493.1 4.78 546.4 5.05 579.2 5.29

Thailand 469.3 3.75 526.0 4.70 560.0 6.43

Tunisia 411.6 3.59 465.8 4.78 501.4 6.80

Uruguay 490.7 3.77 550.0 4.36 583.4 4.67

United Kingdom1 572.6 3.18 628.7 3.55 659.3 4.79

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with

data available. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD average. s.e.

means standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
75

PISA 2003 Results From the U.S. Perspective

Table B-5. Standard deviations of 

15-year-old students' combined

mathematics literacy scores,

by country: 2003

Country

Standard

deviation s.e.

OECD average 100.0 0.41

OECD countries

Australia 95.4 1.50

Austria 93.1 1.67

Belgium 109.9 1.78

Canada 87.1 0.97

Czech Republic 95.9 1.87

Denmark 91.3 1.44

Finland 83.7 1.08

France 91.7 1.80

Germany 102.6 1.77

Greece 93.8 1.76

Hungary 93.5 1.96

Iceland 90.4 1.21

Ireland 85.3 1.26

Italy 95.7 1.87

Japan 100.5 2.75

Korea, Republic of 92.4 2.14

Luxembourg 91.9 0.95

Mexico 85.4 1.85

Netherlands 92.5 2.33

New Zealand 98.3 1.17

Norway 92.0 1.15

Poland 90.2 1.34

Portugal 87.6 1.66

Slovak Republic 93.3 2.32

Spain 88.5 1.26

Sweden 94.7 1.79

Switzerland 98.4 2.05

Turkey 104.7 5.34

United States 95.2 1.29

Non-OECD countries

Hong Kong-China 100.2 3.01

Indonesia 80.5 2.06

Latvia 87.9 1.66

Liechtenstein 99.1 4.43

Macao-China 86.9 2.41

Russian Federation 92.3 1.93

Serbia and Montenegro 84.7 1.55

Thailand 82.0 1.79

Tunisia 82.0 1.95

Uruguay 99.7 1.60

United Kingdom1 92.3 1.35

1

Due to low response rates, data for the United Kingdom are

not discussed in this report.

NOTE: The OECD average is the average of the national

averages of the OECD member countries with data available.

Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of

the OECD countries and are not included in the OECD average. s.e. means standard error.

SOURCE: Organization for Economic Cooperation and

Development (OECD), Program for International Student

Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

76

Table B-6. Percentage distribution of 15-year-old students scoring at each proficiency level on

the combined mathematics literacy scale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 8.2 0.17 13.2 0.16 21.1 0.15 23.7 0.18

OECD countries

Australia 4.3 0.45 10.0 0.51 18.6 0.62 24.0 0.71

Austria 5.6 0.70 13.2 0.84 21.6 0.90 24.9 1.14

Belgium 7.2 0.56 9.3 0.49 15.9 0.65 20.1 0.71

Canada 2.4 0.26 7.7 0.36 18.3 0.61 26.2 0.67

Czech Republic 5.0 0.69 11.6 0.90 20.1 0.96 24.3 0.95

Denmark 4.7 0.50 10.7 0.62 20.6 0.89 26.2 0.88

Finland 1.5 0.23 5.3 0.38 16.0 0.57 27.7 0.65

France 5.6 0.68 11.0 0.77 20.2 0.82 25.9 0.99

Germany 9.2 0.84 12.4 0.81 19.0 1.05 22.6 0.82

Greece 17.8 1.21 21.2 1.15 26.3 1.04 20.2 1.01

Hungary 7.8 0.80 15.2 0.81 23.8 1.05 24.3 0.93

Iceland 4.5 0.40 10.5 0.55 20.2 1.02 26.1 0.88

Ireland 4.7 0.57 12.1 0.84 23.6 0.83 28.0 0.82

Italy 13.2 1.19 18.7 0.93 24.7 1.03 22.9 0.84

Japan 4.7 0.65 8.6 0.72 16.3 0.80 22.4 1.02

Korea, Republic of 2.5 0.32 7.1 0.65 16.6 0.80 24.1 0.98

Luxembourg 7.4 0.41 14.3 0.65 22.9 0.87 25.9 0.79

Mexico 38.1 1.71 27.9 1.02 20.8 0.87 10.1 0.84

Netherlands 2.6 0.65 8.4 0.95 18.0 1.11 23.0 1.14

New Zealand 4.9 0.44 10.1 0.63 19.2 0.71 23.2 0.90

Norway 6.9 0.50 13.9 0.82 23.7 1.16 25.2 1.01

Poland 6.8 0.61 15.2 0.76 24.8 0.75 25.3 0.94

Portugal 11.3 1.11 18.8 0.99 27.1 0.99 24.0 1.03

Slovak Republic 6.7 0.85 13.2 0.86 23.5 0.88 24.9 1.08

Spain 8.1 0.66 14.9 0.87 24.7 0.78 26.7 1.02

Sweden 5.6 0.52 11.7 0.60 21.7 0.84 25.5 0.95

Switzerland 4.9 0.45 9.6 0.57 17.5 0.80 24.3 0.98

Turkey 27.7 2.01 24.6 1.33 22.1 1.12 13.5 1.27

United States 10.2 0.80 15.5 0.81 23.9 0.80 23.8 0.79

Non-OECD countries

Hong Kong-China 3.9 0.72 6.5 0.64 13.9 1.00 20.0 1.25

Indonesia 50.5 2.08 27.6 1.05 14.8 1.07 5.5 0.71

Latvia 7.6 0.86 16.1 1.08 25.5 1.17 26.3 1.15

Liechtenstein 4.8 1.33 7.5 1.66 17.3 2.78 21.6 2.54

Macao-China 2.3 0.60 8.8 1.34 19.6 1.40 26.8 1.77

Russian Federation 11.4 1.03 18.8 1.09 26.4 1.13 23.1 1.02

Serbia and Montenegro 17.6 1.35 24.5 1.08 28.6 1.16 18.9 1.11

Thailand 23.8 1.28 30.2 1.25 25.4 1.12 13.7 0.85

Tunisia 51.1 1.37 26.9 0.95 14.7 0.75 5.7 0.61

Uruguay 26.3 1.30 21.8 0.80 24.2 0.89 16.8 0.68

United Kingdom1 5.2 0.54 12.5 0.67 21.2 1.20 25.6 0.88

See notes at end of table.
77

PISA 2003 Results From the U.S. Perspective

Table B-6. Percentage distribution of 15-year-old students scoring at each

proficiency level on the combined mathematics literacy scale, by

country: 2003—Continued

Country Level 4 Level 5 Level 6

Percent s.e. Percent s.e. Percent s.e.

OECD average 19.1 0.17 10.6 0.13 4.0 0.10

OECD countries

Australia 23.3 0.64 14.0 0.53 5.8 0.45

Austria 20.5 0.84 10.5 0.85 3.7 0.52

Belgium 21.0 0.62 17.5 0.69 9.0 0.48

Canada 25.1 0.60 14.8 0.55 5.5 0.45

Czech Republic 20.8 0.87 12.9 0.80 5.3 0.53

Denmark 21.9 0.83 11.8 0.86 4.1 0.50

Finland 26.1 0.89 16.7 0.64 6.7 0.46

France 22.1 0.97 11.6 0.72 3.5 0.40

Germany 20.6 1.02 12.2 0.87 4.1 0.48

Greece 10.6 0.87 3.4 0.53 0.6 0.17

Hungary 18.2 0.90 8.2 0.73 2.5 0.42

Iceland 23.2 0.81 11.7 0.61 3.7 0.36

Ireland 20.2 1.06 9.1 0.76 2.2 0.33

Italy 13.4 0.73 5.5 0.43 1.5 0.19

Japan 23.6 1.24 16.1 0.96 8.2 1.14

Korea, Republic of 25.0 1.08 16.7 0.81 8.1 0.93

Luxembourg 18.7 0.85 8.5 0.59 2.4 0.31

Mexico 2.7 0.39 0.4 0.10 # †

Netherlands 22.6 1.34 18.2 1.09 7.3 0.58

New Zealand 21.9 0.80 14.1 0.60 6.6 0.44

Norway 18.9 1.00 8.7 0.57 2.7 0.35

Poland 17.7 0.89 7.8 0.49 2.3 0.31

Portugal 13.4 0.94 4.6 0.47 0.8 0.16

Slovak Republic 18.9 0.82 9.8 0.68 2.9 0.38

Spain 17.7 0.65 6.5 0.62 1.4 0.25

Sweden 19.8 0.81 11.6 0.57 4.1 0.49

Switzerland 22.5 0.72 14.2 1.05 7.0 0.90

Turkey 6.8 1.05 3.1 0.82 2.4 1.02

United States 16.6 0.73 8.0 0.53 2.0 0.36

Non-OECD countries

Hong Kong-China 25.0 1.17 20.2 1.00 10.5 0.94

Indonesia 1.4 0.39 0.2 0.09 # †

Latvia 16.6 1.17 6.3 0.70 1.6 0.36

Liechtenstein 23.2 3.09 18.3 3.22 7.3 1.73

Macao-China 23.7 1.71 13.8 1.55 4.8 0.96

Russian Federation 13.2 0.92 5.4 0.58 1.6 0.38

Serbia and Montenegro 8.1 0.88 2.1 0.41 0.2 0.10

Thailand 5.3 0.53 1.5 0.31 0.2 0.10

Tunisia 1.4 0.30 0.2 0.12 # †

Uruguay 8.2 0.65 2.3 0.33 0.5 0.17

United Kingdom1 20.6 0.73 11.0 0.73 3.9 0.43

† Not applicable.

# Rounds to zero.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a

majority of items at that level. Students were classified into mathematics literacy levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a

score greater than 357.77 and less than or equal to 420.07); level 2 (a score greater than 420.07 and less than or

equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68); level 4 (a score greater

than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the

OECD member countries with data available. Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

78

Table B-7. Percentage distribution of 15-year-old students scoring at each proficiency level on

the mathematics literacy quantity subscale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 8.8 0.18 12.5 0.15 20.1 0.15 23.7 0.21

OECD countries

Australia 5.5 0.44 11.0 0.47 19.0 0.76 24.3 0.90

Austria 3.7 0.49 11.2 0.88 20.9 1.00 27.2 1.06

Belgium 7.2 0.57 8.9 0.53 15.1 0.52 20.6 0.62

Canada 3.8 0.29 8.8 0.37 18.1 0.64 25.2 0.64

Czech Republic 4.7 0.72 9.7 0.86 17.2 0.89 23.5 0.95

Denmark 4.7 0.59 10.4 0.58 19.9 0.79 26.3 0.90

Finland 1.4 0.21 5.0 0.53 14.6 0.66 26.9 0.69

France 6.7 0.74 11.1 0.82 20.4 0.98 25.4 1.15

Germany 8.5 0.70 10.4 0.83 17.5 0.90 22.0 1.08

Greece 19.0 1.21 19.8 0.88 25.1 0.89 20.0 0.92

Hungary 7.8 0.73 13.5 0.84 21.6 0.90 25.2 0.89

Iceland 6.2 0.43 10.9 0.63 19.1 1.09 24.3 1.00

Ireland 5.6 0.57 12.3 0.85 23.0 1.00 26.9 1.06

Italy 13.7 1.06 16.1 0.74 22.0 0.76 22.4 0.82

Japan 5.7 0.73 9.2 0.78 16.6 0.81 23.1 1.12

Korea, Republic of 2.6 0.31 7.2 0.70 17.0 0.84 25.2 0.84

Luxembourg 6.5 0.44 12.4 0.78 21.8 0.99 26.2 1.35

Mexico 35.5 1.80 25.0 1.23 21.4 1.12 12.4 0.77

Netherlands 4.1 0.71 10.1 0.96 18.3 1.22 23.0 1.22

New Zealand 6.4 0.55 11.9 0.70 20.1 0.71 23.6 0.78

Norway 7.7 0.54 13.8 0.74 22.8 0.93 25.4 1.06

Poland 7.1 0.68 13.5 0.70 24.2 0.96 27.1 0.87

Portugal 12.9 1.22 18.3 1.06 25.2 0.80 23.4 1.22

Slovak Republic 5.6 0.75 10.6 0.77 20.0 0.80 26.1 0.89

Spain 8.9 0.66 13.2 0.88 22.5 0.78 25.0 0.66

Sweden 4.4 0.45 10.3 0.60 21.4 0.81 27.3 1.00

Switzerland 4.2 0.38 8.6 0.60 16.0 0.81 24.2 1.03

Turkey 32.1 2.07 23.1 1.03 20.2 1.07 12.6 1.06

United States 13.7 1.00 15.6 0.76 22.0 0.70 21.9 0.78

Non-OECD countries

Hong Kong-China 4.1 0.70 7.0 0.67 13.7 1.24 21.5 1.33

Indonesia 51.5 1.92 24.7 0.93 14.9 1.03 6.1 0.64

Latvia 7.4 0.87 15.5 1.20 26.4 1.13 27.7 1.21

Liechtenstein 4.0 1.37 7.6 1.36 16.5 2.88 24.1 2.90

Macao-China 2.4 0.62 8.1 1.29 17.8 1.37 25.8 1.71

Russian Federation 11.1 1.05 16.8 1.01 25.8 0.87 24.6 1.03

Serbia and Montenegro 13.6 1.11 20.6 1.15 27.1 1.15 22.1 1.13

Thailand 27.7 1.39 26.4 1.21 23.3 0.91 13.7 0.84

Tunisia 49.0 1.31 25.2 0.99 16.1 0.91 7.0 0.59

Uruguay 25.6 1.11 19.5 0.82 22.1 0.81 18.1 1.17

United Kingdom1 8.3 0.64 13.7 0.67 20.7 0.96 24.2 0.68

See notes at end of table.
79

PISA 2003 Results From the U.S. Perspective

Table B-7. Percentage distribution of 15-year-old students scoring at each 

proficiency level on the mathematics literacy quantity subscale,

by country: 2003—Continued

Country Level 4 Level 5 Level 6

Percent s.e. Percent s.e. Percent s.e.

OECD average 19.9 0.17 11.0 0.11 4.0 0.09

OECD countries

Australia 22.4 0.64 12.5 0.57 5.2 0.37

Austria 23.1 0.99 11.2 0.81 2.8 0.45

Belgium 22.3 0.59 17.5 0.61 8.5 0.45

Canada 23.7 0.52 14.4 0.52 6.0 0.35

Czech Republic 23.1 0.89 15.0 0.74 6.7 0.62

Denmark 22.7 0.89 12.0 0.71 4.0 0.42

Finland 27.3 0.89 17.9 0.65 7.0 0.44

France 21.9 0.84 11.0 0.71 3.5 0.34

Germany 22.0 1.20 14.1 0.98 5.5 0.43

Greece 11.0 0.80 4.1 0.56 1.0 0.29

Hungary 19.7 0.80 9.7 0.67 2.5 0.31

Iceland 22.5 0.83 12.7 0.68 4.2 0.52

Ireland 20.6 0.84 9.5 0.62 2.2 0.36

Italy 15.2 0.82 7.7 0.49 2.8 0.27

Japan 23.6 1.03 15.1 0.84 6.7 0.85

Korea, Republic of 26.0 0.98 15.6 0.86 6.4 0.79

Luxembourg 21.0 0.82 9.4 0.59 2.7 0.29

Mexico 4.6 0.48 1.0 0.21 0.1 0.06

Netherlands 21.9 1.11 15.9 1.03 6.7 0.58

New Zealand 21.2 0.76 11.9 0.61 5.0 0.33

Norway 18.8 0.91 8.9 0.58 2.6 0.31

Poland 18.7 0.82 7.6 0.62 1.8 0.28

Portugal 13.8 0.78 5.2 0.42 1.2 0.22

Slovak Republic 21.9 0.83 12.3 0.76 3.6 0.44

Spain 18.8 0.77 8.8 0.60 2.6 0.25

Sweden 21.6 0.87 11.1 0.77 3.9 0.56

Switzerland 24.6 0.80 15.7 0.95 6.7 0.88

Turkey 6.5 1.01 3.2 0.71 2.3 0.92

United States 16.0 0.67 8.1 0.67 2.8 0.36

Non-OECD countries

Hong Kong-China 25.8 1.21 18.7 0.95 9.2 0.71

Indonesia 2.1 0.49 0.6 0.19 0.1 0.05

Latvia 16.3 1.12 5.5 0.59 1.2 0.27

Liechtenstein 24.8 2.63 17.1 2.44 6.0 1.49

Macao-China 25.3 1.76 15.6 1.50 5.1 1.06

Russian Federation 14.8 0.99 5.6 0.64 1.4 0.27

Serbia and Montenegro 12.3 0.99 3.7 0.60 0.7 0.18

Thailand 6.3 0.56 2.0 0.38 0.6 0.20

Tunisia 2.2 0.36 0.4 0.17 0.1 0.05

Uruguay 10.0 0.71 3.7 0.40 0.9 0.16

United Kingdom1 19.2 0.73 10.1 0.80 3.8 0.47

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a

majority of items at that level. Students were classified into mathematics literacy levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a

score greater than 357.77 and less than or equal to 420.07); level 2 (a score greater than 420.07 and less than or

equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68); level 4 (a score greater

than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the

OECD member countries with data available. Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

80

Table B-8. Percentage distribution of 15-year-old students scoring at each proficiency level on

the mathematics literacy space and shape subscale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 10.6 0.19 14.2 0.16 20.4 0.14 21.5 0.16

OECD countries

Australia 6.1 0.50 10.8 0.59 18.4 0.52 23.0 0.66

Austria 8.0 0.74 12.0 0.79 18.6 0.82 21.4 0.74

Belgium 6.6 0.53 10.4 0.48 16.7 0.51 20.3 0.70

Canada 4.7 0.36 10.7 0.57 20.4 0.61 25.0 0.51

Czech Republic 8.1 0.88 10.6 0.71 17.0 0.72 19.3 0.71

Denmark 7.1 0.65 11.2 0.74 19.5 0.74 23.8 0.80

Finland 2.5 0.31 7.3 0.47 17.0 0.67 25.5 0.83

France 7.7 0.77 12.0 0.66 19.6 0.85 23.4 1.07

Germany 11.1 0.77 13.3 0.95 18.6 0.86 21.2 0.94

Greece 21.3 1.17 21.7 1.05 24.4 1.02 18.7 0.95

Hungary 13.1 0.99 17.3 0.80 21.8 0.76 20.5 0.72

Iceland 6.5 0.61 12.1 0.66 21.6 0.76 26.0 1.09

Ireland 10.7 0.78 16.9 1.15 25.4 0.87 23.0 1.02

Italy 15.1 0.99 16.8 0.85 22.0 0.66 21.1 0.70

Japan 4.2 0.65 7.4 0.75 13.9 0.70 20.0 0.84

Korea, Republic of 4.8 0.54 8.4 0.57 14.7 0.90 19.7 0.92

Luxembourg 9.5 0.48 15.6 0.62 23.0 0.85 22.6 1.12

Mexico 39.1 1.59 27.8 0.77 20.6 0.95 9.4 0.71

Netherlands 3.7 0.65 10.1 0.78 18.6 1.07 24.9 1.16

New Zealand 5.8 0.50 10.8 0.66 18.1 0.84 21.8 0.77

Norway 11.5 0.65 16.1 0.63 22.2 0.92 22.3 0.79

Poland 10.7 0.81 14.9 0.70 22.0 0.93 22.1 0.88

Portugal 16.4 1.37 21.5 0.78 26.0 0.96 20.2 0.97

Slovak Republic 10.2 0.92 13.4 0.80 19.0 0.83 20.2 0.80

Spain 10.1 0.76 16.7 0.84 25.5 0.77 24.7 0.83

Sweden 7.9 0.63 13.4 0.62 22.1 0.84 24.2 1.02

Switzerland 5.4 0.52 8.6 0.50 15.7 0.85 21.4 0.94

Turkey 28.6 1.87 26.0 1.15 22.3 1.18 12.7 1.11

United States 12.1 0.84 18.2 1.05 24.7 1.09 22.0 0.95

Non-OECD countries

Hong Kong-China 4.1 0.65 7.0 0.91 13.2 1.20 18.7 0.94

Indonesia 49.7 1.74 25.9 1.22 15.5 1.00 6.6 0.67

Latvia 10.7 0.93 15.1 0.99 22.4 0.92 23.3 1.14

Liechtenstein 5.7 1.37 8.1 1.69 14.9 2.79 21.5 3.52

Macao-China 4.0 0.71 9.8 1.48 17.6 2.03 24.5 2.01

Russian Federation 14.9 1.03 16.5 0.79 21.9 0.88 20.4 0.84

Serbia and Montenegro 21.8 1.33 24.4 1.04 24.5 0.77 16.9 0.97

Thailand 23.4 1.20 26.8 0.95 24.7 1.13 15.4 0.91

Tunisia 49.7 1.28 26.0 1.09 15.5 0.66 6.3 0.51

Uruguay 29.3 1.15 23.3 0.86 22.9 0.94 15.2 0.79

United Kingdom1 8.6 0.64 14.1 0.98 21.4 0.78 24.3 0.82

See notes at end of table.
81

PISA 2003 Results From the U.S. Perspective

Table B-8. Percentage distribution of 15-year-old students scoring at each

proficiency level on the mathematics literacy space and shape

subscale, by country: 2003—Continued

Country Level 4 Level 5 Level 6

Percent s.e. Percent s.e. Percent s.e.

OECD average 17.2 0.15 10.4 0.11 5.8 0.10

OECD countries

Australia 21.2 0.75 13.2 0.56 7.3 0.55

Austria 19.1 0.88 12.3 0.87 8.5 0.74

Belgium 20.0 0.90 15.7 0.76 10.2 0.47

Canada 21.4 0.49 12.1 0.46 5.6 0.40

Czech Republic 18.9 0.76 14.4 0.80 11.7 0.79

Denmark 20.0 0.74 12.5 0.69 5.9 0.49

Finland 24.6 0.80 15.2 0.59 7.9 0.60

France 20.0 0.84 12.0 0.79 5.1 0.51

Germany 18.4 0.82 11.4 0.71 6.0 0.44

Greece 9.6 0.75 3.6 0.50 0.8 0.26

Hungary 14.8 0.89 8.0 0.74 4.5 0.56

Iceland 20.5 0.76 10.0 0.63 3.3 0.37

Ireland 15.4 0.77 6.8 0.64 1.8 0.24

Italy 14.5 0.61 7.2 0.46 3.3 0.28

Japan 21.9 1.00 18.2 0.94 14.3 1.20

Korea, Republic of 19.9 1.00 16.5 0.82 16.0 1.27

Luxembourg 17.1 0.73 8.5 0.76 3.6 0.37

Mexico 2.5 0.37 0.5 0.13 # †

Netherlands 21.9 1.10 14.6 0.84 6.2 0.56

New Zealand 20.7 0.90 14.4 0.74 8.5 0.49

Norway 16.4 0.73 8.2 0.51 3.3 0.33

Poland 16.4 0.69 8.8 0.53 5.0 0.52

Portugal 10.9 0.75 4.1 0.44 0.9 0.20

Slovak Republic 17.4 0.84 11.6 0.65 8.2 0.66

Spain 15.3 0.75 6.0 0.48 1.6 0.27

Sweden 18.2 0.79 10.0 0.57 4.2 0.41

Switzerland 21.4 0.89 15.9 0.73 11.7 1.09

Turkey 5.8 1.01 2.5 0.68 2.1 0.94

United States 14.2 0.71 6.5 0.51 2.3 0.33

Non-OECD countries

Hong Kong-China 21.5 1.15 19.9 0.95 15.6 1.05

Indonesia 1.8 0.38 0.4 0.14 0.1 0.05

Latvia 16.8 0.93 8.2 0.71 3.5 0.51

Liechtenstein 23.2 4.21 16.5 2.59 10.1 1.78

Macao-China 23.2 1.75 13.7 1.27 7.2 0.92

Russian Federation 14.2 0.86 7.7 0.74 4.3 0.63

Serbia and Montenegro 8.6 0.92 2.8 0.46 0.9 0.22

Thailand 7.0 0.62 2.2 0.37 0.5 0.18

Tunisia 2.1 0.37 0.5 0.14 # †

Uruguay 6.7 0.50 2.2 0.41 0.4 0.14

United Kingdom1 17.9 0.61 9.7 0.65 3.9 0.40

† Not applicable.

# Rounds to zero.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a

majority of items at that level. Students were classified into mathematics literacy levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a

score greater than 357.77 and less than or equal to 420.07); level 2 (a score greater than 420.07 and less than or

equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68); level 4 (a score greater

than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the

OECD member countries with data available. Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

82

Table B-9. Percentage distribution of 15-year-old students scoring at each proficiency level on

the mathematics literacy change and relationships subscale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 10.2 0.19 13.0 0.15 19.8 0.14 22.0 0.19

OECD countries

Australia 4.8 0.44 9.5 0.53 18.5 0.59 23.8 0.75

Austria 8.6 0.84 14.1 0.95 20.5 0.94 22.5 1.10

Belgium 7.6 0.60 9.7 0.59 14.8 0.63 18.2 0.74

Canada 2.9 0.25 7.6 0.36 17.2 0.57 24.9 0.54

Czech Republic 5.7 0.74 11.8 0.96 20.8 0.88 23.5 0.82

Denmark 6.3 0.61 11.9 0.82 20.4 1.06 24.5 0.85

Finland 2.7 0.32 7.0 0.56 16.1 0.71 24.5 0.94

France 6.4 0.76 9.5 0.71 18.2 0.68 23.9 0.91

Germany 9.5 0.87 12.6 0.74 18.5 0.90 20.6 0.76

Greece 23.3 1.36 19.9 0.94 22.9 0.83 18.0 0.89

Hungary 8.4 0.81 14.5 0.72 22.0 1.16 23.5 0.98

Iceland 6.3 0.44 12.0 0.60 20.2 0.77 24.4 0.79

Ireland 5.1 0.51 11.2 0.86 22.6 0.84 27.0 1.07

Italy 18.2 1.25 19.2 0.84 23.7 0.78 20.4 0.86

Japan 6.4 0.73 8.5 0.72 15.7 0.78 20.6 0.85

Korea, Republic of 3.0 0.40 7.0 0.74 15.7 0.97 22.3 0.93

Luxembourg 10.7 0.60 15.3 0.91 21.5 1.07 22.5 0.88

Mexico 47.2 1.74 24.1 0.85 17.0 0.88 8.6 0.81

Netherlands 1.4 0.37 7.2 0.82 16.4 1.22 22.7 1.13

New Zealand 5.6 0.60 10.2 0.87 17.5 0.69 22.5 0.98

Norway 9.5 0.70 15.1 0.70 22.8 1.05 23.9 0.78

Poland 10.1 0.78 16.1 0.71 23.6 0.81 23.0 0.91

Portugal 13.6 1.28 17.5 1.04 23.8 0.93 22.5 1.05

Slovak Republic 9.7 0.91 14.3 0.86 21.0 0.92 22.4 0.92

Spain 11.3 0.67 14.9 0.96 22.9 0.71 24.0 0.93

Sweden 9.4 0.64 12.6 0.64 19.6 0.87 21.7 0.87

Switzerland 7.6 0.57 10.1 0.58 17.3 1.05 21.3 0.99

Turkey 30.0 2.04 21.1 1.06 20.1 1.16 13.9 1.23

United States 10.4 0.84 14.4 0.65 22.6 0.78 24.3 0.71

Non-OECD countries

Hong Kong-China 5.6 0.85 8.0 0.78 14.5 1.08 20.6 1.01

Indonesia 59.6 1.78 20.2 0.82 12.3 0.81 5.4 0.63

Latvia 10.6 0.98 14.7 1.08 22.2 1.29 23.5 1.24

Liechtenstein 4.6 1.13 10.0 1.89 15.1 2.43 20.7 3.01

Macao-China 5.2 1.13 12.2 1.32 18.2 1.52 23.4 1.84

Russian Federation 11.8 1.14 16.2 0.86 23.7 1.02 23.5 0.89

Serbia and Montenegro 26.5 1.60 24.1 1.13 23.5 0.93 15.7 0.92

Thailand 31.9 1.61 26.4 1.29 22.0 0.90 12.1 0.84

Tunisia 58.8 1.19 20.4 0.70 12.9 0.67 5.8 0.44

Uruguay 29.8 1.31 19.1 0.79 21.6 1.10 16.5 1.03

United Kingdom1 5.7 0.57 11.2 0.85 20.3 0.89 24.6 0.85

See notes at end of table.
83

PISA 2003 Results From the U.S. Perspective

Table B-9. Percentage distribution of 15-year-old students scoring at

each proficiency level on the mathematics literacy change

and relationships subscale, by country: 2003—Continued

Country Level 4 Level 5 Level 6

Percent s.e. Percent s.e. Percent s.e.

OECD average 18.5 0.20 11.1 0.13 5.3 0.11

OECD countries

Australia 22.9 0.65 14.0 0.65 6.5 0.57

Austria 18.8 1.03 10.9 0.81 4.6 0.49

Belgium 19.7 0.71 17.5 0.86 12.4 0.52

Canada 24.4 0.56 15.6 0.58 7.3 0.44

Czech Republic 19.4 0.83 12.5 0.71 6.4 0.61

Denmark 20.7 0.84 11.4 0.78 4.6 0.48

Finland 24.1 0.80 16.7 0.66 8.9 0.53

France 22.2 0.81 14.2 0.69 5.6 0.47

Germany 19.6 0.86 13.2 0.76 6.1 0.52

Greece 10.8 0.95 4.0 0.46 1.1 0.19

Hungary 18.4 0.83 9.6 0.74 3.6 0.42

Iceland 21.0 0.80 11.9 0.67 4.2 0.41

Ireland 21.6 0.85 10.2 0.63 2.3 0.35

Italy 11.8 0.84 5.2 0.42 1.5 0.19

Japan 21.1 1.10 16.4 0.79 11.3 1.18

Korea, Republic of 23.6 0.99 17.5 0.91 10.9 1.07

Luxembourg 18.1 1.00 8.5 0.61 3.4 0.36

Mexico 2.6 0.39 0.4 0.13 0.1 0.03

Netherlands 21.8 1.14 19.2 0.89 11.3 0.71

New Zealand 22.2 0.79 14.0 0.68 7.9 0.46

Norway 17.4 0.89 8.3 0.61 2.9 0.39

Poland 16.1 0.84 7.9 0.59 3.3 0.33

Portugal 15.1 0.91 5.8 0.47 1.7 0.25

Slovak Republic 18.1 0.96 10.1 0.74 4.4 0.46

Spain 17.1 0.62 7.7 0.54 2.0 0.24

Sweden 18.3 0.84 11.6 0.53 6.7 0.63

Switzerland 20.9 0.76 13.9 0.82 8.8 0.91

Turkey 7.9 1.15 3.8 0.80 3.2 1.17

United States 17.7 0.82 8.4 0.62 2.2 0.31

Non-OECD countries

Hong Kong-China 23.0 1.00 18.6 0.98 9.8 0.89

Indonesia 1.9 0.38 0.6 0.19 0.1 0.08

Latvia 17.6 1.16 8.2 0.73 3.2 0.50

Liechtenstein 20.5 3.39 18.6 2.26 10.5 1.65

Macao-China 21.6 1.81 13.8 1.17 5.7 0.97

Russian Federation 15.3 1.09 6.9 0.68 2.6 0.41

Serbia and Montenegro 7.2 0.68 2.5 0.40 0.5 0.13

Thailand 5.3 0.58 1.8 0.38 0.4 0.16

Tunisia 1.8 0.28 0.4 0.14 # †

Uruguay 8.8 0.75 3.4 0.40 0.9 0.23

United Kingdom1 21.4 0.73 11.7 0.72 4.9 0.46

† Not applicable.

# Rounds to zero.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a

majority of items at that level. Students were classified into mathematics literacy levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a

score greater than 357.77 and less than or equal to 420.07); level 2 (a score greater than 420.07 and less than or

equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68); level 4 (a score greater

than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the

OECD member countries with data available. Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

84

Table B-10. Percentage distribution of 15-year-old students scoring at each proficiency level on

the mathematics literacy uncertainty subscale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 7.4 0.15 13.3 0.17 21.5 0.16 23.8 0.17

OECD countries

Australia 4.1 0.40 9.0 0.47 17.5 0.57 23.8 0.59

Austria 7.4 0.74 15.2 0.98 22.9 1.27 24.3 1.11

Belgium 6.2 0.54 11.1 0.54 17.3 0.56 20.4 0.62

Canada 2.0 0.22 6.4 0.37 16.5 0.55 25.6 0.50

Czech Republic 5.2 0.62 14.4 0.85 24.4 1.09 24.2 0.96

Denmark 4.4 0.59 10.4 0.69 20.8 0.78 25.8 0.84

Finland 1.6 0.23 5.5 0.64 15.4 0.65 27.2 0.80

France 6.0 0.68 12.3 0.87 20.9 0.79 25.3 1.03

Germany 8.7 0.80 15.2 0.78 21.8 0.92 22.6 1.01

Greece 12.8 1.13 20.4 1.30 27.3 1.00 23.1 0.91

Hungary 6.0 0.73 15.2 0.93 26.2 1.06 26.5 0.90

Iceland 4.0 0.38 8.9 0.59 18.8 0.69 24.4 1.07

Ireland 3.6 0.45 10.2 0.74 21.2 0.86 26.5 0.93

Italy 13.7 1.05 18.9 0.70 25.6 0.71 22.2 0.88

Japan 4.9 0.61 9.1 0.91 17.5 0.83 23.7 1.08

Korea, Republic of 2.2 0.27 7.2 0.63 17.3 0.83 25.0 0.96

Luxembourg 8.2 0.41 14.6 0.76 22.8 1.05 24.5 1.21

Mexico 35.3 1.66 30.6 1.29 21.3 1.02 9.5 0.84

Netherlands 1.0 0.24 6.7 0.83 17.0 1.00 23.4 1.25

New Zealand 3.9 0.55 9.4 0.77 18.0 1.04 23.3 1.01

Norway 5.7 0.56 11.8 0.79 20.6 0.82 24.4 1.22

Poland 5.2 0.63 13.9 0.88 25.7 1.04 27.4 0.86

Portugal 9.0 1.09 18.4 1.08 27.7 1.01 25.6 1.11

Slovak Republic 8.6 0.99 17.9 0.84 26.8 0.92 24.1 0.93

Spain 7.1 0.64 13.7 0.69 25.5 0.80 26.9 0.78

Sweden 6.4 0.54 11.8 0.72 21.5 0.82 22.9 0.79

Switzerland 6.3 0.49 10.7 0.65 19.1 0.78 24.0 0.88

Turkey 18.6 1.52 25.6 1.37 25.3 1.17 16.6 1.30

United States 9.0 0.76 14.9 0.71 22.3 0.74 23.6 0.75

Non-OECD countries

Hong Kong-China 3.3 0.65 6.3 0.70 12.5 0.86 19.3 0.89

Indonesia 35.3 1.57 36.7 1.01 20.4 1.08 6.2 0.69

Latvia 8.3 0.76 17.8 1.20 28.1 1.25 25.7 1.20

Liechtenstein 5.2 1.60 9.5 1.95 18.4 2.33 23.0 2.90

Macao-China 2.5 0.57 7.2 1.29 18.9 1.62 27.4 2.03

Russian Federation 19.0 1.40 24.8 1.05 26.3 0.99 18.1 1.04

Serbia and Montenegro 20.1 1.30 27.3 1.12 26.8 1.06 17.4 1.29

Thailand 18.1 1.08 32.8 0.95 29.6 1.04 14.1 0.86

Tunisia 47.9 1.27 32.3 0.96 14.8 0.88 4.2 0.61

Uruguay 27.1 1.27 23.5 1.13 23.5 1.26 16.0 0.77

United Kingdom1 3.8 0.42 10.1 0.64 20.4 0.69 25.7 0.76

See notes at end of table.
85

PISA 2003 Results From the U.S. Perspective

Table B-10. Percentage distribution of 15-year-old students scoring at

each proficiency level on the mathematics literacy uncertainty

subscale, by country: 2003—Continued

Country Level 4 Level 5 Level 6

Percent s.e. Percent s.e. Percent s.e.

OECD average 19.2 0.17 10.6 0.14 4.2 0.10

OECD countries

Australia 23.0 0.60 15.1 0.52 7.4 0.52

Austria 17.9 1.14 9.3 0.71 3.0 0.41

Belgium 20.8 0.57 15.8 0.53 8.4 0.43

Canada 26.3 0.56 16.4 0.55 6.8 0.47

Czech Republic 19.2 0.87 9.3 0.85 3.3 0.44

Denmark 22.0 0.82 12.6 0.66 4.0 0.43

Finland 27.0 0.87 16.4 0.82 6.8 0.57

France 21.7 0.75 11.0 0.65 2.8 0.28

Germany 19.0 0.94 9.7 0.78 2.9 0.32

Greece 11.8 0.94 4.0 0.59 0.7 0.15

Hungary 17.3 0.91 7.1 0.70 1.6 0.32

Iceland 22.9 0.86 14.8 0.68 6.1 0.50

Ireland 22.0 0.93 12.4 0.72 4.0 0.39

Italy 13.0 0.79 5.1 0.38 1.4 0.16

Japan 23.5 1.35 14.8 0.95 6.6 0.95

Korea, Republic of 25.7 0.95 15.7 0.81 6.7 0.80

Luxembourg 18.2 0.68 8.7 0.60 2.9 0.41

Mexico 2.7 0.43 0.5 0.11 # †

Netherlands 23.2 1.30 19.1 1.06 9.5 0.78

New Zealand 22.1 0.97 14.6 0.73 8.6 0.54

Norway 20.3 0.84 11.6 0.89 5.6 0.42

Poland 18.7 1.04 7.5 0.75 1.6 0.25

Portugal 14.5 1.00 4.2 0.38 0.6 0.18

Slovak Republic 15.7 0.80 5.6 0.46 1.2 0.23

Spain 18.4 0.71 6.9 0.51 1.5 0.26

Sweden 19.7 0.76 12.1 0.61 5.6 0.54

Switzerland 21.2 0.83 12.9 0.98 5.8 0.74

Turkey 8.0 1.14 3.4 0.77 2.6 1.07

United States 17.4 0.75 9.5 0.70 3.2 0.39

Non-OECD countries

Hong Kong-China 24.8 1.19 21.1 1.10 12.7 1.12

Indonesia 1.3 0.33 0.1 0.05 # †

Latvia 14.6 0.92 4.5 0.52 1.0 0.25

Liechtenstein 23.8 2.97 14.9 2.54 5.1 1.37

Macao-China 23.5 1.70 14.9 1.53 5.4 1.02

Russian Federation 8.6 0.76 2.7 0.43 0.5 0.14

Serbia and Montenegro 6.7 0.72 1.5 0.28 0.2 0.14

Thailand 4.3 0.50 1.1 0.25 0.1 0.06

Tunisia 0.8 0.28 # † # †

Uruguay 7.1 0.54 2.4 0.29 0.4 0.13

United Kingdom1 22.3 0.65 12.8 0.71 4.8 0.51

† Not applicable.

# Rounds to zero.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a

majority of items at that level. Students were classified into mathematics literacy levels according to their

scores. Exact cut point scores are as follows: below level 1 (a score less than or equal to 357.77); level 1 (a

score greater than 357.77 and less than or equal to 420.07); level 2 (a score greater than 420.07 and less than or

equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68); level 4 (a score greater

than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the

OECD member countries with data available. Because PISA is principally an OECD study, the results for

non-OECD countries are displayed separately from those of the OECD countries and are not included in

the OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

86

Table B-11. Average selected mathematics literacy subscale scores of 15-year-old students, by 

country: 2000 and 2003

Country

Space and shape Change and relationships

2000 2003 2000 2003

Average s.e. Average s.e. Average s.e. Average s.e.

OECD average — — 498.8 0.70 — — 496.3 0.65

OECD countries

Australia 519.7 3.14 520.5 2.33 522.2 3.21 525.3 2.30

Austria 510.1 2.85 515.2 3.48 499.3 3.10 499.7 3.60

Belgium 501.9 3.05 529.6 2.26 513.7 3.82 535.3 2.45

Canada 515.3 1.50 517.8 1.81 519.7 1.26 536.7 1.93

Czech Republic 510.1 3.51 528.0 4.17 484.4 2.98 514.1 3.55

Denmark 526.2 2.58 512.4 2.76 499.3 2.70 509.3 2.99

Finland 533.0 2.05 539.0 2.04 529.5 2.12 543.0 2.19

France 501.0 2.69 507.6 2.98 514.5 2.70 519.7 2.62

Germany 485.9 3.10 499.6 3.28 485.1 2.41 507.2 3.73

Greece 450.4 4.35 437.1 3.80 429.9 5.21 435.6 4.31

Hungary 478.4 3.30 479.0 3.34 478.6 4.05 494.6 3.10

Iceland 518.8 2.30 503.5 1.46 507.5 2.79 509.4 1.43

Ireland 473.5 3.23 476.2 2.43 501.1 2.74 505.9 2.45

Italy 454.7 3.61 470.3 3.15 442.8 3.00 452.1 3.21

Japan 564.8 5.06 553.1 4.31 536.2 5.12 536.1 4.33

Korea, Republic of 538.2 3.59 551.7 3.80 530.1 2.55 547.5 3.52

Luxembourg 448.6 3.02 488.2 1.35 423.8 2.63 487.0 1.15

Mexico 399.8 2.62 381.7 3.20 357.6 3.14 364.1 4.14

New Zealand 523.7 4.01 524.9 2.34 527.1 3.04 525.7 2.37

Norway 490.5 3.09 482.6 2.55 493.6 3.05 487.7 2.64

Poland 469.9 5.49 490.3 2.66 451.2 5.66 484.3 2.70

Portugal 439.5 3.54 450.2 3.43 448.5 3.56 467.8 3.96

Spain 472.7 2.57 476.5 2.59 467.8 2.81 480.7 2.80

Sweden 509.8 2.64 498.3 2.57 501.6 2.63 505.0 2.94

Switzerland 538.9 3.58 539.5 3.50 509.6 4.84 522.7 3.65

United States 461.2 4.94 472.0 2.78 485.7 6.02 485.5 3.03

Non-OECD countries

Latvia — — 486.4 4.04 — — 487.2 4.36

Liechtenstein — — 538.2 4.58 — — 539.5 3.67

Russian Federation — — 474.3 4.69 — — 476.8 4.64

United Kingdom1 504.9 2.58 496.0 2.50 519.2 2.21 512.9 2.54

—Not available.

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data available. Because

PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries

and are not included in the OECD average. s.e. means standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2000 and 2003. 
87

PISA 2003 Results From the U.S. Perspective

Table B-12. Average problem-solving

scores of 15-year-old 

students, by country: 2003

Country Average s.e.

OECD average 500.0 0.65

OECD countries

Australia 529.9 1.98

Austria 506.1 3.18

Belgium 524.7 2.21

Canada 529.3 1.74

Czech Republic 516.4 3.42

Denmark 516.8 2.54

Finland 547.6 1.86

France 519.2 2.67

Germany 513.5 3.24

Greece 448.5 3.97

Hungary 501.1 2.86

Iceland 504.7 1.38

Ireland 498.5 2.34

Italy 469.5 3.10

Japan 547.3 4.05

Korea, Republic of 550.4 3.06

Luxembourg 493.7 1.37

Mexico 384.4 4.29

Netherlands 520.2 2.95

New Zealand 532.8 2.17

Norway 489.8 2.60

Poland 486.6 2.78

Portugal 469.9 3.87

Slovak Republic 491.8 3.38

Spain 482.3 2.73

Sweden 508.6 2.44

Switzerland 521.3 3.05

Turkey 407.6 6.03

United States 477.4 3.13

Non-OECD countries

Hong Kong-China 547.9 4.18

Indonesia 361.5 3.29

Latvia 482.5 3.90

Liechtenstein 529.5 3.95

Macao-China 532.4 2.53

Russian Federation 478.6 4.59

Serbia and Montenegro 420.2 3.32

Thailand 425.0 2.72

Tunisia 344.8 2.11

Uruguay 410.7 3.68

United Kingdom1 510.2 2.38

1

Due to low response rates, data for the United

Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the

national averages of the OECD member countries

with data available. Because PISA is principally an

OECD study, the results for non-OECD countries are

displayed separately from those of the OECD countries and are not included in the OECD average. s.e.

means standard error.

SOURCE: Organization for Economic Cooperation and

Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

88

Table B-13. Problem-solving scores of 15-year-old students, by percentiles and country: 2003

Country 5th percentile 10th percentile 25th percentile 50th percentile

Score s.e. Score s.e. Score s.e. Score s.e.

OECD average 328.2 1.61 368.4 1.32 434.1 1.10 503.1 0.28

OECD countries

Australia 370.9 4.07 409.3 3.46 469.5 2.78 535.1 0.70

Austria 356.9 5.14 388.0 4.50 443.0 4.08 508.9 1.30

Belgium 339.4 5.72 381.1 4.66 455.0 3.32 533.6 1.19

Canada 378.8 2.43 414.0 2.78 470.8 2.46 532.1 0.61

Czech Republic 356.5 8.56 394.0 6.17 454.0 4.43 519.0 0.94

Denmark 368.9 5.02 402.3 4.28 458.5 3.06 520.0 1.14

Finland 408.6 4.67 441.6 2.80 494.5 2.48 550.5 1.10

France 358.4 6.09 395.9 4.78 458.6 3.94 523.8 1.65

Germany 350.7 5.91 383.5 5.35 446.9 4.78 520.5 1.20

Greece 283.1 5.57 319.2 5.25 382.7 4.52 450.8 2.25

Hungary 343.0 5.82 377.6 4.05 436.5 3.76 503.9 1.80

Iceland 358.0 5.47 393.4 3.28 449.8 2.22 510.1 1.44

Ireland 364.0 4.48 394.6 3.80 444.9 3.14 500.0 1.06

Italy 288.8 8.75 334.2 6.47 406.1 4.72 475.2 1.53

Japan 362.1 8.27 405.8 6.83 481.4 5.71 555.8 1.56

Korea, Republic of 404.1 4.63 437.9 5.21 493.9 3.85 553.7 1.52

Luxembourg 339.1 3.70 373.4 2.32 432.1 2.44 496.0 1.54

Mexico 226.6 5.44 261.8 5.18 317.2 5.20 384.5 1.93

Netherlands 372.2 5.89 401.5 5.14 456.1 4.89 521.3 1.73

New Zealand 369.8 3.75 405.6 4.23 467.8 3.65 538.0 1.21

Norway 322.3 5.54 360.9 4.64 424.4 3.71 493.6 2.01

Poland 338.0 5.55 372.0 4.07 428.4 3.10 487.8 1.44

Portugal 311.2 7.91 345.3 6.84 408.6 5.65 475.2 2.18

Slovak Republic 336.6 7.13 370.4 5.86 430.1 4.71 494.0 0.79

Spain 321.8 4.78 361.2 4.12 420.9 3.55 486.0 0.93

Sweden 359.8 6.39 395.2 4.41 451.3 3.04 511.4 1.38

Switzerland 358.2 5.73 397.0 4.04 461.3 3.32 526.8 1.87

Turkey 257.2 7.84 290.7 6.58 343.0 5.16 402.3 1.34

United States 312.2 5.63 347.3 4.55 409.7 4.07 479.5 1.71

Non-OECD countries

Hong Kong-China 376.2 10.52 420.2 7.86 487.0 6.12 557.8 1.64

Indonesia 244.7 4.22 269.9 3.77 311.7 3.60 359.3 1.91

Latvia 326.2 6.97 361.9 6.00 419.9 5.38 484.6 1.04

Liechtenstein 369.0 14.93 404.1 11.09 467.6 6.03 532.8 4.89

Macao-China 394.8 6.41 424.8 5.58 477.8 3.72 535.7 1.69

Russian Federation 314.2 7.73 351.0 6.99 412.7 5.72 480.7 1.69

Serbia and Montenegro 278.7 4.15 310.6 4.40 362.8 3.94 420.2 2.53

Thailand 293.0 3.87 322.4 3.37 369.1 2.64 422.0 2.15

Tunisia 213.0 4.30 242.7 3.10 290.7 2.54 344.4 1.01

Uruguay 223.8 5.69 265.0 5.10 334.5 4.66 413.5 1.69

United Kingdom1 352.7 4.40 387.1 3.59 446.1 2.90 513.2 1.03

See notes at end of table.
89

PISA 2003 Results From the U.S. Perspective

Table B-13. Problem-solving scores of 15-year-old students, by percentiles

and country: 2003—Continued 

Country 75th percentile 90th percentile 95th percentile

Score s.e. Score s.e. Score s.e.

OECD average 570.9 0.82 625.0 0.75 655.6 0.82

OECD countries

Australia 593.9 2.13 643.5 2.67 671.7 3.40

Austria 569.4 3.98 620.7 4.18 650.9 4.63

Belgium 601.2 2.24 653.4 2.21 682.0 2.77

Canada 591.1 1.89 640.4 2.13 668.8 2.40

Czech Republic 582.2 3.58 633.7 3.87 663.5 4.01

Denmark 578.4 2.76 627.0 3.44 654.9 3.71

Finland 604.0 2.28 649.6 2.33 676.5 3.58

France 585.8 3.05 634.7 3.66 662.3 4.51

Germany 583.1 4.28 631.6 2.66 658.4 3.19

Greece 516.6 4.58 574.5 5.72 607.5 5.60

Hungary 566.7 3.93 622.1 4.31 653.1 5.37

Iceland 564.3 2.03 609.1 2.33 634.4 3.59

Ireland 555.3 2.73 600.7 2.83 624.5 3.21

Italy 540.1 2.95 595.4 3.41 627.2 3.56

Japan 620.9 4.22 675.3 4.56 705.3 6.01

Korea, Republic of 609.9 3.47 658.3 4.21 686.0 5.53

Luxembourg 558.0 2.17 609.8 2.59 639.9 3.40

Mexico 451.5 5.06 509.3 5.69 541.6 6.48

Netherlands 586.7 3.59 636.4 3.34 662.1 3.71

New Zealand 600.9 2.38 652.7 2.49 682.4 2.76

Norway 558.8 3.27 614.8 4.17 645.5 4.38

Poland 547.6 2.97 600.2 3.53 631.7 4.49

Portugal 534.3 3.60 586.3 3.45 614.5 3.54

Slovak Republic 558.0 3.58 609.0 3.84 638.1 4.16

Spain 547.3 3.22 599.4 3.89 629.4 3.31

Sweden 570.8 3.06 619.0 3.82 647.2 3.56

Switzerland 586.9 3.86 636.9 4.60 666.2 5.18

Turkey 466.5 7.69 531.2 11.94 576.7 18.61

United States 547.9 3.29 603.6 3.97 634.8 4.18

Non-OECD countries

Hong Kong-China 617.4 3.18 663.5 2.93 690.2 3.72

Indonesia 408.6 4.10 456.8 5.51 486.9 5.92

Latvia 547.3 4.57 599.0 4.11 628.4 4.89

Liechtenstein 598.9 9.30 644.1 10.46 671.8 12.04

Macao-China 590.0 4.34 632.9 5.43 658.9 6.54

Russian Federation 546.0 5.12 603.7 5.03 637.2 5.55

Serbia and Montenegro 477.7 4.19 529.9 4.93 559.8 5.08

Thailand 478.5 3.98 531.7 4.01 565.4 5.97

Tunisia 400.0 2.76 445.8 4.11 474.5 4.98

Uruguay 488.4 5.45 552.1 5.04 589.3 5.74

United Kingdom1 576.5 3.07 628.9 3.69 659.2 3.98

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with

data available. Because PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD countries and are not included in the OECD average. s.e.

means standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

90

Table B-14. Standard deviations of 15-year-old

students' problem-solving scores,

by country: 2003

Country Standard deviation s.e.

OECD average 100.0 0.44

OECD countries

Australia 91.4 1.35

Austria 89.9 1.71

Belgium 104.4 1.57

Canada 88.4 0.93

Czech Republic 92.8 1.92

Denmark 87.3 1.51

Finland 82.0 1.15

France 92.9 2.08

Germany 94.7 1.75

Greece 98.8 1.67

Hungary 94.1 2.03

Iceland 84.8 1.15

Ireland 79.6 1.35

Italy 102.1 2.14

Japan 104.9 2.72

Korea, Republic of 86.4 1.95

Luxembourg 91.6 1.03

Mexico 96.1 2.01

Netherlands 89.4 2.03

New Zealand 95.7 1.24

Norway 98.8 1.65

Poland 90.4 1.68

Portugal 92.4 2.11

Slovak Republic 92.8 2.39

Spain 93.6 1.25

Sweden 88.4 1.58

Switzerland 94.0 1.88

Turkey 96.7 4.43

United States 98.1 1.29

Non-OECD countries

Hong Kong-China 97.2 2.90

Indonesia 73.2 1.74

Latvia 92.1 1.75

Liechtenstein 92.7 4.20

Macao-China 81.3 2.55

Russian Federation 98.5 2.11

Serbia and Montenegro 85.8 1.56

Thailand 82.0 1.59

Tunisia 79.5 1.42

Uruguay 111.7 1.93

United Kingdom1 93.2 1.21

1

Due to low response rates, data for the United Kingdom are not

discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data available.

Because PISA is principally an OECD study, the results for nonOECD countries are displayed separately from those of the

OECD countries and are not included in the OECD average. s.e.

means standard error.

SOURCE: Organization for Economic Cooperation and

Development (OECD), Program for International Student

Assessment (PISA), 2003. 
91

PISA 2003 Results From the U.S. Perspective

Table B-15. Percentage distribution of 15-year-old students scoring at each proficiency level on

the problem-solving scale, by country: 2003

Country Below level 1 Level 1 Level 2 Level 3

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 17.3 0.25 30.4 0.20 34.2 0.22 18.2 0.18

OECD countries

Australia 9.4 0.59 25.8 0.67 39.1 0.83 25.7 0.84

Austria 13.6 1.00 32.3 1.07 36.8 1.06 17.2 1.16

Belgium 14.0 0.80 23.8 0.88 34.1 0.82 28.1 0.81

Canada 8.5 0.48 27.0 0.70 40.0 0.69 24.5 0.72

Czech Republic 12.1 1.14 29.4 1.20 37.0 1.12 21.5 1.24

Denmark 10.5 0.76 30.2 0.92 39.2 0.90 20.1 0.93

Finland 4.6 0.51 22.1 0.84 43.3 0.82 30.1 0.92

France 11.7 0.95 28.1 1.04 37.5 1.05 22.7 0.99

Germany 14.2 1.00 27.7 1.14 36.4 1.49 21.7 1.39

Greece 32.7 1.53 36.1 1.01 24.3 1.21 7.0 0.80

Hungary 16.1 0.98 31.8 1.40 34.9 1.20 17.2 1.16

Iceland 12.4 0.66 32.5 1.01 40.2 0.97 14.9 0.64

Ireland 12.5 0.91 36.9 1.17 38.3 1.05 12.3 0.76

Italy 24.7 1.32 34.7 1.16 30.0 0.99 10.6 0.66

Japan 9.9 1.00 20.0 0.98 34.5 1.25 35.6 1.58

Korea, Republic of 5.2 0.54 21.6 1.02 40.8 1.11 32.4 1.35

Luxembourg 17.0 0.71 34.1 0.95 34.7 1.04 14.2 0.56

Mexico 58.1 1.86 29.7 1.07 10.9 0.99 1.3 0.24

Netherlands 10.7 1.13 30.5 1.31 35.8 1.41 23.0 1.13

New Zealand 9.9 0.81 25.3 0.85 36.5 1.01 28.3 0.92

Norway 19.4 0.88 32.6 1.15 33.1 0.97 14.9 0.84

Poland 17.5 0.98 37.2 1.02 33.6 1.14 11.7 0.66

Portugal 23.9 1.73 36.5 1.09 31.0 1.36 8.6 0.63

Slovak Republic 17.5 1.40 34.4 1.16 34.0 1.34 14.1 0.96

Spain 20.1 0.87 35.5 1.08 32.9 1.15 11.6 0.83

Sweden 12.0 0.85 32.4 1.12 38.2 1.02 17.4 0.99

Switzerland 11.4 0.74 26.8 1.02 38.7 1.10 23.1 1.42

Turkey 51.2 2.51 32.5 1.58 12.4 1.64 3.9 1.24

United States 23.7 1.12 33.7 0.83 30.3 1.03 12.4 0.78

Non-OECD countries

Hong Kong-China 8.0 1.05 20.5 0.99 36.5 1.17 35.0 1.39

Indonesia 73.5 1.71 22.9 1.36 3.5 0.64 0.1 0.07

Latvia 20.3 1.48 35.6 1.27 32.5 1.39 11.6 0.96

Liechtenstein 10.2 1.54 26.0 2.35 36.8 3.56 27.1 2.62

Macao-China 6.3 0.82 27.3 1.37 42.1 2.00 24.2 1.59

Russian Federation 22.8 1.69 34.5 1.03 30.6 1.28 12.2 1.02

Serbia and Montenegro 42.6 1.70 39.5 1.16 15.8 1.17 2.1 0.31

Thailand 41.4 1.57 40.5 1.08 15.6 1.10 2.6 0.47

Tunisia 77.1 1.05 20.4 0.82 2.5 0.51 0.1 0.05

Uruguay 47.2 1.58 30.5 1.33 17.5 1.23 4.7 0.54

United Kingdom1 13.7 0.75 30.3 1.09 36.6 1.00 19.5 0.97

1

Due to low response rates, data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular proficiency level, a student must have been able to correctly answer a majority of items at that

level. Students were classified into problem solving levels according to their scores. Exact cut point scores are as follows: below

level 1 (a score less than or equal to 404.06); level 1 (a score greater than 404.06 and less than or equal to 498.08); level 2 (a score greater

than 498.08 and less than or equal to 592.10); level 3 (a score greater than 592.10).The OECD average is the average of the national

averages of the OECD member countries with data available. Because PISA is principally an OECD study, the results for nonOECD countries are displayed separately from those of the OECD countries and are not included in the OECD average. s.e. means

standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

92

Table B-16. Average reading literacy scores of 15-year-old 

students, by country: 2000 and 2003

Country 2000 2003

Average s.e. Average s.e.

OECD average 500.0 0.64 494.2 0.65

OECD countries

Australia 528.3 3.52 525.4 2.13

Austria 507.1 2.40 490.7 3.76

Belgium 507.1 3.56 507.9 2.57

Canada 534.3 1.56 527.9 1.75

Czech Republic 491.6 2.37 488.5 3.46

Denmark 496.9 2.35 492.3 2.82

Finland 546.5 2.58 543.5 1.64

France 504.7 2.73 496.2 2.68

Germany 484.0 2.47 491.4 3.39

Greece 473.8 4.97 472.3 4.10

Hungary 480.0 3.95 481.9 2.47

Iceland 506.9 1.45 491.7 1.56

Ireland 526.7 3.24 515.5 2.63

Italy 487.5 2.91 475.7 3.04

Japan 522.2 5.21 498.1 3.92

Korea, Republic of 524.8 2.42 534.1 3.09

Luxembourg 441.3 1.59 479.4 1.48

Mexico 422.0 3.32 399.7 4.09

Netherlands — — 513.1 2.85

New Zealand 528.8 2.78 521.6 2.46

Norway 505.3 2.80 499.7 2.78

Poland 479.1 4.46 496.6 2.88

Portugal 470.2 4.52 477.6 3.73

Slovak Republic — — 469.2 3.12

Spain 492.6 2.71 480.5 2.60

Sweden 516.3 2.20 514.3 2.42

Switzerland 494.4 4.25 499.1 3.28

Turkey — — 441.0 5.79

United States 504.4 7.05 495.2 3.22

Non-OECD countries

Hong Kong-China — — 509.5 3.69

Indonesia — — 381.6 3.38

Latvia 458.1 5.27 490.6 3.67

Liechtenstein 482.6 4.12 525.1 3.58

Macao-China — — 497.6 2.16

Russian Federation 461.8 4.16 442.2 3.94

Serbia and Montenegro — — 411.7 3.56

Thailand — — 419.9 2.81

Tunisia — — 374.6 2.81

Uruguay — — 434.1 3.43

United Kingdom1 523.4 2.56 507.0 2.46

—Not available.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in

this report.

NOTE: The OECD average is the average of the national averages of the OECD

member countries with data available. Because PISA is principally an OECD

study, the results for non-OECD countries are displayed separately from those

of the OECD countries and are not included in the OECD average. s.e. means

standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD),

Program for International Student Assessment (PISA), 2000 and 2003. 
93

PISA 2003 Results From the U.S. Perspective

Table B-17. Average science literacy scores of 15-year-old 

students, by country: 2000 and 2003

Country 2000 2003

Average s.e. Average s.e.

OECD average 500.0 0.65 499.6 0.60

OECD countries

Australia 527.5 3.47 525.1 2.10

Austria 518.6 2.55 491.0 3.44

Belgium 495.7 4.29 508.8 2.44

Canada 529.4 1.57 518.7 2.02

Czech Republic 511.4 2.43 523.3 3.38

Denmark 481.0 2.81 475.2 2.97

Finland 537.7 2.48 548.2 1.92

France 500.5 3.18 511.2 2.99

Germany 487.1 2.43 502.3 3.64

Greece 460.6 4.89 481.0 3.82

Hungary 496.1 4.17 503.3 2.77

Iceland 495.9 2.17 494.7 1.47

Ireland 513.4 3.18 505.4 2.69

Italy 477.6 3.05 486.5 3.13

Japan 550.4 5.48 547.6 4.14

Korea, Republic of 552.1 2.69 538.4 3.54

Luxembourg 443.1 2.32 482.8 1.50

Mexico 421.5 3.18 404.9 3.49

Netherlands — — 524.4 3.15

New Zealand 527.7 2.40 520.9 2.35

Norway 500.3 2.75 484.2 2.87

Poland 483.1 5.12 497.8 2.86

Portugal 459.0 4.00 467.7 3.46

Slovak Republic — — 494.9 3.71

Spain 490.9 2.95 487.1 2.61

Sweden 512.1 2.51 506.1 2.72

Switzerland 495.7 4.45 513.0 3.69

Turkey — — 434.2 5.89

United States 499.5 7.31 491.3 3.08

Non-OECD countries

Hong Kong-China — — 539.5 4.26

Indonesia — — 395.0 3.21

Latvia 460.1 5.62 489.1 3.89

Liechtenstein 476.1 7.09 525.2 4.33

Macao-China — — 524.7 3.03

Russian Federation 460.3 4.74 489.3 4.14

Serbia and Montenegro — — 436.4 3.50

Thailand — — 429.1 2.70

Tunisia — — 384.7 2.56

Uruguay — — 438.4 2.90

United Kingdom1 532.0 2.69 518.4 2.52

—Not available.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in

this report.

NOTE: The OECD average is the average of the national averages of the OECD

member countries with data available. Because PISA is principally an OECD

study, the results for non-OECD countries are displayed separately from those

of the OECD countries and are not included in the OECD average. s.e. means

standard error.

SOURCE: Organization for Economic Cooperation and Development (OECD),

Program for International Student Assessment (PISA), 2000 and 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

94

Table B-18. Average combined mathematics literacy scores of 15-year-old 

students, by sex and country: 2003

Country

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 505.5 0.76 494.4 0.76 11.1 0.81

OECD countries

Australia 526.9 3.01 521.5 2.69 5.3 3.75

Austria 509.4 3.95 501.8 3.96 7.6 4.41

Belgium 532.9 3.40 525.4 3.23 7.5 4.81

Canada 540.8 2.05 529.6 1.88 11.2 2.13

Czech Republic 523.8 4.27 508.9 4.37 15.00 5.09

Denmark 522.7 3.38 506.2 2.98 16.6 3.21

Finland 548.0 2.46 540.6 2.11 7.4 2.67

France 515.3 3.55 506.7 2.92 8.5 4.16

Germany 507.9 3.98 498.9 3.93 9.0 4.37

Greece 455.0 4.75 435.6 3.85 19.4 3.63

Hungary 493.7 3.33 485.9 3.31 7.8 3.54

Iceland 507.7 2.28 523.1 2.17 -15.4 3.46

Ireland 510.2 3.01 495.4 3.39 14.8 4.19

Italy 474.9 4.56 457.1 3.84 17.8 5.89

Japan 538.5 5.81 530.1 3.97 8.4 5.90

Korea, Republic of 551.7 4.36 528.3 5.35 23.4 6.77

Luxembourg 501.9 1.86 484.8 1.53 17.2 2.81

Mexico 390.8 4.28 380.0 4.08 10.9 3.94

Netherlands 540.3 4.08 535.2 3.48 5.1 4.29

New Zealand 530.7 2.77 516.2 3.20 14.5 3.90

Norway 498.3 2.85 492.0 2.87 6.2 3.21

Poland 493.0 2.97 487.5 2.95 5.6 3.14

Portugal 472.4 4.18 460.2 3.44 12.2 3.31

Slovak Republic 507.3 3.90 488.6 3.59 18.7 3.65

Spain 489.6 3.38 480.7 2.19 8.9 2.98

Sweden 512.3 2.98 505.8 3.09 6.5 3.27

Switzerland 534.6 4.74 518.0 3.65 16.6 4.87

Turkey 430.2 7.88 415.1 6.68 15.1 6.16

United States 485.9 3.33 479.7 3.24 6.3 2.89

Non-OECD countries

Hong Kong-China 552.4 6.52 548.3 4.56 4.1 6.65

Indonesia 361.8 3.92 358.5 4.57 3.3 3.39

Latvia 484.8 4.79 482.0 3.56 2.8 3.97

Liechtenstein 549.8 7.23 521.0 6.28 28.8 10.92

Macao-China 538.2 4.82 516.9 3.28 21.3 5.83

Russian Federation 473.5 5.27 463.4 4.21 10.1 4.36

Serbia and Montenegro 437.5 4.24 436.3 4.45 1.2 4.36

Thailand 414.7 3.96 418.8 3.40 -4.0 4.24

Tunisia 364.9 2.74 352.7 2.93 12.2 2.51

Uruguay 428.4 3.95 416.3 3.84 12.1 4.15

United Kingdom1 511.8 2.90 505.1 3.88 6.7 4.90

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The male-female score point difference is calculated by subtracting the average scores of females

from the average scores of males. The OECD average is the average of the national averages of the OECD

member countries with data available. Because PISA is principally an OECD study, the results for nonOECD countries are displayed separately from those of the OECD countries and are not included in the

OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
95

PISA 2003 Results From the U.S. Perspective

Table B-19. Percentage distribution of 15-year-old students scoring at each proficiency level on

the combined mathematics literacy scale, by sex and country: 2003

Country

Below level 1 Level 1

Male Female Male Female

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 8.1 0.21 8.4 0.19 12.6 0.19 13.8 0.21

OECD countries

Australia 4.6 0.61 4.0 0.53 10.3 0.81 9.7 0.67

Austria 6.1 0.95 5.1 0.72 13.1 0.99 13.3 1.20

Belgium 7.4 0.82 6.9 0.82 9.8 0.87 8.8 0.77

Canada 2.9 0.39 2.0 0.28 7.4 0.48 7.4 0.49

Czech Republic 4.3 0.67 5.7 1.10 10.9 1.07 12.3 1.29

Denmark 3.8 0.57 5.6 0.78 9.6 0.88 11.8 0.93

Finland 1.6 0.33 1.4 0.29 5.8 0.59 4.9 0.56

France 6.1 0.96 5.2 0.73 10.7 1.02 11.3 0.97

Germany 8.9 1.04 9.2 1.05 12.5 0.99 12.1 1.02

Greece 16.4 1.33 19.1 1.53 19.4 1.30 22.8 1.47

Hungary 7.6 0.84 8.0 1.07 14.6 1.04 15.9 1.08

Iceland 6.1 0.62 2.8 0.49 12.1 0.88 8.8 0.79

Ireland 4.2 0.79 5.2 0.74 10.8 1.13 13.5 1.28

Italy 12.5 1.55 13.9 1.70 17.2 1.57 20.1 1.33

Japan 5.2 0.90 4.3 0.74 9.1 0.94 8.1 0.87

Korea, Republic of 2.3 0.42 2.7 0.49 6.2 0.81 8.3 0.99

Luxembourg 6.8 0.64 8.0 0.73 13.2 0.80 15.3 1.13

Mexico 36.2 2.12 39.7 1.87 26.9 1.61 28.8 1.27

Netherlands 2.2 0.70 2.9 0.84 8.0 1.23 8.7 1.18

New Zealand 4.7 0.57 5.2 0.70 9.9 0.81 10.4 1.00

Norway 7.3 0.71 6.5 0.76 13.3 0.90 14.5 1.07

Poland 7.7 0.88 5.9 0.71 14.9 0.92 15.5 1.05

Portugal 12 1.41 10.6 1.21 16.7 1.12 20.6 1.32

Slovak Republic 6.1 0.91 7.4 0.94 12.0 1.07 14.5 1.25

Spain 8.4 0.89 7.8 0.69 14.1 1.11 15.7 0.99

Sweden 5.6 0.63 5.6 0.71 11.1 0.85 12.3 0.81

Switzerland 4.4 0.46 5.5 0.65 9.1 0.76 10.2 0.83

Turkey 26.4 2.35 29.2 2.37 22.9 1.51 26.6 1.82

United States 10.5 0.96 9.9 1.03 14.7 0.82 16.4 1.16

Non-OECD countries

Hong Kong-China 5.1 1.12 2.7 0.71 6.7 0.87 6.3 0.87

Indonesia 49.2 2.22 51.8 2.22 28.8 1.29 26.5 1.30

Latvia 8.1 1.56 7.2 1.05 16.3 1.50 15.9 1.28

Liechtenstein 4.7 1.79 4.9 2.37 5.5 2.16 9.6 3.03

Macao-China 2.3 1.07 2.4 0.77 8.5 1.83 9.1 1.72

Russian Federation 11.4 1.46 11.4 1.02 18.4 1.47 19.2 1.40

Serbia and Montenegro 19.2 1.69 16.1 1.65 24.1 1.44 24.8 1.57

Thailand 25.3 1.70 22.6 1.49 29.7 1.52 30.6 1.82

Tunisia 48.2 1.71 53.8 1.68 28.1 1.38 25.8 1.39

Uruguay 24.7 1.56 27.7 1.58 20.9 0.95 22.7 1.28

United Kingdom1 5.3 0.65 5.2 0.72 12.1 0.96 12.9 0.89

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

96

Table B-19. Percentage distribution of 15-year-old students scoring at each proficiency level on

the combined mathematics literacy scale, by sex and country: 2003—Continued 

Country

Level 2 Level 3

Male Female Male Female

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 20.0 0.20 22.1 0.23 22.9 0.23 24.5 0.22

OECD countries

Australia 17.8 0.95 19.4 0.78 22.7 1.19 25.3 0.79

Austria 20.4 1.11 22.7 1.35 23.3 1.47 26.5 1.39

Belgium 15.1 0.94 16.9 0.88 18.6 0.82 21.8 1.00

Canada 16.1 0.91 18.7 0.80 23.4 0.85 28.6 1.20

Czech Republic 19.8 1.25 20.4 1.42 23.2 1.06 25.4 1.38

Denmark 18.7 1.30 22.3 1.09 26.4 1.28 26.0 1.20

Finland 15.4 0.75 16.7 0.80 25.9 0.93 29.5 1.09

France 18.7 1.01 21.6 1.12 25.1 1.47 26.6 1.37

Germany 18.1 1.21 19.9 1.42 21.4 0.98 23.9 1.37

Greece 24.7 1.36 27.8 1.15 21.0 1.05 19.4 1.53

Hungary 23.6 1.44 24.0 1.61 23.9 1.30 24.7 1.26

Iceland 20.4 1.17 20.1 1.40 25.3 1.31 26.9 1.19

Ireland 22.5 1.44 24.7 1.37 27.8 1.46 28.2 1.36

Italy 22.8 1.27 26.4 1.35 22.7 1.12 23.1 1.17

Japan 15.8 1.15 16.9 1.10 20.2 1.41 24.5 1.19

Korea, Republic of 14.6 1.01 19.6 1.66 22.3 1.04 26.7 1.51

Luxembourg 21.4 1.06 24.4 1.22 24.8 1.09 26.9 1.12

Mexico 21.6 1.46 20.2 1.32 11.4 1.00 8.9 1.15

Netherlands 18.2 1.49 17.9 1.42 22.9 1.57 23.0 1.34

New Zealand 17.7 0.91 20.6 1.15 21.9 1.17 24.5 1.17

Norway 23.2 1.25 24.1 1.54 23.9 1.35 26.5 1.20

Poland 22.9 1.06 26.8 0.99 24.5 1.22 26.2 1.13

Portugal 24.6 1.20 29.4 1.33 23.9 1.20 24.1 1.37

Slovak Republic 22.0 1.10 25.0 1.42 24.5 1.28 25.4 1.46

Spain 23.3 1.17 26.1 0.98 25.6 1.49 27.7 1.06

Sweden 21.3 1.13 22.1 1.02 25.4 1.51 25.6 1.02

Switzerland 16.5 1.15 18.6 1.11 23.2 1.48 25.4 1.17

Turkey 22.2 1.28 21.9 1.75 14.3 1.45 12.4 1.47

United States 23.2 1.05 24.6 1.37 23.1 1.39 24.5 1.11

Non-OECD countries

Hong Kong-China 13.0 0.95 14.9 1.57 18.1 1.02 21.8 2.29

Indonesia 15.2 1.25 14.3 1.23 5.1 0.64 5.8 0.94

Latvia 24.6 1.44 26.3 1.52 25.6 1.48 26.9 1.70

Liechtenstein 15.6 3.07 19.2 3.92 19.6 3.48 23.6 3.71

Macao-China 16.8 1.74 22.3 2.12 23.7 2.59 29.8 2.60

Russian Federation 24.5 1.56 28.3 1.52 22.6 1.48 23.6 1.29

Serbia and Montenegro 26.3 1.47 30.9 1.45 17.5 1.19 20.2 1.47

Thailand 24.5 1.41 26.1 1.37 13.2 1.34 14.0 0.99

Tunisia 15.1 1.00 14.3 1.00 6.3 0.66 5.1 0.87

Uruguay 24.3 1.27 24.1 1.50 17.4 1.05 16.1 1.10

United Kingdom1 19.5 1.54 22.7 1.38 25.8 1.52 25.4 1.04

See notes at end of table.
97

PISA 2003 Results From the U.S. Perspective

Table B-19. Percentage distribution of 15-year-old students scoring at each proficiency level on

the combined mathematics literacy scale, by sex and country: 2003—Continued 

Country

Level 4 Level 5

Male Female Male Female

Percent s.e. Percent s.e. Percent s.e. Percent s.e.

OECD average 19.5 0.22 18.8 0.24 11.8 0.19 9.5 0.15

OECD countries

Australia 22.9 1.07 23.6 1.03 14.6 0.96 13.4 0.81

Austria 20.4 1.28 20.5 1.43 11.9 1.04 9.2 1.04

Belgium 20.1 0.86 22.1 0.84 18.1 1.04 16.7 0.75

Canada 25.0 0.70 25.4 1.04 17.6 0.88 13.6 0.82

Czech Republic 20.3 0.99 21.3 1.33 15.0 1.12 10.8 0.98

Denmark 23.5 1.11 20.4 1.26 13.1 1.02 10.6 1.05

Finland 25.4 1.14 26.9 1.22 17.7 1.08 15.7 0.83

France 21.6 1.51 22.6 1.06 13.3 1.16 10.1 1.02

Germany 20.7 1.32 20.6 1.18 13.0 1.14 11.3 0.97

Greece 12.8 1.24 8.6 0.83 4.8 0.80 2.1 0.46

Hungary 18.3 1.20 18.1 1.11 8.6 0.75 7.7 1.01

Iceland 21.0 1.17 25.5 1.10 11.4 0.92 12.2 0.95

Ireland 21.0 1.63 19.4 1.21 10.8 1.09 7.4 0.83

Italy 15.1 1.06 11.9 0.77 7.1 0.64 3.9 0.39

Japan 22.3 1.42 24.9 1.62 16.5 1.37 15.6 1.19

Korea, Republic of 25.9 1.42 23.6 1.53 18.9 1.20 13.4 1.21

Luxembourg 20.0 1.11 17.4 1.08 10.5 0.93 6.6 0.67

Mexico 3.4 0.53 2.1 0.52 0.5 0.15 0.2 0.07

Netherlands 22.6 1.65 22.5 1.51 18.1 1.51 18.3 1.21

New Zealand 21.9 1.18 21.8 1.08 15.7 1.00 12.4 0.99

Norway 19.1 1.23 18.7 1.17 9.7 0.79 7.7 0.69

Poland 17.9 1.23 17.5 1.06 9.0 0.89 6.5 0.78

Portugal 15.6 1.56 11.5 1.01 5.9 0.82 3.3 0.59

Slovak Republic 20.0 1.20 17.8 1.00 11.4 0.93 8.1 0.76

Spain 18.7 1.08 16.7 0.91 8.0 1.09 5.1 0.48

Sweden 19.4 1.43 20.2 1.19 12.4 1.01 10.9 1.03

Switzerland 22.6 1.25 22.3 1.16 15.2 1.80 13.1 1.15

Turkey 7.5 1.14 5.8 1.25 3.5 0.91 2.6 0.83

United States 16.9 1.08 16.2 1.01 8.9 0.70 7.2 0.79

Non-OECD countries

Hong Kong-China 23.9 1.58 26.1 1.29 20.4 1.52 19.9 1.56

Indonesia 1.4 0.36 1.3 0.49 0.2 0.10 0.2 0.13

Latvia 16.1 1.54 17.0 1.28 7.1 1.01 5.6 0.79

Liechtenstein 22.2 4.93 24.2 4.45 21.5 5.54 15.0 3.60

Macao-China 24.7 3.18 22.7 2.17 17.2 3.28 10.7 1.45

Russian Federation 14.1 1.11 12.3 1.15 6.6 0.86 4.2 0.59

Serbia and Montenegro 9.6 0.97 6.7 1.16 2.9 0.62 1.3 0.35

Thailand 5.6 0.77 5.1 0.72 1.5 0.41 1.4 0.45

Tunisia 2.0 0.42 0.9 0.36 0.3 0.19 0.2 0.10

Uruguay 8.9 0.76 7.4 0.88 3.0 0.45 1.7 0.35

United Kingdom1 21.4 0.87 19.9 1.14 11.6 1.15 10.5 1.14

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

98

Table B-19. Percentage distribution of 15-year-old students scoring at each proficiency level

on the combined mathematics literacy scale, by sex and country: 2003—Continued 

Country

Level 6

Male Female

Percent s.e. Percent s.e.

OECD average 5.1 0.14 2.9 0.10

OECD countries

Australia 7.0 0.74 4.5 0.49

Austria 4.8 0.75 2.7 0.51

Belgium 10.9 0.72 6.8 0.54

Canada 7.5 0.79 4.2 0.36

Czech Republic 6.6 0.72 4.1 0.54

Denmark 4.9 0.62 3.3 0.56

Finland 8.2 0.85 5.1 0.53

France 4.6 0.58 2.5 0.56

Germany 5.3 0.65 2.9 0.57

Greece 1.0 0.33 0.2 0.14

Hungary 3.3 0.56 1.6 0.37

Iceland 3.7 0.53 3.8 0.49

Ireland 2.9 0.50 1.6 0.36

Italy 2.5 0.30 0.7 0.13

Japan 10.9 1.94 5.7 0.78

Korea, Republic of 9.7 1.01 5.7 1.15

Luxembourg 3.4 0.56 1.4 0.32

Mexico # † # †

Netherlands 8.0 0.82 6.6 0.70

New Zealand 8.3 0.73 5.0 0.58

Norway 3.5 0.52 1.9 0.43

Poland 3.1 0.50 1.5 0.30

Portugal 1.3 0.26 0.4 0.18

Slovak Republic 4.1 0.58 1.7 0.30

Spain 1.9 0.39 1.0 0.29

Sweden 4.9 0.70 3.4 0.58

Switzerland 9.0 1.30 4.9 0.89

Turkey 3.0 1.24 1.6 0.86

United States 2.8 0.52 1.2 0.39

Non-OECD countries

Hong Kong-China 12.7 1.46 8.3 0.96

Indonesia # † # †

Latvia 2.3 0.53 1.1 0.33

Liechtenstein 10.8 2.66 3.6 1.78

Macao-China 6.8 1.86 3.0 0.91

Russian Federation 2.3 0.63 1.0 0.31

Serbia and Montenegro 0.4 0.21 0.1 0.09

Thailand 0.2 0.12 0.2 0.11

Tunisia # † # †

Uruguay 0.8 0.26 0.2 0.13

United Kingdom1 4.4 0.59 3.4 0.68

† Not applicable.

# Rounds to zero.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: In order to reach a particular level, a student must have been able to correctly answer a majority of items at that level.

Students were classified into mathematics literacy levels according to their scores. Exact cut point scores are as follows:

below level 1 (a score less than or equal to 357.7); level 1 (a score greater than 357.7 and less than or equal to 420.07); level 2 (a

score greated than 420.07 and less than or equal to 482.38); level 3 (a score greater than 482.38 and less than or equal to 544.68);

level 4 (a score greater than 544.68 and less than or equal to 606.99); level 5 (a score greater than 606.99 and less than or equal to

669.3); level 6 (a score greater than 669.3). The OECD average is the average of the national averages of the OECD member

countries with data available. Because PISA is principally an OECD study, the results for non-OECD countries are displayed

separately from those of the OECD countries and are not included in the OECD average. s.e. means standard error. Detail may

not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003. 
99

PISA 2003 Results From the U.S. Perspective

Table B-20. Average mathematics literacy subscale scores of 15-year-old 

students, by sex and country: 2003

Country

Space and shape

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 504.6 0.81 487.9 0.79 16.7 0.90

OECD countries

Australia 526.4 3.15 514.5 2.91 11.9 3.88

Austria 524.5 4.41 505.8 4.34 18.7 5.18

Belgium 538.1 3.23 520.3 3.25 17.9 4.62

Canada 530.1 2.07 510.6 2.18 19.5 2.52

Czech Republic 542.3 4.76 512.1 5.05 30.2 5.68

Denmark 520.7 3.41 504.4 3.26 16.3 3.73

Finland 540.3 2.63 537.8 2.40 2.4 2.96

France 517.0 4.26 499.2 3.25 17.8 4.72

Germany 505.8 4.03 494.3 3.99 11.5 4.69

Greece 447.1 4.75 427.8 3.82 19.3 4.02

Hungary 486.1 3.78 471.1 3.92 15.0 4.04

Iceland 496.2 2.43 511.3 2.30 -15.1 3.74

Ireland 488.9 2.96 463.4 3.44 25.5 4.28

Italy 479.7 4.74 461.5 4.06 18.1 6.33

Japan 557.8 6.30 548.9 4.20 8.9 6.35

Korea, Republic of 562.7 5.14 535.7 6.24 27.0 7.96

Luxembourg 502.5 2.21 474.3 2.02 28.3 3.28

Mexico 389.8 4.06 374.2 3.48 15.6 3.84

Netherlands 530.1 3.70 522.0 3.45 8.2 4.28

New Zealand 533.9 2.70 516.0 3.35 17.9 3.89

Norway 486.3 3.10 479.0 3.53 7.3 4.27

Poland 496.9 3.20 483.7 3.29 13.1 3.70

Portugal 458.1 4.25 443.0 3.53 15.1 3.52

Slovak Republic 522.4 4.75 487.4 4.11 35.0 4.48

Spain 485.9 3.52 467.4 2.35 18.5 2.96

Sweden 503.5 3.01 493.1 3.21 10.4 3.52

Switzerland 551.7 5.30 526.5 3.70 25.3 5.57

Turkey 422.7 7.59 410.9 6.15 11.7 5.99

United States 479.5 3.34 464.3 3.09 15.2 3.24

Non-OECD countries

Hong Kong-China 560.5 6.79 556.4 4.95 4.1 6.85

Indonesia 368.7 3.69 353.0 4.21 15.7 2.86

Latvia 493.7 5.22 479.7 3.91 14.0 4.19

Liechtenstein 557.0 7.94 518.4 7.08 38.5 12.13

Macao-China 539.8 5.13 516.6 4.30 23.3 6.76

Russian Federation 484.7 5.78 464.1 4.97 20.6 5.03

Serbia and Montenegro 434.1 4.28 430.8 4.93 3.3 4.88

Thailand 426.4 4.35 421.8 3.83 4.5 4.67

Tunisia 367.2 2.81 350.9 3.15 16.3 3.02

Uruguay 422.8 3.59 401.7 3.37 21.1 3.61

United Kingdom1 501.5 3.03 491.2 3.97 10.3 5.04

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

100

Table B-20. Average mathematics literacy subscale scores of 15-year-old 

students, by sex and country: 2003—Continued 

Country

Change and relationships

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 504.3 0.85 493.3 0.83 11.0 0.89

OECD countries

Australia 527.5 3.18 523.1 2.78 4.4 3.81

Austria 502.1 4.36 497.4 4.42 4.6 4.99

Belgium 538.9 3.55 531.3 3.50 7.6 5.08

Canada 545.8 2.20 532.4 2.01 13.5 2.29

Czech Republic 521.2 4.48 508.4 3.98 12.8 4.93

Denmark 519.9 3.67 499.1 3.25 20.8 3.50

Finland 548.8 2.80 537.4 2.39 11.4 2.82

France 522.0 4.02 517.6 3.18 4.4 4.97

Germany 513.5 4.28 501.8 4.37 11.8 4.43

Greece 444.8 5.18 427.0 4.42 17.8 4.18

Hungary 499.2 3.63 489.5 3.62 9.7 3.88

Iceland 504.8 2.43 514.4 2.30 -9.6 3.79

Ireland 512.2 3.05 499.6 3.52 12.6 4.44

Italy 462.9 4.85 442.1 3.99 20.8 6.27

Japan 539.4 6.41 533.1 4.26 6.3 6.56

Korea, Republic of 557.8 4.68 532.5 5.80 25.3 7.29

Luxembourg 494.0 2.51 480.2 1.76 13.8 3.67

Mexico 368.2 4.87 360.3 4.56 7.9 4.43

Netherlands 554.3 3.84 548.4 3.71 5.9 4.27

New Zealand 534.4 2.79 517.0 3.43 17.4 4.06

Norway 489.8 3.18 485.5 3.05 4.3 3.33

Poland 488.2 3.15 480.5 3.36 7.7 3.63

Portugal 474.7 4.82 461.6 4.00 13.1 3.80

Slovak Republic 502.4 4.09 486.0 3.90 16.4 4.16

Spain 485.0 3.81 476.6 2.58 8.4 3.25

Sweden 505.8 3.42 504.4 3.88 1.4 4.33

Switzerland 529.9 5.11 515.0 3.95 14.9 5.30

Turkey 425.5 9.07 419.5 7.43 6.0 7.25

United States 488.3 3.42 482.7 3.29 5.6 2.90

Non-OECD countries

Hong Kong-China 540.2 6.83 539.2 4.77 1.0 7.16

Indonesia 336.0 4.35 331.7 5.38 4.3 3.45

Latvia 486.7 5.27 487.7 4.31 -1.0 3.98

Liechtenstein 552.0 7.45 526.4 6.49 25.6 12.10

Macao-China 529.1 5.03 509.0 4.59 20.1 6.56

Russian Federation 478.5 6.02 475.0 4.47 3.4 5.05

Serbia and Montenegro 419.8 4.47 418.3 4.87 1.4 4.93

Thailand 399.7 4.52 409.3 3.95 -9.6 5.06

Tunisia 342.3 2.99 331.0 3.35 11.3 3.01

Uruguay 419.7 4.24 414.4 4.21 5.2 4.38

United Kingdom1 517.3 2.97 509.0 4.04 8.3 5.04

See notes at end of table.
101

PISA 2003 Results From the U.S. Perspective

Table B-20. Average mathematics literacy subscale scores of 15-year-old 

students, by sex and country: 2003—Continued 

Country

Quantity

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 503.8 0.77 497.6 0.77 6.2 0.84

OECD countries

Australia 517.5 2.85 516.3 2.65 1.2 3.67

Austria 514.7 3.68 511.6 3.66 3.1 4.19

Belgium 530.1 3.30 529.1 3.26 0.9 4.66

Canada 532.7 2.16 528.0 1.93 4.7 2.23

Czech Republic 530.8 4.16 525.1 4.55 5.8 5.11

Denmark 520.3 3.21 511.0 2.89 9.3 3.07

Finland 550.1 2.26 546.9 2.07 3.2 2.33

France 508.1 3.78 505.8 2.88 2.3 4.45

Germany 514.5 4.19 513.9 3.85 0.6 4.40

Greece 457.6 4.91 435.0 3.97 22.6 3.99

Hungary 497.2 3.27 495.3 3.24 1.9 3.61

Iceland 499.6 2.53 528.0 2.32 -28.5 3.89

Ireland 506.1 3.06 497.2 3.47 8.9 4.28

Italy 481.4 4.96 468.7 4.38 12.7 6.54

Japan 528.3 5.59 525.2 3.73 3.1 5.67

Korea, Republic of 546.1 3.96 524.2 4.91 21.9 6.15

Luxembourg 505.8 2.17 497.3 1.62 8.5 3.17

Mexico 400.0 4.81 388.0 4.31 12.0 4.46

Netherlands 526.3 4.18 530.3 3.57 -4.0 4.73

New Zealand 516.9 2.66 505.3 3.24 11.6 3.89

Norway 494.2 2.85 494.2 2.70 # †

Poland 492.6 2.94 491.0 2.98 1.6 3.27

Portugal 472.7 4.12 458.9 3.69 13.8 3.27

Slovak Republic 518.6 4.03 506.1 3.63 12.6 3.64

Spain 494.8 3.60 490.0 2.18 4.8 3.07

Sweden 515.2 2.92 511.9 3.24 3.2 3.63

Switzerland 536.0 4.41 529.0 3.23 7.0 4.60

Turkey 421.1 8.04 403.5 6.62 17.5 6.33

United States 478.5 3.63 474.3 3.57 4.2 3.38

Non-OECD countries

Hong Kong-China 543.9 6.05 546.5 4.12 -2.6 6.09

Indonesia 358.5 4.05 356.5 5.01 2.1 3.12

Latvia 483.2 4.42 480.3 3.55 2.9 3.43

Liechtenstein 543.9 7.00 522.6 5.65 21.4 9.89

Macao-China 541.6 4.28 524.9 4.18 16.7 6.00

Russian Federation 475.7 5.04 469.3 4.18 6.4 4.43

Serbia and Montenegro 454.8 4.22 457.9 4.67 -3.1 4.71

Thailand 412.3 4.15 416.9 3.84 -4.5 4.91

Tunisia 372.3 2.90 356.6 3.30 15.6 2.69

Uruguay 435.9 3.91 423.8 3.76 12.0 4.09

United Kingdom1 499.6 2.87 497.6 4.03 2.1 4.95

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

102

Table B-20. Average mathematics literacy subscale scores of 15-year-old students,

by sex and country: 2003—Continued 

Country

Uncertainty

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 508.3 0.71 495.6 0.75 12.6 0.78

OECD countries

Australia 534.5 3.03 527.2 2.74 7.3 3.72

Austria 497.7 3.81 489.9 3.99 7.8 4.58

Belgium 529.2 3.23 521.8 3.21 7.3 4.68

Canada 551.4 2.20 538.4 1.89 13.0 2.26

Czech Republic 508.5 3.88 491.8 3.77 16.7 4.63

Denmark 526.6 3.40 505.0 3.03 21.6 3.21

Finland 550.8 2.64 538.7 2.27 12.1 2.63

France 511.7 3.53 501.0 2.83 10.7 4.25

Germany 502.0 3.86 483.9 3.79 18.1 3.97

Greece 468.9 4.29 448.7 3.68 20.2 3.68

Hungary 492.8 3.16 484.8 3.00 7.9 3.34

Iceland 524.2 2.45 531.7 2.35 -7.5 3.76

Ireland 524.9 3.24 509.4 3.73 15.5 4.60

Italy 475.1 4.51 451.1 3.80 24.1 5.94

Japan 535.2 5.57 521.1 3.82 14.0 5.69

Korea, Republic of 547.1 4.12 525.4 5.18 21.7 6.62

Luxembourg 503.1 2.23 481.5 1.82 21.7 3.49

Mexico 392.1 3.79 387.7 3.63 4.5 3.51

Netherlands 554.0 3.56 544.5 3.69 9.5 4.10

New Zealand 538.0 2.73 526.5 3.31 11.5 3.93

Norway 517.9 2.97 507.6 3.16 10.3 3.32

Poland 494.8 2.82 492.3 2.84 2.6 3.15

Portugal 475.6 4.10 466.0 3.45 9.6 3.15

Slovak Republic 484.1 3.80 467.1 3.41 17.0 3.49

Spain 493.0 3.34 485.0 2.17 8.0 2.83

Sweden 515.2 3.18 506.4 3.38 8.8 3.66

Switzerland 526.4 4.73 505.9 3.68 20.5 5.16

Turkey 451.1 7.31 432.2 6.07 19.0 5.74

United States 493.0 3.41 489.8 3.15 3.2 2.79

Non-OECD countries

Hong Kong-China 564.1 6.57 552.4 4.58 11.8 6.66

Indonesia 382.1 2.81 386.9 3.38 -4.8 2.42

Latvia 473.7 4.23 473.9 3.08 -0.2 3.32

Liechtenstein 538.4 6.94 507.6 5.62 30.8 10.46

Macao-China 540.7 4.51 523.0 4.20 17.8 5.88

Russian Federation 440.7 5.13 432.3 3.94 8.4 4.19

Serbia and Montenegro 430.6 4.01 425.2 4.19 5.4 4.25

Thailand 419.9 3.44 424.9 3.01 -5.0 4.00

Tunisia 366.7 2.50 360.1 2.77 6.7 2.58

Uruguay 422.9 3.85 414.5 3.60 8.3 4.09

United Kingdom1 523.1 2.90 517.5 3.84 5.6 4.87

† Not applicable. 

# Rounds to zero.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The male-female score point difference is calculated by subtracting the average scores of females from

the average scores of males. The OECD average is the average of the national averages of the OECD member

countries with data available. Because PISA is principally an OECD study, the results for non-OECD countries

are displayed separately from those of the OECD countries and are not included in the OECD average. s.e.

means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student

Assessment (PISA), 2003. 
103

PISA 2003 Results From the U.S. Perspective

Table B-21. Average problem-solving scores of 15-year-old students, by sex

and country: 2003

Country

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 499.2 0.76 500.9 0.77 -1.7 0.82

OECD countries

Australia 526.7 2.68 533.1 2.48 -6.4 3.33

Austria 504.7 3.92 507.6 3.75 -2.9 4.34

Belgium 523.6 3.14 527.1 3.15 -3.5 4.52

Canada 532.7 2.01 532.2 1.82 0.5 2.06

Czech Republic 519.6 4.10 513.1 4.34 6.5 4.99

Denmark 519.3 3.08 514.4 2.93 4.9 3.20

Finland 542.6 2.54 552.6 2.25 -10.0 3.03

France 518.7 3.81 519.5 2.93 -0.8 4.14

Germany 511.0 3.87 516.7 3.67 -5.7 3.90

Greece 449.5 4.94 447.5 4.11 1.9 4.37

Hungary 499.3 3.43 503.0 3.39 -3.7 3.71

Iceland 490.0 2.25 520.4 2.49 -30.5 3.90

Ireland 498.7 2.78 498.2 3.46 0.5 4.20

Italy 467.4 5.04 471.4 3.54 -4.1 6.01

Japan 546.1 5.74 548.5 4.07 -2.4 5.66

Korea, Republic of 553.7 3.99 545.6 4.82 8.1 6.10

Luxembourg 494.9 2.41 492.5 1.85 2.4 3.32

Mexico 387.0 5.04 381.9 4.67 5.1 4.45

Netherlands 522.4 3.60 517.9 3.58 4.5 4.09

New Zealand 531.1 2.65 534.4 3.11 -3.3 3.82

Norway 485.6 3.08 494.1 3.24 -8.5 3.57

Poland 486.1 3.37 487.1 3.00 -1.1 3.13

Portugal 469.8 4.58 469.8 3.90 # †

Slovak Republic 495.1 4.07 488.2 3.56 6.9 3.66

Spain 479.2 3.64 485.2 2.59 -6.0 3.13

Sweden 503.6 2.96 513.5 2.83 -9.9 3.12

Switzerland 520.1 3.98 522.6 3.29 -2.5 4.11

Turkey 408.4 7.31 406.4 5.84 2.0 5.83

United States 476.9 3.44 477.8 3.50 -0.9 3.03

Non-OECD countries

Hong Kong-China 545.4 6.16 550.4 4.05 -5.1 6.27

Indonesia 357.7 3.15 365.0 4.03 -7.3 3.01

Latvia 481.2 5.07 483.7 3.98 -2.6 4.65

Liechtenstein 535.1 6.65 523.5 5.92 11.5 9.84

Macao-China 538.2 4.25 526.9 3.20 11.2 5.55

Russian Federation 479.7 5.92 477.4 4.37 2.3 4.87

Serbia and Montenegro 416.5 3.81 423.9 3.92 -7.4 4.07

Thailand 418.2 3.87 430.5 3.07 -12.4 4.33

Tunisia 346.1 2.50 343.4 2.45 2.7 2.57

Uruguay 412.1 4.59 409.3 4.21 2.7 4.77

United Kingdom1 505.7 2.97 514.1 3.50 -8.4 4.51

† Not applicable.

# Rounds to zero.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The male-female score point difference is calculated by subtracting the average scores of females

from the average scores of males. The OECD average is the average of the national averages of the OECD

member countries with data available. Because PISA is principally an OECD study, the results for nonOECD countries are displayed separately from those of the OECD countries and are not included in the

OECD average. s.e. means standard error. Detail may not sum to totals because of rounding.

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

104

Table B-22. Average combined reading literacy and science literacy scores of 

15-year-old students, by sex and country: 2000 and 2003

Country

Reading literacy in 2000

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 484.8 0.82 516.5 0.75 -31.7 0.94

OECD countries

Australia 512.7 4.04 546.3 4.74 -33.6 5.44

Austria 494.7 3.23 520.3 3.59 -25.6 5.24

Belgium 492.4 4.24 525.2 4.92 -32.8 5.99

Canada 518.9 1.76 551.1 1.70 -32.2 1.63

Czech Republic 472.6 4.11 510.1 2.53 -37.4 4.71

Denmark 485.4 2.95 510.3 2.87 -24.8 3.28

Finland 520.1 3.00 571.4 2.78 -51.3 2.63

France 490.3 3.50 519.1 2.72 -28.8 3.38

Germany 467.6 3.17 502.2 3.88 -34.7 5.21

Greece 455.7 6.07 492.7 4.63 -37.0 5.01

Hungary 464.5 5.34 496.2 4.35 -31.6 5.73

Iceland 488.5 2.12 528.1 2.14 -39.7 3.11

Ireland 512.8 4.18 541.5 3.55 -28.7 4.56

Italy 469.2 5.14 507.4 3.57 -38.2 7.05

Japan 507.3 6.74 536.9 5.39 -29.7 6.44

Korea, Republic of 518.5 3.77 532.7 3.70 -14.2 6.02

Luxembourg 428.8 2.58 455.7 2.30 -26.9 3.77

Mexico 411.5 4.18 431.8 3.84 -20.3 4.34

Netherlands — — — — — —

New Zealand 506.8 4.18 552.6 3.80 -45.8 6.28

Norway 485.6 3.79 528.8 2.86 -43.2 4.04

Poland 461.4 5.99 497.5 5.52 -36.1 6.97

Portugal 457.7 4.98 482.4 4.64 -24.7 3.77

Slovak Republic — — — — — —

Spain 481.2 3.35 505.4 2.76 -24.1 3.17

Sweden 498.6 2.56 535.6 2.48 -37.0 2.70

Switzerland 480.1 4.85 510.0 4.50 -30.0 4.17

Turkey — — — — — —

United States 489.7 8.41 518.2 6.20 -28.6 4.12

Non-OECD countries

Hong Kong-China — — — — — —

Indonesia — — — — — —

Latvia 431.9 5.53 484.7 5.40 -52.8 4.20

Liechtenstein 468.5 7.33 499.6 6.83 -31.2 11.54

Macao-China — — — — — —

Russian Federation 442.8 4.53 481.0 4.09 -38.2 2.92

Serbia and Montenegro — — — — — —

Thailand — — — — — —

Tunisia — — — — — —

Uruguay — — — — — —

United Kingdom1 511.6 3.03 537.2 3.40 -25.6 4.28

See notes at end of table.
105

PISA 2003 Results From the U.S. Perspective

Table B-22. Average combined reading literacy and science literacy scores of 

15-year-old students, by sex and country: 2000 and 2003—Continued

Country

Reading literacy in 2003

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 477.3 0.74 511.4 0.72 -34.1 0.78

OECD countries

Australia 506.1 2.84 545.4 2.55 -39.3 3.60

Austria 467.1 4.54 514.4 4.21 -47.2 5.23

Belgium 489.3 3.76 526.2 3.30 -36.9 5.10

Canada 514.0 2.05 545.5 1.82 -31.5 2.02

Czech Republic 473.1 4.11 504.4 4.42 -31.3 4.94

Denmark 479.4 3.31 504.8 2.98 -25.4 2.88

Finland 521.4 2.21 565.4 1.97 -44.0 2.66

France 476.1 3.79 514.3 3.18 -38.2 4.47

Germany 470.8 4.23 512.9 3.91 -42.1 4.62

Greece 452.9 5.10 490.4 3.96 -37.5 4.12

Hungary 467.2 3.16 498.2 3.05 -31.0 3.76

Iceland 463.8 2.33 521.6 2.22 -57.8 3.47

Ireland 501.1 3.26 530.1 3.71 -29.0 4.56

Italy 455.2 5.06 494.6 3.40 -39.3 6.03

Japan 486.6 5.48 509.0 4.07 -22.4 5.40

Korea, Republic of 525.5 3.69 546.7 4.27 -21.3 5.59

Luxembourg 462.7 2.60 495.7 1.84 -33.0 3.36

Mexico 388.6 4.56 410.1 4.57 -21.5 4.36

Netherlands 502.9 3.66 523.8 3.20 -20.9 3.93

New Zealand 507.7 3.13 535.4 3.29 -27.6 4.37

Norway 475.3 3.36 524.5 3.38 -49.2 3.73

Poland 476.8 3.55 516.3 3.19 -39.6 3.66

Portugal 458.5 4.28 494.9 3.72 -36.3 3.34

Slovak Republic 453.3 3.82 485.8 3.32 -32.5 3.50

Spain 460.7 3.78 499.8 2.48 -39.1 3.88

Sweden 495.9 2.81 532.7 2.89 -36.7 3.15

Switzerland 482.0 4.37 517.5 3.10 -35.5 4.69

Turkey 426.0 6.80 459.3 6.12 -33.3 5.83

United States 479.3 3.66 511.3 3.55 -32.0 3.35

Non-OECD countries

Hong Kong-China 493.8 5.31 525.4 3.49 -31.5 5.51

Indonesia 369.5 3.35 393.5 3.92 -24.0 2.78

Latvia 470.4 4.51 509.1 3.66 -38.7 4.25

Liechtenstein 516.6 7.24 534.0 6.54 -17.4 11.87

Macao-China 490.8 3.62 504.1 2.75 -13.3 4.77

Russian Federation 427.8 4.73 456.4 3.69 -28.5 3.88

Serbia and Montenegro 389.9 3.66 433.0 3.91 -43.1 3.93

Thailand 396.4 3.72 439.2 3.01 -42.7 4.05

Tunisia 361.8 3.31 387.1 3.27 -25.3 3.56

Uruguay 414.0 4.50 453.3 3.72 -39.3 4.71

United Kingdom1 491.8 3.07 520.4 3.62 -28.6 4.81

See notes at end of table.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

106

Table B-22. Average combined reading literacy and science literacy scores of 

15-year-old students, by sex and country: 2000 and 2003—Continued

Country

Science literacy in 2000

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 500.5 0.88 500.7 0.76 0.2 1.01

OECD countries

Australia 526.4 3.91 528.9 4.78 -2.5 5.29

Austria 525.7 3.79 513.9 4.27 11.8 6.25

Belgium 495.9 5.24 498.0 5.61 -2.1 6.68

Canada 529.1 1.91 531.0 1.74 -1.8 1.88

Czech Republic 511.9 3.83 511.4 3.20 0.5 5.07

Denmark 487.6 3.91 475.9 3.54 11.7 4.76

Finland 534.5 3.51 541.0 2.70 -6.5 3.83

France 504.1 4.24 498.1 3.77 6.0 4.80

Germany 489.2 3.38 486.7 3.43 2.5 4.73

Greece 457.0 6.08 464.4 5.16 -7.4 5.75

Hungary 495.7 5.79 497.3 5.02 -1.6 6.90

Iceland 494.8 3.44 499.5 3.01 -4.7 4.73

Ireland 510.7 4.23 516.9 4.17 -6.5 5.52

Italy 473.6 5.62 482.6 3.90 -9.0 7.73

Japan 546.7 7.18 554.1 5.89 -7.4 7.20

Korea, Republic of 560.7 4.34 541.3 5.13 19.4 7.64

Luxembourg 441.0 3.58 447.5 3.25 -6.5 4.99

Mexico 423.3 4.20 419.0 3.85 4.3 4.82

Netherlands — — — — — —

New Zealand 522.9 4.62 534.8 3.80 -11.9 7.00

Norway 498.8 4.07 505.4 3.25 -6.6 4.95

Poland 486.1 6.10 480.0 6.50 6.1 7.37

Portugal 456.2 4.81 462.5 4.24 -6.3 4.35

Slovak Republic — — — — — —

Spain 492.1 3.49 491.4 3.58 0.7 4.01

Sweden 512.2 3.49 512.6 2.87 -0.4 3.92

Switzerland 499.7 5.69 492.7 4.65 7.0 5.38

Turkey — — — — — —

United States 497.0 8.93 501.8 6.49 -4.9 5.31

Non-OECD countries

Hong Kong-China — — — — — —

Indonesia — — — — — —

Latvia 449.3 6.40 472.0 5.83 -22.7 5.36

Liechtenstein 484.1 10.93 468.4 9.29 15.7 14.65

Macao-China — — — — — —

Russian Federation 453.2 5.36 467.4 5.16 -14.2 4.50

Serbia and Montenegro — — — — — —

Thailand — — — — — —

Tunisia — — — — — —

Uruguay — — — — — —

United Kingdom1 535.0 3.44 531.4 3.98 3.6 5.20

See notes at end of table.
107

PISA 2003 Results From the U.S. Perspective

Table B-22. Average combined reading literacy and science literacy scores of 

15-year-old students, by sex and country: 2000 and 2003—Continued

Country

Science literacy in 2003

Malefemale

score point

difference s.e.

Male Female

Average s.e. Average s.e.

OECD average 502.5 0.74 496.8 0.75 5.8 0.90

OECD countries

Australia 524.9 2.91 525.2 2.78 -0.2 3.84

Austria 489.7 4.28 492.3 4.18 -2.6 4.95

Belgium 508.8 3.57 508.9 3.48 -0.1 5.03

Canada 527.4 2.30 516.3 2.22 11.0 2.58

Czech Republic 526.1 4.25 520.4 4.07 5.7 4.90

Denmark 484.0 3.59 466.8 3.16 17.2 3.18

Finland 545.2 2.55 551.2 2.17 -6.1 2.77

France 511.2 4.14 511.3 3.54 -0.1 4.82

Germany 505.7 4.47 500.0 4.19 5.7 4.77

Greece 487.4 4.78 475.1 3.93 12.3 4.20

Hungary 502.7 3.31 504.0 3.35 -1.3 3.68

Iceland 489.8 2.37 500.0 2.42 -10.2 3.79

Ireland 506.4 3.08 504.4 3.88 2.0 4.48

Italy 489.5 5.16 483.6 3.63 5.9 6.31

Japan 550.0 6.00 545.6 4.10 4.4 6.01

Korea, Republic of 545.9 4.73 527.5 5.51 18.4 7.05

Luxembourg 489.2 2.53 476.5 1.88 12.7 3.31

Mexico 409.7 3.87 400.4 4.23 9.3 4.06

Netherlands 526.9 4.16 521.8 3.65 5.1 4.66

New Zealand 528.8 2.96 513.0 3.36 15.8 4.21

Norway 485.0 3.45 483.3 3.33 1.7 3.61

Poland 501.2 3.24 494.4 3.37 6.8 3.30

Portugal 471.0 4.01 464.8 3.64 6.3 3.18

Slovak Republic 502.1 4.28 487.3 3.92 14.8 3.67

Spain 489.1 3.85 485.1 2.59 4.0 3.92

Sweden 508.6 3.07 503.7 3.46 4.9 3.61

Switzerland 518.0 4.99 507.6 3.94 10.3 5.01

Turkey 434.4 6.66 434.0 6.42 0.5 5.79

United States 493.7 3.45 488.7 3.52 5.0 3.27

Non-OECD countries

Hong Kong-China 537.8 6.05 541.2 4.19 -3.4 6.00

Indonesia 395.6 3.07 394.5 3.84 1.1 2.68

Latvia 487.2 5.11 490.9 3.90 -3.8 4.68

Liechtenstein 537.8 7.72 511.9 7.32 26.0 12.48

Macao-China 529.0 5.02 520.6 4.04 8.3 6.78

Russian Federation 493.9 5.35 484.7 3.97 9.2 4.27

Serbia and Montenegro 433.7 3.69 438.9 4.22 -5.2 3.82

Thailand 424.7 3.73 432.7 3.13 -8.0 4.24

Tunisia 379.7 2.75 389.5 2.99 -9.9 2.64

Uruguay 440.5 3.71 436.3 3.59 4.2 4.42

United Kingdom1 520.2 3.14 516.8 3.98 3.4 5.16

—Not available.

1

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The male-female score point difference is calculated by subtracting the average scores of females

from the average scores of males. The OECD average is the average of the national averages of the OECD

member countries with data available. Because PISA is principally an OECD study, the results for nonOECD countries are displayed separately from those of the OECD countries and are not included in the

OECD average. s.e. means standard error. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2000 and 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

108

Table B-23. Mean International Socioeconomic Index (ISEI) score of 15-year-old students,

by quarters of the ISEI index and country: 2003

Country

Mean Bottom quarter Second quarter Third quarter Top quarter

Index

score s.e.

Index

score s.e.

Index

score s.e.

Index

score s.e.

Index

score s.e.

OECD average 48.8 0.08 28.2 0.04 42.3 0.08 53.2 0.09 71.2 0.13

OECD countries

Australia 52.6 0.30 31.6 0.14 48.0 0.07 58.3 0.11 72.5 0.14

Austria 47.1 0.52 27.3 0.19 40.9 0.11 51.4 0.12 68.7 0.28

Belgium 50.6 0.38 29.0 0.13 44.5 0.13 56.4 0.13 72.4 0.16

Canada 52.6 0.27 31.7 0.11 47.7 0.08 58.1 0.09 72.9 0.15

Czech Republic 50.1 0.34 32.3 0.18 45.7 0.12 52.5 0.05 69.7 0.23

Denmark 49.3 0.45 29.4 0.19 44.2 0.11 53.2 0.07 70.3 0.29

Finland 50.2 0.36 28.7 0.12 43.4 0.16 56.4 0.14 72.4 0.18

France 48.7 0.47 27.6 0.20 42.3 0.15 53.6 0.05 71.2 0.26

Germany 49.3 0.42 29.5 0.17 42.6 0.14 53.7 0.06 71.5 0.25

Greece 46.9 0.72 26.9 0.13 38.8 0.13 51.8 0.07 70.3 0.39

Hungary 48.6 0.33 30.2 0.18 42.3 0.08 51.6 0.11 70.2 0.20

Iceland 53.7 0.26 31.5 0.20 48.0 0.13 61.7 0.19 73.7 0.25

Ireland 48.3 0.49 28.5 0.17 42.2 0.11 52.7 0.08 70.0 0.29

Italy 46.8 0.38 26.9 0.16 40.3 0.11 50.6 0.05 69.5 0.38

Japan 50.0 0.31 33.4 0.17 43.9 0.04 50.6 0.08 72.0 0.25

Korea, Republic of 46.3 0.36 28.9 0.20 43.5 0.09 49.4 0.06 63.5 0.43

Luxembourg 48.2 0.22 27.3 0.15 42.1 0.13 52.8 0.06 70.5 0.24

Mexico 40.1 0.68 22.2 0.12 28.9 0.04 42.1 0.28 67.3 0.25

Netherlands 51.3 0.38 30.9 0.26 45.4 0.15 56.9 0.20 71.8 0.25

New Zealand1 51.5 0.36 30.1 0.19 46.2 0.12 56.8 0.17 72.7 0.26

Norway 54.6 0.39 35.0 0.20 49.0 0.12 60.6 0.16 73.9 0.21

Poland 45.0 0.34 26.9 0.21 39.5 0.11 49.1 0.10 64.4 0.34

Portugal 43.1 0.54 26.4 0.14 33.9 0.08 46.6 0.19 65.5 0.53

Slovak Republic 48.8 0.40 29.3 0.17 41.4 0.09 53.1 0.10 71.5 0.21

Spain 44.3 0.58 26.2 0.13 35.5 0.14 49.3 0.11 66.1 0.39

Sweden 50.6 0.38 30.4 0.18 44.1 0.14 56.1 0.17 71.9 0.21

Switzerland 49.3 0.43 29.4 0.14 43.1 0.14 53.5 0.08 71.1 0.27

Turkey 41.6 0.75 23.7 0.29 33.6 0.15 47.2 0.10 61.8 0.77

United States 54.6 0.37 32.6 0.21 49.9 0.15 61.4 0.12 74.3 0.21

Non-OECD countries

Hong Kong-China 41.1 0.45 25.9 0.14 34.9 0.07 45.1 0.13 58.7 0.37

Indonesia 33.6 0.61 16.0 0.00 24.1 0.15 34.6 0.33 59.9 0.42

Latvia 50.3 0.52 29.1 0.23 44.2 0.16 54.8 0.14 73.0 0.30

Liechtenstein 50.7 0.75 30.8 0.63 47.4 0.52 55.0 0.09 70.0 0.67

Macao-China 39.4 0.40 25.8 0.32 34.4 0.12 41.7 0.25 55.9 0.52

Russian Federation 49.9 0.38 30.8 0.16 40.8 0.10 54.2 0.21 73.6 0.20

Serbia and Montenegro 48.1 0.53 28.3 0.20 41.2 0.12 51.4 0.11 71.4 0.38

Thailand 36.0 0.43 22.1 0.14 26.7 0.13 35.6 0.13 59.6 0.41

Tunisia 37.5 0.60 18.0 0.17 29.2 0.18 39.6 0.19 63.1 0.44

Uruguay 46.1 0.48 25.2 0.16 37.8 0.15 50.8 0.12 70.8 0.36

United Kingdom2 49.6 0.39 28.5 0.14 43.0 0.14 53.2 0.09 71.6 0.19

1

The item response rate for ISEI for New Zealand is below 85 percent. Missing data have not been explicitly accounted for. See

also table A-2.

2

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data available. Because

PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD

countries and are not included in the OECD average. The International Socioeconomic Index (ISEI) is an internationally comparable index of occupational status, with a range of approximately 16 to 90, developed by Ganzeboom, De Graaf, and Treiman

(1992). s.e. means standard error. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003. 
109

PISA 2003 Results From the U.S. Perspective

Table B-24. Average combined mathematics literacy scores of 15-year-old students, by

quarters of the International Socioeconomic Index (ISEI) and country: 2003

Country

Mathematics literacy

Bottom quarter Second quarter Third quarter Top quarter

Average s.e. Average s.e. Average s.e. Average s.e.

OECD average 455.5 0.92 493.2 0.75 516.1 0.73 547.7 0.84

OECD countries

Australia 488.5 2.81 520.2 2.74 539.3 2.68 565.6 2.87

Austria 466.7 4.40 492.3 3.65 523.9 3.26 547.7 4.44

Belgium 482.1 3.69 527.3 3.16 555.5 2.82 589.8 3.26

Canada 506.1 1.96 530.9 2.31 544.1 2.11 568.7 2.79

Czech Republic 485.6 4.01 507.7 3.85 530.0 3.89 569.7 4.25

Denmark 480.9 3.42 504.0 3.64 525.5 3.94 554.2 3.53

Finland 515.1 2.66 536.3 2.73 551.8 2.92 576.5 2.89

France 469.4 3.72 507.2 4.22 524.8 2.97 556.6 3.76

Germany 462.7 4.93 505.1 3.32 528.1 3.83 564.6 3.98

Greece 409.3 4.28 435.5 3.75 450.4 4.46 492.9 5.00

Hungary 449.5 3.93 472.8 3.87 503.0 3.36 547.5 3.92

Iceland 497.1 3.11 512.4 3.19 518.6 3.06 537.7 3.11

Ireland 470.8 3.93 495.9 3.22 512.9 3.05 540.9 3.50

Italy 430.5 4.19 456.8 3.93 478.2 3.62 502.2 4.11

Japan 505.1 5.09 534.4 4.75 543.0 4.40 567.6 6.36

Korea, Republic of 511.3 4.44 547.4 3.67 549.0 3.58 567.7 6.09

Luxembourg 448.4 2.96 481.1 2.96 508.8 2.55 542.4 3.14

Mexico 357.4 4.81 373.6 3.87 394.3 3.66 424.1 4.87

Netherlands 501.5 4.33 534.8 3.79 558.6 3.48 584.2 3.90

New Zealand1 484.8 3.80 514.5 3.38 532.4 3.30 564.3 3.42

Norway 460.8 3.49 488.5 3.64 506.7 3.45 532.6 3.52

Poland 455.0 3.94 479.1 3.22 497.7 3.33 534.4 3.14

Portugal 431.3 5.28 447.1 3.38 481.0 3.82 511.5 3.83

Slovak Republic 456.5 4.17 484.2 3.35 523.0 3.54 544.3 3.84

Spain 454.0 3.62 475.1 2.79 495.9 3.18 519.4 3.32

Sweden 477.1 3.69 501.3 3.13 517.8 3.88 550.7 4.15

Switzerland 487.1 4.05 523.9 4.12 537.8 4.86 568.3 3.89

Turkey 395.3 5.65 410.9 6.74 419.6 7.48 478.8 12.51

United States 447.7 3.16 476.9 3.76 496.7 4.03 529.7 3.68

Non-OECD countries

Hong Kong-China 532.1 5.50 547.4 5.11 562.0 4.15 574.7 5.57

Indonesia 335.5 4.30 355.6 4.14 360.9 4.48 397.1 6.28

Latvia 456.6 3.80 474.7 4.31 493.5 4.55 514.0 5.03

Liechtenstein 482.5 10.29 530.3 11.22 552.9 9.56 587.5 10.97

Macao-China 521.8 5.22 523.3 6.33 527.6 7.45 540.5 7.32

Russian Federation 443.1 4.48 459.3 5.27 472.9 4.89 501.3 4.75

Serbia and Montenegro 406.0 3.72 426.1 3.81 448.6 4.29 474.8 4.99

Thailand 396.0 3.60 398.9 3.45 426.5 4.04 456.6 5.17

Tunisia 330.8 3.03 341.9 3.97 360.8 3.84 405.6 6.13

Uruguay 388.2 4.77 415.1 3.98 429.8 4.17 477.6 3.83

United Kingdom2 469.1 2.92 500.0 3.06 519.1 3.53 555.1 3.43

1

The item response rate for ISEI for New Zealand is below 85 percent. Missing data have not been explicitly accounted for. See

also table A-2.

2

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data available. Because

PISA is principally an OECD study, the results for non-OECD countries are displayed separately from those of the OECD

countries and are not included in the OECD average.The International Socioeconomic Index (ISEI) is an internationally comparable index of occupational status developed by Ganzeboom, De Graaf, and Treiman (1992). s.e. means standard error. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003. 
International Outcomes of Learning in Mathematics Literacy and Problem Solving

110

Table B-25. Change in the combined mathematics literacy and problem-solving

scores per one standard deviation change in the International

Socioeconomic Index (ISEI) score, by country: 2003

Country Mathematics literacy Problem solving

Change s.e. Change s.e.

OECD average 33.7 0.40 — —

OECD countries

Australia 30.1 1.35 27.6 1.36

Austria 30.7 1.92 30.2 1.98

Belgium 39.8 1.71 37.0 1.73

Canada 24.4 1.17 21.8 1.19

Czech Republic 37.5 1.97 33.8 2.12

Denmark 28.9 1.71 25.4 1.73

Finland 21.7 1.29 20.0 1.48

France 31.6 1.93 32.9 2.22

Germany 38.0 1.95 36.5 1.85

Greece 29.4 2.11 31.1 2.10

Hungary 40.8 2.17 41.7 2.34

Iceland 14.4 1.51 14.3 1.53

Ireland 27.4 1.89 28.1 1.85

Italy 27.1 1.88 28.4 2.08

Japan 23.0 3.12 21.0 3.01

Korea, Republic of 26.4 3.28 23.0 3.06

Luxembourg 33.7 1.56 33.3 1.75

Mexico 23.5 1.88 28.9 2.40

Netherlands 32.3 2.03 32.9 2.00

New Zealand1 29.4 1.65 30.6 1.88

Norway 29.2 1.62 31.2 1.95

Poland 35.2 1.82 33.8 1.93

Portugal 34.3 1.70 35.4 2.20

Slovak Republic 33.2 1.83 34.4 2.08

Spain 25.4 1.43 28.7 1.58

Sweden 28.7 1.79 26.9 1.73

Switzerland 30.3 1.71 29.4 1.75

Turkey 38.1 5.87 34.9 5.33

United States 30.2 1.37 31.1 1.58

Non-OECD countries

Hong Kong-China 22.6 2.64 24.9 2.60

Indonesia 22.0 2.35 22.2 2.02

Latvia 21.0 1.69 19.8 2.17

Liechtenstein 41.2 5.92 40.3 5.44

Macao-China 11.7 3.97 10.6 4.33

Russian Federation 21.4 1.77 24.0 2.07

Serbia and Montenegro 26.0 1.86 24.9 1.75

Thailand 26.6 2.35 26.0 2.32

Tunisia 28.3 2.56 27.2 2.08

Uruguay 31.4 1.83 35.6 2.37

United Kingdom2 31.8 1.46 29.7 1.63

—Not available.

1

The item response rate for ISEI for New Zealand is below 85 percent. Missing data have not been explicitly

accounted for. See also table A-2.

2

Due to low response rates, 2003 data for the United Kingdom are not discussed in this report.

NOTE: The OECD average is the average of the national averages of the OECD member countries with data

available. Because PISA is principally an OECD study, the results for non-OECD countries are displayed

separately from those of the OECD countries and are not included in the OECD average.The International

Socioeconomic Index (ISEI) is an internationally comparable index of occupational status, with a range of

approximately 16 to 90, developed by Ganzeboom, De Graaf, and Treiman (1992). The overall linkage of ISEI

to mathematics literacy and problem solving is examined using the specific change in score on the combined mathematics literacy scale or problem solving in response to a one standard deviation change in the

ISEI index score for each country. A greater increase in achievement score in a country implies a stronger

relationship between socioeconomic status and performance in that country. s.e. means standard error. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International

Student Assessment (PISA), 2003. 
111

PISA 2003 Results From the U.S. Perspective

Table B-26. Average combined mathematics literacy, problem-solving, reading literacy, and 

science literacy scores of U.S. 15-year-old students, by race/ethnicity: 2003

Race/ethnicity

Mathematics

literacy

Problem 

solving

Reading 

literacy

Science 

literacy

Average s.e. Average s.e. Average s.e. Average s.e.

Total 482.9 2.95 477.4 3.13 495.2 3.22 491.3 3.08

White 511.6 2.51 505.7 2.54 524.8 2.57 521.6 2.60

Black 417.3 5.08 413.2 5.69 429.9 5.62 422.7 4.69

Hispanic 442.7 5.13 435.6 5.54 452.6 5.86 448.1 5.63

Asian 506.3 9.79 505.3 9.94 513.1 9.22 508.9 10.59

More than one race 502.2 6.36 497.5 7.05 515.2 7.35 517.0 7.21

OECD average 500.0 0.63 500.0 0.65 494.2 0.65 499.6 0.60

NOTE: Reporting standards not met for American Indian/Alaska Native and Native Hawaiian/Other Pacific Islander; thus, they are

included in the total, but not reported separately. Black includes African American and Hispanic includes Latino. Racial categories

exclude Hispanic origin. s.e. means standard error. 

SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment

(PISA), 2003.
International Outcomes of Learning in Mathematics Literacy and Problem Solving

112

Appendix C: 

TIMSS-PISA 2003

Expert Panelists
113

PISA 2003 Results From the U.S. Perspective

Ramesh Gangolli

Professor, Department of Mathematics

University of Washington

Seattle, WA

Patricia Harvey

Superintendent

Saint Paul Public Schools

Saint Paul, MN

Kati Haycock

Director

The Education Trust

Washington, DC

S. Anders Hedberg

Director

Bristol-Myers Squibb Foundation

Princeton, NJ

Jack Jennings

Director

Center on Education Policy

Washington, DC

Rolf Blank

Director of Education Indicator Programs

Council of Chief State School Officers

Washington, DC

Betsy Brand

Co-Director

American Youth Policy Forum

Washington, DC

Nancy R. Bunt, Ed.D.

Program Director

Math & Science Collaborative

Allegheny Intermediate Unit

Homestead, PA

Rodger Bybee

Executive Director

Biological Sciences Curriculum Study

Colorado Springs, CO

Joan Ferrini-Mundy

Professor, Department of Mathematics

Michigan State University

East Lansing, MI
International Outcomes of Learning in Mathematics Literacy and Problem Solving

114

Appendix D:

PISA Online Resources

and Publications
115

PISA 2003 Results From the U.S. Perspective

International Publications

The following publications are intended to

serve as examples of some of the numerous

reports that have been produced in relation

to PISA by the OECD and other international organizations. All of the publications listed here are available at

http://www.pisa.oecd.org.

Summary and Achievement Reports

Organization for Economic Cooperation and

Development (OECD). (2001). Knowledge and

Skills for Life: First Results from the OECD

Programme for International Student

Assessment. Paris: Author.

Thematic Reports

Artelt, C., Baumert, J., Julius-McElvany, N.

and Peschar, J. (2003) Learners for Life:

Student Approaches to Learning. Results from

PISA 2000. Paris: OECD.

Kirsch, I., de Jong, J., Lafontaine, D.,

McQueen, J., Mendelovits, J., and Monseur,

C. (2002). Reading for Change: Performance

and Engagement Across Countries. Results

from PISA 2000. Paris: OECD.

Willms, J.D. (2003). Student Engagement in

School: A Sense of Belonging and

Participation. Results from PISA 2000. Paris:

OECD.

Technical Reports and Frameworks

Adams, R (Ed.) (2003). PISA 2000 Technical

Report. Paris: Organization for Economic

Cooperation and Development (OECD).

Organization for Economic Cooperation and

Development (OECD). (2000). Measuring

Student Knowledge and Skills: The PISA 2000

Assessment of Reading, Mathematical and

Scientific Literacy. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (1999). Measuring

Student Knowledge and Skills: A New

Framework for Assessment. Paris: Author.

Organization for Economic Cooperation and

Development (OECD). (2002). Programme for

International Student Assessment (PISA):

Manual for the PISA 2000 Database. Paris:

Author.

Online Resources

The PISA NCES website

(http://nces.ed.gov/surveys/pisa) provides

background information on the PISA surveys, copies of NCES publications that

relate to PISA, and sample PISA items from

previous assessments. 

NCES Publications

The following publications are intended to

serve as examples of some of the numerous

reports that have been produced in relation

to PISA by NCES. All of the publications

listed here are available at

http://nces.ed.gov/surveys/pisa.

Summary Reports

Lemke, M., Calsyn, C., Lippman, L., Jocelyn,

L., Kastberg, D., Liu, Y., Roey, S., Williams, T.,

Kruger, T., and Bairu, G. (2001). Highlights

from the 2000 Program for International

Student Assessment (NCES 2002–116). U.S.

Department of Education, National Center

for Education Statistics. Washington, DC:

U.S. Government Printing Office.

Lemke, M., Calsyn, C., Lippman, L., Jocelyn,

L., Kastberg, D., Liu, Y.Y., Roey, S., Williams,

T., Kruger, T., Bairu, G. (2001). Outcomes of

Learning: Results from the 2000 Program for

International Student Assessment of 15-YearOlds in Reading, Mathematics, and Science

Literacy (NCES 2002–115). U.S. Department

of Education, National Center for Education

Statistics. Washington, DC: U.S.

Government Printing Office.

Thematic Reports

Lemke, M., Sen, A., Pahlke, E., Williams, T.,

Kastberg, D., and Jocelyn, L. (forthcoming).

Characteristics of U.S. 15-Year-Old Low

Achievers in an International Context:

Findings from PISA 2000 (NCES 2002–005).

U.S. Department of Education, National

Center for Education Statistics. Washington,

DC: U.S. Government Printing Office.

Data Products

U.S. Department of Education, National

Center for Education Statistics. (2004).

Program for International Student

Assessment (PISA) 2000 Data File (NCES

2004–006). Washington, DC: Author.