U.S. Department of Education 

Institute of Education Sciences 

NCES 2006–026 

 

 March 2006 

Comparing Science 

Content in the 

National Assessment 

of Educational 

Progress (NAEP) 

2000 and Trends in 

International 

Mathematics and 

Science Study 

(TIMSS) 2003 

Assessments 

Technical Report 

 Teresa Smith Neidorf

 NAEP Education Statistics Services Institute 

 American Institutes for Research 

 Marilyn Binkley 

 National Center for Education Statistics 

 Maria Stephens 

 American Institutes for Research 
U.S. Department of Education 

Margaret Spellings 

Secretary 

Institute of Education Sciences 

Grover J. Whitehurst 

Director 

National Center for Education Statistics 

Mark Schneider 

Commissioner 

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data

related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and 

report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized 

analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical 

systems; and review and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate 

indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department of Education, the 

Congress, the states, other education policymakers, practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. 

You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or 

suggestions about this or any other NCES product or report, we would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 

Institute of Education Sciences 

U.S. Department of Education 

1990 K Street NW 

Washington, DC 20006 

March 2006 

The NCES World Wide Web Home Page is: http://nces.ed.gov

The NCES World Wide Electronic Catalog is: http://nces.ed.gov/pubsearch/

Suggested Citation 

Neidorf, T.S., Binkley, M., and Stephens, M. (forthcoming). Comparing Science Content in the National Assessment of 

Educational Progress (NAEP) 2000 and Trends in International Mathematics and Science Study (TIMSS) 2003 Assessments

(NCES 2006–026). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved [date] 

from http://nces.ed.gov/pubsearch. 

For ordering information on this report, write to 

U.S. Department of Education 

ED Pubs 

P.O. Box 1398 

Jessup, MD 20794-1398 

or call toll free 1-877-4ED-PUBS or order online at http://www.edpubs.org 

Content Contacts: 

Marilyn Binkley Elois Scott 

(202) 502-7484 (202) 502-7489 

Marilyn.Binkley@ed.gov Elois.Scott@ed.gov
iii

Executive Summary 

The National Center for Education Statistics (NCES) collects information on student 

performance in key subject areas through the National Assessment of Educational Progress (NAEP), 

as well as through participation and collaboration in international studies. Information from these 

studies is used to inform policymakers, educators, researchers, and the public about the knowledge 

and skills of U.S. students and how these compare with students in other countries. 

This technical report describes a study that was undertaken to compare the content of two 

recent fourth- and eighth-grade assessments in science: the NAEP 2000 assessment and the Trends in 

International Mathematics and Science Study (TIMSS) 2003 assessment.1

 Its aim is to provide 

information that will be useful for interpreting and comparing the results from the two assessments, 

based on an in-depth look at the content of the respective frameworks2

 and assessment items. 

The report draws upon information provided by the developers of the assessments, as well as 

data obtained from an expert panel convened to compare the frameworks and items from the two 

assessments on various dimensions.3

 The frameworks were compared with respect to 

• how each assessment organizes and defines the science content, cognitive skills, and science 

processes to be assessed at each grade level; 

• the main content areas included and the set of topics covered in each; 

• how each assessment addresses scientific investigation or inquiry; and 

• the types and distribution of item formats used. 

Item comparisons were based on 

• cross-classification of NAEP and TIMSS items to each other’s assessment framework in 

terms of the science content covered and grade-level expectations; 

• classification of all items by cognitive domain (factual knowledge, conceptual understanding, 

or reasoning and analysis) as defined in the TIMSS 2003 framework; 

• classification of all items with respect to whether they assess scientific inquiry skills and 

understandings;4

 and 

• comparisons of the types of item formats and tasks included. 

 

1

 At the time the study was conducted, NAEP 2000 was the most recent science assessment that could be compared 

with TIMSS 2003. Subsequently, NAEP has conducted a science assessment in 2005, with results to be released in 

2006. 

2

 Assessment frameworks define what will be assessed, including the content to be covered, the types of test 

questions, and recommendations for how the test is administered. 

3

 The panel—composed of experts in science, science education, and science assessment, with familiarity and 

experience with the NAEP and TIMSS assessments—is listed in appendix C. 

4 Classifications of scientific inquiry were based on the definitions in the TIMSS 2003 framework. 
iv

Example items are referenced throughout the report to illustrate findings from some of these 

comparisons.5

 

The results of this study reveal that while there is considerable overlap in the content 

included in the NAEP and TIMSS science assessments at fourth and eighth grade, there are some key 

differences between the two assessments both at the framework level and in the assessment items 

developed based on these frameworks. Differences exist in how each assessment defines the main 

science content areas, the specific topics and objectives to be assessed at each grade level, the 

distribution of items with respect to cognitive categories, the relative emphasis on scientific 

investigation or inquiry, and the type and distribution of different item formats. Some of these 

differences reflect the different purposes of the two assessments, with NAEP developed in the 

context of the U.S. system and TIMSS based on a consensus across many countries with different 

science curricula. 

There are some similarities and differences between NAEP and TIMSS at the framework 

level, with NAEP defining science content in three broad fields of science (physical science, life 

science, and Earth science), and TIMSS including five content domains (physics, chemistry, life 

science, Earth science, and environmental science). Both NAEP and TIMSS include content areas 

related to life science and Earth science, which appear to be defined similarly based on the topic 

areas included in the framework. The two frameworks differ, however, in how the physical sciences 

are organized, with TIMSS having separate content domains for chemistry and physics. TIMSS also 

includes a separate content domain for environmental science, which includes topics related to 

environmental and resource issues that go across the fields of science in NAEP. The inclusion in 

TIMSS of separate content areas in chemistry, physics, and environmental science results in broader 

topic coverage in some areas. The differences at the framework level are translated into different 

emphases in the pool of items included in each assessment, even in the content areas of life science 

and Earth science where there is considerable overlap of the topic areas in the frameworks. 

TIMSS reports on each of the five content domains separately at the eighth grade, while at 

fourth grade only three content areas are reported—physical science, life science, and Earth science. 

At the fourth grade, TIMSS items covering topics in physics and chemistry are combined in the 

physical science reporting category, and a few items covering environmental science topics are 

reported on either the life science or Earth science subscales. Similar topics are also included in the 

NAEP framework in the areas of life science and Earth science. Thus, there appears to be greater 

correspondence between NAEP and TIMSS at the fourth-grade level, at least in terms of broad 

content areas. 

Comparing the entire set of items in the assessments overall, there are somewhat different 

emphases placed on each of the broad science content areas defined by either framework. While 

NAEP has a more even distribution of items across the three main content areas of physical science, 

life science, and Earth science, it does include a slightly greater proportion of items devoted to Earth 

science than TIMSS at either grade level. TIMSS, on the other hand, places more emphasis on life 

science at the fourth grade and on physical science at the eighth grade. As only TIMSS includes 

environmental science as a separate content area, there are more items in TIMSS than in NAEP that 

were classified in this area of the TIMSS framework, particularly at the eighth grade. 

 

5

 Additional released item sets from each assessment are available on the NAEP and TIMSS websites: 

http://nces.ed.gov/nationsreportcard and http://isc.bc.edu/timss2003. 
v

Even greater differences emerge with more detailed comparisons of the topics and specific 

objectives covered and the grade-level correspondence between items in each assessment and the 

intentions of the other assessment framework. In general, NAEP items were more frequently 

classified to the TIMSS framework with a higher degree of content match (e.g., to particular topics or 

grade-specific objectives) than TIMSS items to the NAEP framework. More than 90 percent of 

NAEP items (97 percent at grade 4 and 93 percent at grade 8) compared to 83 percent of TIMSS 

items at grade 4 and 82 percent at grade 8 were classified to one of the broad topics in the other 

assessment framework. The difference was even greater when considering items mapped to specific 

objectives, with more than 80 percent of NAEP items at either grade (83 percent at grade 4 and 82 

percent at grade 8) compared to 67 percent of fourth-grade and 60 percent of eighth-grade items in 

TIMSS. While the majority of items in both NAEP and TIMSS were classified to topics and 

objectives in the other assessment framework, a substantial number were not, indicating that both 

NAEP and TIMSS contain items that might not be included in the other assessment at any grade 

level. Also, of the items that were mapped to topics and objectives in the other assessment 

framework, many were at a different grade level or in a different content area. 

The grade-level correspondence between the NAEP and TIMSS assessments overall is quite 

different when comparing the fourth- and eighth-grade items. At the fourth grade, 88 percent of 

TIMSS items were classified at the corresponding grade level of the NAEP framework. In contrast, 

61 percent of NAEP items were classified at the fourth-grade level and 35 percent were classified at 

the eighth-grade level according to the TIMSS framework. The correspondence between the eighthgrade assessments is greater, with more than 80 percent of items in both NAEP and TIMSS classified 

at the eighth-grade level of the other assessment framework. Still, there were 10 percent of eighthgrade NAEP items classified at the fourth-grade level in TIMSS and 18 percent of grade 8 TIMSS 

items classified at either the fourth-grade or twelfth-grade level of the NAEP framework. 

There are also notable differences across the content areas in the level of grade match 

between the two assessments. NAEP fourth-grade items classified at the eighth-grade level in 

TIMSS go across all of the main content areas. TIMSS items classified as a higher grade according 

to the NAEP framework—either fourth-grade items classified at the eighth grade or eighth-grade 

items classified at the twelfth grade—come primarily from Earth science and environmental science, 

with some items from life science. There are no physical science items in TIMSS that were 

classified at a higher grade level in the NAEP framework. 

Within each content area, detailed comparisons of content coverage and grade 

correspondence reveal some key differences. These include the following: 

• Physical science: TIMSS includes a number of items at both fourth and eighth grade that do 

not correspond well to explicit topics in the NAEP framework, although the NAEP 

assessment does include some items matching these TIMSS topic areas (e.g., chemistry items 

related to oxidation, burning, and chemical change, and physics items related to heat 

conductivity and electrical circuits). NAEP and TIMSS both include items at the eighth 

grade that are somewhat beyond the eighth-grade specifications of the other framework (e.g., 

TIMSS chemistry items requiring knowledge of subatomic particles and NAEP physics items 

involving acceleration and momentum). TIMSS also addresses chemistry topics not included 

in NAEP at the eighth grade (e.g., acids and bases). 
vi

• Life science: TIMSS includes human health, a topic not included in the NAEP framework, 

although there are some NAEP items related to this TIMSS topic. NAEP places a greater 

emphasis on ecology and ecosystems in life science than TIMSS at both grade levels, 

although TIMSS includes additional items measuring topics related to ecosystems in its 

environmental science category. Also, the assessments differ with respect to what is 

considered fourth- and eighth-grade content in this area. Both assessments include cells and 

their functions at the eighth-grade level, but the topics included in this area are defined 

differently in TIMSS and NAEP. 

• Earth science: There is a lower level of grade correspondence between NAEP and TIMSS in 

Earth science than in the physical and life sciences. About one-quarter of TIMSS fourthgrade items and almost one-half of TIMSS eighth-grade items were classified at a different 

grade level. Among the TIMSS eighth-grade items, 28 percent were classified at fourth 

grade and 21 percent were classified at twelfth grade in the NAEP framework. Based on the 

TIMSS framework, NAEP and TIMSS generally emphasize different aspects of Earth science 

at the two grade levels, with NAEP having a greater focus on Earth in the solar system and 

the universe and TIMSS on Earth’s structure and physical features at the fourth grade. This 

situation is reversed at the eighth grade. 

• Environmental science: While only TIMSS includes this area as a separate content domain, 

more than 70 percent of TIMSS environmental science items were classified to topics in the 

NAEP framework across the fields of science, but primarily in Earth science. Still, a number 

of TIMSS items in this content domain (29 percent at grade 4 and 30 percent at grade 8) were 

classified at a different grade level in the NAEP framework. Also, several items (29 percent 

at grade 4 and 22 percent at grade 8) were found not to match any of the NAEP topics; these 

items cover a range of TIMSS framework objectives related to human use of natural 

resources as well as global and local environmental issues due to human and natural causes. 

The NAEP and TIMSS assessments have different balances of cognitive skills based on the 

expert panel classification of items to the TIMSS cognitive domains (factual knowledge, conceptual 

understanding, and reasoning and analysis). TIMSS has a larger proportion of items measuring 

factual knowledge (44 percent at fourth grade and 31 percent at eighth grade), while NAEP has more 

emphasis on conceptual understanding (53 percent of fourth-grade items and 61 percent of eighthgrade items). Both NAEP and TIMSS classify at least 20 percent of eighth-grade items as reasoning 

and analysis, while at the fourth grade there are relatively more items in this cognitive domain in 

NAEP than in TIMSS (22 percent compared to 10 percent). 

While both assessment frameworks include the measurement of scientific investigation or 

scientific inquiry, this area receives more emphasis in NAEP than in TIMSS. Both assessments 

include individual items as well as longer tasks designed to measure investigation or inquiry skills. 

TIMSS includes “problem solving and inquiry” tasks as part of its pencil-and-paper assessment, but 

only NAEP includes “hands-on” tasks involving the use of materials to conduct scientific 

investigations. Based on the expert panel classifications, about one-quarter of NAEP items compared 

to about 10 percent of TIMSS items at both the fourth and eighth grades measure scientific inquiry

skills and understandings. This difference was due at least in part to the “hands-on” tasks in NAEP, 

nearly all of which were classified as measuring scientific inquiry. 
vii

NAEP and TIMSS also differ with respect to the types and distribution of item formats. 

While the NAEP assessment is roughly balanced with respect to multiple-choice and constructedresponse items, TIMSS has a larger proportion of multiple-choice items—about two-thirds at both 

grades. TIMSS includes a higher proportion of items classified as extended-response by its 

assessment developers; however, the definition of extended-response items differs between the two 

assessments and some of these items may correspond more closely to items classified as shortanswer items in the NAEP assessment. There are differences between NAEP and TIMSS in the 

cognitive domains covered by multiple-choice items, with more than half of TIMSS multiple-choice 

items classified as factual knowledge compared to 60 percent of the NAEP items classified as 

conceptual understanding. The cognitive domains of the constructed-response items are similar for 

both assessments, with more than 60 percent conceptual understanding, at least 25 percent reasoning

and analysis, and 10 percent or less factual knowledge. 

The similarities and differences between NAEP and TIMSS are presented in this report for 

consideration when making meaningful comparisons and interpretations of the achievement results 

from the two assessments. While NAEP and TIMSS show similarities, particularly at the broad 

content area and topic level, differences are revealed by examining in more detail what the items in 

each assessment measure. In addition to providing information to help explain differences in 

achievement results from NAEP and TIMSS, this report also illustrates the complementary nature of 

the NAEP and TIMSS assessments. As a result of different emphases in NAEP and TIMSS, each 

assessment contributes more information in some content areas as well as some unique components. 
viii

Acknowledgments 

Many people’s contributions made this report possible, and the authors wish to thank all 

those who have assisted with various aspects of the report, including data analysis, reviews, and 

design. 

Members of the expert panel listed in appendix C provided all of the item classification data 

used as the foundation for the results presented in this report. The authors would like to thank the 

panel members for their expertise and contributions to the study. 

Thanks to Patrick Gonzales, Eugene Owen, and Mariann Lemke of the National Center for 

Education Statistics (NCES) for their input on the design and their role in reviewing the report. The 

authors are grateful to NCES technical reviewers, Marilyn Seastrom and Bruce Taylor, as well as to 

NCES program directors (Andrew Malizio and Elois Scott) and associate commissioners (Peggy Carr 

and Val Plisko) who provided direction and support for this publication. The authors wish to thank 

Steve Gorman of NCES, Barbara Lynch of Synergy Enterprises, Christine O’Sullivan of K-12 

Consulting, Duc-Le To of the Institute of Education Sciences, and two anonymous reviewers for 

their very helpful comments. 

Several people, formerly or currently from the Education Statistics Services Institute (ESSI), 

provided support in the form of research assistance and/or review: Ben Dalton, Kristy David, Rachel 

Dinkes, Aaron Douglas, Janine Emerson, J. Lane Glenn, Robin Gurley, Jamie Johnston, Dana Kelly, 

Geeta Kotak, David Miller, Melanie Ouellette, Lisette Partelow, Mike Planty, Anindita Sen, and 

Margaret Woodworth. In particular, we recognize the efforts of David Miller and Jamie Johnston in 

helping to shepherd the report through the final stages of the review and publication process. Thanks 

to Heather Block, also of ESSI, for her work on graphic design for this report, and to Brian Henigin 

of Westat for the final publication formatting of the report. 

For permission to use secure items and publish released items, the authors would like to 

thank the Trends in International Mathematics and Science Study (TIMSS) and the International 

Association for the Evaluation of Educational Achievement (IEA). Thanks also to the staff of the 

Educational Testing Service for participation in the expert panel meeting and for supplying item 

information and review materials from the National Assessment of Educational Progress (NAEP). 
ix

Contents Page 

EXECUTIVE SUMMARY .....................................................................................................................iii

ACKNOWLEDGMENTS .....................................................................................................................viii

LIST OF TABLES................................................................................................................................... xi

LIST OF FIGURES ...............................................................................................................................xiii

LIST OF EXHIBITS.............................................................................................................................. xiv

1. INTRODUCTION ................................................................................................................................ 1

2. OVERVIEW OF THE ASSESSMENTS AND THEIR FRAMEWORKS ......................................... 5

2.1. NAEP 2000 Science Framework........................................................................................ 8

2.2. TIMSS 2003 Science Framework .................................................................................... 10

2.3. Comparing the NAEP and TIMSS Science Frameworks and Assessments..................... 12

3. PROCESS AND METHODS ............................................................................................................ 17

3.1. Organization of the Expert Panel Meeting ....................................................................... 17

3.2. Methods Used for NAEP/TIMSS Comparisons............................................................... 17

4. OVERALL COMPARISONS ............................................................................................................ 21

4.1. Content Coverage............................................................................................................. 21

4.2. Grade Level ...................................................................................................................... 24

4.3. Cognitive Domain ............................................................................................................ 28

4.4. Scientific Inquiry.............................................................................................................. 34

4.5. Item Format...................................................................................................................... 37

5. NAEP/TIMSS COMPARISONS BY MAIN CONTENT AREAS................................................... 41

5.1. Physical Science............................................................................................................... 42

5.2. Life Science...................................................................................................................... 49
x

5.3. Earth Science.................................................................................................................... 56

5.4. Environmental Science..................................................................................................... 62

6. CONCLUSION................................................................................................................................... 67

REFERENCES ....................................................................................................................................... 69

APPENDIX A: Content Framework Summary Documents .................................................................A-1 

APPENDIX B: Summary Information on Cognitive Domains and Scientific Inquiry ........................B-1 

APPENDIX C: Expert Panel ................................................................................................................C-1 

APPENDIX D: Methodological Notes and Supplementary Data.........................................................D-1 

APPENDIX E: Example Items .............................................................................................................E-1 
xi

List of Tables 

Table Page 

1. Target percentage of NAEP assessment time distributed across NAEP framework 

dimensions, by grade: 2000............................................................................................................... 8 

2. Target percentage of TIMSS assessment time distributed across TIMSS framework 

dimensions, by grade: 2003............................................................................................................. 10 

3. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the fields of 

science in the NAEP science framework, by grade and survey ...................................................... 22 

4. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the content 

domains in the TIMSS science framework, by grade and survey ................................................... 22 

5. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the other 

assessment framework at the topic or objective level, by grade and survey ................................... 23 

6. Percentage of NAEP 2000 single-grade and cross-grade science items classified at each 

grade level according to the TIMSS 2003 science framework........................................................ 28 

7. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across TIMSS 

cognitive domains, by grade and survey ......................................................................................... 29 

8. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across TIMSS 

cognitive domains, by grade and content area................................................................................. 31

9. Percentage distribution of TIMSS 2003 science items across TIMSS cognitive domains: 

comparison between classifications by expert panel and assessment developers ........................... 33 

10. Percentage distribution of NAEP 2000 and TIMSS 2003 scientific inquiry items across 

TIMSS cognitive domains............................................................................................................... 37 

11. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across item 

formats, by grade and survey........................................................................................................... 37 

12. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade physical science 

items classified to the other science assessment framework, by level of content/grade 

match ............................................................................................................................................... 44 

13. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade life science items 

classified to the other science assessment framework, by level of content/grade match ................ 51 

14. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade Earth science 

items classified to the other science assessment framework, by level of content/grade 

match ............................................................................................................................................... 59 
xii

15. Percentage of TIMSS 2003 fourth- and eighth-grade environmental science items 

classified to the NAEP 2000 science framework, by level of content/grade match........................ 63 

16. Percentage distribution of TIMSS 2003 environmental science items across 

environmental science topics in the TIMSS science framework, by grade..................................... 64 

17. Percentage of TIMSS 2003 environmental science items classified to topics in the 

NAEP 2000 science framework in each field of science, by grade................................................. 65 

D-1. Reliability of cognitive domain classifications for science items in NAEP 2000 and 

TIMSS 2003, by number of comparisons and percentage agreement ............................................D-2 

D-2. Reliability of cognitive domain classifications for science items in NAEP 2000 and 

TIMSS 2003, by number of items and percentage agreement.........................................................D-3 

D-3. Reliability of scientific inquiry classifications for science items in NAEP 2000 and 

TIMSS 2003, by number of comparisons and percentage agreement ............................................D-3 

D-4. Reliability of scientific inquiry classifications for science items in NAEP 2000 and 

TIMSS 2003, by number of items and percentage agreement.........................................................D-3 
xiii

List of Figures 

Figure Page 

1-A. Percentage distribution of NAEP 2000 science items classified at each grade level 

according to the TIMSS 2003 science framework, by grade........................................................... 26 

1-B. Percentage distribution of TIMSS 2003 science items classified at each grade level 

according to the NAEP 2000 science framework, by grade............................................................ 27 

2. Percentage distribution of NAEP 2000 science items across TIMSS 2003 cognitive 

domains, by NAEP knowing and doing categories ......................................................................... 32 

3. Percentage of NAEP 2000 and TIMSS 2003 fourth-grade items classified as measuring 

scientific inquiry, by science content area....................................................................................... 35 

4. Percentage of NAEP 2000 and TIMSS 2003 eighth-grade items classified as measuring 

scientific inquiry, by science content area....................................................................................... 36 

5. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across TIMSS 

cognitive domains, by item format .................................................................................................. 39 

6. Percentage of NAEP 2000 and TIMSS 2003 physical science items classified to 

physical science topics in the NAEP science framework, by survey and grade.............................. 47 

7. Percentage of NAEP 2000 and TIMSS 2003 physical science items classified to 

chemistry and physics topics in the TIMSS science framework, by survey and grade................... 48 

8. Percentage of NAEP 2000 and TIMSS 2003 life science items classified to life science 

topics in the NAEP science framework, by survey and grade......................................................... 54 

9. Percentage of NAEP 2000 and TIMSS 2003 life science items classified to life science 

topics in the TIMSS science framework, by survey and grade ....................................................... 55 

10. Percentage of NAEP 2000 and TIMSS 2003 Earth science items classified to Earth 

science topics in the NAEP science framework, by survey and grade............................................ 61 

11. Percentage of NAEP 2000 and TIMSS 2003 Earth science items classified to Earth 

science topics in the TIMSS science framework, by survey and grade........................................... 62 
xiv

List of Exhibits 

Exhibit Page 

1-A. NAEP science framework dimensions: 2000 .................................................................................... 7 

1-B. TIMSS science framework dimensions: 2003 ................................................................................... 7 

2. Terminology used in making comparisons across NAEP 2000 and TIMSS 2003 content 

frameworks ...................................................................................................................................... 13 

3. Physical science topics included in the NAEP 2000 and TIMSS 2003 science 

frameworks ...................................................................................................................................... 43 

4. Life science topics included in the NAEP 2000 and TIMSS 2003 science frameworks................. 50 

5. Earth science topics included in the NAEP 2000 and TIMSS 2003 science frameworks............... 57 

A-1. NAEP science framework and specifications summary: 2000......................................................A-2 

A-2. TIMSS science framework and specifications summary: 2003 ..................................................A-20 

B-1. Cognitive domains based on the TIMSS 2003 science framework............................................... B-1 

B-2. Scientific inquiry based on the TIMSS 2003 science framework ................................................. B-6 

E-1. Index of example items from NAEP 2000 and TIMSS 2003........................................................ E-1 
1 

1. Introduction 

The National Center for Education Statistics (NCES) provides information to assist 

policymakers, researchers, educators, and the public in obtaining a comprehensive picture of what 

U.S. students know and can do in key subject areas. Information on such learning outcomes can be 

valuable in helping to inform policy, monitor and benchmark progress, and identify factors for 

improving student achievement, among other benefits. 

National-level data on student achievement in the United States comes primarily from two 

sources: the National Assessment of Educational Progress (NAEP)—otherwise known as the 

Nation’s Report Card—and the United States’ participation and collaboration in international 

assessments, such as the Trends in International Mathematics and Science Study (TIMSS), the 

Program for International Student Assessment (PISA), and the Progress in International Reading 

Literacy Study (PIRLS).1

 NAEP has been assessing fourth-, eighth-, and twelfth-grade students’ 

performance in reading, mathematics, science, and other subjects at regular intervals for over 30 

years. The international assessments allow the United States to benchmark its performance to other 

countries—in fourth- and eighth-grade mathematics and science in TIMSS, in fourth-grade reading 

literacy in PIRLS, and in 15 year-olds’ reading, mathematical and scientific literacy in PISA. This 

comparative element is increasingly important as the marketplace in which U.S. students will have to 

compete becomes more global. 

While these assessments may appear to be similar in terms of the age or grade of the students 

assessed and the subject matter of the assessment, each was designed to serve a different purpose and 

is based on a unique framework. In order to provide information to help interpret and compare 

results, NCES has periodically undertaken studies to examine the similarities and differences among 

the various assessments. The purpose of these studies is to provide an in-depth and comparative look 

at the frameworks and test items used in the different assessments to shed light on what each 

assessment can contribute to the knowledge base on student performance and help explain 

differences in results between the assessments. An understanding of the similarities and differences 

between the assessments is important to avoid misunderstanding or misinterpreting these results. 

In 2003, NCES conducted two content comparison studies with NAEP—one in mathematics 

and one in science—following the 2003 administrations of TIMSS and PISA. This report focuses on 

the comparison of the science assessments—specifically NAEP 2000 and TIMSS 2003—while a 

companion paper (Neidorf, Binkley, Gattis, and Nohara 2006) compares the NAEP, TIMSS and 

PISA 2003 mathematics assessments. PISA is not included in the science comparison study because 

scientific literacy was not a major domain of this assessment in 2003, including only about 30 

scientific literacy items. It is included in the mathematics comparison study, however, because 

mathematical literacy was a major domain in that year, with many more items on which to base 

analyses. Scientific literacy will be the major domain in 2006, when there will be a greater number 

of scientific literacy items and its assessment framework will be fully developed and updated. For 

more information on PISA see http://nces.ed.gov/surveys/PISA. 

 

1

 TIMSS and PIRLS are conducted under the auspices of the International Association for the Evaluation of 

Educational Achievement (IEA). The Organization for Economic Cooperation and Development (OECD) is 

responsible for PISA. 
2 

 The 2003 science and mathematics comparison studies build on several earlier studies, which 

were undertaken to explore the similarities and differences between NAEP and various international 

assessments. Such studies comparing frameworks and items are conducted periodically as NAEP 

and international assessments evolve, improving their frameworks and test items to reflect current 

research, policy, and practice. 

One previous published study of mathematics and science assessments is the NAEP 2000, 

TIMSS 1999, and PISA 2000 mathematics and science assessments (Nohara 2001), which compares 

the underlying frameworks and test items from each assessment in terms of content, item format, and 

thinking skills required. 

There also have been several studies comparing reading assessments. For example, the 

earliest of these compared the NAEP 1992 reading assessment and the 1991 IEA Reading Literacy 

Study (Binkley and Rust 1994). More recently, Binkley and Kelly (2003) examined the frameworks, 

passages, and items from the NAEP 2002 and the Progress in International Reading Literacy Study 

(PIRLS) 2001 reading assessments. 

The goal of this science comparison study is to identify similarities and differences between 

the NAEP 2000 and TIMSS 2003 assessments based on a detailed comparison of their frameworks 

and items. This information may be used to help inform interpretations of students’ performance in 

science based on the two different assessments. While there are other important aspects that might 

be compared, such as item difficulty, sampling, and scaling procedures, this study focuses on a 

comparison of the content of the assessments. This content comparison is based on the main 

dimensions of the assessment frameworks and focuses on a comparison of the set of assessment 

items as a reflection of how the frameworks are implemented. The questions driving the study are as 

follows: 

• How do NAEP and TIMSS define the domain of science to be assessed and its content areas, 

in terms of both the topics that are included and the distribution of items across topics? 

• How do NAEP and TIMSS define the content topics and assessment objectives appropriate 

for the fourth-grade and eighth-grade assessments? How do the items in each assessment 

compare to the grade-level expectations specified by the other framework?2

• How do NAEP and TIMSS compare with respect to the distribution of items across broad 

cognitive domains related to factual knowledge, conceptual understanding, and reasoning

and analysis? 

• How do NAEP and TIMSS define and measure scientific inquiry? To what extent does each 

assessment measure scientific inquiry skills? 

• How do NAEP and TIMSS compare with respect to the types and distribution of item 

formats? 

 

2

 The 2003 mathematics and science comparison studies are the first to compare the assessments in terms of grade 

level—the extent to which items from one assessment map to the same grade level of the framework of the other 

assessment. 
3 

To answer these questions, NCES convened an expert panel to examine the science 

frameworks and items for each assessment. The panel cross-classified NAEP and TIMSS fourth- and 

eighth-grade items to the other assessment’s framework with respect to science content and grade 

level. They also classified all items from both assessments with respect to a common set of cognitive 

domain categories and a common definition of scientific inquiry. In addition to the classification 

data from the panel, the study drew upon information provided by the NAEP and TIMSS assessment 

developers that describes how each item is classified according to the main dimensions of its own 

framework, as well as other relevant characteristics such as item format and scoring rubrics. 

Section 2 of this report opens with an overview of the NAEP and TIMSS assessments and a 

comparison of their respective science assessment frameworks. Section 3 reviews the methods used 

for this comparison study. The results of the study are then presented in two major sections. The 

first results section (section 4) compares the assessments overall with respect to content coverage, 

grade level, cognitive domain, scientific inquiry, and item format. The overall comparisons are 

followed (section 5) by comparisons of the two assessments in each of the main content areas of 

physical science, life science, Earth science, and environmental science. Section 5 provides more 

detailed comparisons of the extent to which items in one assessment map to the science framework of 

the other assessment and compares the content distribution of the items for each of the NAEP and 

TIMSS science subscales. The report concludes with a summary of key findings (section 6). 

5 

2. Overview of the Assessments and their Frameworks 

NAEP 

The National Assessment for Educational Progress (NAEP) is the United States’ source for 

nationally representative and continuing information on what American students know and can do 

and is commonly known as the Nation’s Report Card. NAEP policies and frameworks are 

established by an independent National Assessment Governing Board (NAGB), and the Department 

of Education’s National Center for Education Statistics (NCES) administers the assessment. For 

over 30 years, NAEP has periodically collected and reported data on achievement in reading, 

mathematics, science, and other subjects for students in fourth, eighth, and twelfth grades. The 

comparisons in this report are based on the main NAEP science assessments conducted in 2000.3

The frameworks established by NAGB for all the NAEP subject areas, including science, are 

based on the collaborative input of a wide range of experts and involvement by participants from 

government, education, business, and public sectors; are informed by common curricular practices in 

the nation’s schools; and ultimately are intended to reflect the best thinking about the knowledge, 

skills, and competencies needed for students to have a deep level of understanding of different 

subject areas at different grades. 

TIMSS 

The Trends in International Mathematics and Science Study (TIMSS) is the United States’ 

source for international comparative information on mathematics and science education in the 

primary and middle grades. TIMSS is one of the current studies conducted under the auspices of the 

International Association for the Evaluation of Educational Achievement (IEA), which has been 

conducting international comparative studies since the early 1960s. TIMSS is directed by the 

International Study Center at Boston College and collects achievement and background data to 

provide information on trends in mathematics and science achievement over time as well as on the 

curricular, instructional, and attitudinal factors that may be related to performance. TIMSS collects 

data on a four-year cycle. The first administration was in 1995 (at fourth, eighth, and twelfth 

grades)4

, the second in 1999 (at eighth grade only), and the most recent in 2003, with 25 countries 

participating at fourth grade and 45 countries participating at eighth grade. 

Like NAEP, the TIMSS assessment is based on collaboratively developed frameworks. In 

contrast to NAEP, however, the TIMSS framework development and consensus process involves 

science experts, education professionals, and measurement specialists not only from the United 

States, but from many countries. 

 

3

 At the time of this study, NAEP 2000 was the most recent science assessment. A subsequent science assessment 

was conducted in 2005. 

4 Defined as the upper of the two grades containing the majority of 9-year-olds or 13-year-olds and the final year in 

secondary school. These are the fourth, eighth and twelfth grades in the U.S. and most other countries. TIMSS 1995 

was also administered in third and seventh grades.
6 

Organization of the NAEP 2000 and TIMSS 2003 Science Frameworks 

Exhibits 1-A and 1-B compare schematically the organizing dimensions in the NAEP 2000 

and TIMSS 2003 science frameworks. These organizing dimensions provide the basic framework 

for the development of the pool of items in each assessment, and the framework documents include 

target percentages for the distribution of assessment time across the main categories in each 

dimension to ensure a balanced assessment (discussed in the following sections).5

 As seen in these 

figures, there are some similarities as well as differences between the dimensions in the frameworks 

for NAEP and TIMSS. 

Both the NAEP 2000 and TIMSS 2003 science frameworks are represented in exhibits 1-A 

and 1-B based on two main organizing dimensions—a content dimension and a cognitive dimension, 

as well as overarching dimensions (along the bottom) that define areas of science content 

knowledge—skills, and abilities that go across content and cognitive categories. In the content 

dimension, NAEP includes three broad fields of science (physical science, life science, and Earth 

science), while TIMSS includes five separate content domains (physics, chemistry, life science, Earth 

science, and environmental science). At the eighth grade, TIMSS reports on all five of the content 

domains, while at the fourth grade three content reporting categories are used that correspond to the 

NAEP fields of science (physics and chemistry items are combined into physical science and 

environmental science items are reported in either life science or Earth science). 

In their main cognitive dimensions (knowing and doing in NAEP and cognitive domains in 

TIMSS), both assessment frameworks include three broad categories that appear to be at least 

partially overlapping. The NAEP knowing and doing categories include conceptual understanding, 

scientific investigation, and practical reasoning; while the TIMSS cognitive domains include factual 

knowledge, conceptual understanding, and reasoning and analysis. There are two overarching 

dimensions in NAEP (nature of science and themes) and one overarching dimension in TIMSS 

(scientific inquiry). These overarching dimensions include both content-related and cognitive- or 

skills-based components, and there is overlap between the assessment objectives defined by these 

overarching dimensions and those defined in the other main dimensions in the NAEP and TIMSS 

science frameworks. In particular, both assessments include cognitive categories related to 

conceptual understanding and reasoning. Both assessments also address scientific investigation (a 

knowing and doing category in NAEP) or scientific inquiry (an overarching dimension in TIMSS). 

There also is overlap between understandings related to NAEP’s nature of science dimension and the 

scientific inquiry dimension in TIMSS. All items developed for NAEP and TIMSS are classified 

with respect to which categories in the content and cognitive dimensions they assess. Some items 

across these categories also are developed to assess science content knowledge, abilities and skills 

defined by the overarching dimensions. 

The following sections describe and compare in more detail the science assessment 

frameworks for NAEP and TIMSS. Additional assessment framework summary documents that 

were used for the comparison study are found in appendixes A and B. 

 

5

 The frameworks only provide target percentages of assessment time as guidelines for test development. 
7 

Exhibit 1-A. NAEP science framework dimensions: 2000 

Fields of science Knowing and doing 

Physical science Conceptual understanding 

Life science Scientific investigation 

Earth science Practical reasoning 

Nature of science 

(science and technology) 

Themes 

Models, systems, patterns of change 

NOTE: The NAEP framework is based on two main organizing dimensions—fields of science and knowing and doing—as well as two 

overarching dimensions (nature of science and themes) that go across the fields of science and knowing and doing categories. 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 National 

Assessment of Educational Progress, 1996, 2000. 

Exhibit 1-B. TIMSS science framework dimensions: 2003 

Content domains Cognitive domains

Physics Factual knowledge 

Chemistry Conceptual understanding 

Life science Reasoning and analysis 

Earth science 

Environmental science

Scientific inquiry 

NOTE: The TIMSS framework is based on two main organizing dimensions—content domains and cognitive domains—as well as an 

overarching dimension (scientific inquiry) that goes across the content and cognitive domains. 

SOURCE: International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 

2nd Edition, 2003.
8 

2.1. NAEP 2000 Science Framework 

The framework for the NAEP 2000 science assessment includes a content dimension (fields 

of science) and a cognitive dimension (knowing and doing), as well as two overarching dimensions 

(nature of science and themes) (see exhibit 1-A).6

 The framework stipulates that every item 

developed for the assessment is given a primary classification in the two major content and cognitive 

dimensions according to certain distribution targets. While secondary classifications may be 

assigned for some items, NAEP does not use multidimensional scaling, and these secondary 

classifications are not used in the analysis of results. In addition, some items are also classified as 

addressing understandings and skills specified in the two overarching dimensions. 

The first major dimension is the three content areas, fields of science, which are the same for 

fourth, eighth, and twelfth grade. They are life science, physical science, and Earth science. Within 

these content areas, major topics and subtopics are further identified which, with few exceptions, are 

the same across grades (4, 8, and 12).7

 The framework specifies that the distribution of assessment 

time should be approximately equal across the fields of science at the fourth and twelfth grades, and 

should place a slight emphasis on life science at the eighth grade (table 1). 

Table 1. Target percentage of NAEP assessment time distributed across NAEP framework 

Table 1. dimensions, by grade: 2000 

NAEP framework dimensions Grade 4 Grade 8 Grade 12 

Fields of science 

Life science 33 40 33 

Physical science 33 30 33 

Earth science 33 30 33 

Knowing and doing 

Conceptual understanding 45 45 45 

Scientific investigation 45 30 30 

 Practical reasoning 10 25 25 

NOTE: Percentages reflect the targets specified in the NAEP 2000 science framework. At the 4th and 12th grade, distributions across the 

three fields of science are approximately equal. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 

National Assessment of Educational Progress, 1996, 2000. 

The other major organizing dimension in the NAEP science framework is the cognitive 

dimension, knowing and doing science, which defines the processes and cognitive skills the items are 

designed to require of students. The three categories of knowing and doing in NAEP are conceptual 

understanding, scientific investigation, and practical reasoning. The NAEP 2000 science framework 

specifies that there should be heavy emphasis on conceptual understanding (45 percent at each 

 

6

 The framework for the NAEP 2000 science assessment was developed in the early 1990s and was used as the basis 

for the assessments in 1996, 2000, and 2005 (see NAGB (2000) for additional information). A new science 

framework will be developed for the 2009 assessment. 

7

 A separate assessment specifications document specifies different grade-specific assessment objectives within each 

subtopic that are appropriate for each of the respective grades (NAGB 1994). See section 5 and appendix A for 

more information about the topics and objectives included in the NAEP content framework and specifications 

documents. 
9 

grade), which is the ability to understand basic concepts and tools of science. The category of 

scientific investigation focuses on students’ use of appropriate tools and thinking skills in science to 

acquire new information, plan and conduct appropriate investigations, and communicate the results 

of investigation. These ways of knowing and doing science receive greater emphasis at the lowest 

grade (45 percent at fourth grade compared to 30 percent at eighth and twelfth grade). Practical 

reasoning also involves the application of scientific knowledge and skills but focuses more on 

analyzing and solving practical problems. The proportion of the assessment measuring practical

reasoning also differs across grades (10 percent at fourth grade compared to 25 percent at eighth and 

twelfth grades). 

In addition to the content and cognitive dimensions, the NAEP 2000 science framework 

includes two overarching categories, considered to be important aspects of science knowledge and 

skills to be captured in NAEP. The first is nature of science, which includes topics relating to the 

history of science and technology, habits of mind that characterize these fields, and methods of 

inquiry and problem solving. The framework specifies that at least 15 percent of the assessment 

should measure nature of science. The second category, themes, represents the big ideas or key 

concepts that transcend the scientific disciplines and enable students to consider problems with 

broader implications. Concepts related to themes are specified in each content area. The three 

themes are systems, models, and patterns of change. 

In comparison to nature of science, a much larger proportion of the assessment is specified to 

assess themes (about one-third at the fourth grade and half at the eighth and twelfth grades), spread 

evenly across the three. NAEP includes both individual items that assess concepts related to themes

as well as sets of items all related to a common theme (“theme blocks”) that provide an in-depth 

measure of a particular theme and may include items across content areas. 

The NAEP 2000 science framework specifies that multiple-choice, short-answer, and 

extended-response items are included in the assessment. Less than half of the assessment time 

should be multiple-choice, and about one-third of the constructed-response items should require an 

extended response. The framework also specifies that in addition to pencil-and-paper tests, about 30 

percent of the assessment at each grade will be devoted to “performance” or “hands-on” tasks. For 

this part of the assessment, students are provided with kits and they must manipulate physical 

materials to solve a scientific problem or conduct an investigation involving those materials. All of 

the items contained in the “hands-on” tasks are also classified according to the appropriate categories 

in the content and cognitive dimensions as well as the overarching categories of nature of science

and themes. 
10 

2.2. TIMSS 2003 Science Framework 

The TIMSS 2003 science framework is based on two main organizing dimensions, a content 

dimension and a cognitive dimension, as well an overarching dimension of scientific inquiry (see 

exhibit 1-B earlier in this section).8

 

The TIMSS 2003 science framework specifies five content domains. They include life 

science, chemistry, physics, Earth science, and environmental science. All five content domains are 

included as separate reporting categories at the eighth grade, but at the fourth grade, there are three 

reporting categories (physical science, life science, and Earth science). For fourth grade, chemistry 

and physics items are combined on the physical science scale, and a small number of items assessing 

environmental science topics at the fourth grade are embedded within the life science and Earth 

science reporting scales. As shown in table 2, the TIMSS 2003 science framework specifies that 

assessment time should have greater emphasis on life science at the fourth grade (45 percent at fourth 

grade compared to 30 percent at eighth grade) and physical science at the eighth grade (25 percent 

for physics and 15 percent for chemistry at the eighth grade compared to 35 percent for physical 

science overall at the fourth grade). A smaller proportion of the assessments at both grades are 

specified for Earth science (20 percent at fourth grade and 15 percent at eighth grade) and 

environmental science (15 percent at eighth grade only). Within the content domains, the TIMSS 

framework further specifies topic areas and grade-specific objectives within those topic areas that are 

appropriate for each grade.9

 

Table 2. Target percentage of TIMSS assessment time distributed across TIMSS framework 

Table 2. dimensions, by grade: 2003 

TIMSS framework dimensions Grade 4 Grade 8 

Content domains 

Life science 45 30 

Physical science 35 40 

 Chemistry † 15 

 Physics † 25 

Earth science 20 15 

 Environmental science † 15 

Cognitive domains 

Factual knowledge 40 30 

Conceptual understanding 35 35 

 Reasoning and analysis 25 35 

† Not applicable. At grade 4, target percentages are defined for only three reporting categories—life science, physical science, and Earth science. 

Physical science is assessed as one content area that includes both physics and chemistry topics. Some topics related to environmental science are 

assessed as part of life science and Earth science. 

NOTE: Percentages reflect the targets specified in the TIMSS 2003 framework. 

SOURCE: International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 

2nd Edition, 2003.

 

8

 The TIMSS science framework was revised for 2003 from the original curriculum framework used as the basis for 

the 1995 and 1999 assessments. See Mullis et al. (2003) for additional information. 

9

 See section 5 and appendix A for more information about the topics and objectives included in the TIMSS content 

framework. 
11 

On the cognitive dimension, TIMSS specifies three cognitive domains to describe the range of 

cognitive skills and abilities that the items require students to apply to answer the item successfully. 

The three cognitive domains are factual knowledge, conceptual understanding, and reasoning and 

analysis. The inclusion of items in the factual knowledge category provides a measure of the extent 

and accuracy of students’ factual knowledge and their knowledge of information, tools, and 

procedures, which are the basis of developing scientific understanding and reasoning. The next 

category of conceptual understanding assesses students’ grasp of the relationships that explain the 

behavior of the physical world and their ability to relate observations to more abstract or general 

concepts in science. The third category, reasoning and analysis, captures students’ ability to engage 

in scientific reasoning for the purpose of analyzing and solving problems, developing explanations, 

drawing conclusions, making decisions, and extending their knowledge to new situations. The 

TIMSS 2003 science framework specifies an approximately equal distribution across the cognitive 

domains for the eighth grade assessment. For fourth grade, however, there is a somewhat greater 

emphasis on the factual knowledge category (40 percent) compared to reasoning and analysis (25 

percent). 

All TIMSS items are classified with respect to content domain (topic area and objective) and 

cognitive domain at the broad category level. Items also are classified with respect to whether they 

are intended to measure knowledge and skills necessary to engage in scientific inquiry. All items 

measuring scientific inquiry are set in a science content-based context and assess skills and abilities 

described in the cognitive domain categories. 

The TIMSS 2003 science framework includes scientific inquiry as an overarching dimension 

that overlaps all the content domains and cognitive domains and has both content- and skills-based 

components. In TIMSS, it is expected that students at both grade levels will possess some general 

knowledge of the nature of science and scientific inquiry and demonstrate skills and understandings 

involved in the major phases of the scientific inquiry process, such as formulating questions and 

hypotheses; designing investigations; representing, analyzing, and interpreting data; and drawing 

conclusions based on evidence. The TIMSS 2003 assessment does not include “performance” or 

“hands-on” tasks.10 Rather, a set of “problem solving and inquiry tasks” are included as part of the 

pencil-and-paper assessments. These tasks present students with an inquiry-based situation and ask a 

series of related questions assessing some of the skills specified in the scientific inquiry dimension. 

While these tasks are not intended to be full scientific investigations, they are designed to require a 

basic understanding of the processes of investigation and skills essential to the scientific inquiry 

process. Individual items are also included in the assessment to measure individual inquiry skills. 

The framework specifies that up to 15 percent of the assessment at each grade may assess scientific 

inquiry. Appendix B provides a more detailed definition of the TIMSS cognitive domain categories 

and scientific inquiry. 

The TIMSS 2003 science framework specifies that both multiple-choice and constructedresponse items requiring students to generate their own answers be included in the assessment, with 

up to two-thirds of the assessment time coming from multiple-choice items. About two-thirds of the 

constructed-response items should require a short answer, while the other third should require a more 

extended response. 

 

10 Performance assessment tasks were included as part of the TIMSS 1995 assessment but not repeated in the 1999 

or 2003 assessments. 
12 

2.3. Comparing the NAEP and TIMSS Science Frameworks and Assessments 

The NAEP and TIMSS science frameworks have some obvious similarities, including the 

general structure with content and cognitive dimensions, the inclusion of overarching dimensions, 

and the specification of curriculum-based content areas related to the physical, life, and Earth 

sciences. Both NAEP and TIMSS cover a broad range of science topics and skills and include 

similar numbers of items in their assessments.11 However, there are differences as well, which may 

affect the items developed based on the respective frameworks and thus the content of each 

assessment overall. A first notable structural distinction is TIMSS’s inclusion of a separate 

environmental science category and the disaggregation of physical science into the two disciplines of 

chemistry and physics in the eighth-grade assessment, which differs from the NAEP categorization 

into three main fields of science at both grades. While both NAEP and TIMSS are grade-based 

assessments including two corresponding grades (fourth and eighth), NAEP also includes a twelfthgrade assessment.12 The set of topics and objectives included in the NAEP framework reflect this 

difference. 

The method of specifying the science content to be assessed at each grade level is somewhat 

different in NAEP and TIMSS frameworks. Exhibit 2 shows the different content levels and 

terminology specified in each framework as well as general terminology used in this report to 

facilitate direct comparisons. NAEP has one additional “layer” of content specification than does 

TIMSS—major topic areas used to organize subtopics within each field of science. For example, in 

physical science there are three major topics in NAEP—matter and its transformations, energy and 

its transformations, and motion—with several subtopics within each.13 The major topics and 

subtopics in the NAEP framework are common across grades (4, 8, and 12), and a separate 

assessment specifications document used by assessment developers describes specific objectives in 

each subtopic that are expected at each grade level (NAGB 1994). In contrast, the TIMSS 

framework specifies only two levels—topic areas and grade-specific objectives in each content 

domain. For this report, the NAEP subtopics (within major topics) and specific objectives are 

compared with the two most comparable TIMSS levels (topic area and objective).14 

When making direct comparisons related to item content, this report uses a general 

terminology of content area, topic, and objective to refer to the comparable levels of specification 

used in the NAEP and TIMSS content framework. For the discussion of content or cognitive 

classifications based on a single framework (NAEP or TIMSS), the terminology from that framework 

is used. 

 

 

11 Both NAEP 2000 and TIMSS 2003 include more than 170 items at eighth grade and more than 140 items at fourth 

grade. 

12 Although TIMSS was administered at the twelfth grade in the 1995 survey, it was not administered at this grade 

level in the 2003 survey, which is the focus of this comparison study. 

13 The content area comparisons in section 5 show major topics and subtopics in each NAEP field of science and the 

topic areas within each TIMSS content domain. 14 The topic/subtopic/objective structures are shown in the detailed framework summary documents in appendix A. 
13 

Exhibit 2. Terminology used in making comparisons across NAEP 2000 and TIMSS 2003 

content frameworks 

NAEP framework

Field of science 

Major topic 

Subtopic (within major topic) 

Specific objective 

⇐

⇐

⇐

General terminology

Content area 

Topic 

Objective

⇒

⇒

⇒

TIMSS framework

Content domain 

Topic area 

Objective

Examples of related NAEP and TIMSS content areas, topics, and objectives

 NAEP TIMSS

Field of science: 

Major topic: 

Subtopic: 

Specific objective: 

Physical science 

Energy and its 

transformations 

Energy sources and use, 

including distribution, 

conversion, costs, and 

depletion 

Demonstrate awareness 

that the sun is the ultimate 

source of most energy we 

use 

Content domain: 

Topic area: 

Objective: 

Physics 

Energy types, sources, and 

conversions 

Identify common energy sources 

and forms; know some practical 

uses of energy 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 National 

Assessment of Educational Progress, 1996, 2000; and International Study Center, Lynch School of Education, Boston College, TIMSS 

Assessment Frameworks and Specifications 2003, 2nd Edition, 2003.

Although the TIMSS framework recognizes the importance of unified concepts and topics 

that bridge the domains of science, only NAEP includes some specific science themes (models, 

systems, and patterns of change) as an explicit part of the framework dimensions and assessment 

specifications. 

While both NAEP and TIMSS frameworks define cognitive categories for assessing 

conceptual understanding and reasoning (defined as practical reasoning in NAEP and reasoning 

and analysis in TIMSS), only TIMSS includes an explicit category for factual knowledge. However, 

the NAEP definition of conceptual understanding does include knowledge of facts and information 

about procedures. 

Both the NAEP and TIMSS frameworks recognize scientific investigation or inquiry as an 

important component of science teaching, learning, and assessment. Scientific investigation is 

included as one of the cognitive categories in NAEP; scientific inquiry is included as an overarching 

dimension in the TIMSS framework that has both content- and skill-based components. Based on the 

target percentages in the frameworks, more emphasis on scientific investigation or inquiry would be 

expected for NAEP, particularly at fourth grade. In addition, NAEP includes “hands-on” 

investigational tasks, while TIMSS includes pencil-and-paper “problem-solving and inquiry tasks” 
14 

but no performance task component. Examples of a NAEP “hands-on” task and a TIMSS “problem 

solving and inquiry” task from the eighth-grade assessments are included in appendix E.15

NAEP and TIMSS include both multiple-choice items, in which students choose the correct 

answer from a list of several choices (usually four), and constructed-response items, in which 

students generate their own answers. Some of the constructed-response items in both assessments 

require only a short and often objective response (e.g., an example of a mixture), while others require 

a more extended response. The frameworks specify that up to two-thirds of the TIMSS assessment 

time can be from multiple-choice items compared to half or less in NAEP. Both frameworks allow 

for about one-third of the constructed-response items to be extended response. Some of the 

extended-response items across the two assessments are similar, while others are not. A more 

detailed comparison of item format is included in section 4.5. 

The assessment designs for NAEP and TIMSS result in each individual student taking only a 

portion of the total assessment items, but the testing time for individual students differs across the 

two assessments. NAEP requires 50 minutes at all three grades and TIMSS requires 72 minutes at 

fourth grade and 90 minutes at eighth grade. The hands-on tasks in NAEP require an additional 20 

minutes at fourth grade or 30 minutes at eighth and twelfth grade. 

Finally, the NAEP framework was developed within the specific context of the U.S. system 

and defines a set of achievement levels (basic, proficient, and advanced) that are used as the primary 

means of reporting what students should know and be able to do in science at each grade level from a 

national perspective. In contrast, the TIMSS framework reflects a consensus across diverse 

participating countries about what science topics are most appropriate and important to assess at 

fourth and eighth grades. In general, the topics included are in the curricula for the majority of 

TIMSS countries. 

Some curricular differences that exist across countries are related to the structure and 

sequencing of science instruction (Beaton et al. 1996; Martin et al. 1997; Martin et al. 2000; 

Robitaille 1997). While integrated or general science is taught in many countries (including the 

United States), separate science courses (biology, chemistry, and physics) are taught by the eighth 

grade in a number of other countries. Earth science is not taught as a separate science subject in 

many countries, and topics related to this content area may be included in courses covering the 

physical and life sciences as well as in separate courses such as geography. There is considerably 

more emphasis placed on science instruction in the primary and middle school years in some 

countries than others. In particular, some countries have less formalized science instruction by the 

fourth grade than the United States. Some of the differences in science curricula across countries are 

reflected in differences between the NAEP and TIMSS frameworks and science assessments. In the 

following sections, more detailed analyses of the individual content areas provide some information 

on these possible differences. 

 

15 The example included for the NAEP “hands-on” task is from the 1996 assessment, as none of the “hands-on” 

tasks in the 2000 assessment were released and cannot be reproduced here. This example is included only to 

illustrate the nature of the “hands-on” tasks in NAEP, and this particular task is not reflected in any of the item 

comparisons made in this report which are based on the NAEP 2000 assessment. The NAEP hands-on tasks 

included in the 2000 assessment are similar in style and approach but involve different science content. 
15 

This section provided an overview of the NAEP and TIMSS assessments and a comparison 

of their respective science assessment frameworks. The next section reviews the methods used for 

this comparison study. 

17 

3. Process and Methods 

To conduct comparisons of the NAEP and TIMSS assessments, NCES convened a panel of 

11 experts in science, science education, and science assessment. All panel members had familiarity 

and experience with at least one of the assessments and their frameworks.16 The panel met over a 2-

day period to review the frameworks and classify the items from each assessment. The following 

two sections describe the organization of the expert panel meeting and the methods used for making 

the NAEP/TIMSS comparisons reported in this report. Additional methodological notes are included 

in appendix D. 

3.1. Organization of the Expert Panel Meeting 

The expert panel meeting opened with a plenary session during which the study organizers 

presented the goals of the study, provided an overview of the NAEP and TIMSS frameworks, and 

described the procedures for reviewing items. The expert panel members also had an opportunity 

during the opening plenary session to review, classify, and discuss several practice items in order to 

establish a common understanding of the classification procedures. 

During the 2-day meeting, all of the NAEP and TIMSS fourth- and eighth-grade science 

items were reviewed, reflecting a total of about 630 items across the two assessments and grades. 

The items were divided into three groups according to content, with each group containing all items 

from both NAEP and TIMSS in the content areas of17

• life science; 

• physical science (chemistry and physics); and 

• Earth science and environmental science. 

The panel also was divided into three groups, with each group responsible for reviewing and 

classifying all of the items in one of the content groups. Panelists and staff were assigned to 

subgroups according to their content area expertise and to make sure that each group contained 

participants familiar with each of the assessments. 

The meeting concluded with a plenary session during which panelists shared their thoughts 

on the frameworks, items, and the study overall. While this report draws from these comments, 

where applicable, it reports primarily on the results from the item review and classification sessions, 

which were the focus of the meeting. 

3.2. Methods Used for NAEP/TIMSS Comparisons 

In each content area group, the panel conducted a framework-level review to familiarize the 

panelists with the relevant portions of the content frameworks and to uncover some of the main 

 

16 A list of panel members and associated staff is presented in appendix C. 

17 The division of items was based on the assessment developers’ classifications by content area subscale. 
18 

similarities and differences in how the major content areas covered by each group are interpreted in 

the two frameworks documents. The panels then classified the items, first classifying the TIMSS 

items to the NAEP framework and then classifying the NAEP items to the TIMSS framework. All 

items were classified on the following dimensions: 18

• Content: Each item was classified with respect to the content framework of the other 

assessment (i.e., TIMSS items to the NAEP framework and NAEP items to the TIMSS 

framework) by identifying the content area, topic, and objective with the best match to the 

item content. Some items were classified as matching the other assessment framework at 

only the topic or content area level. Items that could not be classified at any level were also 

identified. 

• Grade level: Each item was classified with respect to the grade level corresponding to the 

best content match in the other framework. For TIMSS items classified to the NAEP 

framework, grade classification was made to grade 4, 8, or 12. However, for NAEP items 

classified to the TIMSS framework, grade classifications were limited to grades 4 and 8 since 

TIMSS does not include grade 12. 

• Cognitive domain: All items were classified in one of the cognitive domain categories of 

factual knowledge, conceptual understanding, or reasoning and analysis according to the 

definitions in the TIMSS 2003 framework. 

• Scientific inquiry (yes/no): All items were classified as to whether or not they measured 

scientific inquiry skills as defined in the TIMSS 2003 framework. 

In conducting their evaluations, panelists were given the following guidelines: 

• Items should be classified to the most detailed content level possible—ideally, to the 

objective level. (Although panelists were allowed to make some logical inferences about 

what a content area, topic, or objective might include, they were instructed not to classify 

items further than they considered appropriate.) 

• Each group should consider all content areas of the framework. The content area in one 

assessment may overlap with another content area in the other assessment (e.g., the best topic 

match for an Earth science item may be in the life science content area of the other 

framework). 

• In cases where items appear to address multiple content areas, topics, or objectives, a primary 

classification for the item should be identified whenever possible. (In cases where this was 

not appropriate, panelists were instructed to indicate multiple or secondary classifications 

which were recorded.)19 

 

18 Additional information about the content categories and definitions of cognitive domains and scientific inquiry is 

provided in appendixes A and B. Considerations in selecting the methods of classification are discussed in appendix 

D. 

19 The results in this report are based on primary classifications in nearly all cases. 
19 

• Instances where a number of items that cannot be placed in a framework are of a similar type 

should be documented. These instances may indicate a potential gap in the framework to 

which the items are being classified. 

• Grade-level classifications should be based on descriptions found in the frameworks rather 

than on common understandings of grade-level content (i.e., items should be placed at the 

grade level where they best match the descriptions in the content framework). (As with other 

content classifications, panelists were allowed to make some logical inferences about what a 

topic or objective might include at a given grade level.)20 

• Classifications to cognitive domain categories and scientific inquiry should be based strictly 

on the definitions in the TIMSS 2003 framework. 

Within each group, panelists classified all items individually and then discussed the 

classifications as a group to arrive at a group classification. In general, consensus was reached, but 

for some items the final classifications reflect the classifications of the majority of panelists. For 

cognitive domain, final classifications for some items reflected the fact that the panel was split 

between two categories or that the consensus of the group was that the item was on the borderline 

between two categories. To monitor consistency in the classifications of cognitive domain and 

scientific inquiry across the three groups, a set of common items was classified by the members of all 

three groups. The degree to which the three groups classified these items in the same categories on 

these two dimensions serves as a measure of the reliability of these classifications. The items in the 

reliability set were not chosen at random, but rather, were a representative set of 60 items (30 from 

NAEP and 30 from TIMSS) selected to cover the main categories addressed in the study (content 

area and grade level). Reliability items were classified at regular intervals throughout the 

classification process. The reliability procedure and results are described in more detail in the 

methodological notes (appendix D). 

Expert panelists typically spent more time reviewing and classifying the items in the 

reliability set that were in their primary content area. Thus the classifications by the primary content 

area expert panel groups are the most valid and used for all of the results in the report. Results from 

the secondary classifications of the reliability set were used to monitor the consistency of 

classification and were not a complete replication of the process used by the primary group, which 

was most familiar with items in the respective content area. 

Panelists’ comments on the items were also recorded during the item review process, 

including observations about specific item characteristics and rationales for the classifications. In 

addition, general comments made by the panel about the assessments and frameworks in plenary or 

during the separate group discussions were recorded and used to inform the discussions in this report. 

 

20 Since the TIMSS framework contains grade-specific objectives, the grade-level classification is concurrent with 

an objective classification. The NAEP framework provides a single set of major topics and subtopics that apply 

across grades; however, the assessment specifications document provides grade-specific objectives that were used to 

determine the specific grade match for the TIMSS items to NAEP framework classifications. For items not 

classified to a grade-specific objective, the grade classification reflects the judgment of the panel of the grade at 

which the item is most consistent with the overall framework. 
20 

This section reviewed the methods used for this comparison study. The next section 

compares the assessments overall with respect to content coverage, grade level, cognitive domain, 

scientific inquiry, and item format. 
21 

4. Overall Comparisons 

The classifications made by the expert panel as well as the information provided by each 

assessment provide rich data that can be organized and analyzed in numerous ways. This section 

compares the assessments overall with respect to content coverage, grade level, cognitive domain, 

scientific inquiry, and item format. 

4.1. Content Coverage 

Tables 3 and 4 compare the distribution of items from each assessment across the main 

content areas in the NAEP and TIMSS frameworks. The tables compare NAEP and TIMSS item 

classifications, according to their own respective frameworks, with item classifications according to 

the framework of the other assessment.21 At the fourth grade, according to the NAEP framework, 

NAEP items are fairly evenly distributed across the three content areas of physical science, Earth 

science, and life science, with a slight emphasis (37 percent) on life science (table 3). When the 

TIMSS grade 4 items were classified according to the NAEP framework, 45 percent were classified 

as life science and 24 percent as Earth science. Using the TIMSS framework as the reference point, a 

similar picture emerges (table 4). Both NAEP and TIMSS have about one-third of their fourth-grade 

items classified as physical science on the TIMSS framework, while TIMSS has more items 

classified as life science (45 percent compared to 37 percent) and NAEP has relatively more items 

classified as Earth science (32 percent compared to 24 percent). NAEP and TIMSS have items 

distributed similarly across the TIMSS chemistry and physics content domains, with both 

assessments having somewhat more emphasis on physics. Also, a small percentage of items in each 

assessment (5 percent or less) were classified in the environmental science content area. There also 

was one NAEP fourth-grade item dealing with technology that could not be placed within the content 

areas as defined in the TIMSS framework. 

At eighth grade, NAEP items are again fairly evenly distributed across the fields of science 

according to the NAEP framework, whereas TIMSS items show a greater emphasis on physical 

science (46 percent in TIMSS compared to 31 percent in NAEP) and a lesser emphasis on Earth 

science (24 percent in TIMSS compared to 33 percent in NAEP) (table 3). These differences in 

emphases are confirmed using the TIMSS framework as the classification system (table 4). The 

comparison based on the TIMSS framework also shows that the increased emphasis on physical 

science in TIMSS is distributed across both the chemistry and physics content areas. TIMSS also has 

a higher percentage of items classified as environmental science, which is consistent with this being 

an explicit content area in TIMSS but not NAEP. Most items classified as environmental science 

were classified as Earth science in the NAEP framework, although some TIMSS environmental 

science items were also classified as life or physical science in NAEP. As with the fourth grade, one 

NAEP eighth-grade technology item could not be placed within the content areas as defined in the 

TIMSS framework. Also, four items (2 percent) from each assessment were classified to multiple 

content areas on the other’s framework. 

 

21 The classifications of items to their own framework were provided by the assessment developers. Cross 

classifications of NAEP and TIMSS items to the other’s assessment framework were done by the expert panel. 
22 

Table 3. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the fields of 

Table 1. science in the NAEP science framework, by grade and survey 

Grade 4 Grade 8 NAEP field of science 

NAEP1 TIMSS2 NAEP1 TIMSS2

Total number of items 144 141 197 177 

Percentage distribution 

Physical science 31 31 31 46 

Life science 37 45 36 32 

Earth science 32 24 33 24 

Classified to multiple fields 0 0 0 2 

1

 NAEP items classified by NAEP developers.

2

 TIMSS items classified by expert panel.

NOTE: Data reflect the percentage of items classified to the NAEP content framework at any level of specificity (field of science, subtopic, or 

specific objective). Multi-part items were treated as one item for classification purposes and only contribute one to the total. However, if multipart items covered multiple fields of science, then they contributed to the percentage for each field of science. Items classified to multiple fields 

of science were counted in each relevant category. Four TIMSS eighth-grade items were classified to multiple fields of science: three to Earth 

science and life science and one to physical science and Earth science. Detail may not sum to totals because of rounding or items classified to 

multiple fields of science. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 

and 2000 National Assessment of Educational Progress, 1996, 2000. 

Table 4. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the content 

Table 2. domains in the TIMSS science framework, by grade and survey 

Grade 4 Grade 8 TIMSS content domain 

NAEP1 TIMSS2 NAEP1 TIMSS2

Total number of items 144 141 197 177 

 Percentage distribution 

Physical science 35 33 31 42 

 Chemistry 15 13 11 17 

 Physics 19 20 21 25 

Life science 37 43 35 29 

Earth science 24 20 27 16 

Environmental science 3 5 7 13 

Classified to multiple domains 1 0 2 0 

Not classified to a content domain 1 0 1 0 

1

 NAEP items classified by expert panel.

2

 TIMSS items classified by TIMSS developers.

NOTE: Data reflect the percentage of items classified to the TIMSS content framework at any level of specificity (content domain, topic area, or 

objective). Multi-part items were treated as one item for classification purposes and only contribute one to the total. However, if multi-part items 

covered multiple content domains, then they contributed to the percentage for each content domain. Items classified to multiple content domains

were counted in each relevant category. Three eighth-grade NAEP items were classified to multiple content domains: two to chemistry and 

physics and one to life science and physics. One fourth-grade NAEP item was also classified to multiple content domains: chemistry and 

physics. Two NAEP Earth science items (one at eighth grade and one at fourth grade) were not classified to a content domain on the TIMSS 

framework. Detail may not sum to totals because of rounding or omitted items or items classified to multiple content domains. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks 

and Specifications 2003, 2nd Edition, 2003. 
23 

At the broad content area level, the classification of virtually all NAEP and TIMSS items was 

consistent with the basic definitions of the other framework. However, a closer examination of the 

degree to which the NAEP and TIMSS items could be classified on the other assessment framework 

(i.e., at the topic or specific objective levels) reveals that the level of content match is less universal 

at the more detailed levels of the framework, particularly when classifying TIMSS items to the 

NAEP framework (table 5). About one-third of TIMSS fourth-grade items and 40 percent of TIMSS 

eighth-grade items could not be mapped to one of NAEP’s specific objectives at any grade, and 

nearly 20 percent of TIMSS items in each grade could not be mapped to the NAEP framework at the 

topic level. Conversely, for NAEP items, the level of match to the TIMSS framework was greater, 

with over 80 percent classified at the specific objective level and over 90 percent classified at the 

topic level for both grades. This suggests that, for each assessment to varying degrees, there are 

specific topics and objectives represented in the items that may not be covered explicitly in the 

framework of the other assessment. In the case of TIMSS items, panelists’ comments bear this out, 

as they highlighted a number of the items for which there was no explicit topic included in the NAEP 

framework, such as those related to chemical change and reactions, burning, heat conductivity, or 

electrical circuits in physical science or to human health in life science. Some NAEP items that 

could not be placed on the TIMSS framework related to momentum and action-reaction in the 

physical science content area. In some areas, the lower level of specific match between TIMSS items 

and the NAEP framework reflects broader content coverage in TIMSS. In other areas, the 

differences are more a reflection of a lack of specificity in the NAEP framework about the content to 

be included, as there are items in the NAEP assessment covering content similar to these TIMSS 

items. More information about the differences in topics covered in NAEP and TIMSS is included in 

the sections devoted to content area comparisons (section 5). NAEP and TIMSS items illustrating 

some of the findings in each content area are included in appendix E. 

Table 5. Percentage of NAEP 2000 and TIMSS 2003 science items classified to the 

 other assessment framework at the topic or objective level, by grade and 

Table 3. survey 

Grade 4 Grade 8 

Level of content 

classification 

NAEP items to 

TIMSS framework 

TIMSS items to 

NAEP framework 

NAEP items to 

TIMSS 

framework 

TIMSS items to 

NAEP framework 

Topic level 97 83 93 82 

Objective level 83 67 82 60 

NOTE: Data reflect the percentage of items that were classified by the expert panel to the topic and objective levels of the other 

assessment framework in any content area. Items classified to multiple topics or objectives are considered to match those levels of 

classification and are counted only once. SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International 

Mathematics and Science Study (TIMSS) 2003 Assessment; U.S. Department of Education, National Assessment Governing Board, 

Science Framework for the 1996 and 2000 National Assessment of Educational Progress, 1996, 2000; and International Study Center, 

Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003.
24 

4.2. Grade Level 

The cross-classification data were used to examine the extent to which items from one 

assessment map to the same grade level framework of the other assessment. Figures 1-A and 1-B 

show the percentage of items in the NAEP and TIMSS assessments overall that were classified at 

each grade level in the other assessment framework. For these overall comparisons, the percentages 

at each grade level of the other’s assessment framework reflect items that were classified at the 

specific objective, topic, or broad content area level.22

The degree of consistency regarding what is considered fourth-grade science content and 

what is considered eighth-grade science content varies between the two assessments and across the 

two grades. Sixty-one percent of NAEP fourth-grade items were classified to the TIMSS fourthgrade framework, and 35 percent were classified to the TIMSS eighth-grade framework (figure 1-A). 

In other words, about one-third of the items included in the NAEP fourth-grade assessment would be 

more likely to be included in the TIMSS eighth-grade assessment. In contrast, 88 percent of TIMSS 

fourth-grade items were judged to be consistent with the NAEP fourth-grade framework and the 

remaining 12 percent with the eighth-grade framework (figure 1-B). As is discussed in the 

following sections, the NAEP fourth-grade items that were classified at the eighth grade in TIMSS 

go across all of the main content areas of physical science, life science, and Earth science. This is 

consistent with the different purposes for the assessments, with TIMSS reflecting a consensus across 

countries including many with a less formalized science curriculum in primary school than that in the 

United States. It also is consistent with the relatively high performance of U.S. fourth-graders on 

past TIMSS science assessments (Martin et al. 1997; Robitaille 1997). 

Compared to fourth grade, grade-level match appears to be more consistent for eighth grade, 

with 87 percent of NAEP items and 82 percent of TIMSS items classified at the corresponding grade 

level of the other assessment framework. The remaining TIMSS items were almost evenly split 

between those mapped to the fourth-grade and twelfth-grade NAEP framework. The remaining 

NAEP items were classified to the TIMSS fourth-grade framework. Comparisons within each 

content area presented in the following sections indicate that the TIMSS items classified as a higher 

grade level in NAEP—either fourth-grade items classified at eighth grade or eighth-grade items 

classified as twelfth grade—come primarily from the Earth science and environmental science 

categories, with some items also from life science. In contrast, there were no physical science items 

in TIMSS that were classified at a higher grade level in the NAEP framework. It should be noted 

that since the TIMSS 2003 framework includes only fourth and eighth grades, it was not possible for 

the panel to classify NAEP items at a grade level higher than the eighth grade on the TIMSS 

framework. 

 

22 The analyses for each of the content area comparisons in section 5 further examine the degree to which items 

match topics and objectives at particular grades. 
25 

While there were few comments recorded that suggested that the NAEP items exceeded the 

TIMSS eighth-grade descriptions, there were some physical science items that the panel judged to be 

generally consistent with the topics in the TIMSS eighth-grade framework but having some item 

characteristics (e.g., specific concept applications or terminology) that are somewhat more advanced 

than the eighth-grade specifications in TIMSS. The same situation was also found with some of the 

TIMSS eighth-grade physical science items, which although they were classified as most consistent 

with the descriptions in the NAEP eighth-grade framework, had aspects that were somewhat beyond 

that grade level but not consistent with the twelfth-grade specifications. These items (NAEP and 

TIMSS), which are treated as eighth-grade classifications in the overall results shown in figure 1-A, 

are discussed further in the content area comparison section on physical science (section 5.1). 
26 

Figure 1-A. Percentage distribution of NAEP 2000 science items classified at each 

Figure 1-A. grade level according to the TIMSS 2003 science framework, by grade 

61

10

35

87

0

20

40

60

80

100

Grade 4 Grade 8

NAEP items

Percent

Grade 4 Grade 8

Grade level according to TIMSS framework

NOTE: Data reflect expert panel classifications of grade level according to the TIMSS content framework at any level of specificity 

(content domain, topic area, or objective). Six NAEP items at each grade that the panel did not classify with respect to grade level on 

the TIMSS framework are not included. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment 

Frameworks and Specifications 2003, 2nd Edition, 2003. 
27 

Figure 1-B. Percentage distribution of TIMSS 2003 science items classified at each Figure 1-B. 

grade level according to the NAEP 2000 science framework, by grade 

88

8

12

82

10

0

20

40

60

80

100

Grade 4 Grade 8

TIMSS items

Percent

Grade 4 Grade 8 Grade 12

Grade level according to NAEP framework

NOTE: Data reflect expert panel classifications of grade level according to the NAEP content framework at any level of specificity 

(field of science, subtopic, or specific objective). Detail may not sum to totals because of rounding. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science 

Framework for the 1996 and 2000 National Assessment of Educational Progress, 1996, 2000. 
28 

 One consideration in comparing the grade match of NAEP and TIMSS is the fact that 

NAEP includes a portion of items that were developed to be used at multiple grades (crossgrade items): 28 items administered at both fourth and eighth grades and 48 items 

administered at both eighth and twelfth grades. These cross-grade items reflect 19 percent of 

the items in the fourth-grade assessment and 38 percent of the eighth-grade assessment items 

(data not presented in tables). As shown in table 6, the inclusion of cross-grade items 

generally does not explain the lack of grade correspondence found for the fourth-grade 

NAEP items. The cross-grade items administered at fourth and eighth grades are fairly 

evenly distributed with respect to their classification to fourth- and eighth-grade levels in the 

TIMSS framework, 50 percent and 43 percent, respectively. All of the cross-grade items 

administered at eighth and twelfth grades are classified to the TIMSS framework at the 

eighth grade, which again may be due in part to the fact that there is no twelfth-grade 

assessment in TIMSS. Example 1 in appendix E illustrates a NAEP cross-grade item 

administered at fourth and eighth grades that was classified at the eighth-grade level on the 

TIMSS framework. 

Table 6. Percentage of NAEP 2000 single-grade and cross-grade science items classified at each 

Table 6. grade level according to the TIMSS 2003 science framework 

NAEP item type 

Total Single-grade items Cross-grade items Grade level according to the 

TIMSS 2003 framework 

 Grade 4 Grade 8 

 Grade 4 

only 

 Grade 8 

only 

 Grades 

4 and 8 

 Grades 

8 and 12 

Grade 4 61 10 64 5 50 0 

Grade 8 35 87 33 92 43 100 

NOTE: Data reflect expert panel classifications of grade level to the TIMSS 2003 content framework at any level of specificity (content domain, 

topic area, or objective). Single-grade items are administered at one grade level; cross-grade items are administered at more than one grade (4 

and 8 or 8 and 12); totals reflect single-grade and cross-grade items included in the assessment at each grade level. Ten NAEP items (four fourthgrade, four eighth-grade, and two cross-grade) that the panel judged as having a poor fit to the TIMSS content framework were not assigned a 

grade level classification, and, therefore, are not included. Detail may not sum to totals because of rounding or omitted items. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000

Science Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003. 

4.3. Cognitive Domain 

All of the NAEP and TIMSS items were classified according the TIMSS 2003 definition of 

cognitive domain as a common metric for comparing the cognitive demands of the items in each 

assessment. Items were classified into the cognitive domain categories of factual knowledge, 

conceptual understanding, and reasoning and analysis according to the expert panel judgment of the 

primary cognitive skills and abilities required by the item. While a range of cognitive skills is 

defined in each of the TIMSS cognitive domains, there is an implicit hierarchy assumed by this 

classification system. 

Table 7 shows the percentage distribution of items in each assessment across the cognitive 

domains at fourth and eighth grades. At fourth grade, 53 percent of the NAEP items were classified 

as conceptual understanding; in addition, 6 percent of items were classified on the borderline of this 

category and another. Similar percentages of items (about 20 percent) were classified in the other 

two categories of factual knowledge and reasoning and analysis. In comparison to NAEP, TIMSS at 

grade 4 is more evenly distributed between factual knowledge and conceptual understanding items, 
29 

44 percent and 39 percent, respectively. Ten percent were classified as reasoning and analysis. As 

with NAEP, there were a number of TIMSS items (6 percent) that were classified on the borderline 

of the cognitive domain categories. 

At the eighth grade level, there were fewer items that the panel had difficulty classifying in a 

single category, with 3 percent “borderline” items for each assessment. NAEP and TIMSS both 

emphasize conceptual understanding items at the eighth grade, but TIMSS does so to a lesser degree 

(47 percent compared to 61 percent for NAEP). At both grades, TIMSS has a substantially greater 

proportion of items classified as factual knowledge than NAEP—44 percent at fourth grade and 31 

percent at eighth grade. In contrast, NAEP has less than 20 percent at both grades. Also, the 

proportion of TIMSS eighth-grade items classified as reasoning and analysis was twice that for the 

fourth grade (20 percent compared to 10 percent), while in NAEP the percentage classified as 

reasoning and analysis was about one quarter at both grade levels. 

For both NAEP and TIMSS, the percentage of items classified as factual knowledge is greater 

at fourth grade than at eighth grade, although this difference is greater in TIMSS than in NAEP (13 

percent and 7 percent, respectively). Both framework documents suggest that some differentiation 

between the grades would be expected on these process-related domains because of different 

expectations for cognitive development. 

Table 7. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across T 

 able 7. TIMSS cognitive domains, by grade and survey 

Grade 4 Grade 8 

Cognitive domain NAEP TIMSS NAEP TIMSS 

Classified in a single domain 

Factual knowledge 18 44 11 31 

Conceptual understanding 53 39 61 47 

Reasoning and analysis 22 10 25 20 

 

Classified on the border of two domains 

Factual knowledge/conceptual understanding 3 5 2 2 

Conceptual understanding/reasoning and analysis 3 1 1 1 

Not classified in a cognitive domain 0 1 0 0 

NOTE: Data reflect expert panel classifications of cognitive domain as defined in the TIMSS 2003 science framework. 

Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International 

Mathematics and Science Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston 

College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003.
30 

Table 8 shows the distribution of items across the cognitive domains broken down by main 

content area.23 Looking at the distributions, there are some differences across the content areas, 

including: 

• The difference between NAEP and TIMSS in terms of the relative proportion of items in 

factual knowledge versus reasoning and analysis items is most pronounced in physical 

science. In this content area, NAEP has about one-third at both grades classified as 

reasoning and analysis (36 percent at the fourth grade and 31 percent at the eighth grade) and 

less than 10 percent as factual knowledge (9 percent at the fourth grade and 5 percent at the 

eighth grade). In comparison, nearly half of fourth-grade (46 percent) and a quarter of 

eighth-grade items in TIMSS are factual knowledge (26 percent) and less than 20 percent are 

reasoning and analysis (13 percent at the fourth grade and 18 percent at the eighth grade). 

• While NAEP and TIMSS have the highest proportion of life science items classified as 

conceptual understanding, more NAEP items are likely to require reasoning and analysis

and more TIMSS items are likely to require factual knowledge. This is true for both grades.24

• The profile of cognitive domain for the Earth science items differs from the profiles of the 

other content areas, particularly at the fourth grade. While NAEP fourth-grade Earth science 

items retain a heavy emphasis on conceptual understanding (61 percent), the proportion of 

factual knowledge items is greater than in other content areas. For TIMSS, there is a higher 

percentage of reasoning and analysis items in Earth Science (21 percent at fourth grade and 

31 percent at eighth grade) than in the physical and life sciences. Earth science is the only 

content area where TIMSS has a higher proportion of reasoning and analysis items than 

NAEP. 

• The TIMSS environmental science items for eighth grade showed heavier emphasis on 

reasoning and analysis compared to the other TIMSS content areas, with 39 percent 

classified in this cognitive domain. Additionally, almost half of the items were classified as 

conceptual understanding. Thirteen percent were classified as factual knowledge—the 

lowest across all the TIMSS content areas. Some of these items address topics that, in 

NAEP, are considered twelfth-grade topics (data not shown in table 8). 

 

23 For the data reflected in table 8 and the remaining figures in this section, items classified on the borderline 

between two cognitive domains have been counted in the “higher” category (factual knowledge/conceptual

understanding as conceptual understanding; conceptual understanding/reasoning and analysis as reasoning and

analysis). This procedure was based on a policy set by the expert panel and reflects an assumed hierarchy indicated 

in the TIMSS framework. 

24 Another difference noted for the life science items but not shown in table 8 is that a higher number of items in life 

science were classified on the border between two cognitive domains, particularly at fourth grade (18 percent of 

NAEP and 10 percent of TIMSS fourth-grade items were not placed in a single category). 
31 

Table 8. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across 

Table 6. TIMSS cognitive domains, by grade and content area 

Grade 4 Grade 8 

Content area and cognitive domain NAEP TIMSS NAEP TIMSS 

Physical science items 

nnFactual knowledge 9 46 5 26 

nnConceptual understanding 56 41 65 56 

nnReasoning and analysis 36 13 31 18 

Life science items1 

nnFactual knowledge 21 45 19 41 

nnConceptual understanding 54 48 60 49 

nnReasoning and analysis 24 5 21 10 

Earth science items 

nnFactual knowledge 24 39 9 41 

nnConceptual understanding 61 39 65 27 

nnReasoning and analysis 15 21 27 31 

Environmental science items 

nnFactual knowledge † † † 13 

nnConceptual understanding † † † 48 

nnReasoning and analysis † † † 39 

† Not applicable. TIMSS does not have a reporting category for environmental science at the fourth grade. TIMSS fourth-grade 

environmental science items have been included in percentages for the content area subscales in which they were reported (life 

science or Earth science). NAEP does not include an environmental science content area at fourth or eighth grade. 1

 One TIMSS fourth-grade life science item (2 percent) not classified by the panel with respect to cognitive domain is not included. 

NOTE: Data reflect expert panel classifications of cognitive domain as defined in the TIMSS 2003 science framework. 

Items classified on the borderline between two cognitive domains have been counted in only one category for the purposes of this 

table (factual knowledge/conceptual understanding as conceptual understanding; conceptual understanding/reasoning and analysis

as reasoning and analysis). Detail may not sum to totals because of rounding or omitted items. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in 

International Mathematics and Science Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of 

Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003.

The NAEP items developed for each of the knowing and doing categories in the NAEP 

framework were compared with respect to the proportion of factual knowledge, conceptual 

understanding and reasoning and analysis. Figure 2 displays the percentage of items classified in 

each cognitive domain for the items from the NAEP categories of conceptual understanding, 

scientific investigation, and practical reasoning. 

While 72 percent of the items in the NAEP conceptual understanding category were also 

classified as conceptual understanding in the TIMSS framework, 19 percent were classified as 

factual knowledge and 10 percent as reasoning and analysis. Examination of the definitions of the 

NAEP framework reveals that the conceptual understanding category does include “facts and 

events” and “information and procedures” which are included in the TIMSS factual knowledge

domain. This NAEP category also includes “application of scientific knowledge in the engagement 

of practical tasks,” which would be consistent with the TIMSS definition of reasoning and analysis. 

Thus, the NAEP category of conceptual understanding is defined broadly to include factual 

knowledge as well as conceptual understanding and reasoning abilities, and this is reflected in the 

items developed based on the framework. 
32 

More than two-thirds of items in the NAEP scientific investigation category were classified 

as reasoning and analysis, which is consistent with the fact that many scientific inquiry skills such as 

hypothesizing, designing investigations, analyzing data, and drawing conclusions are included in this 

TIMSS cognitive domain. The remaining third of NAEP scientific investigation items were 

classified to the other TIMSS cognitive domains. This is consistent with a NAEP definition of 

scientific investigation that includes factual knowledge and conceptual understanding about the 

methods and tools of science. 

The NAEP practical reasoning items share characteristics with the TIMSS categories of 

conceptual understanding and reasoning and analysis, with somewhat less than half of items 

classified in the latter. The two TIMSS domains, conceptual understanding and reasoning and 

analysis, covered by the NAEP practical reasoning items are differentiated in terms of the 

complexity or familiarity of the problem situations, the level of hypothesizing, planning, decision 

making, and analysis required, as well as whether the problem solution requires consideration of 

multiple factors. 

Figure 2. Percentage distribution of NAEP 2000 science items across TIMSS 2003 cognitive 

Figure 2. domains, by NAEP knowing and doing categories 

19

11

72

21

57

10

68

43

0

20

40

60

80

100

Conceptual understanding Scientific investigation Practical reasoning

NAEP knowing and doing categories

Percent

Factual knowledge Conceptual understanding Reasoning and analysis

TIMSS cognitive domains

NOTE: Includes both fourth and eighth grade items combined. Classifications by knowing and doing categories provided by NAEP assessment 

developers; classifications by TIMSS cognitive domain made by expert panel according to definitions in the TIMSS 2003 framework. Items 

classified on the border of two cognitive domains were counted in the “higher” category for the purpose of this figure (factual 

knowledge/conceptual understanding as conceptual understanding; conceptual understanding/reasoning and analysis as reasoning and analysis). 

Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 

2003, 2nd Edition, 2003; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 

National Assessment of Educational Progress, 1996, 2000. 

.
33 

The expert panel’s classifications were compared to those of the assessment developers for 

the TIMSS items since both were based on the cognitive domain categories in the TIMSS framework. 

As shown in table 9, there is general agreement between the classifications, although the expert panel 

classified more items as factual knowledge (37 percent versus 34 percent) and conceptual 

understanding (47 percent versus 41 percent), whereas developers classified more items as reasoning 

and analysis (25 percent versus 16 percent). The more “stringent” classifications by the panel may 

be due in part to the nature of the task. Whereas the experts were reviewing items ex post facto, 

developers use the definitions not as a classifying system per se but as a tool or guide to ensure that a 

range of cognitive skills are represented in assessment items being developed. Differences may also 

be due in part to the fact that some items may require skills from more than one of these categories, 

and how these items were handled may have differed. Also, the TIMSS developers have a greater 

familiarity with the framework definitions and how they are applied in the development of items. 

Table 9. Percentage distribution of TIMSS 2003 science items across TIMSS cognitive 

Table 7. domains: comparison between classifications by expert panel and assessment 

developers 

Classification source TIMSS cognitive domains Classified by expert panel Classified by TIMSS assessment developers 

Factual knowledge 37 34 

Conceptual understanding 47 41 

Reasoning and analysis 16 25 

NOTE: Includes items at both fourth and eighth grades. Items classified by the expert panel on the border of two cognitive domains were counted 

in only one category for the purpose of this table (factual knowledge/conceptual understanding assigned to conceptual understanding; conceptual 

understanding/reasoning and analysis assigned to reasoning and analysis). Items classified by the developers to multiple cognitive domain

categories (e.g., if they were multi-part items) were counted in the “highest” category, assuming an implicit hierarchy of factual 

knowledge/conceptual understanding/reasoning and analysis. Classifications by cognitive domain were made according to the definitions in the 

TIMSS 2003 framework. Detail may not sum to totals because of rounding. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science Study 

(TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003.
34 

4.4. Scientific Inquiry 

The expert panel classified items with respect to whether or not they matched the TIMSS 

definition for measuring scientific inquiry skills.30 Figures 3 and 4 show the percentage of fourthgrade and eighth-grade items from NAEP and TIMSS that were classified by the expert panel as 

measuring scientific inquiry skills overall and broken down by main content area. For the NAEP 

items overall, the distribution of items is roughly the same in both grades, with nearly one-quarter of 

the items classified as measuring scientific inquiry skills. For TIMSS, a slightly higher percentage of 

eighth-grade items were classified as scientific inquiry than fourth-grade items (12 percent compared 

to 9 percent). The higher percentage of NAEP items classified as scientific inquiry is due in part to 

the inclusion of the hands-on performance tasks in NAEP which represent between 10 and 15 percent 

of the NAEP fourth- and eighth-grade assessments overall and about two-fifths of the items classified 

as scientific inquiry (data not shown). In comparison, the TIMSS assessment included two or three 

problem solving and inquiry tasks at each grade. About one-third of the items in the TIMSS problem 

solving and inquiry tasks were classified as measuring scientific inquiry compared to nearly all of the 

NAEP performance task items.31 There was considerable discussion among the panelists about the 

difficulty of assessing scientific inquiry through large-scale assessments such as NAEP and TIMSS. 

Looking at the results by content area, at fourth grade, the NAEP physical science items 

stand out as having the highest proportion of scientific inquiry items (36 percent), compared to 23 

percent in life science and 15 percent in Earth science. This may relate to the fact that the NAEP 

physical science area includes more performance tasks than other content areas. At the eighth grade, 

there is a more even distribution of scientific inquiry across the content areas in NAEP than in 

TIMSS, but the proportion in life science is the lowest. For TIMSS at both grade levels, the Earth 

and environmental science content areas have more scientific inquiry items (close to 20 percent or 

more) than the life and physical science content areas in both grades, with the percentage of life 

science items being particularly low (4 percent or less at both grades). 

Examples 2 and 3 in appendix E show items from the NAEP and TIMSS assessments, 

respectively, that were classified as measuring scientific inquiry skills. Both of these items require 

students to design and plan a scientific procedure. 

 

30 See appendix B for a description of the definition of scientific inquiry. 31 Both the NAEP hands-on tasks and the TIMSS problem solving and inquiry tasks include several items that are 

considered separately when determining the proportion of items in each assessment classified as scientific inquiry. 

Example tasks are show in appendix E. 
35 

Figure 3. Percentage of NAEP 2000 and TIMSS 2003 fourth-grade items classified as 

Figure 3. measuring scientific inquiry, by science content area 

24

36

23

15

11

3

18

9

0 10 20 30 40 50

Overall

Physical science

Life science

Earth science

Content area

Percent

NAEP TIMSS

 1

2

4

3

1

 The overall category refers to all items combined. 

2

 TIMSS physical science category combines items from both the chemistry and physics content domains. 3

 Two TIMSS fourth-grade items from the environmental science content domain are included in the life science reporting category. 4

 Five TIMSS fourth-grade items from the environmental science content domain are included in the Earth science reporting category. 

NOTE: Data reflect expert panel classifications of scientific inquiry as defined in the TIMSS 2003 framework. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks 

and Specifications 2003, 2nd Edition, 2003. 

.
36 

Figure 4. Percentage of NAEP 2000 and TIMSS 2003 eighth-grade items classified 

Figure 4. as measuring scientific inquiry, by science content area 

24

24

21

28

12

11

4

21

26

0 10 20 30 40 50

Overall

Physical science

Life science

Earth science

Environmental

science

Content area

Percent

NAEP TIMSS

1

2

3

1

 The overall category refers to all items combined. 

2

 TIMSS physical science category combines items from both the chemistry and physics content domains. 3

 NAEP does not include an environmental science category. 

NOTE: Data reflect expert panel classifications of scientific inquiry as defined in the TIMSS 2003 framework. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International 

Mathematics and Science Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston 

College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003. 

The panel’s classifications of scientific inquiry and cognitive domain may be used to further 

compare the nature of the scientific inquiry items in each assessment. The TIMSS definition of 

scientific inquiry includes both content- and skills-based components, and many of the skills and 

abilities associated with the inquiry process (formulating hypotheses, designing investigations, 

collecting, interpreting and analyzing data, and drawing conclusions) overlap the cognitive domain of 

reasoning and analysis. In addition to these process skills, the TIMSS scientific inquiry dimension 

also includes general knowledge and understanding about the methods and nature of science. To 

investigate the extent to which the scientific inquiry items are focused on process skills, the 

percentage of NAEP and TIMSS scientific inquiry items classified in each cognitive domain category 

were computed (table 10). NAEP and TIMSS have similar profiles, with the vast majority of 

scientific inquiry items classified as reasoning and analysis and slightly more than 10 percent as 

factual knowledge or conceptual understanding. 
37 

Table 10. Percentage distribution of NAEP 2000 and TIMSS 2003 scientific inquiry items 

Table 8. across TIMSS cognitive domains 

TIMSS cognitive domains NAEP scientific inquiry items TIMSS scientific inquiry items1

Factual knowledge 5 3 

Conceptual understanding 6 9 

Reasoning and analysis 88 86 

1

 One TIMSS item (3 percent) that the panel did not classify with respect to cognitive domain is not included. 

NOTE: Includes both fourth- and eighth-grade items. Data reflect classifications made by the expert panel according to definitions in the TIMSS 

2003 framework of scientific inquiry and cognitive domains. Items classified on the borderline between two cognitive domains have been 

counted in only one category for the purposes of this table (factual knowledge/conceptual understanding as conceptual understanding; 

conceptual understanding/reasoning and analysis as reasoning and analysis). Detail may not sum to totals because of rounding or omitted items. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003.

4.5. Item Format 

The items in the NAEP and TIMSS assessments were also compared with respect to the types 

of item formats used and their proportion of the assessments. Table 11 shows the percentage 

distribution of NAEP and TIMSS items by item format, including multiple choice and constructed 

response (short answer and extended response). Items can vary in difficulty and cognitive demand 

regardless of format, though extended constructed-response items can be particularly important in 

assessing students’ abilities to generate ideas and solutions and communicate their depth of scientific 

understanding. Including a variety of item types ensures that a range of knowledge and skills is 

being assessed. 

Table 11. Percentage distribution of NAEP 2000 and TIMSS 2003 science items across item 

Table 9. formats, by grade and survey 

Grade 4 Grade 8 Item format 

NAEP TIMSS NAEP TIMSS 

Multiple choice 49 65 48 62 

Constructed response 51 35 52 38 

Short answer 44 18 45 20 

Extended response 7 17 7 18 

NOTE: The breakdown of constructed-response items as short answer or extended response was provided by the assessment developers for the 

NAEP items. For the TIMSS items, the assignment was based on examination of the items and level of score points in the scoring guides in 

accordance with information provided by the TIMSS assessment developers—extended-response items reflect multi-part items and items that 

were scored with 3-level scoring rubrics. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; and International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS) 2003 Assessment. 

On both the fourth- and eighth-grade assessments, the NAEP items are roughly balanced 

between multiple-choice and constructed-response items, while nearly two-thirds of the TIMSS items 

are multiple-choice items. Of the constructed-response items, TIMSS has a substantially higher 

proportion of items defined as extended constructed response (about half). However, the definition 

and nature of extended-response items is not always the same across the two assessments. In both 
38 

assessments, the constructed-response items are scored with rubrics that are customized for each 

item. In TIMSS the short-answer items are scored with a 2-level rubric (Correct/Incorrect) and 

extended-response items with a 3-level rubric (Correct/Partial/Incorrect). In NAEP, the short-answer 

items may be scored with either a 2-level or 3-level rubric, while extended-response items are scored 

with a 4-level rubric (Correct/Partial/Minimal/Incorrect). Of particular note is that TIMSS includes a 

number of items at each grade level that ask for two responses to the same question (e.g., give two 

examples, write two reasons, etc.). While each response requires a short-answer and is scored with a 

2-level rubric, the items as a whole are treated as having three score levels (two correct answers, one 

correct answer, and no correct answers), and therefore, are classified as extended response in TIMSS. 

Items of this type in NAEP are also scored with a 3-level rubric, but are classified as short-answer. 

To compare the cognitive demand placed on students by the items of different formats, figure 

5 shows the percentage of multiple-choice and constructed-response items in each assessment that 

were classified as factual knowledge, conceptual understanding, or reasoning and analysis. More 

than half of the multiple-choice items in TIMSS were classified as factual knowledge compared to 22 

percent of NAEP multiple-choice items classified as such. Sixty percent of NAEP 

multiple-choice items were classified as measuring conceptual understanding compared to 36 

percent of TIMSS multiple-choice items. NAEP also had a somewhat higher proportion of multiplechoice items classified as reasoning and analysis (18 percent compared to 11 percent in TIMSS). 

Example 4 in appendix E illustrates a NAEP eighth-grade multiple-choice item classified as 

measuring conceptual understanding. Relative to the multiple-choice items, the distribution of 

constructed-response items across the cognitive domains was generally more comparable between the 

two assessments, although a larger proportion of the NAEP items were classified as reasoning and 

analysis (34 percent compared to 25 percent) and a smaller proportion as factual knowledge (6 

percent compared to 10 percent) and conceptual understanding (61 percent compared to 64 percent). 

 This section compared the assessments overall with respect to content coverage, grade level, 

cognitive domain, scientific inquiry, and item format. In the next section, more detailed comparisons 

of the content of the two assessments are made in each of the main content areas of physical science, 

life science, Earth science, and environmental science. 
39 

Figure 5. Percentage distribution of NAEP 2000 and TIMSS 2003 science items 

Figure 6. across TIMSS cognitive domains, by item format

Multiple-choice items 

 

Constructed-response items 

 

TIMSS cognitive domains 

Factual Knowledge Conceptual Understanding Reasoning and analysis

1

 One TIMSS multiple-choice item that the panel did not classify with respect to cognitive domain is not included. 

NOTE: Data reflect expert panel classifications of cognitive domain according to definitions in the TIMSS 2003 framework. Graphics 

for NAEP and TIMSS include both fourth- and eighth-grade items combined. Items classified on the border of two cognitive domains

were counted in the “higher” category for the purpose of this figure (factual knowledge/conceptual understanding as conceptual 

understanding; conceptual understanding/reasoning and analysis as reasoning and analysis). Detail may not sum to totals because of 

rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International 

Mathematics and Science Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston 

College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003. 

NAEP

22

60

18

TIMSS

53 36

11

1

NAEP

6

61

34

TIMSS

10

64

25

41 

5. NAEP/TIMSS Comparisons by Main Content Areas 

The overall comparisons section highlighted that, among other differences, NAEP and 

TIMSS have somewhat different emphases across the main science content areas. This section 

provides more detailed comparisons of the content coverage of the items for each of the NAEP and 

TIMSS science content area subscales. There are four sections, including: 

• physical science (which includes chemistry and physics for TIMSS); 

• life science; 

• Earth science; and 

• environmental science (which includes only TIMSS items). 

Each content area section includes 

• a comparison of the relevant parts of the content frameworks;32 

• an analysis of the level of match between the items from one assessment and the topics and 

objectives at particular grades in the other assessment framework; and 

• a comparison of how items are distributed across topics within the content areas defined by 

each framework. 

For these analyses, the NAEP and TIMSS items are divided according to subscale and then 

comparisons are made within the content areas that are the same or similar across the two 

assessments. Grade and content classification are examined simultaneously in the analyses for this 

section. For each content area, this section reports on the percentage of items that were classified to 

the other framework at the corresponding grade level or at another grade level. For items classified 

at the corresponding grade level, there are three levels of content match, including: 

• specific match (to a specific objective in the same content area); 

• general match (at the broader topic level in the same content area but not the objective level); 

and 

• match to another content area (at either the topic or objective level). 

For items classified at another grade level in the other assessment framework, there are two 

types, including: 

• lower grade (grade 8 items classified as grade 4 topics or subtopics); and 

 

32 Framework comparison exhibits and figures in this section list the topics included in each content area. For 

NAEP, topics are organized by major topic areas as described in section 2. Additional information about the 

specific objectives included for each of the topics is given in appendix A. 
42 

• higher grade (grade 4 items classified to grade 8 topics or subtopics or grade 8 TIMSS items 

classified to grade 12 NAEP topics or subtopics).33

This section also reports the percentages of items not classified to topics in the other 

assessment framework (i.e., those that could not be classified to a topic at a specific grade).34 The 

text in this section may refer to items classified to specific objectives, but this level of detail is not 

shown in the tables in the report. Subtopics are listed in appendix A and example items illustrating 

various features that are referenced in this section are shown in appendix E. 

5.1. Physical Science 

TIMSS has a greater emphasis on physical science than NAEP, particularly at the eighth 

grade. As discussed in section 4, thirty-one percent of NAEP items at both fourth and eighth grades 

are from the field of physical science (table 3). In TIMSS, 33 percent of fourth-grade items and 42 

percent of eighth-grade items are from the content domains of chemistry and physics (table 4). The 

results in the physical science section are based on 45 fourth-grade and 62 eighth-grade items in 

NAEP, and 46 fourth-grade and 74 eighth-grade items in TIMSS. 

Framework comparison in physical science 

The most visible structural difference between the NAEP and TIMSS physical science 

frameworks is that TIMSS includes separate content areas for chemistry and physics topics and 

reports achievement in these two areas on separate subscales at the eighth grade. Exhibit 3 shows a 

comparison of the physical science topics included in the NAEP and TIMSS science frameworks. 

NAEP includes topics related to chemistry in the major topic area of matter and its transformations, 

and a comparison of the frameworks at the topic level only would not necessarily reveal whether the 

specific content covered by the two assessments differed greatly. Closer examination of the topics 

and specific objectives included in each assessment, however, reveals a broader range of chemistry 

content covered in the TIMSS framework. Some topics within the content area of chemistry (e.g., 

acids and bases and chemical change) are explicit in the TIMSS framework but not the NAEP 

framework. As discussed in the next sections, the panel was not always able to classify some of the 

TIMSS chemistry items to topics within the NAEP framework. 

 

The NAEP and TIMSS frameworks include 14 and 12 topics, respectively, in physical 

science, with a slight emphasis on topics relating to physics. One particular difference panelists 

noted between the frameworks is that the TIMSS conception of physical science explicitly includes 

knowledge about the properties and uses of water, whereas the NAEP physical science framework 

does not refer to water specifically but rather to properties and uses of common materials. The NAEP 

framework does, however, include some specific objectives in Earth science related to the special 

properties of water. 

 

33 Because the NAEP 2000 framework is used to guide a twelfth-grade assessment and the TIMSS 2003 framework 

is not, the classification of grade 8 items to the twelfth-grade level is only applicable for the classification of TIMSS 

items to the NAEP framework. 

34 The method for determining grade-level match in this section differs somewhat from what was used for the 

overall comparisons in section 4.2. Overall comparisons of grade level include items classified at any level of 

content match (specific objective, topic, or broad content area). In this section, grade level was not assigned unless 

items could be classified to at least the topic level. 
43 

Exhibit 3. Physical science topics included in the NAEP 2000 and TIMSS 2003 science 

Exhibit 5. frameworks 

NAEP TIMSS 

Matter and its transformations 

Diversity of materials: classification, types, and 

particulate nature of matter 

Temperature and states of matter 

Properties and uses of materials: modifying properties 

and the synthesis of materials with new properties 

Resource management (grade 12 only) 

Energy and its transformations 

Forms of energy 

Energy transformations in living systems, natural 

physical systems, and artificial systems constructed by 

humans 

Energy sources and use, including distribution, energy 

conversion, and energy costs and depletion 

Motion 

Frames of reference 

Force and changes in position and motion 

Action and reaction (grades 8 and 12 only)

Vibrations and waves as motion 

General wave behavior (grade 12 only)

Electromagnetic radiation 

Interactions of electromagnetic radiation with matter 

Chemistry 

Classification and composition of matter 

Particulate structure of matter (grade 8 only)

Properties and uses of water 

Acids and bases (grade 8 only)

Chemical change 

Physics 

Physical states and changes in matter 

Energy types, sources, and conversions 

Heat and temperature 

Light 

Sound and vibration (grade 8 only)

Electricity and magnetism 

Forces and motion 

NOTE: Unless otherwise noted, all topics are intended for all grades (grades 4, 8, and 12 in NAEP and grades 4 and 8 in TIMSS). The 

number of grade-specific objectives and level of detail varies across topics and assessments. 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 National 

Assessment of Educational Progress, 1996, 2000; U.S. Department of Education, National Assessment Governing Board, Science 

Assessment and Exercise Specifications for the 1994 National Assessment of Educational Progress, 1994; and International Study 

Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003. 

Content and grade match in physical science 

At the fourth grade, there is a noticeable difference in the grade level correspondence 

between NAEP and TIMSS in the area of physical science. Although all the NAEP fourth-grade 

physical science items can be classified to topics in the TIMSS framework, these items embody some 

of what is considered eighth-grade content on the TIMSS framework (table 12). Whereas 80 percent 

of the TIMSS fourth-grade items were classified to the NAEP fourth-grade framework, nearly 40 

percent of NAEP fourth-grade items were classified to the eighth-grade framework in TIMSS, 
44 

distributed across nearly all the topics. The remaining NAEP fourth-grade items were classified to 

physical science topics in the TIMSS fourth grade framework at varying levels of specificity. While 

the grade-level match of TIMSS fourth-grade items to the NAEP framework is closer, there are about 

20 percent of items that were not clearly placed in NAEP framework topics at any grade. Example 5 

in appendix E presents a NAEP fourth-grade physical science item placed at the eighth-grade level 

on the TIMSS science framework. 

Table 12. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade physical 

Table 10. science items classified to the other science assessment framework, by level of 

Table 12. content/grade match 

Grade 4 Grade 8 

Level of content/grade match NAEP items to 

TIMSS framework

TIMSS items to 

NAEP framework

NAEP items to 

TIMSS framework 

TIMSS items to 

NAEP framework

Total number of physical science items 45 46 62 74

 Percentage distribution 

Classified as same grade 62 80 66 68

Specific match1

 in physical science 40 54 53 45

General match2

 in physical science 22 24 5 23

Match to another content area3

 0 2 8 0

Classified as another grade4

 38 0 13 3

Lower grade5

 † † 13 3

Higher grade6

 38 0 † 0

No classification to topics7

 0 20 21 30

† Not applicable. Grade 4 is the lowest grade in both frameworks; grade 8 is the highest grade in the TIMSS 2003 framework. 

1

 Includes items that were classified to an objective at the same grade. 

2

 Includes items that were classified to a topic but not to an objective at the same grade. 

3

 Includes items that were classified to a topic or objective at the same grade. 

4

 Includes items that were classified to a topic or objective in any content area at another grade. 

5

 Includes grade 8 items classified to grade 4 topics/objectives. 

6

 Includes grade 4 items classified to grade 8 topics/objectives or grade 8 TIMSS items classified to grade 12 NAEP topics/objectives. 

7

 Includes items that the panel did not classify to a topic at a specific grade level. 

NOTE: Data reflect the percentage of items classified by the expert panel at each level. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 

1996 and 2000 National Assessment of Educational Progress, 1996, 2000. 

More than half of TIMSS fourth-grade items (54 percent) and 40 percent of NAEP fourthgrade items map to one of the specific objectives in the other’s fourth-grade framework in physical 

science. For both assessments, many of the fourth-grade items that do not have a specific match in 

the other’s framework are items covering topics in chemistry. For example, there is no specific 

match for a number of TIMSS items relating to burning and chemical change. Conversely, several 

NAEP items from the topics of properties and uses of new materials and the diversity of materials

have only a general match in the TIMSS chemical change topic. There are also TIMSS items related 

to electrical circuits and heat conductivity which were not explicitly included in the NAEP 

framework, although there are some items in NAEP that deal with these topics. Example 6 in 
45 

appendix E shows a TIMSS fourth-grade item involving chemical change that was not classified to a 

topic in the NAEP framework. 

The proportion of TIMSS and NAEP eighth-grade physical science items classified at the 

corresponding grade level is similar—about two-thirds of the items are classified at the eighth-grade 

level of the other framework. For NAEP, however, this includes 8 percent of items that were 

classified to the TIMSS environmental science content area, mostly in the topic of conservation and 

use of resources. Also, 13 percent of NAEP eighth-grade items were classified to the TIMSS fourthgrade framework, concentrated in the chemistry topic relating to the classification of matter and 

spread across several topics in physics. 

Both assessments have a substantial number of eighth-grade items that were not mapped to 

topics in the other’s framework. For TIMSS, as in the fourth grade, this includes items related to 

burning and oxidation, heat conduction, and electrical circuits, which are topics not explicitly 

addressed in the NAEP framework. In addition, there are TIMSS chemistry items covering topics 

such as acids and bases that are included only in the TIMSS eighth-grade framework. Example 7 in 

appendix E shows a TIMSS eighth-grade item that was not classified to a topic in the NAEP 

framework. NAEP includes a number of items which, although the panel could assign them to a 

TIMSS topic, did not match well to the particular topic descriptions in the TIMSS eighth-grade 

framework. In many cases, this is because the panel thought the items addressed topics that were 

somewhat beyond the eighth-grade level descriptions in TIMSS. These included items involving 

acceleration, momentum, and induction as well as physics items that used terminology that was 

beyond the level specified for eighth-grade TIMSS, such as amplitude and frequency. While there 

were no TIMSS physical science items classified at the twelfth-grade level of the NAEP framework, 

there were some items considered to be somewhat above the eighth-grade level in the NAEP 

framework, in particular those dealing with the particulate structure of matter that require basic 

knowledge of subatomic particles, which is not included in NAEP at the eighth grade. These items 

were classified as “no classification to topics,” since they did not clearly fit the grade-level 

expectations for the topics in the other assessment framework. For the overall grade-level 

comparisons presented in section 4.2, these items were still treated as classified at the eighth grade. 

In the case of NAEP items, there was no higher grade possible in the TIMSS framework. For the 

TIMSS items, the panel did not believe that the items warranted a classification at the twelfth-grade 

level, despite being somewhat beyond the specifications for eighth grade. 

In terms of specificity of content match within physical science, a higher percentage of 

NAEP eighth-grade items can be mapped to TIMSS at the specific objective level than vice-versa (53 

percent compared to 45 percent). Many of the TIMSS items that have a general match within 

physical science but were not classified to a specific objective in the NAEP framework are physics 

items related to light as well as items covering some chemistry topics. Some NAEP items that do not 

have a specific match to the TIMSS physics framework are related to momentum and action/reaction. 

Item distribution across physical science topics 

Looking at the distribution of physical science items across the topics in the respective NAEP 

and TIMSS frameworks shows that NAEP and TIMSS emphasize different topics (see figures 6 and 

7). Using the NAEP framework as the basis for comparison, NAEP has relatively more focus at the 

fourth grade on the diversity of matter and properties and uses of materials (figure 6). In fact, more 

than one-third of NAEP items are classified to the latter topic area (36 percent). While TIMSS had 

almost 30 percent of items classified to the topic of properties and uses of materials (28 percent), it 
46 

also had a larger proportion of items (20 percent) related to temperature and states of matter than 

NAEP. NAEP had more coverage of energy and its transformations at fourth grade than TIMSS. A 

fourth-grade NAEP item from the energy and transformation topic area is shown in example 8 in 

appendix E. Within the physics topics, TIMSS has a larger proportion of items in the NAEP topic 

related to electromagnetic radiation, which focuses on basic properties and behavior of light at this 

grade (13 percent versus 7 percent). On the other hand, NAEP has items related to sound in the topic 

of vibrations and waves as motion, which is a topic not assessed at the fourth grade in TIMSS. This 

is confirmed by the distribution of items across topics in the TIMSS framework, which also shows 

relatively more TIMSS items in the topic of light (13 percent versus 9 percent) and none in the topic 

of sound and vibration (figure 7). Also, TIMSS has relatively more physical science items covering 

properties and uses of water at the fourth grade (7 percent versus 2 percent). Interestingly, NAEP 

has a higher percentage of fourth-grade items classified to the TIMSS topic of chemical change (20 

percent versus 11 percent), despite the fact that the panel felt that this was an area not explicitly 

covered in the NAEP framework. As noted previously, the NAEP items were judged as having only 

a general match to this TIMSS topic. 

As with the fourth-grade physical science assessments, the eighth-grade science assessments 

have some differences between them in the distribution of items across topics. According to the 

NAEP framework, there are more items in TIMSS than in NAEP that measure temperature and 

states of matter, there are more items in NAEP than in TIMSS related to properties and uses of 

materials, vibrations and waves as motion, and topics related to forces and motion (figure 6). 

Compared to TIMSS, NAEP also has three times the percentage of items in topics relating to energy 

and transformations. These differences in emphasis are also reflected in the corresponding topic 

areas in the TIMSS framework (figure 7). The comparisons based on the TIMSS framework reveal 

that TIMSS has a greater emphasis than NAEP on chemical change and particulate structure of 

matter at the eighth grade. Also, only TIMSS includes items related to acids and bases, a topic not 

included in the NAEP framework.
47 

Figure 6. Percentage of NAEP 2000 and TIMSS 2003 physical science items classified to 

Figure 7. physical science topics in the NAEP science framework, by survey and grade 

1

 NAEP items classified by NAEP developers.

2

 TIMSS items classified by expert panel. 

3 The topic used for classification is based on the NAEP Assessment Specifications document (NAGB 1994), which combines the two framework 

topics related to electromagnetic radiation (electromagnetic radiation and interactions of electromagnetic radiation with matter). 

NOTE: Topics may be abbreviated for graphical clarity. Two NAEP framework topics included for assessment only at the twelfth grade are not 

reflected in this figure, as no grade 4 or grade 8 items in either assessment were classified to these topics: resource management and general 

wave behavior. Percentages reflect the proportion of physical science items classified at either the topic level or the specific objective level at 

any grade level. Items that were classified to multiple topics were counted in all relevant topics. Bars not shown indicate that no items from that 

particular grade and assessment were classified to the topic. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 

and 2000 National Assessment of Educational Progress, 1996, 2000. 
48 

Figure 7. Percentage of NAEP 2000 and TIMSS 2003 physical science items classified to 

Figure 8. chemistry and physics topics in the TIMSS science framework, by survey and grade

1

 NAEP items classified by expert panel. 

2

 TIMSS items classified by TIMSS developers. 

NOTE: Percentages reflect the proportion of physical science items classified at either the topic level or the specific 

objective level at any grade level. Items that were classified to multiple topics were counted in all relevant topics. Bars not 

shown indicate that no items from that particular grade and assessment were classified to the topic. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International 

Mathematics and Science Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston 

College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003. 
49 

5.2. Life Science 

TIMSS places a greater emphasis on life science at the fourth grade than does NAEP. As 

discussed in section 4, in NAEP, 37 percent of fourth-grade items and 36 percent of eighth-grade 

items were from the field of life science (table 3). In TIMSS, 43 percent of fourth-grade items and 29 

percent of eighth-grade items were from the content domain of life science (table 4). The results in 

the life science section are based on 53 fourth-grade and 70 eighth-grade NAEP items, and 60 fourthgrade and 51 eighth-grade TIMSS items. 

Framework comparison in life science 

The NAEP and TIMSS frameworks for life science show some structural similarities. 

Exhibit 4 shows a comparison of the life science topics included in the NAEP and TIMSS science 

frameworks. Each includes seven topics at the fourth-grade level and eight topics at the eighth-grade 

level, with considerable overlap across the two frameworks at least at the topic level. Although 

labeled and structured somewhat differently, the topics included in both frameworks cover the broad 

major topic areas identified in NAEP—change and evolution, cells and their functions, organisms, 

and ecology. In both assessments, cells and their functions are not included at the fourth grade. 

Further examination of the specific objectives included in each framework (appendix A) reveal other 

differences between NAEP and TIMSS in terms of the content specified in the framework at each 

grade level. 

One of the notable differences in the frameworks is that human health is an explicit topic in 

the TIMSS framework, whereas this does not appear to be addressed even within any of the topics in 

NAEP (although there is some language related to disease within the objectives under life cycles). 

Panelists noted the difficulty of mapping TIMSS human health items to the NAEP framework and 

suggested they were a special type of item that is more common in TIMSS than in NAEP. Panelists 

also noted the difficulty of classifying items related to the biosphere and indicated that this topic was 

not well specified in either framework. However, there are some specific life science objectives 

related to the role of organisms in the flow of energy and cycling of materials through Earth’s surface 

in both NAEP and TIMSS topics related to ecology and ecosystems. In addition, there are some 

related objectives in the TIMSS environmental science content area.
50 

Exhibit 4. Life science topics included in the NAEP 2000 and TIMSS 2003 science frameworks

NAEP TIMSS 

Change and evolution 

Diversity of life on Earth 

Genetic variation within a species 

Theories of adaptation and natural selection 

Changes in diversity over time (grade 12 only) 

Cells and their functions1

Cells (grade 8 only) 

Cells as systems (grade 12 only) 

Information transfer (grade 12 only)

Energy transfer for the construction of proteins 

(grade 12 only)

Communication among cells (grade 12 only) 

Organisms 

Reproduction, growth, and development 

Life cycles 

Functions and interactions of systems within 

organisms 

Ecology 

Interdependence of life: populations, communities, 

and ecosystems 

Types, characteristics, and classification of living things 

Structure, function, and life processes in organisms 

Cells and their functions (grade 8 only)

Development and life cycles of organisms 

Reproduction and heredity 

Diversity, adaptation, and natural selection 

Ecosystems 

Human health 

1

 Subtopics reflect those included at eighth and/or twelfth grade in the major topic of cells and functions in the NAEP Assessment 

Specifications document (NAGB 1994). 

NOTE: Unless otherwise noted, all topics are intended for all grades (grades 4, 8, and 12 in NAEP and grades 4 and 8 in TIMSS). The 

number of grade-specific objectives and level of detail varies across topics and assessments. 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 National 

Assessment of Educational Progress, 1996, 2000; U.S. Department of Education, National Assessment Governing Board, Science 

Assessment and Exercise Specifications for the 1994 National Assessment of Educational Progress, 1994; and International Study Center, 

Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003.

Content and grade match in life science 

Table 13 shows the level of content and grade match between NAEP and TIMSS in life 

science. As with physical science, NAEP and TIMSS life science items have different levels of 

grade match to the other’s framework. Seventy-seven percent of the TIMSS fourth-grade items were 

classified at the fourth-grade level in the NAEP framework, and 62 percent of NAEP fourth-grade 

items were classified at the fourth-grade level in the TIMSS framework (table 13). On the other 

hand, 30 percent of NAEP fourth-grade items were considered to best match eighth-grade content 

according to the TIMSS framework, compared to 10 percent of TIMSS fourth-grade items classified 
51 

at the eighth-grade level in the NAEP framework. The NAEP items that were classified to the 

eighth-grade TIMSS framework relate almost exclusively to the topic areas of ecosystems and the 

types, characteristics and classifications of living things. Of the 10 percent of TIMSS items 

classified to the NAEP eighth-grade framework, half are in the topic of interdependence of life. 

Most of the NAEP and TIMSS fourth-grade items that were classified at the corresponding 

grade level had a specific match to a life science objective in the other’s framework (60 percent of 

NAEP items and 68 percent of TIMSS items). An additional 2 percent of NAEP items and 8 percent 

of TIMSS items had a general match in life science. 

Notably, none of the fourth-grade life science items from either assessment were classified to 

other content areas, though 8 percent of NAEP and 13 percent of TIMSS items did not match topics 

in any area of the other’s framework. Included in these are TIMSS items related to human health

(e.g., nutritious food sources) and some NAEP items in which the life science content was considered 

to provide a context for the problem but was not a primary part of the knowledge that was required to 

answer the item (e.g., data interpretation). 

Table 13. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade life science 

Table 13. items classified to the other science assessment framework, by level of content/grade 

Table 13. match 

Grade 4 Grade 8 

Level of content/grade match NAEP items to 

TIMSS framework 

TIMSS items to 

NAEP framework 

NAEP items to 

TIMSS framework 

TIMSS items to 

NAEP framework 

Total number of life science items 53 60 70 51 

 Percentage distribution 

Classified as same grade 62 77 71 76 

Specific match1

 in life science 60 68 59 57 

General match2

 in life science 2 8 10 16 

Match to another content area3

 0 0 3 4 

Classified as another grade4 30 10 11 12 

Lower grade5

 † † 11 0 

Higher grade6

 30 10 † 12 

No classification to topics7

 8 13 17 12 

† Not applicable. Grade 4 is the lowest grade in both frameworks; grade 8 is the highest grade in the TIMSS 2003 framework

1

 Includes items that were classified to an objective at the same grade. 

2

 Includes items that were classified to a topic but not to an objective at the same grade. 

3

 Includes items that were classified to a topic or objective at the same grade. 

4

 Includes items that were classified to a topic or objective in any content area at another grade. 

5

 Includes grade 8 items classified to grade 4 topics/objectives. 

6

 Includes grade 4 items classified to grade 8 topics/objectives or grade 8 TIMSS items classified to grade 12 NAEP topics/objectives. 

7

 Includes items that the panel did not classify to a topic at a specific grade level. 

NOTE: Data reflect the percentage of items classified by the expert panel at each level. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 

1996 and 2000 National Assessment of Educational Progress, 1996, 2000. 
52 

At the eighth-grade level, 71 percent of NAEP and 76 percent of TIMSS life science items 

were classified at the same grade level of the other’s framework. About 60 percent had a specific 

match to an objective in life science, and 10 percent of NAEP items and 16 percent of TIMSS items 

classified to the other’s framework had a general match at only the topic level. Slightly more than 10 

percent of both TIMSS and NAEP items were classified as a different grade level on the other’s 

framework. The off-grade TIMSS items were classified to the NAEP twelfth-grade framework (half 

related to the life science topic of interdependence of life) and the NAEP items were classified to the 

TIMSS fourth-grade framework (mostly in the ecosystem and structure, function and life processes

topics). These results, together with those above for fourth grade, indicate a lack of agreement across 

NAEP and TIMSS with respect to what is considered fourth-, eighth- or twelfth-grade content in the 

area of ecology and ecosystems. Although the content of this item may not necessarily be beyond 

the level of the NAEP eighth-grade life science items, the NAEP framework includes a specific 

objective related to the cycling of matter in ecosystems in the general topic area of ecology at the 

twelfth grade but not the eighth grade. Example 9 in appendix E shows a TIMSS eighth-grade life 

science item that was classified to a twelfth grade objective in the NAEP life science framework. 

Seventeen percent of NAEP items and 12 percent of TIMSS items at the eighth grade have no 

clear match to any topic in the other’s framework. As with fourth grade, these include TIMSS items 

in human health and NAEP items that are focused on data interpretations and investigations where 

the life science content is only a context for the item. Each assessment also has a small number of 

eighth-grade items that were placed in other content areas, including the Earth, environmental and 

physical sciences. Example 10 in appendix E shows a TIMSS eighth-grade human health item that 

was not classified to a NAEP topic. 

Item distribution across life science topics 

There are some similarities between the NAEP and TIMSS life science items based on their 

distribution across topics in the NAEP and TIMSS frameworks (see figures 8 and 9). Both NAEP 

and TIMSS include a larger proportion of fourth-grade than eighth-grade items covering the topics of 

life cycles; types, characteristics and classification of living things; and diversity of life. In 

comparison, there are higher proportions of eighth-grade items in both assessments covering the 

NAEP topic of functions and interactions of systems, and the related NAEP and TIMSS topics 

devoted to ecology and ecosystems. Also, neither assessment includes items related to cells and their 

functions at the fourth-grade level. 

Despite these similarities, there are some important differences between the NAEP and 

TIMSS assessments. There is considerably more emphasis in NAEP than in TIMSS on topics related 

to ecosystems or interdependence of life at both fourth and eighth grades. This difference is 

particularly noticeable in the TIMSS framework comparisons, with more than 30 percent of NAEP 

items compared to less than 20 percent of TIMSS items classified to the ecosystems topic area at 

either grade level (figure 9). A NAEP eighth-grade item from the interdependence of life topic is 

shown in Example 11 in appendix E. NAEP also has more items classified to the NAEP topic related 

to life cycles, particularly at eighth grade (13 percent compared to 2 percent of TIMSS items) (figure 

8). Based on the TIMSS framework, TIMSS includes relatively more items in reproduction and 

heredity. In fact, this topic (as defined by TIMSS) is addressed by 10 percent of fourth-grade items 

in TIMSS but none in NAEP (figure 9). In contrast, there are similar proportions of fourth-grade and 

eighth-grade items from each assessment classified to the closest NAEP topic of reproduction, 

growth and development. As expected, the TIMSS framework comparison shows substantially more 
53 

TIMSS items in the human health topic. Despite the lack of an explicit human health topic in the 

NAEP framework, NAEP still has some items devoted to human health as defined in TIMSS. Most 

of these items were from the life cycles topic in NAEP. 

One unusual aspect of the eighth-grade comparison is the apparent discrepancy between the 

NAEP and TIMSS distributions related to the topic cells and their functions. Based on the NAEP 

framework, NAEP has 14 percent of items in this topic area, while TIMSS has none (figure 8). 

However, using the TIMSS framework as the classification system, there are relatively more TIMSS 

items classified to the corresponding topic than NAEP (14 percent compared to 7 percent) (figure 9). 

These differences are due to the differences in the definitions of what is included in the cells and 

their functions topic in the two frameworks. In TIMSS, cellular processes (respiration and 

photosynthesis) are included within the objectives under cells and their functions, while in NAEP 

these understandings are included in the interdependence of life topic. NAEP, on the other hand, 

includes sexual and asexual reproduction in the cells topic, which is addressed in the TIMSS topic of 

reproduction and heredity. Also, the basic cellular make-up of living organisms is included in the 

cells and their functions topic in TIMSS but not NAEP. In the NAEP framework, there is a twelfthgrade objective related to the cell as the fundamental unit of living organisms under the major topic 

of organisms. 
54 

Figure 8. Percentage of NAEP 2000 and TIMSS 2003 life science items classified to life science 

Figure 9. topics in the NAEP science framework, by survey and grade 

1

 NAEP items classified by NAEP developers. 

2

 TIMSS items classified by expert panel. 3

 Structure-function is included in the framework topic of theories of adaptation and natural selection based on subtopics in the NAEP 

Assessment Specifications document used for item classifications (NAGB 1994). 

NOTE: Topics may be abbreviated for graphical clarity . Four NAEP framework topics included for assessment only at the twelfth grade are not 

reflected in this figure, as no grade 4 or grade 8 items in either assessment were classified to these topics: one from change and evolution

(changes in diversity over time) and three from cells and their functions (cells as systems, information transfer, and communication among cells). 

Percentages reflect the proportion of life science items classified at either the topic level or the specific objective level at any grade level. Items 

that were classified to multiple topics were counted in all relevant topics. Bars not shown indicate that no items from that particular grade and 

assessment were classified to the topic. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 

and 2000 National Assessment of Educational Progress, 1996, 2000. 
55 

Figure 9. Percentage of NAEP 2000 and TIMSS 2003 life science items classified to life science 

Figure 9. topics in the TIMSS science framework, by survey and grade 

1

 NAEP items classified by expert panel. 2

 TIMSS items classified by TIMSS developers. 

NOTE: Percentages reflect the proportion of life science items classified at either the topic level or the specific objective level at any grade level. 

Items that were classified to multiple topics were counted in all relevant topics. Bars not shown indicate that no items from that particular grade 

and assessment were classified to the topic. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks 

and Specifications 2003, 2nd Edition, 2003. 
56 

5.3. Earth Science 

Earth science items make up a larger proportion of the NAEP assessment than of the TIMSS 

assessment, at both grades. As discussed in section 4, in NAEP, 32 percent of fourth-grade items and 

33 percent of eighth-grade items were from the field of Earth science (table 3). In TIMSS, 20 percent 

of fourth-grade items and 16 percent of eighth-grade items were from the content domain of Earth 

science (table 4). The results in the Earth science section are based on 46 fourth-grade and 65 

eighth-grade NAEP items, and 28 fourth-grade and 29 eighth-grade TIMSS items. 

Framework comparison in Earth science 

The NAEP and TIMSS frameworks for Earth science are different in the breadth of coverage 

and the level of detail specified at the topic level (exhibit 5). The TIMSS Earth science framework 

has three broad topics focusing on the structure of the Earth, its processes, and its place in the solar

system and universe. The NAEP framework, on the other hand, contains a larger set of topics at a 

more detailed level (e.g., solid earth, water, air) and also includes a separate topic on Earth in space. 

These structural differences between the two frameworks do not necessarily translate to major 

differences in content specified across the full set of objectives included in each. The set of topics 

within each sphere (lithosphere, hydrosphere, and atmosphere) in the NAEP framework contain 

objectives related to both structures and features and to processes. Similarly, the broad topics in the 

TIMSS framework contain specific objectives related to lithosphere, hydrosphere, and atmosphere. 

As noted in the life science section, neither NAEP nor TIMSS has an explicit topic area related to the 

biosphere within their Earth science frameworks.
57 

Exhibit 5. Earth science topics included in the NAEP 2000 and TIMSS 2003 science 

Exhibit 7. frameworks 

NAEP TIMSS 

Solid earth (lithosphere) 

Composition of the Earth 

Forces that alter Earth's surface 

Rocks: their formation, characteristics, and uses 

Soil: its changes and uses 

Natural resources used by humankind 

Forces within the Earth (grades 8 and 12 only) 

Water (hydrosphere) 

Water cycle 

Nature of the oceans and their effects on water and 

climate 

Location, distribution, and characteristics of water, 

and its effect and influence on human activity 

Air (atmosphere) 

Composition and structure of the atmosphere, 

including energy transfer (grades 8 and 12 only) 

Nature of weather 

Common weather hazards

Air quality and climate 

Earth in space 

Setting of Earth in the solar system 

Setting and evolution of the solar system in the 

universe (grades 8 and 12 only) 

Tools and technology used to gather information 

about space 

Apparent daily motions of the Sun, Moon, planets, 

and stars 

Rotation of the Earth about its axis and the Earth’s 

revolution around the Sun 

Tilt of Earth's axis that produces seasonal variations 

in climate 

Earth History: Earth as a unique member of the 

solar system that may be approximated in other 

galaxies in the universe, and that evolved at least 4.5 

billion years ago. 

 

Earth's structure and physical features (lithosphere, 

hydrosphere, and atmosphere) 

Earth’s processes, cycles, and history 

Earth in the solar system and the universe 

NOTE: Unless otherwise noted, all topics are intended for all grades (grades 4, 8, and 12 in NAEP and grades 4 and 8 in TIMSS). The 

number of grade-specific objectives and level of detail varies across topics and assessments. 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 2000 National 

Assessment of Educational Progress, 1996, 2000; U.S. Department of Education, National Assessment Governing Board, Science 

Assessment and Exercise Specifications for the 1994 National Assessment of Educational Progress, 1994; and International Study Center, 

Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd Edition, 2003. 
58 

Content and grade match in Earth science 

Compared to the other content areas, there is less correspondence between the NAEP and 

TIMSS assessments with respect to what is included in Earth science, both in terms of grade level 

expectations and the specific content covered. At the fourth grade, the level of grade match for both 

NAEP and TIMSS Earth science items is not high (table 14), with less than 60 percent of Earth science 

items from each assessment classified to a fourth-grade topic in the other framework in Earth science 

or another content area. Thirty-seven percent of NAEP fourth-grade items and 25 percent of TIMSS 

fourth-grade items in Earth science were classified to the other assessment’s eighth-grade framework, 

with a number of these NAEP items classified to other TIMSS content areas. Another 18 percent of 

TIMSS items did not have a clear match to a topic anywhere in the NAEP framework at any grade 

level, including some items that panelists noted were related to water but did not match the description 

of the NAEP topic relating to the water cycle very well. Example 12 in appendix E illustrates a NAEP 

fourth-grade Earth science item classified to an eighth-grade environmental science topic on the 

TIMSS science framework, while example 13 illustrates a TIMSS fourth-grade Earth science item 

classified at the eighth-grade level on the NAEP science framework. Example 14 presents a TIMSS 

fourth-grade Earth science item not classified to a topic on the NAEP science framework. 

For the eighth-grade Earth science items, the patterns of grade and content match are different 

between the two assessments. Over 90 percent of the NAEP items were classified to the TIMSS 

eighth-grade framework, which is the highest percentage of NAEP items classified at the same grade 

level for any of the content areas. Of these, about 70 percent had a specific match to a TIMSS 

objective, and 14 percent were classified to topics in other content areas, primarily environmental 

science. Conversely, less than half of the TIMSS eighth-grade items were classified to the NAEP 

eighth-grade framework, which is the lowest degree of grade match of TIMSS items for any of the 

content areas, and only about one-third were classified to a specific objective. Similar proportions of 

the eighth-grade TIMSS items were classified to the fourth- and twelfth-grade NAEP framework, 

across multiple topics. Example 15 in appendix E shows a TIMSS eighth-grade Earth science item 

placed at the twelfth-grade level on the NAEP science framework. 
59 

Table 14. Percentage of NAEP 2000 and TIMSS 2003 fourth- and eighth-grade Earth 

Table 12. science items classified to the other science assessment framework, by level of 

Table 12. content/grade match 

Grade 4 Grade 8 

Level of content/grade match NAEP items to 

TIMSS framework 

TIMSS items to 

NAEP framework 

NAEP items to 

TIMSS framework

TIMSS items to 

NAEP framework 

Total number of Earth science items 46 28 65 29 

 Percentage distribution 

Classified as same grade 59 57 92 48 

Specific match1

 in Earth science 39 46 69 31 

General match2 in Earth science 13 11 9 14 

Match to another content area3

 7 0 14 3 

Classified as another grade4

 37 25 6 48 

Lower grade5

 † † 6 28 

Higher grade6 37 25 † 21 

No classification to topics7

 4 18 2 3 

† Not applicable. Grade 4 is the lowest grade in both frameworks; grade 8 is the highest grade in the TIMSS 2003 framework. 1

 Includes items that were classified to an objective at the same grade. 

2

 Includes items that were classified to a topic but not to an objective at the same grade. 

3

 Includes items that were classified to a topic or objective at the same grade. 

4

 Includes items that were classified to a topic or objective in any content area at another grade. 

5

 Includes grade 8 items classified to grade 4 topics/objectives. 

6

 Includes grade 4 items classified to grade 8 topics/objective or grade 8 TIMSS items classified to grade 12 NAEP topics/objectives. 

7

 Includes items that the panel did not classify to a topic at a specific grade level. 

NOTE: Data reflect the percentage of items classified by the expert panel at each level. Detail may not sum to totals because of 

rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 

and 2000 National Assessment of Educational Progress, 1996, 2000. 

Item distribution across Earth science topics 

The distribution of NAEP and TIMSS Earth science items across topics from each framework 

indicates that the two assessments generally emphasize different aspects of Earth science at the fourth 

and eighth grades (figures 10 and 11). Across the 18 topics in the NAEP Earth science framework, 

there are seven topics at the fourth grade and six topics at the eighth grade that are assessed by items in 

one assessment but not the other. In particular, items classified to NAEP topics of soil, the

composition of the Earth, the nature of the oceans and their effects, and the location, distribution, and 

characteristics of water are included in the fourth-grade assessment in the TIMSS but not until the 

eighth-grade assessment in NAEP. In contrast, the topic related to forces that alter Earth’s surface is 

reflected in the fourth-grade items in NAEP but only in eighth-grade items in TIMSS. Other topics 

related to interactions of society with the atmosphere and tools and technology to gather information

about space include only NAEP items at either grade level. 
60 

Based on the NAEP framework topics, the NAEP fourth-grade assessment has relatively 

greater emphasis on the water cycle, the nature of weather, and the tilt of the Earth’s axis, its rotation 

and revolution (figure 10). In contrast, TIMSS has a greater emphasis on the setting of the Earth in the 

solar system. Using the TIMSS framework as the basis for comparison, NAEP has a somewhat greater 

focus on Earth in the solar system and the universe, while TIMSS has a greater focus on Earth’s 

structure and physical features (figure 11). Examples 13 and 14 in appendix E present a TIMSS 

fourth-grade item from each of those topics. Neither item was classified to a grade 4 topic in the 

NAEP framework. 

Again, there are some notable differences in the distribution of NAEP and TIMSS items across 

the Earth science topics in the eighth grade. While both NAEP and TIMSS have an increased 

emphasis on the TIMSS topic of Earth’s processes, cycles, and history in their eighth-grade 

assessments compared to those at the fourth grade (figure 11), NAEP has somewhat more emphasis 

than TIMSS on forces that alter the Earth’s surface, while TIMSS has relatively more on the water 

cycle, which was a focus in the NAEP fourth-grade assessment (NAEP topics in figure 10). As noted 

earlier, NAEP also has a number of items that pertain to the composition of the Earth and to soil: its 

changes and uses, topics that are not explicitly addressed by TIMSS Earth science items at the eighth 

grade. This is consistent with a higher proportion of the NAEP eighth-grade assessment in the TIMSS 

topic of Earth’s structure and physical features. This topic area received more focus at the fourth 

grade in the TIMSS assessment. In comparison, TIMSS has a greater percentage of its eighth-grade 

assessment devoted to Earth in the solar system according to its own framework. Based on the NAEP 

framework, these topics are focused on the setting of the Earth in the solar system and the tilt of 

Earth’s axis, its rotation and revolution, which is a focus of NAEP at the fourth grade. Based on the 

grade-specific objectives in the TIMSS framework, however, this topic area deals primarily with the 

earth/moon/sun system at the fourth grade and is substantially broadened at the eighth-grade level. 

Only NAEP has any eighth-grade items classified to the topic of the setting and evolution of the solar 

system in the universe. 
61 

Figure 10. Percentage of NAEP 2000 and TIMSS 2003 Earth science items classified to Earth 

Figure 11. science topics in the NAEP science framework, by survey and grade 

1 NAEP items classified by NAEP developers. 

2 TIMSS items classified by expert panel. 

3 Air topics listed above are slightly different from the framework version: nature of weather includes common weather hazards; the combined topic 

of air quality and climate is separated into two topics—interactions of society with the atmosphere and climate. 4 Earth in space topics listed above are slightly different from the framework version: setting of Earth in the solar system includes apparent daily

motions of the Sun, Moon, planets, and stars; tilt of Earth’s axis, its rotation and revolution combines two topics—rotation of the Earth about its 

axis and the Earth’s revolution around the sun and tilt of earth’s axis that produces seasonal variations in climate. 

NOTE: Topics may be abbreviated for graphical clarity or revised to reflect the NAEP assessment specifications document (NAGB 1994). 

Percentages reflect the proportion of Earth science items classified at either the topic level or the specific objective level at any grade level. Items 

that were classified to multiple topics were counted in all relevant topics. Bars not shown indicate that no items from that particular grade and 

assessment were classified to the topic. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 

2000 National Assessment of Educational Progress, 1996, 2000. 
62 

Figure 11. Percentage of NAEP 2000 and TIMSS 2003 Earth science items classified to Earth 

Figure 12. science topics in the TIMSS science framework, by survey and grade 

1

 NAEP items classified by expert panel. 2

 TIMSS items classified by TIMSS developers. 

NOTE: Percentages reflect the proportion of Earth science items classified at either the topic level or the specific objective level at any grade level. 

Items that were classified to multiple topics were counted in all relevant topics. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP) 2000 

Science Assessment; International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition, 2003. 

5.4. Environmental Science 

Only TIMSS includes environmental science as a separate content domain in its framework and 

as a separate reporting category at the eighth-grade level. The TIMSS items that are discussed in this 

section for fourth grade are included in the life science and Earth science subscales for reporting 

purposes. This section examines the TIMSS environmental science items—7 at the fourth-grade level 

and 23 at the eighth-grade level. 

The TIMSS environmental science framework 

The TIMSS framework in environmental science focuses on three topics, including: 

• changes in population (included for the eighth grade only); 

 

• use and conservation of natural resources; and 

• changes in environments. 

Some of the objectives within the TIMSS environmental science topics are somewhat similar to 

topics and objectives included in the NAEP framework across the fields of science. These NAEP 

topics are predominantly in life science and Earth science, and sometimes are at the twelfth-grade level 

of the framework. One of the key differences, however, is that the TIMSS environmental science 

objectives (versus those similar objectives embedded in NAEP) focus more on the identification and 

possible solutions for global problems. 
63 

Content and grade match of TIMSS environmental science items to the NAEP framework 

The TIMSS fourth-grade environmental science items cover topics and objectives on both the 

fourth- and eighth-grade NAEP framework, primarily in Earth science. Forty-three percent of the 

items are classified to NAEP fourth-grade topics and 29 percent to eighth-grade topics (table 15). All 

the items that map to the fourth-grade NAEP framework have a specific match in some content area. 

However, two of the TIMSS fourth-grade items (29 percent) were not classified to any NAEP topic. 

These items were related to human use of natural resources and the impact of human activity on the 

environment. 

Table 15. Percentage of TIMSS 2003 fourth- and eighth-grade environmental science items 

Table 13. classified to the NAEP 2000 science framework, by level of content/grade match 

TIMSS items to NAEP framework 

Level of content/grade match 

Grade 4 Grade 8 

Total number of environmental science items 7 23 

Percentage distribution 

Classified as same grade 43 48 

Specific match1

 in any content area 43 22 

General match2

 in any content area 0 26 

Classified as another grade3 29 30 

Lower grade4 † 13 

Higher grade5 29 17 

No classification to topics6 29 22 

† Not applicable. Grade 4 is the lowest grade in the NAEP framework. 

1

 Includes items that were classified to an objective at the same grade. 

2

 Includes items that were classified to a topic but not to an objective at the same grade. 

3

 Includes items that were classified to a topic or objective at another grade. 

4

 Includes grade 8 items classified to NAEP grade 4 topics/objectives. 

5

 Includes grade 4 items classified to NAEP grade 8 topics/objectives or grade 8 items classified to NAEP grade 12 topics/objectives. 

6

 Includes items that the panel did not classify to a topic at a specific grade level. 

NOTE: Data reflect the percentage of items classified by the expert panel at each level. Detail may not sum to totals because of rounding. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science Study 

(TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 and 

2000 National Assessment of Educational Progress, 1996, 2000. 

Similar to the fourth-grade findings, the eighth-grade TIMSS environmental science items 

cross grade boundaries on the NAEP framework, with 48 percent mapping to the corresponding grade 

and 30 percent to the fourth- and twelfth-grade frameworks—the latter of which are primarily matched 

to NAEP topics in life science or physical science. The eighth-grade environmental science items 

show a different pattern from the fourth-grade environmental science items. The eighth-grade items 

are divided fairly evenly between those with a specific match and those that were classified only to the 

general topic level. There also are five eighth-grade items (22 percent) not classified to a NAEP topic. 

These items cover a range of TIMSS objectives and address both global and local environmental issues 

due to human or natural causes. Example 16 in appendix E illustrates a TIMSS eighth-grade 

environmental science item classified as a fourth-grade physical science item on the NAEP science 

framework. 
64 

Distribution of TIMSS environmental science items across TIMSS and NAEP science topics 

When classified to their own framework, the fourth-grade TIMSS environmental science items 

are split four to three between topics on the use and conservation of natural resources and changes in 

environments (table 16). The NAEP topics to which these items map (table 17) include Earth science 

topics that address relationships between humans and the world—such as oceans and their effects and

natural resources used by humankind—and the life science topic related to ecology (interdependence 

of life). Example 17 in appendix E presents a TIMSS fourth-grade environmental science item 

classified as life science on the NAEP science framework. 

At the eighth-grade level, equal proportions of items (43 percent) are included in the topics 

covering the use and conservation of natural resources and changes in environments, and a much 

smaller number are in the changes in population topic (table 16). Almost half the TIMSS 

environmental science items are classified to NAEP topics in Earth science, again focused on the 

interaction between humans and the environment (table 17). Example 18 in appendix E presents a 

TIMSS eighth-grade environmental science item classified as Earth science on the NAEP framework. 

A number of TIMSS items at the eighth grade also were classified to life science and physical science 

topics in NAEP (about 20 percent in each). The items classified as physical science deal with energy 

sources and uses of water and some were classified at the fourth-grade level in NAEP (Example 16). 

The items classified in life science involve predictions related to long-term environmental effects and 

changes in population. A number of these items were classified at the twelfth-grade level in NAEP. 

Table 16. Percentage distribution of TIMSS 2003 environmental science items 

Table 16. across environmental science topics in the TIMSS science framework, by grade 

 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and International Study Center, Lynch School of Education, Boston College, TIMSS Assessment 

Frameworks and Specifications 2003, 2nd Edition, 2003. 

TIMSS items to TIMSS framework Environmental science topic 

Grade 4 Grade 8

Changes in population 0 13

Use and conservation of natural resources 57 43

Changes in environments 43 43
65 

Table 17. Percentage of TIMSS 2003 environmental science items classified to topics in 

Table 18. the NAEP 2000 science framework in each field of science, by grade 

TIMSS items to NAEP framework NAEP topic within each field of science 

Grade 4 Grade 8

Earth science 

 Soil: its changes and uses 14 4

 Natural resources used by humankind 14 4

 Nature of the oceans and their effects 14 0

 Water: location, distribution, characteristics 14 22

 Interactions of society with atmosphere 0 17

Life science 

 Interdependence of life 14 22

Physical science 

 Temperature and states of matter 0 4

 Energy sources and uses of water 0 13

NOTE: Percentages reflect the proportion of TIMSS environmental science items classified by the expert panel at either the objective 

level (specific match) or topic level (general match) at any grade level. Items that were classified to multiple topics were counted in all 

relevant topics. Two fourth-grade items and five eighth-grade items that were not classified at the topic level are not included. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS) 2003 Assessment; and U.S. Department of Education, National Assessment Governing Board, Science Framework for 

the 1996 and 2000 National Assessment of Educational Progress, 1996, 2000. 

While the NAEP framework does not include a separate content category related to 

environmental science, some of the NAEP items from other fields of science were found by the panel 

to best match this part of the TIMSS framework (3 percent at grade 4 and 7 percent at grade 8, as 

shown in table 4). These NAEP items (14 items at eighth grade and 5 items at fourth grade) are not 

reflected in the percentage of items across TIMSS topics shown in the content area sections for Earth 

science, life science and physical science. The fourth-grade items all came from Earth science and 

cover topics related to human use of natural resources as well as the impact of both human activity and 

natural events on the environment. In contrast, the eighth-grade items came from all three of the 

NAEP fields of science and covered topics related to human use of natural resources and impact on the 

environment, energy resources, and genetic engineering. All of these NAEP items at both grades were 

classified to environmental topics in the TIMSS framework at the eighth-grade level. One of the 

NAEP items classified as environmental science is shown in example 12 in appendix E. 

This section provided a comparison of the two assessments in each of the main content areas of 

physical science, life science, Earth science, and environmental science. The last section includes a 

summary and conclusion of the findings of this comparison study. 

67 

6. Conclusion 

The content comparisons between NAEP and TIMSS reveal some key differences in the 

science topics covered, grade-level correspondence, and the characteristics of the item pools on other 

dimensions. All of these factors together may result in differences in student performance, and it is 

important to consider these differences when interpreting the results from the different assessments. 

Differences in the science content included in each assessment can be seen at both the framework 

level and in the pool of items developed based on these frameworks. Even in content areas where 

there is considerable overlap of the frameworks, such as in life science and Earth science, a closer 

examination of the topics and specific objectives covered by the items in each assessment reveals 

some important differences as well as similarities. In comparison to NAEP, whose framework was 

developed in the context of the U.S. system, the TIMSS framework reflects a consensus across many 

countries. Some of the differences in curricula across these countries are reflected in the frameworks 

and the differences in content between the two assessments. In particular, the inclusion in TIMSS of 

separate content areas in chemistry, physics, and environmental science results in broader topic 

coverage in some areas. While there is a considerable overlap in the topics included in some content 

areas, the items included in each assessment place different emphases at the topic level. As a result, 

both NAEP and TIMSS assessments may each contribute more information in some areas as well as 

some unique components to the larger picture of what students at fourth and eighth grades know and 

can do in science. In addition, the “hands-on” tasks in NAEP provide further complementary 

information to the pencil-and-paper portions of both assessments, enabling the measurement of 

student performance in this area of knowing and doing science. 

69 

References 

American Association for the Advancement of Science (AAAS). (1993). Benchmarks for Science 

Literacy. New York: Oxford University Press. 

Beaton, A., Martin, M., Mullis, I., Gonzalez, E., Smith, T., and Kelly, D. (1996). Science 

Achievement in the Middle School Years: IEA’s Third International Mathematics and Science 

Study. International Association for the Evaluation of Educational Achievement (IEA). Chestnut 

Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College. 

Binkley, M., and Kelly, D. (2003). A Content Comparison of the NAEP and PIRLS Fourth-Grade 

Reading Assessments (NCES 2003–10). U.S. Department of Education, National Center for 

Education Statistics. Washington, DC: U.S. Government Printing Office. 

Binkley, M., and Rust, K. (Eds.). (1994). Reading Literacy in the United States: Technical Report of 

the U.S. Component of the IEA Reading Literacy Study (NCES 94–259). U.S. Department of 

Education, National Center for Education Statistics. Washington, DC: U.S. Government 

Printing Office. 

Martin, M., Mullis, E., Beaton, A., Gonzalez, E., Smith, T., and Kelly, D. (1997). Science 

Achievement in the Primary School Years: IEA’s Third International Mathematics and Science 

Study. International Association for the Evaluation of Educational Achievement (IEA). Chestnut 

Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College. 

Martin, M., Mullis I., Gonzalez, E., Gregory, K., Smith, T., Chrostowski, S., Garden, R., and 

O’Connor, K. (2000). TIMSS 1999 International Science Report: Findings from IEA’s Repeat of 

the Third International Mathematics and Science Study at the Eighth Grade. International 

Association for the Evaluation of Educational Achievement (IEA). Chestnut Hill, MA: 

International Study Center, Lynch School of Education, Boston College. 

Mullis, I.V.S., Martin, M.O., Smith, T.A., Garden, R.A., Gregory, K.D., Gonzalez, E.J., 

Chrostowski, S.J., and O’Connor, K.M. (2003). TIMSS Assessment Frameworks and 

Specifications 2003, 2nd Edition. Chestnut Hill, MA: International Study Center, Lynch School of 

Education, Boston College. 

National Assessment Governing Board (NAGB). (1994). Science Assessment and Exercise 

Specifications for the 1994 National Assessment of Educational Progress. Washington, DC: 

Author. 

National Assessment Governing Board (NAGB). (2000). Science Framework for the 1996 and 2000 

National Assessment of Educational Progress. Washington, DC: Author. 

National Research Council (NRC). (1996). National Science Education Standards. Washington, DC: 

National Academy Press. 
70 

Neidorf, T.S., Binkley, M., Gattis, K., and Nohara, D. (forthcoming). Comparing Mathematics 

Content in the National Assessment of Educational Progress (NAEP), Trends in International 

Mathematics and Science Study (TIMSS), and Program for International Student Assessment 

(PISA) 2003 Assessments (NCES 2006-029). U.S. Department of Education. Washington, DC: 

National Center for Education Statistics. 

Nohara, D. (2001). A Comparison of the National Assessment of Educational Progress (NAEP), the 

Third International Mathematics and Science Study Repeat (TIMSS-R), and the Program for 

International Student Assessment (PISA) (NCES 2001–07). U.S. Department of Education, 

National Center for Education Statistics. Washington, DC: U.S. Government Printing Office. 

Robitaille, D. (Ed.). (1997). National Context for Mathematics and Science Education: An 

Encyclopedia of the Educational Systems Participating in TIMSS. Vancouver, BC: Pacific 

Educational Press. 
A-1 

Appendix A. Content Framework Summary Documents 

This appendix presents information about the NAEP 2000 and TIMSS 2003 science 

content frameworks used for item classifications at the expert panel meeting. 

Exhibit A-1 is the summary document that was used by the expert panel for the 

classification of items to the fields of science, major topics, subtopics, and specific objectives

in the NAEP 2000 science framework. 

Exhibit A-2 is the summary document that was used by the expert panel for the 

classification of items to the content domains, topic areas, and objectives in the TIMSS 2003 

science framework. 

These summary documents are based on the NAEP 2000 and TIMSS 2003 

framework and assessment specifications documents, but have been reformatted and adapted 

slightly to facilitate the classification process.
A-2 

Exhibit A-1. NAEP science framework and specifications summary: 2000 

LIFE SCIENCE (LS) 

A Change and evolution Grade(s) 

A1: Diversity of life on earth (variations between taxons) 4 8 12

A2: Genetic variation within a species (4) 8 12

A3: Theories of adaptation, including structure-function, and natural selection (4) 8 12

A4: Changes in diversity over time (evolution) • • 12

B Cells and their functions 

B1: Cells as systems • 8 12

B2: Information transfer in cells • • 12

B3: Energy transfer for the construction of proteins • • 12

B4: Communication among cells • • 12

C Organisms 

C1: Reproduction, growth and development 4 8 12

C2: Life cycles 4 8 12

C3: Functions and interactions of systems within organisms 4 8 12

D Ecology 

D1: The interdependence of life: populations, communities, and ecosystems 4 8 12

See notes at end of exhibit. 
A-3 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

LIFE SCIENCE (LS) 

A Change and evolution 

 A1 Diversity of life on earth (variations between taxons) 4 8 12 

Grade 4 Grade 8 Grade 12 

A1a: Using common (not scientific) names, 

identify/classify common plants and animals 

into the major taxonomic groups (e.g., 

mammal, bird, seed plant) based on their 

physical characteristics. 

A1b: Associate the physical appearance of a 

living thing with the environment it lives in. 

A1a: Classify common plants and animals into 

both the major taxonomic groups [tested at 

fourth grade] and finer taxonomic groups. 

A1b: Associate the physical appearance of a 

living thing with the environment it lives in. 

A1c: Relate associations of plants and animals 

with habitats. 

A1a: Recognize, classify, or enumerate key 

characteristics of major groups of animals and plants, 

e.g., algae, fungi, mammals, arthropods, mosses, etc., 

and explain their importance or know some 

significant information that pertains to them. 

A1b: Arrange the animal and plant phyla in the order 

in which they are thought to have evolved, e.g., 

worms prior to insects, reptiles prior to birds. 

A2 Genetic variation within a species (4) 8 12 

Grade 4 Grade 8 Grade 12 

A2a: Describe/identify random differences 

between individuals of the same kind of plant 

or animal. 

A2b: Recognize the males, females, and 

young of common species. 

A2c: Describe/identify similarities and 

differences between multiple offspring of 

same parents, and between parents and 

offspring. 

A2a: Describe/identify random differences 

between individuals of the same kind of plant or 

animal. 

A2b: Demonstrate an introductory knowledge of 

the genetic basis for variation within a species. 

A2a: Explain some of the mechanisms of genetic 

variation. 

A2b: Discuss ethical issues related to genetic 

variation in humans. 

A3 Theories of adaptation, including structure-function, and natural selection (4) 8 12 

Grade 4 Grade 8 Grade 12 

A3a: Identify major body structures of some 

common organisms. 

A3b: Relate the structure of body parts, as 

well as the overall shape of an organism, to 

function. 

A3a: Demonstrate awareness that adaptation 

may be to either the living or nonliving 

components of the environment. 

A3b: Demonstrate awareness that members of a 

population that survive long enough to 

reproduce may differ in some ways from 

members that do not survive to reproductive 

age, and that their offspring may inherit the 

anatomical, chemical, and/or behavioral 

characteristics that enabled the parents to reach 

reproductive age. 

A3c: Identify natural selection as the process by 

which organisms with characteristics that allow 

them to survive to reproductive age become 

better represented in future generations. 

A3d: Identify both adaptation and natural 

selection as processes which, operating over 

very long periods of time, have resulted in the 

diversity of plant and animal life present today. 

A3a: Identify, describe, and distinguish among 

mechanisms of evolution, i.e., adaptation, natural and 

artificial selection, transfer of genetic material 

through generations, and the appearance of new traits 

in individuals and through changes in genetic 

material (recombination and mutation). 

A3b: Identify factors such as the isolation of 

populations, the increase in genetic diversity, and the 

loss of the ability to interbreed and explain how they 

cause new species to arise. 

See notes at end of exhibit. 
A-4 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

LIFE SCIENCE (LS) 

A Change and evolution 

A4 Changes in diversity over time (evolution) · · 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. A4a: Describe patterns of evolution and 

consequences of evolution, e.g., adaptation and 

radiation, and identify evidence for evolution. 

A4b: Distinguish and explain the principal methods 

of evolution, e.g., sexual isolation, adaptation, 

artificial selection, mutation. 

B Cells and their functions 

B1 Cells (8) / Cells as systems (12) · 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. B1a: Describe observations of cells under the 

microscope. 

B1b: Explain, in a general way, the advantages 

of cellular interdependence vs. independence 

(multicellular animals vs. single-celled animals).

B1c: Describe, in general terms, the difference 

between asexual and sexual reproduction in 

cells and the advantages and disadvantages of 

each. [The stages of mitosis are not to be tested.] 

B1a: Demonstrate an understanding of the cell as a 

living system, including the physical structure and 

chemical activities of the cell. 

B2 Information transfer in cells · · 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. B2a: Explain, in general terms, the role DNA plays in 

controlling cell functions. (CU; S) 

B2b: Describe/identify examples of the role DNA 

plays in cell reproduction. (CU; S) 

B3 Energy transfer for the construction of proteins · · 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. B3a: Describe the transformations of matter and 

energy during photosynthesis and cellular respiration, 

and explain how cells use food for growth. 

B4 Communication among cells · · 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. B4a: Demonstrate an understanding of the roles of 

specialized cells in carrying out life functions, e.g., 

respiration, digestion, immune protection, nervous 

and hormonal control. 

B4b: Explain how cells communicate. Offer or 

identify several examples of intercellular 

communication, i.e., that what goes on in one cell 

may affect many. 

See notes at end of exhibit.
A-5 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

LIFE SCIENCE (LS) 

C Organisms 

C1 Reproduction, growth and development 4 8 12 

Grade 4 Grade 8 Grade 12 

C1a: Identify/explain that only adults can 

reproduce and that not all young live to 

adulthood. 

C1b: Identify examples (of species or larger 

taxons) of animals that produce hundreds of 

eggs or young, and animals that produce few 

or only a single offspring at any given time. 

C1c: Identify reproductive structures of plants 

and animals. 

C1a: Describe growth, development, and 

reproduction of the human organism. 

C1a: Demonstrate knowledge that a cell is the 

fundamental unit of a living organism. 

C1b: Demonstrate an understanding of the growth of 

multicellular organisms by cell growth and 

reproduction, i.e., mitosis (asexual) and meiosis 

(sexual) with corresponding transfer of one-half of 

DNA information from each parent. 

C2 Life cycles 4 8 12 

Grade 4 Grade 8 Grade 12 

C2a: Describe life cycles, including growth 

and metamorphosis, of familiar organisms. 

C2a: Identify some major influences on the 

human life cycle, such as diet and disease. 

C2a: Describe the life cycles of representative 

organisms that cause human diseases. 

C2b: Describe the use of technology in the 

prevention, diagnosis, and treatment of disease. 

C2c: Discuss ethical issues related to disease and 

disease prevention, such as: What is the relationship 

between hunger and disease? If the world could 

eliminate hunger, what might be the consequences? 

C3 Functions and interactions of systems within organisms 4 8 12 

Grade 4 Grade 8 Grade 12 

C3a: Identify major internal systems of both 

plants and animals and associate them with 

their functions. 

C3b: Demonstrate an introductory knowledge 

of interdependence of body systems, i.e., 

when something happens in one part of the 

body, it affects what goes on in other parts of 

the body. 

C3c: Describe/identify the relationship 

between the structure of a body part and its 

function. 

C3a: Demonstrate awareness that while different 

systems of the body have different functions, the 

functioning of each system affects other 

systems, e.g., describe/identify major organ 

systems of the human body, state their major 

functions, and describe some of their 

interactions. 

C3b: Demonstrate an understanding of the 

functions and interactions of organ systems to 

maintain a stable internal environment that can 

resist disturbance from within or without 

(homeostasis). 

C3a: Describe/identify major organ systems of the 

human body, state their major functions, and describe 

some of their interactions. 

C3b: Demonstrate an understanding of the functions 

and interactions of organ systems to maintain a stable 

internal environment that can resist disturbance from 

within or without (homeostasis). 

C3c: Answer questions about health issues based on 

knowledge of body systems and functions. 

C3d: Discuss ethical issues related to health and 

physical well-being. 

See notes at end of exhibit.
A-6 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

LIFE SCIENCE (LS) 

D Ecology 

D1 The interdependence of life: populations, communities, and ecosystems 4 8 12 

Grade 4 Grade 8 Grade 12 

D1a: Demonstrate introductory knowledge of 

photosynthesis, i.e., that green plants make 

their own food with sunlight, water and air 

(also under C 3). 

D1b: Describe basic requirements for living 

things, e.g., plants and animals need food for 

energy and growth. 

D1c: Describe positive and negative effects of 

human beings on the environment. 

D1a: Describe the flow of energy in ecosystems, 

i.e., identify plants as the source of food energy 

for all animals, describe the process by which 

plants use energy in sunlight to assemble food 

molecules from water and carbon dioxide, and 

describe the processes by which plants and 

animals break down food molecules to obtain 

food energy. 

D1b: Demonstrate an understanding of the 

patterns of relationships among populations and 

the effects of changes in one population in a 

food web on another, including the 

environmental effects of human activity. 

D1c: Design systems that encourage growth of 

certain plants and animals. 

D1a: Describe the complexity of ecosystems, i.e., 

how interactions between living and nonliving 

components of an ecosystem affect the functioning of 

that system as a whole and how ecosystems respond 

to natural and human changes in the environment 

(ecological succession). 

D1b: Make predictions about changes in the size or 

growth rate of a population using mathematical 

models. 

D1c: Describe matter cycles and energy flow 

(nutrient cycles) in ecosystems. 

D1d: Describe/predict how human activity impacts 

nutrient cycles in ecosystems, e.g., impact of building 

a dam on a selected ecosystem. 

See notes at end of exhibit.
A-7 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

A Matter and its transformations Grade(s) 

A1: Diversity of matter (materials): classification and types, particulate nature of matter, 

conservation of matter 

4 8 12

A2: Temperature and states of matter (physical changes) 4 8 12

A3: Properties and uses of materials: modifying properties, synthesis of materials with new 

properties 

4 8 12

A4: Resource management • • 12

B Energy and its transformations 

B1: Forms of energy 4 8 12

B2: Energy transformations in living systems, natural physical systems, and artificial 

systems constructed by humans 

4 8 12

B3: Energy sources and use, including distribution, conversion, costs, and depletion 4 8 12

C Motion 

C1: Frames of reference 4 8 12

C2: Force and motion 4 8 12

C3: Action and reaction • 8 12

C4: Vibrations and waves as motion (includes sound) 4 8 12

C5: General wave behavior • • 12

C6: Electromagnetic radiation, including its interactions with matter 4 8 12

See notes at end of exhibit. 
A-8 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

A Matter and its transformations 

A1 Diversity of matter (materials): classification and types, particulate nature of matter, conservation 

of matter 4 8 12 

Grade 4 Grade 8 Grade 12 

A1a: Classify/identify common objects and 

substances by physical characteristics such as a 

state of matter, texture, color, size, shape, 

hardness, and opacity. 

A1b: Use metric devices to measure linear 

dimensions of objects, weight, volume, 

temperature. 

A1c: Demonstrate understanding that changing 

the shape or physical state of an object does not 

change in weight (mass) of the object. 

A1a: Know that matter is composed of 

extremely small particles (atoms). 

Demonstrate awareness that atoms combine to 

form molecules and complex structures. 

[Factual information about the infrastructure 

of the atom is not to be tested]. 

A1b: Classify substances as elements, 

compounds, or mixtures. 

A1c: Describe/demonstrate how mixtures can 

be separated into their component parts, e.g., 

boiling, filtering, screening, use of tweezers, 

magnets, etc. 

A1d: Demonstrate an understanding of 

conservation of matter. 

A1e: Make appropriate measurements in 

situations requiring distinctions between mass 

and volume for liquids and gases and between 

linear dimensions and volume of solids. 

A1a: Distinguish/classify objects, both regular and 

irregular; pure substances, both elements and 

compounds; and mixtures, both homogeneous 

(solutions, liquids, and gases) and 

nonhomogeneous. 

A1b: Describe, measure, and compare substances in 

terms of mass, volume, and density/specific gravity.

A1c: Identify evidence that matter is composed of 

tiny particles (atoms, molecules), and that the 

particles are in motion (kinetic molecular theory). 

A1d: Define, describe, and contrast physical, 

chemical, and nuclear changes in molecular terms. 

A1e: Discuss the conservation of matter in physical, 

chemical, and nuclear changes. [Can also be tested 

under temperature states of matter, or energy and 

its transformations] 

A2 Temperature and states of matter (physical changes) 4 8 12 

Grade 4 Grade 8 Grade 12 

A2a: Identify/describe freezing, melting, 

boiling, evaporation, and condensation and the 

resulting changes in size and appearance of 

common substances, e.g., water/ice, wax, 

butter, sugar, etc. 

A2a: Discuss changes in the physical state of 

matter (solid, liquid, gaseous) in terms of the 

arrangement and motion of molecules and 

how these changes are related to temperature. 

A2b: Explain other common physical changes, 

such as dissolving and thermal expansion, in 

molecular terms. 

A2c: Identify/distinguish between chemical 

and physical changes in natural and 

technological systems and describe those 

changes in molecular terms. 

A2a: Discuss/identify the relation of physical states 

of matter to molecular energy. 

A2b: Discuss/identify the relation of physical 

changes in substances (i.e., melting, boiling, 

thermal expansion and contraction, compression 

and expansion under pressure, increase or decrease 

in density) to changes in the structural organization 

of the atoms or molecules of which they are 

composed. 

See notes at end of exhibit.
A-9 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

A Matter and its transformations 

A3 Properties and uses of materials: modifying properties, synthesis of materials with new properties 4 8 12 

Grade 4 Grade 8 Grade 12 

A3a: Identify useful properties of common 

materials. 

A3b: Sort/classify materials by useful 

properties such as magnetism, and conductivity, 

density, solubility. 

A3c: Describe the properties of magnets. 

A3a: Explain that all diverse substances that 

are found or made are arrangements of a small 

number of pure elements. 

A3b: Examine useful properties of materials, 

e.g., density, solubility, acidity, etc. 

A3c: Select/suggest appropriate uses of 

combinations of materials, considering their 

properties, e.g., alloys. 

A3a: Relate the physical properties (e.g., 

compressibility, structural rigidity) of pure 

substances in solid, liquid, and gaseous states to the 

structural organization of particles in the substance 

and their freedom of motion. 

A3b: Examine/Utilize useful properties of 

materials. 

A3c: Describe how common artificial materials are 

made, recognizing that substances can be designed 

to have certain properties, and that the addition of 

relatively small amounts of some substances can 

significantly alter the properties. 

A3d: Describe how common artificial materials are 

disposed of or recycled and discuss the 

technological and environmental issues involved in 

these processes. 

A4 Resource management • • 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. A4a: Discuss scientific, technological, 

environmental, and social issues involved in 

resource management. 

B Energy and its transformations 

B1 Forms of energy 4 8 12 

Grade 4 Grade 8 Grade 12 

B1a: Explain that energy appears in various 

forms, and that each form of energy has its own 

characteristics. 

B1b: Explain that heat is a form of energy often 

produced as a byproduct when one form of 

energy is converted to another form, and that 

reducing the amount of heat produced saves 

energy. 

B1c: Explain how any change in a system can 

be traced to the addition, transformation, or 

subtraction of energy from the system. 

B1a: Describe common forms of energy (e.g., 

light, heat, sound, kinetic, potential, food) 

found familiar systems. 

B1b: Recognize the relationship between 

magnetism and electricity. Demonstrate 

awareness that magnetism can produce 

electricity and that electricity can produce 

magnetism. 

B1a: In addition to the energy forms (heat, light, 

sound, motion, food) tested at eighth grade, 

identify/explain/describe the electromagnetic, 

electrical, chemical, mechanical, and nuclear forms 

of energy. 

B1b: Relate forms of energy with common uses of 

those energy forms in daily living. 

See notes at end of exhibit.
A-10 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

B Energy and its transformations 

B2 Energy transformations in living systems, natural physical systems, and artificial systems 

constructed by humans 4 8 12 

Grade 4 Grade 8 Grade 12 

B2a: Explain that energy is required when work 

is done on a system or when matter changes its 

form. 

B2b: Understand that energy transfer is 

essential to all living organisms. 

B2c: Identify common energy changes, e.g., a 

light bulb converts electricity into heat and 

light, a golf club converts human energy into 

the motion of the club and golf ball, etc. 

B2a: Describe common energy 

transformations found in familiar systems, 

both technological and natural. 

B2b: Demonstrate an understanding of total 

energy conservation when energy conversion 

takes place. 

B2a: Explain/describe energy transformations in 

natural and technological systems. 

B2b: Explain the relationship, in energy 

transformations, between loss of input energy as 

heat and lowered efficiency of the transformation. 

B2c: Demonstrate both qualitative and quantitative 

understanding of the conversation of energy in 

physical and chemical changes. 

B3 Energy sources and use, including distribution, conversion, costs, and depletion 4 8 12 

Grade 4 Grade 8 Grade 12 

B3a: Demonstrate awareness that the Sun is the 

ultimate source of most energy we use. 

B3b: Explain that energy is used to do 

mechanical work and identify examples. 

B3c: Explain the difference between renewable 

and nonrenewable resources. 

B3d: Describe a variety of ways that people use 

energy. 

B3e: Describe/list a variety of ways that energy 

can be stored, e.g., springs, batteries, body fat, 

plants. 

B3a: Discuss that energy conversions used in 

technology may produce undesirable side 

effects on the surroundings that should be 

minimized. 

B3b: Describe/identify limits on our abilities 

to conserve energy resources and recognize 

renewable and nonrenewable resources. 

B3c: Demonstrate awareness that balancing 

energy requirements in a technological society 

with resource limits also requires balancing 

scientific, technical, political, economic, and 

social factors. 

B3a: Describe specific chemical changes in the 

formation and use of fossil fuels. 

See notes at end of exhibit.
A-11 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

C Motion 

C1 Frames of reference 4 8 12 

Grade 4 Grade 8 Grade 12 

C1a: Demonstrate an understanding that 

everything moves. 

C1b: Demonstrate awareness that positions of 

things may be described in reference to 

something else. 

C1c: Demonstrate awareness that positions 

change (motion), and that monitoring changes 

of position in time yields information about 

speed. [Should be tested only qualitatively and 

in familiar contexts, such as running races] 

C1d: Describe and compare motions of 

common objects in terms of speed and 

direction. [Should be tested only qualitatively 

and in familiar contexts] 

C1a: Demonstrate awareness that everything 

is moving, and that descriptions of motion 

depend on the frame of reference being used, 

i.e., are relative to whatever point, object, or 

path is defined. 

C1b: Demonstrate qualitative and quantitative 

understanding of speed and velocity, i.e., 

relate speed to time and distance. 

C1c: Demonstrate awareness of the kinds of 

motion that are characterized by both speed 

and direction (velocity). 

C1a: Demonstrate awareness that everything is 

moving, and that descriptions of motion depend on 

the frame of reference being used. 

C1b: Isolate one component of a complex system 

and describe the motion of that component relative 

to the system as a whole. 

C1c: Represent and analyze motion both 

quantitatively and graphically. 

 C2 Force and motion 4 8 12 

Grade 4 Grade 8 Grade 12 

C2a: Relate changes in motion to the effects of 

forces, including gravitational force. 

C2a: Describe balanced and unbalanced forces 

and explain their impact on an object's state of 

motion, including state of rest. 

C2b: Describe qualitatively the relationship 

between the net force exerted on a body and 

the resultant acceleration due to the force. 

C2c: Identify frictional forces and describe 

their impact on motion. 

C2d: Define/Describe momentum. 

C2a: Demonstrate both qualitative and quantitative 

understanding of the relationship between the net 

force exerted on a body and the resultant 

acceleration due to the force (including 

gravitational). 

C2b: Demonstrate both qualitative and quantitative 

understanding of momentum. 

C3 Action and reaction • 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. C3a: Understand and give examples (colliding 

cars, billiard balls, swinging pendulum) of the 

principle that when X exerts a force on Y, Y 

exerts an equal force on X in the opposite 

direction. 

C3a: Demonstrate qualitative and quantitative 

understanding of pressure. 

C3b: Demonstrate qualitative understanding of the 

principle of equal and opposite forces. 

See notes at end of exhibit.
A-12 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

PHYSICAL SCIENCE (PS) 

C Motion 

C4 Vibrations and waves as motion (includes sound) 4 8 12 

Grade 4 Grade 8 Grade 12 

C4a: Identify some common vibrations and 

waves and describe their motion. 

C4b: Explain that vibrations may set up a 

traveling disturbance (wave) that spreads away 

from its source. 

C4a: Recognize that vibrations move through 

systems as waves. 

C4b: Describe vibration in terms of frequency 

and amplitude. 

C4c: Relate characteristics of sounds, i.e., 

loudness and pitch, to amplitude and 

frequency. 

C4d: Describe/demonstrate how sound travels 

through different materials, i.e., how sound is 

transmitted, reflected, and absorbed. 

C4e: Describe the motions of pendulums and 

other vibrating objects. 

C4a: Explain how apparent changes in wavelength 

provide information about relative motion. 

C4b: Identify the properties of sound, i.e., intensity, 

frequency, and harmonic content, and relate them to 

the effects, i.e., loudness, pitch, and quality. 

C4c: Explain how echoes occur. 

C5 General wave behavior • • 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. Not to be tested at this grade level. C5a: Describe wave behavior in terms of speed, 

wavelength, and frequency. [Can also be tested 

under vibrations and waves as motion] 

C5b: Describe the relationship between wavelength 

and how well a wave is transmitted, absorbed, 

reflected, or diffracted. [Can also be tested under 

electromagnetic radiation, below] 

C5c: Explain how media affect the motion of 

waves. 

C6 Electromagnetic radiation, including its interactions with matter 4 8 12 

Grade 4 Grade 8 Grade 12 

C6a: Describe basic properties of light, e.g., 

brightness and colors. 

C6b: Explain how light illuminates objects and 

how it causes them to cast shadows. 

C6c: Explain that things appear to have 

different colors because they reflect or scatter 

light of some colors more than others. 

C6a: Cite evidence of the wave properties of 

light, i.e., changing direction, bouncing off 

surfaces, spreading out, speeding up, slowing 

down, changing wavelength. 

C6b: Explain how objects are seen, i.e., light 

is reflected from them to the eye. 

C6c: Show awareness of electromagnetic 

spectrum, especially the visible region. 

C6d: Understand that objects and media 

reflect, transmit, or absorb light. 

C6e: Understand that lenses bend (refract) 

light to magnify, reduce, and/or project 

images. 

C6a: Demonstrate an understanding of 

electromagnetic waves, i.e., the electromagnetic 

spectrum. 

C6b: Describe the interaction of electromagnetic 

radiation with matter, i.e., that matter reflects, 

transmits, or absorbs light. 

See notes at end of exhibit.
A-13 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

A Solid Earth (lithosphere) Grade(s) 

A1: Composition of the Earth 4 8 12

A2: Forces that alter the Earth's surface 4 8 12

A3: Rocks: their formation, characteristics and uses 4 8 12

A4: Soil: its changes and uses 4 8 12

A5: Resources from the Earth used by humankind 4 8 12

A6: Forces within the Earth • 8 12

B Water (hydrosphere) 

B1: The water cycle 4 8 12

B2: Nature of the oceans and their effects 4 8 12

B3: The location of water, its distribution, characteristics, effect of and influence on 

human activity 4 8 12

C Air (atmosphere) 

C1: The composition and structure of the atmosphere, including energy transfer • 8 12

C2: The nature of weather 4 8 12

C3: Climate • 8 12

C4: Interactions of human society with atmosphere 4 8 12

D Earth in Space 

D1: The setting of the Earth in the solar system 4 8 12

D2: The setting and evolution of the solar system in the universe • 8 12

D3: Tools and technology that are used to gather information about space 4 8 12

D4: The tilt of the Earth's axis, its rotation about its axis, and its revolution around the Sun 4 8 12

D5: Earth history: Includes the ideas that the Earth is a unique member of our solar 

system; it may be approximated in other star systems and galaxies in the universe; and 

that it evolved at least 4.5 billion years ago. 

(4) 8 12

See notes at end of exhibit. 
A-14 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

A Solid Earth (lithosphere) 

A1 Composition of Earth 4 8 12 

Grade 4 Grade 8 Grade 12 

A1a: Classify substances as soil, sand or rock. 

A1b: Identify common geographic features of 

landscapes. 

A1a: Identify some elements and minerals 

that are abundant in the Earth's crust. 

A1b: Know that the interior of the Earth is hot 

and is composed of two major volumes: the 

mantle and the core. 

A1c: Understand that the components of the 

solid Earth undergo transformations over very 

long periods of time. 

A1d: Know that the solid Earth is composed 

of a finite number of elements. 

 

A1a: Describe the interior composition of the 

Earth, including its core, mantle, and crust. 

A1b: Explain how scientists use seismographic 

evidence in determining the structure and 

composition of the Earth's interior. 

A2 Forces that alter the Earth's surface 4 8 12 

Grade 4 Grade 8 Grade 12 

A2a: Describe/identify/explain basic facts 

about major features of the Earth's surface and 

natural changes in those features, e.g., 

volcanoes, glaciers. 

A2a: Use maps to identify surface features of 

the Earth. 

A2b: Demonstrate an understanding of the 

ways that gravity, forces in the interior of the 

Earth, weather, water, plants, animals, and 

civilizations impact the Earth's features. [Can 

also be tested under rocks, below.] 

A2a: Use maps to identify surface features of the 

Earth. 

A2b: Demonstrate understanding of ways that 

gravity, forces in the interior of the Earth, weather, 

water, plants, animals, and civilizations impact the 

Earth's features. 

A3 Rocks: their formation, characteristics and uses 4 8 12 

Grade 4 Grade 8 Grade 12 

A3a: Identify common rocks and minerals and 

explain how we can investigate what they are 

made of and how they form. 

A3a: Demonstrate an understanding of 

geologic and climatic changes over time (e.g., 

formation of rocks, minerals and fossils) and 

their use in explaining the age of the Earth. 

A3b: Describe characteristics of common 

rocks. 

A3a: Understand rock cycles. 

A3b: Discuss the uses of knowledge about rock 

cycles. 

A4 Soil: its changes and uses 4 8 12 

Grade 4 Grade 8 Grade 12 

A4a: Know some facts about the composition 

of soil. 

A4b: Recognize that plants grow in soil and 

that the soil provides both nutrients and 

support for plants. 

A4a: Know that soils are typed by the relative 

proportions of inorganic and organic 

components. 

A4b: Identify common soil conservation 

methods. 

A4a: Trace the factors in soil formation from lava 

to mature productive soils. 

A4b: Discuss problems associated with agriculture 

and lithosphere. 

See notes at end of exhibit.
A-15 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

A Solid Earth (lithosphere) 

A5 Resources from the Earth used by humankind 4 8 12 

Grade 4 Grade 8 Grade 12 

A5a: Identify Earth resources used in 

everyday life. 

A5b: Explain/identify that some Earth 

resources must be processed to make them 

useful. 

A5a: Identify Earth materials that people use 

and where to find them. 

A5b: Identify how and where we get energy 

from the Earth. 

A5c: Describe the reasons for using and not 

using (conserving) the Earth's resources. 

A5a: Identify Earth materials that people use and 

where to find them. 

A5b: Identify how and where we get energy from 

the Earth. 

A5c: Discuss issues related to the effects of human 

activity on Earth systems, including the importance 

of conservation and recycling, the limits of the 

Earth's natural resources, and the impact of 

technology on the use of these resources. 

A6 Forces within the Earth • 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. A6a: Describe how earthquake occurrences 

are recorded and note some positional 

regularities, e.g., locate earthquake belts of the 

Earth. 

A6b: Describe the effect of volcanic activity 

on short-term climate changes. 

A6c: Demonstrate/explain how to use a 

compass and explain how its workings are 

related to the Earth's magnetic field. 

A6a: Describe the interior of the Earth. 

A6b: Identify/explain the effect of the movement of 

crustal plates that are moving apart, that are moving 

together, and that are scraping against each other on 

the continental landforms (moving apart: sea floor 

spreading; moving together: mountain building and 

subduction; scraping against: earthquakes). 

A6c: Discuss continental drift. 

B Water (hydrosphere) 

B1 The water cycle 4 8 12 

Grade 4 Grade 8 Grade 12 

B1a: Describe the water cycle. Describe how 

water enters and leaves the atmosphere and 

explain the flow of water after precipitation. 

B1a: Relate common interactive cycles such 

as the water cycle (flow of water after 

precipitation), the nitrogen cycle, and the 

carbon cycle. 

B1a: Discuss how water or the lack of it influences 

climates. 

B1b: Describe the energy that drives the water 

cycle. 

B1c: Describe the natural and manmade events that 

may change the water cycle, and the effects that 

these changes may have on society. 

B2 Nature of the oceans and their effects 4 8 12 

Grade 4 Grade 8 Grade 12 

B2a: Know that most of the Earth's surface is 

covered by water. 

B2b: Locate the Atlantic and Pacific Oceans on 

a map or globe. 

B2c: Identify salt as the major difference 

between fresh and ocean waters. 

B2d: Describe some of the effects of oceans on 

climate. 

B2a: Identify the correct ratio between the 

area of the Earth's surface that is covered by 

oceans and the area that is dry land. 

B2b: Describe the motions of ocean waters 

and identify their causes. 

B2c: Identify/explain the effects of oceans on 

climate. 

B2a: Identify/describe resources provided by the 

oceans (food, minerals, recreation, transportation). 

B2b: Identify/describe the effects of the oceans on 

global climates. 

B2c: Identify/describe the effects of human activity 

on the oceans (waste dumps, oil spills, global 

warming). 

See notes at end of exhibit.
A-16 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

B Water (hydrosphere) 

B3 The location of water, its distribution, characteristics, effect of and influence on human activity 4 8 12 

Grade 4 Grade 8 Grade 12 

B3a: Know that water exists not only on the 

Earth's surface but beneath the Earth's surface 

as well. 

B3b: Understand that water exists in three 

physical states on Earth. [Can also be tested 

under water cycle, above]

B3c: Identify land features that are shaped by 

water and design simple models to illustrate 

the action of water in shaping the Earth's 

surface. [Can also be tested under forces that 

alter the Earth's surface, above] 

B3d: Identify common sources of water and 

explain/describe people's dependence on water 

for daily activities. 

B3e: List some ways we can use water more 

wisely. 

B3a: Demonstrate awareness that water is 

found in the air, on the surface of the Earth, 

and under the ground. 

B3b: Describe the three physical states of 

water on Earth and the conditions under which 

they exist. [Can also be tested under Physical 

Science] 

B3c: Describe physical characteristics of 

lakes, oceans, and rivers and their relation to 

habitats for plant and animal life. [Can also be 

tested under Life Science.] 

B3d: Describe what properties make water 

special.

B3e: Discuss some common problems related 

to water, e.g., availability, purity, relationship 

to supply and demand, effects of 

overpopulation on availability, and quality of 

water. 

B3f: Describe ways scientists explore the 

water environment. 

B3a: Trace the movement of water in the air, on the 

surface of the Earth, and under the ground. 

B3b: Describe the three physical states of water on 

Earth, conditions under which they exist, and their 

uses. [Can also be tested under Physical Science]

B3c: Describe physical characteristics of lakes, 

oceans, and rivers and their relation to habitats for 

plant and animal life. [Can also be tested under 

Life Science]

B3d: Discuss some common problems that concern 

water, e.g., availability, purity, relationship to 

supply and demand, effects of overpopulation on 

availability and quality of water. 

B3e: Describe ways scientists explore the water 

environment. 

C Air (atmosphere) 

C1 The composition and structure of the atmosphere, including energy transfer • 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. C1a: Describe/identify the composition and 

physical characteristics of the atmosphere. 

C1b: Describe the structure of the Earth's 

atmosphere. 

C1a: Know that the chemical balance of the 

atmosphere is maintained by cycles. 

C1b: Identify/describe levels or layers of the 

atmosphere. 

C1c: Discuss energy transfer in the atmosphere. 

See notes at end of exhibit.
A-17 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

C Air (atmosphere) 

C2 The nature of weather 4 8 12 

Grade 4 Grade 8 Grade 12 

C2a: Describe the different phenomena of 

weather conditions, such as clouds, 

temperature, and types of precipitation. 

C2b: Explain the relationship of seasonal 

changes to weather conditions. 

C2c: Describe weather changes, list ways of 

measuring them, and offer simple explanations 

for how the weather changes. 

C2d: Use and make weather charts and 

temperature measurements. 

C2a: Describe/identify interactions of water 

and the Sun's heat energy in cycles of 

precipitation and evaporation. 

C2b: Explain weather-related phenomena such 

as thunderstorms, tornados, hurricanes, 

cyclones, drought, or acid precipitation. 

C2c: Describe/explain patterns of changing 

weather. 

C2d: Describe/use weather measurement 

methods such as charts, barometers, or 

anemometers. 

C2a: Describe patterns of circulation of air around 

the planet and explain how they affect weather 

conditions. 

C2b: Explain and predict general weather patterns, 

based on knowledge of phenomena that determine 

weather. 

C3 Climate • 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. C3a: Define climate as the long-term average 

weather of a region and describe/explain 

climates of major Earth regions. 

C3b: Demonstrate an understanding of how 

relatively small changes in global 

temperatures can have dramatic effects on the 

Earth's climate. 

C3a: Identify major climatic zones of the world 

(polar, middle latitude, and tropical) and identify 

relationships of weather conditions to these zones. 

C3b: Explain changes in climate over long periods 

of time, i.e., atmospheric phenomena and long-term 

effects. 

C3c: Explain how scientists monitor atmospheric 

events over time. 

C4 Interactions of human society with atmosphere 4 8 12 

Grade 4 Grade 8 Grade 12 

C4a: Describe ways human beings protect 

themselves from adverse weather conditions. 

C4b: Identify/explain some effects human 

activities have on weather. 

C4a: Describe ways human beings protect 

themselves from adverse weather conditions. 

C4b: Identify/explain/discuss some effects 

human activities have on weather and 

atmosphere. 

C4a: Explain the impact of human activities on the 

atmosphere, demonstrating knowledge of the 

products of air pollution. 

C4b: Discuss the causes of air pollution and 

possible solutions in relation to their consequences 

and tradeoffs. 

See notes at end of exhibit.
A-18 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

D Earth in Space 

D1 The setting of the Earth in the solar system 4 8 12 

Grade 4 Grade 8 Grade 12 

D1a: Explain how the Earth differs from the 

Sun and the Moon. 

D1b: Explain how the Earth relates to the Sun: 

periodicity, seasons, night and day. [May also 

be tested under tilt of the Earth, below]

D1a: Describe the location and motion of the 

Earth and its Moon in the solar system. 

D1b: Identify the other planets in the solar 

system and describe their motions, as well as 

the motions of moons and comets. 

D1c: Describe the characteristics of Earth and 

other planets in the solar system in terms of 

their abilities to support life. [Can also be 

tested under Earth history]

D1a: Describe the energy that reaches Earth from 

the Sun, identify the process by which the Sun 

generates its energy, and relate the Sun's energy to 

its effects on Earth. 

D1b: Discuss issues related to efficient use of the 

Sun's energy. 

D2 The setting and evolution of the solar system in the universe • 8 12 

Grade 4 Grade 8 Grade 12 

Not to be tested at this grade level. D2a: Describe the current scientific theory of 

the origin and evolution of the Earth and the 

solar system. 

D2b: Know the characteristics of stars (large, 

hot, energy radiators). 

D2c: Know that a galaxy is a group of many 

stars and that the Sun is one of many stars in 

our galaxy. 

D2a: Describe how stars form and how they 

produce energy. 

D2b: Demonstrate awareness of the similarity of 

materials and forces found everywhere in the 

universe. 

D2c: Demonstrate awareness of 

observations/theories of the structure and evolution 

of the universe. 

D3 Tools and technology that are used to gather information about space 4 8 12 

Grade 4 Grade 8 Grade 12 

D3a: Describe the use of telescopes, satellites, 

space shuttles, etc., to gather information about 

space. 

D3a: Show familiarity with common 

instruments used to study objects in space, 

e.g., telescopes, spectrographs. 

D3b: In general terms, explain the role of 

gravity in orbital motion of both natural and 

man made satellites; i.e., gravitational forces 

hold objects in orbit around another object; 

gravity must be overcome in order to achieve 

or leave orbit; or the Earth is held in place 

around the Sun by gravitational force. 

D3c: Describe space explorations and the 

knowledge gained from them. 

D3a: Understand that the majority of the 

knowledge about the universe is based on analysis 

of the electromagnetic radiation reaching the Earth.

D3b: Discuss "spinoffs" from space exploration 

that affect daily life. 

See notes at end of exhibit.
A-19 

Exhibit A-1. NAEP science framework and specifications summary: 2000—Continued 

EARTH SCIENCE (ES) 

D Earth in Space 

D4 The tilt of the Earth's axis, its rotation about its axis, and its revolution around the Sun 4 8 12 

Grade 4 Grade 8 Grade 12 

D4a: Explain some consequences of the Earth's 

rotation about its axis. 

D4b: Explain the relation of seasonal 

phenomena to the revolution of the Earth, 

spinning on a tilted axis, around the Sun. 

D4a: Explain phases of the Moon in terms of 

relative positions of the Earth, Moon, and Sun.

D4b: Explain/describe the relation of seasonal 

phenomena to the revolution of the Earth, 

spinning on a tilted axis, around the Sun. 

D4c: Explain how the apparent motions of the 

stars have been used in navigation. 

D4d: Explain the association of time 

measurement with celestial motions. 

D4a: Explain the relationship of the tides to Sun 

and Moon positions. 

D4b: Use latitude and longitude in determining 

locations on the Earth's surface. 

D5 Earth history: Includes the ideas that the Earth is a unique member of our solar system; it may 

be approximated in other star systems and galaxies in the universe; and that it evolved at least 4.5 

billion years ago. 

(4) 8 12 

Grade 4 Grade 8 Grade 12 

D5a: Identify/cite evidence that the Earth is 

very old. 

D5a: Identify/cite evidence that the Earth is 

very old. 

D5b: Identify phases in the Earth's history, 

including cooling from a molten state, 

development of the atmosphere, and the 

collection of water in liquid phase on the 

Earth's surface. 

D5c: Discuss the role of life in the Earth's 

history. 

D5a: Identify/cite evidence that the Earth is very 

old. 

D5b: Identify phases in the Earth's history, 

including cooling from the molten state, 

development of the atmosphere, and the collection 

of water in liquid phase on the Earth's surface. 

NOTE: In each field of science (life science, physical science, etc.) major topics are identified by capital letters (A, B, C,...), subtopics are identified by 

numbers (1, 2, 3,...), and specific objectives are identified by lowercase letters (a, b, c,...). Subtopics can be assessed at those grade levels indicated by 4, 

8, and 12 on the right side of the table. Parentheses around a grade level indicate that a topic may be introduced at a simple level at that grade. If a 

subtopic should not be addressed at a specific grade level, it is indicated by a dot (•). 

SOURCE: U.S. Department of Education, National Assessment Governing Board, Science Assessment and Exercise Specifications for the 1994 National 

Assessment of Educational Progress, 1994; and U.S. Department of Education, National Assessment Governing Board, Science Framework for the 1996 

and 2000 National Assessment of Educational Progress, 2000. 
A-20 

Exhibit A-2. TIMSS science framework and specifications summary: 2003 

LIFE SCIENCE (LS) Grade(s) 

1 Types, Characteristics, and Classification of Living Things 4 8 

 

2 Structure, Function, and Life Processes in Organisms 4 8 

3 Cells and Their Functions • 8 

 

4 Development and Life Cycles of Organisms 4 8 

 

5 Reproduction and Heredity (4) 8 

6 Diversity, Adaptation, and Natural Selection 4 8 

7 Ecosystems 4 8 

 

8 Human Health 4 8 

See notes at end of exhibit. 
A-21 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

LIFE SCIENCE (LS) 

1 Types, Characteristics, and Classification of Living Things 4 8 

Grade 4 Grade 8 

1a: 

 - Explain differences between living and nonliving things 

based on common features (movement, basic needs for 

air/food/water, reproduction, growth, response to stimuli). 

1b: 

 - Compare and contrast physical and behavioral 

characteristics of humans and other major groups of 

organisms (e.g., insects, birds, mammals, plants). 

 - Identify/provide examples of plants and animals belonging 

to these groups. 

1a: 

 - State the defining characteristics that are used to differentiate 

among the major taxonomic groups and organisms within these 

groups. 

 - Classify organisms on the basis of a variety of physical and 

behavioral characteristics. 

2 Structure, Function, and Life Processes in Organisms 4 8 

Grade 4 Grade 8 

2a: 

- Relate major body structures in humans and other 

organisms (plants and animals) to their functions 

(e.g., digestion takes place in the stomach, plant roots absorb 

water, teeth break down food, bones support the body, lungs 

take in oxygen). 

2b: 

- Demonstrate knowledge of bodily actions in response to 

outside conditions (e.g., heat, cold, danger) and activities 

(e.g., exercise). 

2a: 

 - Locate major organs in the human body. 

 - Identify the components of organ systems. 

 - Compare/contrast organs and organ systems in humans and 

other organisms. 

2b: 

- Relate the structure and function of organs and organ systems to 

the basic biological processes required to sustain life (sensory, 

digestive, skeletal/muscular, circulatory, nervous, respiratory, 

reproductive). 

2c: 

- Explain how biological actions in response to specific 

external/internal changes work to maintain stable bodily 

conditions (e.g., sweating in heat, shivering in cold, increased 

heart rate during exercise). 

See notes at end of exhibit. 
A-22 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

LIFE SCIENCE (LS) 

3 Cells and Their Functions • 8 

Grade 4 Grade 8 

Not Assessed. 3a: 

- Describe the cellular make-up of all living organisms (both 

single-celled and multi-cellular). 

- Demonstrate knowledge that cells carry out life functions and 

undergo cell division during growth/repair in organisms. 

- Demonstrate knowledge that tissues, organs, and organ systems 

are formed from groups of cells with specialized structures and 

functions. 

3b: 

- Identify cell structures and some functions of cell organelles (cell 

wall, cell membrane, nucleus, cytoplasm, chloroplast, 

mitochondria, vacuoles), including a comparison of plant and 

animal cells. 

3c: 

- Provide a general description of the process of photosynthesis 

that takes place in plant cells (the need for light, carbon dioxide, 

water, and chlorophyll, production of food, and release of oxygen).

3d: 

- Describe the process of respiration that takes place in plant and 

animal cells (the need for oxygen, breaking down of food to 

produce energy, and release of carbon dioxide). 

4 Development and Life Cycles of Organisms 4 8 

Grade 4 Grade 8 

4a: 

- Trace the general steps in the life cycle of organisms 

(birth, growth and development, reproduction, and death). 

- Know and compare life cycles of familiar organisms (e.g., 

humans, butterflies, frogs, plants, mosquitoes). 

4a: 

- Compare and contrast how different organisms grow and develop 

(e.g., humans, plants, birds, insects). 

5 Reproduction and Heredity (4) 8 

Grade 4 Grade 8 

5a: 

- Recognize that plants and animals reproduce with their 

same kind to produce offspring with features that closely 

resemble those of the parents. 

5a: 

- Explain that reproduction (asexual or sexual) occurs in all living 

organisms and is important for the survival of species. 

- Compare/contrast biological processes in asexual and sexual 

reproduction in general terms (e.g., cell division to produce an 

identical offspring versus combination of egg and sperm from 

female/male parents to produce offspring that are similar but not 

identical to either parent). 

- State advantages and disadvantages of each type of reproduction.

5b: 

- Relate the inheritance of traits to the passing on of genetic 

material contained in the cells of the parent(s) to their offspring. 

- Distinguish inherited characteristics from physical/behavioral 

features that are acquired/learned. 

See notes at end of exhibit. 
A-23 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

LIFE SCIENCE (LS) 

6 Diversity, Adaptation, and Natural Selection 4 8 

Grade 4 Grade 8 

6a: 

- Associate physical features and patterns of behavior of 

plants and animals with the environments in which they live.

- Identify/provide examples of certain physical or behavioral 

characteristics of plants/animals that make them better suited 

for survival in different environments and explain why (e.g., 

camouflage, color change, fur thickness). 

6a: 

- Relate the survival/extinction of different species to variation in 

physical/behavioral characteristics in a population and 

reproductive success in changing environments. 

6b: 

- Demonstrate knowledge of the relative time major groups of 

organisms have existed on the earth (e.g., humans, reptiles, fish, 

plants). 

- Describe how similarities and differences among living species 

and fossils provide evidence of the changes that occur in living 

things over time. 

7 Ecosystems 4 8 

Grade 4 Grade 8 

7a: 

- Explain that all plants and animals need food to provide 

fuel for activity and material for growth and repair. 

- Understand that plants need the sun to make their own 

food, while animals consume plants and/or other animals as 

food. 

7b: 

- Explain relationships in a given community (e.g., forest, 

tidepool) based on simple food chains, using common plants 

and animals and predator/prey relationships. 

[Assessment objectives related to the effects of human behavior on 

environments are described in the Environmental Science section.]

7a: 

- Demonstrate knowledge of the flow of energy in an ecosystem 

(the role of photosynthesis and respiration and the storage of 

food/energy products in organisms). 

- Identify different organisms as producers, consumers, and 

decomposers. 

- Draw/interpret food pyramids or food web diagrams. 

7b: 

- Describe the role of organisms in the cycling of materials 

through the earth’s surface (e.g., oxygen/carbon dioxide, water) 

and the decomposition of organisms and recycling of elements 

back into the environment. 

7c: 

- Discuss the interdependence of populations of organisms in an 

ecosystem in terms of the effects of competition and predation. 

- Identify factors that can limit population size (e.g., disease, 

predators, food resources, drought). 

- Predict effects of changes in an ecosystem (e.g., climate, water, 

supply, food supply, population changes, migration) on the 

available resources and the balance among populations. 

See notes at end of exhibit. 
A-24 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

LIFE SCIENCE (LS) 

8 Human Health 4 8 

Grade 4 Grade 8 

8a: 

- Recognize ways that common communicable diseases 

(e.g., colds, influenza) are transmitted. 

- Identify signs of health/illness and some methods of 

preventing and treating illness. 

8b: 

- Describe ways of maintaining good health, including the 

need for a balanced/varied diet, identification of common 

food sources (e.g., fruits and vegetables, grains), and the 

effect of personal habits on health (e.g., using sunscreen, 

preventing injury, personal hygiene, exercise, drug, alcohol, 

and tobacco use). 

8a: 

- Describe causes of common infectious diseases, methods of 

infection/transmission, prevention, and the importance of the 

body’s natural resistance (immunity) and healing capabilities. 

8b: 

- Explain the importance of diet, hygiene, exercise, and lifestyle in 

maintaining health and preventing illness (e.g., heart disease, 

diabetes, skin cancer, lung cancer). 

- Identify the dietary sources and role of nutrients in a healthy diet 

(vitamins, minerals, proteins, carbohydrates, fats). 

See notes at end of exhibit. 
A-25 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

CHEMISTRY (CH) Grade(s)

1 Classification and Composition of Matter 4 8 

 

2 Particulate Structure of Matter • 8 

 

3 Properties and Uses of Water (4) 8 

 

4 Acids and Bases • 8 

 

5 Chemical Change (4) 8 

See notes at end of exhibit. 
A-26 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

CHEMISTRY (CH) 

1 Classification and Composition of Matter 4 8 

Grade 4 Grade 8 

1a: 

- Compare/classify/order different objects and materials on the 

basis of observable physical properties (e.g., weight/mass, 

shape, volume, color, hardness, texture, odor, taste, magnetic 

attraction). 

1b: 

- Identify some properties of metals and relate them to their use 

(e.g., conduct heat and electricity, are hard, are shiny, can be 

molded). 

1c: 

- Identify/describe mixtures on the basis of physical 

appearance. 

- Demonstrate understanding that mixtures can be separated 

based on the observable properties of their parts (e.g., particle 

size, shape, color, magnetic attraction). 

1d: 

- Give examples of some materials that will dissolve in water 

and some that will not. 

- Identify common conditions that increase the amount of 

material that will dissolve or the speed at which materials 

dissolve (hot water, stirring, small particles). 

1a: 

- Classify/compare substances on the basis of characteristic 

physical properties that can be demonstrated or measured (e.g., 

density, thermal/electrical conductivity, solubility, melting/boiling 

point, magnetic properties). 

1b: 

- Recognize that substances may be grouped according to similar 

chemical and physical properties. 

- Describe properties of metals that distinguish them from other 

common substances (nonmetals). 

1c: 

- Differentiate between pure substances (elements and 

compounds) and mixtures (homogeneous and heterogeneous) on 

the basis of their formation and composition, and provide/identify 

examples of each (solid, liquid, gas). 

1d: 

- Select/describe physical methods for separating mixtures into 

their components (e.g., filtration, distillation, sedimentation, 

magnetic separation, flotation, dissolution). 

1e: 

- Define solutions in terms of substance(s) (solid, liquid, or gas 

solutes) dissolved in a solvent. 

- Apply knowledge of the relationship between 

concentration/dilution and the amounts of solute/solvent and the 

effect of factors such as temperature, stirring, and particle size. 

2 Particulate Structure of Matter • 8 

Grade 4 Grade 8 

Not Assessed. 2a: 

- Describe the structure of matter in terms of particles, including 

molecules as combinations of atoms and atoms as being composed 

of subatomic particles (electrons surrounding a nucleus containing 

protons and neutrons). 

3 Properties and Uses of Water (4) 8 

Grade 4 Grade 8 

3a: 

- Identify common uses of water in each of its forms (e.g., 

solvent, coolant, heat source). 

3a: 

- Identify water as a compound with molecules composed of one 

oxygen atom and two hydrogen atoms. 

- Relate the behavior/uses of water to its physical properties (e.g., 

melting point and boiling point, ability to dissolve many 

substances, thermal properties, expansion upon freezing). 

See notes at end of exhibit. 
A-27 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

CHEMISTRY (CH) 

4 Acids and Bases • 8 

Grade 4 Grade 8 

Not Assessed. 4a: 

- Compare the properties and uses of common acids and bases 

(acids have a sour taste and react with metals; bases usually have a 

bitter taste and slippery feel; strong acids and bases are corrosive; 

both acids and bases dissolve in water and react with indicators to 

produce different color changes; acids and bases neutralize each 

other). 

5 Chemical Change (4) 8 

Grade 4 Grade 8 

5a: 

- Identify some familiar changes in materials that produce other 

materials with different characteristics (e.g., decaying of 

animal/plant matter, burning, rusting, cooking). 

5a: 

- Differentiate chemical from physical changes in terms of the 

transformation (reaction) of one or more substances (reactants) 

into different substances (products). 

- Provide evidence that a chemical change has taken place based 

on common examples (e.g., temperature change, gas production, 

color change, light emission). 

5b: 

- Recognize that although matter changes form during chemical 

change, its total amount is conserved. 

5c: 

- Recognize the need for oxygen in common oxidation reactions 

(combustion, rusting). 

- Compare the relative tendency of familiar substances to undergo 

these reactions (e.g., combustion of gasoline versus water, 

corrosion of steel versus aluminum). 

5d: 

- Demonstrate understanding that some chemical reactions give 

off while others absorb heat/energy. 

- Classify familiar chemical transformations as either releasing or 

absorbing heat/energy (e.g., burning, neutralization, cooking). 

See notes at end of exhibit. 
A-28 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

PHYSICS (PH) Grade(s) 

1 Physical States and Changes in Matter 4 8 

 

2 Energy Types, Sources and Conversions (4) 8 

3 Heat and Temperature 4 8 

4 Light 4 8 

5 Sound and Vibration • 8 

6 Electricity and Magnetism 4 8 

7 Forces and Motion 4 8 

See notes at end of exhibit. 
A-29 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

PHYSICS (PH) 

1 Physical States and Changes in Matter 4 8 

Grade 4 Grade 8 

1a: 

- Describe that all objects/materials are made up of matter that 

exists in three major states (solid, liquid, gas). 

- Describe differences in the observable physical properties of 

solids, liquids, and gases in terms of shape and volume. 

1b: 

- Demonstrate knowledge that water exists in different physical 

states and can be changed from one state to another by heating 

or cooling, and describe these changes in familiar terms 

(melting, freezing, boiling). 

1a: 

- Use knowledge about the movement of and distance between 

particles to explain differences in the physical properties of solids, 

liquids, and gases (volume, shape, density, compressibility). 

1b: 

- Describe the processes of melting, freezing, evaporation, and 

condensation as changes of state resulting from the supplying or 

removing of heat/energy. 

- Relate the rate/extent of these processes to common physical 

factors (surface area, dissolved substances, temperature, 

altitude/pressure). 

1c: 

- Demonstrate understanding of the melting/boiling point of 

substances. 

- Explain why temperature remains constant during phase change 

(melting, boiling, freezing). 

1d: 

- Illustrate understanding that matter (mass) is conserved during 

familiar physical changes (e.g., change of state, dissolving solids, 

thermal expansion). 

2 Energy Types, Sources and Conversions (4) 8 

Grade 4 Grade 8 

2a: 

- Identify common energy sources and forms (e.g., wind, sun, 

electricity, burning fuel, water wheel, food). 

- Know some practical uses of energy. 

2a: 

- Identify different forms of energy (e.g., mechanical, light, sound, 

electrical, thermal, chemical). 

- Describe simple energy transformations (e.g., combustion in an 

engine to move a car, electrical energy to power a lamp, 

hydroelectric power, changes between potential and kinetic 

energy). 

- Apply knowledge of the concept of conservation of total energy. 

3 Heat and Temperature 4 8 

Grade 4 Grade 8 

3a: 

- Demonstrate knowledge that heat flows from a hot object to a 

cold object and causes materials to change temperature and 

volume. 

- Identify common materials that conduct heat better than 

others. 

- Recognize the relationship between temperature 

measurements and how hot/cold an object is. 

3a: 

- Relate heat to the transfer of energy from an object at a high 

temperature to one at a lower temperature. 

- Compare the relative thermal conductivity of different materials.

- Compare/contrast methods of heat transfer (conduction, 

convection, and radiation). 

3b: 

- Explain thermal expansion in terms of change in volume and/or 

pressure (e.g., thermometers, balloons). 

3c: 

- Relate temperature and changes in volume and/or pressure to the 

movement/speed of particles. 

See notes at end of exhibit. 
A-30 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

PHYSICS (PH) 

4 Light 4 8 

Grade 4 Grade 8 

4a: 

- Identify common sources of light (e.g., bulb, flame, sun). 

- Relate familiar physical phenomena to the presence/absence 

and behavior of light (e.g., appearance of rainbows; colors 

produced from prisms, oil slicks, soap bubbles, etc.; formation 

of shadows; visibility of objects; mirrors). 

4a: 

- Describe/identify some basic properties/behaviors of light 

(transmission from a source through different media; speed of 

light compared to sound; reflection, refraction (bending), 

absorption, and transmission by different materials; splitting of 

white light into its component colors by prisms and other 

dispersive media). 

4b: 

- Relate the appearance/color of objects to the properties of 

reflected/absorbed light. 

4c: 

- Solve practical problems involving the reflection of light from 

plane mirrors and the formation of shadows. 

- Use/interpret ray diagrams to identify the path of light and 

locate reflected/projected images. 

5 Sound and Vibration • 8 

Grade 4 Grade 8 

Not Assessed. 5a: 

- Explain how sound with varying loudness (intensity) and pitch 

is produced by vibrations with different properties (amplitude, 

frequency).* 

- Recognize that sound is transmitted away from a source through 

different materials and can be reflected by surfaces. 

* Knowledge/use of the specific terms amplitude and frequency is not 

expected at grade 8.

6 Electricity and Magnetism 4 8 

Grade 4 Grade 8 

6a: 

- Know common uses of electricity. 

- Identify a complete electrical circuit using batteries, bulbs, 

wires, and other common components that conduct electricity. 

6b: 

- Know that magnets have north and south poles, that like poles 

repel and opposite poles attract, and that magnets can be used to 

attract some other materials/objects. 

6a: 

- Describe the flow of current in an electrical circuit. 

- Draw/identify diagrams representing complete circuits (series 

and parallel). 

- Classify materials as electrical conductors or insulators. 

- Recognize that there is a relationship between current and 

voltage in a circuit. 

6b: 

- Demonstrate knowledge of the properties of permanent magnets 

and the effects of magnetic force. 

- Identify essential features and practical uses of electromagnets. 

See notes at end of exhibit. 
A-31 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

PHYSICS (PH) 

7 Forces and Motion 4 8 

Grade 4 Grade 8 

7a: 

- Identify familiar forces that cause objects to move (e.g., 

gravity acting on falling objects, push/pull forces). 

7b: 

- Describe how the relative weight of objects can be determined 

using a balance. 

- Relate the weight* of different objects to their ability to float 

or sink. 

* Although buoyancy is a function of density, knowledge of the term and 

concept of density and the distinction between weight and mass is not 

expected at grade 4. At this level, students may be assessed on their 

knowledge of flotation using objects of comparable size but different 

weight/mass.

7a: 

- Represent the motion of an object in terms of its position, 

direction, and speed in a given reference frame. 

- Compute speed from time and distance using standard units. 

- Use/interpret information in distance versus time graphs. 

7b: 

- Describe general types of forces (e.g., weight as a force due to 

gravity, contact force, buoyant force, friction). 

- Predict changes in motion (if any) of an object based on the 

forces acting on it. 

- Demonstrate basic knowledge of work and the function of 

simple machines (e.g., levers) using common examples. 

7c: 

- Explain observable physical phenomena in terms of density 

differences (e.g., floating/sinking objects, rising balloons, ice 

layers). 

7d: 

- Demonstrate knowledge of effects related to pressure 

(e.g., atmospheric pressure as a function of altitude, ocean 

pressure as a function of depth, evidence of gas pressure in 

balloons, spreading force over a large/small area, fluid levels). 

See notes at end of exhibit. 
A-32 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

EARTH SCIENCE (ES) Grade(s)

1 Earth's Structure and Physical Features 4 8 

 

2 Earth's Processes, Cycles, and History 4 8 

3 Earth in the Solar System and the Universe 4 8 

See notes at end of exhibit. 
A-33 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

EARTH SCIENCE (ES) 

1 Earth’s Structure and Physical Features 4 8 

Grade 4 Grade 8 

1a: 

- Know that the surface of the earth is composed of rocks, 

minerals, sand, and soil. 

- Compare physical properties, locations, and uses of these 

materials. 

1b: 

- Recognize that most of the earth’s surface is covered with 

water. 

- Describe the locations/types of water found on the earth (e.g., 

salt water in oceans, fresh water in lakes and rivers, clouds, 

snow, ice caps, icebergs). 

1c: 

- Provide evidence for the existence/nature of air, including the 

fact that air contains water (e.g., cloud formation, dew drops, 

evaporation of ponds), examples of the uses of air, and the 

importance of air for supporting life. 

1d: 

- Identify/describe common features of the earth’s landscape 

(e.g., mountains, plains, rivers, deserts) and relate them to 

human use (e.g., farming, irrigation, land development). 

1a: 

- Demonstrate knowledge of the structure and physical 

characteristics of the earth’s crust, mantle, and core. 

- Use/interpret topographic maps. 

- Describe the formation, characteristics, and/or uses of soil, 

minerals, and basic rock types. 

1b: 

- Compare the physical state, movement, composition and relative 

distribution of water on the earth (e.g., oceans, rivers, ground 

water, glaciers, ice caps, clouds). 

1c: 

- Know that the earth’s atmosphere is a mixture of gases, and 

identify the relative abundance of its main components. 

- Relate changes in atmospheric conditions (temperature, 

pressure, composition) to altitude. 

[Assessment objectives related to the use and conservation of earth’s 

natural resources are described in the Environmental Science section. ]

See notes at end of exhibit. 
A-34 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

EARTH SCIENCE (ES) 

2 Earth’s Processes, Cycles, and History 4 8 

Grade 4 Grade 8 

2a: 

- Draw/describe the movement of water on the earth’s surface 

(e.g., flowing in rivers/streams from mountains to 

oceans/lakes). 

- Relate the formation of clouds and rain/snow to a change of 

state of water. 

2b: 

- Describe changes in weather conditions from day to day or 

over the seasons in terms of observable properties such as 

temperature, precipitation (rain/snow), clouds, and wind. 

2c: 

- Recognize that fossils of animals and plants that lived on the 

earth a long time ago can be found in rocks and provide 

evidence that the earth is very old. 

2a: 

- Demonstrate knowledge of the general processes involved in the 

rock cycle (weathering/erosion, deposition, heat/pressure, 

melting/cooling, lava flow) resulting in the continuous formation 

of igneous, metamorphic, and sedimentary rock. 

2b: 

- Diagram/describe the steps in the earth’s water cycle 

(evaporation, condensation, and precipitation), referencing the 

sun as the source of energy and the role of cloud movement and 

water flow in the circulation and renewal of fresh water on the 

earth’s surface. 

2c: 

- Interpret weather data/maps, and relate changing weather 

patterns to global and local factors in terms of temperature, 

pressure, precipitation, wind speed/direction, cloud 

types/formation, and storm fronts. 

2d: 

- Compare seasonal climates of major regions on the earth, 

considering effects of latitude, altitude and geography (e.g., 

mountains and oceans). 

- Identify/describe long- and short-term climatic changes (e.g., ice 

ages, global warming trends, volcanic eruptions, changes in ocean 

currents). 

2e: 

- Identify/describe physical processes and major geological 

events that have occurred over billions of years (e.g., 

weathering, erosion, deposition, volcanic activity, 

 earthquakes, mountain building, plate movement, 

 continental drift). 

- Explain the formation of fossils and fossil fuels. 

See notes at end of exhibit. 
A-35 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

EARTH SCIENCE (ES) 

3 Earth in the Solar System and the Universe 4 8 

Grade 4 Grade 8 

3a: 

- Describe the solar system as a group of planets (including 

earth) each revolving around the sun. 

- Identify the sun as the source of heat and light for the solar 

system. 

3b: 

- Relate daily patterns observed on the earth to the earth’s 

rotation on its axis and its relationship to the sun (e.g., 

day/night, appearance of shadows). 

3c: 

- Draw/describe the phases of the moon. 

3a: 

- Explain phenomena on the earth (day/night, tides, year, phases 

of the moon, eclipses, seasons in the northern/southern 

hemisphere, appearance of sun, moon, planets, and constellations) 

in terms of the relative movements, distances, and sizes of the 

earth, moon, and other bodies in and outside the solar system. 

3b: 

- Recognize the role of gravity in the solar system (e.g., tides, 

keeping the planets and moons in orbit, pulling us to the earth’s 

surface). 

3c: 

- Compare and contrast the physical features of the earth with the 

moon and other planets (e.g., atmosphere, temperature, water, 

distance from sun, period of revolution/rotation, ability to support 

life). 

3d: 

- Recognize the sun as an “average” star, and know that there are 

billions of other stars in the universe outside and very distant 

from the earth’s solar system. 

See notes at end of exhibit. 
A-36 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

ENVIRONMENTAL SCIENCE (EV) Grade(s)

1 Changes in Population • 8 

 

2 Use and Conservation of Natural Resources 4 8 

3 Changes in Environments 4 8 

See notes at end of exhibit. 
A-37 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

ENVIRONMENTAL SCIENCE (EV) 

1 Changes in Population • 8 

Grade 4 Grade 8 

Not Assessed. 1a: 

- Analyze trends in human population, identifying that the world 

population is growing at an increasing rate, and comparing the 

population distribution, growth rate, and consumption/availability 

of resources in different regions. 

1b: 

- Discuss effects of population growth on the environment (e.g., 

use of natural resources, food supply/demand, health, water 

supply/demand, growth of cities/suburbs, land use/development, 

hunting/fishing). 

2 Use and Conservation of Natural Resources 4 8 

Grade 4 Grade 8 

2a: 

- Identify some of the earth’s physical resources that are used in 

everyday life and their common sources (e.g., water, soil, wood, 

minerals, fuel, food). 

- Explain the importance of using these resources wisely. 

Note: Environmental Science is not reported separately at Grade 4. 

Items measuring understandings related to the use and conservation of 

natural resources are reported in Life Science or Earth Science.

2a: 

- Know common examples of renewable and nonrenewable 

resources. 

- Discuss advantages and disadvantages of different types of 

energy sources (e.g., fossil fuels, wood, solar, wind, geothermal, 

nuclear, hydroelectric, chemical batteries). 

- Describe methods of conservation and waste management (e.g., 

recycling/reuse, use of biodegradable materials). 

2b: 

- Relate effects of human use of land/soil resources (e.g., farming, 

ranching, mining, tree harvesting) to methods used in agriculture 

and land management (e.g., crop rotation, terracing/contour 

farming, fertilization, irrigation, pest control, grazing 

management, reclamation/recycling, reforesting). 

2c: 

- Discuss factors related to the supply/demand of fresh water and 

use of water resources (e.g., renewable but limited supply of fresh 

water, purification, desalination, irrigation, water treatment/reuse, 

conservation, use of dams, fishing practices). 

See notes at end of exhibit. 
A-38 

Exhibit A-2. TIMSS science framework and specifications summary: 2003—Continued 

ENVIRONMENTAL SCIENCE (EV) 

3 Changes in Environments 4 8 

Grade 4 Grade 8 

3a: 

- Present ways in which human behavior can have a positive or 

a negative effect on environments. 

- Provide general descriptions and examples of the effects of 

pollution on humans, plants, animals, and their environments, 

and ways of preventing or reducing pollution. 

Note: Environmental Science is not reported separately at grade 4. 

Items measuring understandings related to changes in environments are 

reported in Life Science or Earth Science.

3a: 

- Discuss ways in which human activity can both contribute to 

and help solve environmental problems, including both short- and 

long-term effects on ecosystems. 

- Describe sources, effects, and ways of preventing/reducing air, 

water, and land pollution. 

- Explain the role of science and technology in addressing 

environmental issues. 

3b: 

- Relate some global environmental concerns to their possible 

causes and/or effects (e.g., global warming, acid rain, depletion of 

the ozone layer, deforestation, desertification). 

- Present ways in which science and technology can be used to 

address these concerns. 

3c: 

- Describe some natural hazards and their impact on humans, 

wildlife, and the environment in terms of changes to habitat, 

resources, food webs, and life cycles (e.g., earthquakes, 

landslides, wildfires, volcanic eruptions, floods, storms). 

NOTE: In each science content domain (life science, chemistry, etc.) topic areas are identified by numbers (1, 2, 3,...) and specific objectives are 

identified by lowercase letters (a, b, c,...). Topic areas can be assessed at those grade levels indicated by 4 and 8 on the right side of the table. 

Parentheses around a grade level indicate that a topic area may be introduced at a simple level at that grade. If a topic area should not be addressed at a 

specific grade level, it is indicated by a dot (•). 

SOURCE: International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd

Edition, 2003.
B-1 

Appendix B. Summary Information on Cognitive Domains and Scientific 

Inquiry 

Exhibit B-1 is the document that was used by the expert panel for the classification of items 

to the cognitive domains in the TIMSS 2003 science framework. Exhibit B-2 is the document that 

was used for the scientific inquiry classifications. Both summary documents are based on the TIMSS 

2003 framework but have been reformatted and adapted slightly to facilitate the classification 

process. 

Exhibit B-1. Cognitive domains based on the TIMSS 2003 science framework 

Factual Knowledge (FK) Conceptual Understanding (CU) Reasoning and Analysis (RA) 

Items in the FK category 

assess students’ knowledge 

base of relevant science facts, 

information, tools, 

vocabulary, symbols, units, 

and procedures. 

Items in the CU category assess 

students’ understanding of the 

relationships that explain the 

behavior of the physical/natural 

world and the ability to relate the 

observable to more abstract or 

general scientific concepts. Items in 

this category involve more 

straightforward applications of 

concepts and require less analysis 

and integration than those in the RA 

category. 

Items in the RA category assess 

students’ scientific reasoning and 

analysis skills used to solve 

problems; develop, evaluate and 

justify explanations and problemsolving strategies; draw 

conclusions; make decisions; and 

extend knowledge to new 

situations. Items may involve less 

familiar or more complicated 

contexts and require students to 

bring together knowledge and 

understanding from different 

areas. 

Specific demands that items 

in this category might make 

include: 

• Recall/Recognize 

• Define 

• Describe 

• Use Tools & 

Procedures 

Specific demands that items in this 

category might make include: 

• Illustrate with examples 

• Compare/contrast/classify 

• Represent/model 

• Relate 

• Extract/apply information 

• Find solutions 

• Explain 

Specific demands that items in this 

category might make include: 

• Analyze/interpret/solve 

problems 

• Integrate/synthesize 

• Hypothesize/predict 

• Design/plan 

• Collect/analyze/interpret 

data 

• Draw conclusions 

• Generalize 

• Evaluate 

• Justify 

See notes at end of exhibit.
B-2 

Exhibit B-1. Cognitive domains based on the TIMSS 2003 science framework—Continued 

 Factual Knowledge 

Recall/Recognize • Make or identify accurate statements about science facts, relationships 

processes, and concepts. 

• Identify the characteristics or properties of specific organisms, materials, and 

processes. 

Define • Provide or identify definitions of scientific terms. 

• Recognize and use scientific vocabulary, symbols, abbreviations, units, and 

scales in relevant contexts. 

Describe • Recognize or describe organisms, physical materials, and science processes 

that demonstrate knowledge of properties, structure, function, and 

relationships. 

Use Tools & 

Procedures 

• Demonstrate knowledge of the use of science apparatus, equipment, tools, 

procedures, and measurement devices/scales. 

See notes at end of exhibit. 
B-3 

Exhibit B-1. Cognitive domains based on the TIMSS 2003 science framework—Continued 

 Conceptual Understanding 

Illustrate with Examples • Support or clarify statements of facts/concepts with appropriate 

examples. 

• Identify or provide specific examples to illustrate knowledge of 

general concepts. 

Compare/Contrast/Classify • Identify or describe similarities and differences between groups of 

organisms, materials, or processes. 

• Distinguish, classify or order individual objects, materials, 

organisms, and processes based on characteristics and properties. 

Represent/Model • Use/draw diagrams and/or models to demonstrate understanding of 

science concepts, structures, relationships, processes, and 

biological/physical systems and cycles (e.g., food webs, electrical 

circuits, water cycle, solar system, atomic structure). 

Relate • Relate knowledge of underlying biological and physical concepts 

to the observed or inferred properties/behaviors/uses of objects, 

organisms, and materials. 

Extract/Apply Information • Identify/extract/apply relevant textual, tabular, or graphical 

information in light of science concepts/principles. 

Find Solutions • Identify/use science relationships, equations, and formulas to find 

qualitative or quantitative solutions involving the direct 

application/demonstration of concepts. 

Explain • Provide or identify reasons/explanations for observations or natural 

phenomena, demonstrating understanding of the underlying science 

concept, principle, law, or theory. 

See notes at end of exhibit. 
B-4 

Exhibit B-1. Cognitive domains based on the TIMSS 2003 science framework—Continued 

 Reasoning and Analysis 

Analyze/Interpret/Solve 

Problems 

• Analyze problems to determine the relevant relationships, 

concepts, and problem-solving steps. 

• Develop/explain problem-solving strategies. 

• Interpret/use diagrams and graphics to visualize and/or solve 

problems. 

• Give evidence of deductive and inductive reasoning processes 

used to solve problems. 

Integrate/Synthesize • Provide solutions to problems that require consideration of a 

number of different factors or related concepts. 

• Make associations/connections between concepts in different 

areas of science. 

• Demonstrate understanding of unified concepts and themes across 

the domains of science. 

• Integrate mathematical concepts/procedures in the solutions to 

science problems. 

Hypothesize/Predict • Combine knowledge of science concepts with information from 

experience or observation to formulate questions that can be 

answered by investigation. 

• Formulate hypotheses as testable assumptions using knowledge 

from observation and/or analysis of scientific information and 

conceptual understanding. 

• Make predictions about the effects of changes in biological or 

physical conditions in light of evidence and scientific 

understanding. 

Design/Plan • Design/plan investigations appropriate for answering scientific 

questions or testing hypotheses. 

• Describe/recognize the characteristics of well-designed 

investigations in terms of variables to be measured and controlled 

and cause-and-effect relationships. 

• Make decisions about measurements/procedures to use in 

conducting investigations. 

Collect/Analyze/Interpret Data • Make/record systematic observations and measurements, 

demonstrating appropriate applications of apparatus, equipment, 

tools, procedures, and measurement devices/scales. 

• Represent scientific data in tables, charts, graphs, and diagrams 

using appropriate format, labeling, and scales. 

• Select/apply appropriate mathematical computations/techniques to 

data to obtain derived values necessary to draw conclusions. 

• Detect patterns in data, describe/summarize data trends, and 

interpolate/extrapolate from data or given information. 

Draw Conclusions • Make valid inferences on the basis of evidence and/or 

understanding of science concepts. 

• Draw appropriate conclusions that address questions/hypotheses 

and demonstrate understanding of cause and effect. 

See notes at end of exhibit. 
B-5 

Exhibit B-1. Cognitive domains based on the TIMSS 2003 science framework—Continued 

 Reasoning and Analysis—Continued 

Generalize • Make/evaluate general conclusions that go beyond the experimental 

or given conditions. 

• Apply conclusions to new situations. 

• Determine general formulas for expressing physical relationships. 

Evaluate • Weigh advantages and disadvantages to make decisions about 

alternative processes, materials, and sources. 

• Consider scientific and social factors to evaluate the 

impact/consequences of science and technology in biological and 

physical systems. 

• Evaluate alternative explanations and problem-solving strategies 

and solutions. 

• Evaluate results of investigations with respect to sufficiency of data 

to support conclusions. 

Justify • Use evidence and scientific understanding to justify explanations 

and problem solutions. 

• Construct arguments to support the reasonableness of solutions to 

problems, conclusions from investigations, or scientific 

explanations. 

SOURCE: International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd

Edition, 2003.
B-6 

Exhibit B-2. Scientific inquiry based on the TIMSS 2003 science framework 

How is scientific inquiry defined? 

• Treated as an overarching dimension that overlaps all the fields of science and has both 

content- and skills-based components. 

• Includes knowledge, skills, and abilities assessed by items or tasks set in different contentrelated contexts and covering a range of cognitive demands. 

• Items/tasks assessing scientific inquiry are associated with both a content and cognitive 

domain as well as identified as engaging students in knowledge and/or process skills 

involved in scientific inquiry. 

Includes two components: 

General knowledge about the nature of science and 

scientific inquiry: 

• Scientific knowledge is subject to change 

• Importance of using different methods in 

verifying/testing knowledge 

• Use of basic “scientific methods” 

• Communication of results 

• Interaction of science, mathematics and 

technology 

Knowledge and process skills involved in five 

major phases of the scientific inquiry process: 

• Formulating questions and hypotheses 

• Designing investigations 

• Collecting and representing data 

• Analyzing and interpreting data 

• Drawing conclusions and developing 

explanations 

Understandings and abilities increase across grades: 

Grade 4 Grade 8 

• Demonstrate knowledge of a “fair test”. 

• Describe/conduct an investigation. 

• Make systematic observations or measurements 

using simple tools, equipment, and procedures. 

• Represent findings using simple charts and 

diagrams. 

• Apply routine mathematical computations of 

measured values. 

• Identify simple relationships. 

• Briefly describe results of investigations. 

• State simple conclusions from investigations to 

answer a specific question. 

• Formulate hypotheses or predictions based on 

observation or scientific knowledge that can be 

tested. 

• Specify variables to be controlled and varied in 

well-designed investigations. 

• Make decisions about measurements, 

equipment and procedures. 

• Collect/represent data using appropriate 

terminology, units, precision, format, and 

scales. 

• Analyze data using appropriate mathematical 

techniques. 

• Describe patterns in data. 

• Demonstrate understanding of cause and 

effect. 

• Draw conclusions based on evidence and in 

light of scientific understanding. 

• Evaluate results of investigations, including 

sufficiency of data for supporting conclusions. 

• Consider alternative explanations 

• Apply/extend conclusions to new situations. 

SOURCE: International Study Center, Lynch School of Education, Boston College, TIMSS Assessment Frameworks and Specifications 2003, 2nd

Edition, 2003.
C-1 

Appendix C. Expert Panel 

Members and Staff 

Expert Panel Members

Alicia Cristina Alonzo 

California Institute of Technology 

Rodger W. Bybee 

Biological Sciences Curriculum Study (BSCS) 

Audrey B. Champagne 

University at Albany 

State University of New York 

W. Tony Heiting 

Iowa Department of Education 

Tricia R. Kerr 

The Ohio State University 

Gregory Kirchhofer 

Columbia Public Schools (MO) 

Mary Lauko 

Educational Testing Service 

Brett D. Moulding 

Utah State Office of Education 

Christine O’Sullivan 

K-12 Consulting 

(Consultant for Educational Testing Service) 

Senta A. Raizen 

National Center for Improving Science 

Education 

Ann Marie Zolandz 

Educational Testing Service 

Staff

National Center for Education Statistics

Marilyn Binkley 

Patrick Gonzales 

Andrew Malizio 

Eugene Owen 

Elois Scott 

Education Statistics Services Institute

Kristy David 

Janine Emerson 

Kim Gattis 

Dana Kelly 

Teresa Smith Neidorf 

Margaret Woodworth 

American Institutes for Research

Maria Stephens 

David Nohara 

Independent Project Consultant 

D-1 

Appendix D. Methodological Notes and Supplementary Data 

Considerations in Selecting Classification Methods 

The cross-classification approach (classification of items in one assessment to the other 

assessment framework) was selected for the examination of content and grade fit so that there would 

be multiple content profiles for each assessment. This method also prevents either assessment from 

being evaluated through only one perspective, which may or may not be reflective of its purposes. 

This approach takes advantage of the multiple ways of describing content found in the frameworks 

and enables direct comparisons across the assessments. 

For the classifications based on cognitive processes and scientific inquiry skills, common 

classification systems were chosen—the TIMSS 2003 cognitive domain and scientific inquiry

dimensions. There were several reasons for deciding, first, to use common classification systems 

and, second, for selecting the TIMSS definitions as the common systems. With regard to the first 

decision, the study organizers recognized that classifications in these dimensions likely would require 

more discretionary judgment than those in content areas and thought a single rubric for each would 

be the most realistic to implement under the time constraints. With regard to the second decision, 

although other classification systems for both dimensions were reviewed,1

 the study organizers 

determined that, because the descriptions of cognitive domain and scientific inquiry in TIMSS are 

among the most recently developed and the most explicitly defined, they would be the most relevant 

for the comparison study and would provide sufficient guidance for panelists in making 

classifications. They also allow the panel’s ratings to be compared with the developer’s ratings, at 

least for the TIMSS items. 

Reliability Analyses 

For the classification of items, the expert panel was divided into three groups to review items 

by content area (as described in section III). To measure the extent to which the different content area 

groups were interpreting the common rubrics in similar ways, a common set of items was classified 

by all three groups with respect to cognitive domain and scientific inquiry. The degree to which the 

three groups classified these items in the same categories on these two dimensions serves as a 

measure of the reliability of these classifications. The set of 60 items (30 from NAEP and 30 from 

TIMSS), which reflects approximately 10 percent of the total item classifications across both 

assessments, was taken from across the science content areas and grade levels. This was not a 

random sample, but a representative set chosen to cover the main categories addressed in the study 

(content area and grade level). Some effort was also made to ensure that there were at least some 

items from each of the cognitive categories based on the original assessment developers’ 

classifications (cognitive domain in TIMSS and knowing and doing in NAEP). In addition, some 

items classified as scientific inquiry/investigation by the original developers were included. 

Reliability items were classified at regular intervals throughout the classification process. Given the 

 

1

 In looking for classification systems for cognitive domain, study organizers reviewed the National Science 

Education Standards (NRC 1996), the Benchmarks For Science Literacy (AAAS 1993), several studies on the 

alignment of state assessments and standards being conducted by the Council of Chief State School Officers 

(CCSSO), and select state assessments, in addition to TIMSS and NAEP. The first two sources also addressed 

scientific inquiry and so were considered for that rubric as well. 
D-2 

limited time available for the expert panel meeting, the 30 items from each assessment was the 

maximum number of items that could be included in the reliability set. 

The multiple classification data for the reliability set were analyzed based on the percentage 

of classifications where there was agreement. Classification reliability statistics were computed in 

two ways, as follows: 

• The percentage of total comparisons: based on the number of comparisons where there was 

agreement between any two groups (i.e., groups 1 and 2, groups 2 and 3, and groups 1 and 3) 

across ALL items and 

• The percentage of items: based on the number of items where there was agreement across 

ALL three groups. 

The results for these two types of analysis for the cognitive domain classifications are shown 

in tables D-1 and D-2, and the corresponding results for scientific inquiry classifications are shown in 

tables D-3 and D-4. 

There was reasonably high agreement across groups on the classification to the three 

cognitive domain categories (factual knowledge, conceptual understanding, and reasoning and 

analysis). The lack of perfect agreement reflects the fact there is considerable overlap of 

skills/abilities covered by these types of cognitive categories. The results show 72 percent agreement 

for all comparisons between any two groups across all items (table D-1). These results reflect 

agreement across all three groups for 55 percent of all items (table D-2). The disagreements were 

always between “adjacent” categories (i.e., factual knowledge/conceptual understanding or 

conceptual understanding/reasoning and analysis). That is, there were no instances of disagreement 

between factual knowledge and reasoning and analysis. For all items where there was not total 

agreement across all three groups (45 percent overall), two of the three groups agreed and the third 

classified the item to an adjacent category. In some cases items were classified as being on the 

border between two adjacent categories by some groups; these were considered to be in agreement if 

they overlapped the classification of other groups. Collectively, when the reliability analyses are 

broken down into the NAEP and TIMSS items, the results are nearly the same. 

Table D-1. Reliability of cognitive domain classifications for science items in NAEP 2000 and 

Table D-1. TIMSS 2003, by number of comparisons and percentage agreement 

 Number of comparisons and percentage agreement NAEP 2000 TIMSS 2003 Overall 

Total number of comparisons across items 90 90 180 

Number of comparisons with agreement between groups 65 64 129 

Percentage agreement 72 71 72 

NOTE: Data are based on 30 NAEP items and 30 TIMSS items that were classified by three expert panel groups and reflect all comparisons 

between any two groups (i.e., groups 1 and 2; groups 2 and 3; and groups 1 and 3). Items that were classified on the border between two 

categories by some groups were counted as being in agreement with other groups if their categories overlapped. 

SOURCE: Expert panel classifications of selected fourth- and eighth-grade science items from the National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment and the Trends in International Mathematics and Science Study (TIMSS) 2003 Assessment.

 
D-3 

Table D-2. Reliability of cognitive domain classifications for science items in NAEP 2000 and 

Table D-2. TIMSS 2003, by number of items and percentage agreement 

 Number of items and percentage agreement NAEP 2000 TIMSS 2003 Overall 

Total number of items 30 30 60 

Number of items with agreement across all groups 17 16 33 

Percentage agreement 57 53 55 

NOTE: Data are based on 30 NAEP items and 30 TIMSS items that were classified by three expert panel groups and reflect comparisons across 

all three groups for each item. Items that were classified on the border between two categories by some groups were counted as being in 

agreement if their categories overlapped both of the other groups. 

SOURCE: Expert panel classifications of selected fourth- and eighth-grade science items from the National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment and the Trends in International Mathematics and Science Study (TIMSS) 2003 Assessment.

Compared to the cognitive domain classifications, there was higher agreement across groups 

on the scientific inquiry classifications (yes/no), but the agreement was higher for NAEP items than 

for TIMSS items. There was 83 percent agreement between any two groups across all items on the 

scientific inquiry classifications, with 90 percent agreement for the NAEP items and 76 percent 

agreement for the TIMSS items (table D-3). At the item level, 75 percent of items overall had 

agreement across all three groups (table D-4). This reflects 87 percent of NAEP items compared to 

63 percent of TIMSS items. In a few cases items were classified as being borderline inquiry (yes/no) 

by some groups; these were considered to be in agreement with groups that classified the items as 

yes. 

Table D-3. Reliability of scientific inquiry classifications for science items in NAEP 2000 and 

Table D-3. TIMSS 2003, by number of comparisons and percentage agreement 

 Number of comparisons and percentage agreement NAEP 2000 TIMSS 2003 Overall 

Total number of comparisons across items 90 90 180 

Number of comparisons with agreement between groups 81 68 149 

Percentage agreement 90 76 83 

NOTE: Data are based on 30 NAEP items and 30 TIMSS items that were classified by three expert panel groups and reflect all comparisons 

between any two groups (i.e., groups 1 and 2; groups 2 and 3; and groups 1 and 3). Items that were classified as borderline (yes/no) by some 

groups were counted as being in agreement with other groups that classified the items as yes. 

SOURCE: Expert panel classifications of selected fourth- and eighth-grade science items from the National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment and the Trends in International Mathematics and Science Study (TIMSS) 2003 Assessment. 

Table D-4. Reliability of scientific inquiry classifications for science items in NAEP 2000 and 

Table D-4. TIMSS 2003, by number of items and percentage agreement 

 Number of items and percentage agreement NAEP 2000 TIMSS 2003 Overall 

Total number of items 30 30 60 

Number of items with agreement across all groups 26 19 45 

Percentage agreement 87 63 75 

NOTE: Data are based on 30 NAEP items and 30 TIMSS items that were classified by three expert panel groups and reflect comparisons across 

all three groups for each item. Items that were classified as borderline (yes/no) by some groups were counted as being in agreement with other 

groups that classified the items as yes. 

SOURCE: Expert panel classifications of selected fourth- and eighth-grade science items from the National Assessment of Educational Progress 

(NAEP) 2000 Science Assessment and the Trends in International Mathematics and Science Study (TIMSS) 2003 Assessment. 
D-4 

In sum, the main focus of the present study is a content comparison—classification of items 

to the content framework of the other assessment—which was done by the separate content-area 

subpanels. The reliability tables are included only to provide some indication of the extent to which 

the expert panelists agreed on the other metrics that used a common rubric (cognitive domain and 

scientific inquiry). Expert panelists typically spent less time reviewing and classifying the items in 

the reliability set that were outside of their primary content area, and the results from these secondary 

classifications should not be viewed as a complete replication of the process used by the primary 

group which was most familiar with the items in the respective content area. Therefore, only the 

primary group classifications were used in the reporting of results for cognitive domain and scientific 

inquiry. 

Data Processing 

After the expert panel meeting, the facilitators of each group met to review the methods used 

and the data collected to ensure consistency. In some cases, methods or reporting conventions were 

slightly different between groups. For these cases, the facilitators reviewed their notes and the notes 

of individual panel members to standardize the data. Datasets were produced that included the 

standardized expert panel classifications for all items from each assessment (including multiple 

classifications on the reliability set) as well as original classification information for each item 

provided by the assessment developers. The raw data containing all original panelist classifications 

and comments from each subgroup were also available for analysts and were consulted in the writing 

of this report. 
E-1 

Appendix E. Example Items 

Exhibit E-1. Index of example items from NAEP 20001 and TIMSS 2003 

 

1

 A NAEP task from the 1996 assessment (Task 1) is included as an example to illustrate the type of hands-on tasks used in NAEP. 

No hands-on tasks from the 2000 NAEP assessment have been released. 

Example 

number Description of characteristics illustrated in text 

1 NAEP cross-grade life science item administered at grades 4 and 8 classified at the 

eighth-grade level on TIMSS 2003 science framework 

2 NAEP fourth-grade item classified as scientific inquiry 

3 TIMSS eighth-grade environmental science item classified as scientific inquiry 

4 NAEP eighth-grade multiple-choice physical science item classified as measuring 

conceptual understanding 

5 NAEP fourth-grade physical science item classified at the eighth-grade level on the 

TIMSS 2003 science framework 

6 TIMSS fourth-grade chemistry item not classified to a topic on the NAEP 2000 

science framework 

7 TIMSS eighth-grade chemistry item not classified to a topic on the NAEP 2000 

science framework 

8 NAEP fourth-grade physical science item from the topic of energy and transformations 

9 TIMSS eighth-grade life science item classified at the twelfth-grade level on the 

NAEP 2000 science framework 

10 TIMSS eighth-grade life science item from the topic of human health not classified to 

a topic in the NAEP 2000 science framework 

11 NAEP eighth-grade life science item from the topic of interdependence of life 

12 NAEP fourth-grade Earth science item classified as eighth-grade environmental 

science on the TIMSS 2003 science framework 

13 TIMSS fourth-grade Earth science item classified at the eighth-grade level on the 

NAEP 2000 science framework 

14 TIMSS fourth-grade Earth science item not classified to a topic on the NAEP 2000 

science framework 

15 TIMSS eighth-grade Earth science item classified at the twelfth-grade level on the 

NAEP 2000 science framework 

16 TIMSS eighth-grade environmental science item classified as fourth-grade physical 

science on the NAEP 2000 science framework 

17 TIMSS fourth-grade environmental science item classified as life science on the 

NAEP 2000 science framework 

18 TIMSS eighth-grade environmental science item classified as Earth science on the 

NAEP 2000 science framework 

 

Full 

tasks Description 

Task 1 NAEP eighth-grade hands-on task from the NAEP 1996 science assessment 

(Salt Solutions) 

Task 2 TIMSS eighth-grade problem solving and inquiry task (Metal Crown) 
E-2 

EXAMPLE 1 

NAEP short constructed-response item – grades 4 and 8 

Scoring guide 

Complete 

Student provides a response showing clear understanding of competition and its effect on population. 

Partial 

Student provides a response that shows an understanding of competition, but not of the consequences. 

OR 

Student provides a response that shows a reasonable consequence and gives a plausible reason not based on 

competition. 

Unsatisfactory/Incorrect 

Student provides a response that shows no understanding of the concept of interspecific competition, 

answering that there will be more of the large fish already in the pond, or that the two kinds of fish would be 

friends, etc. 

OR 

Student responds that one type of fish would die and gives no reason or an implausible reason. 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Life science Life science 

Ecology 

The interdependence of life: populations, 

communities, and ecosystems 

Ecosystems 

Grade 4 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:2 no 1

Classified by NAEP assessment developers 

2

 Classified by expert panel 

Suppose that one spring a new type of large fish was put into the pond. So many were put in 

that there were twice as many fish as before. By the end of the summer, what would happen 

to the large fish that were already in the pond? 

Explain why you think these new large fish would have this effect.

 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment.
E-3 

EXAMPLE 2 

NAEP short constructed-response item – grade 4 

You are going to the park on a hot day and need to take some water with you. You have 

three different bottles, as shown in the picture below. You want to choose the bottle that will 

hold the most water. Explain how you can find out which bottle holds the most water. 

____________________________________________ 

____________________________________________ 

____________________________________________ 

____________________________________________ 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment. 

Scoring guide 

Complete 

Student demonstrates an understanding of how to measure and compare the volumes of three different bottles 

by outlining a method for finding which bottle holds the most water.

Partial 

Student demonstrates some understanding, but does not state a specific method.

Unsatisfactory/Incorrect 

Student response is based on the shape or height of the bottle, not on its volume. Student may also compare 

the time required to pour water out of each bottle.

Framework classifications 

NAEP content framework1 TIMSS content framework2

Physical science Chemistry 

Matter and its transformations 

Diversity of matter (materials): classification and 

types, particulate nature of matter, conservation of 

matter 

Classification and composition of matter 

Grade 4 Grade 4 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 yes 1

Classified by NAEP assessment developers 

2

Classified by expert panel 
E-4 

EXAMPLE 3 

TIMSS extended constructed-response item – grade 8 

Sea water contains dissolved salts and is not suitable for drinking. Describe a procedure 

that can be used to obtain a cup of drinking water from a bucket of sea water. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment.

Scoring guide 

Correct

Describes a correct procedure that includes the following basic steps (may use diagrams). 

i) Boiling/evaporation to separate water from salt. 

ii) Collecting the distilled water (condensation). 

[May describe other correct procedures such as freezing method or reverse osmosis method.] 

Partial 

Describes boiling/evaporation step to separate water from salt; condensation step is omitted. 

OR 

States "distillation" or similar but no description of the process is given. 

Incorrect 

Mentions boiling process but with no or incorrect indication of separation included. 

OR 

Mentions filtering to separate salt. 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

 Multiple classification

Environmental science Physical science Earth science 

 Matter and its transformations Water (hydrosphere) 

Use and conservation of natural 

resources 

Temperature and states of matter 

(physical changes) The location of water, its 

distribution, characteristics, effect of 

and influence on human activity 

Grade 8 Grade 8 Grade 8 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 yes 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-5 

EXAMPLE 4 

NAEP multiple choice item – grade 8 

Two boys wearing in-line skates are standing on a smooth surface with the palms of their 

hands touching and their arms bent, as shown above. If Boy X pushes by straightening his 

arms out while Boy Y holds his arms in the original position, what is the motion of the two 

boys? 

A) Boy X does not move and Boy Y moves backward. 

B) Boy Y does not move and Boy X moves backward. 

C) Boy X and Boy Y both move backward. 

D) The motion depends on how hard Boy X pushes. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment. 

Answer Key: C 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Physical science Physics 

Motion 

Action and reaction Forces and motion 

Grade 8 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by NAEP assessment developers 

2

Classified by expert panel 
E-6 

EXAMPLE 5 

NAEP multiple-choice item – grade 4 

Answer Key: A 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Physical science Physics 

Motion 

Vibration and waves as motion (includes sound) Sound and vibration 

Grade 4 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by NAEP assessment developers 

2

Classified by expert panel 

► Key 

There is a thunderstorm close to your house. The windows rattle at the same time that you 

hear the thunder. What causes the windows to rattle? 

A) Sound waves from the thunder 

B) Light from the lightning 

C) Rain from the clouds 

D) The high humidity during the storm 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment
E-7 

EXAMPLE 6 

TIMSS multiple-choice item – grade 4 

Which of these activities will result in a different kind of material being formed? 

A) A nail is left outside and it rusts. 

B) A glass is dropped and it shatters into small pieces. 

C) A rubber band is stretched until it breaks. 

D) A pencil is sharpened to a point. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: A 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Chemistry Physical science

Matter and its transformations

Chemical change No match at subtopic level 

Grade 4 Grade 4 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-8 

EXAMPLE 7 

TIMSS short constructed-response item – grade 8 

A solution of hydrochloric acid (HCl) in water will turn blue litmus paper red. A solution 

of the base sodium hydroxide (NaOH) in water will turn red litmus paper blue. If the acid 

and base solutions above are mixed in the right proportion, the resulting solution will cause 

neither red nor blue litmus paper to change color. 

Explain why the litmus paper does not change color in the mixed solution. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Scoring guide 

Correct 

Explanation refers explicitly to the formation of water (and salt) from the neutralization reaction. [Responses 

may use words or a chemical equation. The equation does not need to be completely correct for credit as long 

as neutralization is clear.] 

OR 

Explanation refers explicitly to neutralization (or equivalent), but the specific reaction is not mentioned. 

OR 

Explanation refers to a chemical reaction taking place (implicitly or explicitly) to form products that do not 

react with litmus paper (or similar). [Neutralization is not explicitly mentioned.] 

Incorrect 

Mentions only that acid and base are “balanced”, “opposites”, “cancel each other”, or similar. 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Chemistry Physical science 

 Matter and its transformations 

Acids and bases No match at subtopic level 

Grade 8 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-9 

EXAMPLE 8 

NAEP multiple-choice item – grade 4 

Beans and coal have stored energy. Where did the energy come from that is stored in beans 

and coal? 

A) From the Earth’s gravity 

B) From the Sun’s light 

C) From the heat in the Earth’s core 

D) From the air’s carbon dioxide 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment. 

Answer Key: B 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Physical science Physics 

Energy and its transformations 

Energy sources and use, including distribution, 

conversion, costs, and depletion 

Energy types, sources and conversions 

Grade 4 Grade 4 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by NAEP assessment developers 

2

Classified by expert panel 
E-10 

EXAMPLE 9 

TIMSS multiple-choice item – grade 8 

Animals and plants are made up of a number of different chemical elements. What happens 

to all of these elements when animals and plants die? 

A) They die with the animal or plant. 

B) They evaporate into the atmosphere. 

C) They are recycled back into the environment. 

D) They change into different elements. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment

Answer Key: C 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Life science Life science 

 Ecology 

Ecosystems The interdependence of life: populations, 

communities, and ecosystems 

Grade 8 Grade 12 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-11 

EXAMPLE 10 

TIMSS multiple-choice item – grade 8 

Eating leafy vegetables is important for human health. This is because leafy vegetables are 

a good source of which of the following? 

A) protein 

B) carbohydrates 

C) minerals 

D) fat 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: C 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Life science Life science 

 Organisms 

Human health No match at subtopic level 

Grade 8 Grade 8 

TIMSS cognitive domain:2 factual knowledge 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-12 

EXAMPLE 11 

NAEP multiple-choice item – grade 8 

If air pollution causes the rain that falls on this pond to become much more acidic, after two 

years how will this acidity affect the living things in this pond? 

A) There will be more plants and animals because the acid is a source of food. 

B) There will be fewer plants and animals because the acid will dissolve many of them. 

C) There will be fewer plants and animals because many of them cannot survive in 

water with high acidity. 

D) There will be more plants and animals because the acid will kill most of the diseasecausing microorganisms. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment. 

Answer Key: C 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Life science Life science 

Ecology 

The interdependence of life: populations, 

communities and ecosystems Ecosystems 

Grade 8 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by NAEP assessment developers 

2

Classified by expert panel 
E-13 

EXAMPLE 12 

NAEP extended constructed-response item – grade 4 

Garbage is a big problem. In many cities and towns, garbage is taken away to landfills, 

which are often called dumps. Some landfills are very big and may cover hundreds of 

acres. But even these big landfills are getting full and may have to be closed. 

Here are some ideas for solving the garbage problem. Write what you think is a good point 

about each idea and what you think is a bad point about each idea. 

Ideas for Solving 

Garbage 

Problem Good Points Bad Points

Recycling 

 

Burning garbage 

 

Dumping garbage in 

the ocean 

 

Sending garbage to a 

landfill in another state 

 

Shipping garbage to 

outer space 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National 

Assessment of Educational Progress (NAEP), 2000 Science Assessment. 
E-14 

EXAMPLE 12 — continued 

Scoring guide 

Complete 

The student is asked to provide ten explanations, one good and one bad for five proposed solutions for 

dealing with garbage. A complete response provides eight to ten correct explanations. 

Essential 

An essential response provides six or seven correct explanations. 

Adequate 

An adequate response provides three to five correct explanations. 

Partial 

A partial response provides one or two correct explanations. 

Unsatisfactory 

An unsatisfactory response provides no correct explanations. 

Framework classifications 

NAEP content framework1

 TIMSS content framework2

Earth science Environmental science 

Solid Earth (lithosphere) 

Resources from the Earth used by humankind Use and conservation of natural resources 

Grade 4 Grade 8 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 no 1

Classified by NAEP assessment developers 

2

Classified by expert panel 
E-15 

EXAMPLE 13 

TIMSS multiple-choice item – grade 4 

Katie sees a full moon. About how much time will go by before the next full moon? 

A) one week 

B) two weeks 

C) one month 

D) one year 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: C 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Earth science Earth science 

 Earth in space 

Earth in the solar system and the universe The setting of the Earth in the solar system 

Grade 4 Grade 8 

TIMSS cognitive domain:2 factual knowledge / conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-16 

EXAMPLE 14 

TIMSS multiple-choice item – grade 4 

Look at the picture above. Where is the best location to grow crops? 

A) Location A 

B) Location B 

C) Location C 

D) Location D 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: B 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Earth science Earth science 

 Solid Earth (lithosphere) 

Earth’s structure and physical features No match at the subtopic level 

Grade 4 Grade 8 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-17 

EXAMPLE 15 

TIMSS multiple-choice item – grade 8 

The diagram above shows a map of the world with the lines of latitude marked. Which of 

the following places marked on the map is the most likely to have an average yearly 

temperature similar to location X? 

A) location A 

B) location B 

C) location C 

D) location D 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: A 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Earth science Earth science 

 Air (atmosphere) 

Earth's processes, cycles, and history Climate 

Grade 8 Grade 12 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 yes 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-18 

EXAMPLE 16 

TIMSS multiple-choice item – grade 8 

Which group of energy sources are ALL renewable? 

A) coal, oil, and natural gas 

B) solar, oil, and geothermal 

C) wind, solar, and tidal 

D) natural gas, solar, and tidal 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Answer Key: C 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Environmental science Physical science 

 Energy and its transformations 

Use and conservation of natural resources Energy sources and use, including distribution, 

conversion, costs, and depletion 

Grade 8 Grade 4 

TIMSS cognitive domain:2 factual knowledge 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-19 

EXAMPLE 17 

TIMSS short constructed-response item – grade 4 

Write down what happens to plants and fish in a river when a factory pours large amounts 

of hot water into the river. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Scoring guide

Correct Response 

Explains that some species will die; others will be more abundant. 

OR 

Explains that many plants and fishes die. 

OR 

Explains that fish try to leave. 

Incorrect 

The description or explanation given is not adequate. 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

 Primary classification

Environmental science Life science 

 Ecology 

Changes in environments No match at the subtopic level 

Grade 4 Grade 8 

TIMSS cognitive domain:2 reasoning and analysis 

Scientific inquiry classification:

2

 yes 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-20 

EXAMPLE 18 

TIMSS extended constructed-response item – grade 8 

The surface of Earth has more water than land. Write down TWO reasons why some 

people still do not have enough water to drink. 

1. 

2. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and 

Science Study (TIMSS), 2003 Assessment. 

Scoring guide 

Note: Each of the two responses is coded separately. The same code may be used twice if they are based on 

general categories. However, if the two responses are essentially the same, the second response should be 

coded as an incorrect response. 

Correct 

Mentions unsuitability of salt water for human consumption. 

OR 

Mentions climate or uneven distribution of rain/water. 

OR 

Mentions pollution as a cause. 

OR 

Mentions reasons related to population, water consumption, or waste of water. 

OR 

Mentions economic/technical factors (cost of transportation, water treatment). 

OR 

Mentions that much of the Earth's water is frozen in icebergs, glaciers, etc. 

Incorrect response 

Mentions only water in clouds. 

Response too vague. 

Framework classifications 

TIMSS content framework1

 NAEP content framework2

Environmental science Earth science 

 Water (hydrosphere) 

Use and conservation of natural resources 

The location of water, its distribution, characteristics, 

effect of and influence on human activity 

Grade 8 Grade 8 

TIMSS cognitive domain:2 conceptual understanding 

Scientific inquiry classification:

2

 no 1

Classified by TIMSS assessment developers 

2

Classified by expert panel 
E-21 

TASK 1

NAEP 1996 hands-on task – grade 8 

SALT SOLUTIONS

Estimating the Salt Concentration of an Unknown Salt Solution

Using the “Floating Pencil Test”

For this task, you have been given a kit that contains materials that you will use to perform an 

investigation during the next 30 minutes. Please open your kit now and use the following 

diagram to check that all of the materials in the diagram are included in your kit. If any 

materials are missing, raise your hand and the administrator will provide you with the materials 

that you need. 
E-22 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

Every body of water in natural ecosystems has salts and other substances dissolved in it. The 

concentration of dissolved salt varies from less than 0.2 percent in most freshwater streams and 

lakes to about 3.5 percent in most of the world's oceans. In this task, you will observe and 

measure how much of the length of a pencil floats above the water surface in water with very 

low salt concentration and in water with very high salt concentration. You will then use the 

same procedures to estimate the salt concentration of an unknown solution. Follow the 

directions step-by-step and write your answers to the questions in the space provided in your 

booklet. 
E-23 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

1. Open the plastic bottle labeled Distilled Water. The salt concentration of this water is 

very close to 0 percent. Pour the distilled water into the cylinder up to the black line. Put the 

cap back on the bottle. 

Now take the pencil and put it in the water in the cylinder, eraser-end down. Part of the pencil 

will float above the water, as shown in the picture below. 

Explain why the pencil floats when it is put in the water. 

 

 
E-24 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

2. Look at the pencil in the water. There are letters along the side of the pencil. Make sure that 

the pencil is not touching the side of the cylinder. Note the exact level where the water surface 

meets the side of the pencil, as shown in Picture A. Then draw a line on Picture B where the 

water surface comes to on your pencil. This line will help you to remember where the water 

level came to on your pencil for the next step (3). 
E-25 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

3. Now take the pencil out of the water and dry it with a paper towel. Use the ruler to measure 

the length of the pencil that was above the water. Record the length in Table 1 below under 

Measurement 1. 

TABLE 1 

Length of Pencil Above Water Surface (cm) Type of Solution 

Measurement 1 Measurement 2 Average 

Distilled Water 

Salt Solution 

Unknown 

Salt Solution 

 

4. Now place the pencil back in the distilled water and repeat steps 2 and 3. Record your 

measurement in Table 1 under Measurement 2. 

5. Calculate the average of Measurements 1 and 2 and record the result in the data table. 

(You can calculate the average by adding Measurement 1 + Measurement 2 and then dividing 

by two.) 
E-26 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

6. Explain why it is better to measure the length of the pencil that was above the water more 

than once. 

 

 

Now pour the distilled water out of the cylinder into the large plastic bowl. Later you will 

throw this water away. 

Open the plastic bottle labeled Salt Solution. This solution contains 25% salt. Pour the salt 

solution into the cylinder up to the black line. Put the cap back on the bottle. 

7. Now take the pencil and put it in the 25% salt solution in the cylinder, eraser-end down. How 

does the pencil float in this solution compared to how it floated in the distilled water? (Fill in 

the oval in front of the correct answer.) 

 In the salt solution, more of the pencil is above the surface. 

 In the salt solution, more of the pencil is below the surface. 
E-27 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

8. Now use the same procedure that you used with the pencil in the distilled water to obtain two 

measurements of the length of the pencil that floats above the surface of the 25% salt solution. 

Record these two measurements in Table 1. Then calculate the average and record this result in 

the table. 

9. Why does the pencil float at a different level in the salt solution than in the distilled water? 

 

 

10. If you added more salt to the 25% salt solution and stirred the solution until the salt was 

dissolved, how would this change the way that the pencil floats? (Fill in the oval in front of the 

correct answer.) 

Less of the pencil would be above the surface. 

More of the pencil would be above the surface. 

 There would be no difference in the amount of pencil above the surface. 

Now pour the 25% salt solution out of the cylinder into the large plastic bowl. Later you will 

throw this solution away. 
E-28 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

Now open the plastic bottle labeled Unknown Salt Solution. You will now estimate the 

concentration of this unknown salt solution. Pour the unknown solution into the cylinder up to 

the black line. Put the cap back on the bottle. 

11. Put the pencil in the solution in the cylinder, eraser-end down. Then repeat the same 

procedure that you used for the distilled water and the 25% salt solution. Obtain two 

measurements of the length of the pencil that floats above the surface of the unknown salt 

solution. Record these two measurements in Table 1. Then calculate the average and record 

this result in the table. 
E-29 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

12. On the graph below, plot the average values you obtained for the distilled water and the 

25% salt solution. Draw a straight line between the two data points. Assume that this line 

represents the relationship between the length of pencil that is above the water surface and the 

concentration of salt in the water.

 

 3.0 

 

 2.5 

 2.0 

 1.5 

 1.0 

 0.5 

Average Length of Pencil Above Water Surface (cm) 

 0 5 10 15 20 25 

 % Salt in Water 

13. Based on the graph that you plotted, how does the length of the pencil that is above the 

surface change when the salt concentration changes? (Fill in the oval in front of the correct 

answer.) 

It increases as the salt concentration increases. 

It decreases as the salt concentration increases. 

 It remains constant as the salt concentration increases. 
E-30 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

14. Based on the graph that you plotted, what is the salt concentration of the unknown solution? 

 

 
E-31 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

Scoring guides 

Item 1 

Complete 

Student response explains that the pencil floats because it (or the wood of the pencil) is less dense than water.

Essential 

Student response mentions density in some correct reasonable way, but does not clearly say that the pencil is 

less dense than water.

Partial 

Student response demonstrates partial understanding of why the pencil floats or a reference to buoyancy, 

weight, pressure, forces, or lightness is made, but no mention is made of the key concept of density. 

Unsatisfactory/Incorrect 

Student response indicates a lack of understanding of the fact that the wood of the pencil is less dense than 

water.

Items 3, 4, 8, and 11 

Rationale: Student demonstrates an ability to accurately observe, measure, and record the length of the pencil 

floating above the water surface. In each case, the student measures two lengths that are within +/- 0.2 cm or 

+= 1/16 inch of each other. The distilled water should have the lowest value, the salt solution the highest.

Complete 

All three sets of measurements agree within tolerances, and the relative order of the solutions is correct.

Essential 

All three sets of measurements agree within tolerances, but the relative order is incorrect.

Partial 

Only two sets of measurements agree within tolerances. 

Unsatisfactory/Incorrect 

One or no sets of measurements agree within tolerances.

Items 5, 8, and 11, which are scored together 

Rationale: Student demonstrates an ability to accurately calculate the average of two measurements. 

Complete 

All three of the student-calculated averages is within +/- .01 cm. or +/- 1/32 inch of the correct average, as 

calculated from the measurements taken by the student. 

Partial 

Two of the three student-calculated averages is within +/- .01 cm. or +/- 1/32 inch of the correct average, as 

calculated from the measurements taken by the student. 

OR 

One of the three student-calculated averages is within +/- .01 cm. or +/- 1/32 inch of the correct average, as 

calculated from the measurements taken by the student. 

Unsatisfactory/Incorrect 

None of the three student-calculated averages is within +/- .01 cm. or +/- 1/32 inch of the correct average, as 

calculated from the measurements taken by the student. 
E-32 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

Scoring guides 

Item 6 

Rationale: Student demonstrates an understanding of the concept of uncertainty (and error) in measurement 

by explaining that error in measurement can be reduced by taking several measurements and calculating their 

average. 

Complete 

Student response acknowledges the differences between consecutive measurements and the fact that error is 

reduced when an average of several measures is obtained. 

Partial 

Student response makes a reference to variability of measurement without providing a complete explanation. 

OR 

Student response refers to making a mistake when measuring. 

OR 

Student response refers to taking an average without mentioning the variability of measurements. 

Unsatisfactory/Incorrect 

Student response fails to acknowledge that a different answer might be obtained every time a measurement is 

made (i.e., that by making three measurements and dividing by three, error is reduced). 

Item 7 

Correct answer: In the salt solution, more of the pencil is above the surface. 

Item 9 

Rationale: Student demonstrates an understanding of the concept of relative density by explaining that the 

salt in the 25% salt solution causes this solution to have a higher density relative to the wood in the pencil 

than the distilled water does, and that the higher the relative density, the higher the pencil will float. 

Complete 

Student response indicates that the 25% salt solution is more dense that the distilled water, and may say that 

the pencil floats higher in the solution that is more dense. 

Partial 

Student response fails to identify the difference in density between the 25% salt solution and the distilled 

water, but does make reference to a difference between the two. The student may also refer to density of the 

salt solution, but compare it to the pencil instead of the distilled water. 

Unsatisfactory/Incorrect 

Student response fails to indicate that the 25% salt solution is more dense than the distilled water, and that 

this difference accounts for the pencil floating higher in the salt solution than in the distilled water. 

Item 10 

Correct answer: More of the pencil would be above the surface. 
E-33 

Task 1—continued 

NAEP 1996 hands-on task – grade 8 

Scoring guides 

Item 12 

Rationale: Student demonstrates an ability to accurately plot two data points and connect these points with a 

straight line. The graph should match student data. 

Note: Points will be accepted as correct if they are within one line of exact value. Vertical lines or bars also 

okay. The points should be clearly marked on the graph – if there are more than two and there is no line 

drawn, score as Unsatisfactory/Incorrect. 

Complete 

Student correctly plots the results for both the distilled water and the salt solution and draws a line between 

the two data points. 

Partial 

Student correctly plots one point required to draw the line but not both. 

OR 

Student plots both points correctly but fails to connect them with a straight line, or does not connect them. 

Unsatisfactory/Incorrect 

Student fails to accurately plot the results for both the distilled water and the salt solution. 

Item 13 

Correct answer: It increases as the salt concentration increases. 

Item 14 

Rationale: Student demonstrates an ability to accurately make inferences from a line graph (or from data if 

the student does not have a good graph). 

Complete 

Student gives a salt concentration consistent with the data and a satisfactory explanation is provided as to 

how the answer was obtained (i.e., by reading off the graph the point on the X-axis (% salt concentration) at 

which the point on the Y-axis (length of pencil above the water) intersects the line drawn by the student). 

Proportional reasoning is also appropriate. If attempt to explain is not clear or exactly correct, a point or 

other indication on the graph is acceptable. 

Essential 

Student gives a salt concentration consistent with the graph, but does not give a correct explanation as to how 

the answer was obtained, or explains how to use the graph but makes an error in the value. 

Partial 

Student response shows a use of proportional reasoning in the explanation, or an unclear explanation of how 

to use the graph, but does not have a graph that could be used to interpolate, or student uses the graph 

incorrectly. 

Unsatisfactory/Incorrect 

Student does not obtain a value consistent with the graph and does not give a correct explanation of the 

acceptable method of interpolation. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress 

(NAEP), 1996 Science Assessment.
E-34 

TASK 2 

TIMSS 2003 problem solving and inquiry task – grade 8 

Metal Crown 

Instructions: Questions 1,2, 3, and 4 are about an investigation of the properties of a metal 

crown. To answer these questions, you may use any information shown on the pages in the 

Metal Crown section. 

A king gave a jeweler a block of pure metal. He asked the jeweler to make him a crown out 

of the metal. 

After the jeweler delivered the crown, the king observed it carefully. He thought that the 

jeweler might have substituted another pure metal or a mixture of metals to make the crown. 

He weighed the crown, and it had the same mass as the original block, 2400 grams. Still not 

satisfied, the king asked some scientists to help him find out what the crown was made of. 

 Questions for Metal Crown begin on the next page. 
E-35 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Question 1 

The scientists decided to compare the densities of the crown and a block of metal just like the 

original block. The density of a substance is the mass of a sample of the substance divided by 

its volume (density = mass/volume). 

The scientists found the volume of the block and computed its density based on its known 

mass (2400g). The diagram below shows the dimensions of the block of metal that the 

scientists measured. 

What is the density of the block of metal? 

Answer: ___________ g/cm3 
E-36 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Question 2 

The scientists then needed to find the volume of the crown in order to determine its density. 

The following equipment and materials were available for them to use. 

Describe a procedure that the scientists could use to find the volume of the crown using some 

or all of the equipment and materials shown above. You may use diagrams to help explain 

your procedure. 
E-37 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Question 3 

The scientists measured the volume of the crown five times. They computed the density for 

each volume measurement. Their results are shown in the table below. 

A. Why did the scientists measure the volume five times? 

B. The scientists reported to the kind that the density of the crown was 12.0 g/cm3

. Show how 

the scientists used their results to obtain this value for the density. 
E-38 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Question 4 

The table below lists the density for different metals. 

A. Look at the density you computed for the block of metal. What was the block of metal 

most likely made of? 

Answer: ________________ 

Explain your answer. 

B. The density of the crown was found to be 12.0 g/cm3

. What would you report to the king 

about what metal or mixture of metals the jeweler used to make the crown? 
E-39 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Scoring guides 

Question 1 

Correct response 

19.2 g/cm3 [extra trailing zeroes may also be added]. 

OR 

19 g/cm3 [rounds to nearest whole unit]. 

Incorrect response 

Shows the set up for density (mass/volume) but does not compute density or makes a computational error. 

OR 

125 [Computes volume but not density]. 

OR 

19.3 [No work shown; indicates density copied from table.] 

Question 2 

Correct Response 

Describes or diagrams a procedure based on displacement of water using measured water level differences: 

i) Adding water to the beaker (sink or tray) and marking the water level.

ii) Placing the crown in the beaker (sink or tray) and marking the new water level.

iii) Measuring the volume difference before/after adding the crown using the graduated cylinder.

OR 

Describes or diagrams a procedure based on displacement of water using measured overflow:

i) Filling the beaker (or tray) with water 

ii) Placing the crown in the beaker (or tray) and collecting the overflow. 

iii) Measuring the volume of the overflow using the graduated cylinder. 

Partial Response 

Describes or diagrams a partial procedure that includes displacement of water but with inadequate or no 

description of the steps/measurements to determine the volume. 

Incorrect Response 

Mentions putting the crown in the beaker (sink or tray) of water with no explicit mention that the water level 

will rise/overflow and no or incorrect procedure given for measuring the volume. 

Question 3a 

Correct Response 

Refers to accuracy, precision, reliability, experimental uncertainty, estimation of measurement error (or 

similar). 

OR 

Refers only to computing an average or mean value (or median or range). 

Incorrect Response

Refers only to 'mistakes' or changes in the measurements (or similar); no explicit mention of accuracy, 

precision, experimental uncertainty, etc. 

OR 

Refers only to a 'fair test' or similar; no explicit mention of computation of average, accuracy, precision, 

experimental uncertainty, etc. 
E-40 

Task 2—continued 

TIMSS 2003 problem solving and inquiry task – grade 8 

Scoring guides 

Question 3b 

Correct Response 

Shows (or describes) a correct method for computing the average (mean) value. 

OR 

Shows (or describes) a correct method for determining the median value. 

Incorrect Response 

States that it is the average, mean or median value with no or incorrect work shown. 

OR 

Shows a computation of density (mass/volume). [No detemination of average or median included.] 

Question 4a 

Note: To receive credit, responses must identify gold AND give an explanation based on density. Responses 

that identify gold with no or incorrect explanation are scored as incorrect. It is possible that a different metal 

or metal(s) may be identified based on an incorrect density computation in the previous question. These 

types of responses may be scored as correct, provided the explanation is reasonable based on the computed 

density. 

Correct Response 

GOLD with an explanation based on correct density computed in previous question (19.2/cm3). 

Incorrect Response 

GOLD with no explanation or incorrect explanation that is NOT based on density. 

OR 

SILVER (alone or mixed). [Confuses density of crown with density of metal block.] 

Question 4b 

Note: To receive credit, responses must indicate that the crown is composed of a mixture of metals (alloy) 

AND identify the metals that might be included based on the density (crown density between the densities of 

pure metals). Responses that indicate that the crown is made of a mixture (alloy) or is not pure gold with no 

further information about what other metals are included are scored as incorrect. If responses indicate that 

the crown is made of Palladium (not shown in the table but with a density of 12 g/cm3), they should be 

scored as correct. 

Correct Response 

Reports that the crown is made of a mixture (alloy) AND names specific metal(s) that might be included 

(reasonable composition based on density). 

Incorrect Response 

Reports only that the crown is made of a mixture or is NOT pure gold (or similar); NO specific materials are 

named. 

OR 

Reports SILVER (density closest to 12 g/cm3). 

OR 

Reports an incorrect mixture of metals based on additive densities. 

SOURCE: International Association for the Evaluation of Educational Achievement, Trends in International Mathematics and Science 

Study (TIMSS), 2003 Assessment.