*Assessment and Item Specifications for the NAEP 2009 Mathematics Assessment*

National Assessment of Educational Progress

June 2007

##### National Assessment Governing Board

**U.S. Department of Education**

1Chapter One:Introduction

1What Is An Assessment specifications document?

1The Need for a New Framework and Specifications at Grade 12

2The Framework and Specifications Development Process

2Changes from the 2005 Framework and Specifications

2Conclusion and Preview of Document

4Chapter Two: Item Specifications by Grade level and Content area

4Content Areas

52009 NAEP Mathematics Objectives Organization

6Mathematical Content Areas

47Chapter Three:Mathematical Complexity of Items

48Low Complexity

52Moderate Complexity

56High Complexity

61Chapter Four:Item Development

61Item Characteristics

61General Principles of Item Writing

71Inclusion of English Language Learners and Students with Disabilities

77Item Formats

91Item Tryouts and Reviews

94Chapter Five: Design of the Assessment

94Alignment of the Assessment and the Framework

95Accessibility

95Reporting Requirements

96Balance of Content

96Balance of Mathematical Complexity

97Balance of Item Formats

97Balance of Item Contexts

99NAEP Administration and Student Sampling

100Calculators

103Manipulatives and Tools

104Item Families

108References

110Appendix A: NAEP Mathematics Achievement Level Descriptions

114Appendix B: NAEP Item Development and Review Policy Statement

124Appendix C: NAEP Mathematics Project Staff and Committees

132Acknowledgments

# Chapter One

## Introduction

Since 1973 the National Assessment of Educational Progress (NAEP) has gathered information about student achievement in mathematics. Results of these periodic assessments, produced in print and web-based formats, provide valuable information to a wide variety of audiences. They inform citizens about the nature of students’ comprehension of the subject, curriculum specialists about the level and nature of student achievement, and policymakers about factors related to schooling and its relationship to student proficiency in mathematics. 

The NAEP assessment in mathematics is accomplished in two different ways. One assessment measures long-term trends in achievement among 9, 13, and 17-year old students using the same basic design each time. This unique measure has allowed for comparisons of students’ knowledge of mathematics since the assessment was first administered in 1973. The main NAEP assessment is administered at the national, state, and selected urban district levels. Results are reported on student achievement in grades 4, 8, and 12 at the national level and for grades 4 and 8 at the state and trial urban district level. The main NAEP assessment is based on a framework that can be updated periodically. The *Assessment and Item Specifications for the NAEP 2009 Mathematics Assessment*only reflects changes in grade 12 from 2005. Mathematics content objectives for grades 4 and 8 have not changed from 2005. Therefore, main NAEP trendlines from the early 1990’s can continue at 4th and 8th grades for the 2009 assessment. Taken together, the NAEP assessments provide a rich, broad, and deep picture of student mathematics achievement in the United States. 

###  What Is An Assessment Specifications Document?

This document is a companion to the NAEP *Mathematics Framework for 2009.*The framework lays out the basic design of the assessment by describing the mathematics content that should be tested and the types of assessment questions that should be included. It also describes how the various design factors should be balanced across the assessment. The assessment and item specifications give more detail about development of the items and conditions for the 2009 NAEP mathematics assessment. It contains much of the same information that is in the framework about the mathematics content and other dimensions of the assessment, but adds further detail. The intended audience for the specifications is test developers and item writers. 

### The Need For a New Framework and Specifications at Grade 12

For several years, the National Assessment Governing Board has been focusing special attention on ways to improve the assessment of 12th graders by the National Assessment of Educational Progress. The goal for this 12th grade initiative is to enable reporting on how well prepared 12th grade students are for post-secondary education and training. To accomplish this goal the content of the assessments, as described in the 2005 mathematics framework, was analyzed and revisions considered. The challenge was to find the essential mathematics that can form the foundation for these post-secondary paths. This must include use of quantitative tools, broad competence in mathematical reasoning, mathematics required for postsecondary courses, and the ability to integrate and apply mathematics to diverse problem-solving contexts. Analysis of the 2005 framework revealed that some revisions would be necessary to meet this challenge.

### The Framework and Specifications Development Process

To implement this change at the 12th grade, the Governing Board contracted with Achieve, Inc. to examine NAEP’s mathematics assessment framework in relation to benchmarks set by the American Diploma Project. An Achieve panel of mathematicians, mathematics educators, and policymakers proposed increasing the scope and rigor of 12th grade NAEP. New assessment objectives were developed by Achieve, and then a panel of mathematicians and mathematics educators (including classroom teachers) reviewed and revised the objectives and matched them against the current set of objectives for grades 4 and 8. The panel conducted focus groups with the Association of State Supervisors of Mathematics and survey reviews with various NAEP constituents, using repeated rounds of reviews. The final set of grade 12 objectives was approved by the Governing Board in August 2006.

### Changes from the 2005 Framework and specifications

The chart below compares the 2009 to the 2005 mathematics framework and specifications:

### Conclusion and Preview of Document

The assessment and item specifications for the NAEP 2009 mathematics assessment follow. The bullets below summarize each chapter:

· **Mathematics content**

Chapter Two contains descriptions of the five major content areas of mathematics (Number Properties and Operations, Measurement, Geometry, Data Analysis, Statistics, and Probability, and Algebra), as well as the specific objectives for grades 4, 8 and 12 that will be assessed. Further specifications are added to some objectives to clarify the intent for item writers.

· **Mathematical complexity**

Each NAEP mathematics test item is designed to measure a specific level of thinking, called the mathematical complexity of the item. Chapter Three describes the three levels and offers examples of each.

· **Item development**

Chapter Four describes considerations for good item writing, with multiple examples of how each characteristic of an item might be met. The chapter also contains a description of the item tryout and review process.

· **Design of the assessment**

Each form of the NAEP mathematics assessment must be balanced according to a number of different factors, including content, level of complexity, and format. In Chapter Five the guidelines for balancing each factor are described. The chapter also addresses other issues of design, such as sampling, use of calculators, tools and manipulatives, and accessibility for all students.

A valuable resource for learning more about NAEP can be found on the Internet at http://nces.ed.gov/nationsreportcard/. This site contains reports describing results of recent assessments, as well as a searchable tool for viewing released items. The items can be searched by many different features, such as grade level and content area. Information about the items includes student performance and any applicable scoring rubrics. NAEP released items that are used as examples in this document are marked with the designation that matches the item name in the NAEP Sample Question Tool, found on the website.

Chapter Two

## Item Specifications by Grade level and Content Area

This chapter presents an overview of the content areas, a description of the matrix format, and a detailed description of each content area followed by the specific objectives of the mathematics framework. In addition there are guidelines provided for item writers intended to help clarify content area specifications as well as individual objectives.

### Content Areas

Although the names of the content areas in previous NAEP frameworks, as well as some of the topics in those areas, may have changed somewhat from one assessment to the next, there has remained a consistent focus toward collecting information on student performance in five key areas. The framework for the 2009 Mathematics Assessment is anchored in these same five broad areas of mathematical content:

**Number Properties and Operations** (including computation and the understanding of number concepts)

**Measurement** (including use of instruments, application of processes, and concepts of area and volume)

**Geometry** (including spatial reasoning and applying geometric properties)

**Data Analysis, Statistics, and Probability** (including graphical displays, and statistics)

**Algebra**(including representations and relationships)

These divisions are not intended to separate mathematics into discrete elements. Rather, they are intended to provide a helpful classification scheme that describes the full spectrum of mathematical content assessed by NAEP. Classifying items into one primary content area is not always clear cut, but doing so brings us closer to the goal of ensuring that important mathematical concepts and skills are assessed in a balanced way.

At grade 12, the five content areas are collapsed into four, with geometry and measurement combined into one for assessment development purposes. This reflects the fact that the majority of measurement topics suitable for twelfth-grade students are geometrical in nature. Separating these two areas of mathematics at grade 12 becomes forced and unnecessary. 

It is important to note that there are certain aspects of mathematics that occur in all the content areas. The best example of this is computation. Computation is the skill of performing operations on numbers. It should not be confused with the content area of NAEP called Number Properties and Operations, which encompasses a wide range of concepts about our numeration system. Certainly the area of Number Properties and Operations includes a variety of computational skills, ranging from operations with whole numbers to work with decimals and fractions and finally real numbers. But computation is also critical in Measurement and Geometry, such as in calculating the perimeter of a rectangle, estimating the height of a building, or finding the hypotenuse of a right triangle. Data analysis often involves computation, such as calculating a mean or the range of a set of data. Probability often entails work with rational numbers. Solving algebraic equations usually involves numerical computation as well. Computation, therefore, is a foundational skill in every content area. While the main NAEP assessment is not designed to report a separate score for computation, results from the long-term NAEP assessment can provide insight into students’ computational abilities.

As described in Chapter One, one of the changes made from the 2005 framework is the addition of a subtopic for mathematical reasoning that appears in Number, Geometry, Data Analysis, Statistics and Probability, and Algebra. At grades 4 and 8, no new objectives were written, but some of the objectives from the 2005 framework were moved into this new subtopic area. This reflects a new emphasis on the importance of mathematical reasoning across each of the content areas.

### 2009 NAEP Mathematics Objectives Organization 

The specifications matrix is organized by the five NAEP content areas: Number Properties and Operations; Measurement, Geometry; Data Analysis, Statistics and Probability; and Algebra. Though such an organization brings with it the danger of fragmentation, the intent is the test items will, in many cases, cross some boundaries of these content areas, although an item will emphasize a primary content area.

The specifications matrix depicts the particular objectives appropriate for assessment under each subtopic. Within Number, for example, and the subtopic of Number Sense, specific objectives are listed for assessment at grade 4, grade 8, and grade 12. The same topic at different grade levels depicts a developmental sequence for that concept or skill. An empty cell in the matrix is used to convey the fact that a particular objective is not appropriate for assessment at that grade level. Guidelines for item writers (in italics) are included when needed to clarify the scope or measurement intent of the objectives.

**In order to fully understand the objectives and their intent, please note the following:**

The objectives included in the matrix describe what is to be assessed on the 2009 NAEP. They should not be interpreted as a complete description of mathematics that should be taught at these grade levels.

· Some of the 12th grade objectives are marked with a “*”. This denotes objectives that describe mathematics content that is beyond that typically taught in a standard 3-year course of study (the equivalent of one year of geometry and two years of algebra). Therefore, these objectives will be selected less often than the others for inclusion on the assessments.

· While all test items will be assigned a primary classification, some test items could potentially fall into more than one objective or more than one content area.

· When the guidelines use “include,” this means that items can include these features, not that most items should include these features.

When the word “or” is used in an objective, it should be understood as inclusive; that is, an item may assess one or more of the concepts included. 

A valuable resource for learning more about NAEP can be found on the Internet at http://nces.ed.gov/nationsreportcard/. This site has reports describing results of recent assessments, as well as a searchable tool for viewing released items. The items can be searched by different features, such as grade level and content area. Information about the items includes student performance and any applicable scoring rubrics.

### Mathematical Content Areas

#### NUMBER PROPERTIES AND OPERATIONS

Numbers are our main tools for describing the world quantitatively. As such they deserve a privileged place in the NAEP Mathematics framework. With whole numbers, we can count collections of discrete objects of any type. We can also use numbers to describe fractional parts, and even to describe continuous quantities such as length, area, volume, weight, and time, and more complicated derived quantities such as rates—speed, density, inflation, interest, and so forth. Thanks to Cartesian coordinates, we can use pairs of numbers to describe points in a plane or triples of numbers to label points in space. Numbers let us talk in a precise way about anything that can be counted, measured, or located in space.

Numbers are not simply labels for quantities. They form systems with their own internal structure. The arithmetic operations (addition and subtraction, multiplication and division) help us model basic real world operations. For example, joining two collections, or laying two lengths end to end, can be described by addition, while the concept of rate depends on division. Multiplication and division of whole numbers lead to the beginnings of number theory, including concepts of factorization, remainder, and prime number. Besides the arithmetic operations, the other basic structure of the real number system is its ordered nature. This allows comparison of numbers, as to which is greater or lesser. Ordering and comparing reflect our intuitions about the relative size of quantities and provide a basis for making sensible estimates.

The accessibility and usefulness of arithmetic is greatly enhanced by our efficient means for representing numbers — the Hindu-Arabic decimal place value system. In its full development, this remarkable system includes decimal fractions, which let us approximate any real number as closely as we wish. Decimal notation allows us to do arithmetic by means of simple, routine algorithms, and it also makes size comparisons and estimation easy. The decimal system achieves its efficiency through sophistication, as all the basic algebraic operations are implicitly used in writing decimal numbers. To represent ratios of two whole numbers exactly, we supplement decimal notation with fractions.

Comfort in dealing with numbers effectively is called *number sense*. It includes firm intuitions about what numbers tell us; an understanding of the ways to represent them symbolically (including facility with converting between different representations); ability to calculate, either exactly or approximately, and by several means—mentally, with paper and pencil, or with a calculator, as appropriate; and skill in estimation. Ability to deal with proportion, including percents, is another important part of number sense.

Number sense is a major expectation of the NAEP Mathematics Assessment. At 4th grade, students are expected to have a solid grasp of the whole numbers, as represented by the decimal system, and to have the beginnings of understanding fractions. By 8th grade, they should be comfortable with rational numbers, represented either as decimal fractions (including percents) or as common fractions. They should be able to use them to solve problems involving proportionality and rates. In middle school also, number should begin to coalesce with geometry, via the idea of the number line. This should be connected with ideas of approximation and the use of scientific notation. Eighth graders should also have some acquaintance with naturally occurring irrational numbers, such as square roots and pi. By twelfth grade, students should be comfortable dealing with all types of real numbers.

##### Number Properties and Operations

The word “expressions” refers to *numerical* expressions in this content area. 

Italicized print in the matrix indicates item development guidelines.

Measurement

Measuring is the process by which numbers are assigned in order to describe the world quantitatively. This process involves selecting the attribute of the object or event to be measured, comparing this attribute to a unit, and reporting the number of units. For example, in measuring a child, we may select the attribute of height and the inch as the unit for the comparison. In comparing the height to the inch, we may find that the child is about 42 inches. If considering only the domain of whole numbers, we would report that the child is 42 inches tall. However, since height is a continuous attribute, we may consider the domain of rational numbers and report that the child is 41 3/16 inches tall (to the nearest sixteenth of the inch). Measurement also allows us to model positive and negative numbers as well as the irrational numbers.

This connection between measuring and number makes measuring a vital part of the school curriculum. Measurement models are often used when students are learning about number and operations. For example, area and volume models can help students understand multiplication and the properties of multiplication. Length models, especially the number line, can help students understand ordering and rounding numbers. Measurement also has a strong connection to other areas of school mathematics and to the other subjects in the school curriculum. Problems in algebra are often drawn from measurement situations. One can also consider measurement to be a function or a mapping of the attribute to a set of numbers. Much of school geometry focuses on the measurement aspect of geometric figures. Statistics also provides ways to measure and to compare sets of data. These are some of the ways that measurement is intertwined with the other four content areas.

In this NAEP mathematics framework, attributes such as capacity, weight/mass, time and temperature are included as well as the geometric attributes of length, area, and volume. Although many of these attributes are included in the grade 4 framework, the emphasis is on length, including perimeter, distance, and height. More emphasis is placed on area and angle in grade 8. By grade 12, volumes and rates constructed from other attributes, such as speed, are emphasized. 

Units involved in items on the NAEP assessment include non-standard, customary, and metric units. At grade 4, common customary units such as inch, quart, pound, and hour and the common metric units such as centimeter, liter, and gram are emphasized. Grades 8 and 12 include the use of both square and cubic units for measuring area, surface area, and volume, degrees for measuring angles, and constructed units such as miles per hour. Converting from one unit in a system to another such as from minutes to hours is an important aspect of measurement included in problem situations. Understanding and using the many conversions available is an important skill. There are a limited number of common, everyday equivalencies that students are expected to know.

Items classified in this content area depend on some knowledge of measurement. For example, an item that asks the difference between a 3-inch and a 1 3/4 inch line segment is a number item, while an item comparing a 2-foot segment with an 8-inch line segment is a measurement item. In many secondary schools, measurement becomes an integral part of geometry; this is reflected in the proportion of items recommended for these two areas. 

##### General Guidelines for Measurement

Any attribute, unit, instrument, conversion factor, or formula included in the list at a lower grade is also appropriate for the higher grade(s).

Attributes

The following attributes may be included in items:

· Grade 4 – Length, time, temperature, capacity, weight, and area with emphasis on length (length includes perimeter, height, and distance).

· Grade 8 – Angle and volume. Emphasis is on area. Attributes such as speed, measured in terms of the attributes of time and distance, are also appropriate.

· Grade 12 – The emphasis is on area (including surface area) and volume, but any attribute used in grade 8 is appropriate. Rates constructed from other attributes such as speed or flow rate are appropriate.

Units

· Grade 4 – Non-standard units, common customary units (inch, foot, mile, cup, quart, gallon, pound, hour, minute, day, year) and metric units (centimeter, millimeter, meter, liter, gram) for the allowed attributes at this grade level. 

· Grade 8 – Square units and cubic units, degrees of angles, and constructed units such as miles per hour; metric units most commonly used for each of the attributes are appropriate also.

· Grade 12 – Same as grade 8.

Instruments

The following instruments are commonly found in curricula; variations based on the same principles could be used (e.g., graduated cup measures):

· Grade 4 – Ruler, clock, thermometer, graduated cylinder, balance scales, scales.

· Grade 8 – Protractor.

· Grade 12 – Same as grade 8.

Conversions

Items should be based on students’ knowing specific equivalences, as follows:

· Grade 4 – Feet/inches, hours/minutes, and meters/centimeters; other simple conversions should be given, such as 2 pints = 1 quart.

· Grade 8 – Square and cubic unit conversions; students should also know all common time equivalences, all common metric equivalence. Otherwise, conversions should be based on provided equivalences.

· Grade 12 – Conversions from constructed units such as miles per hour to feet per minute.

Formulas

Grade 4 students are not expected to know any measurement formulas; however, they are expected to know how to find the perimeter and area of a rectangle. Both grade 8 and grade 12 students should know formulas for the area of a rectangle, triangle, and circle, the circumference of a circle, and the volume of a cylinder and rectangular solid. If other formulas are used, they should be given. (See General Guidelines for Geometry for more information about formulas for area, circumference, and volume.)

##### Measurement

Italicized print in the matrix indicates item development guidelines.

Geometry

Geometry began as a practical collection of rules for calculating lengths, areas, and volumes of common shapes. In classical times, the Greeks turned it into a subject for reasoning and proof, and Euclid organized their discoveries into a coherent collection of results, all deduced using logic from a small number of special assumptions, called postulates. Euclid’s *Elements* stood as a pinnacle of human intellectual achievement for over 2000 years.

The 19th century saw a new flowering of geometric thought, going beyond Euclid, and leading to the idea that geometry is the study of the possible structures of space. This had its most striking application in Einstein's theories of relativity, which described the behavior of light, and also of gravity, in terms of a four-dimensional geometry, which combines the usual three dimensions of space with time as an additional dimension.

A major insight of the 19th century is that geometry is intimately related to ideas of symmetry and transformation. The symmetry of familiar shapes under simple transformations—that our bodies look more or less the same if reflected across the middle, or that a square looks the same if rotated by 90o—is a matter of everyday experience. Many of the standard terms for triangles (scalene, isosceles, equilateral) and quadrilaterals (parallelogram, rectangle, rhombus, square) refer to symmetry properties. Also, the behavior of figures under changes of scale is an aspect of symmetry with myriad practical consequences. At a deeper level, the fundamental ideas of geometry itself (for example, congruence) depend on transformation and invariance. In the 20th century, symmetry ideas were seen to underlie much of physics also, not only Einstein's relativity theories, but atomic physics and solid state physics (the field that produced computer chips).

School geometry roughly mirrors the historical development through Greek times, with some modern additions, most notably symmetry and transformations. By grade 4, students are expected to be familiar with a library of simple figures and their attributes, both in the plane (lines, circles, triangles, rectangles, and squares), and in space (cubes, spheres, and cylinders). In middle school, understanding of these shapes deepens, with study of cross-sections of solids, and the beginnings of an analytical understanding of properties of plane figures, especially parallelism, perpendicularity, and angle relations in polygons. Right angles and the Pythagorean Theorem are introduced, and geometry becomes more and more mixed with measurement. The basis for analytic geometry is laid by study of the number line. In high school, attention is given to Euclid's legacy and the power of rigorous thinking. Students are expected to make, test, and validate conjectures. Via analytic geometry, the key areas of geometry and algebra are merged into a powerful tool that provides a basis for calculus and the applications of mathematics that helped create the modern technological world in which we live.

Symmetry is an increasingly important component of geometry. Elementary students are expected to be familiar with the basic types of symmetry transformations of plane figures, including flips (reflection across lines), turns (rotations around points), and slides (translations). In middle school, this knowledge becomes more systematic and analytical, with each type of transformation being distinguished from other types by their qualitative effects. For example, a rigid motion of the plane that leaves at least two points fixed (but not all points) must be a reflection in a line. In high school, students are expected to be able to represent transformations algebraically. Some may also gain insight into their systematic structure, such as the classification of rigid motions of the plane as reflections, rotations, or translations, and what happens when two or more isometries are performed in succession (composition).

General Guidelines for Geometry

Students are expected to know the basic formulas for shapes as indicated in the chart below.

**Key:                      Not tested**

**Students are expected to know the formula**

**Formula should be provided**

Geometry

Italicized print in the matrix indicates item development guidelines.

Data Analysis, Statistics, and Probability

 Data analysis covers the entire process of collecting, organizing, summarizing, and interpreting data. This is the heart of the discipline called statistics and is in evidence whenever quantitative information is used in determining a course of action. To emphasize the spirit of statistical thinking, data analysis should begin with a question to be answered—not with the data. Data should be collected only with a specific question (or questions) in mind and only after a plan (usually called a design) for collecting data relevant to the question is thought out. Beginning at an early age, students should grasp the fundamental principle that looking for questions in an existing data set is far different from the scientific method of collecting data to verify or refute a well-posed question. A pattern can be found in almost any data set if one looks hard enough, but a pattern discovered in this way is often meaningless, especially from the point of view of statistical inference.

In the context of data analysis, or statistics, probability can be thought of as the study of potential patterns in outcomes that have not yet been observed. We say that the probability of a balanced coin coming up heads when flipped is one half because we believe that about half of the flips would turn out to be heads if we flipped the coin many times. Under random sampling, patterns for outcomes of designed studies can be anticipated and used as the basis for making decisions. If the coin actually turned up heads 80% of the time, we would suspect that it was not balanced. The whole probability distribution of all possible outcomes is important in most statistics problems because the key to decision-making is to decide whether or not a particular observed outcome is unusual (located in a tail of the probability distribution) or not. For example, four as a grade point average is unusually high among most groups of students, four as the pound weight of a baby is unusually low, and four as the number of runs scored in a baseball game is not unusual in either direction. 

By grade 4, students should be expected to apply their understanding of number and quantity to pose questions that can be answered by collecting appropriate data. They should be expected to organize data in a table or a plot, and summarize the essential features of center, spread, and shape both verbally and with simple summary statistics. Simple comparisons can be made between two related data sets, but more formal inference based on randomness should come later. The basic concept of chance and statistical reasoning can be built into meaningful contexts, though, such as “If I draw two names from among those of the students in the room, am I likely to get two girls?”  Such problems can be addressed through simulation. 

Building on the same definition of data analysis and the same principles of describing distributions of data through center, spread, and shape, grade 8 students will be expected to use a wider variety of organizing and summarizing techniques. They can also begin to analyze statistical claims through designed surveys and experiments that involve randomization, with simulation being the main tool for making simple statistical inferences. They will begin to use more formal terminology related to probability and data analysis. 

Students in grade 12 will be expected to use a wide variety of statistical techniques for all phases of the data analysis process, including a more formal understanding of statistical inference (but still with simulation as the main inferential analysis tool). In addition to comparing univariate data sets, students at this level should be able to recognize and describe possible associations between two variables by looking at two-way tables for categorical variables or scatterplots for measurement variables.  Association between variables is related to the concepts of independence and dependence, and an understanding of these ideas requires knowledge of conditional probability. These students should be able to use statistical models (linear and non-linear equations) to describe possible associations between measurement variables and be familiar with techniques for fitting models to data. 

##### General Guidelines for Data Analysis, Statistics, and Probability

Data Representation

· Items should include interpretation of uncommon representations of data such as those found in newspapers and magazines.

· Bar and line graphs should increase in complexity (e.g., through using more complex scales) from grade to grade.

· Descriptions of data sets at grade 4 may be informal.

The following representations of data are indicated for each grade level. Objectives in which only a subset of these representations is applicable are indicated in the parenthesis associated with the objective.

*Grade 4*

Pictographs, bar graphs, circle graphs, line graphs, line plots, tables, and tallies

*Grade 8*

Histograms, line graphs, scatterplots, box plots, bar graphs, circle graphs, stem and leaf plots, frequency distributions, and tables

*Grade 12*

Histograms, line graphs, scatterplots, box plots, bar graphs, circle graphs, stem and leaf plots, frequency distributions, and tables, including two-way tables 

##### Data Analysis, Statistics, and Probability

Italicized print in the matrix indicates item development guidelines.

ALGEBRA

Algebra was pioneered in the Middle Ages by mathematicians in the Middle East and Asia as a method of solving equations easily and efficiently by manipulation of symbols, rather than by the earlier geometric methods of the Greeks. The two approaches were eventually united in the analytic geometry of René Descartes. Modern symbolic notation, developed in the Renaissance, greatly enhanced the power of the algebraic method, and from the 17th century forward, algebra in turn promoted advances in all branches of mathematics and science.

The widening use of algebra led to study of its formal structure. Out of this were gradually distilled the “rules of algebra,” a compact summary of the principles behind algebraic manipulation. A parallel line of thought produced a simple but flexible concept of function and also led to the development of set theory as a comprehensive background for mathematics. When it is taken liberally to include these ideas, algebra reaches from the foundations of mathematics to the frontiers of current research.

These two aspects of algebra, a powerful representational tool and a vehicle for comprehensive concepts such as function, form the basis for the expectations throughout the grades. By grade 4, students are expected to be able to recognize and extend simple numeric patterns as one foundation for a later understanding of function. They can begin to understand the meaning of equality and some of its properties, as well as the idea of an unknown quantity, as a precursor to the concept of variable.

As students move into middle school, the ideas of function and variable become more important. Representation of functions as patterns, via tables, verbal descriptions, symbolic descriptions, and graphs can combine to promote a flexible grasp of the idea of function. Linear functions receive special attention. They connect to the ideas of proportionality and rate, forming a bridge that will eventually link arithmetic to calculus. Symbolic manipulation in the relatively simple context of linear equations is reinforced by other means of finding solutions, including graphing by hand or with calculators.

In high school, students should become comfortable in manipulating and interpreting more complex expressions. The rules of algebra should come to be appreciated as a basis for reasoning. Non-linear functions, especially quadratic functions, and also power and exponential functions, are introduced to solve real-world problems. Students should become accomplished at translating verbal descriptions of problem situations into symbolic form. Expressions involving several variables, systems of linear equations, and the solutions to inequalities are encountered by grade 12.****

General Guidelines for Algebra

Overall, items at grade 4 emphasize informal algebra. For example, there is an emphasis on “completing number sentences” instead of “solving equations.”  At grade 8, items cover some formal algebra, but the expectation is that less formal algebra content will be included. For example, solution of higher degree polynomial equations or systems of linear or non-linear equations is not expected at the 8th grade level, but these topics are included at the 12th grade level.

At grade 12, the types of functions eligible for use in all items are linear, quadratic, rational, exponential, and trigonometric. Rational functions are limited to the following set: those with a constant or linear numerator and a linear or quadratic denominator. Rational expressions are limited in the same way. Trigonometric functions are limited to sine, cosine and tangent.

Logarithmic functions can be used in * items only.

##### Algebra

Italicized print in the matrix indicates item development guidelines.

# Chapter Three

## Mathematical Complexity of Items

Each NAEP item assesses an objective that can be associated with a content area of mathematics, such as number or geometry. The item also makes certain demands on students’ thinking. These demands determine the mathematical complexity of the item, which is the second dimension of the mathematics framework. The three levels of mathematical complexity in NAEP assessment are low, moderate, and high.

The demands on thinking that an item expects—what it asks the student to recall, understand, reason about, and do—assume that students are familiar with the mathematics of the task. For example, a task with low complexity might ask students simply to state the formula to find the distance between two points. Those students who had never learned anything about distance formula would not be successful on the task even though the demands were low. Items are developed for administration at a given grade level on the basis of the framework, and complexity of those items is independent of the particular curriculum a student has experienced.

Mathematical complexity deals with *what* the students are asked to do in a task. It does not take into account *how* they might undertake it. In the distance formula task, for instance, some students who had studied the formula might simply reproduce it from memory. Others, however, who could not recall the exact formula, might end up deriving it from the Pythagorean theorem, engaging in a different kind of thinking than the task presupposed.

The categories—low complexity, moderate complexity, and high complexity—form an ordered description of the demands an item may make on a student. Items at the low level of complexity, for example, may ask a student to recall a property. At the moderate level, an item may ask the student to make a connection between two properties; at the high level, an item may ask a student to analyze the assumptions made in a mathematical model. This is an example of the distinctions made in item complexity to provide balance in the item pool. The ordering is not intended to imply that mathematics is learned or should be taught in such an ordered way. Using the levels of complexity to describe that dimension of each item allows for a balance of mathematical thinking in the design of the assessment.

The mathematical complexity of an item is not directly related to its format (multiple-choice, short constructed response, or extended constructed response). Items requiring that the student generate a response tend to make somewhat heavier demands on students than items requiring a choice among alternatives, but that is not always the case. Any type of item can deal with mathematics of greater or less depth and sophistication. There are multiple-choice items that assess complex mathematics, and constructed response items can be crafted to assess routine mathematical ideas. 

The remainder of this chapter gives brief descriptions of each level of complexity as well as examples from previous NAEP assessments to illustrate each level. A brief rationale is included to explain why an item is so classified. All example items found in this chapter can also be found in the companion document, the NAEP *Mathematics Framework for 2009*.

Items in the NAEP assessment should be balanced according to levels of complexity, as described in more detail in Chapter Five. The ideal balance should be as follows:

###### Percent of Testing Time at Each Level of Complexity

### Low Complexity

Low-complexity items expect students to recall or recognize concepts or procedures specified in the framework. Items typically specify what the student is to do, which is often to carry out some procedure that can be performed mechanically. It is not left to the student to come up with an original method or to demonstrate a line of reasoning. The following examples are items that have been classified at the low complexity level.

Moderate Complexity

Items in the moderate-complexity category involve more flexibility of thinking and choice among alternatives than do those in the low-complexity category. The student is expected to decide what to do, and how to do it, bringing together concepts and processes from various domains. For example, the student may be asked to represent a situation in more than one way, to draw a geometric figure that satisfies multiple conditions, or to solve a problem involving multiple unspecified operations. Students might be asked to show or explain their work, but would not be expected to justify it mathematically. The following examples are items that have been classified at the moderate complexity level.

### High Complexity

High-complexity items make heavy demands on students, who are expected to use reasoning, planning, analysis, judgment, and creative thought. Students may be expected to justify mathematical statements or construct a mathematical argument. Items might require students to generalize from specific examples. Items at this level take more time than those at other levels due to the demands of the task, but not due to the number of parts or steps. In the example items at the moderate level, several suggestions were made in the rationale that would make the items at a high level of complexity. The following examples are items that have been classified at the high complexity level. 

Additional examples of items and their classifications can be found in *2009 NAEP Mathematics Assessment and Item Specifications* as well as on the National Center for Education Statistics’ website: http://nces.ed.gov/nationsreportcard/itmrls/. All the released items from recent mathematics assessments can be accessed from this site. The complexity classification is available only for items beginning with the 2005 assessment since this was the first year that the framework specified this dimension.

# Chapter Four

## Item Development

This chapter discusses specifications that apply to all grade levels assessed by the NAEP in mathematics. Chapter Two: Item Specifications by Content Area contains specifications by content area for each grade level. Chapter Three: Mathematical Complexity of Items describes the three levels of complexity of NAEP mathematics items. The guidelines in these three chapters are focused on translating the intent of the content framework into development of items used on the assessment.

These guidelines highlight only some of the critical considerations in item development and concentrates on topics specific to the NAEP mathematics assessment. Item writers should refer to directions for developing items provided by the assessment development contractor in addition to the information in Chapters Two, Three, and Four.

### Item Characteristics

Each item written for the NAEP mathematics assessment reflects two major dimensions: mathematical content area (see Chapter Two) and mathematical complexity (see Chapter Three). 

Items will also vary by 

· format (see Chapter Five, page xx, and this chapter, pages xx-xx),

· calculator status (see Chapter Five, page xx),

· whether manipulatives are required (see Chapter Five, page xx), and 

· whether they are part of a family of items (see Chapter Five, page xx). 

Each of these characteristics is discussed in the specifications, and examples of items illustrating these characteristics are given.

### General Principles of Item Writing

The following principles of good item writing should be used as appropriate, depending upon the measurement intent of the item. For example, a principle under graphics states, “represent each important part of the item in the visual images.”  This principle is appropriate for items with graphics that are used to support students’ interpretation of the text but may not be appropriate for graphics that are used in other ways. The guidelines should be followed unless the targeted construct precludes doing so.

1. Clear Measurement Intent

A critical step in good item writing is making sure that the measurement intent of the item is clear and that students understand what is being measured and what type of response is expected.

a)Clear intent in development 

· Item writers should provide a clear description of what objective(s) and, if applicable, the portion(s) of the objective(s) each item is intended to measure, as well as specify the mathematical complexity of the item along with a rationale for the level of complexity. This will help classify items according to assessment specifications, help develop clear scoring rubrics and scoring materials, reduce confusion in reviews, and provide evidence of the degree of alignment of the assessment to the framework.

· In order to clearly measure the targeted objective(s), each item should be independent. The response to one item should not depend upon the response to another. For example, students should not be asked to determine the cost of an article in one item and then use that cost to determine sales tax in another item. Items can be related to one another, for example through being based on the same graphic display or by having the same theme, but their response requirements must be independent.

b) Clear intent for test takers 

· It should be clear to the student what is being asked in each item. Writers should be careful not to make assumptions about how students will interpret an item’s implicit requirements.  Directions should be as straightforward as possible.

· Constructed-response items should contain clear directions to students about how they can respond. For example, can the response incorporate graphics or does it require the student to produce a verbal description?  Is more than one type of response appropriate?  Is more credit given for providing multiple solutions to a problem?

· If a constructed-response item requires students to show their work, the directions must clearly specify that students should include their work, provide direction for how students can show their work, and note that student work will be used in scoring the response.

Examples 20, 21, and 22 illustrate clear directions for students. 

Example 21 has directions to students in how they should respond and includes required characteristics of the response. 

Example 21 asks students to explain their answers, and it provides structure for them to do so. The item also includes direction to students about the format of their responses.

Example 22 specifies that students should give mathematical evidence for their answer.

2. Plain Language

The purpose of using plain language is to clearly convey meaning without altering what items are intended to measure. All items should use plain language. Even when the intent of the item is for students to define, recognize, or use mathematics vocabulary correctly, the surrounding text should be in plain language. Plain language guidelines often increase access and minimize confusion for students. 

· Write questions using brief, “simple” sentences or stems.

· Use the same structure for paragraphs throughout the assessment as much as possible (e.g., topic sentence, supporting sentences, and concluding sentence).

· Use present tense and active voice.

· Minimize paraphrasing.

· Avoid using pronouns.

· Use high-frequency words as much as possible.

· Avoid colloquialisms.

· When using words with multiple meanings, make sure the intended meaning is clear.

· Avoid using unnecessary descriptive information.

· Use format to clarify text (e.g., use bullets, allow space between pieces of text, and use boxes and lines judiciously).

Example 23 illustrates the use of present tense and active voice. No unnecessary descriptive information is included. 

Example 24 uses a simple, straightforward format.

3. Graphics

Graphics such as pictures, charts, and diagrams can be used to help students understand item text by depicting information in an item or a set of items. Graphics can be very effective in supporting text, illustrating mathematical concepts in the text, and increasing item access with minimal cost. If used improperly, however, graphics can add substantial confusion and distract test takers from what items are asking students to do. When using graphics,

· use visuals that mirror and parallel the wording and expectations of the text.

· illustrate only necessary information in the graphics.

· represent each important part of the item in the visual images.

Example 25 is an item that illustrates only necessary information and parallels the expectations of the text. 

Example 26 shows the effective use of graphics as response options. 

Example 27 uses pictures to illustrate important information in the text.

4. Using Contextual Information Appropriately

Often, items will be designed so that they measure mathematics in context. Contextual information includes problem scenarios, explanations, more thorough directions, and background text. Using contextual information judiciously can place mathematical concepts in fuller, often more realistic, conditions, measure students’ ability to apply mathematical concepts, and provide necessary background information. However, the contextual information should not interfere with the mathematics being assessed or become a barrier to a student’s ability to demonstrate his or her mathematical knowledge. The context should be the minimum necessary to set up the problem.  Guidelines for using contextual information appropriately include the following:

· Use contexts that are meaningful to the mathematics being assessed.

· Use contexts that are appropriate for the grade level assessed.

· Use realistic contexts.

· Use familiar contexts; avoid contexts that may confuse or be unfamiliar to some students taking the assessment.

Examples 27 (above) and 28 show the use of contexts that are appropriate for the grade level. The contexts are likely to be familiar to students in 4th and 12th grade, respectively. 

.

5. Writing Items with Multiple Access Points

Students vary in their abilities to access information and respond to tasks through visual, spatial, auditory, kinesthetic, and tactile pathways. When possible, items should be designed to allow students to approach and respond to the item in different ways. Incorporating multiple pathways appropriately in both constructed-response and multiple-choice items can increase the ability of the assessment to elicit responses from students across the range of achievement without affecting the mathematical content being assessed.

a)Multiple access in how items are written or presented to the students 

· Use visuals to accompany and explain the text when appropriate.

· Write both multiple-choice and constructed-response items that allow for various approaches to determining the solution.

· Use manipulatives.

· Use a brief activity when appropriate to address students with auditory, kinesthetic, or tactile strengths.

b)Multiple response opportunities 

· Write constructed-response items that allow for multiple response formats. For example, students might be allowed to show their answers through illustrations, diagrams, or formulas.

· When possible, write constructed-response items and scoring rubrics that measure the mathematical knowledge and skills of students with a range of achievement.

Example 27 (shown earlier) allows for a variety of response formats. Students can give verbal explanations, draw, or construct charts, for example. 

Example 29 provides access through the use of dialogue, personalizing the mathematics. The item uses dialogue in a grade-appropriate way. 

Example 30 provides access through the use of manipulatives, which allow students to use both tactile and kinesthetic pathways. 

### Inclusion of English Language Learners and Students with Disabilities

The mathematics assessments should be developed to allow for the participation of the widest possible range of students, so that interpretation of scores lead to valid inferences about levels of performance of the nation’s students as well as valid comparisons across states. All students should have the opportunity to demonstrate their knowledge of the concepts and ideas that the NAEP Mathematics Assessment is intended to measure. 

According to the National Research Council:

Fairness, like validity, cannot be properly addressed as an afterthought once the test has been developed, administered, and used. It must be confronted throughout the interconnected phases of the testing process, from test design and development to administration, scoring, interpretation, and use (1999, pp. 80-81). 

Current NAEP inclusion criteria for students with disabilities and English language learners are as follows (from NCES, 2007, Current Policy section):

Inclusion in NAEP of an SD [student with disabilities] or ELL student is encouraged if that student (a) participated in the regular state academic assessment in the subject being tested, and (b) if that student can participate in NAEP with the accommodations NAEP allows. Even if the student did not participate in the regular state assessment, or if he/she needs accommodations NAEP does not allow, school staff are asked whether that student could participate in NAEP with the allowable accommodations. (Examples of testing accommodations not allowed in NAEP are giving the reading assessment in a language other than English, or reading the reading passages aloud to the student. Also, extending testing over several days is not allowed for NAEP because NAEP administrators are in each school only one day.)

Most students with disabilities are eligible to be assessed in the NAEP program. Similarly, most students who are learning English as their second language are also eligible to participate in the NAEP assessment. There are two ways that NAEP addresses the issue of accessibility. One is to follow careful item and assessment development procedures to build accessibility into the standard assessment. The other is to provide accommodations for students with disabilities and for English language learners.

#### Accommodations

For many students with disabilities and students whose native language is other than English, the standard administration of the NAEP assessment will be most appropriate. For some students with disabilities and some English language learners, the use of one or more accommodations will be more suitable. How to select and provide appropriate accommodations is an active area of research, and new insights are emerging on how to best apply accommodation guidelines to meet the needs of individual students. The NAEP mathematics accommodations policy allows for a variety of accommodations, depending upon the needs of each student. Most accommodations that schools routinely provide in their own testing programs are allowed in the mathematics assessment, as long as they do not affect the construct tested. For example, it would NOT be appropriate to allow a student to use a calculator on a non-calculator block. 

The most frequently used accommodations included (from the *NAEP 2006 Assessment Administrator Manual*, p. 2.28):

· Bilingual booklet (mathematics operational and science only)

· Bilingual dictionary (mathematics operational and science only)

· Large-print booklet 

· Extended time in regular session 

· Read aloud (mathematics and science only) 

· Small group 

· One-on-one 

· Scribe or use of a computer to record answers 

· Other—includes format or equipment accommodations such as a sign language translator or amplification devices (if provided by the school)

· Breaks during test

· Magnification device

· School staff administers

Accommodations are offered in combination as needed; for example, students who receive one-on-one testing generally also use extended time. 

In a very small number of cases, students will not be able to participate in the assessment, even with the accommodations offered by NAEP:  

· Students with disabilities whose Individualized Education Plan (IEP) teams or equivalent groups have determined that they cannot participate, **or** whose cognitive functioning is so severely impaired that they cannot participate, **or** whose IEP requires an accommodation that NAEP does not allow.

· Limited English proficient students who have received mathematics instruction primarily in English for less than three school years **and** who cannot participate in the assessment when it is administered in English, with or without an accommodation, or when the bilingual English/Spanish form is used.

#### Item Writing Considerations to Maximize Accessibility 

Appropriate specialists (e.g., specialists in educating students with disabilities, school psychologists, linguists, specialists in second language acquisition) should be involved in the entire process of test development, along with psychometricians, teachers, and content specialists. While such specialists often participate in item review, their expertise can be useful in the item development process. For example, in the case of addressing accessibility for English language learners, many issues regarding vocabulary (meaning of individual words), semantics (meaning of words in a sentence), syntax (grammatical structure of sentences), and pragmatics (interpretation of words and sentences in context) cannot be properly addressed without applying the methods and reasoning from linguistics. 

This section addresses techniques for maximizing access to items for English language learners and students with disabilities. Chapter Five includes a section on maximizing accessibility at the assessment level. In addition, many of guidelines in the section, General Principles of Item Writing, address accessibility issues.

##### Item Writing Considerations for English Language Learners

Some students who are learning English as their second language will take the standard, English-only version of the test. These students are diverse both across and within their language groups. This is particularly the case with Spanish language speakers who come from various countries in Latin America and the Caribbean. Among the Spanish speaking population there are linguistic differences (mainly in vocabulary), cultural differences, and differences in educational and socio-economic backgrounds. English language learners may have trouble understanding what items are asking for on assessment forms administered in English. 

Although this section is specific to making items accessible to English language learners, the guidelines below can be applied to the development of mathematics items for all students. Many are extensions of the plain language principles found in the section, General Principles of Item Writing. 

*Vocabulary*

The vocabulary of both mathematical English and ordinary English must be considered when developing mathematics items (Shorrocks-Taylor and Hargreave, 1999). In general, familiar words and natural language should be used in problems (Prins and Ulijn, 1998); mathematics-specific vocabulary should be used when appropriate, ordinary vocabulary when feasible, that is, when vocabulary knowledge is not part of targeted measurement construct and when ordinary English will not convey the precise meaning that a mathematical term will. For English language learners, the problem context or conventions that allow “shorthand” communication may not be enough to provide the meaning of specific words (Wong Fillmore and Snow, 2000). Avoid when possible the use of words with different meanings in ordinary English and mathematical English (e.g., odd), and clarify meanings of multi-meaning words when it is necessary to use them (e.g., largest can refer to size or quantity) (Kopriva, 2000; Brown, 1999). 

· Avoid the use of idioms.

· Use concrete rather than abstract words. For example, instead of using “object” in a stem, refer to an actual object.

· Use shorter words, as long as they convey precise meaning.

· Use active verbs whenever possible. 

· Limit the use of pronouns.

· Avoid using ambiguous referents as shorthand (for example, It is a good idea to...).

· Use cognates when appropriate (cognates are words that are similar in form and meaning in two languages).

· Avoid the use of false cognates (words that are similar in form in two languages but have different meanings). For example, “once” means 11 in Spanish.

*Text Structure*

English and other languages often have different rules of syntax or word order. While students may know the basic differences between their primary language and English, subtle differences that can lead to confusion based on syntax or word order should be avoided.  There are several features of test items that can impact comprehension. 

· Use active voice (Abedi, 2006). For example, use “Laura drew a circle with a radius of 7 inches.” rather than “A circle was drawn with a radius of 7 inches.”

· Avoid long noun phrases (nouns with several modifiers) (Abedi, 2006).

· Use simple sentences as much as possible. However, do not compress sentences to the extent that important connecting information is omitted (Wong Fillmore and Snow, 2000; Shorrocks-Taylor and Hargreave, 1999).

· Avoid long stems that contain adverbials (e.g., Although ...., Gerald ....) and conditional (e.g., if/then) clauses (Abedi, 2006). 

· Avoid relative clauses and prepositional phrases (Abedi, 2006).For example, use “Marcie pays $2.00 for pens. Each pen costs $0.50. How many pens does Marcie buy?” rather than “If Marcie pays $2.00 for pens and each pen costs $0.50, how many pens does Marcie buy?”

· Write questions in the positive; avoid using negatives in the stem and item options.

· Avoid mixing mathematical symbols and text (Prins & Ulijn, 1998).

· Use formatting to keep items clear, separating key ideas (e.g., use bullets or frames).

##### Item Writing Considerations for Students with Disabilities

Most students with disabilities will take the standard assessment without accommodations, and those who take the assessment with accommodations will use the standard version of the test also. Item writers and the assessment developer should minimize item characteristics that could hinder accurately measuring the mathematics achievement of students with disabilities. Using the item writing considerations for English language learners will minimize some linguistic characteristics that can affect the responses of some students with disabilities. In addition, item writers should attend to the following recommendations.

· Avoid layout and design features that could interfere with the ability of the student to understand the requirements and expectations of the item.

· Use illustrations carefully. Thompson, Johnstone, and Thurlow (2002) provide guidance for the use of illustrations:

· Minimize the use of purely decorative illustrations, since they can be distracting.

· Use illustrations that can be enlarged or viewed with a magnifier or other assistive technology.

· Use simple black and white line drawings when possible.

· Avoid item contexts that assume background experiences that may not be common to some students with sensory or physical disabilities.

· Develop items so that they can be used with allowed accommodations.

· Address alternatives for students who are not able to use the manipulatives necessary for responding to an item.

#### Scoring Responses from English Language Learners

Students’ literacy status and varied background experiences have an impact on how well scorers can properly read, understand, and evaluate the responses of English language learners to constructed-response items. 

##### Literacy Issues

Responses sometimes can be difficult to read because language confusion arises between the students’ native language and English. While this is developmentally appropriate in terms of language acquisition, many scorers are not trained to interpret these types of systematic errors. The following procedures should be used to score responses from English language learners appropriately:

· Scoring leaders should have additional training in recognizing and properly interpreting responses from English language learners.

· Experts in reading responses of English language learners should be available to scorers throughout the scoring process.

· Scorer training materials and benchmark student responses should illustrate features typically observed in the writing of ELLs in English. These features include:

1. Code-switching—intermittent use of the student’s native language and English.

2. Use of native language or beginning-stage English phonetic spelling in attempting to respond in English or (e.g., *de ticher sed*—“the teacher said”).

3. Use of writing conventions from the native language when students are responding in English (e.g., *today is monday*—the names of the weekdays are not capitalized in standard Spanish).

4. Word mergers (the condensing of words into one mega-word), transposition of words, and omission of tense markers, articles, plurals, prepositions, or other words.

5. Use of technical notation conventions from the student’s culture (e.g., $25,00 to express twenty-five dollars).

6. Substitution of common words for more precise terminology. For instance, a student may use the word “fattest” to mean “greatest.”  If the item is not intended to evaluate students’ knowledge and use of correct terminology, then this substitution may be acceptable. On the other hand, if the intent **is** to measure the students’ knowledge and use of such terminology in an applied setting, then the substitution would be incorrect.

7. Inappropriate use of unfamiliar words.

8. Unusual sentence and paragraph structures that reflect discourse structures in the native language.

9. Other features such as the transposition of words (e.g., *the cat black*) and omission of tense markers (e.g., *yesterday**he learn at lot*), articles (*I didn’t see it in notebook)*, plurals (*the horse are gone*), prepositions (e.g., *explain me what you said*), or other words.

10. Over-reliance on non-verbal forms of communication, such as charts or pictures.

##### Varied Background Experiences

Novel interpretations and responses are common for English language learners and often reflect background experiences quite different from that of most native English speakers. Scorers should evaluate responses based on the measurement intent of the item and recognize when an unusual response is actually addressing that intent.

At times, scoring rubrics implicitly or explicitly favor writing styles that mirror what is taught in language arts curricula in the U.S. schools. However, circular, indirect, deductive, and abbreviated reasoning writing styles are encouraged by some cultures, and scorers should be trained to appropriately score these types of responses. In addition, some cultures discourage children from giving long responses to questions, especially when authority figures ask the questions. Such a pattern of communication can be reflected in the written responses of ELLs to constructed-response questions. Despite being short, these responses to constructed-response items may be correct. 

When a specific writing style is not the measurement intent of the item, scorers need to understand the nature, conventions, and approaches of these kinds of styles and how to separate the structure and sophistication of the written response from the substantive content being evaluated.

### Item Formats

There are three types of items on the NAEP mathematics assessment: multiple-choice, short constructed-response, and extended constructed-response. 

· **Multiple-choice** items require students to select one correct or best answer to a given problem. These items are scored as either correct or incorrect.

· **Short constructed-response** items require students to give a short answer such as a numerical result or the correct name or classification for a group of mathematical objects, draw an example of a given concept, or perhaps write a brief explanation for a given result. Short constructed-response items are scored according to scoring rubrics with two or three categories describing increasing degrees of knowledge and skill.

· **Extended constructed-response**items require students to consider a situation that demands more than a numerical response or a short verbal or graphic communication. If it is a problem to solve, for example, the student may be asked to carefully consider a situation choose a strategy to “solve” the situation, carry out the strategy, and interpret the solution derived in terms of the original situation. Extended constructed-response items are typically scored according to scoring rubrics with five categories. 

Item writers should carefully consider the content and skills they intend to assess when deciding whether to write a multiple-choice or constructed-response item. Each content area includes knowledge and skills that can be measured using each of the three item formats and each level of mathematical complexity can be measured by any of the item formats. Carefully constructed multiple-choice items, for example, can measure any of the levels of mathematical complexity. Although a level of mathematical complexity may lend itself more readily to one item format, each type of item—multiple-choice, short constructed-response, and extended constructed-response—can deal with mathematics of greater or less depth and sophistication.

#### Developing Multiple-Choice Items

Multiple-choice items are an efficient way to assess knowledge and skills, and they can be developed to measure any of the levels of mathematical complexity. In a well-designed multiple-choice item, the stem clearly presents the problem to the student. The stem may be in the form of a question, a phrase, or a mathematical expression, as long as it conveys what is expected of the student. The stem is followed by either four or five answer choices, or options, only one of which is correct. In developing multiple-choice items, item writers should ensure that

· the stem includes only the information needed to make the student’s task clear or needed to set the problem in an appropriate context.

· options are as short as possible.

· options are parallel in structure, syntax, and complexity.

· options do not contain inadvertent cues to the correct answer such as repeating a word from the stem in the correct answer or using specific determiners (e.g., all, never) in the distractors (incorrect options).

· distractors are plausible, but not so plausible as to be possible correct answers.

· distractors are designed to reflect the measurement intent of the item, not to trick students into choices that are not central to the mathematical idea being assessed.

Example 31 illustrates a straightforward stem with a direct question. The distractors are plausible, but only one is clearly correct.

#### Developing Constructed-Response Items and Scoring Rubrics

The type of constructed-response item, short or extended, that is written should depend on the mathematical construct that is being assessed—the objectives and the level of complexity. Item writers should draft the scoring rubric as they are developing the item so that both the item and rubric reflect the construct being measured.

In developing the scoring rubric for an item, writers should think about what kind of student responses would show increasing degrees of knowledge and understanding. Writers should sketch condensed sample responses for each score category. Item writers also should include a mathematical justification or explanation for each rubric category description. Doing so will assist the writer in drafting a clear scoring rubric as well as provide guidance for scoring the item. 

Short Constructed-Response Items

Some short constructed-response items are written to be scored dichotomously (that is, as correct or incorrect). Short constructed-response items with two scoring categories should measure knowledge and skills in a way that multiple-choice items cannot or provide greater evidence of students’ understanding. Such short constructed-response items might be appropriate for measuring some computation skills, for example, to avoid guessing or estimation, which could be a factor if a multiple-choice item were used. They are also useful when there is more than one possible correct answer, when there are different ways to display an answer, or when a brief explanation is required. Item writers should take care that short constructed-response items would not be better or more efficiently structured as multiple-choice items—they should not be simply multiple-choice items without the options. Constructed-response items should be developed so that the knowledge and skills they measure are “worth” the additional time and effort to respond on the part of the student and the time and effort it takes to score the response.

Some short constructed-response items are written to be scored on a three-category scale. Short constructed-response items with three scoring categories should measure knowledge and skills that require students to go beyond giving an acceptable answer. These items allow for degrees of accuracy in a response so that a student can receive some credit for demonstrating partial understanding of the concept or skill measured by the item.

Item writers must draft a scoring rubric for each short constructed-response item. For dichotomous items, the rubrics should define the following two categories:

Correct

Incorrect

Examples 32 and 33 show dichotomous constructed-response items and their rubrics. 

Example 32 requires students to perform a calculation, and the rubric simply defines a correct result.

Example 33 asks students to explain a concept, and the rubric lists some examples and provides guidance for determining whether other student responses are acceptable.

For items with three score categories, the rubrics should define the following categories:

Correct

Partial

Incorrect

Examples 34 and 35 show constructed-response items with three score categories. 

Example 34 requires students to demonstrate understanding of a concept, and the scoring rubric lists acceptable responses.

Example 35 requires a completely correct answer for the top score category and gives a “partial” score for an answer that demonstrates an understanding of the appropriate ratio. 

##### Extended Constructed-Response Items

In general, extended constructed-response items ask students to solve a problem by applying and integrating mathematical concepts or require that students analyze a mathematical situation and explain a concept, or both. Extended constructed-response items typically have five scoring categories:

Extended

Satisfactory

Partial

Minimal

Incorrect

Examples 36, 37, and 38 are extended constructed-response items. 

Example 36 asks students to solve a problem and analyze a situation. 

Example 37 requires students to explain and justify a solution. 

Example 38 requires students to demonstrate understanding of a concept by explaining its use.

Item writers must develop a draft scoring rubric specific to each extended constructed-response item. The rubric should clearly reflect the measurement intent of the item. The next section describes some requirements for writing scoring rubrics.

##### Aligning Items and Rubrics

Item writers should refer back to the measurement intent of the item when they are developing its scoring rubric. The number of categories used in the rubric should be based upon the mathematical demand of the item. 

1. Defining the Score Categories

Each score category must be distinct from the others; descriptions of the score categories should clearly reflect increasing understanding and skill in the targeted mathematics constructs. Distinctions among the categories should suggest the differences in student responses that would fall into each category; the definitions must be clear enough to use in training scorers. Each score level should be supported by the mathematical intent of the item. Factors unrelated to the measurement intent of the item should not be evaluated in the rubric. For example, if an item is not meant to measure writing skills, then the scoring rubric should be clear that the demonstration of mathematics in the response does not need to be tied to how well the response is written. However, if an explanation is part of the item requirement, the rubric should reflect that such explanations should be clear and understandable. 

The three extended response items above (Examples 36, 37, and 38) all have distinct, well-defined score categories. Example 36 is a good illustration of a rubric that clearly differentiates among score categories. The categories describe increasing mathematical understanding as the scores increase, and the descriptions are in terms of student responses. 

2. Measuring More than One Concept

If an item is measuring more than one skill or concept, the description of the score categories in the rubric should clearly reflect increasing understanding and achievement in each area. For instance, if the item is measuring both students’ understanding of a topic in algebra and their skill in developing an appropriate problem solving approach, then the description of each category in the rubric should explain how students’ understanding and skill are evaluated. If an item requires both an acceptable solution and an explanation, the rubric should show how these two requirements are addressed in each score category.

Example 38 requires two correct answers accompanied by justifications, and the scoring rubric clearly addresses both. 

3. Specifying Response Formats

Unless the item is measuring whether or not a student can use a specified approach to a problem, each score category should allow for various approaches to the item. It should be clear in the rubric that different approaches to the item are allowed. Varied approaches may include

· different mathematical procedures; for example, adding or multiplying may lead to the same solution. 

· different representations; for example, using a diagram or algorithms to solve or explain a solution may be appropriate.

Scorer training materials should include examples of student work that illustrate various appropriate approaches to the item.

### Item Tryouts and Reviews

Appendix B contains the *NAEP Item Development and Review Policy Statement* (National Assessment Governing Board, 2002), which explicates the following six guiding principles:

**Principle 1**

NAEP test questions selected for a given content area shall be representative of the content domain to which inferences will be made and shall match the NAEP assessment framework and specifications for a particular assessment.

**Principle 2**

The achievement level descriptions for basic, proficient, and advanced performance shall be an important consideration in all phases of NAEP development and review.

**Principle 3**

The Governing Board shall have final authority over all NAEP test questions. This authority includes, but is not limited to, the development of items, establishing the criteria for reviewing items, and the process for review. 

**Principle 4**

The Governing Board shall review all NAEP test questions that are to be administered in conjunction with a pilot test, field test, operational assessment, or special study administered as part of NAEP.

**Principle 5**

NAEP test questions will be accurate in their presentation and free from error. Scoring criteria will be accurate, clear, and explicit.

**Principle 6**

All NAEP test questions will be free from racial, cultural, gender, or regional bias, and must be secular, neutral, and non-ideological. NAEP will not evaluate or assess personal or family beliefs, feelings, and attitudes, or publicly disclose personally identifiable information. 

The test development contractor should build careful review and quality control procedures into the assessment development process. Although large-scale field testing provides critical statistical item-level information for test development, other useful information about the items should be collected before and after field testing. Before field testing, items and scoring rubrics should be reviewed by experts in mathematics and educational measurement, including mathematics teachers and representatives of state education agencies, and by reviewers trained in sensitivity review procedures. After field testing, the items and the assessment as a whole should be reviewed to make sure that they are as free as possible from irrelevant variables that could interfere with students’ demonstrating their mathematical knowledge and skills. 

Sensitivity reviews are a particularly important part of the assessment development process. Reviewers should include educators and community members who are experts in the schooling or cultural backgrounds of students in the primary demographic groups taking the assessment, including special needs students and English language learners. The reviewers focus on checking that test items are fair for all students, identifying any items that contain offensive or stereotypical subject matter and other irrelevant factors. They provide valuable guidance about the context, wording, and structure of items, and they identify flaws in the items that confound the validity of the inferences for the groups of students they represent. 

Classroom tryouts and cognitive labs are two particularly useful procedures for collecting information about how specific items or item prototypes are working. The information collected is valuable for determining whether items are measuring the construct as intended and for refining the items and scoring procedures before field testing. These two techniques can be targeted to specific samples of students (for example, to see if items are appropriately written for English language learners) or to students representing the characteristics of the assessment population. The information that the test development contractor collects from classroom tryouts and cognitive labs should be provided to item writers to help them develop new items and revise existing items before field testing, and it can be used to enhance item writing training and reference materials.

#### Classroom Tryouts

Classroom tryouts are an efficient and cost-effective way to collect information from students and teachers about how items and directions are working. Tryouts allow the test developer to troubleshoot the items and scoring rubrics. Classroom tryouts usually involve a non-random but carefully selected sample; the students should reflect the range of student achievement in the target population and represent the diversity of examinees. For example, the tryout sample should include urban and rural schools; schools in low, middle, and high economic communities; schools from different regions; and schools with students in all the major racial/ethnic categories in the population. The more the sample represents various groups in the testing population, the more likely the tryout will identify areas that can be improved in the items. 

In addition to providing student response data, tryouts can provide various kinds of information about the items, including what students and their teachers think the items are measuring, the appropriateness of the associated test materials, and the clarity of the instructions. Students can be asked to evaluate the items, for example by circling words, phrases, or sentences they find confusing and making suggestions for improvement. Teachers can ask students what they thought each item was asking them to do and why they answered as they did, and provide the information to the test developer. Teachers can be asked to edit items and associated test materials. Item tryouts also are an efficient way to test how accommodations work and to try out new manipulatives and other materials.

Student responses to the items should be reviewed by content and measurement experts to detect any problems in the items and should be used along with the other information gathered to refine the items and scoring rubrics. Using a sample that includes various subgroups in the population will allow reviewers to look for issues that might be specific to these groups. Responses also are useful in developing training materials and notes for item scorers.

#### Cognitive Labs

In cognitive labs, students are interviewed individually while they are taking or shortly after they have completed a set of items. Because cognitive labs highlight measurement considerations in a more in-depth fashion than other administrations can, their use can provide important information for item development and revision. For example, cognitive labs can identify if and why an item is not providing meaningful information about student achievement, provide information about how new formats are working, or identify why an item was flagged by differential item functioning (DIF) analyses. 

The student samples used in cognitive labs are much smaller than those used in classroom tryouts. Students are selected purposefully to allow an in-depth understanding of how an item is working and to provide information that will help in revising items or in developing a particular type of item.

Chapter Five

## Design of the Assessment 

The NAEP mathematics assessment is complex in its structure. The design of the assessment demands that multiple features stay in balance. A broad range of content should be adequately covered by the test items, balanced at each grade level according to the required distribution for each content area. At the same time, items make differing cognitive demands on students according to how mathematically complex they are. This, too, requires balance. The assessments also need to be balanced according to the three item formats: multiple-choice, short constructed response, and extended constructed response. An additional balance issue involves the mathematical setting of the item, whether it is purely mathematical or set in a real-world context. 

There are other features of both the test and the items that are important in the design of a valid and reliable assessment. These include how sampling is used in the design of NAEP, the use of calculators, and the use of manipulatives and other tools. Of critical importance is the issue of accessibility for all students, which is addressed in several different ways on the NAEP mathematics assessments. A final design feature of NAEP is the use of families of items. Each of these features and issues is described in this chapter.

### Alignment of the Assessment and the Framework

The assessment should be developed so that it is aligned with the content expectations and complexity levels defined by the NAEP *Mathematics Framework for 2009.*Drawing upon Webb and others, five interrelated dimensions are considered in structuring the NAEP assessment so that it is aligned with the framework:

1. The match between the content of the assessment and the content of the framework: The assessment as a whole should reflect the breadth of knowledge and skills covered by the topics and objectives in the framework.

2. The match between the complexity of mathematical knowledge and skills on the assessment and in the framework: The assessment should represent the balance of levels of mathematical complexity at each grade as described in the framework.

3. The match between the emphasis of the assessment and the emphasis of topics, objectives, and contextual requirements in the framework: The assessment should represent the balance of content and item formats specified in the framework and give appropriate emphasis to the conditions in which students are expected to demonstrate their mathematics achievement, reflecting the use of calculators, manipulatives, and real-world settings.

4. The match between the assessment and how scores are reported and interpreted: The assessment should be developed so that scores will reflect both the framework and the performance described in the NAEP achievement levels.

5. The match between the assessment design and the characteristics of the targeted assessment population: The assessment should give all students tested a reasonable opportunity to demonstrate their knowledge and skills in the topics and objectives covered by the framework.

These five dimensions are the foundation for the NAEP assessment and item specifications. The principles in these dimensions are used in each of the sections that follow.

### Accessibility

The NAEP mathematics assessment is designed to measure the achievement of students across the nation. Therefore, it should allow students who have learned mathematics in a variety of ways, following different curricula and using different instructional materials; students who have mastered the content to varying degrees; students with disabilities; and students who are English-language learners to demonstrate their content knowledge and skill. The design issue for the assessment is: what is a reasonable way to measure the same mathematics for students who come to the assessment with different experiences, strengths, and challenges, who approach the mathematics from different perspectives, and who have different ways of displaying their knowledge and skill?

Two methods NAEP uses to design an accessible assessment program are (1) developing the standard assessment so that it is accessible and (2) providing accommodations for students with special needs. In Chapter Four each of these design issues is described in greater detail.

### Reporting Requirements

The NAEP mathematics assessment reports results for the nation’s students in fourth, eighth, and twelfth grades and participating states’ students in grades 4 and 8, as well as for subgroups of the population defined by specific demographic characteristics such as gender, type of school attended, and eligibility for free and reduced-priced lunch (see the NAEP Report Card for more information about subgroup scores). Scores are not reported for individual students or schools.

Results are reported in two ways: scale scores and achievement levels. Scale scores can range from 0 to 500. Reports identify the percentage of students who reach three achievement levels—basic, proficient, and advanced. Briefly,

· **Basic**denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. 

· **Proficient**represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter. 

· **Advanced** represents superior performance. 

These levels are intended to provide descriptions of what students *should* know and be able to do in mathematics. Established for the 1992 mathematics scale through a broadly inclusive process and adopted by the National Assessment Governing Board, the three levels per grade are the primary means of reporting NAEP data. The updated mathematics framework was developed with these levels in mind to ensure congruence between the levels and the test content. See Appendix A for the NAEP Mathematics Achievement Level Descriptions.

The assessment should be designed so that results can be reliably and validly reported for the population and for subgroups by scale scores and in terms of the achievement levels. Because the results are intended to describe the achievement of all students in the nation, results should provide an accurate picture of achievement across the entire scoring scale.

The test developer should design the assessment so that the content is aligned with the content framework and the knowledge and skills described in the achievement levels. In addition, before new items are developed, the test developer should review the items available for the assessment and the test information functions for the assessment at each grade level to identify the need for new items that measure performance across the achievement scale, including at the upper and lower ends.

### Balance of Content

As described in Chapter Two, each NAEP mathematics item is developed to measure one of the objectives, which are organized into the five major content areas of mathematics. The table below shows the distribution of items by grade and content area. See Chapter Two for more details.

**Table 2. Percentage Distribution of Items by Grade and Content Area**

### Balance of Mathematical Complexity

As described in Chapter Three, items are classified according to the level of demands they make on students. This is known as the mathematical complexity of the item. Each item is considered to be at one of three levels of complexity: Low, Moderate, or High.

The ideal balance sought for the 2009 NAEP is not necessarily the balance one would wish for curriculum or instruction in mathematics education. Balance here must be considered in the context of the constraints of an assessment such as NAEP. These constraints include the timed nature of the test and its paper-and-pencil format. Items of high complexity, for example, often take more time to complete. At the same time, some items of all three types are essential to assess the full range of students’ mathematical achievement. 

The ideal balance would be that half of the total testing time on the assessment is spent on items of moderate complexity, with the remainder of the total time spent equally on items of low and high complexity. This balance would apply for all three grade levels.

###### Percent of Testing Time at Each Level of Complexity

### Balance of Item Formats

Items consist of three formats: multiple-choice, short constructed response, and extended constructed response (see Chapter Three for an in-depth discussion of each type). Testing time on NAEP is divided evenly between multiple-choice items and both types of constructed response items, as shown below:

###### Percent of Testing Time by Item Formats

The design of the assessment, then, must take into account the amount of time students are expected to spend on each of the three formats of items. 

### Balance of Item Contexts

Just as mathematics can be separated into “pure” and “applied” mathematics, NAEP items should seek a balance of items that measure students’ knowledge within both realms. Therefore some items will deal with purely mathematical ideas and concepts, while others will be set in the contexts of real-world problems. 

In the two pairs of examples below, the first item is purely mathematical, while the second is set in the context of a real-world problem.

Both items involve computation. In the first item the operation is specified. In the other item, the students must interpret the contextual situation, recognizing that it calls for the finding the difference between 74 and 30, and then compute. 

### NAEP Administration and Student Sampling

As currently planned, the 2009 mathematics assessment will be administered between January and early April, with the administration conducted by trained field staff. Each mathematics assessment booklet will contain two separately timed, 25-minute sections of mathematics items. Three types of items will be used: multiple-choice, short constructed-response, and extended constructed-response. The assessment is designed so that there are multiple forms of the test booklets. The items will be distributed across the booklets using a matrix sampling design so that students taking part in the assessment do not all receive the same items. In addition to the mathematics items, the assessment booklets will include background questionnaires, administered in separately timed sessions.

The assessment is designed to measure mathematics achievement of students in the nation’s schools in grades 4, 8, and 12 and report the results at the national, regional, and state levels. To implement this goal, schools throughout the country are randomly selected to participate in the assessment. The sampling process is carefully planned to select schools that accurately represent the broad population of U.S. students and the populations of students in each state participating in State NAEP.

The selection process is designed to include schools of various types and sizes from a variety of community and geographical regions, with student populations that represent different levels of economic status, racial, ethnic and cultural backgrounds, and instructional experiences. Students with disabilities and English language learners are included to the extent possible, with accommodations as necessary (see Chapter Four for more information about inclusion criteria and accommodations). The sophisticated sampling strategy helps to ensure that the NAEP program can generalize the assessment findings to the diverse student populations in the nation and participating jurisdictions. This allows the program to present information on the strengths and weaknesses in aggregate student understanding of mathematics and the ability to apply that understanding in problem-solving situations; provide comparative student data according to race/ethnicity, type of community, and geographic region; describe trends in student performance over time; and report relationships between student achievement and certain background variables.

### Calculators

The assessment contains blocks (sets of items) for which calculators are not allowed, and calculator blocks, which contain some items that would be difficult to solve without a calculator. At each grade level, approximately two-thirds of the blocks measure students’ mathematical knowledge and skills without access to a calculator; the other third of the blocks allow the use of a calculator. The type of calculator students may use on a calculator block varies by grade level, as follows:

· At grade 4, a four-function calculator is supplied to students, with training at the time of administration.

· At grade 8, a scientific calculator is supplied to students, with training at the time of administration.

· At grade 12, students are allowed to bring whatever calculator, graphing or otherwise, they are accustomed to using in the classroom with some restrictions for test security purposes (see below). A scientific calculator is supplied to students who do not bring a calculator to use on the assessment.

No items on the 2009 NAEP at either grade 8 or grade 12 will be designed to provide an advantage to students with a graphing calculator. Estimated time required for any item should be based on the assumption that students are not using a graphing calculator. 

The assessment developer will propose restrictions on calculator use in grades 8 and 12 to (1) help ensure that items in calculator blocks cannot be solved in ways that are inconsistent with the knowledge and skills the items are intended to measure and (2) to maintain the security of NAEP test materials. These restrictions will address issues such as calculators with QWERTY keyboards, communication between students during testing, and the use of stored formulas, algorithms, and other procedures.

Items are categorized according to the degree to which a calculator is useful in responding to the item:

· A calculator inactive item is one whose solution neither requires nor suggests the use of a calculator.

· A calculator is not necessary for solving a calculator neutral item; however, given the option, some students might choose to use one.

· A calculator is necessary or very helpful in solving a calculator active item; a student would find it very difficult to solve the problem without the aid of a calculator.

### Manipulatives and Tools

The assessment uses reasonable manipulative materials, where possible, in measuring students’ ability to represent their understandings and to use tools to solve problems. Such manipulative materials and accompanying tasks are carefully chosen to cause minimal disruption of the test administration process. Examples of such materials are number tiles, geometric shapes, rulers, and protractors.

In the following example, number tiles are provided to the student.

In the next example, students are provided with a protractor.

### Item Families

Item families are groups of related items designed to measure the depth of student knowledge within a particular content area (vertical item families) or the breadth of student understanding of specific concepts, principles, or procedures across content areas (horizontal item families). Within a family, items may cross content areas, vary in mathematical complexity, and cross grade levels.

Using item families in different ways provides for a more in-depth analysis of student performance than would a collection of discrete, unrelated items. The results might show the degree to which students can solve problems in a given content area at increasing levels of complexity. Or they might give evidence about how a mathematical concept is more accessible to students through one type of model than another. These two types of families are illustrated below.

The first example illustrates a family designed to measure depth of grade 8 students’ knowledge of the relationship of area of a square and the length of the sides of the square.

The first item is straightforward, asking the students to determine the length of a side given the area of the square. It would be classified as low complexity. The second item, classified as moderate complexity, requires the students to first determine the area and then find the length of a side. The third item brings together measurement and algebra, but relies on student knowledge about area of a square and length of its sides. Students must attend to several conditions and write an equation to describe a general situation. It would be classified as high complexity.

The second example illustrates a family designed for grade 4 to measure breath of understanding of the meaning of fractions. Two models, area and sets, are used. For each model, two versions are given. In the first version, the model has the same number of parts or pencils as the denominator. In the second version the number of parts or pencils is a multiple of the denominator. 

Other item families might examine the same mathematical idea in different content areas. For example, a family of items might be designed to see how students use proportional thinking in different mathematical contexts such as geometry, algebra, and measurement. 

# References

Abedi, J. (2006). Language Issues in Item-Development. In Downing, S. M. and Haladyna, T. M. (Ed.) *Handbook of Test Development*. New Jersey: Lawrence Erlbaum Associates, Publishers.

Abedi, J. (2004). *Inclusion of students with limited English proficiency in NAEP: Classification and measurement issues*(CSE Report 629). Los Angeles: University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation, Graduate School of Education and Information Studies. 

Kopriva, R.J., & Carr, T.G. (2006). STELLA: A computerized system for individually assigning test accommodations to ELLs. Presentation at the 2006 National Conference on Large-Scale Assessment, San Francisco, CA.

Kopriva, R.J. (in press). *Improving testing for English language learners: A comprehensive approach to designing, building, implementing, and interpreting better academic assessments.*NJ. Lawrence Erlbaum Publishers.

Kopriva, R., & Saez, S. (1997). *Guide to scoring LEP student responses to open-ended mathematics items*. Washington, DC: Council of Chief State School Officers.

Koretz, D., & Barton, K. (2003-2004). Assessing students with disabilities: Issues and evidence. *Educational Assessment, 9*(1 & 2), 29-60.

National Center for Education Statistics. (2007). NAEP inclusion policy. Retrieved June 6, 2007, from http://nces.ed.gov/nationsreportcard/about/inclusion.asp

National Center for Education Statistics. (2006). NAEP assessment administrator manual. Retrieved June 6, 2007, from http://nces.ed.gov/nationsreportcard/about/inclusion.asp

National Research Council (2004). *Keeping score for all: The effects of inclusion and accommodation policies on large-scale educational assessments*. Washington, DC: National Academy Press.

National Research Council. (1999). *High stakes: Testing for tracking, promotion, and graduation.* Committee on Appropriate Test Use. J. P. Heubert & R. M. Hauser (Eds.). Board on Testing and Assessment, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

Perie, M., Grigg, W., & Dion, G. (2005). *The nation’s report card: Mathematics 2005* (NCES 2006-453). U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Washington, DC: Government Printing Office.

Solano-Flores, G., Lara., J., Sexton, U., & Navarrete, C. (2001). *Testing English language learners: A sampler of student responses to science and mathematics test items*. Washington, DC: Council of Chief State School Officers.

South Carolina Department of Education (2006). Adding value to accommodations decision-making for English language learners and students with Disabilities (AVAD). Proposal for the US Department of Education Enhanced Assessment Grant Program.

Thompson, S.J., Johnstone, C.J., Thurlow, M. L. (2002). Universal design applied to large-scale assessments. Minneapolis, MN: National Center on Educational Outcomes.

Webb, N.L., Ely, T., Cormier, M., & Vesperman B., (2006). The Web Alignment Tool: Development, refinement, and dissemination. In Council of Chief State School Officers, *Aligning assessment to guide the learning of all students* (pp. 1-30).*.* Washington, DC: Council of Chief State School Officers.

Appendix A

## NAEP Mathematics Achievement Level Descriptions

The achievement levels are cumulative; therefore, students performing at the *Proficient* level also display the competencies associated with the *Basic* level, and students at the *Advanced* level also demonstrate the skills and knowledge associated with both the *Basic* and the *Proficient* levels. The cut score indicating the lower end of the score range for each level is noted in parentheses.

### NAEP Mathematics Achievement Levels—Grade 4

### NAEP Mathematics Achievement Levels—Grade 8

### NAEP Mathematics Achievement Levels—Grade 12

Appendix B

## NAEP Item Development and Review Policy Statement 

**Adopted: May 18, 2002**

It is the policy of the National Assessment Governing Board to require the highest standards of fairness, accuracy, and technical quality in the design, construction, and final approval of all test questions and assessments developed and administered under the National Assessment of Educational Progress (NAEP). All NAEP test questions or items must be designed and constructed to reflect carefully the assessment objectives approved by the National Assessment Governing Board. The final assessments shall adhere to the requirements outlined in the following Guiding Principles, Policies and Procedures for NAEP Item Development and Review.

The Governing Board’s Assessment Development Committee, with assistance from other Board members as needed, shall be responsible for reviewing and approving NAEP test questions at several stages during the development cycle. In so doing, the Guiding Principles, Policies and Procedures must be adhered to rigorously.

### Introduction

The No Child Left Behind Act of 2001 (P.L. 107-110) contains a number of important provisions regarding item development and review for the National Assessment of Educational Progress (NAEP). The legislation requires that:  

· “the purpose [of NAEP] is to provide...a fair and accurate measurement of student academic achievement”

· “[NAEP shall]...use widely accepted professional testing standards, objectively measure academic achievement, knowledge, and skills, and ensure that any academic assessment authorized....be tests that do not evaluate or assess personal or family beliefs and attitudes or publicly disclose personally identifiable information;”

· “[NAEP shall]...only collect information that is directly related to the appraisal of academic achievement, and to the fair and accurate presentation of such information;”

· “the Board shall develop assessment objectives consistent with the requirements of this section and test specifications that produce an assessment that is valid and reliable, and are based on relevant widely accepted professional standards;”

· “the Board shall have final authority on the appropriateness of all assessment items;”

· “the Board shall take steps to ensure that all items selected for use in the National Assessment are free from racial, cultural, gender, or regional bias and are secular, neutral, and non-ideological;” and

· “the Board shall develop a process for review of the assessment which includes the active participation of teachers, curriculum specialists, local school administrators, parents, and concerned members of the public.”

Given the importance of these mandates, it is incumbent upon the Board to ensure that the highest standards of test fairness and technical quality are employed in the design, construction, and final approval of all test questions for the National Assessment. The validity of educational inferences made using NAEP data could be seriously impaired without high standards and rigorous procedures for test item development, review, and selection.

Test questions used in the National Assessment must yield assessment data that are both valid and reliable in order to be appropriate. Consequently, technical acceptability is a necessary, but not a sufficient condition, for judging the appropriateness of items. In addition, the process for item development must be thorough and accurate, with sufficient reviews and checkpoints to ensure that accuracy. The Guiding Principles, Policies, and Procedures governing item development, if fully implemented throughout the development cycle, will result in items that are fair and of the highest technical quality, and which will yield valid and reliable assessment data.

Each of the following Guiding Principles is accompanied by Policies and Procedures. Full implementation of this policy will require supporting documentation from the National Center for Education Statistics (NCES) regarding all aspects of the Policies and Procedures for which they are responsible.

This policy complies with the documents listed below which express acceptable technical and professional standards for item development and use. These standards reflect the current agreement of recognized experts in the field, as well as the policy positions of major professional and technical associations concerned with educational testing.

Standards for educational and psychological testing. (1999). Washington, DC: American Educational Research Association (AERA), American Psychological Association (APA), and National Council on Measurement in Education (NCME).                              

Code of fair testing practices in education. (1988). Washington, DC: Joint Committee on Testing Practices. 

National Center for Education Statistics (NCES) Statistical Standards, DRAFT, February 2002.

Guiding Principles – Item Development and Review Policy

Principle 1

NAEP test questions selected for a given content area shall be representative of the content domain to which inferences will be made and shall match the NAEP assessment framework and specifications for a particular assessment.

Principle 2

The achievement level descriptions for basic, proficient, and advanced performance shall be an important consideration in all phases of NAEP development and review.

Principle 3

The Governing Board shall have final authority over all NAEP test questions. This authority includes, but is not limited to, the development of items, establishing the criteria for reviewing items, and the process for review. 

Principle 4

The Governing Board shall review all NAEP test questions that are to be administered in conjunction with a pilot test, field test, operational assessment, or special study administered as part of NAEP.

Principle 5

NAEP test questions will be accurate in their presentation and free from error. Scoring criteria will be accurate, clear, and explicit.

Principle 6

All NAEP test questions will be free from racial, cultural, gender, or regional bias, and must be secular, neutral, and non-ideological. NAEP will not evaluate or assess personal or family beliefs, feelings, and attitudes, or publicly disclose personally identifiable information. 

#### Policies and Procedures for Guiding Principles

##### Principle 1

NAEP test questions selected for a given content area shall be representative of the content domain to which inferences will be made and shall match the NAEP assessment framework and specifications for a particular assessment.

##### Policies and Procedures

1. Under the direction of the Board, the framework for each assessment will be developed in a manner that defines the content to be assessed, consistent with NAEP’s purpose and the context of a large-scale assessment. The framework development process shall result in a rationale for each NAEP assessment, which delineates the scope of the assessment relative to the content domain. The framework will consist of a statement of purpose, assessment objectives, format requirements, and other guidelines for developing the assessment and items.

2. In addition to the framework, the Board shall develop assessment and item specifications to define the: a) content and process dimensions for the assessment; b) distribution of items across content and process dimensions at each grade level; c) stimulus and response attributes (or what the test question provides to students and the format for answering the item); d) types of scoring procedures; e) test administration conditions; and f) other specifications pertaining to the particular subject area assessment. 

3. The Board will forward the framework and specifications to NCES, in accordance with an appropriate timeline, so that NCES may carry out its responsibilities for assessment development and administration.

4. In order to ensure that valid inferences can be made from the assessment, it is critical that the pool of test questions measures the construct as defined in the framework. Demonstrating that the items selected for the assessment are representative of the subject matter to which inferences will be made is a major type of validity evidence needed to establish the appropriateness of items.

5. A second type of validity evidence is needed to ensure that NAEP test items match the specific objectives of a given assessment. The items must reflect the objectives, and the item pool must match the percentage distribution for the content and cognitive dimensions at each grade level, as stated in the framework. Minor deviations, if any, from the content domain as defined by the framework will be explained in supporting materials.

6. Supporting material submitted with the NAEP items will provide a description of procedures followed by item writers during development of NAEP test questions. This description will include the expertise, training, and demographic characteristics of the groups. This supporting material must show that all item writing and review groups have the required expertise and training in the subject matter, bias, fairness, and assessment development. 

7. In submitting items for review by the Board, NCES will provide information on the relationship of the specifications and the content/process elements of the pool of NAEP items. This will include procedures used in classifying each item. 

8. The item types used in an assessment must match the content requirements as stated in the framework and specifications, to the extent possible. The match between an objective and the item format must be informed by specifications pertaining to the content, knowledge or skill to be measured, cognitive complexity, overall appropriateness, and efficiency of the item type. NAEP assessments shall use a variety of item types as best fit the requirements stated in the framework and specifications.

9. In order to ensure consistency between the framework and specifications documents and the item pools, NCES will ensure that the development contractor engages a minimum of 20% of the membership of the framework project committees in each subject area to serve on the item writing and review groups as the NAEP test questions are being developed. This overlap between the framework development committees and the item developers will provide stability throughout the NAEP development process, and ensure that the framework and specifications approved by the Board have been faithfully executed in developing NAEP test questions. 

##### Principle 2

The achievement level descriptions for basic, proficient, and advanced performance shall be an important consideration in all phases of NAEP development and review.

##### Policies and Procedures

1. During the framework development process, the project committees shall draft preliminary descriptions of the achievement levels for each grade to be assessed. These preliminary descriptions will define what students should know and be able to do at each grade, in terms of the content and process dimensions of the framework at the basic, proficient, and advanced levels. Subsequent to Board adoption, the final achievement level descriptions shall be an important consideration in all future test item development for a given subject area framework.

2. The achievement level descriptions will be used to ensure a match between the descriptions and the resulting NAEP items. The achievement level descriptions will be examined, and appropriate instruction provided to item writers to ensure that the items represent the stated descriptions, while adhering to the content and process requirements of the framework and specifications. The descriptions will be used to evaluate the test questions to make certain that the pool of questions encompasses the range of content and process demands specified in the achievement level descriptions, including items within each achievement level interval, and items that scale below basic. 

3. As the NAEP item pool is being constructed, additional questions may need to be written for certain content/skill areas if there appear to be any gaps in the pool, relative to the achievement level descriptions.  

4. Supporting materials will show the relationship between the achievement levels descriptions and the pool of NAEP test questions.

##### Principle 3

The Governing Board shall have final authority over all NAEP test questions. This authority includes, but is not limited to, the development of items, establishing the criteria for reviewing items, and the process for review.

##### Policies and Procedures

1. Under the No Child Left Behind Act, a primary duty of the Governing Board pertains to “All Cognitive and Noncognitive Assessment Items.” Specifically, the statute states that, “The Board shall have final authority on the appropriateness of all assessment items.” Under the law, the Board is therefore responsible for all NAEP test questions as well as all NAEP background questions administered as part of the assessment.

2. To meet this statutory requirement, the Board’s Policy on NAEP Item Development and Review shall be adhered to during all phases of NAEP item writing, reviewing, editing, and assessment construction. The National Center for Education Statistic (NCES), which oversees the operational aspects of NAEP, shall ensure that all internal and external groups involved in NAEP item development activities follow the Guiding Principles, Policies and Procedures as set forth in this Board policy.

3. Final review of all NAEP test questions for bias and appropriateness shall be performed by the Board, after all other review procedures have been completed, and prior to administration of the items to students.  

##### Principle 4

The Governing Board shall review all NAEP test questions that are to be administered in conjunction with a pilot test, field test, operational assessment, or special study administered as part of NAEP.

##### Policies and Procedures

1. To fulfill its statutory responsibility for NAEP item review, the Board shall receive, in a timely manner and with appropriate documentation, all test questions that will be administered to students under the auspices of a NAEP assessment. These items include those slated for pilot testing, field testing, and operational administration. 

2. The Board shall review all test items developed for special studies, where the purpose of the special study is to investigate alternate item formats or new technologies for possible future inclusion as part of main NAEP, or as part of a special study to augment main NAEP data collection. 

3. The Board shall not review items being administered as part of test development activities, such as small-scale, informal try-outs with limited groups of students designed to refine items prior to large-scale pilot, field, or operational assessment. 

4. NCES shall submit NAEP items to the Board for review in accordance with a mutually agreeable timeline. Items will be accompanied by appropriate documentation as required in this policy. Such information shall consist of procedures and personnel involved in item development and review, the match between the item pool and the framework content and process dimensions, and other related information. 

5. For its first review, the Board will examine all items prior to the pilot test or field test stage. In the case of the NAEP reading assessment, all reading passages will be reviewed by the Board prior to item development. For each reading passage, NCES will provide the source, author, publication date, passage length, rationale for minor editing to the passage (if any), and notation of such editing applied to the original passage. NCES will provide information and explanatory material on passages deleted in its fairness review procedures. 

6. For its second review, the Board will examine items following pilot or field testing. The items will be accompanied by statistics obtained during the pilot test or field test stage. These statistics shall be provided in a clear format, with definitions for each item analysis statistic collected. Such statistics shall include, but shall not be limited to: p-values for multiple-choice items, number and percentage of students selecting each option for a multiple-choice item, number and percentage not reaching or omitting the item (for multiple-choice and open-ended), number and percentage of students receiving various score points for open-ended questions, mean score point value for open-ended items, appropriate biserial statistics, and other relevant data. 

7. At a third stage, for some assessments, the Board will receive a report from the calibration field test stage, which occurs prior to the operational administration. This “exceptions report” will contain information pertaining to any items that were dropped due to differential item functioning (DIF) analysis for bias, other items to be deleted from the operational assessment and the rationale for this decision, and the final match between the framework distribution and the item pool. If the technology becomes available to perform statistically sound item-level substitutions at this point in the cycle (from the initial field test pool), the Board shall be informed of this process as well.

8. All NAEP test items will be reviewed by the Board in a secure manner via in-person meetings, teleconference or videoconference settings, or on-line via a password-protected Internet site. The Board’s Assessment Development Committee shall have primary responsibility for item review and approval. However, the Assessment Development Committee, in consultation with the Board Chair, may involve other NAGB members in the item review process on an *ad**hoc* basis. The Board may also submit items to external experts, identified by the Board for their subject area expertise, to assist in various duties related to item review. Such experts will follow strict procedures to maintain item security, including signing a Nondisclosure Agreement. 

9. Items that are edited between assessments by NCES and/or its item review committees, for potential use in a subsequent assessment, shall be re-examined by the Board prior to a second round of pilot or field testing. 

10. Documentation of the Board’s final written decision on editing and deleting NAEP items shall be provided to NCES within 10 business days following completion of Board review at each stage in the process. 

##### Principle 5

NAEP test questions will be accurate in their presentation, and free from error. Scoring criteria will be accurate, clear, and explicit. 

Policies and Procedures

1. NCES, through its subject area content experts, trained item writers, and item review panels, will examine each item carefully to ensure its accuracy. All materials taken from published sources must be carefully documented by the item writer. Graphics that accompany test items must be clear, correctly labeled, and include the data source where appropriate. Items will be clear, grammatically correct, succinct, and unambiguous, using language appropriate to the grade level being assessed. Item writers will adhere to the specifications document regarding appropriate and inappropriate stimulus materials, terminology, answer choices or distractors, and other requirements for a given subject area. Items will not contain extraneous or irrelevant information that may differentially distract or disadvantage various subgroups of students from the main task of the item. 

2. Scoring criteria will accompany each constructed-response item. Such criteria will be clear, accurate, and explicit. Carefully constructed scoring criteria will ensure valid and reliable use of those criteria to evaluate student responses to maximize the accuracy and efficiency of scoring.

3. Constructed-response scoring criteria will be developed initially by the item writers, refined during item review, and finalized during pilot or field test scoring. During pilot or field test scoring, the scoring guides will be expanded to include examples of actual student responses to illustrate each score point. Actual student responses will be used as well, to inform scorers of unacceptable answers. 

4. Procedures used to train scorers and to conduct scoring of constructed-response items must be provided to the Board, along with information regarding the reliability and validity of such scoring. If the technology becomes available to score student responses electronically, the Board must be informed of the reliability and validity of such scoring protocol, as compared to human scoring. 

##### Principle 6

All NAEP test questions will be free from racial, cultural, gender, or regional bias, and must be secular, neutral, and non-ideological. NAEP will not evaluate or assess personal or family beliefs, feelings, and attitudes, or publicly disclose personally identifiable information. 

##### Policies and Procedures

1. An item is considered biased if it unfairly disadvantages a particular subgroup of students by requiring knowledge of obscure information unrelated to the construct being assessed. A test question or passage is biased if it contains material derisive or derogatory toward a particular group. For example, a geometry item requiring prior knowledge of the specific dimensions of a basketball court would result in lower scores for students unfamiliar with that sport, even if those students know the geometric concept being measured. Use of a regional term for a soft drink in an item context may provide an unfair advantage to students from that area of the country. Also, an item that refers to a low-achieving student as “slow” would be unacceptable.

2. In conducting bias reviews, steps should be taken to rid the item pool of questions that, because of their content or format, either appear biased on their face, or yield biased estimates of performance for certain subpopulations based on gender, race, ethnicity, or regional culture. A statistical finding of differential item functioning (DIF) will result in a review aimed at identifying possible explanations for the finding. However, such an item will not automatically be deleted if it is deemed valid for measuring what was intended, based on the NAEP assessment framework. Items in which clear bias is found will be eliminated. This policy acknowledges that there may be real and substantial differences in performance among subgroups of students. Learning about such differences, so that performance may be improved, is part of the value of the National Assessment. 

3. Items shall be secular, neutral, and non-ideological. Neither NAEP nor its questions shall advocate a particular religious belief or political stance. Where appropriate, NAEP questions may deal with religious and political issues in a fair and objective way. The following definitions shall apply to the review of all NAEP test questions, reading passages, and supplementary materials used in the assessment of various subject areas: 

· Secular – NAEP questions will not contain language that advocates or opposes any particular religious views or beliefs, nor will items compare one religion unfavorably to another. However, items may contain references to religions, religious symbolism, or members of religious groups where appropriate. Examples: The following phrases would be acceptable: “shaped like a Christmas tree”, “religious tolerance is one of the key aspects of a free society,” “Dr. Martin Luther King, Jr. was a Baptist minister,” or “Hinduism is the predominant religion in India.”  

· Neutral and Non-ideological - Items will not advocate for a particular political party or partisan issue, for any specific legislative or electoral result, or for a single perspective on a controversial issue. An item may ask students to explain both sides of a debate, or it may ask them to analyze an issue, or to explain the arguments of proponents or opponents, without requiring students to endorse personally the position they are describing. Item writers should have the flexibility to develop questions that measure important knowledge and skills without requiring both pro and con responses to every item. Examples: Students may be asked to compare and contrast positions on states rights, based on excerpts from speeches by X and Y; to analyze the themes of Franklin D. Roosevelt’s first and second inaugural addresses; to identify the purpose of the Monroe Doctrine; or to select a position on the issue of suburban growth and cite evidence to support this position. Or, students may be asked to provide arguments either for or against Woodrow Wilson’s decision to enter World War I. A NAEP question could ask students to summarize the dissenting opinion in a landmark Supreme Court case. The criteria of neutral and non-ideological also pertain to decisions about the pool of test questions in a subject area, taken as a whole. The Board shall review the entire item pool for a subject area to ensure that it is balanced in terms of the perspectives and issues presented.

4. The Board shall review both stimulus materials and test items to ensure adherence to the NAEP statute and the polices in this statement. Stimulus materials include reading passages, articles, documents, graphs, maps, photographs, quotations, and all other information provided to students in a NAEP test question.

5. NAEP questions will not ask a student to reveal personal or family beliefs, feelings, or attitudes, or publicly disclose personally identifiable information. 

Appendix C

## NAEP Mathematics Project Staff and Committees

**Members of NAGB’s Grade 12 Mathematics Panel**

**Herbert Clemens**

Professor, Department of Mathematics

Ohio State University

Columbus, Ohio

**Mary Ann Huntley**

Assistant Professor, Mathematics

Department of Mathematical Sciences

University of Delaware

Newark, Delaware

**Jeremy Kilpatrick**

Regents Professor

University of Georgia

Athens, Georgia

**Mary Lindquist**

Fuller E. Callaway Professor, Emeritus

Columbus State University

Lewisburg, West Virginia

**Mary Jo Messenger**

Chair, Department of Mathematics (retired)

River Hill High School

Clarksville, Maryland

**William Schmidt**

University Distinguished Professor

Michigan State University

East Lansing, Michigan

### NAEP Grade 12 Mathematics Project

**Achieve NAEP Grade 12 Mathematics Panel**

**Sue Eddins**

Mathematics Teacher (retired)

Illinois Mathematics and Science Academy

Aurora, Illinois

**William McCallum**

University Distinguished Professor of MathematicsDepartment of Mathematics

University of Arizona

Tucson, Arizona

**Fabio Milner**

Professor of Mathematics

Purdue University

West Lafayette, Indiana

**William Schmidt**

University Distinguished Professor

Michigan State University

East Lansing, Michigan

**Lynn Steen**

Professor of Mathematics

St. Olaf College

Northfield, Minnesota

**Norman Webb**

Senior Research Scientist

Wisconsin Center for Education Research

University of Wisconsin

Madison, Wisconsin

**Reviews Received on the Draft of NAEP 12th Grade Mathematics Objectives**

Achieve, Inc.

American Mathematical Society

Association of State Supervisors of Mathematics

Thomas B. Fordham Institute

State Mathematics Supervisors from various states

National Council of Teachers of Mathematics

State Testing Directors from various states

**2009 NAEP Mathematics Specifications Working Group**

**Mary Lindquist**

Fuller E. Callaway Professor, Emeritus

Columbus State University

Lewisburg, West Virginia

**Mary Jo Messenger**

Chair, Department of Mathematics (retired)

River Hill High School

Clarksville, Maryland

**Linda Dager Wilson**, Chair

Mathematics Consultant

Washington, D.C.

**Phoebe C. Winter**, Editor

Measurement Consultant

Richmond, Virginia

### 2005 NAEP Mathematics Project Steering Committee

**Eileen Ahearn**

Project Director

National Association of State Directors of Special Education

Alexandria, Virginia

**Charles Allan**

Mathematics Education Consultant

Michigan Department of Education

Lansing, Michigan

**B. Marie Byers**

National School Boards Association

Hagerstown, Maryland

**Randy DeHoff**

Colorado State Board of Education

6th Congressional District–Littleton

Denver, Colorado

**M.B. “Sonny” Donaldson**

Superintendent

Aldine ISD

Houston, Texas

**Janice Earle**

Senior Program Director

National Science Foundation

Arlington, Virginia

**Lou Fabrizio**

Director

Division of Accountability Services

North Carolina Department of Public Instruction

Raleigh, North Carolina

**Bettye Forte**

Mathematics Consultant

Arlington, Texas

**Matt Gandal**

Vice President

Achieve, Inc.

Washington, D.C.

**Alice Gill**

Associate Director

Educational Issues

American Federation of Teachers

Washington, D.C.

**M. Kathleen Heid**

The Pennsylvania State University

University Park, Pennsylvania

**Audrey Jackson**

Assistant Principal

Claymont Elementary School

Parkway City Schools

Fenton, Missouri

**James M. Landwehr**

Director

Data Analysis Research Department

Avaya Labs

Basking Ridge, New Jersey

**Sharon Lewis**

Research Director

Council of the Great City Schools

Washington, D.C.

**Dane Linn**

Policy Studies Director

National Governors’ Association

Washington, D.C.

**Eddie Lucero**

Principal

Griegos Elementary School

Albuquerque, New Mexico

**Lee McCaskill**

Principal

Brooklyn Technical High School

Brooklyn, New York

**Barbara Montalto**

Assistant Director of Mathematics

Texas Education Agency

Austin, Texas

**Judy Rohde**

Mathematics Instructor

John Glenn Middle School

Maplewood, Minnesota

**Wilfried Schmid**

Professor of Mathematics

Harvard Department of Mathematics

Cambridge, Massachusetts

**Sr. Mary Frances Taymans**

Associate Executive Director

Secondary Schools Department

National Catholic Education Association

Washington, D.C.

**Zalman Usiskin**

Professor of Education

Director, University of Chicago School Mathematics Project

Chicago, Illinois

**Judy Walter**

Association for Supervision & Curriculum Development

Alexandria, Virginia

**Diana Wearne**

Associate Professor

School of Education

University of Delaware

Newark, Delaware

**Hung-Hsi Wu**

Professor of Mathematics

Department of Mathematics

University of California–Berkeley

Berkeley, California

2005 NAEP Mathematics Project Planning Committee 

**Dayo Akinsheye**

Mathematics Resource Teacher

Seaton Elementary School

Washington, D.C.

**Geri Anderson-Nielsen**

Mathematics Specialist

Georgetown Day School

Washington, D.C.

**Cindy Chapman**

Elementary Teacher

Albuquerque Public Schools

Albuquerque, New Mexico

**Herbert Clemens**

Professor of Mathematics

Department of Mathematics

University of Utah

Salt Lake City, Utah

**Carl Cowen**

Professor of Mathematics

Purdue University

West Lafayette, Indiana

**Jim Ellingson**

Assistant Professor

Concordia College

Moorhead, Minnesota

**Joan Ferrini-Mundy**

Associate Dean/Director of Science and Mathematics

College of Natural Science

Michigan State University

East Lansing, Michigan

**Kim Gattis**

Education Program Consultant

Kansas Department of Education

Association of State Supervisors of Mathematics

Topeka, Kansas

**Anne Gonzales**

Middle School Mathematics Teacher

South Gate Middle School

South Gate, California

**Jeremy Kilpatrick**

Professor of Mathematics Education

University of Georgia

Athens, Georgia

**Gerald Kulm**

Curtis D. Robert Professor of Mathematics Education

Texas A & M University

College Station, Texas

**Mary Lindquist**

Fuller E. Callaway Professor of Mathematics Education

Columbus State University

Columbus, Georgia

**Mary Jo Messenger**

Chair, Department of Mathematics

River Hill High School

Clarksville, Maryland

**Marjorie Petit**

Senior Associate

National Center for the Improvement of Educational Assessment

(The Center for Assessment)

Portsmouth, New Hampshire 

**Edward Silver**

Professor

School of Education

University of Michigan

Ann Arbor, Michigan

**Debra Vitale**

Mathematics Specialist

Arlington Public Schools

Fairfax, Virginia

**Frank Wang**

President/CEO

Saxon Publishing, Inc.

Norman, Oklahoma

**Norman Webb**

Senior Research Scientist

Wisconsin Center for Education Research

Madison, Wisconsin

**John Wisthoff**

Member, Maryland State Board of Education and Mathematics Professor

Anne Arundel Community College

Pasadena, Maryland

### 2005 NAEP Mathematics Project Technical Advisory Panel

**Fen Chou**

Psychometrician

Louisiana Department of Education

Baton Rouge, Louisiana

**Eugene Johnson**

Chief Psychometrician

American Institutes for Research

Washington, D.C.

**Edward Kifer**

Professor and Chairperson

Department of Educational Policy Studies and Evaluation

College of Education 

University of Kentucky

Lexington, Kentucky

**Ina Mullis**

Co-Director

International Study Center

Boston College

Chestnut Hill, Massachusetts

**Barbara Plake**

Director

Buros Center for Testing

University of Nebraska–Lincoln

Lincoln, Nebraska

**Roger Trent**

Ohio State Assessment Director (Emeritus)

Ohio Department of Education

Columbus, Ohio

**CCSSO Staff**

**Rolf Blank**

Director of Indicators Project

State Education Assessment Center

Council of Chief State School Officers

Washington, D.C.

**Wayne Martin**

Director

State Education Assessment Center

Council of Chief State School Officers

Washington, D.C.

**John Olson**

Director of Assessment

State Education Assessment Center 

Council of Chief State School Officers

Washington, D.C.

**Frank Philip**

Senior Project Associate

State Education Assessment Center

Council of Chief State School Officers

Washington, D.C.

**Linda Dager Wilson**

Consensus Coordinator Consultant

Council of Chief State School Officers

Washington, D.C.

**Phoebe Winter**

Project Director

State Education Assessment Center

Council of Chief State School Officers

Richmond, Virginia

****Committee Representation

**Policy Organizations**

Achieve, Inc.

American Association of School Administrators (AASA)

American Federation of Teachers (AFT)

American Mathematical Society (AMA)

American Statistical Association (ASA)

Association for Supervision and Curriculum Development (ASCD)

Association of State Assessment Programs (ASAP)

Association of State Supervisors of Mathematics (ASSM)

Business Roundtable/National Alliance of Business

Council of the Great City Schools

Education Leaders Council (ELC)

National Association of Elementary School Principals (NAESP)

National Association of Secondary School Principals (NASSP)

National Association of State Boards of Education (NASBE)

National Association of State Directors of Special Education (NASDE)

National Catholic Education Association (NCEA)

National Education Association (NEA)

National Governors’ Association (NGA)

National Science Foundation (NSF)

National School Boards Association (NSBA)

Representative from national textbook publisher

**Mathematical Associations and Groups**

Mathematically Correct

Mathematics Association of America (MAA)

National Council of Teachers of Mathematics (NCTM)

Third International Mathematics and Science Study (TIMSS)

**Educators**

Classroom mathematics teachers from public and non-public schools 

Principals

District and state mathematics specialists

Mathematics and mathematics education professors from public and private universities, colleges, and community colleges

**Technical Experts**

University professors

State testing specialists

Representatives from private research organizations

# Acknowledgments

The following people were the primary authors of the introductions to the content areas:

Roger Howe, Yale University (Number Properties and Operations, Geometry, and Algebra) 

Richard Scheaffer, University of Florida (Data Analysis, Statistics, and Probability)

Mary Lindquist, Columbus State University (Measurement)

High Complexity

25%

Moderate

Complexity

50%

25%

Low

Complexity

Correct Answer: 4

Correct Answer: B

Correct Answer:  C

Correct Answer:  B

Correct Answer:  D  1010

Correct Answer: 4x2 + 30x + 56  

Correct Answer:  D

Correct Answer: Any two of these combinations.

	(0,9)          (9,0)

	(1,8)          (8,1)

	(2,7)          (7,2)

	(3,6)          (6,3)

	(4,5)          (5,4)

Correct Answer:  A

Correct Answer: A

Correct Answer: E

Correct Answer: E

Correct Answer:  C

Correct Answer:  D

Correct Answer: 200 

Correct Answer:  C

Correct Answer:  B

Number of Pockets

Number of Pockets

Number of Pockets

.

Correct Answer: B

Correct Answer: C

Correct Answer: B

High Complexity

25%

Moderate

Complexity

50%

Low

Complexity

25%

50%

50%

Multiple 

Choice

Constructed 

Response

Correct Answer: 926

Correct Answer: B

B

C

A

Correct Answer: 42 feet

Correct Answer:  C

Correct Answers:

Rectangle or Square

Correct Answer: B

Correct Answer: C

9           6         3

             4         2 

    Correct Answer:            �

 Correct Answer:

     �

Sharif Shakrani

Director, Education Policy Research Center

Michigan State University

East Lansing, Michigan

Linda Dager Wilson, Chair

Mathematics Consultant

Washington, D.C.

NAEP

Subcontractors and Consultants

Patricia Kenney

Senior Research Associate

University of Michigan

Ann Arbor, Michigan 

Rebecca Kopriva

Director

Center for the Study of Assessment Validity in Education

Department of Measurement & Statistics

University of Maryland 

College Park, Maryland

Christopher Cross

President (Former)

Council for Basic Education

Washington, DC 

Kim Gattis

President, Association of State Supervisors of Mathematics

Education Program Consultant, Kansas Department of        	Education

Topeka, Kansas

Linda Plattner

Director of Policy, Standards & Instruction

Council for Basic Education

Washington, DC

Correct Answer:  11

gate

Illustration: Corner of fence showing span and gate. 

|   span   |

Correct Answer: B

Correct Answer: B

Correct Answer: s = 2(A – 1

or equivalent. 

Correct Answer: 3 pencils marked with X

Correct Answer: 9 of the squares shaded

Correct Answer: 3 of the smaller rectangles shaded

Correct Answer: 9 pencils marked with X

� In its review of the literature on effects of accommodations, the National Research Council  suggests that “...language simplification may have removed factors that were construct-irrelevant for all test-takers.” (p. 38)

� Identification of special needs students is problematic in and of itself. While NAEP defines inclusion criteria, student identification and classification is under the purview of states and localities. Consequently, there is wide variation from state to state on the percentage of students identified and classified as either SWD or ELL status. For example, the percentage of students identified as having a qualifying disability for the 2005 NAEP mathematics assessment varied across states from 10% to 24% at grade 4 and from 9% to 18% at grade 8 (Perie, Grigg, & Dion, 2005, p. 36). Similarly, the percentage of students identified as ELL for the 2005 NAEP mathematics assessment varied across states from 1% to 33% at grade 4 and from 1% to 21% at grade 8 (Perie, et al., 2005, p. 36). Of course, much of the ELL variation depends on the overall demographics of particular states, but some of the variation is influenced by identification criteria and procedures. Indeed, Koretz and Barton (2003-2004) and Abedi (2004) have raised questions about the reliability and validity of student classification into SWD and ELL, respectively. 

� For example, the South Carolina Department of Education is working with a consortium of states to refine a research-based decision-making  process for assigning accommodations to ELLs based on individual student characteristics (Kopriva and Carr, 2006) and to develop a decision-making framework for IEP teams to use in assigning accommodations to SWDs  (South Carolina Department of Education, 2006).

� For more information about designing assessments that are accessible to English language learners, see Kopriva (in press) and Abedi (2006).

� For more information about scoring responses from English language learners, see Kopriva and Saez (1997) and Solano-Flores, Lara., Sexton, and Navarrete (2001).

� In some cases, it may be appropriate to have four scoring categories for an extended constructed-response item, depending upon the construct assessed and the nature of expected student responses to the item.

� For example, Webb, Ely, Cormier, and Vesperman (2006).. 

� See The Nation’s Report Card: Mathematics 2005, available at http://nces.ed.gov/nationsreportcard/mathematics/,and NAEP technical reports produced by NCES for more information about administration conditions and procedures.

# _1242553697.unknown

# _1242635569.unknown

# _1100520761.unknown

# _1100520819.unknown

# _1100519991.unknown