NAEP 2014-2016 System Clearance: Appendix C page C-1 

NATIONAL CENTER FOR EDUCATION STATISTICS

NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS

Appendix C 

Example of Sample Design Document (2013 Assessment) 

Request for System Clearance for 

NAEP Assessments for 2014-2016 

OMB# 1850-0790 

 

 
Date: May 24, 2012 Memo: 2013-

1.3A/1.3B/1.3G/1.3D/1.E/1.3R

To: William Ward, NCES

Nicole Beaulieu, ETS

Ed Kulick, ETS

David Freund, ETS

Amy Dresher, ETS

Connie Smith, Pearson

Dianne Walsh

Lauren Byrne

Debby Vivari

Rob Dymowski

William Wall

John Burke

Lloyd Hicks

Jennifer Kali

David Ferraro

Erin Wiley

Dwight Brock

Amy Lin

Chris Averett 

Kavemuii Murangi

David Hubble

Sarah Shore

From: Keith Rust

Subject: Sample Design for 2013 NAEP

I. Introduction 

For 2013, the NAEP assessment involves the following components:

n National assessments in reading and math at grades 4, 8, and 12;

n State-by-state and Trial Urban District Assessment (TUDA) assessments in reading and 

math for public schools at grades 4 and 8;

n State-by-state assessments in reading and math for 13 selected states at grade 12;

n A special study to link NAEP and the High School Longitudinal Study (HSLS), for 

mathematics at grade 12;
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 2 of 24

n A special study to link NAEP and the Lexile assessment, for reading at grade 8;

n A special study of Knowledge and Skills Appropriate (KaSA) items in mathematics, at 

grades 4 and 8, including Puerto Rico;

n A special study of Accessible Booklets, for reading at grades 4 and 8;

n A special study of Read-Aloud Accommodations, for reading at grade 4;

n A special study to link NAEP and PISA, for mathematics, in certain schools selected for 

grade 12 NAEP;

n Pilot tests in reading and math at grades 4 and 8;

n A pilot test of the Technology and Engineering Literacy (TEL) Assessment at grade 8.

Below is a summary list of the features of the 2013 sample design.

1. As in recent NAEP studies, Trial Urban District Assessment (TUDA) samples will form 

part of the corresponding state samples, and the state samples will form part of the 

national sample. The same twenty-one Trial Urban District Assessment (TUDA)

participants as in 2011 will be involved.

2. At grade 8 there will be two public school samples, and two private school samples. A 

relatively small PSU-based sample, known as the beta sample (for public schools) and 

the epsilon sample (for private schools), will be used for the computer-based TEL pilot 

assessment. The alpha samples for grades 4 and 8 public, the gamma sample for grade 

12 public, and the delta samples for private schools at grades 4, 8, and 12 will be used 

for the operational assessments in reading and mathematics, and for the pilot test 

samples for reading and mathematics, and all special studies except for the Read-Aloud 

Accommodations Study.

3. Oversampling of private schools at grades 4 and 8 will be scaled back to the level of 

2009. Response rates permitting, this will allow separate reporting, for reading and 

mathematics, for Catholic, and non-Catholic schools, but not further breakdown. As in

2009 there will be no oversampling of private schools at grade 12.

4. There will be no samples in territories, other than for Puerto Rico at grades 4 and 8 for 

the KaSA study.

5. For the first time since 2003, there is no National Indian Education Study (NIES). This 

means that less extensive sampling of BIE schools is required. To ensure sound results 

for AIAN students at the national level, at grades 4 and 8 BIE students will be sampled 

at the same rate as students in Oklahoma, the state with the largest AIAN population.

6. As in 2011, the Department of Defense Schools are expected to be reported as a single 

jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and 
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 3 of 24

overseas (DoDDS). However, for design purposes, we will still sample these as two 

separate entities.

7. At grade 12, there will state-level samples for 13 states. Eleven of these also participated 

at the state level in 2009. As at grades 4 and 8, state and national samples will be 

integrated. The 13 states participating at the state level for reading and mathematics are 

Arkansas, Connecticut, Florida, Idaho, Illinois, Iowa, Massachusetts, Michigan, New 

Hampshire, New Jersey, South Dakota, Tennessee, and West Virginia. The NAEPPISA linking study will be conducted in the grade 12 sample schools in Connecticut, 

Florida, and Massachusetts. These three states will conduct state-level PISA in the fall 

of 2012.

8. The Read-Aloud Accommodations study will be conducted in special sample of grade 4 

public schools, to be referred to as the rho sample. This sample will be restricted to a 

small number of states, and will not have any overlap schools with any of the other 

2013 NAEP assessments. The sample will not be a nationally representative probability 

sample of schools.

9. The sample sizes of assessed students for these various components are shown in Table 

1 (which also shows the approximate numbers of participating schools). 

Note that the figures in Table 1 (and also Tables 3 and 4) do not reflect the fact that, at grade 4, 

certain schools in the pencil and paper assessments may elect to include all students in the sample, 

rather than the sample size targeted by NAEP. This will lead to an increase is sample size, likely to 

be about five percent overall. The increase will vary by state, depending on the size of the state and 

the size of its schools – obviously there is no addition in states where the design calls for the 

selection of all students.

Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2013 NAEP

Spiral Jurisdictions Students 

Total 

Spiral 

Indic. 

States (incl. 

DC, BIE, 

DoDEA) 

Urban 

districts 

Public 

school 

students 

Private 

school 

students 

Grade 4

nat'l/state math RM 53 21 163,000 3,000 166,000

nat'l/state reading RM 53 21 163,000 3,000 166,000

math Pilot RM 4410 90 4500

reading Pilot RM 5145 105 5250

math KaSA RM 3920 80 4000

Reading accessible booklet study RM 2940 60 3000

Puerto Rico KaSA PR 1 5000 0 5000

Total - alpha 2 

 

347,415

Total- delta 1 

 

6,335

 
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 4 of 24

Table 1. Target sample sizes of assessed students, and expected number of participating 

schools, for 2013 NAEP (Continued)

Spiral Jurisdictions Students 

Total 

Spiral 

Indic. 

States (incl. 

DC, BIE, 

DoDEA) 

Urban 

districts 

Public 

school 

students 

Private 

school 

students 

Grade 4 (Continued)

Typical max. no. students/school 60 64

Average assessed 

students/school 44 24

Total schools - alpha, delta 

 

7,900 265 8,183 

Reading Aloud Study

RA, RB, 

RC 2,100 2,100

Total - Rho 3 2,100 

Typical max. no. students/school 27

Average assessed 

students/school 27

Total schools - rho 78 0 78 

Total number of students grade 4 349,515 6,335 355,850 

Total number of schools grade 4 7,970 265 8,243 

Grade 8 

nat'l/state math RM 53 21 159,080 3,000 162,080

nat'l/state reading RM 53 21 159,080 3,000 162,080

math Pilot RM 4,410 90 4,500

reading Pilot RM 5,145 105 5,250

math KaSA RM 3,920 80 4,000

Reading accessible booklet study RM 2,940 60 3,000

Lexile Linking Study RM 7,840 160 8,000

Puerto Rico KaSA PR 1 5,000 0 5,000

Total - alpha 2 

 

347,415

Total - delta 1 

 

6,495

Typical max. no. students/school 60 65

Average assessed 

students/school 50 25

Total schools - alpha, delta 

 

6,950 260 7,210

TEL Pilot TL 13,500 1,500 15,000

Total - beta 1 

 

13,500

Total - epsilon 1 

 

1,500

Typical max. no. students/school 30 30

Average assessed 

students/school 25 25

Total schools - beta, epsilon 

 

540 60 600

Total number of schools grade 8 

 

7,410 320 7,810

Total number of students grade 8 

 

360,915 7,995 368,910
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 5 of 24

Table 1. Target sample sizes of assessed students, and expected number of participating 

schools, for 2013 NAEP (Continued)

Spiral Jurisdictions Students 

Total 

Spiral 

Indic. 

States (incl. 

DC, BIE, 

DoDEA) 

Urban 

districts 

Public 

school 

students 

Private 

school 

students 

Grade 12 

national and state reading RM 13 41,120 1,500 42,620

National and state math RM 13 38,380 1,400 39,780

HSLS Linking Study ML 6,000 6,000

NAEP-PISA Linking Study MP 3 18,000 18,000

Total - gamma 3 103,500 

Total- delta 1 2,900

Typical max. no. students/school

(excluding HSLS) 60 60

Average assessed 

students/school 50 40

Total schools - gamma, delta 1,690 73 1,763

GRAND TOTAL STUDENTS 813,930 17,230 831,160

GRAND TOTAL SCHOOLS 17,070 658 17,736

II. Assessment Types 

The assessment spiral types are shown in Table 2. At grade 4 there will be five different spirals used, 

with three at grade 8 and three at grade 12. Session IDs contain six digits, traditionally. The first two 

digits identify the assessment “type” (subjects and type of spiral in a general way). Grade is 

contained in the second pair of digits, and the session sequential number (within schools) in the last 

two digits. For example, session RM 0401 denotes the first grade 4 reading and mathematics 

operational assessment in a given school.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 6 of 24

Table 2. NAEP 2013 assessment types and IDs

ID Type Subjects Grades Schools Comments 

RM

Operational

Pencil & Paper

Reading (including pilot, 

accessible booklets), math

(including pilot, KaSA) 4

Public and 

Private

Includes all grade 4 assessment 

components, except for Puerto 

Rico and Reading Aloud

RM

Operational–

Pencil & Paper

Reading (including pilot, 

accessible booklets, Lexile 

linking), math (including 

pilot and KaSA) 8

Public and 

Private

Includes all grade 8 

components except Puerto Rico 

and TEL pilot

RM

Operational–

Pencil & Paper Reading, math 12

Public and 

private

TL

PilotComputerbased

Technology and 

Engineering Literacy 8

Public and 

Private

National sample, not in Puerto 

Rico

ML

Special Study -

Pencil & Paper HSLS Linking Study 12 Public

MP

Special Study 

– Pencil & 

Paper PISA Linking Study

9, 10, 

11 Public In three states only

RA

Special Study 

– Pencil & 

Paper Reading Aloud Study 4 Public

In a small number of states; no 

special accommodations

RB

Special Study 

– Pencil & 

Paper Reading Aloud Study 4 Public

In a small number of states; 

Read Aloud Items

RC

Special Study 

– Pencil & 

Paper Reading Aloud Study 4 Public

In a small number of states; 

Read Aloud Everything

PR Puerto Rico KaSA math 4 & 8 Public

Booklets will not be used in any 

other jurisdiction.

III. Sample Types and Sizes 

In similar fashion to past years (but somewhat different), we will identify six different types of 

school samples: Alpha, Beta, Gamma, Delta, Epsilon, and Rho. These distinguish sets of schools 

that will be conducting distinct portions of the assessment. The Rho sample is new for 2013; this 

will be used for the grade 4 Read-Aloud Accommodations study which has a unique design.

1. Alpha Samples at Grades 4 and 8 

These are public school samples for grades 4 and 8. They will be used for the operational state-bystate assessments in reading and math, and contribute to the national samples for these subjects as 
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 7 of 24

well. The will also be used for pilot tests and special studies, and outlined in Tables 1 and 2. There 

will be alpha samples for each state, DC, DoDEA, BIE, and Puerto Rico.

The details of the target student sample sizes for the alpha samples are as follows:

n With the exception of BIE, the target sample size for each state and grade is 6,600

students. For BIE, the sample will be selected at the same rate as Oklahoma. This is to 

ensure adequate samples of American Indian students for national reporting. The rate 

from Oklahoma is being used since it has the largest American Indian population of any 

state. Therefore using this rate ensures that the BIE sample will not be so small as to 

unduly inflate the standard errors of estimates for American Indian/Alaska Native 

students, while at the same time not placing undue burden on BIE schools, for which 

no jurisdiction level results are to be reported, in the absence of the NIES.

n There will be samples for twenty-one TUDA districts. For the five largest (New York 

City, Los Angeles, Chicago, Miami-Dade, and Houston), the student target sample sizes 

for reading and mathematics are three-quarters the size of a state sample (2,475 per 

subject). For the other fifteen districts the student target sample sizes for reading and 

math are one-half the size of a state sample (1,650 per subject). An exception will be 

made in the case of Detroit, where the target sample sizes will be increased in 

anticipation of substantial numbers of schools closings where the particular schools 

affected have not yet been identified.

Note, that above, there is a conflict between sample size requirements at the state level, and the 

TUDA district level. This will be resolved as in previous years: the districts will have the target 

samples indicated in B, and reflected in Table 3. For the states that contain one or more of these 

districts, the target sample size indicated in A (and shown in Table 3) will be used to determine a 

school sampling rate for the state, which will be applied to the balance of the state outside the 

TUDA district(s). Thus the target student sample sizes, shown in Table 3, for states that contain a 

TUDA district, are only ‘design targets’, and are smaller than the final total sample size for the state, 

but larger than the sample for the balance of the state, exclusive of its TUDA districts. In the case of 

the District of Columbia, the state sample size requirement is that all schools and students be 

included. This renders moot any requirements for the DC TUDA sample, which by default consists 

of all schools operated by the DCPS district (but excludes charter schools in DC, even though those 

are all included in the state sample, as these are not operated by DCPS).

In Puerto Rico, the target sample size is 5,800 per grade (grades 4 and 8), with the goal of assessing 

5,000 students. Only KaSA mathematics will be assessed in Puerto Rico (PR session type).
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 8 of 24

As in past state-by-state assessments, schools with fewer than 20 students in the grade in question 

will be sampled at a moderately lower rate than other schools (at least half, and often higher, 

depending upon the size of the school). This is in implicit recognition of the greater cost and burden 

associated with surveying these schools.

Table 3 shows the target student sample sizes, and the approximate counts of schools to be selected 

in the alpha samples, along with the school and student frame counts, by state and TUDA districts 

for grades 4 and 8. The table also identifies the jurisdictions where we take all schools and where we 

take all students.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 9 of 24

Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for 

the 2013 state-by-state and TUDA district assessments (alpha samples)

 Grade 4 Grade 8 

Jurisdiction 

Schools in 

frame 

Schools in 

sample 

Students 

in frame 

Overall 

target 

student 

sample 

size 

 

Schools in 

frame 

Schools in 

sample 

Students in 

frame 

Overall 

target 

student 

sample size 

Alabama 748 117 59,269 6,600 484 112 57,283 6,600 

Alaska 365 197 9,827 6,600 285 144 9,701 6,600

Arizona 1,179 120 83,555 6,600 769 118 81,283 6,600

Arkansas 496 119 37,017 6,600 315 112 35,831 6,600

Bureau Of Indian Education 135 16 3,246 411 109 13 2,785 375

California 5,846 113 463,973 6,600 2,818 113 456,125 6,600

Colorado 1,013 122 63,266 6,600 520 116 59,357 6,600

Connecticut 604 116 41,489 6,600 306 110 42,667 6,600

Delaware 115 95 9,647 6,600 60 60 9,398 6,600 *

District Of Columbia 131 131 4,907 4,907 ** 79 79 4,307 4,307 **

DoDEA Schools 108 108 7,507 7,507 ** 63 63 5,589 5,589 **

Florida 2,097 110 202,702 6,600 1,150 112 196,792 6,600

Georgia 1,242 107 130,245 6,600 548 106 124,307 6,600

Hawaii 202 117 14,155 6,600 80 60 12,843 6,600

Idaho 368 128 21,351 6,600 200 103 20,466 6,600

Illinois 2,343 123 154,495 6,600 1,603 119 154,308 6,600

Indiana 1,101 116 80,245 6,600 485 107 79,856 6,600

Iowa 674 137 35,421 6,600 387 118 34,991 6,600

Kansas 729 141 35,907 6,600 412 126 34,645 6,600

Kentucky 731 117 51,685 6,600 391 114 49,403 6,600

Louisiana 789 120 55,300 6,600 531 119 50,584 6,600

Maine 341 159 13,945 6,600 211 115 14,205 6,600

Maryland 887 115 61,870 6,600 356 109 60,987 6,600

Massachusetts 973 117 71,298 6,600 482 108 71,962 6,600

Michigan 1,856 121 118,242 6,600 1,042 114 120,377 6,600

Minnesota 952 126 61,086 6,600 695 125 61,246 6,600

Mississippi 436 114 38,958 6,600 290 108 36,999 6,600

Missouri 1,170 131 68,189 6,600 727 124 68,007 6,600
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 10 of 24

Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for 

the 2013 state-by-state and TUDA district assessments (alpha samples) (Continued)

 Grade 4 Grade 8 

Jurisdiction 

Schools in 

frame 

Schools in 

sample 

Students 

in frame 

Overall 

target 

student 

sample 

size 

 

Schools in 

frame 

Schools in 

sample 

Students in 

frame 

Overall 

target 

student 

sample size 

Montana 400 199 10,863 6,600 289 151 10,912 6,600 

Nebraska 568 165 22,085 6,600 339 131 21,503 6,600

Nevada 379 112 33,851 6,600 158 88 33,028 6,600

New Hampshire 265 133 14,495 6,600 137 88 15,191 6,600

New Jersey 1,366 117 100,453 6,600 741 110 99,535 6,600

New Mexico 431 127 25,731 6,600 203 105 24,183 6,600

New York 2,368 113 192,295 6,600 1,335 109 193,235 6,600

North Carolina 1,414 112 118,129 6,600 694 110 111,343 6,600

North Dakota 260 260 6,995 6,995 ** 188 188 7,330 7,330 **

Ohio 1,877 119 133,358 6,600 1,090 112 132,686 6,600

Oklahoma 895 137 49,300 6,600 593 128 46,433 6,600

Oregon 767 129 42,827 6,600 414 120 42,949 6,600

Pennsylvania 1,742 116 129,550 6,600 914 109 132,799 6,600

Puerto Rico 1,017 160 38,842 5,800 407 123 37,363 5,800

Rhode Island 170 117 10,437 6,600 57 57 10,842 6,600 *

South Carolina 618 112 55,228 6,600 298 107 52,433 6,600

South Dakota 328 194 9,380 6,600 254 153 9,306 6,600

Tennessee 998 116 75,934 6,600 565 112 71,570 6,600

Texas 4,277 112 373,404 6,600 2,165 112 348,907 6,600

Utah 582 112 46,508 6,600 232 106 42,593 6,600

Vermont 224 224 6,419 6,419 ** 122 122 6,364 6,364 **

Virginia 1,137 112 93,610 6,600 388 106 92,179 6,600

Washington 1,214 119 77,826 6,600 606 115 77,099 6,600

West Virginia 425 146 20,875 6,600 202 109 20,637 6,600

Wisconsin 1,111 131 60,997 6,600 633 118 61,063 6,600

Wyoming 188 188 6,849 6,849 ** 93 93 6,568 6,568 **

Albuquerque 99 56 7,594 3,300 42 42 6,903 3,300 *

Atlanta 61 61 4,264 4,264 ** 26 26 3,369 3,369 **
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 11 of 24

Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for 

the 2013 state-by-state and TUDA district assessments (alpha samples) (Continued)

 Grade 4 Grade 8 

Jurisdiction 

Schools in 

frame 

Schools in 

sample 

Students 

in frame 

Overall 

target 

student 

sample 

size 

 

Schools in 

frame 

Schools in 

sample 

Students in 

frame 

Overall 

target 

student 

sample size 

Austin 80 54 6,862 3,300 24 24 5,360 3,300 * 

Baltimore City 124 68 6,292 3,300 93 62 5,501 3,300

Boston 79 79 4,106 4,106 ** 38 38 3,808 3,808 **

Charlotte 105 54 11,245 3,300 38 38 9,852 3,300 *

Chicago 488 96 29,942 4,950 470 95 29,469 4,950

Cleveland 83 83 3,573 3,573 ** 81 81 3,550 3,550 **

Dallas 147 53 12,932 3,300 38 38 10,113 3,300 *

Detroit 121 81 7,584 4,470 74 65 5,001 4,240

Fresno 70 54 5,823 3,300 26 26 5,457 3,300 *

Hillsborough County 164 55 15,161 3,300 80 48 15,020 3,300

Houston 179 82 16,525 4,950 64 49 12,738 4,950

Jefferson County 96 54 7,617 3,300 42 42 7,030 3,300 *

Los Angeles 516 82 51,516 4,950 162 77 46,818 4,950

Miami-Dade 274 83 26,903 4,950 160 76 25,573 4,950

Milwaukee 115 67 5,807 3,300 89 56 5,371 3,300

New York City 709 82 63,731 4,950 458 82 61,278 4,950

Philadelphia 177 60 12,098 3,300 142 58 10,970 3,300

San Diego 138 57 10,040 3,300 63 40 9,691 3,300

District of Columbia PS 87 87 3,369 3,369 ** 37 37 2,357 2,357 **

Counts for states do not reflect the oversampling for their constituent TUDA districts, nor the grade 4 ‘take all’ option.

Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are included in the NAEP sample.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 12 of 24

Table 4 consolidates the target student (and resulting school) sample size numbers, to show the total 

target sample sizes in each state, combining the TUDA targets with those for the balance of the 

state.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 13 of 24

Table 4. Total sample sizes, combining state and TUDA samples

 Grade 4 Grade 8 

Jurisdiction 

Schools 

in frame 

Schools 

in sample 

Students 

in frame 

Overall 

target 

student 

sample 

size 

 

Schools in 

frame 

Schools in 

sample 

Students in 

frame 

Overall 

target 

student 

sample size 

Alabama 748 117 59,269 6,600 484 112 57,283 6,600 

Alaska 365 197 9,827 6,600 285 144 9,701 6,600

Arizona 1,179 120 83,555 6,600 769 118 81,283 6,600

Arkansas 496 119 37,017 6,600 315 112 35,831 6,600

Bureau Of Indian Education 135 16 3,246 411 109 13 2,785 375

California 5,846 290 463,973 17,189 2,818 241 456,125 17,251

Colorado 1,013 122 63,266 6,600 520 116 59,357 6,600

Connecticut 604 116 41,489 6,600 306 110 42,667 6,600

Delaware 115 95 9,647 6,600 60 60 9,398 6,600 *

District Of Columbia 131 131 4,907 4,907 ** 79 79 4,307 4,307 **

DoDEA Schools 108 108 7,507 7,507 ** 63 63 5,589 5,589 **

Florida 2,097 225 202,702 13,479 1,150 213 196,792 13,486

Georgia 1,242 164 130,245 10,648 548 129 124,307 9,790

Hawaii 202 117 14,155 6,600 80 60 12,843 6,600

Idaho 368 128 21,351 6,600 200 103 20,466 6,600

Illinois 2,343 194 154,495 10,267 1,603 190 154,308 10,285

Indiana 1,101 116 80,245 6,600 485 107 79,856 6,600

Iowa 674 137 35,421 6,600 387 118 34,991 6,600

Kansas 729 141 35,907 6,600 412 126 34,645 6,600

Kentucky 731 156 51,685 8,929 391 139 49,403 8,963

Louisiana 789 120 55,300 6,600 531 119 50,584 6,600

Maine 341 159 13,945 6,600 211 115 14,205 6,600

Maryland 887 170 61,870 9,229 356 160 60,987 9,305

Massachusetts 973 188 71,298 10,326 482 140 71,962 10,059

Michigan 1,856 194 118,242 10,646 1,042 174 120,377 10,566

Minnesota 952 126 61,086 6,600 695 125 61,246 6,600

Mississippi 436 114 38,958 6,600 290 108 36,999 6,600

Missouri 1,170 131 68,189 6,600 727 124 68,007 6,600

Montana 400 199 10,863 6,600 289 151 10,912 6,600
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 14 of 24

Table 4. Total sample sizes, combining state and TUDA samples (Continued)

 Grade 4 Grade 8 

Jurisdiction 

Schools 

in frame 

Schools 

in sample 

Students 

in frame 

Overall 

target 

student 

sample 

size 

 

Schools in 

frame 

Schools in 

sample 

Students in 

frame 

Overall 

target 

student 

sample size 

Nebraska 568 165 22,085 6,600 339 131 21,503 6,600 

Nevada 379 112 33,851 6,600 158 88 33,028 6,600

New Hampshire 265 133 14,495 6,600 137 88 15,191 6,600

New Jersey 1,366 117 100,453 6,600 741 110 99,535 6,600

New Mexico 431 149 25,731 7,938 203 119 24,183 8,008

New York 2,368 159 192,295 9,361 1,335 156 193,235 9,456

North Carolina 1,414 156 118,129 9,271 694 138 111,343 9,315

North Dakota 260 260 6,995 6,995 ** 188 188 7,330 7,330 **

Ohio 1,877 197 133,358 9,996 1,090 189 132,686 9,973

Oklahoma 895 137 49,300 6,600 593 128 46,433 6,600

Oregon 767 129 42,827 6,600 414 120 42,949 6,600

Pennsylvania 1,742 165 129,550 9,283 914 156 132,799 9,355

Puerto Rico 1,017 160 38,842 5,800 407 123 37,363 5,800

Rhode Island 170 117 10,437 6,600 57 57 10,842 6,600 *

South Carolina 618 112 55,228 6,600 298 107 52,433 6,600

South Dakota 328 194 9,380 6,600 254 153 9,306 6,600

Tennessee 998 116 75,934 6,600 565 112 71,570 6,600

Texas 4,277 293 373,404 17,665 2,165 216 348,907 17,792

Utah 582 112 46,508 6,600 232 106 42,593 6,600

Vermont 224 224 6,419 6,419 ** 122 122 6,364 6,364 **

Virginia 1,137 112 93,610 6,600 388 106 92,179 6,600

Washington 1,214 119 77,826 6,600 606 115 77,099 6,600

West Virginia 425 146 20,875 6,600 202 109 20,637 6,600

Wisconsin 1,111 185 60,997 9,270 633 163 61,063 9,321

Wyoming 188 188 6,849 6,849 ** 93 93 6,568 6,568 **

Sample sizes for each state do reflect the samples in the TUDA districts within the state, but not the grade 4 take-all option.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are included in the NAEP sample.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 15 of 24

Stratification 

Each state and grade will be stratified separately, but using a common approach in all cases. TUDA 

districts will be separated from their state, and each part stratified separately. The first level of 

stratification will be based on urban-centered type of location. This variable has 12 levels (some of 

which may not be present in a given state or TUDA district), and these will be collapsed so that each 

of the resulting location categories contains at least nine percent of the student population. Within 

each of the resulting location categories, schools will be assigned a minority enrollment status. This 

is based on the two race/ethnic groups that are the second and third most prevalent within the 

location category. If these groups are both low in percentage terms, no minority classification will be 

used. Otherwise three (or occasionally four) equal-sized groups (generally high, medium, and low 

minority) will be formed based on the distribution across schools of the two minority groups.

Within the resulting location and minority group classes (of which there are likely to be from three 

to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school 

level results from the most recent available state achievement tests at the relevant grade. In general, 

mathematics test results will be used, but where these are not available, reading results will be used. 

In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to 

match the results to the NAEP school frame), instead of achievement data, schools will be sorted 

using a measure of socio-economic status. This is the median household income of the 5-digit ZIP 

Code area where the school is located, based on 2000 Population Census data.

Once the schools are sorted by location class, minority enrollment class, and achievement data (or 

household income), a systematic sample of schools will be selected using a random start. Schools 

will be sampled with probability proportional to size. The exact details of this process are described 

in the individual sampling specification memos.

2. Beta Sample 

The beta sample at grade 8 is a public school sample. This sample will be used to conduct the TEL 

pilot assessment. Although only a pilot test sample, this will be a nationally representative sample, 

with the exception that schools that are included in the grade 8 alpha sample with certainty will not 

be represented. This is because there is to be no overlap with the grade 8 alpha school sample.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 16 of 24

The first stage of sampling for the beta sample is the selection of a sample of geographic primary 

sampling units (PSUs). These will be selected using the same design as for other recent PSU-based

assessments. However, with the advent of county-level census and ACS data following the 2010 

Census, a new PSU frame is being created, and stratified. A total 67 PSUs will be selected, 

representing the U.S. (but not including Puerto Rico, or DODEA schools that are located outside 

the 50 states and D.C.). This PSU component is needed because of the operational complexities of 

administering the computer-based assessment. A select group of staff will be trained to administer 

those assessments. The school stratification of the beta sample within PSUs will be by type of 

location and median household income. Schools with more than 15 percent black or Hispanic 

students will be sampled at twice the rate of other schools, so as to increase the student sample sizes 

for these two groups.

3. Gamma Sample 

This is the public school sample at grade 12, used for the reading and mathematics assessments. It 

will consist of a single sample of schools covering both national- and state-level estimates.

As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in 

this sample, in the balance of the country outside the 13 participating states. This will be carried out 

at the school level. Each school with more than 15 percent Black and Hispanic students will be 

given twice the selection probability of other schools of comparable size. This means that while 

about 53 percent of the student population (including over 90 percent of the Black and Hispanic 

students) are in the oversampled schools, about 70 percent of the sample students will come from 

these schools.

Table 5 shows the sample size targets for each of the 13 state-specific samples. Note that the target 

student sample sizes vary slightly be state. These are target sampled, rather than assessed, and have 

been estimated based on experience from 2009. Of the 13 states, 11 participated at the state level in 

2009, and so ineligibility, nonresponse, and exclusion rates from 2009 have been used to derive 

targets for those states. For the other two states (Michigan and Tennessee), national rates from 2009 

were used. The goal in each case is to assess about 2,300 students in each of the two subjects 

(reading and mathematics).
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 17 of 24

Table 5. Grade 12 school and student frame counts, expected school sample sizes, and 

initial target student sample sizes for the 2013 state-by-state assessments (gamma 

samples)

 Grade 12 

Jurisdiction Schools in frame Schools in sample Students in frame 

Overall target 

student sample size 

Arkansas 297 101 32,035 6,200

Connecticut 245 110 41,607 6,750

Florida 965 116 176,821 6,600

Idaho 210 95 19,057 6,250

Illinois 954 125 149,998 7,250

Iowa 408 121 37,793 6,850

Massachusetts 371 107 67,923 6,250

Michigan 1032 136 126,382 7,400

New Hampshire 89 77 15,749 8,350

New Jersey 432 110 97,690 6,500

South Dakota 191 140 8,796 6,500

Tennessee 369 126 67,111 7,400

West Virginia 147 92 18,796 6,650

Stratification 

Unlike the other samples, the Gamma sample will have an explicit stratification at the highest level. 

The schools will be stratified by state, for the thirteen state-by-state participants, and the balance of 

the US, giving 14 explicit strata. In each of the thirteen states, stratification will be carried out in the 

same way as for the alpha samples. The last level sorting variable will be median household income, 

as there are no achievement data available at the school level for grade 12.

For the explicit stratum containing the balance of the US, the highest level of sorting will be by 

Census Division. This gives 8 implicit strata (the balance of New England is too small to be an 

effective stratum, and is combined with the balance of the Mid-Atlantic Division). The next stratifier 

in the hierarchy is type of location, which has twelve categories. Many of the type of location strata 

nested within Census divisions will be collapsed with neighboring type of location cells (this will 

occur if the expected school sample size within the cell is less than 4.0). These geographic strata will 

be subdivided using a dichotomous high minority status category. Schools are in the oversampling

stratum if they had more than 10 Black or Hispanic students and greater than 15 percent Black or 

Hispanic students. Otherwise the school will be put in a regular sampling stratum. If the expected 

sample size within these strata is less than 8.0, they will be left as is. If the expected sample size is 

greater than 8.0, then each of these two categories will be subdivided into up to four substrata (two 

for expected sample size up to 12.0, three for expected sample size up to 16.0, and four for expected 
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 18 of 24

sample size greater than 16.0). For the category of no oversampling, the subdivision will be by state 

or groups of contiguous states. For the oversampling strata, the subdivision will be by percentage

Black and Hispanic. Within these substrata, the schools are to be sorted by school type (public, BIE, 

DoDEA) and median household income from the 2000 Census (using a serpentine sort within the 

school type substrata).

High School Longitudinal Study (HSLS) Linking Study Sample 

The sample of students for the HSLS linking study will be obtained as follows. For the whole 

gamma sample, the NAEP school sample will be selected so as to maximize the overlap with the 

HSLS school sample. Then at the time of student sampling, any student who is in the HSLS sample, 

but is not selected for NAEP (the RM spiral), will be included in the study and assigned the ML 

spiral. This is expected to be about 6,000 students from about 300 schools, after accounting for 

schools that are not responding to the HSLS.

NAEP-PISA Linking Study Sample 

The sample of students for this study will be obtained as follows. For all gamma sample schools in 

Connecticut, Florida, and Massachusetts, samples of students will be selected from each of grades 9, 

10, and 11 (from among those grades offered by the school). The procedure for selecting the 

student samples will be similar to that used to select grade 12 students, and is described in the 

student sampling section below. The selected students will be assigned the MP spiral appropriate for 

their grade.

4. Delta Samples 

These are the private school samples at grades 4, 8, and 12 for conducting the operational 

assessments in reading and math, as well as the special studies that are incorporated into the RM 

spiral. The number of students targeted per schools will be 64 at grade 4, 65 at grade 8, and 60 at 

grade 12 – 30 for reading, 30 for math, with 4 at grade 4 and 5 at grade 8 for pilot tests and special 

studies. The sample sizes are large enough to report results by Catholic and non-Catholic at grades 4 

and 8, but at grade 12 only for private schools as a whole. Approximately half the sample at each 

grade will be from Catholic schools. This is very similar to the design used in 2009, at each grade.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 19 of 24

Stratification 

The private schools are to be explicitly stratified by private school type (Catholic/Other). Within 

each private school type, stratification will be by Census region (4 categories), type of location (12

categories), and enrollment size. In general, where there are few or no schools in a given stratum, 

categories will be collapsed together, always preserving the private school type.

5. Epsilon Sample 

This sample is analogous to the beta sample, but for private schools, at grade 8. The same PSUs will

be used as for the beta sample, and the TL session type will be conducted. There will be no 

oversampling of private schools for this sample. The epsilon sample schools will not overlap with 

the delta sample schools.

6. Rho Sample 

This sample will be used to conduct the grade 4 Reading Aloud Accommodations study. It will 

consist of about 70 schools from a small number of states, with the selected schools each having 

enough SD and ELL students so that nine students can be selected who are SD, nine who are ELL, 

and nine who are neither. These students will then be randomly assigned to one of three conditions, 

and assessed with a NAEP reading assessment. The three conditions are a) standard conditions; b) 

read aloud the test questions, but not the stimulus passage; c) read aloud all of the assessment 

material. These schools will not be included in any other NAEP 2013 sample, at any grade.

IV. New Schools 

To compensate for the fact that files used to create the NAEP school sampling frames are at least 

two years out of date at the time of frame construction, we will supplement the Alpha, Gamma, and 

Delta sample with new school samples at each grade. 

The new school samples will be drawn using a two-stage design. At the first stage, a minimum of ten 

school districts (in states with at least ten districts) will be selected from each state for public 
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 20 of 24

schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled 

districts and dioceses will be asked to review lists of their respective schools and identify new 

schools. Frames of new schools will be constructed from these updates, and new schools will be 

drawn with probability proportional to size using the same sample rates as their corresponding 

original school samples.

The school sample sizes in the above tables do not reflect new school samples.

V. Substitute Samples 

Substitute samples will be selected for each of the Beta, Gamma, Delta, and Epsilon samples. The 

substitute school for each original will be the next “available” school on the sorted sampling frame, 

with the following exceptions:

n Schools selected for any NAEP samples will not be used as substitutes.

n Private schools whose school affiliation is unknown will not be used as substitutes. 

Also, unknown affiliated private schools in the original samples will not get substitutes.

n A school can be a substitute for one and only one sample. (If a school is selected as a 

substitute school for grade 12, for example, it cannot be used as a substitute for either 

grade 4 or grade 8.)

n A public school substitute will always be in the same state as its original school. 

VI. Contingency Samples 

The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at 

some point over the next few months, a given district might choose to opt out of the TUDA 

program for 2013. However, it is not acceptable for all schools in such a district to decline NAEP, 

as then the state estimates will be adversely affected. Thus to deal with this possibility, in each 

TUDA district, subsamples of the alpha sample schools will be identified as contingency samples. In 

the event that the district withdraws from the TUDA program prior the selection of the student 

sample, all alpha sampled schools from that district will be dropped from the sample, with the 

exception of those selected in the contingency sample. The contingency sample will provide a 

proportional representation of the district, within the aggregate state sample. Student sampling in 

those schools will then proceed in the same way as for the other schools within the same state.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 21 of 24

Similarly, in the thirteen states participating at grade 12, in the gamma sample contingency samples 

will be identified in those states. If the state decides to withdraw from the state-level assessment, the 

contingency sample schools will be retained, as part of the grade 12 national sample.

VII. Student Sampling 

Students within the sampled schools will be selected with equal probability. The student sampling 

parameters vary by sample type (Alpha, Beta, Gamma, Delta, Epsilon, and Rho) and grade, as 

described below.

Alpha Sample, Grade 4 Schools (Except Puerto Rico) 

n All students, up to 70, will be selected.

n If the school has between 71 and 90 students (inclusive), the school will be asked if it 

wishes to have all students selected, or a sample of 63. If the school elects to have all 

students selected, all students will be selected. If the school asks for a sample of 63

students, a systematic sample of 63 students will be selected.

n If the school has more than 90 students, a systematic sample of 63 students will be 

selected.

n In some schools, the school may be assigned more than one ‘hit’ in sampling. In these 

schools we will select a sample of size 63 times the number of hits, taking all students if 

this target is greater than or equal to 63/70 of the total grade 4 enrolment.

n All students will be assigned to assessment type RM.

Alpha Sample, Grade 8 Schools (Except Puerto Rico) 

n All students, up to 70, will be selected.

n If the school has more than 70 students, a systematic sample of 63 students will be 

selected.

n In some schools, the school may be assigned more than one ‘hit’ in sampling. In these 

schools we will select a sample of size 63 times the number of hits, taking all students if 

this target is greater than or equal to 63/70 of the total grade 8 enrolment.

n All students will be assigned to assessment type RM.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 22 of 24

Alpha Sample, Puerto Rico Grades 4 and 8 

n All students, up to 55, will be selected.

n If the school has more than 55 students, a systematic sample of 50 students will be 

selected with no oversampling.

n All students will be assigned to assessment type PR.

Delta Samples, Grades 4 and 8 

n All students, up to 72, will be selected.

n If the school has more than 72 students, a systematic sample of 64 students (grade 4), or 

65 students (grade 8) will be selected.

n All students will be assigned to assessment type RM.

Beta and Epsilon Samples, Grade 8 

n In each school, a sample for computer-based testing will be selected as follows: All 

students up to 30 will be selected. If there are more than 30 students enrolled, a sample 

of 30 students will be selected.

n All students will be assigned to the TL session.

Gamma and Delta Samples, Grade 12 

n All students, up to 66, will be selected.

n If the school has more than 66 students, a systematic sample of 60 students will be 

selected.

n In some schools, the school may be assigned more than one ‘hit’ in sampling. In these 

schools we will select a sample of size 60 times the number of hits, taking all students if 

this target is greater than or equal to 60/66 of the total grade enrollment.

n All students so selected will be assigned to assessment type RM.

n For the gamma sample, any grade 12 HSLS students not selected in the above steps will 

be added to the sample and assigned assessment type ML.

n For the gamma sample, in Connecticut, Florida, and Massachusetts, in each school a 

sample of students from grades 9, 10, and 11 will be selected. For grades 10 and 11, the 

sample will be drawn as for grade 12, but using 30 in place of 60, and 33 in place of 66,
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 23 of 24

in the above algorithm. For grade 9, the same procedure will be used, but using figures 

of 18 and 20. These students will be assigned assessment type MP.

Rho Sample, Grade 4 

Details of the student sampling procedures for the grade 4 Reading Aloud Accommodations study 

have yet to be finalized. However, they will be something like the following. On the basis of e-filed 

student data for all grade 4 students in the school, students will be classified into four groups: SD 

only, ELL only, neither SD nor ELL, and Other (including students who are both SD and ELL, and 

students for whom complete data on SD and ELL status is not available). A random sample of nine

students will be selected from each of the first three of these groups. The students from each group 

will be randomly assigned in equal numbers to each of the three assessment conditions.

VIII. Weighting Requirements 

Weighting activities for the 2013 NAEP assessments are not covered under the current NAEP 

contracts. However, based on past experience, below is a description of the likely weighting 

requirements.

The Operational Samples 

These samples will have a single set of weights for each subject (reading and math at grades 4, 8, and 

12) applied to reflect probabilities of selection, school and student nonresponse, any trimming, and 

the random assignment to the particular subject. There will be a separate replication schemes by 

grade and public/private. Such weights will also be derived for the Puerto Rico KaSA assessment.

The Pilot Test and Special Study Samples 

Fully adjusted weights will not be provided for the students in the pilot test studies and special 

studies. However, with the exception of the grade 4 Reading Aloud Accommodations study,

preliminary weights will be available for these samples. These will reflect the school and student 

selection probabilities, but with no adjustments for nonresponse.
Memorandum: 2013- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24, 2012

Page 24 of 24

The HSLS Linking Sample 

Weights may be required for the linking sample, which account for the sampling procedures and 

nonresponse for both the HSLS and the NAEP assessment. Clearly this will require collaboration 

between the next NAEP contractor responsible for weighting, and RTI International, the contractor 

for the HSLS study. Once the analysis plan for this study has been specified, a weighting plan can be 

developed.