NAEP 2014 -2 016 System Clearance: Appendix C page C-1 

# N ATIONAL 

# C ENTER FOR 

# E DUCATION 

# S TATISTICS



# N ATIONAL 

# A SSESSMENT OF 

# E DUCATIONAL 

# P ROGRESS



# Appendix C

Example of Sample D esign D ocument (2013 Assessment )

Request for System Clearance for

NAEP Assessments for 2014-2016

OMB# 1850

- 0790

### Date: May 24 , 201 2 Memo : 2013

-

1.3 A/ 1.3 B/ 1.3 G/ 1. 3 D /1. E /1.3 R

To: William Ward, NCES

Nicole Beaulieu, ETS

Ed Kulick, ETS

David Freund, ETS

Amy Dresher, ETS

Connie Smith, Pearson

Dianne Walsh

Lauren Byrne

Debby Vivari

Rob Dymowski

William Wall

John Burke Lloyd Hicks

Jennifer Kali

David Ferraro

Erin Wiley

Dwight Brock

Amy Lin

Chris Averett

Kavemuii Murangi

David Hubble

Sarah Shore

From: Keith Rust

Subject: Sample Design for 2013 NAEP



# I. Introduction

For 2013 , the NAEP assessment involves the following components:

n 

### National assessments in reading and math at grades 4, 8 , and 12 ;

n 

### State

- by

- state and Trial Urban District Assessment (TUDA) assessments in reading and

math for public schools at grades 4 and 8;

n 

### State

- by

- state assessments in reading and math for 13 selected states at grade 12;

n 

### A special study to link NAEP and the High School Longitudinal Study (HSLS), for mathematics at grade 12 ;

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 2 of 24 n 

### A special study to link NAEP a nd the Lexile assessment, for reading at grade 8;

n 

### A special study of Knowledge and Skills Appropriate (KaSA) items in mathematics , at grades 4 and 8, including Puerto Rico ;

n 

### A special study of Accessible Booklets, for reading at grades 4 and 8;

n 

### A special s tudy of Read

- Aloud A ccommodations, for reading at grade 4 ;

n 

### A special study to link NAEP and PISA, for mathematics , in certain schools selected for grade 12 NAEP ;

n 

### Pilot tests in reading and math at grades 4 and 8 ;

n 

### A pilot te st of the Technology and Engineer ing Literacy (TEL) Assessment at grade 8.

Below is a summary list of the features of the 2013 sample design.

1. As in recent NAEP studies, Trial Urban District Assessment (TUDA) samples will form part of the corresponding state samples, and the state samples will form part of the national sample. The same twenty

- one Trial Urban District Assessment ( TUDA ) participants as in 2011 will be in volved

.

2. At grade 8 there will be two public school samples, and two private school samples. A

r elatively s mall PSU

- based sa mple , known as the beta sample (for public schools) and the epsilon sample (for private schools), will be used for the computer

- based TEL pilot

assessment . The alpha sample s for grades 4 and 8 public

, the gamma sample for grade 12 public, and the delta sam ples for privat e schools at grades 4, 8, and 12 will be used for the ope rational assessments in reading and mathematics, and for the pilot test samples for reading and mathematics, and all special studies except for the Read

- Aloud Accommodations Study

.

3. Oversampling of private schools at grades 4 and 8 will be scaled back to the level of

2009

. Response rates permitting, this will allow separate reporting, for reading and
mathematics, for Catholic, and non

- Catholic schools, but not further breakdown

. As in

2009 t here will be no oversampling of private schools at grade 12.

4. There will be no samples in territories, other than for Puerto Rico at grades 4 and 8 for the KaSA study

.

5. For the first time since 2003, there is no National Indian Education Study (NIES). This means that less extensive sampling of BIE schools is required. To ensure sound results
for AIAN students at the national level, at grades 4 and 8 BIE students will be sampled at the same rate as students in Oklahoma, the state with the largest AIAN p opulation.

6. As in 2011 , the Department of Defense Schools are expected to be reported as a single jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 3 of 24 overseas (DoDDS). However, for design purposes, we will still sample these as two

s eparate entities

.

7. At grade 12, there will state

- level samples for 13 states. Eleven of these also participated at the state level in 2009. As at grades 4 and 8, state and national samples will be integrated. The 13 states participating at the state level f or reading and mathematics are Arkansas, Connecticut, Florida, Idaho, Illinois, Iowa, Massachusetts, Michigan, New Hampshire, New Jersey, South Dakota, Tennessee, and West Virginia. The NAEP

PISA linking study will be conducted in the grade 12 sample schoo ls in Connecticut,
Florida, and Massachusetts. These three states will conduct state

- level PISA in the fall

of 2012.

8. The Read

- Aloud Accommodations study will be conducted in special sample of grade 4 public schools, to be referred to as the rho sample. Thi s sample will be restricted to a small number of states, and will not have any overlap schools with any of the other

2013 NAEP asses sments. The sample will not be a nationally representative probability sample of schools.

9. The sample sizes of assessed stude nts for these various components are shown in Table 1 (which also shows the approximate numbers of participating schools).

Note that the figures in Table 1 (and also Table s 3 and 4 ) do not reflect the fact that, at grade 4,

certain schools in the pencil and paper assessments may elect to include all students in the sample,

rather than the sample size targeted by NAEP. This will lead to an increase is sample size, likely to

be about five percent overall. The increase will vary by state, depending on the size of the state and

the size of its schools

– obviously there is no addition in states where the design calls for the

selection of all students

.

Table 1. Target sample sizes of asse ssed students, and expected number of participating

schools, for 2013 NAEP

Spiral Jurisdictions Students

Total Spiral

Indic. States (incl.

DC, BIE,
DoDEA) Urban

districts Public

school

students Private

school

students

Grade 4

nat'l/state math RM 5 3 2 1 163,0 00 3 ,000 166,0 00

nat'l/state reading RM 53 2 1 163,0 00 3 ,000 166,0 00

math Pilot RM 4 4 10 90 4500

reading Pilot RM 5 1 45 10 5 5250

math KaSA RM 3 92 0 80 4000

Reading accessible booklet study RM 294 0 6 0 3 000

Puerto Rico KaSA PR 1 5000 0 5000

Total

- alpha 2 34 7,41 5

Total

- delta 1 6,33 5

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 4 of 24 Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2013 NAEP ( Continued )

Spiral Jurisdictions Students

Total Spiral

Indic. States (incl.

DC, BIE,
DoDEA) Urban

districts Public

school

students Private

school

students

Grade 4 ( Continued )

Typical max. no. students/school 6 0 6 4

Average assessed

students/school 4 4 24

Total schools

- alpha, delta 7, 900 26 5 8,1 83

Reading Aloud Study RA, RB,

RC 2,1 00 2,1 00

Total

- Rho 3 2,100

Typical max. no. students/school 27

Average assessed

students/school 27

Total schools

- rho 7 8 0 7 8

Total number of s tudents grade 4 349,51 5 6,33 5 355 ,850

Total number of s chools grade 4 7,970 265 8,2 43

Grade 8

nat'l/state math RM 53 21 159,08 0 3 ,000 162,080

nat'l/state reading RM 53 21 159,08 0 3 ,000 162,080

math Pilot RM 4

, 410 90 4

, 500

reading Pilot RM 5

, 145 105 5

, 250

math KaSA RM 3

, 920 80 4

, 000

Reading accessible booklet study RM 2

, 94 0 6 0 3 ,000

Lexile Linking Study RM 7,840 160 8 ,000

Puerto Rico KaSA PR 1 5,000 0 5,000

Total

- alpha 2 347, 41 5

Total

- delta 1 6, 495

Typical max. no. students/school 60 6 5

Average assessed

students/school 50 25

Total schools

- alpha, delta 6,9 50 260 7,2 10

TEL Pilot TL 13,500 1,500 15,000

Total

- beta 1 13,500

Total

- epsilon 1 1,500

Typical max. no. students/school 30 30

Average assessed

students/school 25 25

Total schools

- beta, epsilon 540 60 600

Total number of schools grade 8 7, 410 320 7,8 10

Total number of students grade 8 360,91 5 7,995 368 ,910

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 5 of 24 Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2013 NAEP ( Continued )

Spiral Jurisdictions Students

Total Spiral

Indic. States (incl.

DC, BIE,
DoDEA) Urban

districts Public

school

students Private

school

students

Grade 12

national and state reading RM 13 41,120 1,500 42,620

National and state math RM 13 38,380 1,400 39,780

HSLS Linking Study ML 6 ,000 6 ,000

NAEP

- PISA Linking Study MP 3 18,000 18,000

Total

- gamma 3 103 ,500

Total

- delta 1 2,900

Typical max. no. students/school
(excluding HSLS) 60 60

Average assessed

students/school 50 40

Total schools

- gamma, delta 1,690 73 1, 763

GRAND TOTAL STUDENTS 813 ,930 17,230 8 31 ,160

GRAND TOTAL SCHOOLS 17,070 658 17,7 36



# II. Assessment Types

The assessment spiral types are shown in Table 2. At grade 4 there will be five different spirals use d,

with three at grade 8 and three at grade 12. Session IDs contain six digits, traditionally. The first two

digits identify the assessment “type” (subjects and type of spiral in a general way). G rade is

contained in the second pair of digits, and the session sequential number (within schools) in the last

two digits. For example, session RM 0401 denotes the first grade 4 reading and mathematics

operational assessment in a given school.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 6 of 24 Table 2. NAEP 2013 assessment types and IDs

ID Type Subjects Grades Schools Comments

RM Operational

Pencil & Paper Reading (including pilot

,

accessible booklets )

, math

(including pilot, KaSA) 4 Public and

Private Includes all grade 4 assessment

components, except for Puerto

Rico and Reading Aloud

RM Operational

– Pencil & Paper Reading (including pilot,

accessible booklets, Lexile

linking)

, math (including
pilot and KaSA) 8 Public and

Private Includes all grade 8

components except Puerto Rico

and TEL pilot

R M Operational

– Pencil & Paper Reading, math 12 Public and

private

TL Pilot

Computer

based Technology and

Engineering Literacy 8 Public and

Private National sample , not in Puerto

Rico

ML Special Study

- Pencil & Paper HSLS Linking Study 12 Public

MP Special Study

– Pencil &

Paper PISA Linking Study 9, 10,

11 Public In three states only

RA Special Study

– Pencil &

Paper Reading Aloud Study 4 Public In a small number of states; no

special accommodations

RB Special Study

– Pencil &

Paper Reading Aloud Study 4 Public In a small number of states;

Read Aloud Items

RC Special Study

– Pencil &

Paper Reading Aloud Study 4 Public In a small number of states;

Read Aloud Everything

PR Puerto Rico KaSA math 4 & 8 Public Booklets will not be used in any

other jurisdiction.



# III. Sample Types and Sizes

In similar fashion to past years (but somewhat different ) , we will identify six different types of

school samples : Alpha, Beta, Gamma, Delta, Epsilon , and Rho . These distinguish sets of schools

that will be conducting distinct portions of the assessment

. The Rho sample is new for 201 3; this

will be used for the g rade 4 Read

- Aloud Accommodations s tudy which has a unique design.



## 1. Alpha Samples at Grades 4 and 8

These are public school samples for grades 4 a nd 8. They will be used for the operational state

- b y

state assessments in reading and math , and contribute to the national samples for these subjects as

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 7 of 24 well. The will also be used for pilot tests and special studies, and outlined in Tables 1 and 2. There

will be alpha samples for each state, DC, DoDEA, BIE, and Puerto Rico.

The details of the target student sample sizes for the alpha samples are as follows:

n 

### With the exception of BIE, the target sample size for each state and grade is 6,600

students. For BIE, the sample will be selected at the same rate as Oklahoma. This is to ensure adequate samples of American Indian students for national reporting. The rate from Oklahoma is being used since it has the largest American Indian population of any

state. The refore using this rate ensures that the BIE sample will not be so small as to unduly inflate the standard errors of estimates for American Indian/Alaska Native students, while at the same time not placing undue burden on BIE schools, for which

no jurisdiction level results are to be reported, in the absence of the NIES.

n 

### There will be samples for twenty

- one TUDA districts. For the five largest (New York City, Los An geles, Chicago, Miami

- Dade, and Houston ), the student target sample sizes for reading and mathematics are three

- quarters the size of a state sample ( 2,475 per subject ) . For the other fifteen districts the student target sample sizes for reading and

math are one

- half the size of a state sample ( 1,650 per subject )

. An exception will be

made in the case of Detroit, where the target sample sizes will be increased in anticipation of substantial numbers of schools closings where the particular schools affected have not yet been identified.

Note

, that above

, there is a conflict between sample size requirements at the state level, and the

TUDA district level. This will be resolved as in previous years: t he districts will have t he target

samples indicated in B , an d reflected in Table 3 . For the states that contain one or more of these

districts, the target sample size indicated in A (and shown in Table 3 ) will be used to determine a

school sampling rate for the state, which will be applied to the balance of the state outside the

TUDA district(s). Thus the target studen t sample sizes, shown in Table 3

, for states that contain a

TUDA district, are only ‘design targets ’ , and are smaller than the final total sample size for the state,

but larger than the sample for the balance of the state, exclusive of its TUDA districts

. In the case of

the District of Co lumbia, the state sample size requirement is that all schools and students be

included. This renders moot any requirements for the DC TUDA sample, which by default consists

of all schools operated by the DCPS district (but excludes charter schools in DC, e ven though those

are all included in the state sample, as these are not operated by DCPS).

In Puerto Rico, the target sample size is 5, 800 per grade (grades 4 and 8), with the goal of assessing

5,000 students

. Only KaSA mathematics will be assessed in Pue rto Rico (PR session type).

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 8 of 24 As in past state

- by

- state assessments, schools with fewer than 20 students in the grade in question

will be sampled at a moderately lower rate than other schools (at least half, and often higher,

depending upon the size of the school). This is in implicit recognition of the greater cost and burden

associated with surveying these schools.

Table 3 shows the target student sample sizes, and the approximate counts of schools to be selected

in the alpha samples, along with the schoo l and student frame counts, by state and TUDA districts

for grades 4 and 8. The table also identifies the jurisdictions where we take all schools and where we

take all students.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3R May 24 , 2012

Page 9 of 24 Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (alpha samples)

Grade 4 Grade 8

Jurisdiction Schools in

frame Schools in

sample Students

in frame Overall

target

student
sample

size Schools in

frame Schools in

sample Students in

frame Overall

target

student
sample size

Alabama 748 117 59,269 6,600 484 112 57,283 6,600

Alaska 365 197 9,827 6,600 285 144 9,701 6,600

Arizona 1,179 120 83,555 6,600 769 118 81,283 6,600

Arkansas 496 119 37,017 6,600 315 112 35,831 6,600

Bureau Of Indian Education 135 16 3,246 411 109 13 2,785 375

California 5,846 113 463,973 6,600 2,818 113 456,125 6,600

Colorado 1,013 122 63,266 6,600 520 116 59,357 6,600

Connecticut 604 116 41,489 6,600 306 110 42,667 6,600

Delaware 115 95 9,647 6,600 60 60 9,398 6,600 *

District Of Columbia 131 131 4,907 4,907 ** 79 79 4,307 4,307 **

DoDEA Schools 108 108 7,507 7,507 ** 63 63 5,589 5,589 **

Florida 2,097 110 202,702 6,600 1,150 112 196,792 6,600

Georgia 1,242 107 130,245 6,600 548 106 124,307 6,600

Hawaii 202 117 14,155 6,600 80 60 12,843 6,600

Idaho 368 128 21,351 6,600 200 103 20,466 6,600

Illinois 2,343 123 154,495 6,600 1,603 119 154,308 6,600

Indiana 1,101 116 80,245 6,600 485 107 79,856 6,600

Iowa 674 137 35,421 6,600 387 118 34,991 6,600

Kansas 729 141 35,907 6,600 412 126 34,645 6,600

Kentucky 731 117 51,685 6,600 391 114 49,403 6,600

Louisiana 789 120 55,300 6,600 531 119 50,584 6,600

Maine 341 159 13,945 6,600 211 115 14,205 6,600

Maryland 887 115 61,870 6,600 356 109 60,987 6,600

Massachusetts 973 117 71,298 6,600 482 108 71,962 6,600

Michigan 1,856 121 118,242 6,600 1,042 114 120,377 6,600

Minnesota 952 126 61,086 6,600 695 125 61,246 6,600

Mississippi 436 114 38,958 6,600 290 108 36,999 6,600

Missouri 1,170 131 68,189 6,600 727 124 68,007 6,600

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 10 of 24 Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (alpha samples) ( Continued )

Grade 4 Grade 8

Jurisdiction Schools in

frame Schools in

sample Students

in frame Overall

target

student
sample

size Schools in

frame Schools in

sample Students in

frame Overall

target

student
sample size

Montana 400 199 10,863 6,600 289 151 10,912 6,600

Nebraska 568 165 22,085 6,600 339 131 21,503 6,600

Nevada 379 112 33,851 6,600 158 88 33,028 6,600

New Hampshire 265 133 14,495 6,600 137 88 15,191 6,600

New Jersey 1,366 117 100,453 6,600 741 110 99,535 6,600

New Mexico 431 127 25,731 6,600 203 105 24,183 6,600

New York 2,368 113 192,295 6,600 1,335 109 193,235 6,600

North Carolina 1,414 112 118,129 6,600 694 110 111,343 6,600

North Dakota 260 260 6,995 6,995 ** 188 188 7,330 7,330 **

Ohio 1,877 119 133,358 6,600 1,090 112 132,686 6,600

Oklahoma 895 137 49,300 6,600 593 128 46,433 6,600

Oregon 767 129 42,827 6,600 414 120 42,949 6,600

Pennsylvania 1,742 116 129,550 6,600 914 109 132,799 6,600

Puerto Rico 1,017 160 38,842 5,800 407 123 37,363 5,800

Rhode Island 170 117 10,437 6,600 57 57 10,842 6,600 *

South Carolina 618 112 55,228 6,600 298 107 52,433 6,600

South Dakota 328 194 9,380 6,600 254 153 9,306 6,600

Tennessee 998 116 75,934 6,600 565 112 71,570 6,600

Texas 4,277 112 373,404 6,600 2,165 112 348,907 6,600

Utah 582 112 46,508 6,600 232 106 42,593 6,600

Vermont 224 224 6,419 6,419 ** 122 122 6,364 6,364 **

Virginia 1,137 112 93,610 6,600 388 106 92,179 6,600

Washington 1,214 119 77,826 6,600 606 115 77,099 6,600

West Virginia 425 146 20,875 6,600 202 109 20,637 6,600

Wisconsin 1,111 131 60,997 6,600 633 118 61,063 6,600

Wyoming 188 188 6,849 6,849 ** 93 93 6,568 6,568 **

Albuquerque 99 56 7,594 3,300 42 42 6,903 3,300 *

Atlanta 61 61 4,264 4,264 ** 26 26 3,369 3,369 **

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 11 of 24 Table 3. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (alpha samples) ( Continued )

Grade 4 Grade 8

Jurisdiction Schools in

frame Schools in

sample Students

in frame Overall

target

student
sample

size Schools in

frame Schools in

sample Students in

frame Overall

target

student
sample size

Austin 80 54 6,862 3,300 24 24 5,360 3,300 *

Baltimore City 124 68 6,292 3,300 93 62 5,501 3,300

Boston 79 79 4,106 4,106 ** 38 38 3,808 3,808 **

Charlotte 105 54 11,245 3,300 38 38 9,852 3,300 *

Chicago 488 96 29,942 4,950 470 95 29,469 4,950

Cleveland 83 83 3,573 3,573 ** 81 81 3,550 3,550 **

Dallas 147 53 12,932 3,300 38 38 10,113 3,300 *

Detroit 121 81 7,584 4,470 74 65 5,001 4,240

Fresno 70 54 5,823 3,300 26 26 5,457 3,300 *

Hillsborough County 164 55 15,161 3,300 80 48 15,020 3,300

Houston 179 82 16,525 4,950 64 49 12,738 4,950

Jefferson County 96 54 7,617 3,300 42 42 7,030 3,300 *

Los Angeles 516 82 51,516 4,950 162 77 46,818 4,950

Miami

- Dade 274 83 26,903 4,950 160 76 25,573 4,950

Milwaukee 115 67 5,807 3,300 89 56 5,371 3,300

New York City 709 82 63,731 4,950 458 82 61,278 4,950

Philadelphia 177 60 12,098 3,300 142 58 10,970 3,300

San Diego 138 57 10,040 3,300 63 40 9,691 3,300

District of Columbia PS 87 87 3

, 369 3

, 369 ** 37 37 2

, 357 2

, 357 **

Counts for states do not reflect the oversampling for their constituent TUDA districts, nor the grade 4 ‘take all’ option

.

Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are incl uded in the NAEP sample.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 12 of 24 Table 4 consolidates the target student (and resulting school) sample size numbers, to show the total

target sample sizes in each state, combining the TUDA targets with those for the balance of the

state.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 13 of 24 Table 4. Total sample s izes, combining s tate and TUDA samples

Grade 4 Grade 8

Jurisdiction Schools

in frame Schools

in sample Students

in frame Overall

target

student
sample

size Schools in

frame Schools in

sample Students in

frame Overall

target

student
sample size

Alabama 748 117 59,269 6,600 484 112 57,283 6,600

Alaska 365 197 9,827 6,600 285 144 9,701 6,600

Arizona 1,179 120 83,555 6,6 00 769 118 81,283 6,6 00

Arkansas 496 119 37,017 6,600 315 112 35,831 6,600

Bureau Of Indian Education 135 16 3,246 411 109 13 2,785 375

California 5,846 290 463,973 17,189 2,818 241 456,125 17,251

Colorado 1,013 122 63,266 6,600 520 116 59,357 6,600

Connecticut 604 116 41,489 6,600 306 110 42,667 6,600

Delaware 115 95 9,647 6,600 60 60 9,398 6,600 *

District Of Columbia 131 131 4,907 4,907 ** 79 79 4,307 4,307 **

DoDEA Schools 108 108 7,507 7,507 ** 63 63 5,589 5,589 **

Florida 2,097 225 202,702 13,479 1,150 213 196,792 13,486

Georgia 1,242 164 130,245 10,648 548 129 124,307 9,790

Hawaii 202 117 14,155 6,600 80 60 12,843 6,600

Idaho 368 128 21,351 6,600 200 103 20,466 6,600

Illinois 2,343 194 154,495 10,267 1,603 190 154,308 10,285

Indiana 1,101 116 80,245 6,600 485 107 79,856 6,600

Iowa 674 137 35,421 6,600 387 118 34,991 6,600

Kansas 729 141 35,907 6,600 412 126 34,645 6,600

Kentucky 731 156 51,685 8,929 391 139 49,403 8,963

Louisiana 789 120 55,300 6,600 531 119 50,584 6,600

Maine 341 159 13,945 6,600 211 115 14,205 6,600

Maryland 887 170 61,870 9,229 356 160 60,987 9,305

Massachusetts 973 188 71,298 10,326 482 140 71,962 10,059

Michigan 1,856 194 118,242 10,646 1,042 174 120,377 10,566

Minnesota 952 126 61,086 6,600 695 125 61,246 6,600

Mississippi 436 114 38,958 6,600 290 108 36,999 6,600

Missouri 1,170 131 68,189 6,600 727 124 68,007 6,600

Montana 400 199 10,863 6,600 289 151 10,912 6,600

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 14 of 24 Table 4. Total sample s izes, combining s tate and TUDA samples ( Continued )

Grade 4 Grade 8

Jurisdiction Schools

in frame Schools

in sample Students

in frame Overall

target

student
sample

size Schools in

frame Schools in

sample Students in

frame Overall

target

student
sample size

Nebraska 568 165 22,085 6,600 339 131 21,503 6,600

Nevada 379 112 33,851 6,600 158 88 33,028 6,600

New Hampshire 265 133 14,495 6,600 137 88 15,191 6,600

New Jersey 1,366 117 100,453 6,600 741 110 99,535 6,600

New Mexico 431 149 25,731 7,938 203 119 24,183 8,008

New York 2,368 159 192,295 9,361 1,335 156 193,235 9,456

North Carolina 1,414 156 118,129 9,271 694 138 111,343 9,315

North Dakota 260 260 6,995 6,995 ** 188 188 7,330 7,330 **

Ohio 1,877 197 133,358 9,996 1,090 189 132,686 9,973

Oklahoma 895 137 49,300 6,600 593 128 46,433 6,600

Oregon 767 129 42,827 6,600 414 120 42,949 6,600

Pennsylvania 1,742 165 129,550 9,283 914 156 132,799 9,355

Puerto Rico 1,017 160 38,842 5,800 407 123 37,363 5,800

Rhode Island 170 117 10,437 6,600 57 57 10,842 6,600 *

South Carolina 618 112 55,228 6,600 298 107 52,433 6,600

South Dakota 328 194 9,380 6,600 254 153 9,306 6,600

Tennessee 998 116 75,934 6,600 565 112 71,570 6,600

Texas 4,277 293 373,404 17,665 2,165 216 348,907 17,792

Utah 582 112 46,508 6,600 232 106 42,593 6,600

Vermont 224 224 6,419 6,419 ** 122 122 6,364 6,364 **

Virginia 1,137 112 93,610 6,600 388 106 92,179 6,600

Washington 1,214 119 77,826 6,600 606 115 77,099 6,600

West Virginia 425 146 20,875 6,600 202 109 20,637 6,600

Wisconsin 1,111 185 60,997 9,270 633 163 61,063 9,321

Wyoming 188 188 6,849 6,849 ** 93 93 6,568 6,568 **

Sample sizes for each state do reflect the samples in the TUDA districts within the state , but not the grade 4 take

- all option

.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are included in the NAEP sample

.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 15 of 24 Stratification

Each state and grade will be stratified separately, but using a common approach in all cases. TUDA

districts will be separated from their state, and each part stratified separately. The first level of

stratification will be based on urban

- centered type of location. This variable has 12 levels (some of

which may not be present in a given state or TUDA district), and these will be collapsed so that each

of the resulting location categories contains at least nine percent of the student population. Within

each of the resulting location categories, schools will be assigned a minority enrollment status. This

is based on the two race/ethnic groups that are the second and third most prevalent within the

location category. If these groups are both low in percentage t erms, no minority classification will be

used. Otherwise three (or occasionally four) equal

- sized groups (generally high, medium, and low

minority) will be formed based on the distribution across schools of the two minority groups.

Within the resulting lo cation and minority group classes (of which there are likely to be from three

to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school

level results from the most recent available state achievement tests at the relevant grade. In general,

mathematics test results will be used, but where these are not available, reading results will be used.

In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to

match the results to t he NAEP school frame), instead of achievement data, schools will be sorted

using a measure of socio

- economic status. This is the median household income of the 5

- digit ZIP

Code area where the school is located, based on 2000 Population Census data.

Once t he schools are sorted by location class, minority enrollment class, and achievement data (or

household income), a systematic sample of schools will be selected using a random start. Schools

will be sampled with probability proportional to size. The exact d etails of this process are described

in the individual sampling specification memos.



## 2. Beta Sample

The beta sample at grade 8 is a public school sample

. This sample will be used to conduct the TEL

pilot assessment. Although only a pilot test sample, thi s will be a nationally representative sample

, with the exception that schools that are included in the grade 8 alpha sample with certainty will not

be represented

. This is because there is to be no overlap with the grade 8 alpha school sample.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 16 of 24 The first s tage of sampling for th e beta sample is the selection of a sample of geographic primary

sampling units (PSUs). These will be selected using the same design as for other recent PSU

- based

assessment s. However, with the advent of county

- level census and ACS d ata following the 2010

Census, a new PSU frame is being created, and stratified . A total 67 PSUs will be selected,

representing the U.S. (but not including Puerto Rico, or DODEA schools that are located outside

the 50 states and D.C.). This PSU component i s needed because of the operational complexit ies of

administerin g the computer

- based assessment . A select group of staff w ill be trained to administer

those assessments

. The school stratification of the beta sample within PSUs will be by type of

location a nd median household income. Schools with more than 15 percent black or Hispanic

students will be sampled at twice the rate of other schools, so as to increase the student sample sizes

for these two groups.



## 3. Gamma S ample

This is the public school sample at grade 12, used for the reading and mathematics assessments. It

will consist of a single sample of schools covering both national

- and state

- level estimates

.

As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in

this sample , in the balance of the country outside the 13 participating states . This will be carried out

at the school level. Each school with more than 15 percent Black and Hispanic students will be

given twice the selection probability of other school s of comparable size. This means that while

about 53 percent of the student population (including over 90 percent of the Black and Hispanic

students) are in the oversampled schools, about 70 percent of the sample student s will come from

these schools.

Table 5 shows the sample size targets for each of the 13 state

- specific samples. Note that the target

student sample sizes vary slightly be state. These are target sampled, rather than assessed, and have

been estimated based on experience from 2009. Of the 13 states, 11 participated at the state level in

2009, and so ineligibility, nonresponse, and exclusion rates from 2009 have been used to derive

targets for those states . For the other two states (Michigan and Tennessee), national rates from 2009

were use d. The goal in each case is to assess about 2,300 students in each of the two subjects

(reading and mathematics).

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 17 of 24 Table 5

. Grade 12 school and student frame counts, expected school sample sizes, and

initial target student sample sizes for the 2013 state

- b y

- state assessments ( gamma samples)

Grade 12

Jurisdiction Schools in frame Schools in sample Students in frame Overall target

student sample size

Arkansas 297 101 32,035 6,200

Connecticut 245 110 41,607 6,750

Florida 965 116 176,821 6,600

Idaho 210 95 19,057 6,250

Illinois 954 125 149,998 7

, 250

Iowa 408 121 37,793 6,850

Massachusetts 371 107 67,923 6,250

Michigan 1032 136 126,382 7,400

New Hampshire 89 77 15,749 8,350

New Jersey 432 110 97,690 6,500

South Dakota 191 140 8,796 6,500

Tennessee 369 126 67,111 7,400

West Virginia 147 92 18,796 6,650



### Stratification

Unlike the other samples, the Gamma sample will have an explicit stratification at the highest level.

The schools will be stratified by state, for the thirteen state

- by

- state participants, and the balance of

the US, giving 14 explicit strata. In each of the thirteen states, stratification will be carried out in the

same way as for the alpha samples. The last level sorting variable will be median household income

, as there are no achievement data available at the school level for grade 12.

For the explicit stratum containing the balance of the US, the highest level of sorting will be by

Census Division. This gives 8 implicit strata (the balance of New England is too small to be an

effective stratum, and is combined with the balance of the Mid

- Atlantic Division). The next stratifier

in the hierarchy is type of location, which has twelve categories. Many of the type of location strata

nested within Census divisions will be collapsed with neighboring type of location cells (this will

occur if the expected school sample size within the cell is less than 4.0). These geographic strata will

be subdivided using a dichotomous high minority status category. Schools are in th e oversampling

stratum if they had more than 10 Black or Hispanic students and greater than 15 percent Black or

Hispanic students. Otherwise the school will be put in a regular sampling stratum. If the expected

sample size within these strata is less than 8.0, they will be left as is. If the expected sample size is

greater than 8.0, then each of these two categories will be subdivided into up to four substrata (two

for expected sample size up to 12.0, three for expected sample size up to 16.0, and four for expected

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 18 of 24 sample size greater than 16.0). For the category of no oversampling , the subdivision will be by state

or groups of contiguous states. For the oversampling strata, the subdivision will be by percentage

Black and Hispanic . Within these substrata, th e schools are to be sorted by school type (public, BIE,

DoDEA) and median household income from the 2000 Census (using a serpentine sort within the

school type substrata).

High School Longitudinal Study (HSLS) Linking Study Sample

The sample of students f

or the HSLS linking study will be obtained as follows. For the whole

gamma sample , the NAEP school sample will be selected so as to maximize the overlap with the

HSLS school sample. Then at the time of student sampling, any student who is in the HSLS sampl e,
but is not selected for NAEP (the RM spiral), will be included in the study and assigned the ML

spiral . This is expected to be about 6 ,000 students from about 30 0 schools , after accounting for

schools that are not responding to the HSLS

.

NAEP

- PISA Linking Study Sample

The sample of students for this study will be obtained as follows. For all gamma sample schools in

Connecticut, Florida, and Massachusetts, samples of students will be selected from each of grades 9,

10, and 11 (from among those grades offered by the school). The procedure for selecting the

student samples will be similar to that used to select grade 12 students, and is described in the

student sampling section below. The selected students will be assigned the MP spiral appropriate for

their grade.



## 4. Delta S amples

These are the priv ate school samples at grades 4, 8, and 12 for conducting the oper ational

assessments in reading and math, as well as the special studies that are incorporated into the RM

spiral

. The number of students targ eted per schools will be 6 4 at grade 4, 65 at grade 8 , and 60 at

grade 12

– 30 for reading, 30 for math, with 4 at grade 4 and 5 at grade 8 for pilot tests and special

studies

. The sample sizes are large enoug h to report results by Catholic and non

- Catholi c at grades 4

and 8, but at grade 12 only for private schools as a whole. Approximately half the sample at each

grade will be from Catholic schools. This is very similar to the design used in 2009, at each grade.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 19 of 24 Stratification

The private schools are to be explicitly stratified by private school type (Catholic /Other ). W ithin

each private school type, stratification will be by Census region ( 4 categories), type of location (12

categories), and enrollment size. In general, where there are few or no schools in a given stratum,

categories will be collapsed together

, always preserving the private school type

.



## 5. Epsilon Sample

This sample is analogous to the beta sampl e , but for private schools , at grade 8 . The same PSUs will

be used as for the beta sample , and the TL session type will be conducted. There will be no

oversampling of private schools for this sample

. The epsilon sample schools will not overlap with

the delta sample schools.



## 6. Rho Sample

This samp le will be used to conduct the g rade 4 Readin g Aloud Accommodations s tudy. It will

consist of about 70 schools from a small number of states, with the selected schools each having

enough SD and ELL students so that nine students can be selected who are SD, nine who are ELL,

and nine who are neither. These students will then be randomly assigned to one of three conditions,

and assessed with a NAEP reading assessment. The three conditions are a) standard conditions; b)

read aloud the test questions, but not the stimulus passage; c) read aloud all of the assessment

material. These schools will not be included in any other NAEP 2013 sample, at any grade.



# IV. New Schools

To compensate for the fact that files used to create the NAEP school sampling frames are at least

two years out of date at the time of frame construction, we will supplement the Alpha, Gamma

, and

Delta sample with new school samples at each grade.

The new school samples will be drawn using a two

- stage design. At the first stage, a minimum of ten

school districts (in states with at least ten districts) will be selected from each state for public

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 20 of 24 schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled

districts and dioceses will be asked to review lists of their respective schools and identify ne w
schools. Frames of new schools will be constructed from these updates, and new schools will be

drawn with probability proportional to size using the same sample rates as their corresponding

original school samples.

The school sample sizes in the above t ables do not reflect new school samples.



# V. Substitute Samples

Substitute samples will be sele cted for each of the Beta

, Gamma, Delta, and Epsilon samples. The

substitute school for each original will be the next “available” school on the sorted sampling frame,

with the following exceptions:

n 

### Schools selected for any NAEP samples will not be used as substitutes.

n 

### Private schools whose school affiliation is unknown will not be used as substitutes. Also, unknown affiliated private schools in the original sam ples will not get substitutes.

n 

### A school can be a substitute for one and only one sample. (If a school is selected as a

substitute school fo r grade 12, for example, it can not be used as a substitute for either grade 4 or grade 8.)

n 

### A public school substitute will always be in the same state as its original school.



# VI. Contingency Samples

The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at

some point over the next few months, a given district might choose to opt out of the TUDA

program for 2013 . However, it is not acceptable for all schools in such a district to decline NAEP,

as then the state estimates will be adversely affected. Thus to deal with this possibility, in each

TUDA district, subsamples of the alpha sample schools will be identified as contingency samples. In

the event that the district withdraws from the TUDA program prior the selection of the student

sample, all alpha sampled schools from that district will be dropped from the sample, with the

excep tion of those selected in the contingency sample. The contingency sample will provide a

proportional representation of the district, within the aggregate state sample. Student sampling in

those schools will then proceed in the same way as for the other sch ools within the same state.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 21 of 24 Similarly, in the thirteen states participating at grade 12, in the gamma sample contingency samples

will be identified in those states. If the state decides to withdraw from the state

- level assessment, the

contingency sample sc hools will be retained, as part of the grade 12 national sample.



# VI I

. Student Sampling

Students within the sampled schools will be selected with equal probability. The student sampling

parameters vary by sample type (Alpha, Beta

, Gamma, Delta, Epsilon

, and Rho ) and grade, as

described below.

Alpha Sample , Grade 4 Schools (Except Puerto Rico )

n 

### All students, up to 70 , will be selected.

n 

### If the school has between 71 and 90 students (inclusive), the school will be asked if it

wishes to have all stud ents selected, or a sa mple of 63 . If the school elects to have all
students selected, all students will be selected. If the school asks for a sa mple of 63

stu dents, a systematic sample of 63 students will be selected.

n 

### If the school has more than 90 stu dents, a systematic sample of 63 students will be

selected.

n 

### In some schools, the school may be assigned more than one ‘hit’ in sampling. In these

schools we will select a sample of size 63 times the number of hits, taking all students if this targe t is greater than or equal to 63 / 70 of the total grade 4 enro l ment.

n 

### All students will b e assigned to assessment type RM

.

Alpha Sample , Grade 8 Schools (E xcept Puerto Rico)

n 

### All students, up to 70 , will be selected.

n 

### If the school has more than 70 stu dents, a systematic sampl e of 63 students will be

selected.

n 

### In some schools, the school may be assigned more than one ‘hit’ in sampling. In these

schools we will select a sample of size 63 times the number of hits, taking all students if this targe t is greater than or equal to 63 / 70 of t he total grade 8 enro lment.

n 

### All students will be assigned to assessment type RM.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 22 of 24 Alpha Sample, Puerto Rico Grades 4 and 8

n 

### All students, up to 55, will be selected.

n 

### If the school has more than 55 students, a systematic sample of 50 students will be

s elected with no oversampling.

n 

### All students will be assigned to assessment type PR.

Delta Samples , Grade s 4 and 8

n 

### All students, up to 72 , will be selected.

n 

### If the school has more than 72 stu dents, a systematic sample of 64 students (grade 4), or 65 students (grade 8) will be selected.

n 

### All students will b e assigned to assessment type RM

.

Beta and Epsilon Samples, Grade 8

n 

### In each school, a sample for computer

- based testing will be selected as follows: All students up to 30 will be selected. If th ere are more than 30 students enrolled, a sample

of 30 students will be selected.

n 

### All students will be assigned to the TL session.

Gamma and Delta Samples, Grade 12

n 

### All students, up to 66, will be selected.

n 

### If the school has more than 66 students, a system atic sample of 60 students will be

selected.

n 

### In some schools, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 60 times the number of hits, taking all students if this target is greater than or eq ual to 60/66 of the total grade enrollment.

n 

### All students so selected will be assigned to assessment type RM.

n 

### For the gamma sample, any grade 12 HSLS students not selected in the above steps will be added to the sample and assigned assessment type ML.

n 

### For t he gamma sample, in Connecticut, Florida, and Massachusetts, in each school a sample of students from grades 9, 10, and 11 will be selected. For grades 10 and 11, the sample will be drawn as for grade 12, but using 30 in place of 60, and 33 in place of 66,

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 23 of 24 in the above algorithm. For grade 9, the same procedure will be used, but using figures of 18 and 20. These students will be assigned assessment type MP.

Rho Sample, Grade 4

Details of the studen t sampling procedures for the grade 4 Reading Aloud Accommodations study

have yet to be finalized. However, they will be something like the following. On the basis of e

- filed

student data for all grade 4 students in the school, students will be classified into four groups: SD

only, ELL only, neither SD nor ELL, and Other (including students who are both SD and ELL, and

students for whom complete data on SD and ELL status is not a vailable). A random sample of nine

students will be selected from each of the first three of these group s . Th e students from each group

will be randomly assigned in equal numbers to each of the three assessment conditions

.



# VIII

. Weighting Requirements

Weighting activities for the 2013 NAEP assessments are not covered under the current NAEP

contracts. However, b ased on past experience, below is a description of the likely weighting

requirements

.

The Operational Samples

These samples will have a single set of we ights for each subject (reading and math at grades 4, 8

, and

12 ) applied to reflect probabilities of selection, school and student nonresponse, any trimming, and

the random assignment to the particular subject. There will be a separate replication scheme s by

grade and public/private

. Such weights will also be derived for the Puerto Rico K aSA assessment

.

The Pilot Test and Special Study Samples

Fully adjusted weight s will not be provided for the students in the pilot test studies and special

studies . However, with the exception of the grade 4 Reading Aloud Accommodations study

, preliminary weights will be available for these samples. These will reflect the school and student

selection probabilities, but with no adjustments for nonresponse.

### Memorandum : 2013

- 1.3A/1.3B/1.3G/1.3D/1.3E/1.3 R May 24 , 2012

Page 24 of 24 The HSLS Linking Sample

Weights may be required for the linking sample

, which account for the samplin g procedures and

nonresponse for both the HSLS and the NAEP assessment

. Clearly this will require collaboration

between the next NAEP contractor responsible for weighting, and RTI International, the contractor

for the HSLS study

. Once the analysis plan for this study has been specified, a weighting plan can be

developed.