# Appendix G

NAEP 2015 Sample Design Memo\\ wes tat.com \ dfs \NAEPLIB\ 2015 \ Memos \ School Sampling \ Public Grades 4 & 8 -A\ 2015 -m01v01a.docx



### Dat e: February 2 6 , 201 4 Memo : 201 5-

1. 1 A/ 1. 1 B/ 1.1G / 1.1D /1. 1E

To: William Ward, NCES

Nicole Beaulieu, ETS

Ed Kulick, ETS

David Freund, ETS

Amy Dresher, ETS

Connie Smith, Pearson

Dianne Walsh

Lauren Byrne

Debby Vivari

Rob Dymowski Jennifer Kali

David Ferraro

Erin Wiley

Dwight Brock

Amy Lin

Chris Averett

Kavemuii Murangi

David Hubble

Sarah Shore

William Wall

From: Lloyd Hicks and John Burke

Reviewer: Keit h Rust

Subject: Sample Design for 2015 NAEP - Draft



# I. Introduction

For 201 5 , the NAEP assessment involves the following components:

A. 

### National assessments in reading, math ematics, and science at grades 4, 8 , and 12 ;

B. 

### State- by

- state and Trial Urban District Assessment (TUDA) assessments in reading and

mathematics for public schools at grades 4 and 8;

C. 

### A state

- by

- state assessment in science for public schools at grades 4 and 8 ;

D. 

### No state

- by

- state assessments in any states at grade 12;

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 2 of 23 E. 

### Techno logy

- based assessment (TBA) start -ups in reading, mathematics, and science at grades 4, 8, and 12;

F. 

### Pilot assessments of science interactive computer tasks (ICTs) and hybrid hands

- on tasks (HOTs) at grades 4 , 8, and 12 ;

G. 

### A special national study of Knowledg e and Skills Appropriate (KaSA) items in mathematics, at grades 4 and 8, including Puerto Rico.

Below is a summary list of the features of the 201 5 sample design.

1. The alpha samples for grades 4 and 8 public, the gamma sample for grade 12 public, and the delta samples for private schools at grades 4, 8, and 12 will be used for the operational assessments in reading, mathematics, science, and meaning vocabulary. These samples will also be used for the national KaSA special studies at grades 4 and 8.

2. There w

ill be two sets of relatively small PSU

- based samples for the TBA start -up assessments and for the pilot assessments of science ICTs/HOTs at grades 4, 8, and 12. The public school samples for these assessments are known as the Beta samples and the

correspo nding private samples are known as the Epsilon samples.

3. As in recent NAEP studies, Trial Urban District Assessment (TUDA) samples will form part of the corresponding state samples, and the state samples will form part of the national sample. The re are twenty

- one Trial Urban District Assessment ( TUDA ) participants . Twenty of the twenty

- one participated also participated in 201 3. Milwaukee WI is no longer participating, but Duvall County FL has been added in.

4. All states but Alaska, California, Colorado, P ennsylvania, and Vermont have signed on to do science at grades 4 and 8. BIE schools will not be doing science either

.

5. Most schools in states doing science will receive a 1:1:1 RMS spiral (1/3 reading, 1/3 math, and 1/3 science). Most schools in states not doing science will receive a 10:10:1 RMS spiral (10/21 reading, 10/21 math, and 1/21 science) to ensure that these states are sufficiently represented in the national science sample.

6. All TUDAs but Albuquerque, NM, Chicago, IL, and New York City, NY

– thos e that constitute a relatively large proportion of their state population

– will receive the 10:10:1 RMS spiral. Albuquerque, Chicago, and New York will receive the 1:1:1 RMS spiral (note that they are in states that have signed on for science)

.

7. A small s ubset of schools from each jurisdiction, regardless of whet her they are doing

science , will receive a spiral that includes meaning vocabulary and KaSA booklets at a rate that will support national level reporting. This spiral, the NL spiral, will consist o f 6/21 reading, 6/21 math, 6/21 science, 2/21 meaning vocabulary, and 1/21 KaSA booklets. The subsample of schools selected for the NL spiral in each state is proportional to the representation of each state in the nation.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 3 of 23 8. There will be no samples in terri tories, other than for Puerto Rico at grades 4 and 8 for the KaSA study.

9. Most BIE schools will receive a 10:10:1 RMS spiral and a small subset will receive the NL spiral

.

10. A ll BIE schools and students will be included in the operational samples at grades 4 and

8. This is because, after a hiatus in 2013, the National Indian Education Study (NIES) is resuming . Having all BIE students in sample is designed to provide detailed national results for American Indian and Alaskan Native (AIAN) students in reading and mathematics, as part of the National Indian Education Study (NIES). Because science will be spiraled with math and reading, the science sample will be spread across almost all BIE schools at grades 4 and 8.

11. As in 201 3 , the Department of Defense Schools are expected to be reported as a single jurisdiction (DoDEA), instead of the two components of domestic (DDESS) and overseas (DoDDS). DoDEA schools will receive either the 1:1:1 RMS spiral or the NL spiral.

12. At grade 12, there will be no state

- level samples.

13. All private schools will receive a NL spiral.

14. Oversampling of private schools at grades 4 and 8 will be done at the same level as 2013. Response rates permitting, this will allow separate reporting for reading,
mathematics, and science for Catholic and non

- Catholic schools, but no further. As in 2013 there will be no oversampling of private schools at grade 12.

15. The sample sizes of assessed students for these various components are shown in Table 1 (which also shows the approximate numbers of participatin g schools).

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 4 of 23 Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2015 NAEP

Spi ral Jur isdictions Stu dents

Tot al Spi ral Indic. Sta tes (incl.

DC, BIE,

DoDE A) Urb an

dis tricts Pub lic school

stu dents Pri vate

sch ool

stu dents

Gra de 4

nat 'l/state reading RS, RM, NL 53 21 133,000 3,000 136,000

nat 'l/state math RS, RM, NL 53 21 133,000 3,000 136,000

nat 'l/state science RS, RM, NL 47 105,000 3,000 108,000

rea ding - vocabulary NL 7,200 800 8,000

mat h KaSA NL 3,600 400 4,000

Pue rto Rico KaSA PR 1 5,000 0 5,000

Tot al

- alpha 4 386,800 386,800

Tot al

- delta 1 10,200 10,200

Typ ical max. no. students/school 90 105

Avera ge assessed students/school 51 25 50

Tot al schools

- alpha, delta 7,550 410 7,960

Rea ding - TBA Start

- up TB 10,800 1,200 12,000

Mat hematics - TBA Start

- up TB 10,800 1,200 12,000

Scie nce - TBA Start

- up TB 7,200 800 8,000

Scie nce ICT Pilot TB 3,600 400 4,000

Scie nce HOT Pilot TB 2,700 300 3,000

Tot al

- beta 1 35,100 35,100

Tot al

- epsilon 1 3,900 3,900

Typ ical max. no. students/school 50 50

Avera ge assessed students/school 41 25 39

Tot al schools

- beta, epsilon 855 155 1,010

Tot al number of students grade 4 421,900 14,100 436,000

Tot al number of schools grade 4 8,405 565 8,970

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 5 of 23 Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2015 NAEP (C ontinued)

Spi ral Jur isdictions Stu dents

Tot al Spi ral Indic. Sta tes (incl.

DC, BIE,

DoDE A) Urb an

dis tricts Pub lic school

stu dents Pri vate

sch ool

stu dents

Gra de 8

nat 'l/state reading RS, RM, NL 53 21 133,000 3,000 136,000

nat 'l/state math RS, RM, NL 53 21 133,000 3,000 136,000

nat 'l/state science RS, RM, NL 47 105,000 3,000 108,000

rea ding - vocabulary NL 7,200 800 8,000

mat h KaSA NL 3,600 400 4,000

Pue rto Rico KaSA PR 1 5,000 0 5,000

Tot al

- alpha 4 386,800 386,800

Tot al

- delta 1 10,200 10,200

Typ ical max. no. students/school 90 105

Avera ge assessed students/school 64 26 62

Tot al schools

- alpha, delta 6,020 400 6,420

Rea ding - TBA Start

- up TB 15,300 1,700 17,000

Mat hematics - TBA Start

- up TB 10,800 1,200 12,000

Scie nce - TBA Start

- up TB 7,200 800 8,000

Scie nce ICT Pilot TB 3,600 400 4,000

Scie nce HOT Pilot TB 2,700 300 3,000

Tot al

– beta 1 39,600 39,600

Tot al

– epsilon 1 4,400 4,400

Typ ical max. no. students/school 50 50

Avera ge assessed students/school 44 25 41

Tot al schools

- beta, epsilon 900 175 1,075

Tot al number of students grade 8 426,400 14,600 441,000

Tot al number of schools grade 8 6,920 575 7,495

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 6 of 23 Table 1. Target sample sizes of assessed students, and expected number of participating

schools, for 2015 NAEP (C ontinued)

Spi ral Jur isdictions Stu dents

Tot al Spi ral Indic. Sta tes (incl.

DC, BIE,

DoDE A) Urb an

dis tricts Pub lic school

stu dents Pri vate

sch ool

stu dents

Gra de 12

nat 'l/state reading NL 11,700 1,300 13,000

nat 'l/state math NL 12,600 1,400 14,000

nat 'l/state science NL 9,900 1,100 11,000

rea ding - vocabulary NL 5,400 600 6,000

Tot al

– gamma 1 39,600 39,600

Tot al

- delta 1 4,400 4,400

Typ ical max. no. students/school 100 100

Avera ge assessed students/school 76 80 77

Tot al schools

- gamma, delta 520 55 575

Rea ding - TBA Start

- up TB 13,500 1,500 15,000

Mat hematics - TBA Start

- up TB 10,800 1,200 12,000

Scie nce - TBA Start

- up TB 8,100 900 9,000

Scie nce ICT Pilot TB 3,600 400 4,000

Scie nce HOT Pilot TB 2,700 300 3,000

Tot al

– beta 1 38,700 38,700

Tot al

– epsilon 1 4,300 4,300

Typ ical max. no. students/school 50 50

Avera ge assessed students/school 42 43 42

Tot al schools

- beta, epsilon 920 100 1,020

Tot al number of students grade 12 78,300 8,700 87 ,000

Tot al number of schools grade 12 1,440 155 1,595

GRAND TOTAL STUDENTS 926,600 37,400 964,000

GRAND TOTAL SCHOOLS 17,510 1,990 19,500



# II. As sessment Types

The assessment spiral types are shown in Table 2. F ive different spirals will be use d at each of the

grades 4 and 8, and two at grade 12 . Session IDs contain six digits, traditionally. The first two digits

identify the assessment “type” (subjects and type of spiral in a general way). Grade is contained in

the second pair of digits, and the session sequential number ( within schools) in the last two digits.

For example, session RS 0401 de notes the first grade 4 reading, mathematics , and science operational

assessment in a given school.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 7 of 23 Table 2. NAEP 2015 assessment types and IDs

ID Typ e Sub jects Gra des Sch ools Com ments

RS Ope rational

P&P Reading, math, science

(1:1:1) 4, 8 Public Mos t public schools in states

doi ng science, plus TUDAs in

some of those states

RM Ope rational

P&P Reading, math, science

(10:10:1) 4, 8 Public Mos t public schools in states not

doi ng science, plus TUDAs in

those states and most others

NL Ope rational

P&P Reading, math, science,

Meaning Vocab, KaSA
(6:6:6:2:1, grades 4 & 8;
13:14:11:6:0, grade 12) 4, 8, 12 Public,

private All private schools; all public

sch ools at Grade 12; some

public schools in each
jurisdiction (except Puerto Rico
and DoD overseas) at grades 4

and 8

PR Ope rational

P&P KaSA Mathematics 4, 8 Public Puerto Rico only

TB TBA start

- ups & p ilots Rea ding, math, science,

sci ence ICT, science HOT 4, 8, 12 Publi c, pri vate Boo klets will not be used in any

othe r jurisdiction.



# III. Sam ple Types and Sizes

In similar fashion to past years (but somewhat different ) , we will identify five different types of

school samples : Alpha, Beta, Gamma, Delta, and Epsilon . These distinguis h sets of schools that will

be conducting distinct portions of the assessment

.



## 1. Alpha Samples at Grades 4 and 8

These are public school samples for grades 4 and 8. They will be used for the operational state -bystate assessments in reading

, math , and s cience and contribute to the national samples for these

subjects as well . The y will also be used for the national meaning vocabulary assessment and for the

national KaSA special studies. There will be alpha samples for each state, DC, DoDEA, BIE, and

Puert o Rico. The alpha samples will not be used for any TBA start -up assessments or pilot

assessments.

The details of the target student sample sizes for the alpha samples are as follows:

A. 

### At each grade, the target student sample size for a state depends on whether or not it signed on for science. For each state that signed on for science, the target student sample size is

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 8 of 23 7,600: roughly 2

, 200 assessed each for math, reading, and science. In these states the RS session type will be used. For the four states not signed on for science and BIE, the target student sample sizes are 5,300 : 2, 200 (assessed) each for math and reading and 220 for science. In these jurisdictions the R M session type wil l be used.

B. 

### A national

- level subsample will be applied to the sample schools at each grade with a target student sample size of 86,600. Schools that are selected for this subsample will be given the NL spiral.

C. 

### There will be samples for twenty

- one TUDA distr icts. For two of the five largest (New York City and Chicago) where we require a sizeable science sample so as to ensure an adequate state sample , the student target samples sizes are three -quarters the size of a science state (5,670). For the remaining three large TUDA districts (Los Angeles, Miami

- Dade, and Houston ) where only a smaller science sample is required , the student target sample sizes are three

quarters the size of a non

- science state sample ( 4,000 ). For Albuquerque, a small TUDA

where a sizeab le science sample is required , the student target sample size is half the size of a science state (3,780), whereas the remaining small TUDA districts where only a smaller science sample is required

, the student target sample sizes are one

- half the size of a non

- science state sample (2 ,650 ).

D. 

### Note, that above, there is a conflict between sample size requirements at the state level, and the TUDA district level. This will be resolved as in previous years: the districts will have the target samples indicated in B, and reflected in Table 4. For the states that contain one or more of these districts, the target sample size indicated in A (and shown in Table 4) will be used to determine a school sampling rate for the state, which will be applied to the balance of the state outside the TUDA district(s). Thus the target student sample sizes, shown in Table 4, for states that contain a TUDA district, are only ‘design targets’, and are smaller than the final total sample size for the state, but larger than the sample fo r the balance of the state, exclusive of its TUDA districts. In the case of the District of Columbia, the state sample size requirement is that all schools and students be included. This renders moot any requirements for the DC TUDA sample, which by defaul t consists of all schools operated by the DCPS district (but excludes charter schools in DC, even though those are all included in the state sample, as these are not operated by DCPS).

E. 

### In Puerto Rico, the target sample size is 5,750 per grade (grades 4 and 8), with the goal of assessing 5,000 students. Only KaSA mathematics will be assessed in Puerto Rico (PR session type).

As in past state

- by

- state assessments, schools with fewer than 20 students in the grade in question

will be sampled at a moderately low er rate than other schools (at least half, and often higher,

depending upon the size of the school). This is in implicit recognition of the greater cost and burden

associated with surveying these schools.

As mentioned above, the NAEP 2015 design includes an oversample of high proportion American

Indian schools in certain states (as part of the NIES design). These schools will be sampled at higher

rates than the other schools. The NIES oversample will take place in Arizona, Minnesota, North

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 2 6 , 2014

Page 9 of 23 Carolina, Oregon , Utah, and Washington. Schools with relatively large percentages of American

Indian students will be separately stratified, as explained below, and oversampled by factors ranging

from 2 to 6 based on state and grade. Table 3 below shows the thresholds use d to define the NIES

oversampling strata along with their corresponding oversampling factors.

Table 3. Percent American Indian thresholds and oversampling factors for the NIES school

oversample by state and grade

Sta te Gra de 4 Gra de 8

Per cent American

Ind ian thresholds Ove rsampling

fac tor Per cent American

Ind ian thresholds Ove rsampling

fac tor

Ari zona 50 3 50 2

Uta h 5 5 5 5

Min nesota 10 5 10 4

Nort h Carolina 15 6 10 6

Ore gon 10 6 15 6

Was hington 10 6 15 6



### Tab le 4 shows the target student sample sizes, and the approximate counts of schools to be selected

in the alpha samples, along with the school and student frame counts, by state and TUDA districts

for grades 4 and 8. The table also identifies the jurisdictions where we tak e all schools and where we

take all students.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26 , 2014

Page 10 of 23 Table 4. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (A lpha samples)

Gra de 4 Gra de 8

Juri sdiction Sch ools in

fra me Sch ools in

sam ple Stu dents

in frame Ove rall

targ et

studen t

sam ple

size Sch ools in

fra me Sch ools in

sam ple Stu dents in

fra me Ove rall

targ et

studen t

sam ple size

Ala bama 719 110 57,293 7,600 462 96 58,020 7,600

Ala ska 361 158 9,870 5,300 276 118 9,865 5,300

Ari zona 1,184 111 83,596 7,600 784 104 82,121 7,600

Ark ansas 488 113 36,667 7,600 300 99 35,481 7,600

Bur eau of Indian Education 136 136 3,330 3,330 ** 112 112 2,904 2,904 **

Cal ifornia 5,907 103 470,328 7,600 2,876 96 454,580 7,600

Col orado 1,041 97 65,699 5,300 554 93 62,665 5,300

Con necticut 575 114 40,276 7,600 315 96 41,529 7,600

Dela ware 118 94 10,099 7,600 61 61 9,885 7,600 *

Dist rict of Columbia 122 122 5,113 5,113 ** 73 73 4,539 4,539 **

DoDE A Schools 108 108 7,514 7,514 ** 63 63 5,596 5,596 **

Flo rida 2,183 96 204,096 7,600 1,173 94 201,815 7,600

Geo rgia 1,248 92 130,599 7,600 552 87 129,337 7,600

Haw aii 203 112 14,902 7,600 83 62 13,132 7,600

Ida ho 373 127 22,095 7,600 212 101 21,610 7,600

Ill inois 2,310 121 152,571 7,600 1,610 110 153,611 7,600

Ind iana 1,059 110 78,537 7,600 496 90 80,280 7,600

Iow a 647 138 36,039 7,600 374 108 35,713 7,600

Kansas 704 145 36,223 7,600 395 115 35,689 7,600

Kent ucky 734 113 51,728 7,600 407 99 50,978 7,600

Lou isiana 754 115 54,823 7,600 479 102 52,027 7,600

Mai ne 326 169 13,609 7,600 199 116 13,687 7,600

Mar yland 893 110 64,759 7,600 379 93 60,562 7,600

Mas sachusetts 955 111 70,874 7,600 477 90 71,996 7,600

Mic higan 1,731 119 113,008 7,600 1,071 102 116,734 7,600

Min nesota 934 117 62,923 7,600 702 114 62,522 7,600

Mis sissippi 422 106 38,029 7,600 286 95 37,697 7,600

Mis souri 1,160 130 68,565 7,600 691 110 68,104 7,600

### Mem orandum : 2015

- 1.1 A/1. 1 B/1. 1 G/1. 1 D/1. 1E February 26 , 2014

Page 11 of 23 Table 4

. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (A lpha samples) ( Continued )

Gra de 4 Gra de 8

Juri sdiction Sch ools in

fra me Sch ools in

sam ple Stu dents

in frame Ove rall

targ et

studen t

sam ple

size Sch ools in

fra me Sch ools in

sam ple Stu dents in

fra me Ove rall

targ et

studen t

sam ple size

Mon tana 392 214 11,146 7,600 273 157 10,789 7,600

Neb raska 536 167 22,939 7,600 303 126 21,793 7,600

Nevad a 385 97 33,934 7,600 161 84 33,313 7,600

New Hampshire 268 141 13,975 7,600 141 90 14,348 7,600

New Jersey 1,360 114 99,003 7,600 750 97 98,145 7,600

New Mexico 440 130 25,977 7,600 225 102 25,261 7,600

New York 2,397 104 199,215 7,600 1,443 94 198,874 7,600

Nort h Carolina 1,422 103 116,094 7,600 703 91 116,743 7,600

Nort h Dakota 258 258 7,760 7,760 ** 183 183 7,403 7,403 **

Ohi o 1,778 115 129,775 7,600 1,118 100 132,975 7,600

Okl ahoma 872 136 50,101 7,600 584 120 47,828 7,600

Ore gon 750 131 42,558 7,600 422 108 42,917 7,600

Pen nsylvania 1,649 91 129,728 5,300 900 87 133,900 5,300

Puer to Rico 988 171 33,865 5,750 401 125 32,888 5,750

Rho de Island 166 121 10,751 7,600 59 59 10,248 7,600 *

Sou th Carolina 633 101 55,665 7,600 307 89 55,154 7,600

Sou th Dakota 314 197 10,084 7,600 245 160 9,476 7,600

Ten nessee 989 109 76,003 7,600 581 97 73,679 7,600

Tex as 4,373 99 386,022 7,600 2,224 95 372,168 7,600

Uta h 613 102 48,529 7,600 244 91 45,931 7,600

Ver mont 218 218 6,247 6,247 ** 122 122 6,138 6,138 **

Vir ginia 1,115 100 95,536 7,600 384 86 94,826 7,600

Was hington 1,231 116 79,005 7,600 610 99 78,960 7,600

Wes t Virginia 423 157 20,728 7,600 196 98 20,744 7,600

Wisc onsin 1,090 136 61,306 7,600 642 109 61,502 7,600

Wyo ming 185 185 7,185 7,185 ** 87 87 6,902 6,902 **

### Mem orandum : 2015

- 1.1 A/1. 1 B/1. 1 G/1. 1 D/1. 1E February 26 , 2014

Page 12 of 23 Table 4

. Grade 4 and 8 school and student frame counts, expected school sample sizes, and initial target student sample sizes for

the 2013 state

- by

- state and TUDA district assessments (A lpha samples) ( Continued )

Gra de 4 Gra de 8

Juri sdiction Sch ools in

fra me Sch ools in

sam ple Stu dents

in frame Ove rall

targ et

studen t

sam ple

size Sch ools in

fra me Sch ools in

sam ple Stu dents in

fra me Ove rall

targ et

studen t

sam ple size

Alb uquerque 95 52 7,426 3,780 40 32 6,721 3,780

Atl anta 55 42 4,238 2,650 24 24 3,509 3,509 **

Aus tin 79 44 6,988 2,650 24 24 5,542 2,650 *

Bal timore City 127 56 6,385 2,650 102 53 5,654 2,650

Bos ton 72 52 4,165 2,650 41 41 3,867 2,650 *

Cha rlotte 103 43 11,447 2,650 45 32 10,717 2,650

Chi cago 501 102 29,331 5,670 482 100 28,794 5,670

Cle veland 71 71 2,970 2,970 ** 66 66 2,755 2,755 **

Dalla s 147 43 13,031 2,650 42 42 10,799 2,650 *

Det roit 66 48 4,213 2,650 50 50 3,224 3,224 **

Duv al County, FL 115 44 9,999 2,650 48 31 9,098 2,650

Fre sno 71 44 5,854 2,650 23 23 5,086 2,650 *

Hills borough County, FL 171 44 15,515 2,650 85 43 15,159 2,650

Hou ston 176 64 17,149 4,000 64 46 13,400 4,000

Jef ferson County, KY 100 44 7,589 2,650 46 29 7,345 2,650

Los Angeles 556 66 50,563 4,000 191 64 45,215 4,000

Mia mi 271 67 26,278 4,000 162 65 27,130 4,000

New York City 768 74 72,429 5,670 527 71 68,327 5,670

Phi ladelphia 162 48 11,438 2,650 126 46 9,573 2,650

San Diego 145 47 10,378 2,650 62 36 9,231 2,650

Dist rict of Columbia PS 80 80 3,291 3,291 ** 32 32 2,289 2,289 **

Tot al 52,240 7,831 3,776, 296 443,343 28,770 6,215 3,725, 584 441,811

Cou nts for states do not reflect the oversampling for their constituent TUDA districts.

Target student sample sizes reflect sample sizes prior to attrition due to exclusion, ineligibility, and nonresponse.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are included in the NAEP sample.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 13 of 23 Table 5 consolidates the target student (and resulting school) sample size numbers, to show the total

target sample sizes in each state, combining the TUDA targets with those for the balance of the

state.

### Mem orandum : 2015

- 1.1 A/1. 1 B/1. 1 G/1 .1 D/1. 1E February 26 , 2014

Page 14 of 23 Table 5

. Total sample s izes, combining s tate and TUDA samples

Gra de 4 Gra de 8

Juri sdiction Sch ools

in frame Sch ools

in sample Stu dents

in frame Ove rall

targ et

studen t

sam ple

size Sch ools in

fra me Sch ools in

sam ple Stu dents in

fra me Ove rall

targ et

studen t

sam ple size

Ala bama 719 109 57,293 7,600 462 96 58,020 7,600

Ala ska 361 158 9,870 5,300 276 118 9,865 5,300

Ari zona 1,184 111 83,596 7,600 784 104 82,121 7,600

Ark ansas 488 113 36,667 7,600 300 99 35,481 7,600

Bur eau Of Indian Education 136 136 3,330 3,330 ** 112 112 2,904 2,904 **

Cal ifornia 5,907 277 470,328 15,818 2,876 229 454,580 15,902

Col orado 1,041 97 65,699 5,300 554 93 62,665 5,300

Con necticut 575 114 40,276 7,600 315 96 41,529 7,600

Dela ware 118 94 10,099 7,600 61 61 9,885 7,600 *

Dist rict Of Col umbia 122 122 5,113 5,113 ** 73 73 4,539 4,539 **

DoDE A Schools 108 108 7,514 7,514 ** 63 63 5,596 5,596 **

Flo rida 2,183 227 204,096 14,969 1,173 209 201,815 14,962

Geo rgia 1,248 130 130,599 10,004 552 108 129,337 10,903

Haw aii 203 112 14,902 7,600 83 62 13,132 7,600

Ida ho 373 127 22,095 7,600 212 101 21,610 7,600

Ill inois 2,310 197 152,571 11,805 1,610 185 153,611 11,842

Ind iana 1,059 110 78,537 7,600 496 90 80,280 7,600

Iow a 647 138 36,039 7,600 374 108 35,713 7,600

Kansas 704 145 36,223 7,600 395 115 35,689 7,600

Kent ucky 734 142 51,728 9,137 407 113 50,978 9,157

Lou isiana 754 115 54,823 7,600 479 102 52,027 7,600

Mai ne 326 169 13,609 7,600 199 116 13,687 7,600

Mar yland 893 150 64,759 9,501 379 133 60,562 9,541

Mas sachusetts 955 155 70,874 9,803 477 125 71,996 9,842

Mic higan 1,731 162 113,008 9,966 1,071 149 116,734 10,614

Min nesota 934 117 62,923 7,600 702 114 62,522 7,600

Mis sissippi 422 106 38,029 7,600 286 95 37,697 7,600

Mis souri 1,160 130 68,565 7,600 691 110 68,104 7,600

Mon tana 392 213 11,146 7,600 273 157 10,789 7,600

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1 E February 26 , 2014

Page 15 of 23 Table 5

. Total sample s izes, combining s tate and TUDA samples ( Continued )

Gra de 4 Gra de 8

Juri sdiction Sch ools

in frame Sch ools

in sample Stu dents

in frame Ove rall

targ et

studen t

sam ple

size Sch ools in

fra me Sch ools in

sam ple Stu dents in

fra me Ove rall

targ et

studen t

sam ple size

Neb raska 536 167 22,939 7,600 303 126 21,793 7,600

Nevad a 385 97 33,934 7,600 161 84 33,313 7,600

New Hampshire 268 141 13,975 7,600 141 90 14,348 7,600

New Jersey 1,360 114 99,003 7,600 750 97 98,145 7,600

New Mexico 440 151 25,977 9,188 225 111 25,261 9,347

New York 2,397 142 199,215 10,507 1,443 132 198,874 10,659

Nort h Carolina 1,422 137 116,094 9,500 703 116 116,743 9,551

Nort h Dakota 258 258 7,760 7,760 ** 183 183 7,403 7,403 **

Ohi o 1,778 182 129,775 10,396 1,118 162 132,975 10,198

Okl ahoma 872 136 50,101 7,600 584 120 47,828 7,600

Ore gon 750 131 42,558 7,600 422 108 42,917 7,600

Pen nsylvania 1,649 131 129,728 7,482 900 126 133,900 7,571

Puer to Rico 988 171 33,865 5,750 401 125 32,888 5,750

Rho de Island 166 121 10,751 7,600 * 59 59 10,248 7,600 *

Sou th Carolina 633 101 55,665 7,600 307 89 55,154 7,600

Sou th Dakota 314 197 10,084 7,600 245 160 9,476 7,600

Ten nessee 989 109 76,003 7,600 581 96 73,679 7,600

Tex as 4,373 250 386,022 16,167 2,224 202 372,168 16,291

Uta h 613 102 48,529 7,600 244 91 45,931 7,600

Ver mont 218 218 6,247 6,247 ** 122 122 6,138 6,138 **

Vir ginia 1,115 100 95,536 7,600 384 86 94,826 7,600

Was hington 1,231 116 79,005 7,600 610 99 78,960 7,600

Wes t Virginia 423 157 20,728 7,600 196 98 20,744 7,600

Wisc onsin 1,090 136 61,306 7,600 642 109 61,502 7,600

Wyo ming 185 185 7,185 7,185 ** 87 87 6,902 6,902 **

Tot al 52,240 7,831 3,776, 296 443,343 28,770 6,215 3,725, 584 44 1

, 811

Samp le sizes for each state do reflect the samples in the TUDA districts within the state.

* identifies jurisdictions where all schools (but not all students) for the given grade are included in the NAEP sample.

** identifies jurisdictions where all students for the given grade are included in the NAEP sample

.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 16 of 23 Stratification

Each state and grade will be stratified separately, but using a common approach in all cases. TUDA

districts will be separated from their state, and each part stratified separately. The first level of

stratification will be based on urban

- centered type of location. This variable has 12 levels (some of

which may not be present in a given state or TUDA district), and these will be collapsed so that each

of the resulting location categories contains at least nine percent of the student populat ion. Within

each of the resulting location categories, schools will be assigned a minority enrollment status. This

is based on the two race/ethnic groups that are the second and third most prevalent within the

location category. If these groups are both lo w in percentage terms, no minority classification will be

used. Otherwise three (or occasionally four) equal- sized groups (generally high, medium, and low

minority) will be formed based on the distribution across schools of the two minority groups.

Within the resulting location and minority group classes (of which there are likely to be from three

to fifteen, depending upon the jurisdiction), schools will be sorted by a measure derived from school

level results from the most recent available state achievem ent tests at the relevant grade. In general,

mathematics test results will be used, but where these are not available, reading results will be used.

In the few states that do not have math or reading tests at grades 4 and 8 (or where we are unable to

match the results to the NAEP school frame), instead of achievement data, schools will be sorted

using a measure of socio- economic status. This is the median household income of the 5

- digit ZIP

Code area where the school is located, based on the 2012 ACS (5

- yea r) data.

Once the schools are sorted by location class, minority enrollment class, and achievement data (or

household income), a systematic sample of schools will be selected using a random start. Schools

will be sampled with probability proportional to s ize. The exact details of this process are described

in the individual sampling specification memos.



## 2. Beta Sample

The beta sample comprises the national public school sample s at grades 4, 8, and 12

. This sample

will be used to conduct the TBA start -ups in reading, mathematics, and science and the pilot

assessment s in science interactive computer tasks (ICTs) and hybrid hands

- on tasks (HOTs) at all

three grades

. Each o f t hese samples will be nationally representative , with the exception that schools

that are included in the grades 4 and 8 alpha sample s with certainty will not be represented

. This is

because there is to be no overlap with the grade 4 and 8 alpha school sample s.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 17 of 23 The first stage of sampling for th e beta sample is the selection of a sample o f geographic primary

sampling units (PSUs). These will be selected using the same design as for the 2014 assessment s , but

minimizing the overlap with PSUs in that assessment . A total 67 PSUs will be selected, representing

the U.S. (but not including Puerto Rico, or Do DEA schools that are located outside the 50 states

and D.C.). This PSU component is needed because of the operational complexit ies of administerin g
the computer

- based assessment . A select group of staff w ill be trained to administer those

asses sments

. The school stratification of the beta sample within PSUs will be by type of location and

median household income. Schools with more than 15 percent black or Hispanic students will be

sampled at twice the rate of other schools, so as to increase the student sample sizes for these two

groups.

The number of students targeted per school will be 50 (either 25 start -up and 25 ICT/HOT or 50

start -up).



## 3. Gamma Sample

This is the public school sample at grade 12, used for the reading, mathematics, scienc e, and

meaning vocabulary assessments. It will consist of a single sample of schools covering only national

level estimates

. The number of students targeted per school will be 95 (28 reading, 30 for math, 24

science, and 13 meaning vocabulary).

As in past assessments, modest oversampling of Black and Hispanic students will be undertaken in

this sample. This will be carried out at the school level. Each school with more than 15 percent

Black and Hispanic students will be given twice the selection probability of other schools of

comparable size. This means that while about 53 percent of the student population (including over

90 percent of the Black and Hispanic students) are in the oversampled schools, about 70 percent of

the sample student s will co me from these schools.

Stratification

The Gamma sample will have an implicit stratification, using a hierarchy of stratifiers and a

serpentine sort. The highest level of the hierarchy is Census division (9 implicit strata). The next

stratifier in the hie rarchy is type of location, which has twelve categories. Many of the type of

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 18 of 23 location strata nested within Census divisions will be collapsed with neighboring type of location

cells (this will occur if the expected school sample size within the cell is les s than 4.0). These

geographic strata will be subdivided using a dichotomous high minority status category for

oversampling purposes . Schools with more than 10 Black or Hispanic students and greater than 15

percent Black or Hispanic students will be conside red high minority and placed in an oversampling

stratum . All other schools will be considered low minority and placed in a regular sampling stratum

. If the expected sample size for a high or low minority stratum is less than 8.0, it will be left as is. If

the expected sample size is greater than 8.0, then the stratum will be subdivided into up to four

substrata (two for expected sample size up to 12.0, three for expected sample size up to 16.0, and

four for expected sample size greater than 16.0). For the r egular sampling strata , the subdivision will

be by state or groups of contiguous states. For the oversampling strata, the subdivision will be by

percentage Black and Hispanic. Within these substrata, the schools are to be sorted by school type

(public, BIE , DoDEA) and median household income from the 2012 5

- year ACS (using a serpentine

sort within the school type substrata).



## 4. Delta Samples

These are the priv ate school samples at grades 4, 8, and 12 for conducting the oper ational

assessments in reading, mathematics, science, and meaning vocabulary , as well as the KaSA special

studies at grades 4 and 8 that are incorporated into the NL spiral

. The number of students targeted

per schools will be 105 (30 reading, 30 math, 30 science, 10 meaning voca bulary, and 5 KaSA) at

grades 4 and 8 , and 95 (28 reading, 30 for math , 24 science, and 13 meaning vocabulary) at grade 12.

The sample sizes are large enoug h to report results by Catholic and non

- Catholic at grades 4 and 8,

but at grade 12 only for private schools as a whole. Approximately half the sample at each grade will

be from Catholic schools. This is very similar to the design used in 2013 , at each grade.

Stratification

The private schools are to be explicitly stratified by private school type (Cath olic /Other ). W ithin

each private school type, stratification will be by Census region (4 categories), type of location (12

categories), and enrollment size. In general, where there are few or no schools in a given stratum,

categories will be collapsed toge ther

, always preserving the private school type

.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 19 of 23 

## 5. Epsilon Sample

This sample is analogous to the beta sampl e , but for private schools, at grade s 4, 8 , and 12. The

same PSUs will be used as for the beta sample

. There will be no oversampling of private schools for

this sample

. The epsilon sample schools will not overlap with the delta sample schools.



# IV. New Schools

To compensate for the fact that files used to create the NAEP school sampling frames are at least

two years out of date at the time of frame construction, we will supplement the Alpha, Gamma

, and

Delta sample with new school samples at each grade.

The new school samples will be drawn using a two

- stage design. At the first stage, a minimum of ten

school districts (in states with at least ten districts) will be selected from each state for public

schools, and ten Catholic dioceses will be selected nationally for the private schools. The sampled

districts and dioceses will be asked to review lists of their respective schools and ident ify new

schools. Frames of new schools will be constructed from these updates, and new schools will be

drawn with probability proportional to size using the same sample rates as their corresponding

original school samples.

The school sample sizes in the a bove tables do not reflect new school samples.



# V. Substitute Samples

Substitute samples will be sele cted for each of the Beta

, Gamma, Delta, and Epsilon samples. The

substitute school for each original will be the next “available” school on the sorted sampling frame,

with the following exceptions:

A. 

### Schools selected for any NAEP samples will not be used as substitutes.

B. 

### Private schools whose school affiliation is unknown will not be used as substitutes. Also, unknown affiliated private schools in the orig inal samples will not get substitutes.

C. 

### A school can be a substitute for one and only one sample. (If a school is selected as a

substitute school fo r grade 12, for example, it can not be used as a substitute for either grade 4 or grade 8.)

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 20 of 23 D. 

### A public school su bstitute will always be in the same state as its original school.



# VI. Contingency Samples

The districts that are taking part in the TUDA program are volunteers. Thus it is possible that at

some point over the next few months, a given district might choos e to opt out of the TUDA

program for 2015 . However, it is not acceptable for all schools in such a district to decline NAEP,

as then the state estimates will be adversely affected. Thus to deal with this possibility, in each

TUDA district, subsamples of th e alpha sample schools will be identified as contingency samples. In

the event that the district withdraws from the TUDA program prior the selection of the student

sample, all alpha sampled schools from that district will be dropped from the sample, with t he

exception of those selected in the contingency sample. The contingency sample will provide a

proportional representation of the district, within the aggregate state sample. Student sampling in

those schools will then proceed in the same way as for the other schools within the same state.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 21 of 23 

# VI I

. Student Sampling

Students within the sampled schools will be selected with equal probability. The student sampling

parameters vary by sample type (Alpha, Beta

, Gamma, Delta, and Epsilon) and grade, as described

below.

Alpha Sample , Grade s 4 and 8 Schools (Except Puerto Rico)

A. 

### The sample size for each school will depend upon wh ich spiral the school was assigned, ei ther
RS, RM or NL.

B. 

### In schools assigned the RS spiral, all students will be selected, up to 100 . If th e school has

more than 100 students, 90 will be selected. In some schools, the school may be assigned more than one ‘hit’ in sampling. In these schools we will select a sample of size 90 times the number of hits, taking all students if this target is great er than or equal to 90 / 100 of the total

enrollment

.

C. 

### In schools assigned the R M spiral, all students, up to 70, will be selected. If the school has more than 70 students, 63 will be selected. In some schools, the school may be assigned more

than one ‘hit’ i n sampling. In these schools we will select a sample of size 63 times the number of hits, taking all students if this target is greater than or equal to 63/70 of the total

enrollment.

D. 

### In schools assigned the NL spiral, all students will be selected, up to 100. If the school has
more than 100 students, 90 will be selected.

E. 

### The student sampling procedures for DC at grade s 4 and 8 have yet to be determined.

Alpha Sample, Puerto Rico Grades 4 and 8

A. 

### All students, up to 55, will be selected.

B. 

### If the school has more than 55 students, a systematic sample of 50 students will be selected with no oversampling.

C. 

### All students will be assigned to assessment type PR.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 22 of 23 Delta Samples, Grades 4 and 8

A. 

### All students, up to 115 , will be selected.

B. 

### If the school has more than 115 students, a systematic sample of 105 students will be selected.

C. 

### All students will b e assigned to assessment type NL

.

Beta and Epsilon Samples, Grade 8

A. 

### In each school, a sample for computer

- based testing will be selected as follows: All students

up to 50 will be se lected. If there are more than 50 students enrolled, a sample of 5 0 students

will be selected.

B. 

### All stu dents will be assigned to the TB session.

Gamma and Delta Samples, Grade 12

A. 

### All students, up to 105 , will be selected.

B. 

### If the school has more than 105 students, a systematic sample of 95 students will be selected.

C. 

### All students will be assigned to assessment type NL

.



# VIII

. Weighting Requirements

Weighting activities for the 2015 NAEP assessments are not covered under the current NAEP

contracts. However, based on past experience, below is a description of the likely weighting

requirements

.

The Operational Samples

These samples will have a single set of weights for each subject (reading

, math , science, and meanin g
vocabulary at grades 4, 8

, and 12) applied to reflect probabilities of selection, school and student

nonresponse, any trimming, and the random assignment to the particular subject. There will be a

separate replication scheme s by grade and public/private. Such weights will also be derived for the

Puerto Rico KaSA assessment and the national KaSA special studies at grades 4 and 8.

### Mem orandum : 2015

- 1.1A/1.1B/1.1G/1.1D/1.1E February 26, 2014

Page 23 of 23 The Pilot Test Samples

Fully adjusted weight s will not be provided for the students in the TBA start

- up and science ICT

and HOT pilot test studies. However, preliminary weights will be available for these samples. These

will reflect the school and student selection probabilities, but with no adjustments for nonresponse.

The NIES Samples

The NIES survey samples consist of two gra de

- specific samples, comprising students selected for

each of the grade 4 and 8 operational samples. We will create one set of weights for each grade

specific sample. The NIES weights are designed for any aggregation of the NIES data, not involving

NAEP ac hievement data. NIES analyses involving NAEP achievement data should use the

appropriate NAEP operational weights.