# N

ATIONAL 

# C

ENTER FOR 

# E

DUCATION 

# STATISTICS 

# N ATIONAL 

# A SSESSMENT OF 

# E DUCATIONAL 

# P

ROGRESS 

# SUPPORTING STATEMENT PART B Request for System Clearance for NAEP Assessments f

or 2014- 2016 OMB# 1850- 0790 v.36

November 1 , 2012

### Table of **Contents**

B. COLLECTIONS OF INFORMATION EMPLOYING STATI STICALMETHODS .......... 1 1. Potential respondent universe ................................ ................................ ................................ 1 2. Procedures for collection of information ................................ ................................ ............... 1 3. Methods to maximize response rates and deal with issues of nonresponse ........................... 3 4. Tests of procedures or methods to be undertaken ................................ ................................ .. 3 5. Consultants on statistical aspects of the design ................................ ................................ ..... 4 Appendix A Statute Authorizing NAEP ................................ ................................ .................... A - 1 Appendix B External Consultants NAEP Design and Anal ysis Committee ................................ ................................ ................. B - 1 NAEP Validity

Studies Panel ................................ ................................ ................................ .. B - 2 NAEP Quality Assurance Technical Panel ................................ ................................ .............. B - 3 NAEP Socio - Economic Status Panel ................................ ................................ ....................... B - 4 NAEP National Indian Education Study Technical Review Panel ................................ .......... B - 5 NAEP Civics Standing Committee ................................ ................................ .......................... B - 6 NAEP Economics Standing Committee ................................ ................................ ................. B - 7 NAEP Geography Standing Committee ................................ ................................ ................. B - 8 NAEP Mathematics Standing Committee ................................ ................................ .............. B - 9 NAEP Reading Standing Committee ................................ ................................ .................... B - 10 NAEP Science Standing Committee ................................ ................................ ..................... B - 11 NAEP Technology

and Engineering Literacy Standing Committee ................................ .... B - 12 NAEP U.S. History

Standing Committee ................................ ................................ ............. B - 13 NAEP Writing Standing Committee ................................ ................................ ...................... B - 14 Appendix C Example of Sample Design Document (2013 Assessment) ................................ .. C - 1NAEP 2014 - 2016 System Clearance: Supporting Statement Part B page - 1 

### B. **COLLECTIONS OF**  **INFORMA**

**TION EMPLOYING ST AT**  **ISTICAL METHODS**  **1. Potential respondent universe**  The possible universe of student respondents for main NAEP is estimated to be 12 million at grades 4, 8, 12, attending approximately 154,000 public and private elementary and secondary

schools. NAEP test booklets are administered in selected public and private schools to a sample of fourth - , eighth - , and twelfth - grade students . Respondents are selected according to student sampling procedures with these possible exclusions:  The student is identified as an English language learner (ELL), but is prevented from participation in NAEP, even with accommodations allowed in NAEP.  The student is identified as having a disability which prevents participation in NAEP, even with accommodations as allowed in NAEP, and has an Individualized Education Plan (IEP) or equivalent classification, such as a Section 504 plan. NAEP relies upon the professional judgment of school administrators as to how students or schools should be classified. 

### 2. **Procedures for collection of information** Sampling The sampling information in this system clearance package is an overview of the sampling techniques and criteria used b y the current sampling contractor for the sampling in NAEP assessments. Each specific assessment will involve different selected samples based on the number of students and subjects in that particular assessment. Planned sample sizes are based on the need to obtain representative samples on which to report achievement information. For s ampling f

rames, NAEP uses the most current versions of the NCES Common Core of Data ( C CD ; public schools) and Private School Universe Survey (PSS ; private schools) files. Inaddition, to address the fact that the CCD file necessarily does not include the most recent changes to schools by the time ofNAEP 2014 - 2016 System Clearance: Supporting Statement Part B page - 2

the assessment , NAEP also conducts a surv ey of NAEP State Coordinators to check for additional schools in a sample of public school districts. Design Features As in the past, NAEP samples are based on multistage designs. The state assessment designs consist of stratified samples of public schools , where the stratification is derived from ty pe of location (urban/suburban/large town/small town/rural), proportion minority enrollment, school level achievement on statewide testing programs, and a measure of household income in the zip code area of the school. The second stage of sampling is the selection of the students from within each selected school. This is an equal probability

s ys tematic sample from among all students in the appropriate grade. For the national samples, a three - stage design is used . The first stage is the selection of primary sampling units (PSUs), which are individual counties or groups of contiguous counties. The second stage is the selection of schools within PSUs, and the third stage is the selection of students within schools. The following are characteristic features of NAEP sampling designs:  for state - level assessments, approximately equal sample sizes (2,500

–3,000 assessed students) from each participating state’s public schools, for e ach subject ;  for district - level assessments, sample sizes of approximately 1,000

–2,500 from each participating district ’s public schools, for each subject ;  sample sizes of approximately 6,000

– 12,000 for national - only operational subjects, depending on the size of the item pool;  in each school, some students to be assessed in each subject;  lists of schools obtained from the NCES CCD and PSS files;  schools grouped into strata;  schools assigned a measure of size;  sample selected with probability proportional to the measure of size; and  school stratification based on characteristics such as: ty pe of location, enrollment by race/ethnicity , and school achievement .NAEP 2014 - 2016 System Clearance: Supporting Statement Part B page - 3

Refer to a ppendix C for an example of the sampling procedures performed for the 2013 assessment. 

### 3. **Methods to maximize response rates and deal with issues of nonresponse**  NAEP attempts to minimize nonresponse of both students and schools. Chief state school officers and Local Education Agency

(LEA) superintendents are provided with lists of schools in the sample in their jurisdiction and their cooperation is requested. For the assessments, schools within each state will be selected and the chief state school officer and the NAEP state coordinator will be asked to solicit their cooperation. NCES will provide letters to states and districts in support of the operational and pilot tests. Since states and school districts receiving Title I funds are required to participate in the NAEP reading and mathematics assessments (grades 4 and 8) under the National Assessment of Educational Progress Act, NAEP response rates have improved for these assessments. Not all of the students in the sample will respond. Some will be unavailable during the sample time period because of absenteeism or other reasons. If a student decides no t to participate , the action will be recorded, but no steps will be taken to obtain participation . Response rates , in percentages, from a recent NAEP assessment are shown below: Grade 4 Grade 8 Grade 12 Student Response Rates 94 93 87 School Response Rates Public Schools 100 100 96 Private Schools 74 74 67 Note: The public school response rates for grades 4 and 8 rounded to 100, but were actually slightly lower (i.e., 99.8 percent). 

### 4. **Tests of procedures or methods to be undertaken** The 2014

–2016 administration procedures will be similar to those of previous NAEP assessments. If the final design for an assessment requires new procedures o r methods, they will be tested in a special study or a pilot test prior to the operational assessment.NAEP 2014 - 2016 System Clearance: Supporting Statement Part B page - 4 

### 5. **Consulta nts on statistical aspects of the design**  ETS, Fulcrum, Westat, and NCES staffs have collaborated on the statistical aspects of the design. The primary persons responsible are the following: Jay Campbell NAEP Executive Director, ETS Peggy Carr Associate Commissioner, NCES Patricia Etienne Program Director, Design, Analysis, and Reporting and Assessment Coordination, NCES Scott Ferguson NAEP Project Director, Fulcrum Arnold Goldstein Statistician, Assessment Reporting and Dissemination, NCES Andrew Kolstad Senior Technical Advisor, NCES Andreas Oranje NAEP Psychometric Director, ETS Keith F. Rust Vice - President, Westat Holly Spurlock Program Director, Assessment Operations, NCES Dianne Walsh Vice - President, Westat In addition, the NAEP Design and Anal ys is Committee and the NAEP Validity Studies panel members (see appendix B) have also contributed to NAEP designs on an on - going basis.