NATIONAL CENTER FOR 

EDUCATION STATISTICS (NCES)

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

WAVE 2 SUBMITTAL FOR 2014

VOLUME I 

SUPPORTING STATEMENT

OMB# 1850–0790 v.38

Grade 8

Student Technology and Engineering Literacy (TEL) Survey Questions

School TEL Questionnaires

Assessment Feedback Questions

August 1, 2013
ii

Contents

1. Explanation for This Submittal..............................................................................................1

Table 1: List of Components in Wave 2 ............................................................................................................1

2. Overview of Wave 2 NAEP 2014 Assessments....................................................................1

3. Sampling Information ............................................................................................................2

4. Information Pertaining to the Materials in This Submittal....................................................2

5. 2014 Wave 2 Burden Information .........................................................................................5

Table 2: Total Burden for Wave 2 .....................................................................................................................5

Table 3: Wave 2 Questionnaires - Per Respondent Burden Time......................................................................6

Table 4: Estimated Burden for NAEP 2014 Questionnaires Contained in This Submittal (Wave 2)................7

6. How, by Whom, and for What Purpose the Data Will Be Used............................................7

7. Estimates of Costs to the Federal Government......................................................................7

Table 5: Administration Cost Estimates (Wave 2 subjects)...............................................................................8

Appendix A: Committee Lists...................................................................................................9

Appendix B: Sample Parent/Guardian Notification Letter......................................................11

Appendix C: Sample Description of School Coordinator Responsibilities

Appendix D: Usability Review Recommendations on e-PAV

Appendix E: TEL Reports

Appendix F: 2014 Sampling Design Memo

Appendix G: Sample 2013 SD and ELL Worksheets and Instructions

Note: The core questions for grade 8 and the charter school questions were submitted and 

approved (OMB# 1850-0790 v.37, July 2013) as part of Wave 1 but are included 

here because they are a component of TEL.
1

1. Explanation for This Submittal

This document contains supplemental information pertaining to the 2014-2016 National Assessment of 

Educational Progress (NAEP) system clearance proposal. NAEP is administered by the National Center for 

Education Statistics (NCES) in the Institute for Education Sciences of the U.S. Department of Education. The 

National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP 

and determines the content framework for each assessment. The system clearance package was submitted in 

November 2012 and approved in March 2013 (OMB #1850-0790 v.36). The terms of clearance for OMB 

approvals state that NCES will publish a notice in the Federal Register allowing a 30-day public comment 

period on the details of each collection concurrent with submission to OMB. 

The 2014 submittal is divided into two waves to meet scheduling and question development requirements. 

The first wave contained the core,

1

social studies (civics, geography, U.S. history), and science (paperand-pencil) descriptions, burden, and questionnaires. Wave 2 contains the technology and engineering 

literacy (TEL) descriptions, burden, and materials, as well as information regarding school coordinator 

activities, including the collection of information on students with disabilities (SD) and English language 

learners (ELL), and the assessment feedback materials.

This Wave 2 submittal contains burden information and survey questions (also referred to as noncognitive 

questions) for the components of the NAEP 2014 assessments listed in Table 1.

Table 1: List of Components in Wave 2

Component Part of Volume II

Student Surveys: TEL - Grade 8 Part 1

School Surveys: School Characteristics and Policies (SCP), TEL, Charter School- Grade 8 Part 2

Assessment Feedback Surveys: School Coordinator Debriefing Interview, PostAssessment Follow-up Survey, Teacher/School Online Completion Feedback

Part 3

Note: There is no teacher questionnaire for TEL.

2. Overview of Wave 2 NAEP 2014 Assessments 

Much of the following broad overview of the 2014 NAEP assessments was included as part of the 2014–

2016 system clearance submittal. The Governing Board determines NAEP policy and the assessment 

schedule, and future Governing Board decisions may result in changes to some aspects of an assessment 

(e.g., which subjects are assessed in which years). However, the overall methodology and assessment 

process remains consistent. The NAEP assessments contain two kinds of questions: cognitive, or 

assessment, questions that measure student knowledge of an academic subject, and survey, or background, 

questions that gather information on demographic as well as classroom instructional procedures. The 

Governing Board is responsible for selecting and approving all of NAEP’s survey and cognitive questions. 

1 NCES collects student question data that are required by law (20 U.S.C. § 9622; i.e., race, ethnicity, and socioeconomic 

status) and provide a context for reporting student performance. These questions are referred to as core questions.
2

The NAEP assessments are currently supported by an alliance of organizations under contract with 

NCES, within the U.S. Department of Education. The main NAEP assessment reports current 

achievement levels and trends in student achievement at grades 4, 8, and 12 for the nation and, for certain 

assessments, states and select urban districts. These assessments follow subject-area frameworks

developed by the Governing Board and use the latest advances in assessment methodology. The subjectarea frameworks evolve to match instructional practices. 

3. Sampling Information

The 2014 Sampling Design Memo2(see appendix F) provides specific sampling information regarding the 

2014 NAEP assessments. It includes information on the following:

• Assessment types

• Primary Sampling Units Selection

• Stratification and Oversampling

• Overlap Control of Samples

• Substitute Samples

• Student Sampling Information

• Weighting Requirements 

4. Information Pertaining to the Materials in This Submittal 

Student Questionnaires 

The 2014 NAEP Technology and Engineering Literacy (TEL) Framework broadly defines technology and 

engineering literacy as the capacity to use, understand, and evaluate technology, and to understand 

technological principles and strategies needed to develop solutions and achieve goals. In 2014, the firstever national TEL assessment will be conducted at grade 8. This assessment will be administered on 

computers and include interactive scenario-based tasks, as well as discrete items. The TEL survey

questions were previously pretested in cognitive interviews (2011), tryouts (2012)

3

, and a pilot assessment 

(2013). Reviews of the TEL pilot results, as well as documentation regarding final decisions for inclusion 

in 2014, are provided in appendix E. Committees that helped define the cognitive and survey questions

pertaining to the 2014 subjects in Wave 2 are listed in appendix A. Core questions for grade 8 were 

submitted and approved as part of Wave 1 but are included here because they are a component of TEL.

Teacher Questionnaires

Teachers of students participating in the TEL assessments will not be administered teacher questionnaires.

2 The 2014 Sampling Memo (dated 3-25-2013) contains information related to the grades 4 and 12 social studies assessments, 

which were subsequently dropped.

3 Reports for the TEL cognitive interviews and tryouts were submitted in 2012 as part of the submittal for the pilot assessment 

(OMB #1850-0790 v.35).
3

School Questionnaires

School questionnaires are comprised of multiple sections, which include: 

School Characteristics and Policies (SCP; also referred to as the school core) – One version of SCP 

questions will be given at each grade. The SCP questions are comprised of trend or previously piloted 

questions. SCP questions were submitted and approved as part of Wave 1 but are included here because 

they are a component of the TEL school questionnaire.

TEL – The operational school questions are previously piloted questions.

Charter School – Charter school questions are comprised of trend questions that were administered in 

previous assessments. Note that the charter school section is only completed if the school is a charter 

school. Charter school questions were submitted and approved as part of Wave 1 but are included here 

because they are a component of the TEL school questionnaire.

All 2014 school questionnaires will be completed online. 

School Personnel Assessment Responsibilities

E-filing – Demographic data are collected for potential students who may participate in NAEP. This 

sample information is gathered at the state, district, or school level. 

Pre-Assessment Activities – Prior to the assessment, school personnel must prepare for the assessment. 

Part of the preparation includes logistical preparation for the upcoming assessment, information gathering, 

communication of teacher and student information, and review of NAEP procedures. 

NAEP is moving in the direction of paperless administrations. In previous assessment years, in 

preparation for conducting assessments, NAEP field staff conducted an in-person pre-assessment visit 

with each school. The in-person pre-assessment visit (PAV) involved the following procedures:

• Sending a package of hardcopy materials to the schools in early January

• Schools preparing materials for an in-person visit by NAEP field staff

• NAEP field staff spending at least an hour at each school conducting the pre-assessment visit

• Follow-up phone calls and emails to obtain all required information before the assessment day

In 2014, a new electronic pre-assessment visit (e-PAV) system is being developed so that school 

coordinators will provide requested administration information online, including logistical information, 

updates of student and teacher information, and the completion of inclusion and accommodation 

information. The benefits of adopting this approach include enhanced security of assessment items and 

student and school information, efficiencies in data collection, reduced burden on school staff, improved 

quality control, and reduced costs. The electronic process will provide the following enhancements:

• Materials will be provided to the school via a secure electronic application in early December

• Schools will provide information in a secure electronic application at their own pace

• NAEP field staff will review school entries and conduct a confirmation phone call with schools in 

January
4

The e-PAV system is being developed in consultation with the state coordinators, so that the system 

considers the needs of the states and the schools. In addition, the system has undergone a usability review, 

the recommendations of which are included in appendix D. These recommendations have been 

incorporated into subsequent development of the e-PAV system. 

Multiple sources of support will be available to schools as part of the electronic process, including:

• dedicated Help Desk, accessible by email, phone, or live chat;

• NAEP field staff, accessible by phone or email; and

• NAEP state coordinators, accessible by phone or email.

In some instances, NAEP field staff will visit schools that are having trouble accessing the e-PAV system 

and help the schools complete the electronic process in-person.

In order to evaluate the success of the e-PAV and inform refinements to the e-PAV process and 

procedures for 2015, NAEP will be conducting a number of evaluation activities, including

• debriefing with NAEP state coordinators, field staff, and school coordinators;

• analyzing observable data from the electronic system (such as time spent on each activity);

• monitoring in-person visits made by field staff for frequency, issues that prompted the visits, and 

feedback from both school and field staff; and

• reviewing Help Desk issues.

A sample brochure communication from the NAEP state coordinators to the participating schools 

describing the pre-assessment and e-filing activities is included as appendix C and will be finalized prior 

to the 2014 assessment year. It outlines what the school coordinator will be responsible for.

SD and ELL information – The SD and ELL inclusion information is provided by the teachers or school 

personnel most knowledgeable about the school’s SD and ELL students. This information will then be 

entered into the e-PAV system by the school coordinator rather than completed via hardcopy forms. With 

the e-PAV system, inclusion guidelines can now be customized on a state-by-state basis. The SD and ELL 

information provided can be formatted with appropriate skip patterns and consistency checks, allowing 

field staff to monitor exclusions in real time. The e-PAV system is currently under development but the 

types of SD and ELL information gathered will be the same as was previously obtained on the SD and 

ELL worksheets. Examples of the 2013 SD and ELL worksheets and instructions, along with samples of 

state specific summary guidelines, are included for reference purposes in appendix G.

Assessment Feedback Activities

School Coordinator Debriefing Interview – After each assessment, the field staff will meet with the school 

coordinator for a debriefing interview. The purpose of this interview is to obtain feedback on

• how well the assessment went in that school,

• usefulness of NAEP materials (publications, letters, etc.),

• school staff feedback,
5

• preparation activities,

• strategies utilized for increasing participation, and

• any issues that were noted.

Post-assessment Follow-up Survey – As part of the on-going quality control of the assessment process, 

25% of the schools will be randomly selected for an additional follow-up survey. Survey questions solicit 

feedback on pre-assessment, assessment, and procedural processes. A sample survey, which will be 

updated for 2014, is included in Volume II, Part 3.

Teacher/School Online Completion Feedback – As discussed earlier, in 2014, all teacher and school 

questionnaires will be completed online. An evaluation question will be added to the end of the 

questionnaires, seeking feedback regarding the online completion process. 

5. 2014 Wave 2 Burden Information

The Wave 2 burden includes TEL student and school survey burden, school coordinator burden for the 

electronic pre-assessment, electronic filing processes, debriefing interview, and post-assessment followup survey, and school personnel burden for completion of SD and ELL information. Note that while 

Wave 2 does also include a question to teachers and school administrators regarding feedback on the 

online completion process, the addition of one question does not significantly impact the already 

approved burden and, therefore, there is no additional burden associated with the administration of this 

question. The total burden for the Wave 2 materials is shown in Table 2.

Table 2: Total Burden for Wave 2

Respondent Category Number of Respondents Total Burden Hours

Student 20,000 10,000

School 865 433

School Coordinator (for pre-assessment activities,

e-filing and assessment feedback)

1,398 3,745

School Personnel (for SD & ELL data completion) 1,398 1,434

Total 23,661 15,612

Further breakdown of the survey per-respondent and overall burden times are included in Tables 3 and 4, 

respectively, and include the following categories:

Students – For the grade 8 TEL assessment, students are presented two 30-minute cognitive sections 

followed by a 15-minute survey questionnaire section, which includes core and TEL-specific items.

Additional student burden time is included for receiving instructions, reading directions, and taking a 

tutorial for using the system. For computer-based assessments, this additional burden is estimated at 15 

minutes, thus, the total burden for students is 30 minutes4for computer-based assessments.

4 The time for the cognitive sections is not part of the burden calculation. 
6

Appendix B includes a sample parental notification letter regarding NAEP. The letter will be adapted for 

each grade/subject combination and the school principal may edit it. However, the information regarding 

confidentiality and the appropriate law reference will remain unchanged.

Principals/Administrators – The school administrators in the sampled TEL schools will be asked to 

complete a school questionnaire. The school questionnaire burden is estimated to be 30 minutes.

Table 3: Wave 2 Questionnaires – Survey Per Respondent Burden Time

Respondent Category Time per Respondent

Grade 8

Student Core + TEL 15 minutes

Student Directions and Set-up 15 minutes

School Questionnaire 30 minutes

School Personnel: Pre-Assessment, Electronic Filing Activities, and Assessment Feedback Activities

Pre-Assessment Activities – As described in section 4, school personnel perform multiple activities in 

preparation for the assessment, including information gathering, communication of teacher and 

student information, and review of NAEP procedures. These pre-assessment activities are estimated to 

require two hours of school personnel time. 

E-filing –If the survey sample information needed for e-filing is obtained at the school or district 

level, school or district personnel will incur some burden. This e-filing burden is estimated at one hour

per school. For the most recent national-only NAEP assessment, approximately 52 percent of schools 

and districts participated in the e-filing process. Thus, 52 percent of total schools were used for efiling burden calculation purposes.

Assessment Feedback Activities – As described in section 4, school personnel will participate in a 

School Coordinator Debriefing Interview immediately following the assessment. The estimated 

burden for this interview is 7 minutes. In addition, 25% of the school personnel will be contacted for a 

more in-depth Post-assessment Follow-up Survey. The burden for this survey is estimated at 10 

minutes per school.

School Personnel: SD and ELL – SD and ELL burden is the average number of hours school personnel 

spend on completing the SD/ELL information for students identified as SD and/or ELL (estimated at 10 

minutes per student). The overall SD/ELL burden is a factor of the number of students identified as 

SD/ELL. Based on most recent data, 22 percent of grade 4 students, 17 percent of grade 8, and 13 percent 

of grade 12 students were identified as SD/ELL. 

All 2014 school personnel activities (pre-assessment activities, e-filing, feedback activities, and SD and 

ELL information collection) are included as part of this Wave 2 submittal. Therefore, the burden for 

school personnel who complete them is included with Wave 2 for all 2014 assessments (i.e., those 

included in Wave 1 and those included in Wave 2). 
7

Table 4: Estimated Burden for NAEP 2014 Activities Described in This Submittal (Wave 2)

Assessment / Grade

 # of 

Students

Avg. 

time per

resp.

Student

Burden

# of 

Teachers 

Avg. 

time per

resp.

Teacher

Burden

# of 

Schools

Avg. 

time per

resp.

School

Burden

# of 

Schools

School

Burden

1

# of 

Schools

Avg. 

time per 

resp.

SD/ELL 

Burden

2

Total 

Burden

(in 

hours)

(in 

hours)

(in 

hours)

(in 

hours)

(in hours) (in hours)

TEL (grade 8) 20,000 30 min 10,000 N/A N/A N/A 865 30 min 433 865 2,317 865

10 min

per 

student

567 13,317

Additional school 

personnel burden

associated with preassessment and 

assessment feedback

activities for Wave 1 

schools3

533 1,428 533

10 min

per 

student

867 2,295

 Totals 20,000 10,000 0 0 865 433 1,398 3,745 1,398 1,434 15,612

Total Respondents: 23,661 Total Burden: 15,612

Due to rounding, some totals may differ slightly from sum of subtotals.

3

 SD and ELL burden for Wave 1 is based on 30,500 total students (1,500 at grades 4 and 12 and 27,500 at grade 8). 

2

SD and ELL burden is the number of hours that school personnel spend to complete the SD/ELL information for students identified as SD and/or ELL (estimated at 10 minutes per 

student). The overall SD/ELL burden is a factor of the number of students identified as SD/ELL. Based on most recent data, 22 percent of grade 4 students, 17 percent of grade 8, and 13 

percent of grade 12 students were identified as SD/ELL.

SD/ELL (school personnel)

School Questionnaire (school 

principals)

pre-assessment, 

e-filing,

& assessment 

feedback

Students Teachers (school coor)

1

 Pre-assessment burden is 2 hrs, e-filing is 1 hr (for 52% of schools, based on recent data), school coordinator debriefing is 7 min, and post-assessment follow-up survey is 10 min (for 

25% of the schools).

6. How, by Whom, and for What Purpose the Data Will Be Used

The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas 

assessed for use in monitoring educational progress. In addition to reporting overall results of student 

performance and achievement, NAEP also reports student performance results for various subgroups of 

students and on various educational factors. The Governing Board sets guidance for what is asked in the 

questions. NCES is responsible for developing the questions and for selecting the final set of questions. 

The questions are designed to (a) provide the information for disaggregating data according to categories 

specified in the legislation,

5

(b) provide contextual information that is subject specific (e.g., reading, 

mathematics) and has an impact and known relationship to student achievement, and (c) provide policyrelevant information specified by the Governing Board.

In the original request to OMB for system clearance, NCES requested approval of the data-gathering 

instruments to be used in the 2014–2016 national- and state-level NAEP assessments. An expanded 

description of the purposes of the data is provided in the system clearance submittal. 

7. Estimates of Costs to the Federal Government 

Table 5 provides estimates for the Wave 2 portion of the 2014 administrations.

5 Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).
8

Table 5: Administration Cost Estimates (Wave 2 subjects) 

Activity Provider Estimated Costs

Web support and maintenance related to the 

TEL assessments

Fulcrum IT Services $983K

Field administration, including administration of

assessments and data collection from the field

Westat (the Data Collection 

contractor and the NAEP 

Support and Service Center 

[NSSC] contractor)

$864K

Totals $1.847 million
2014 Wave 2 Supporting Statement: Appendix A 9

Appendix A: Committee Lists

NAEP Background Variable Committee 

Name Affiliation

Patricia Alexander University of Maryland 

Arthur Applebee State University of New York at Albany 

Claudia Buchman Ohio State University

Lizanne Destefano University of Illinois

Robert Hauser University of Wisconsin-Madison

Kathleen Heid Penn State University

Henry Levin Columbia University

Peter Levine Tufts University

Linda Levstik University of Kentucky

Samuel Lucas University of California-Berkeley

Senta Raizen WestEd,

Washington, DC

Andrew Sum Northeastern University 

William Walstad University of Nebraska-Lincoln 
2014 Wave 2 Supporting Statement: Appendix A 10

NAEP Technology and Engineering Literacy (TEL) Committee 

Name Affiliation

Keith Barton Indiana University

John Behrens Pearson e-LEADS Center,

Mishawaka, IN 

Brooke Bourdelat-Parks BSCS (Biological Sciences Curriculum Study),

Colorado Springs, CO

Barbara Bratzel Shady Hill School

Cambridge, MA 

Lewis Chappelear James Monroe High School,

North Hills, CA

Britte Haugan Cheng SRI International,

Menlo Park, CA 

Meredith Davis North Carolina State University

Chris Dede Harvard Graduate School of Education

Richard Duran University Of California, Santa Barbara

Maurice Frazier Oscar Smith High School,

Chesapeake, VA 

Camilla Gagliolo Arlington Public Schools,

Arlington, VA 

Christopher Hoadley New York University

Eric Klopfer Massachusetts Institute of Technology

Beth McGrath Stevens Institute of Technology,

Hoboken, NJ

Greg Pearson National Academy of Engineering,

Washington, DC 

John Poggio University of Kansas

Erin Reilly Annenberg School for Communication & Journalism,

Los Angeles, CA 

Troy Sadler Missouri University Science Education Center,

Columbia, MO 

Kimberly Scott Arizona State University

Teh-Yuan Wan New York State Education Department,

Albany, NY 
2014 Wave 2 Supporting Statement: Appendix B 11

Appendix B: Sample Parent/Guardian Notification Letter

NAEP (NAEP Assessment Year) PARENT/GUARDIAN NOTIFICATION LETTER

(School Letterhead)

(Insert Date Here)

Dear Parent or Guardian:

We are pleased to notify you that (school name) has been selected to represent schools across 

the nation by participating in the National Assessment of Educational Progress (NAEP). First 

administered in 1969, NAEP is the largest nationally representative assessment of what students 

know and can do in various subject areas. It is administered by the National Center for Education 

Statistics, within the U.S. Department of Education. NAEP is different from state assessments 

because it provides a common measure of student achievement across the country. The results of 

NAEP are released as The Nation's Report Card, which provides information about student 

achievement to educators, parents, policymakers, and the public.

In our school, the NAEP assessment will be given on (date) in (subject). Your child (may be/has 

been) selected to take the assessment. In addition to subject area questions, students will be 

asked some questions about themselves and their educational experience. These questions 

provide contextual information for the assessment, as well as information that may be related to 

students’ learning. If you would like to view sample subject area and student questions, please 

visit http://nationsreportcard.gov/parents.asp.

It will take about (assessment time) minutes for most students to participate in the assessment. 

The results are completely confidential (in accordance with the Confidential Information 

Protection provisions of Title V, Subtitle A, Public Law 107-347), and the information provided will 

be used for statistical purposes only. Your child’s grades will not be affected. Your child may be 

excused from participation for any reason, is not required to complete the assessment, and may 

skip any question. While NAEP is voluntary, we depend on student participation to provide an 

accurate measure of student achievement that will inform improvements in education. Your child 

will represent many other students, so participation is very important. However, if you do not 

want your child to participate, please notify me in writing by (date). 

There is no need to study in preparation for NAEP. We do, however, ask parents to encourage 

their children to do their best and get plenty of rest the night before the assessment. A brochure 

that explains what participation in NAEP means for you and your child is available at 

http://nces.ed.gov/nationsreportcard/pdf/parents/2012469.pdf. Should you have questions, 

please contact me at (telephone number) or at (e-mail address).

We are excited that our school will be participating in NAEP and pleased that your child (may 

be/has been) selected. We know that (school name)'s students will help us show what our 

nation’s students know and can do.

Sincerely,

School Principal