# NATIONAL CENTER FOR

**EDUCATION STATISTICS (NCES)**  **NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS**  **WAVE 2**  **SUBMITTAL FOR 2014** 

**VOLUME I**  **SUPPORTING STATEMENT OMB# 1850** 

– 

### 0790 **v.38**

**Grade 8** Student Technology

and Engineering Literacy ( TEL) Survey Questions School TEL Questionnaire s Assessment Feedback Questions

August 1 , 2013ii

**Contents** 1. Explanation for This Submittal ................................ ................................ .............................. 1 Table 1: List of Components in Wave 2 ................................ ................................ ................................ ............ 1 2. Overview of Wave 2 NAEP 2014 Assessments ................................ ................................ .... 1 3. Sampling Information ................................ ................................ ................................ ............ 2 4. In formation Pertaining to the Materials in This Submittal ................................ .................... 2 5. 2014 Wave 2 Burden Information ................................ ................................ ......................... 5 Table 2: Total Burden for Wave 2 ................................ ................................ ................................ ..................... 5 Table 3: Wave 2 Questionnaires - Per Respondent Burden Time ................................ ................................ ...... 6 Table 4: Estimated Burden for NAEP 2014 Questionnaires Contained in This Submittal (Wave 2) ................ 7 6. How, by

Whom, and for What Purpose the Data Will Be Used ................................ ............ 7 7. Estimates of Costs to the Federal Government ................................ ................................ ...... 7 Table 5: Administration Cost Estimates (Wave 2 subjects) ................................ ................................ ............... 8 Appendix A: Committee Lists ................................ ................................ ................................ ... 9 Ap pendix B: Sample Parent/Guardian Notification L etter ................................ ...................... 11 Appendix C: Sample Description of School Coordinator Responsibilities Appendix D: Usability

Review Recommendations on e - PAV Appendix E: TELReports Appendix F: 2014 Sampling Design Memo Appendix G: Sample 2013 SD and ELL Worksheets and Instructions **Note** : The core questions for grade 8 and the charter sch ool questions were submitted and approved ( OMB# 1850 - 0790 v.37, July 2013) as part of Wave 1 but are included here because they

are a component of TEL .1

**1. Explanation**  **for This Submittal** This document contains supplemental information pertaining to the 2014 - 2016 National Assessment of Educational Progress (NAEP) s ys tem clearance proposal. NAEP is administered by the National Center for Education Statistics (NCES) in the I nstitute for Education Sciences of the U.S. Department of Education. The National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP and determines the content framework for each assessment. The system clearance package was submitted in November 2012 and approved in March 201 3 (OMB #1850 - 0790 v. 36 ). The terms of clearance for OMB approvals state that NCES will publish a notice in the Federal Register allowing a 30 - day public comment period on the details of each collection concurrent with submission to OMB. The 2014 submittal is divided into two waves to meet scheduling and question development requirements. The first wave contain ed the core , 1 social studies ( c ivics, g eograph y, U.S. h istory), and science (paper and - pencil) descriptions, burden , and questionnaires . W ave 2 contain s the technology and engineering literacy (TEL) descriptions, burden, and materials, as well as informati on regarding school coordinator activities, including the collection of information on students with disabilities (SD) and English language learners (ELL) , and the assessment feedback materials . This Wave 2 submittal contains burden information and survey questions ( also referred to as noncognitive questions) for the components of the NAEP 201 4 assessments listed in Table 1 . **Table 1: List of Components in Wave**  **2 Component**  **Part of**  **Volume II Student Surveys**  : TEL - Grade 8 Part 1 **School Surveys**  **:** School Characteristics and Policies (SCP), TEL, Charter School - Grade 8 Part 2 **Assessment Feedback Surveys :**  School Coordinator Debriefing Interview, Post A ssessment Follow -up Survey, Teacher/School Online Completion Feedback Part 3 Note : There is no t eacher questionnaire for TEL. **2. Overview of Wave 2**  **NAEP 201**  **4 Assessments**  Much of the following broad overview of the 2014 NAEP assessments was included as part of the 201 4

– 201 6 sy stem clearance submittal. The Governing Board determines NAEP policy and the assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, t he overall methodology

and assessment process remain s consistent. The NAEP as sessments contain two kinds of questions: cognitive , or assessment , questions that measure student knowledge of an academic subject, and survey , or background , questions that gather information on demographic as well as classroom instructional procedures. The Governing Board is responsible for selecting and approving all of NAEP’s survey and cognitive questions .

1 NCES collects student question data that are required by law ( 20 U.S.C. § 9622 ; i.e., race, ethnicity, and socioeconomic status) and provide a context for reporting student performance. These questions are referred to as core questions.2

The NAEP assessments are currentl y supported by an alliance of organizations under contract with NCES, within the U.S. Department of Education. The main NAEP assessment reports current achievement levels and trends in student achievement at grades 4, 8, and 12 for the nation and , for certain assessments, states and select urban districts. These assessments follow subject - area frameworks developed b y the Governing Board and use the latest advances in assessment methodology . The subject area frameworks evolve to match instructional practices. **3. Sampling Information** The 2014 Samplin g Design Memo 2 (see appendix F ) provides specific sampling information regarding the 2014 NAEP assessments. It includes information on the following :

• Assessment ty

pes • Primary

Sampling Units Selection

• Stratification and Oversampling

• Overlap Control of Samples

• Substitute Samples

• Student Sampling Information

• Weighting Requirements **4 . Information Pertaining to**  **t he**  **Materials in This Submittal**  **Student Question**  **naires** The 2014 NAEP Technology and Engineering Literacy (TEL) Framework broadly defines technol ogy

and engineering literacy

as the capacity to use, understand, and evaluate technology , and to understand technological principles and strategies needed to develop solutions and achieve goals

. In 2014, the first ever national TEL assessment will be condu cted at grade 8 . This assessment will be administered on computers and include interactive scenario - based tasks , as well as discrete items. The TEL survey questions were previously pretested in cognitive interviews (2011) , tryout s ( 2012 ) 3 , and a pilot assessment (2013) . Review s of the TEL pilot results, as well as documentation regarding final decisions for inclusion in 2014, are provided in appendix E . Committees that helped define the cognitive and survey questions pertaining to the 2014 subjects in W ave 2 are listed in appendix A . Core questions for grade 8 were submitted and approved as part of Wave 1 but are included here because they are a component of TEL . **Teacher Questionnaires** Teachers of students partic ipating in the TEL assessments will not be administered teacher questionnaires.

2 The 2014 Sampling Memo (dated 3 - 25 - 2013) contains information related to the grades 4 and 12 social studies assessments, wh ich were subsequently dropped. 3 Reports for the TEL cognitive intervie ws and tryouts we re submitted in 2012 as part of the submittal for the pilot assessment (OMB #1850 - 0790 v.35).3

**School Questionnaires** School questionnaires are comprised of multiple sections, which include: School Characteristics and Policies (SCP; also referred to as the school core)

– One version of SCP questions will be given at each grade. The SCP questions are comprised of trend or previousl y piloted questions. SCP questions were submitted and approved as part of Wave 1 but are included here because they are a component of the TEL school questionnaire. TEL

– The operational school questions are previously piloted questions. Charter School

– Charter school questions are comprised of trend questions that were administered in previous assessments. Note that the charter school section is only completed if the school is a charter school. Charter school questions were submitted and approved as part of Wave 1 but are included here because they

are a component of the TEL school questionnaire. All 2014 school questionnaires will be completed online. **School Personnel Assessment**  **Responsibilities** E - filing

– Demographic data are collected for potential students who may participate in NAEP. This sample information is gathered at the state, district, or school level. Pre - Assessment Activities

– P rior to the assessment, school personnel must prepare for the assessment. Part of the preparation includes logistic al preparation for the upcoming assessment , information gathering, communication of teacher and student information, and review of NAEP proce dures . NAEP is moving in the direction of paperless administrations. Inprevious assessment years, in preparation for conducting assessments, NAEP field staff conducted an in - person pre - assessment visit with each school. The in - person pre - assessment visit (PAV) involved the following procedures:

• Sending a package of hardcopy materials to the schools in early January

• Schools preparing materials for an in - person visit by NAEP field staff

• NAEP field staff spending at least an hour at each school conducting th e pre - assessment visit

• Follow - up phone calls and emails to obtain all required information before the assessment day In 2014, a new electronic pre - assessment visit (e - PAV) s ys tem is being developed so that school coordinators will provide requested administration information online , including logistical information, updates of student and teacher information, and the completion of inclusion and accommodation information. The benefits of adopting this approach include enhanced security of assessment i tems and student and school information, efficiencies in data collection, reduced burden on school staff, improved quality control , and reduced costs . The electronic process will provide the follo wing enhancements:

• Materials will be provided to the school via a secure electronic application in earl y December

• Schools will provide information in a secure electronic application at their own pace

• NAEP field staff will review school entries and conduct a confirmation phone call with schools in January4

The e - PAV sy stem is being developed in consultation with the state coordinators, so that the sy stem considers the needs of the states and the schools. In addition, the sy stem has undergone a usability review, the recommendations of which are included in appendix D . These recommendations have been incorporated into subsequent development of the e - PAV s ys tem. Multiple sources of support will be available to schools as part of the electronic process, including :

• d edicated Help Desk, accessible by email, phone, or live chat ; • NAEP field staff, accessible by phone or email ; and

• NAEP state c oordinators, accessible by phone or email . In some instances, NAEP field staff will visit schools that are having trouble accessing the e - PAV system and help the schools complete the ele ctronic process in - person. In order to evaluate the success of the e - PAV and inform refinements to the e - PAV process and procedures for 2015 , NAEP will be conducting a number of evaluation activities, including

• d ebriefing with NAEP s tate c oordinators , fiel d staff , and school coordinators ; • anal yz ing observable data from the electronic s ys tem (such as time spent on each activity) ; • m onitoring in - person visits made by field staff for frequency , issues that prompted the visits, and feedback from both school and field staff ; and

• reviewing Help Desk issues . A sample brochure communication from the NAEP state coordinators to the participating schools describing the pre - assessment and e - filing activities is included as appendix C and will be finalized prior to the 2 014 assessment year . It outlines what the school coordinator will be responsible for . SD and ELL information

– The SD and ELL inclusion information is provided by the teachers or school personnel most knowledgeable about the school’s SD and ELL students . This information will then be entered into the e - PAV sy stem by the school coordinator rather than completed via hardcopy forms. With the e - PAV s ys tem , i nclusion guidelines can now be customized on a state - by- state basis. The SD and ELL information provided can be formatted with appropriate skip patterns and consistency checks, allowing field staff to monitor exclusions in real time. The e - PAV sy stem is currentl y under development but the ty pes of SD and ELL information gathered will be the same as was previ ously obtained on the SD and ELL worksheets. Examples of the 2013 SD and ELL worksheets and instructions, along with samples of state specific summary

guidelines, are included for reference purposes in appendix G . **Assessment Feedback Activities** School Coordinator Debriefing Interview

– After each assessment , the field staff will meet with the school coordinator for a debriefing interview. The purpose of this interview is to obtain feedback on

• how well the assessment went in that school , • usefulness of NAEP materials (publications, letters, etc.) , • school staff feedback ,5

• preparation activities , • strategies utilized for increas ing participation , and

• any issues that were noted . Post - assessment Follow - up Survey

– As part of the on - going quality control of the assessment process, 25 % of the schools will be randoml y selected for an additional follow - up survey . Survey questions solicit feedback on pre - assessment, assessment, and procedural processes. A sample survey , which will be updated for 2014, is inclu ded in Volume II, Part 3. Teacher/School Online Completion Feedback

– As discussed earlier, i n 2014 , all teacher and school questionnaires will be completed online. An evaluation question will be added to the end of the questionnaires , seeking feedback reg arding the online completion process. **5 .**  **2014 Wave**  **2 Burden Information**  The Wave 2 burden includes TEL student and school survey burden , school coordinator burden for the electronic pre - assessment , electronic filing processes , debriefing interview, and po st - assessment follow up survey , and school personnel burden for completion of SD and ELL information. Note that while Wave 2 does also include a question to t eachers and school administrators regarding feedback on the online completion process, the addition of one question does not significantly impact the already approved burden and, therefore, ther e is no additional burden associated with the administration of this question. The total burden for the Wave 2 materials is shown in Table 2. **Table 2**  **: Total B**  **urden for Wave 2** 

**Respondent Category Number of Respondents**  **Total Burden Hours** Student 20,000 10,000 School 865 433 School Coordinator (for pre - assessment activities , e - filing and assessment feedback) 1 , 398 3, 745 School Personnel (for SD & ELL data completion) 1 , 398 1,434 **Total 23,661**  **15, 61**  **2** Further breakdown of the survey per - respondent and overall burden times are included in Tables 3 and 4 , respectivel y, and include the following categories: Students

– For the grade 8 TEL assessment , s tudents are presented two 30 - minute cognitive sections followed b y a 15 - minute survey questionnaire section, which includes core and TEL - specific items. Additional student burden time is included for receiv ing instructions, reading directions , and taking a tutorial for using the s ys tem . F or computer - based assessments, this additional burden is estimated at 15 minutes , thus , the total burden for students is 30 minutes 4 for computer - based assessments.

4 The time for the cognitive sections is not part of the burden calculatio n.6

Appendix B i ncludes a sample parental notification letter regarding NAEP. The letter will be ad a pted for each grade/subject combination and the school principal may edit it. However, the information regarding confidentiality

and the appropriate law reference will remain unchanged . Principals/Administrators

– The school administrators in the sampled TEL schools will be asked to complete a school questionnaire. The school questionnaire burden is estimated to be 3 0 minutes . **Table 3**  **: Wave 2**  **Questionnaires –**  **Survey Per**  **Respondent Burden Time Respondent Category**  **Time per Respondent Grade 8**  Student Core + TEL 1 5 minutes Student Directions and Set - up 15 minutes School Questionnaire 30 minutes School Personnel : Pre - Assessment , Electronic Filing Activities , and Assessment Feedback Activities Pre - Assessment Activities

– As described in section 4, school personnel perform multiple activities in preparation for the assessment, including information gathering, communication of teacher and student information, and re view of NAEP procedures . The se pre - assessment a ctivities are estimated to require two hour s of school personnel time. E - filing

–If the survey

sample information needed for e - filing is obtained at the school or district level, school or district personnel will incur some burden . This e - filing burden is estimated at one hour per school . For the most recent national - onl y NAEP assessment, a pproximately 52 percent of schools and districts participated in the e - filing process. Thus, 52 percent of total schools were used for e filing burden calculation purposes. Assessment Feedback Activities

– As described in section 4, school personnel will participate in a School Coordinator Debriefing Interview immediately following the assessment. The estimated burden for th is interview is 7 minutes. Inaddition, 25% of the school personnel will be contacted for a more in - depth Post - assessment Follow - up Survey . The burden for this survey is estimated at 10 minutes per school. School Personnel: SD and ELL

– SD and ELL burden i s the average number of hours school personnel spend on completing the SD/ELL information for students identified as SD and/or ELL (estimated at 10 minutes per student). The overall SD/ELL burden is a factor of the number of students identified as SD/ELL. Based on most recent data, 22 percent of grade 4 students, 17 percent of grade 8, and 13 percent of grade 12 students were identified as SD/ELL. All 2014 school personnel activities (pre - assessment activities, e - filing, feedback activities, and SD and ELL information collection) are included as part of this Wave 2 submittal . Therefore , the burden for school personnel who complete them is included with Wave 2 for all 2014 assessment s (i.e., those included in Wave 1 and those included in Wave 2) .7

**Table 4**  **: Estimated Burden for NAEP 2014 Activities Described**  **in This Submittal (Wave 2**  **)**

**As se**  **ssm en**  **t / Gr ad** **e # of** 

**Stu de**  **nt s**  **Avg.**

**ti me**  **p er**  **re sp** **. Stu** **de nt**  **Bu rde**  **n # of**  **Te ach** **er s**  **Avg. ti** **me p** **er re**  **sp .**  **Te ach** **er Bu** **rde n**  **# of**

**Sch ools**  **Av g.**  **ti me**  **p er**  **re sp** **. Sch** **ool Burde**  **n # of** 

**Sch ools**  **Sch ool**  **Burde n** **1 # of** 

**Sch ools**  **Avg. ti** **me pe**  **r re**  **sp .**  **SD/ ELL**  **Burde n** 

**2 Total**  **Burde n**  **(i n**  **ho urs**  **) (i** **n ho**  **urs )**  **(i n**  **ho urs**  **) (i** **n**

**ho urs**  **) (i** **n ho**  **urs )**  **(i n**  **ho urs**  **)** TEL (g rade 8) 20,000 30 m in 10,000 N/A N/ A N/A 865 30 m in 433 865 2,317 865 10 m in per st ud en t 567 13,317 Additional school person ne l bu rden associated with

preassessm en t an d assessm en t feedback activit ies f or Wave 1 sch ools 3 533 1,428 533 10 m in per st ud en t 867 2,295

**Total s**  **20,000 10,000**  **0 0**  **865 433**  **1,398 3,745**  **1,398 1,434**  **15,612 Total Re**  **sp on** **de nt**  **s :**  23,661 **Total Bu rde**  **n:** 15,612 Due to rou

ndin g, som e t ot als m ay dif fer slig htl y fr om su m of su btotals.

3SD an d EL L bu rden f or Wave 1 is based on

30,500 t otal stu den ts (1,500 at

g rades 4 an

d 12 and 27,500 at

g rade 8). 2 SD an d EL L bu rden is t hen umb er of h ou rs t hat sch ool person

ne l spen d to com plet e th e SD/EL L in formati on f or st ud en ts i den ti fi ed as SD an

d/ or EL L (estim at ed at 10 m in ute s per st ud en t) . Th e overall SD/EL

L bu rden is a f actor of t hen umb er of stu den ts iden ti fi ed as SD/ EL

L. Based on

m ost recen t data, 22 percen

t of grade 4 students, 17 percent of g rade 8, and 13 percen t of g rade 12 stu

den ts w ere iden

ti fi ed as SD/EL

L. **SD/ ELL (sch** **ool p** **er son** **ne l)**  **Sch ool Q**  **ue sti** **on na**  **ire (sch** **ool princ**  **ip als**  **) pre**  **-asse ssm**  **en t,**  **e-fi ling,**  **& asse ssm** 

**en t**  **feed ba**  **ck (sch** **ool coor**

**) Stu** **de nt**  **s Te**  **ach er**  **s**

1 Pre-assessm

en t bu rden is 2 h rs, e-f ilin g i s 1 h r (f or 52% of sch ool s, based on

recen t data), sch

ool coordinator debrief

in g is 7 m in, an d post-assessm

en t fol low-u p su rvey i s 10 m in (f or 25% of t hesch ools).

**6 . How,**  **by Whom, and for What Purpose the Data Will Be Used** The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring education al progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. The Governing Board sets guidance for what is asked in the questions . NCES is responsible for developing the question s and for selecting the final set of questions. T h e questions are designed to (a) provide the information for di saggregating data according to categories specified in the legislation , 5 (b) provide contextual information that is subject specific (e.g. , reading, mathematics) and has an impact and known relationship to student achievement, and (c) provide policyrelevant information specified by the Governing Board. In the original request to OMB for s ys tem clearance, NCES requested approval of the data - gathering instruments to be used in the 2014

–2016 national - and state - level NAEP assessments. An expanded description of the purposes of the data is provided in the sy stem clearance submittal. **7 . Estimates of Costs to the Federal Government**  Table 5 provides estimates for the Wave 2 portion of the 2014 administrations.

5 Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).8

**Table 5**  **: Administration Cost Estimates**  **(Wave 2**  **subje cts)**  **Activity Provider**  **Estimated Costs** W eb support and maintenance related to the TEL assessments Fulcrum IT Services $ 983K Field administration , including administration of assessments and data collection from the field Westat (the Data Collection contractor and the NAEP Support and Service Center [ NSSC ] contractor) $ 864K **Totals $1.847 million** 201 4 Wave 2 Supporting Statement : Appendix A 9

**Appendix A: Committee Lists NAEP Background Variable Committee**  **Name Affiliation**  Patricia Alexander University of Maryland Arthur Applebee State University of New York at Albany Claudia Buchman Ohio State University Lizanne Destefano University of Illinois Robert Hauser University of Wisconsin - Madison Kathleen Heid Penn State Unive rsity Henry L evin Columbia University Peter Levine Tufts University Linda Levstik University of Kentucky Samuel Lucas University of California - Berkeley Senta Raizen WestEd, Washington, DC Andrew Sum Northeastern University William Walstad University of Nebraska - Lincoln201 4 Wave 2 Supporting Statement : Appendix A 10

**NAEP Technology and Engineering Literacy (TEL)**  **Committee Name**  **Affiliation** Keith Barton Indiana University John Behrens Pearson e - L EADS Center , Mishawaka, IN Brooke Bourdelat - Parks BSCS (Biological Sciences Curriculum Study) , Colorado Springs, CO Barbara Bratzel Shady Hill School Cambridge, MA Lewis Chappelear James Monroe High School , North Hills, CA Britte Haugan Cheng SRI International , Menlo Park, CA Meredith Davis North Carolina State University Chris Dede Harvard Graduate School of Education Richard Duran University Of California, Santa Barbara Maurice Frazier Oscar Smith High School , Chesapeake, VA Camilla Gagliolo Arlington Public Schools , Arlington, VA Christopher Hoadley New York University Eric Klopfer Massachusetts Institute of Technology Beth McGrath Stevens Institute of Technology , Hoboken, NJ Greg Pearson National Academ y of Engineering , Washington, DC John Poggio University of Kansas Erin Reilly Annenberg School for Communication & Journalism , Los Angeles, CA Troy Sadler Missouri University Science Education Center , Columbia , MO Kimberly Scott Arizona State University Teh - Yuan Wan New York State Education Department , Albany , NY201 4 Wave 2 Supporting Statement : Appendix B 11

**Appendix B**  **: Sample**  **Pa rent** **/Guardian Notification Letter** 

**NAEP (NAEP Assessment Year)**  **PARENT/GUARDIAN NOTIFICATION LETTER** (School Letterhead) **(Insert Date Here)** Dear Parent or Guardian: We are pleased to notify you that (school name) has been selected to represent schools across the nation by participating in the National Assessment of Educational Progress (NAEP). First administered in 1969, NAEP is the largest nationally representative a ssessment of what students know and can do in various subject areas. It is administered by the National Center for Education Statistics, within the U.S. Department of Education. NAEP is different from state assessments because it provides a common measure of student achievement across the country. The results of NAEP are released as The Nation's Report Card, which provides information about student achievement to educators, parents, policymakers, and the public. In our school, the NAEP assessment will be gi ven on (date) in (subject) . Your child (may be/has been) selected to take the assessment. In addition to subject area questions, students will be asked some questions about themselves and their educational experience. These questions provide contextual inf ormation for the assessment, as well as information that may be related to students’ learning. If you would like to view sample subject area and student questions, please visit http://nationsreportca rd.gov/parents.asp . It will take about (assessment time) minutes for most students to participate in the assessment. The results are completely confidential (in accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107 - 347), and the information provided will be us ed for statistical purposes only. Your child’s grades will not be affected. Your child may be excused from participation for any reason, is not required to complete the assessment, and may skip any question. While NAEP is voluntary, we depend on student pa rticipation to provide an accurate measure of student achievement that will inform improvements in education. Your child will represent many other students, so participation is very important. However, if you do not want your child to participate, please n otify me in writing by (date). There is no need to study in preparation for NAEP. We do, however, ask parents to encourage their children to do their best and get plenty of rest the night before the assessment. A brochure that explains what participation in NAEP means for you and your child is available at http://nces.ed.gov/nationsreportcard/pdf/parents/2012469.pdf . Should you have questions, please contact me at (telephone numbe r) or at (e - mail address) . We are excited that our school will be participating in NAEP and pleased that your child (may be/has been) selected. We know that (school name) 's students will help us show what our nation’s students know and can do. Sincerely, S chool Principal