# Appendix C

NAEP Science Cognitive Interview Reports

a nd

List of New Questions

1. Science survey question cognitive interview study results (January 2013)

2. List of new/revised questions for Wave 1 materials

### Listening

. Learning

. Leading

. 

# NAEP Item Development (ID)

2015 Science Survey Questionnaires

2012

– 2013 Cognitive Interview Study

Grades 4 , 8, and 12

Student s , Teacher s , & School Administrators

Deliverable in response to ID Task 3. 1

. 3

Submitted: January 23, 20132015 Science Survey Questionnaire Cognitive Interview Study, January 2013

1 

### Table of Contents

Overview

................................ ................................ ................................ ................................ ......... 2

Project Objectives

................................ ................................ ................................ ........................... 3

Study Design and Implementation

................................ ................................ ................................ . 4

Data Collection and Analytic Methods

................................ ................................ ........................... 6

ETS Proposed Recommendations

................................ ................................ ................................ ... 8

Student

................................ ................................ ................................ ................................ ... 9

Teacher

................................ ................................ ................................ ................................ 58

School Administrator

................................ ................................ ................................ ........... 72

Cognitive Laboratories Report and Findings (Eureka Facts)

................................ ......................... 86

Appendices

................................ ................................ ................................ ................................ .. 18 32015 Science Survey Questionnaire Cognitive Interview Study, January 2013

2 

### Overview

As required by the National Assessment Governing Board (Governing Board), the National Center for
Education Statistics (NCES) of the U.S. Department of Education will conduct nationwide assessments at
grades 4, 8, and 12 for the National Assessment of Educational Progress (NAEP). In addition to assessing
subject

- area achievement, NAEP collects question naire data to provide context for the reporting and
interpretation of assessment results. Questionnaire data comes from one of three respondent types:
students, teachers, and school administrators. NAEP questionnaires serve to fulfill reporting
requirement s of federal legislation and to provide a context for reporting student performance.

This document presents findings and ETS recommendations based on results from a recent cognitive
interview study of the 2015 Science survey questionnaire items. The report presents project objectives,
study design and implementation, research design, cognitive interview findings, and recommendations
for new or revised items. The report concludes with an appendix, including interview protocols, scripts,

and forms.

As part of the item development process, items undergo several forms of pretesting including expert
panel reviews and cognitive interviews (or cog labs). Cognitive interviews are a form of in

- depth, one

on

- one pretesting used to assess whether items are being unders tood as intended and to identify any
problems in respondent comprehension. A goal of cog labs is to identify any problems that warrant
adding, revising, or dropping of items prior to the pilot assessment that will be administered in 2014.
These survey item s will be included in all science administrations: paper

- and pencil, Hands

- on Tasks (HOTs), and Interactive Computer Tasks (ICTs).

Note that proposed item revisions in the section “ ETS Proposed Item Recommendations ” are
recommendations for changes to the v ersions that were administered during the cognitive laboratories.
Deleted text is indicated by strikethrough and new text is highlighted yellow

.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

3 

### Project Objectives

This cognitive interview study investigates the cognitive processes that respondents use to answer survey questions. The ability of the respondent to comprehend the questions and provide valid
responses is helpful in identifying the problems and limitations associated with the questions prior to
the 2014 pilot. Early identification of such con cerns, prior to administration to a large number of
respondents, will increase the quality of the information collected from the questionnaires by reducing potentially confusing language and/or improving response categories.

In cognitive interviews, an int erviewer uses a structured protocol in a one

- on

- one interview using two

methods: think

- aloud interviewing and verbal probing techniques. With think

- aloud interviewing, respondents are explicitly instructed to “think aloud” (i.e., describe what they are thi nking) as they
determine their answers to the survey questions. The interviewer reads each question to the
respondent and then records the cognitive processes that the respondent uses in arriving at an answer
to the question. With verbal probing techniques , the interviewer asks probing questions, as necessary, to clarify points that are not evident from the “think aloud” process. These probes might include, for
example, asking the respondents to rephrase the question in their own words or assess whether the
response categories are relevant. The specific protocols used in this study are presented later in this

document.

Cognitive interview studies are largely observational. The data collected are predominantly qualitative
verbal reports given in response to p robes, think

- aloud tasks, and volunteered comments. The objective
is to identify and correct problems of ambiguity or misunderstanding, or other difficulties respondents
have answering questions. The result should be questionnaires that are easier to under stand and
therefore less burdensome for respondents while also yielding more accurate information.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

4 

### Study Design and Implementation

Recruitment process

EurekaFacts, a subcontractor to ETS, recruited student, teacher, and school administrator participants
for this study within the greater Washington, DC and Baltimore, MD metropolitan areas through email
and telephone outreach. Using an approved screener script via telephone, trained EurekaFacts
interviewers screened parents of student candidates who respon ded to the outreach.

During the telephone screening, interviewers confirmed the interview date, time, and location with
parents. Student interviews took place at the EurekaFacts cognitive interview laboratory in Rockville,
MD. Legal guardians of the parti cipating students completed consent forms in advance of the cognitive
interviews (see Appendix for recruitment scripts, consent forms, etc.) Students age 18 or older signed

their own consent forms. Teacher and school administrator interviews were conducted at the respondents’ school or via telephone.

A monetary incentive aimed at ensuring participation and motivation of the participants was provided. Each participating student earned a $25 Visa gift card in compensation for time and effort. In addition, a

V isa gift card of $25 for the students’ parent(s) was provided to remunerate parents for their time and
to help offset the travel/transportation costs of bringing the participating student to and from the
EurekaFacts cognitive laboratory site. Teachers and school administrators were provided with a $40 Visa
gift card.

Study Sample and Population

Existing research and practice have failed to offer a methodological or practical consensus regarding the
minimum or optimal sam ple size necessary to provide valid r esults for background question
development. 1 Nonetheless, a sample size of five to fifteen individuals has become the standard. Several
researchers have confirmed the standard of five as the minimum number of participants per subgroup for analysis (i.e., c ell) for the purposes of exploratory cognitive interviewing for question development. 2

Other researchers have indicated that although a sample size of five per cell will likely identify major
problems with a question, more is better, up to approximately fi fteen per cell. 3

Interview Protocols and Processes

Participants were first welcomed, introduced to the interviewer and the observer, and told they were
there to help answer questions about how people answer survey questions. Participants were reassured
that their participation was voluntary and that their answers may be used only for research purposes

1 Almond, P. J., Cameto , R., Johnstone, C. J., Laitusis, C., Lazarus, S., Nagle, K., Parker, C. E., Roach, A. T., & Sato, E. (2009). White paper: Cognitive interview methods in reading test design and development for alternate
assessments based on modified academic achievement s tandards (AA

- MAS). Dover, NH: Measured Progress and Menlo Park, CA: SRI International.

2 Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think

- aloud method: A practical guide to modeling cognitive processes. San Diego, CA: Academic Pre ss.

3 Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

5 and may not be disclosed, or used, in identifiable form for any other purpose except as required by law
[Education Sciences Reform Act of 2002, 20 U.S.C §9573].

Interviewers explained the cognitive interview ing process, conducted a practice question, and then
participants answered questions verbally.

Interviewers used several different cognitive interviewing techniques, including general think

- aloud and
question

- specific probes, observation, and debriefing q uestions.

The interview protocols were developed to facilitate capturing respondents’ thought processes while
answering the questionnaire items. The interviewers encouraged participants to speak aloud all their
thoughts while answering the questions and a sked follow

- up probes after each item. The probes were
designed to help participants elaborate their responses.

If a participant did not appear to fully think aloud their response, interviewers used “continuers” to
encourage the participant to be more desc riptive and verbalize their thoughts without “putting words in
their mouth.” Interviewers were instructed to remain as objective and unbiased as possible, but to offer a verbal “nudge,” such as:

 What are you thinking now?

 Any other thoughts?

 Tell me how yo u came to pick that answer.

If a participant exhibited difficulty reading or understanding an item, interviewers used probes similar to
the examples below:

 Can you tell me in your own words what that question was asking? [comprehension and interpretation p robe]

 What does the word [term] mean to you as it is used in this question? [comprehension and interpretation probe]

 How much would you say you know about [topic]? [confidence probe]

 How sure are you of your answer? [confidence probe]

 How did you come up w ith that answer? [recall strategy and bias probe]

 Was this easy or hard to answer?” [comprehension and recall probe]

 How easy or difficult is it to remember [topic]? [recall probe]

 Did you find the list of answer choices easy or hard? [response probe]

 How easy or hard was it to choose an answer from that list of choices? [response probe]

 The question uses the word [term]. Does that sound OK to you, or would you choose something different? [sensitivity probe]

 In general, how do you feel about this question? [sensitivity probe]

The generic probes were designed to understand why the question as written was not clear, and
interviewers were instructed to reword the question in a way that the participant would understand
based on shared interviewer feedback or pre vious interview experiences. Any instances of reworded
questions that were tested have been documented in the “Findings” section of this report.

In addition to generic probes, interviewer protocols included item

- specific probes. Not all items had an

item

- s pecific probe. Item

- specific probes were provided only if the researchers deemed it important to2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

6 learn more about a cognitive process or issue that might not be raised naturally from the think

- al oud
process or generic probing.

The interview protocols can be found in the Appendix.

Data Collection and Analytic Methods

The primary objective of the analysis was to identify items with problems and understand the nature of
those problems. In the course of cognitive interviewing, some type of response difficulty will likely occur
for almost every question. Our analytical approach is designed to identify the items that posed difficulties to respondents in ways that (1) appeared to reduce the validity of the response, (2) could be
predicted to occur at a statistical ly significant rate in a larger population, and (3) exhibited a logical basi s for why the problem occurred.

Interview teams recorded their notes into a Microsoft Excel data entry spreadsheet. The spreadsheet is
based on a template developed by ETS research er Zhitong Yang. The spreadsheet is designed to facilitate the collection and analysis of cognitive interview notes based on Roger Tourangeau’s four

stage task

- analytical cognitive model, as elaborated and adapted by Gordon Willis, 4 described below.

There are four primary questions regarding respondents’ survey response processes that are explored
through cognitive interviewing:

1. Comprehension: What does the respondent believe the question to be asking?

2. Memory: How does the respondent recall the information needed to answer the question?

3. Decision: How does the respondent approach (e.g., effort, sentiment) answering the question accurately and truthfully?

4. Response: Does the respondent’s answer match the given or planned response categories?

To collect information to identify problems with respondents’ survey response processes, we asked
interviewers to explicitly code the occurrence of six different threats to obtaining valid and reliable

survey responses.

1. Did the respondent skip over, misread, or rerea d any word(s)?

2. Did the respondent exhibit difficulty reading an item?

3. Did the respondent exhibit difficulty understanding the question?

4. Did the respondent exhibit difficulty understanding the meaning of particular words or concepts?

5. Did the respondent exhi bit difficulty remembering the question or answer choices?

6. Did the respondent exhibit difficulty or discomfort answering the question?

In addition to the explicit coding to identify problems, interviewer data entry sheets also include
descriptions of ident ified problems, summarizations of respondent verbal reports, and respondent
responses to item

- specific probes.

For the purposes of th is report, the analysts focused on providing and presenting evidence to better

assist in the item

- level adjudication process. Consequently, most of the findings reported in this report

4 Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

7 are centered on potential problems with items and discussion of the revisions to items that could best
correct for the identified problems.

A four

- stage process was used to identify problems with items. In the first stage, explicit codes related
to the six aforementioned threats to obtaining valid and reliable survey responses were evaluated to
highlight problematic items. In the second stage, the descriptions of identified threats we re then reviewed to determine the nature of the problem. In the third stage, respondents’ verbal reports and respondents’ responses to item

- specific probes were then reviewed to locate logical bases for the
problems. In the fourth stage, analysts and item developers met to discuss the findings to identify
potential approaches to item revision. These data collection and analytical methods are the basis for the
results presented in the following findings section.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

8 

# ETS Proposed Item Recommendations9 Student Questionnaire

– Grade 4

[ Grade 4 Science

– Student #1 ]

Cognitive lab results:
1. There was much variation in how students understood the phrase “circuits and energy, ” and two of the ten students reported not knowing what the phrase “circuits and energy” meant. They reported
tha t “batteries and light” is clear er to them.
2. Four students reported that they would have responded to the item differently if the phrase “batteries and light” w as used in place of “circuits and energy. ”
Rationale:
1. Cognitive lab results show that grade 4 students’ understanding of the phrase “circuits and energy” varies, and some students do not know what the phrase meant.
2. Cognitive lab results also show that grade 4 students shared a common understanding of the phrase “batteries and light” and correctly l ink ed the example to learning about electricity.
Recommendation:
1. ETS recommends using the trend phrase “batteries and light” instead of revised phrase “circuits and energy. ” New (Revised based on VC519362) In this school year, how often have you done activities or projects to learn about electricity (for exampl e, circuits and energy)?

A. Never

B. Rarely

C. Sometimes

D. Often10 [ Grade 4 Science

– Student #2 ]

Cognitive lab results:
1. When asked how they would respond to the item if they had done activities with chemicals only at certain times during the school year, the methods used by grade 4 students to calculate the rate vary. Some provided the average rates; some others provided a frequency based on the particular period of time when they had the activities. O ther grade 4 students did not understand the probe.
2. Compared with the revised version of this item (using “Never

, ” “Rarely, ” “Sometimes, ” and “Often” as options), more than half of students (6 of 10) preferred the response options in this item, and three stated they liked these response options because they are more concrete.
Rationale:
1. Cognitive lab shows that grade 4 students recall the information based on two major methods: frequency within the particular period of time when they have the activities, and averaging the frequency across the whole school year. Reporting frequency within a particular period is more common among grade 4 students.
2. Grade 4 students’ interpretation of the revised response options (“Never, ” “Rare ly, ” “Sometimes, ” and “Often”) vary and are vague.
Recommendation:
1. Keep the trend item in the 2014 pilot administration

. VC315219 In this school year, how often have you done activities or projects to learn about chemicals (for example, mixing sugar or salt in water)?

A. Never or hardly ever

B. Once ever y few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day11 [ Grade 4 Science

– Student #3 ]

Cognitive lab results:
1. Grade 4 students’ interpretation of the revised response options (“Never, ” “Rarely, ” “Sometimes, ” and “Often”) vary and are vague.
2. When asked how they would respond to the item if they had done act ivities with chemicals only at certain times during the school year, most students chose the more frequent response choice, implying that they tend to report the frequency based on the particular period time when they had
the activities.
Rationale:
1. Grade 4 students’ interpretation of the revised response options (“Never, ” “Rarely, ” “Sometimes, ” and “Often”) vary and are vague.
2. Fewer students preferred these response options in this item.
Recommendation:
1. D rop the revised version of the item in the 2014 pilot administration

. New (Revised based on VC315219) In this school year, how often have you done activities or projects to learn about chemicals (for example, mixing sugar or salt in water)?

A. Never

B. Rarely

C. Sometimes

D. Often12 [ Grade 4 Science

– Student #4 ]

Cognitive lab results:
1. All students displayed a good understanding of the item.
2. Most students provided examples of “sc ientific tools. ” The “mag nifying glass ” was the most cited
tool ; other s included “pH testing strip” and “dropper. ”
3. When being questioned which of the three versions of the answer choices made the most sense, the opinions were divide d . Three students said the answer choices in this item made more sense to them because it provide more details; two students preferred response options “Never, ” “Rarely, ” “Sometimes, ” and “Often” stating they are easier; five students reported that the response options
“No ,” “ Ye s, a little ,” and “ Y es, a lot” made more sense because they are simple and fit their spontaneously generated answers.
4. When asked how they would respond if they had only used tools at certain times of the year, students typically use d two major methods to r ecalculate the frequency of use 1) recall the frequency within the particular period , and

, 2) averag e the frequency across the whole school year.
5. The cognitive lab results show that most students answer ed the question based on the frequency within the part icular period of time when they have science classes.
Rationale:
1. Since all students displayed a good understanding of the item, the revised item stem works well.
2. Most students’ spontaneous answers were based on the frequency of scientific tool use within the particular period of time when they had the science activities.
3. Students’ are divide d among which of the three sets of response options they prefer

. Therefore, t he current response options should be used to maintain trend.
Recommendation:
1. Replace current items VC315222, VC315227, and VC315229 with this item in the 2014 pilot
administration

. New (Revised based on VC315222, VC315227, and VC315229) In this school year, how often have yo u done science activities using scie ntific tools?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day13 [ Grade 4 Science

– Student #5 ]

Cognitive lab results:
1. All students displayed a good understanding of the item.
2. When asked to define “rarely, ” “sometimes, ” and “often, ” students either used subjective, vague terms, or referred to a frequency similar to the trend response options (e.g., “ o nce very few
weeks” ) to explain.
Rationale:
1. Although a few students preferred the response options in this item, when asked to define the options, they either used vague terms or used a frequency similar to the trend options.
Recommendation:
1. Do not use this item in the 2014 pilot administration

. New (Revised based on VC315222, VC315227, and VC315229) In this school year, how often have yo u done science activities using scie ntific tools?

A. Never

B. Rarely

C. Sometimes

D. Often14 [ Grade 4 Science

– Student #6 ]

Cognitive lab results:
1. All students displayed a good understanding of the item.
2. When asked to define “Yes, a little

, ” and “Yes, a lot, ” students either used vague terms, or referred to the frequency response similar to the trend response options (e.g., “ o nce very few weeks”) to explain.
3. Half of the students (5 of 10) preferred the response options in this item because they are easier.
Rationale:
1. Although half of the students (5 of 10) preferred the response o ptions in this item, when being asked to define the options, they either used vague terms or used the frequency similar to the trend options to provide explanation.
Recommendation:
1. Do not use this item in the 2014 pilot administration

. New (Revised based on VC315222, VC315227, and VC315229) H ave yo u used scientific tools in school this year?

A. No

B. Yes, a little

C. Yes, a lot15 [ Grade 4 Science

– Student #7 ]

Cognitive lab results:
1. All students displayed a good understanding of the item.
2. Out of 4 students who stated that they read books about science that are not for school work, only
one said that he/she included it in selec ting the choices.
3. Of the five students who responded that they had read a book or magazine about science in this school year, all indicated that they read most often in the classroom.
Rationale:
1. The cognitive lab results show that grade 4 student s read boo ks or a magazine about science most
often in school.
2. There is no need to specify whether grade 4 students read books or magazines that are for school
or not for school.
Recommendation:
1. Keep the trend item in the 2014 pilot administration. VC315239 In this school year, how often have you read a book or magazine about science?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day16 [ Grade 4 Science

– Student #8 ]

Cognitive lab results:
1. Two students showed hesitation in pronouncing the word “presented.”
2. All students displayed a good understanding of the item.
3. When asked if “to show what you have learned” was different from “to present what you have learned, ” most students (6 of 10) said that these were the same . Only one student said that they differ, and one student did not understand what “show what you have learned, ” meant.
4. Six of the ten stude nts indicated that they have presented something to their class. Of these six, four presented information that they learned, and the other two presented a project that they
worked on.
Rationale:
1. The cognitive lab results show that grade 4 student s understa nd the word “presented, ” and interpreted the word similar to the word “showed. ”
Recommendation:
1. Keep the trend item in the 2014 pilot administration. VC315259 In this school year, how often have you presented what you learned about science to your class?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day17 [ Grade 4 Science

– Student #9 ]

Cognitive lab results:
1. When asked to define “talked about,” the majority of students (8 of 10) answered in a similar manner, stating that it meant to “say something” or “talk to someone.” Only one student did not
know how to respond.
2. When asked to define the term “measurements or results,” the students showed good understanding of these terms, and explained by referring to results, the end

- effect, or product of experiments, and solutions to problems. Two students used e xamples to help illustrate what they were trying to say.
3. Seven of the ten students stated that they “never or hardly ever” talked about measurements or results. The remaining three reported that they talked about measurements or results with a teacher, cla ssmates, or the whole class.
Rationale:
1. The cognitive lab results show that grade 4 student s understand the phrase “talked about. ”
Recommendation:
1. Keep the trend item in the 2014 pilot administration. VC315288 In this school year, how often have you talked about measurements or results from your science activities or projects?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day18 [ Grade 4 Science

– Student #10 ]

Cognitive lab results:
1. Interviewer notes report no major problems wi th comprehension in this item. When asked to paraphrase the item, the students displayed a good understanding of the item.
2. All students but one defined “report” in a similar manner as writing or sharing information from measurements or science projects with other people.
3. Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this school year; two responded “once every few weeks ,” and three said, “two or three times a week. ”
Rationale:
1. The cognitive lab results show that grade 4 student s understand the word “ report

. ”
2. The revised version of the item (item 11) produced response that is more thorough

. Recommendation:
1. Replace the item wit h the revised version of the item (item 11) in the 2014 pilot administration

. VC315265 In this school year, how often have you written a report on your science activities or projects?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day19 [ Grade 4 Science

– Student #11 ]

Cognitive lab results:
1. Most students (9 of 10) displayed a good understanding of the item.
2. All students had a similar understanding of the phrase “science journal,” describing it as something you write in to keep track o f information you have for science topics.
3. Four of the ten students would call the written work they do for a science activity or project a “science journal” (these same four students were unable to define a “lab write

- up”). Two students
would call the wor k a “report,” while two students would call it something else (a “writing piece” and “results”), and one was unsure what to call it

. Only one student said he/she would call the
work a “lab write

- up,” but did not know why.
4. The descriptions that students pro vided for the phrase “lab write

- up” were less consistent.
Rationale:
1. The cognitive lab results show that, for grade 4 student s, this item captured more comprehensive information than the trend item in which only the frequency of report writing was asked. Although the interviewer notes for the previous item (item ten) show that all but one student had a good understanding of the term “report,” the more generic formulation of item eleven (“asked to write about your science activities or projects” ) produced a different answer pattern among students

. This suggests that the item ten did not capture all possible information from students on this topic.
2. To further clarify the intent of the item, ETS suggest s adding common examples (i.e., “such as

repo rts, science journals, or lab write

- ups”) to the item stem.
Recommendation:
1. Replace the trend item with this revised version in the 2014 pilot administration

. New (Revised based on VC315265 ) In this school year, how often have you been asked to write about your science activities or projects ( such as reports, science journals, or lab write

- ups) ?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day20 [ Grade 4 Science

– Student #12 ]

Cognitive lab results:
1. Two students could not define the phrase “library resources.” Of the remaining eight students, all gave books as an example of a library resource. Three also gave the example of computers, and

two others gave the example of magazines.
2. Only after being probed, did students acknowledge the internet as being a library resource. After being asked, six students said that internet was a library resource, while three said that it was not. When probed, two students also acknowledged computers as library resources; another student was unsure, because computers in his/her school were not for fourth graders.
3. Nine students were asked whether they thought of their school library, the public library, or both
in se lecting the options

. Of the nine, seven thought only of the school library, one thought of both, and one answered that they “do not go to the library.”
4. When explicitly asked about individual resources, three students included encyclopedias, and three
inclu ded videos, as library resources.
Rationale:
1. The cognitive lab results show that a majority of students thought only of books as library

resources. 2. To capture all the other library resources used by student, ETS suggest s adding common examples to the item stem (i.e., “such as books, magazines, computers, and audio

- video materials”).
Recommendation:
1. Revise and keep the item in the 2014 pilot administration

. VC315266 In this school year, how often have you us ed your school library resources for science (such as books, magazines, computers, and audio

- video materials) ?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day21 [ Grade 4 Science

– Student #13 ]

Cognitive lab results:
1. Overall, the students understood the phrase “science activities” in a similar manner: eight of the

ten students described it as activities or projects that one does, that are related to a science topic. Only two did not explicitly refer to a science topic.
2. Interviewer notes demonstrated a good and consistent understanding of the phrase “not for schoolwork” among the students: five students said it meant activities done at home, and three said it meant being “out of school.”
3. When asked to provide examples of activities they had done that were not for schoolwork, two students gave technology/engineering examples (e.g., fixing a TV), two gave natural science examples (e.g., studying bugs), and two gave chemical examples (e.g., mixing substances together). Two students who answered “never or hardly ever” could not remember any examples of activities they had done that were not for schoolwork.
Rationale:
1. The cognitive lab results show that a majority of students have a good understanding of the item as
intended

. Recommendatio n: 1. Keep the trend item in the 2014 pilot administration. VC315410 How often do you do science activities that are not for schoolwork?

A. Never or hardly ever

B. Sometimes

C. Often

D. Always or almost always22 [ Grade 4 Science

– Student #14 ]

Cognitive lab results:
1. When asked to define a “science fair,” six students described it as related to making experiments; of these six, two also stated that “science fair” involved showing their work to other people. One student defined it as showing something, but did not mention experiments.
2. Two students defined science fairs as having a competiti ve element; one said first place gets a prize, and the other said science fair was synonymous with science competition.
3. Two s tudents did not understand what a “science fair” was; they knew only that it had something
to do with science.
4. When asked to provide an example of a science fair, there were few prevailing trends. Six students stated that they had not participated in one; two referred to “event nights” at their schools where science activities took place, but they did not present their own experiments.
Rationale:
1. Cognitive lab results show that grade 4 student s do not differentiate between the three scientific activities (i .e., “science fair, ” “science competition, ” and “science club”).
2. ETS s uggest s combining the three scientific activities in one question for grade 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration. New In this school year, have you participated in a science club, a science fair

, or a science competition ?

A. Yes
B. No23 [ Grade 4 Science

– Student #15 ]

Cognitive lab results:
1. When asked to define a “science club,” all students provided responses indicating that science clubs had to do with people meeting together to learn more about science or do science activities.
Add itionally, four students explicitly said they were after

- school activities.
2. When asked to provide an example of a science club, four students provided concise examples of specific clubs, three could only give examples of activities they had done in the clu bs, and one student had never participated in a club. Two students could not recall any examples of science
clubs.
Rationale:
1. Cognitive lab results show that grade 4 student s do not differentiate between the three scientific activities (i.e., “science fair, ” “science competition, ” and “science club”).
2. Suggest combining the three scientific activities in one question for grade 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration. New In this school year, have you participated in a science club?

A. Yes
B. No24 [ Grade 4 Science

– Student #16 ]

Cognitive lab results:
1. When asked to define a “science competition,” five students gave responses indicating that this activity involves “prizes” or “first place, second place, etc.” Six students said it involved either doing your own project/doing science, or that it was like a science fair.
2. When asked to provide an example of a science competition, three students either gave “science fair” as their example, or gave their same example from item fourteen (which asked for an example of a science fair). One student said fairs and competitions were similar, but not the same.
3. Students were asked whether science fairs, science clubs, and science competitions were different things. Over half (6 of 10) said that they were all dif ferent things; three said they were all the same; one said that fairs and competitions were the same, but clubs were different

. All three of the students who indicated that they were all the same gave the reason that they all dealt with science.
Rationale:
1. Cognitive lab results show that some grade 4 student s do not differentiate between the three scientific activities (i.e., “science fair, ” “science competition, ” and “science club”).
2. Suggest combining the three scientific activities in one question for gra de 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration. New In this school year, have you participated in a science competition?

A. Yes
B. No25 [ Grade 4 Science

– Student #17 ]

Cognitive lab results:
1. The cognitive results show that there were some variations in the definitions of “science museum” among grade 4 students . However, most (7 of 10) indicated that it invol ved scientific activities, lessons, and/or technology, or gave appropriate examples

. Two students were unsure of the meaning, and one student seemed to include non

- scientific museums in their definition.
2. Nine students were asked what specific museums they considered science and non

- science museums. Two students could not remember any science museums; two students could not remember any non

- science museums.
3. When students were asked what museums they thought of as science museums, examples
included: The Museu m of Natural History, the Space Museum, and an animal museum called Amazing Animals.
4. When students were asked what museums they thought of as non

- science museums, examples included: The National History Museum, the Wax Museum, and the Native American Muse um. Rationale:
1. Cognitive lab results show that grade 4 student s have a basic understanding of the term “science
museum” with some variations.
2. Since the intent of the item is to measure students’ “out of school” science activities, ETS suggest s
not us ing the “not on a school trip” version of this item.
Recommendation:
1. Do not use this item in the 2014 pilot administration. New In this school year, have you visited a science museum?

A. Yes
B. No26 [ Grade 4 Science

– Student #18 ]

Cognitive lab results:
1. When asked to define “on a school trip,” three students stated it was a trip involving their class and/or grade; three students mentioned riding on a school bus; three students said it was a field trip; two students simply re

- worded the phrase (e.g., “a trip for school”).
2. Of the five students who provided examples, two referenced museums, and two could only discuss the activities they did at the museums, (one simply refe renced a field trip to the Chesapeake Bay).
The referenced museums were The Animal Museum and the Kid’s Museum in Baltimore Harbor.
3. Only one student had been to a science museum on a school trip this year.
Rationale:
1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science
museum” with some variations.
2. Since the inten t ion of the item is to measure students’ “out of school” science activities, ETS suggest s omitting the “not on a school trip” version

. Recommendation:
1. Do not use this item in the 2014 pilot administration. New In this school year, have you visited a science museum on a school trip ?

A. Yes
B. No27 [ Grade 4 Science

– Student #19 ]

Cognitive lab results:
1. Seven of the ten student s displayed good understanding of the item.
2. When asked to define “not on a school trip,” seven of the ten students stated it was a trip that did
not include school individua ls or classmates, while five students described it as involving family
and/or friends.
3. Only one student indicated that they had visited a science museum this year that was not on a school trip they went once.
4. Five students provided the following examples o f trips that were not for school: The Air and Space Museum, the Natural History Museum, and the American Indian Museum.
Rationale:
1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science
museum” with some variations

. 2. Most of the students (7 of 10) displayed good understanding of the phrase “not on a school trip” as the phrase intended.
3. Since the inten t ion of the item is to measure students’ “out of school” science activities, ETS suggest s us ing this version of the item

. Recommendation:
1. Add this item to the 2014 pilot administration. New In this school year, have you visited a science museum that was not on a school trip ?

A. Yes
B. No28 Student Questionnaire

– Grade 8

[ Grade 8 Science

– Student #1 ]

Cognitive lab results:
1. In regards to the meaning of the phrase “circuits and energy,” students’ responses varied greatly. While two students understood a circuit as something consisting of wires, another two understood it as something that electricity moves through. Two students understood energy to be synonymous with the term “power.” Four students provided a limited and vague definition for the phrase, showing a partial understanding that “circuits and energy” referred to electricity. Two of the students were unable to provide a definition.
2. Of the four students who answered “yes” to sub

- item b, three recalled that the activity they thought of when formulating their answer was related to using wires to light a light bulb. The remaining student recalled learning about electricity , but could not provide examples of specific activities related to this topic.
3. When asked how they would respond to sub

- item b if the example had been “batteries and light” instead of “circuits and energy,” half (5 of 10) responded that they would have cha nged their answers. Four of these students would have changed their answers from “no” to “yes,” while the fifth would have changed from “yes” to “no.” VC304986 In your science class this year, have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Yes No a. Living things (for example, plants, animals, bacteria) A B VC304988

b. Electricity (for example, circuits and energy) A B Revised sub

- item
based on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B VC304991

d. Rocks or minerals (for example, identifying types) A B VC305007

e. Magnifying glass or microscope (for looking at small things) A B VC305008

f. Thermometer or barometer (for making measurements) A B VC305009

g. Simple machines (for example, pulleys and levers) A B VC30501229 Rationale:
1. There was significant variation in how students understood the phrase “circuits and energy.” I n terms of their recollections of activities relating to electricity, students predominantly recalled activities where they had used batteries, wires, and/or light bulbs. In addition, probing revealed
that if sub

- item b included the example of “batteries a nd light,” instead of “circuits and energy,” twice as many students would have answered “yes” to this sub

- item. Thus, “batteries and light” appears to be the most relevant examples for grade 8 students.
2. Although grade 8 student s comprehend the item, there could be better distinction between response options as shown in the following two items.
Recommendation:
1. For su b

- item b, ETS recommends using the trend version examples “batteries and light” for the grade 8 student questionnaire in the 2014 pilot administration.
2. ETS recommends using the revised version of the item as shown in the student item #3 (with response options “Never, ” “ Rarely, ” “ Sometimes, ” and “ Often” ) in the 2014 pilot administration.30 [ Grade 8 Science

– Student #2 ]

Cognitive lab results:
1. All students displayed a good comprehension of the item.
2. When asked how they would respond if they had only done activities or projects at certain times during the school year (e.g., doing an activity every day for three weeks, but not since then), students provided a variety of responses.
3. Cognitive lab results show that grade 8 student s typically use two methods to respond to the hypothetical question if they had only done activities or projects at certain times during the school year: recall an experience within a particular period , or averaging the frequency across the entire school year.
4. Four of the 10 students expressed that the response options are hard to follow and do not align
well with their science course schedule.
Rationale:
1. Although grade 8 students comprehend the item, the response options are difficult to follow.
Recommendation:
1. ETS recommends not using this item in the 2014 pilot administration. New (Revised based on VC30498 6) In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never or hardly ever Once every
few weeks About once
a week Two or
three times

a week Every day
or almost every day a. Living things (for example, plants, animals, bacteria) A B C D E VC304988

b. Electricity (for example, circuits and energy) A B C D E Revised sub

item based on

VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D E VC304991

d. Rocks or minerals (for example, identifying types) A B C D E VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D E VC305008

f. Thermometer or barometer (for making measurements) A B C D E VC305009

g. Simple machines (for example, pulleys and levers) A B C D E VC30501231 [ Grade 8 Science

– Student #3 ]

Cognitive lab results:
1. Six students defined the response options using more subjective and vague terminology, with “rarely” meaning “not much,” “sometimes” meaning “once in a while,” and “often” meaning “a
lot.” Of these six students, four defined “rarely” in terms of comparisons to the other categories (“not often,” or “less than sometimes”). The definitions for “sometimes” and “often” varied among these six students, but their definitions showed that they understood the concept in all

cases. 2. Four students defined the response options according to more objective descriptions. Among these respondents, the term “rarely” was defined by all four students as either “once

, ” or “once a
month ”; the term “sometimes” had varying definitions; the term “often” was defined by all four students as once or multiple times a week.
3. When asked which version of the response options they preferred, nine of the ten students said they preferred the response op tions used in this item. Many implied that answering the item with the dichotomous response options used in item one was easy and these options were too broad to assess their internally generated answers. Moreover, the students explained that it was diffic ult to answer the item with frequencies, as used in item two, because the response options were too specific. New (Revised based on VC304986 ) In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never Rarely Sometimes Often a. Living things (for example, plants, animals, bacteria) A B C D VC304988

b. Electricity (for example,
circuits and energy) A B C D Revised sub

- item
based on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D VC304991

d. Rocks or minerals (for example, identifying types) A B C D VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D VC305008

f. Thermometer or barometer (for making measurements) A B C D VC305009

g. Simple machines (for example, pulleys and levers) A B C D VC30501232 Rationale:
1. Verbal reports and answers to probing questions show that the students strongly preferred the response option scale used in item three , as compared to item s one and two. It appears that the students could more easily fit their internally generated answers into the response options provided in item three; in addition, these response options placed a lower overall cognitive burden upon

stu dents.
Recommendation:

1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item in the 2014 pilot administration

. 2. For sub

- item b, ETS recommends using the trend wordin g of the sub

- item (using examples “batteries and light”) in the 2014 pilot administration.33 [ Grade 8 Science

– Student #4 ]

Cognitive lab results:
1. When asked to rephrase the item in their own words, the students displayed a good understanding
of the item or instructions.
2. When asked what the phrase “science topics” meant, five students implie d that this meant subjects studied in science class, two students gave discrete examples of specific science topics, and two answered that this phrase meant “anything” that has to do with science.
3. When asked if they would respond differently if the item u sed the term “science” instead of “science topics,” seven said they would answer differently. The most common reason given for this change of response among students (4 of 7) was that the term “science” was broader than “science topics,” and would allow th em to consider more types of media as related to “science” in their responses.
Rationale:
1. The item is classified under the issue “Organization of Instruction” and sub

- issue “Instructional Strategies.” Adding the word “topics” to the relevant sub

- items wi ll help clarify the intent of the
item.
Recommendation:
1. ETS recommends using the revised version of the three sub

- items in the 2014 pilot administration. VC305292 In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly

ever Once

every few weeks About

once a week Two or
three
times
a week Every day

or almost every day a. Read a science textbook, in class or at home A B C D E VC546510

b. Read a book or magazine about science topics A B C D E Revised sub

item based on
VC305295 c. Use the I nternet to learn about science topics A B C D E Revised sub

item based on
VC720562 d. Watch a movie, video, or DVD about science topics A B C D E Revised sub

item based on
VC30530734 [ Grade 8 Science

– Student #5 ]

Cognitive lab results:
1. When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub

- items c and d implied summarizing what they did or learned in class. The other five students defined the ph rase as generally talking with people (students or teachers) in their class.
2. When asked what is meant by the term “lab write

- up,” half of the students (5 of 10) described it as something akin to a report, describing a science lab that they had complete d. Three students described it as another type of class assignment unrelated to a science lab. Two students were unable to provide a definition.
3. In regards to the meaning of the term “science journal,” responses varied greatly, and included the following: where they write their “warm

- up” exercise (2 students); where to write notes (2 students); where to write about experiments (2 students); something that is used to describe what happened in class (2 students); where one can write about science in general ( 2 students). VC720622 In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly

ever Once

every few weeks About

once a week Two or
three
times
a week Every day

or almost every day a. Identify questions that can be addressed through science experiments A B C D E VC720623

b. Design a science experiment A B C D E VC720634 c. Talk about measurements you took for your science project or activity A B C D E VC720635

d. Talk about the results of your
science project or activity A B C D E VC720636 e. Watch your teacher do a science experiment or activity A B C D E VC720638 f. Make graphs or charts of the results from your science project or activity A B C D E VC720639

g. Write a report on your science project or activity (such as reports, science journals, or lab
write

- ups) A B C D E VC720641

h. Write about your science activities or projects (such as reports, science journals, or lab
write

- ups ) A B C D E New

h. Write a lab write

- up on your science project or activity A B C D E New i. Write a science journal A B C D E New35 Rationale:
1. Interview notes revealed considerable variation in how students understood the phrases “lab write

up” and “science journal.” Considerable overlap between the two phrases exists in some students’ interpretations. To avoid confusio n and los s of data, ETS suggests revising the two sub

- items by using the more generic phrase to describe the activities, i.e., “write about your science activities or projects (such as reports, science journals, or lab write

- ups).”
Recommendation:
1. ETS recommends using the revised sub

- item h in the 2014 pilot administration.36 [ Grade 8 Science

– Student #6 ]

Cognitive lab results:
1. When probed, seven students stated that they were thinking about their school’s library when answering this item and three stated that they were thinking of both their school’s library and a public library.
2. In regards to what students thought of as library resources, the top library resource identified by students was books (9 of 10). Other resources that students mentioned, unaided, were computers (4 students), internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1 student), and magazines (1 student).
3. When asked directly about each resource, eight students said they thought of computers as resources while answering the item, seven students thought of encyclopedias, five thought of the internet, five of magazines, and four thought of videos.
Rationale:
1. Not all students’ spontaneous responses included library resources other than books. ETS suggests adding specific examples at the end of the item to be consistent with the item used in grade 4 and grade 12 questionnaires.
2. Since the item is under issue “ Availability and Use of Instructional Recourses” and sub

- issue “Products, ” ETS suggests clarifying that the question as asking about “school library” uses

. Recommendation:
1. Revise and keep the item in the 2014 pilot administration. VC315266 In this school year, how often have you us ed your school library resources for science (such as books, magazines, computers, and audio

- video materials) ?

A

. Never or hardly ever

B

. Once every few weeks

C

. About once a week

D

. Two or three times a week
E. Every day or almost every day37 [ Grade 8 Science

– Student #7 ]

Cognitive lab results:
1. Interviewer notes do not indicate any consistent problems in students’ understanding of the item or
instructions.
2. When asked what was meant by the phrase “science

- related activities” in sub

- item a, the students either provided general, broad descriptions related to activities that had “anything to do with science,” or offered some examples such as measurements, experiments, and research.
3. When asked what was meant by the phrase “not for schoolwork, ” five students emphasized in their responses that these activities were done outside of school, or were not assigned schoolwork. Four students emphasized in their explanations that these activities were done based on personal
interests.
Rationale:
1. Cogniti ve lab results show that all students understood and interpreted the item as intended.
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration. VC305330 Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill

in one oval on each line.

Strongly disagree Disagree Agree Strongly

agree a. I do science

- related activities that are not for
schoolwork. A B C D VC305348

b. I like science. A B C D VC305350 c. Science is one of my favorite subjects. A B C D VC305351 d. I take science only because I have to. A B C D VC305352 e. I take science only because it will help me in
the future. A B C D VC30535338 [ Grade 8 Science

– Student #8 ]

Cognitive lab results:
1. When asked what was meant by the term “covered” in the item stem, seven students provided a general, broad definition for the term (“gone over,” “what we learned”). In their definitions, three
of t he students included a term that described the extent of the learning (“we learned about it completely,” “we know it thoroughly”).
2. Students that answered “yes” to any of the sub

- items were asked to describe how the material was
“covered” in their class. A majority of the students (8 of 9) answered this by providing the names of the topics that were covered. All of the topics mentioned by students fit within the existing sub

item’s category. One student described the types of assignments and materials that were used to learn each topic.
3. When probed, seven students said that they would prefer a larger scale that asked to what extent the topics were covered, as opposed to “yes” and “no” response options. The main reason for this preference, according to five of these seven students, was the ability to more accurately assess the extent to which these topics were covered. Of the three students who preferred the “ yes/no ” format, two had previously included a term that described the extent of learning in their
des cription of the term “covered.”
Rationale:
1. Cognitive lab results show that all students displayed a good understanding of the item and the
word “covered.”
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration. VC304978 In your science class this year, which of the following topics have been covered? Fill in one oval on each

line.

Yes No

a. Life science (for example, biology, the human body, or ecology) A B VC304982 b. Physical science (for example, energy, physics, or chemistry) A B VC304983

c. Earth and space science (for example, geology or astronomy) A B VC304984 d. Engineering and technology (for example, designing solutions to problems) A B VC30498539 [ Grade 8 Sc ience

– Student #9 ]

Cognitive lab results:
1. When asked what was meant by the t erm “science fair” in sub

- item a, all ten students provided explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven of the ten explanations also indicated that these projects were shown or talked about to others.

a. Eight students were able to provi de an example of a science fair in which they or a classmate had participated.
2. When asked what was meant by the t erm “science club” in sub

- item b, students had varying answers. Five students indicated that these clubs occurred after school; one student said that they occur annually, and the other four students did not discuss when clubs occurred. Four students included in their definition that the p urpose of the club was to do science activities, and three noted that the purpose of the club was to discuss or talk about science.

a. Five students were able to provide an example of a science club in which they or a classmate had participated.
3. When asked w hat was meant by the term “sc ience competition” in sub

- item c, nine students provided a definition that described it as similar to a science fair, where one makes a project, invention, or experiment and competes against others. One student mentioned that p eople also compete by answering questions. One student was unable to define the term.

a. Only one student was able to provide an actual example of a science competition, which they described as a “science club competition, where they had to build things to c ompete in certain challenges.” One student answered that a geography bee was a science competition. The other eight students were unable to retrieve a cogent example.
4. All ten students answered “no” to sub

- items a and c, and nine answered “no” to sub

- item b.
Rationale:
1. Most students comprehend the item as intended.
2. Although almost all students indicated that they have not participated in any of the activities, ETS suggests that the item could be still used to measure the growth of out

- of

- school science act ivities.
Recommendation:
1. ETS recommends adding the item to the 2014 pilot administration. New In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B
b. Science club A B

c. Science competition A B40 [ Grade 8 Science

– Student #10 ]

Cognitive lab results:
1. All ten students interpreted the item as asking about trips to museums that involve science.
2. When asked what was meant by the term “science museum,” a majority of students (6 of 10) gave general an d vague definitions for the term : “a museum that has to do with science of any kind” and
“a museum about science. ” The other four students provided examples of what might be found in a science museum (“a collection of experiments” and “rocks and dinosaurs” ).
3. When asked what they think of as science museums, eight students were able to provide examples of actual science museums, including Air and Space, Smithsonian, and Port Discovery. Only one student provided an example that was not an actual science m useum. Rationale:
1. Cognitive lab results show that grade 8 students comprehend the item as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school trip” and “not on a school trip

, ” as shown in the followi ng two items, ETS suggests using the “not on a school trip” version of the item to measure students’ out

- of

- school science

- related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration. New In this school year, have you visited a science museum?

A. Yes
B. No41 [ Grade 8 Science

– Student #11 ]

Cognitive lab results:
1. The students interpreted the item as asking if they have visited a museum on a school trip, or under the supervision of the school.
2. When asked what was meant by the phrase “on a school trip,” all t en students either found the term synonymous with the term “field trip,” or described traveling to a location with people from
their school.
3. When asked if they had been on a school trip to a science museum, six students described a situation where they h ad been on a school trip. Of those six, three provided descriptions of the specific trip, including traveling to a science museum with classmates, the class, or the teacher on
a bus.
Rationale:
1. Cognitive lab results show that grade 8 students understand t he item as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school trip” and “not on a school trip” ( as compared to the following item ) , ETS suggests using the “not on a school trip” version of the item to measure students’ out

- of

- school science

- related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration. New In this school year, have you visited a science museum on a school trip ?

A. Yes
B. No42 [ Grade 8 Science

– Student #12 ]

Cognitive lab results:
1. When asked what was meant by the phrase “not on a school trip,” seven of the students stated that it meant visiting the museum without others from their school, and three said it meant visiting in the presence of family or friends.
2. Six students described a time when they visited a science museum that was not on a school trip, and four students said they could not remember a visit to a science museum that was not on a school trip.
Rationale:
1. Cognitive lab results show that grade 8 students understand the it em as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school trip” and “not on a school trip," ETS suggests using the “not on a school trip” version of the item
to measure students’ out

- of

- school science

- related activities.
Recommendation:
1. ETS recommends adding this item to the 2014 pilot administration. New In this school year, have you visited a science museum that was not on a school trip ?

A. Yes
B. No43 Student Questionnaire

– Grade 12

[ Grade 12 Science

– Student #1 ]

Cognitive lab results:
1. When the students were asked what they thought was meant by the term “circuits and energy,” half the students (5 of 10) attempted to explain the concept of how circuits and energy are related (“circuits carry electrical energy from one place to another”). The other half of students named tangible components of circuits such as wires, batteries, and light bulbs.
2. Of the three students that answered “yes” to sub

- item b, two recalled activities in class related to electricity or circuits. The other describe d a chemistry lab where reactions were measured as a function of heat applied by an electric heater.
3. When probed, eight of the ten students would not change their answer if they would change their response to sub

- item b if instead of “circuits and energy ” the example was “batteries and light.” Five students noted they believed that asking the item this way would have changed its intent. Three of these five thought that “light” did not fit with the concept of electricity.
Rationale:
1. Cognitive lab results s how that, for grade 12 students, the phrase “circuits and energy” appear s to be a better example because this phrase includes “batteries and light, ” in its overall meaning

. Recommendation:
1. For sub

- item b, ETS recommends using the revised examples “circuits and energy” for the grade 12 student questionnaire instead of using “batteries and light” in the 2014 pilot administration.
2. ETS recommends using the revised version of the item as shown in the student item #3 (with response options “Never, ” Rarely, ” Somet imes, ” and “ Often” ) in the 2014 pilot administration. VC304986 In your science class this year, have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Yes No a. Living things (for example, plants, animals, bacteria) A B VC304988

b. Electricity (for example, circuits and energy) A B Revised sub

- item
based on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B VC304991

d. Rocks or minerals (for example, identifying types) A B VC305007

e. Magnifying glass or microscope (for looking at small things) A B VC305008

f. Thermometer or barometer (for making measurements) A B VC305009

g. Simple machines (for example, pulleys and levers) A B VC30501244 [ Grade 12 Science

– Student #2 ]

Cognitive lab results:
1. All students displayed a good comprehension of the item.
2. When asked how they would respond if th ey had only done activities or projects at certain times during the school year (e.g., if they had done an activity every day for three weeks, but they have not done the activity since then) , students typically use two methods to respond: recall experience within a particular period of time, and averaging the frequency across the whole school year

.
a. Of the five students who would answer o n the lower end of the scale, three explained their choices by describing a process similar to averaging the frequency of the activity over the course of a school year.
b. The other two students explained that the response option “about once a week” or higher would be a poor way of describing this situation.

c. One student chose “every day or almost every day” explaining that they would consider the period of time that they were working on that project rather than the entire year. New (Revised based on VC30498 6) In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never or hardly ever Once every
few weeks About once
a week Two or
three times

a week Every day
or almost every day a. Living things (for example, plants, animals, bacteria) A B C D E VC304988

b. Electricity (for example, circuits and energy) A B C D E Revised sub

item based on

VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D E VC304991

d. Rocks or minerals (for example, identifying types) A B C D E VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D E VC305008

f. Thermometer or barometer (for making measurements) A B C D E VC305009

g. Simple machines (for example, pulleys and levers) A B C D E VC30501245 d. The remaining three students did not specify the response option they would choose and explained that their answer would depend on how often they studied that topic or how well they would remember it.
Rationale:
1. Cognitive lab results show that grade 12 students used two methods to respond to the item: recall experiences within a particular period of time and averaging the frequency across the whole school

year. 2. Two students explicitly stated that the response opt ion “about once a week” or higher would be a poor way of describing the situation.
Recommendation:
1. ETS recommends not using this item in the 2014 pilot administration

.46 [ Grade 12 Science

– Student #3 ]

Cognitive lab results:
1. The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to them. Five of the students provided objective, quantifiable answers, and five students provided subjecti ve answers.

a. For the term “rarely, ” three of the five who answered objectively indicated that the term means “once,” and two described it as “once every few weeks.” Of the five who answered subjectively, three compared the term to the other response categ ories (“between never and sometimes”).
b. For the term “sometimes,” three of the five who answered objectively indicated that it meant more than once during the week, and two described it as more than once in a month. Of the five that answered subjectively, t hree compared it to the other response categories (between “ rarely ” and “ often ” ).

c. For the term “often,” three of the five who answered objectively indicated that it meant every day, and two described it as once or more in a week. Of those who answered
sub jectively, all used a term to describe a large amount of use as “very frequently.”
2. When asked how they would respond if they had only done activities or projects at certain times during the school year, six of the ten students provided a clear indic ation of which of the response options they would choose.

a. All six students provided answers at the higher end of the scale by choosing either “sometimes” (4 of 6) or “often” (2 of 6). New (Revised based on VC304986 ) In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never Rarely Sometimes Often a. Living things (for example, plants, animals, bacteria) A B C D VC304988

b. Electricity (for example, circuits and energy) A B C D Revised sub

- item
based on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D VC304991

d. Rocks or minerals (for example, identifying types) A B C D VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D VC305008

f. Thermometer or barometer (for making measurements) A B C D VC305009

g. Simple machines (for example, pulleys and levers) A B C D VC30501247 b. Two of these students explained that they based their answers on the a mount of time that they were working on that project rather than the entire school year. Two explained that if they had spent three weeks on a project they would have learned a lot, and one stated that “rarely” would not be applicable in the situation desc ribed in the probe.
3. When asked which of the first three item formats they preferred, six students preferred item three, two preferred item two, one preferred item one and one said that all are easy to understand

.
a. Of the six students that preferred item three, five stated they believed these response options better matched their answers compared to the other items.
b. The two students that preferred item two stated that they liked the specificity of the response optio ns. Rationale:
1. Verbal reports and answers to probing questions show that the students strongly preferred the response option scale used in item three, as compared to item s one and two. It appears that the students could more easily fit their internally gen erated answers into the response options provided in item three; in addition, these response options placed a lower overall cognitive burden upon
students.
Recommendation:

1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item in the 2014 pilot administration.

2. For sub

- item b, ETS recommends using the revised wording of the sub

- item (using examples “circuits and energy”) in the 2014 pilot administration.48 [ Grade 12 Science

– Student #4 ]

Cognitive lab results:
1. When asked what the phrase “science topics” meant, six students explained that it means academic subjects such as physics, chemistry, and biology. Four students answered that this phrase means anyt hing that has to do with science.
2. Four of the five students who indicated they read books or magazines that were not for schoolwork did not include these books or magazines when answering sub

- item b. The reason provided for not including these books or ma gazines was that they were not being read for their
science classes.
3. When asked if they would respond differently if the item used the term “science” instead of “science topics,” four of the ten students responded that they would answer the item differentl y. One student explained that he/she would have included activities done at home in addition to those completed in class. The explanations of the other two students suggest that the phrase “science topics” was interpreted as referring to more narrow and st rictly defined science subjects, while the term ”science” would include a wider range of subjects.
Rationale:
1. The item is classified under issue “Organization of Instruction” and sub

- issue “Instructional Strategies.” Adding the word “topics” to the relevant sub

- items, helps further clarify the intent of
the item.
Recommendation:
1. ETS recommends using the revised version of the three sub

- items in the 2014 pilot administration. VC305292 In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly

ever Once

every few weeks About

once a week Two or
three
times
a week Every day

or almost every day a. Read a science textbook, in class or at home A B C D E VC546510

b. Read a book or magazine about science topics A B C D E Revised sub

item based on
VC305295 c. Use the I nternet to learn about science topics A B C D E Revised sub

item based on
VC720562 d. Watch a movie, video, or DVD about science topics A B C D E Revised sub

item based on
VC30530749 [ Grade 12 Science

– Student #5 ]

Cognitive lab results:
1. When asked what the phrase “talk about” meant in sub

- item c and d, all ten students, described it as “discussing” with the entire class, classmates, or teachers.
2. When probed, most students (7 of 10) provided similar definitions of the term “lab write

- up” in
sub

- item h.

a. These seven students provided explanations that were centered on the idea that it was a written piece that summarized a science lab. However, three of these students expressed some confusion about the meaning of the term “lab write

- up.”
b. Two students believed that the term “lab write

- up” meant answering questions.

c. One student was unable provide a definition.
3. When initially answering, three students expressed some confusion about the meaning of the term “science journal” in sub

- item i. When asked directly what the term meant, the students had varying responses. While providing a definition, six of the students expressed that they were
unsure what the term meant; of these six, two were unable to guess a definition. Two students believed the term could mean a long science report. Two other students believed a science journal might be where students keep their notes for science. Two students thought the t erm might mean a short daily class or homework assignment. One student thought a science journal might be where VC720622 In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly

ever Once

every few weeks About

once a week Two or
three
times
a week Every day

or almost every day a. Identify questions that can be addressed through science experiments A B C D E VC720623

b. Design a science experiment A B C D E VC720634 c. Talk about measurements you took for your science project or activity A B C D E VC720635

d. Talk about the results of your
science project or activity A B C D E VC720636 e. Watch your teacher do a science experiment or activity A B C D E VC720638 f. Make graphs or charts of the results from your science project or activity A B C D E VC720639

g. Write a report on your science project or activity (such as reports, science journals, or lab
write

- ups) A B C D E VC720641

h. Write a lab write

- up on your science project or activity A B C D E New i. Write a science journal A B C D E New50 short brief constructed response items were answered. Another student thought that it might be used to provide a summary of an experiment for pu blication.
Rationale:
1. Interview notes revealed considerable variation in how students understood the phrases “lab write

up” and “science journal.” Considerable overlap between the two phrases exists in some students’ interpretations. To avoid confusion and los s of data, ETS suggests revising the two sub

- items by using the more generic phrase to describe the activities : “write about your science activities or projects (such as reports, science journals, or lab write

- ups).”
Recommendation:
1. ETS recommends u sing the revised sub

- item in the 2014 pilot administration.51 [ Grade 12 Science

– Student #6 ]

Cognitive lab results:
1. When answering the probe, “did you think about your school’s library, a public library, or both,” there was an even split among students.

a. Five students thought only of the school library resources when answering.
b. Five students thought of both the school and public libraries and their resources.
2. When asked to provide examples of library resources that they have used for science class, all students were able to provide at least one example. Some of these included:

a. Biographies, websites, online journals, internet databases, magazines, autobiographies, encyclopedia Britannica, and textbooks.
3. A majority of students (9 of 10) considered computers and the internet as library resources.

a. While thin king about his answer, one student was not sure whether the probe was asking about the use of computers in general, or solely as a means to access the internet.
4. A majority of students (9 of 10) did not think of videos as a library resource. One of these s tudents responded to the probe by indicating that while not mentioning videos as a library resource originally, he had used the library computers to watch online videos.
Rationale:
1. Not all students’ spontaneous responses included library resources other th an books. ETS suggests adding specific examples at the end of the item to be consistent with the item used in grade 4 and grade 8 questionnaires.
2. Since the item is under issue “Availability and Use of Instructional Recourses” and sub

- issue “Products, ” ET S suggests clarifying the question as asking about “school library” use.
Recommendation:
1. Revise and keep the item in the 2014 pilot administration. VC315266 In this school year, how often have you us ed your school library resources for science (such as books, magazines, computers, and audio

- video materials) ?

A

. Never or hardly ever

B

. Once every few weeks

C

. About once a week

D

. Two or three times a week
E. Every day or almost every day52 [ Grade 12 Science

– Student #7 ]

Cognitive lab results:
1. Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10) stated that the item asked if they agreed or disagreed with these statements about science. Three
studen ts stated that the item is asking about their opinions and feelings about science.
2. When probed what the phrase “science

- related activities” meant to them, a majority of students (9 of 10) reported that it meant doing anything related to science such as wat ching a movie or conducting an experiment.
3. When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10) stated that it was something that was not required by the school (something extracurricular).
4. When asked to provide a n example of “science

- related activities that you or your classmates have done or might do that are not for schoolwork,” more than half of the students (6 of 10) were able to provide an example, such as:

a. Using a telescope, working at NIH, reading Stephen H awking books, nature treks, and
science fairs.
b. More than half of the students (6 of 10) stated that these activities were not for schoolwork because a teacher did not instruct them to perform these activities, they would not be graded, and the activities w ould not be discussed in class.
Rationale:
1. Cognitive lab results show that all students understood and interpreted the item as intended.
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration. VC305330 Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill

in one oval on each line.

Strongly disagree Disagree Agree Strongly

agree a. I do science

- related activities that are not for
schoolwork. A B C D VC305348

b. I like science. A B C D VC305350 c. Science is one of my favorite subjects. A B C D VC305351 d. I take science only because I have to. A B C D VC305352 e. I take science only because it will help me in
the future. A B C D VC305353

f. When I graduate from high school, I would like to have a job related to science. A B C D VC72059653 [ Grade 12 Science

– Student #8 ]

Cognitive lab results:
1. A majority of students (7 of 10) indicated that the item asked whether they had done the listed activities during the current school year. Three students thought it referred to all extracurricular
activities.
2. A majority of students (8 of 10) stated that they did not participate in any of the listed activities. Two students had participated in the listed activities; one had participated in a science club and a science competition. The other student h ad participated in a science fair. Those who reported participating in the listed activities also specified that they participated in the activities during the current school year.
3. When asked what the phrase “science fair” means to them, most students (7 o f 10) explained that it refers to creating a research project and presenting it to other people.

a. Two students did not differentiate the terms “science fair” and “science competition.”
b. One student had never participated in a science fair but had attended o ne.

c. Over half of students (6 of 10) were able to provide an example of a science fair, but only two students mentioned science fairs that were taking place this year. Four students provided examples of science fairs ranging chronologically from elementary school to their first year of high school. Four students were unable to provide specific examples of science
fairs.
4. When asked what the phrase “science club” meant to them, students had varied ideas about whether the club was a school sponsored, extracurri cular, or an activity separate from the school altogether. All students stated that a science club was a club where students meet and discuss
science

- related topics. When asked to provide an example of a science club that they or their classmates have part icipated in or might participate in, students provided the following examples:

a. Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club, biology, physics, or chemistry clubs, and an environmental club.
b. Three students were unable t o provide an example, or stated that a science club was the same as a science fair or competition.
5. Most students (7 of 10) stated that a science competition includes conducting an experiment and being judged against other students, explaining that it was d ifferent from a fair. Two students considered a science competition to be the same as a science fair or club. When asked to provide an example of a science competition that they or their classmates have participated in or might participate in, some examples included:

a. Science bowl, Siemens competition, NIH competition, and county competitions. New In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B
b. Science club A B

c. Science competition A B54 Rationale:
1. Most students comprehend the item as intended.
2. Although almost all students indicated that they have not participate d in any of the activities, ETS

s uggests that item could be still used to measure the growth of out

- of

- school science activities.
Recommendation:
1. ETS recommends adding the item to the 2014 pilot administration.55 [ Grade 12 Science

– Student #9 ]

Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. A majority of students (9 of 10) stated that they were being asked if t hey had been to a museum that focuses on science. Only two students indicated that the visit had to take place during the current school year.
3. When asked what the phrase “science museum” meant to them, all students stated that it is a
museum that focuses o n any field of science.
4. When asked to provide examples of science museums, all students responded and listed:

a. The National Air and Space Museum, the Natural History Museum, and the Cryptologic
Museum.
5. When asked to provide examples of museums that are not for science, all students responded and
listed:

a. The National Gallery of Art, Spy Museum, Native American Museum, and Holocaust
Museum.
6. When asked to provide an example of a science museum that they or their classmates have visited or might visit, all stude nts responded and provided these examples:

a. The Baltimore Science Museum, the Natural History Museum, the National Air and Space Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but
was unable to name it.
7. Of the four studen ts who were asked who they went to the science museum with this school year, two went with their class, one went with friends, and one went with family.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip” as shown in the following two items, ETS suggests using the “not on a school trip” version of the
item to measure students’ out

- of

- school science

- related activit ies.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration. New In this school year, have you visited a science museum?

A. Yes
B. No56 [ Grade 12 Science

– Student #10 ]

Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. All students understood that the item was asking about field trips for science class.
3. All students stated that “on a school trip” means a trip that is sponsored by the school with school supervision. Most students (8 of 10) were able to report an example of a school trip

. They
included:

a. The Medical Museum in Walter Reed, the Natural History Museum, the National Aquarium, and the Air and Space Museum.
4. None of the students had visited a museum with their school during the current school year.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly dif ferentiate between “on a school trip” and “not on a school trip” as compared to the following item, ETS suggests using the “not on a school trip” version of the
item to measure students’ out

- of

- school science

- related activities.
Recommendation:
1. ETS recom mends not using the item in the 2014 pilot administration. New In this school year, have you visited a science museum on a school trip ?

A. Yes
B. No57 [ Grade 12 Science

– Student #11 ]

Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. A majority of students (9 of 10) were able to identify that the item was asking if they have visited
a science museum on a trip that was not sponsored by the school.
3. All students stated that “not on a school trip” meant that the visit was not sponsored by the school,
and done on the students own time.
4. More than half (6 of 10) of students gave examples of museum trips with the ir families. Some of
the museums mentioned were:

a. The National Air and Space Museum, and the Smithsonian.
5. A majority of students (9 of 10) reported that they have not visited a museum this year.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip” as compared to the previous item, ETS suggests using the “not on a school trip” version of the
item to measure student s’ out

- of

- school science

- related activities.
Recommendation:
1. ETS recommends adding this item to the 2014 pilot administration. New In this school year, have you visited a science museum that was not on a school trip ?

A. Yes
B. No58 Teacher Questionnaire

– Grade 4

[ Grade 4 Science

– Teacher #1 ]

Cognitive lab results:
1. The teachers shared a common understanding of the item as asking about how students were placed in smaller groups to work during class and how this related to students’ learning.

2. All teachers interpreted the phrase “students’ learning preferences” in sub

- item b to refer to students’ learning style.

3. Teachers interpreted “students’ readiness needs” in sub

- item b to mean students’ ability levels related to reading or content comprehension, though one teacher noted some confusion with the
term “students’ readiness needs, ” c alling it “weird wording

, ” and wondered if this referred to how English learners are grouped with other students.

4. Compared to the original version of the item (i.e., “Do you creat e groups within this class for science instruction on the basis of ability, ” solely with “yes/no” options), all five teachers responded that they preferred the way the item was currently asked, noting that this would allow them to better explain how they t each.

Rationale:
1. The cognitive lab report shows that all five teachers interpreted the items as intended, and compared to the trend version of the item, they preferred the cognitive lab tested version.

2. To address the confusion regarding the wording “learning preferences” and “readiness needs, ” and further clarify the intention of the two sub

- items, ETS recommends revising the two phrases to “learning preferences or styles” and “readiness or ability levels” respectively.

Recommendation : 1. Replace the trend version of the item (VC305014) with the cognitive lab tested and further revised
version. New (Revised based on VC305014 ) To what extent do you use each of the following student groupings for science instruction in your

classroom ? Fill in one oval on each line

.

Not at
all Small

e xtent Moderate

e xtent Large

e xtent

a. Groupings b ased on students’ interest
in science/science

- related topics A B C D New

b. Groupings based on s tudents’ learning preferences or styles A B C D New

c. Groupings b ased on students’
readiness needs or ability levels A B C D New59 [ Grade 4 Science

– Teacher #2 ]

Cognitive lab results:
1. Some phrases, such as “inquiry skills

, ” “develop skills in lab techniques

, ” and “scientific writing skills, ” were difficult to comprehend.
2. One teacher expressed that sub

- item f “developing problem

- solving (design) skills” is asking two different questions at the same time.
3. A majority of teacher s (4 of 5) interpreted the phrase “upper grade levels” in sub

- item d as intended, including any grade above fourth grade and middle school.
4. All teache rs referred to school, county, or science standards and curricula to explain what knowledge and skills that student would need for studying science in upper grade levels.
5. One teacher felt that the phrase “knowledge and skills” was necessary for clarificati on. VC970876 To what extent do you emphasize each of the following objectives in teaching scien ce to your fourth

grade class? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Increase students’ interest in

science A B C D VC970917

b. Increase awareness of the importance of science in daily life A B C D VC970928

c. Learn about applications of science

to environmental issues A B C D VC970930

d. Teach scientific facts and principles A B C D VC970919

e. Teach scientific methods A B C D VC970920 f. Equip students with the knowledge and skills needed for studying science in upper grade levels A B C D Revised sub

- item
based on
VC970922

g. Develop systematic observation

skills A B C D VC970929

h. Develop inquiry skills A B C D VC970923 i. Develop skills in lab techniques A B C D VC970926 j. Develop problem

- solving (design)

skills A B C D VC970925

k. Develop scientific writing skills A B C D VC97093160 Rationale:
1. The difficulties that teachers have with understanding some phrases might result from not grouping the sub

- items t ogether categorically. If these sub

- items were grouped together in a logical order, it would potentially clarify the issue.

2. Indeed, sub

- item f “develop problem

- solving (design) skills” is confusing in that it implies that problem

- solving skills equate experimental design skills. Removing the word “design” and the parenthesis would clarify the issue. Since experimental design s kills should have been included in “inquiry skills, ” there is no need to develop a sub

- item to address it.

Recommendation : 1. Re

- order the sub

- items (as shown above) and remove the word “design” and the parenthesis from
sub

- item f “develop problem

- solving (d esign) skills. ”61 [ Grade 4 Science

– Teacher #3 ]

Cognitive lab results:
1. One teacher found sub

- item j and the term “audiovisual” to be vague.
2. There are two occasions where teachers appeared confused, likely due to forgetting what the item stem was originally asking.
3. Teachers differentiated between the “school” and the “school system. ” They interpreted “school system” as including the school district and the county school office.
4. One teacher found it difficult to choose between options B (little) and C (some) for sub

- i tem s c (su pplies or equipment for science demonstrations ) and d (supplies or equipment for science labs)

. Rationale:
1. Cognitive lab results reveal that teachers have a consistent understanding on the revised version of the item (see next item) and preferre d the response options in the revised version of the item

. Recommendation : 1. Replace the item with the revised version of the item (Teacher, #4)

. VC970932 How much of the following instructional materials and other resources does your school system provide you with to teach science to your fourth

- grade class? Fill in one oval on each line.

None Little Some A lot

a. Science textbooks A B C D VC970953 b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957 f. Computers for students’ use in class A B C D VC970958
g. Computer labs A B C D VC970959 h. Computers for teachers’ use A B C D VC970960 i. Computerized science labs for classroom use A B C D VC970961 j. Audiovisual materials A B C D VC970962 k. Science kits A B C D VC970963 l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC97096462 [ Grade 4 Science

– Teacher #4 ]

Cognitive lab results:

1. Teachers differentiated between “school” and “school system” and the role of each entity in providing materials. Teachers would respond to the sub

- items differently between this item (the revised version) and previous item (the original version).

2. All teachers indicated that the options in this item (i.e., “ not at all, ” “ small extent, ” “ moderate
extent, ” “ large extent ” ) appropriate and sufficient for describing the resources available, and had a similar interpretation of each option.

3. Most teachers implied that they preferred this item (the revised version) compared to the previous item (the original item).

Rationale:
1. Cognitive lab results show th at this item (the revised version) perform ed better than the previous item (the original item).

2. To address the potential of losing information regarding resources provided by the county and

school district s caused by revising the item stem from “school system” to “schoo l

, ” ETS recommends revising the item stem from “school” to “school system (including your school,

county office , and school district)

. ”

Recommendation:

1. Replace the original item (VC970932) with th is item (the revised version ). New (Revised based on VC970932 ) To what extent does your school system (including your school, county office, and school district) provide the following to you ? Fill in one oval on each line.

Not at all Small

extent Moderate

extent Large

extent

a. Science textbooks A B C D VC970953 b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957 f. Computers for students’ use in class A B C D VC970958
g. Computer labs A B C D VC970959 h. Computers for teachers’ use A B C D VC970960 i. Computerized science labs for classroom

use A B C D VC970961

j. Audiovisual materials A B C D VC970962 k. Science kits A B C D VC970963 l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC97096463 [ Grade 4 Science

– Teacher #5 ]

Cognitive lab results:

1. All teachers understood the item was asking about different instructional methods they use with
their students.

2. All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to whether students could learn at a faster or slower pace based on how quickly they understood the
material or lesson.

3. All teachers stated that they would have answered sub

- item c “ v ary the pace of instruction for
some students ” the same way if it had instead read “pace my teaching differently for some
students. ” O ne teacher , however, said she would prefer the question asked in the alternate wording.

Rationale:
1. Cognitive lab results show that teachers consider the alternate wording of the sub

- item c as asking the same question, slightly preferring the alternate wording to (i.e., the original wording “ pace my teaching differently for some students

. ”

2. ETS recommen ds replacing the original wording for sub

- item c with this one (the revised wording) to be consistent with grade 8 questionnaire.

Recommendation:

1. Replace the original wording of the sub

- item c with this one (the revised wording). VC767811 When you teach science to your fourth

- grade class, do you do any of the following? Fill in one oval on

each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Use a different set of methods
in teaching some students A B C D VC767814

b. Supplement the regular course
curriculum with additional
material for some students A B C D VC767820

c. Vary the pace of instruction for
some students A B C D Revised sub

- tem based on
VC767821
d. Have some students engage in
different classroom activities A B C D VC767823

e. Set different achievement
standards for some students A B C D VC76782464 [ Grade 4 Science

– Teacher #6 ]

Cognitive lab results:

1. Two teachers interpreted the item as asking what outside opportunities teachers are providing, while others indicated that it asked what the teachers or the school provided.

2. All teachers answered the questions in regard s to activities or opportunities that the school organized, rather than what the teacher personally provided.

3. Teachers made a distinction between “science fair” and “science competition. ”

4. All teachers felt the response options provid ed were appropriate, al though one teacher commented that it would be better to ask teachers to quantify the frequency of events per school year. When
asked the follow

- up probe of whether it would better to use frequencies such as certain times a week, a s emester, or a school year, a majority of teachers (4 of 5) said this would be better.

Rationale:
1. To address a majority of teachers’ (4 of 5) preference on frequency options, and to be consistent

with the school questionnaires, ETS recommends revising the options to “1

– 3 time per year, ” “4

– 7

times per year, ” and “8 or more times per year. ”

2. ETS recommends revising the wording of sub

- item c to be consistent with school questionnaire.

Recommendation:

1. Add this item to the 2014 pilot questionnaire. New To what extent do you provide students with the following learning experiences? Fill in one oval on each

line.

Not at all Small
Extent
1

– 3 times

per year Moderate
Extent
4

– 7 times

per year Large
Extent

8 or

more times

per year

a. Science fairs A B C D New b. Science clubs A B C D New

c. Science competitions A B C D New d. Trips to science museums (including traditional museums and other similar sites) A B C D New65 Teacher Questionnaire

– Grade 8

[ Grade 8 Science

– Teacher #1 ]

Cognitive lab results:
1. All teachers interpreted the term “student groupings” to mean separating the students into work groups to collaborate on assignments.

2. Three of the five teachers interpreted the phrase “students’ le arning preferences” as referring to learning style, while the remaining two teachers interpreted the phrase as referring generally to interests and preferences.

3. Three of the five teachers expressed confusion regarding the phrase ‘students’ readiness needs” in

sub

- item c. One teacher noted that "students' readiness needs" could mean “the desire and willingness to learn. ”

4. Four of the five teachers stated that they use all three grouping methods in their teaching day.

5. All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” are appropriate for describing the extent of instructional grouping strategies.

Rationale:
1. The cognitive lab report shows that all five teachers interpreted the items as intende d. Three teachers further indicated that it is necessary to ask about various grouping methods.

2. To address the confusion regarding the wording “learning preferences” and “readiness needs, ” and further clarify the intention of the two sub

- items, ETS recomme nds revising the two phrases to “learning preferences or styles” and “readiness or ability levels” respectively.

Recommendation : 1. Replace the trend version of the item (VC305014) with th is cognitive lab tested and further
revised version. New (Revised based on VC305014 ) To what extent do you use each of the following student groupings for science instruction in your

classroom ? Fill in one oval on each line

.

Not at
all Small

e xtent Moderate

e xtent Large

e xtent

a. Groupings b ased on students’ interest
in science/science

- related topics A B C D New

b. Groupings based on s tudents’ learning preferences or styles A B C D New

c. Groupings b ased on students’
readiness needs or ability levels A B C D New66 [ Grade 8 Science

– Teacher #2 ]

Cognitive lab results:
1. A majority teachers (4 of 5) stated that the item asks about how much they focus on different areas and topics when teacher science.

2. All teachers interpreted the phrase “upper grade levels” as high school, and one of them included honors courses, AP courses, and college.

3. All teachers explained the phrase “knowledge and skills needed for study science in upper grade levels” as the princ iples and techniques that students need to be successful in high school science.

4. A majority of teachers (4 of 5) referred to state curriculum standards to determine what knowledge and skills will be needed beyond 8 th grade.

5. Most teachers (3 of 5) reported that their answer would change slightly but remain similar if the item had been phrased “prepare students for further study in science. "

Rationale:
1. The cognitive lab report on this matrix item did not reveal any difficulty among teachers with the wording of the item.

2. Since issues have been identified in grade 4 version of the item and recommended revision has been made, ETS recommends revising the item to be consistent with the grade 4 version. VC976013 To what extent do you emphasize each of the following objectives in teaching scien ce to your eighth

grade class? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Increase students’ interest in

science A B C D VC976015

b. Increase awareness of the importance of science in daily life A B C D VC976023

c. Learn about applications of science

to environmental issues A B C D VC970926

d. Teach scientific facts and principles A B C D VC976017

e. Teach scientific methods A B C D VC976018 f. Equip students with the knowledge and skills needed for studying science in upper grade levels A B C D Revised sub

- item
based on VC976019

g. Develop systematic observation

skills A B C D VC97625

h. Develop inquiry skills A B C D VC976020 i. Develop skills in lab techniques A B C D VC976022 j. Develop problem

- solving (design)

skills A B C D VC976021

k. Develop scientific writing skills A B C D VC97092767 Recommendation : 1. Re

- order the sub

- items (as shown above) and remove the word “design” and the parenthesis from
sub

- item f “develop problem

- solving (design) skills. ”68 [ Grade 8 Science

– Teacher #3 ]

Cognitive lab results:
1. The teachers displayed inconsistent interpretations of the item. Two interpreted the item as asking about the resources they have available; three interpret ed the item as asking about what resources were provided by their school system, including county or school district.
Rationale:
1. Cognitive lab results reveal that grade 8 teachers do not have a preference between the current
revised versions of the item ste m

. 2. A majority of teachers (4 of 5) indicated that the response options in the revised version of the item (i.e., “ not at all, ” “ small, ” “ moderate extent, ” “ large extent ” ) are appropriate and sufficient for describing the extent to which resources are provi ded by the school.

Recommendation:

1. Replace the item with the revised version of the item to be consistent with grade 4 teacher questionnaire. VC97 6028 How much of the following instructional materials and other resources does your school system provide you with to teach science to your eighth

- grade class? Fill in one oval on each line.

None Little Some A lot

a. Science textbooks A B C D VC976031 b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036 f. Computers for students’ use in class A B C D VC976037
g. Computer labs A B C D VC976039 h. Computers for teachers’ use A B C D VC976040 i. Computerized science labs for classroom use A B C D VC976041 j. Audiovisual materials A B C D VC976042 k. Science kits A B C D VC976043 l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC97604569 [ Grade 8 Science

– Teacher #4 ]

Cognitive lab results:

1. Teachers differentiated between “school” and “school system” and the role of each entity in providing materials.
2. Three teachers reported that the way this item is phrased (the revised version) and the previous item (the original version) would not make a difference in how they would answer any of the sub

items.
3. A majority of teachers (4 of 5) stated that the options in this item (i.e., “ not at all, ” “ small extent, ”

“ moderate extent, ” “ large extent ” ) are appropriate and sufficient for describing the extent to which resources are provided by the school.
4. All teachers shared a consistent understanding of the options and implied that they prefer these options compared to the ones in the previous item (th e original item).
Rationale:
1. Cognitive lab results show that this item (the revised version) perform ed better than the previous item (the original item).

2. To address the potential los s of information regarding resources provided by county and school district caused by revising item from “school system” to “school,” ETS recommends revising the item stem from “school” to “school system (including your school, county office, and school

distri ct)

. ”

Recommendation:

1. Replace the original item with this item (the revised version). New (Revised based on VC97 6028) To what extent does your school system (including your school, county office, and school district) provide the following to you ? Fill in one oval on each line.

Not at all Small

extent Moderate

extent Large

extent

a. Science textbooks A B C D VC976031 b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036 f. Computers for students’ use in class A B C D VC976037
g. Computer labs A B C D VC976039 h. Computers for teachers’ use A B C D VC976040 i. Computerized science labs for classroom use A B C D VC976041 j. Audiovisual materials A B C D VC976042 k. Science kits A B C D VC976043 l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC97604570 [ Grade 8 Science

– Teacher #5 ]

Cognitive lab results:

1. A majority of teachers (4 of 5) stated that the question is asking about different instructional methods, and how they plan and change lesson presentation methods to accommodate students with varying levels of understanding.

2. A majority of teachers (4 of 5) stated that the phrase “vary the pace of instruction fo r some students” meant providing students with different amounts of time to complete tasks, based on their different performance levels.

3. Three teachers stated that the y would have answered sub

- item c “ v ary the pace of instruction for
some students” the sam e way if it had instead said “pace my teaching differently for some students.” Two stated that their answers would be different because they interpreted the first
version to be more student

- specific, instead of referring to their overall teaching approach

. 4. All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some
students.”

Rationale:
1. Two teachers stated that their answer on sub

- item c would be different if it had instead asking “pace my teaching differently for som e students. ”

2. All teachers stated that they prefer the current phrasing, “vary the pace of instruction for some

students. ”

Recommendation:

1. Replace the original wording of the sub

- item c with this one (the revised wording). VC976085 When you teach science to your eighth

- grade class, do you do any of the following? Fill in one oval on

each line.

Not at all Small

extent Moderate

extent Large

extent

a. Use a different set of methods in teaching some students A B C D VC976086

b. Supplement the regular course
curriculum with additional
material for some students A B C D VC976088

c. Vary the pace of instruction for
some students A B C D Revised sub

- item
based on VC976091
d. Have some students engage in
different classroom activities A B C D VC976092

e. Set different achievement
standards for some students A B C D VC97609471 [ Grade 8 Science

– Teacher #6 ]

Cognitive lab results:

1. A majority of teachers (4 of 5) stated that the item asks about science

- related activities outside of

the classroom.

2. Three teachers made a distinction between “science fair” and “science competition, ” while the

other two did not.

3. Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” were sufficient to describe the extent of the learning experience provided to the

students.

4. One thought the choices were sufficient but felt that the term “science museum” was too limiting and suggested to add science field trips.

5. In regards to whether frequencies, such as certain times a week/a semester/a school year, would be

better than the response options provided for this item, opinions were divided. While two teachers rejected the idea that frequencies would be more relevant response options, two noted that they would be preferred. One respondent specifically suggested the following opt ions: per quarter, per
semester, or per year.
Rationale:
1. To address some teachers’ preference on frequency options and to be consistent with grade 4 questionnaires, ETS recommends revising the options to “1

– 3 time s per year, ” “4

– 7 times per year,” and “8 or more times per year.”

2. ETS recommends revising the wording of sub

- item c to be consistent with school questionnaire.

Recommendation:

1. Add this item to the 2014 pilot questionnaire. New To what extent do you provide students with the following learning experiences? Fill in one oval on each

line.

Not at all Small
Extent
1

– 3 times

per year Moderate
Extent
4

– 7 times

per year Large
Extent

8 or

more times

per year

a. Science fairs A B C D New b. Science clubs A B C D New

c. Science competitions A B C D New d. Trips to science museums (including traditional museums and other similar sites) A B C D New72 School, Grade 4:

[ Grade 4 Science

– School #1 ]

Cognitive lab results:

1. School administrators were not unified in their understanding of what the role and responsibilities would be for a “science specialist or coach. ”

2. Although a majority of administrators (4 of 5) indi cated that they did not have a part

- time

or full

- time science specialist or coach in their school, they emphasized that the responsibilities they associated with the role were taken on by classroom teachers and/or other types of specialists (e.g.

, instruct ional support teacher).

3. All administrators felt that the sub

- items in the following matrix item covered the majority of the responsibilities they would consider part of the role of a science specialist

or coach.

Rationale:

1. ETS posits that the intention of the item should be whether or not there are personnel

available in school performing the function of a “science specialist or coach,” not whether the particular title of “science specialist or coach” is available in the sch ool.

Since all administrators felt that the sub

- items in the following matrix item (item 2) covered the majority of the responsibilities of science personnel, the content of the sub

items provide a better context for administrators to comprehend the questi on than providing an abstract definition of “science specialist or coach.” Therefore, ETS recommends collecting the information through revising the following item in which the majority of the responsibilities of a “science specialist or coach” have been e xplained in

more detail.

2. It appears to be difficult for some school administrators to choose between the options of

“full

- time” and “part

- time” since the role may be filled by school, district, or county staff; or by someone who is not a school employee. T herefore, in the revised version of the following item, the availability of a “science specialist or coach” will not differentiate between “part

- time” and “full

- time.”

Recommendation : 1. Remove item from the 2014 pilot administration

. New (Revised based on VE013369)

Is t here a science specialist or coach available (full

- or part

- time) to fourth

- grade teachers at your school?

A. Yes, available full

- time at my school

. B. Yes, available part

- time at my school

. C. No73 [ Grade 4 Science

– School #2 ]

Cognitive lab results:

1. Although a majority of administrators (4 of 5) indicated that they did not have a part

- time

or full

- time science specialist or coach in their school, they emphasized that the responsibilities they associated with the role were taken on by classroom teachers and/or other types of specialists (e.g.

, instructional support teacher).

2. All administrators felt that the sub

- items covered the majority of the responsibilities they would consider part of the role of a science specialist or coach. New (Revised stem based on VE013372)
To what extent are any of the following a responsibility of the science specialist or coach , or other instructional support personnel, available to fourth

- grade science instruction teachers at your school?

Fill in one oval on each line.

Not available at my school Not
at all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about

science content or the teaching of science A B C C E VE013373

b. Conduct professional development for groups of

teachers about science content or the teaching of

science A B C C E VE013382

c. Provide science instruction to students on various topics A B C C E VE013383

d. Interpret and incorporate

Translate science standards

and current research into classroom practice A B C C E Revised
sub

- item
based on
VE013384

e. Provide science enrichment to some student groups A B C C E VE01338574 Rationale:

1. ETS posits that the intention of the item should be whether or not there are personnel

available for fourth

- grade science instruction functioning as a “science specialist or
coach”, not whether the particular title of “science specialist or coach” exists in school. ETS recommends adding “other instructional support personnel” to the item stem. Since

all administrators felt that the sub

- items covered the majority of the responsibility of a “science sp ecialist or coach” these sub

- items would provide administrators better context to comprehend the question than providing abstract definitions of “science specialist or coach.” Therefore, ETS recommends combining the matrix item with the previous skip
patte rn item and adding “ Not available at my school ” to the option list to capture the information originally collected through the previous skip pattern item (VE013369).

2. For sub

- item b, the word “translate” is vague (used in an analogous sense). ETS

recommends revising the word “translate” to the phrase “interpret and incorporate” to clearly reflect the kind of service and support that a “science specialist or coach” provides.

Recommendation:

1. Replace the trend item (VE013372) with this revised version of the i tem in the 2014 pilot

administration.75 [ Grade 4 Science

– School #3 ]

Cognitive lab results:

1. The definition for “science fair” and “science competition” differed among

administrators.

2. A majority of administrators (4 of 5) preferred the response options of “large extent, ”

“ moderate extent, ” and “ small extent,” versus frequencies (i.e., certain times a week/semester/school years )

. Rationale:

1. The Science Subject

- Area Standing Committee (06/19/2012) has suggested developing more items asking about students’ science activities outside of school. The Assessment Development Committee (ADC) (07/05/2012) specifically suggested that out

- of

- school activities could include science fairs, national competitions, robotics competitions, etc. To address these suggestions, ETS de veloped relevant items for all student, teacher, and school administrator questionnaires and tested these items in cognitive laboratories.

2. Since cognitive lab results revealed that school administrators differentiate between the

terms “science fair” and “s cience competition,” ETS recommends keeping them as two separate sub

- items.

3. To be consistent with grade 12 questionnaire, the phrase “in the last school year” and frequency ranges have been added to the item stem and response options to further quantify th e answers. Recommendation:

1. A dd to the 2014 pilot administration New
In the last school year, t o what extent did does you r school provide students with the following opportunities ? Fill in one oval on each line.

Not at all Small

e xtent

(1

– 3 times) Moderate

e xtent (4

–

7 times) Large

e xtent (8 times

or more)

a. Science fairs A B C D New b. Science clubs A B C D New

c. Science competitions A B C D New d. Trips to science museums (including traditional museums and other similar sites) A B C D New76 School, Grade 8 :

[ Grade 8 Science

– School #1 ]

Cognitive lab results:

1. Three of the five administrators indicated that they did not have a part

- time or full

- time science specialist or coach in their school

, and also emphasized the responsibilities they associated with the role taken on by classroom teachers and/or other types of specialists

such as “instructional lead teacher, ” “instructional coach, ” or “mentor teacher

. ”

2. Three of the five administrators agreed that the sub

- items in the following matrix item

covered all main responsibilities of a science specialist or coach.

Rationale:

1. ETS posits that the intention of the item should be whether there are personnel within school performing the function of a “science specialist or coach”, not whether personnel exists within the school with the specific title of “science specialist or coach”. Since most

administrators felt that the sub

- items in the following matrix item covered the majority of the responsibility of the personnel, the content of the sub

- items would provide better context for administrators to comprehend the question than providing abstract definitions of “science specialist or coach”. Therefore, ETS recommends collecting the information through revising t he following item in which the majority of the responsibilities of a “science specialist or coach” have been provided in detail.

2. It appears to be difficult for some school administrators to choose between “full

- time” and “part

- time” even though there are p ersonnel available fulfilling the role of a “science specialist or coach”, because not all personnel who function as a “science specialist or coach” are from the school. They could be also from the school district or the county (not a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the revised version of the following item, the availability of a “science specialist or coach” will not be differentiated between “part

- time” and “full

- time”.

Recommendation :
1. Remove the item f rom the 2014 pilot administration New (Revised based on VE013964)

Is there a science specialist or coach available (full

- or part

- time) to eighth

- grade teachers at your school?

A. Yes, available full

- time at my school

. B. Yes, available part

- time at my school

. C. No77 [ Grade 8 Science

– School #2 ]

Cognitive lab results:

1. The use of the word “translate” in sub

- item “d” is confusing to one administrator.

2. Although two administrators indicated that they did not have a part

- time or full

- time science specialist or coach in their school, they were able to answer the question based on

other available personnel performing the function of a “science specialist or coach. ”

3. Mo st (3 of 5) administrators felt that the sub

- items covered the majority of the responsibilities they would consider part of the role of a science specialist or coach.

4. Two of the five administrators agreed that the four answer choices are sufficient for

des cribing the extent of a science specialist or coach’s responsibilities in their school. New (Revised stem based on VE013966)
To what extent are any of the following a responsibility of the science specialist or coach , or other instructional support personnel available to eigh th

- grade science instruction teachers at your school?

Fill in one oval on each line.

Not available at my school Not
at all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about

science content or the teaching of science A B C C E VE013967

b. Conduct professional development for groups of

teachers about science content or the teaching of

science A B C C E VE013968

c. Provide science instruction to students on various topics A B C C E VE013969

d. Interpret and incorporate

Translate science standards

and current research into classroom practice A B C C E Revised

sub

- item

based on

VE013970

e. Provide science enrichment to some student groups A B C C E VE01397178 Rationale:

1. ETS posits that the intention of the item should be whether there are personnel available

for fourth

- grade science instruction functioning as a “science specialist or coach”, not whether the particular title of “science specialist or coach” exists in school. ETS recommends adding “other instructional support personnel” to the item stem. Since all

administrators felt that the sub

- items covered the majority of the responsibility of a “science specialist or coach”, these sub

- items would provide administrators better context to comprehend the question than providing abstract definitions of “science specialist or coach”. Therefore, ETS recommends combining the m atrix item with the previous skip pattern item and adding “ Not available at my school ” to the option list to capture the information originally collected through the previous skip pattern item ( VE013964 ).

2. For sub

- item b, the word “translate” is vague (used in an analogous sense). ETS recommends revising the word “translate” to the phrase “interpret and incorporate” to clearly reflect the kind of service that a “science specialist or coach” provides.

Recommendation:

1. Replace the trend item (VE013966) with th is revised version of the item in the 2014 pilot

administration.79 [ Grade 8 Science

– School #3 ]

Cognitive lab results:

1. All administrators m ade a clear distinction between “science f air” and “science competition. ”

2. A majority of administrators (4 of 5) agreed that the response options of “ not at all, ”

“ large extent, ” “ moderate extent, ” and “ small extent,” are sufficient for describing the extent of the opportunities their school provides.

Rationale:

1. Standing Committee (06/19/2012) has suggested developing more it ems asking about students’ science activities outside of school. ADC (07/05/2012) specifically suggested

out

- of

- school activities could include science fairs, national competitions, robotics competitions, etc. To address these suggestions, ETS developed r elevant items for all student, teacher, and school administrator questionnaires and tested these items in cognitive lab.

2. Cognitive lab results reveals that school administrators differentiate “science fair” and “science competition”, ETS recommends keeping them as two separate sub

- items.

3. To be consistent with grade 12 questionnaire, phrase “in the last school year” and range of the option “small extent ,” “moderate extent ,” and “large extent” haven been added to the item stem and options to further quantify the answers.

Recommendation:

1. A dd to the 2014 pilot administration. New
In the last school year, t o what extent did does you r school provide students with the following opportunities ? Fill in one oval on each line.

Not at all Small

e xtent

(1

– 3 times) Moderate

e xtent (4

–

7 times) Large

e xtent (8 times

or more)

a. Science fairs A B C D New b. Science clubs A B C D New

c. Science competitions A B C D New d. Trips to science museums (including traditional museums and other similar sites) A B C D New80 School, Grade 12 :

[ Grade 12 Science

– School #1 ]

Cognitive lab results:

1. Two of the five administrators stated that although they did have an individual fulfilling the role of a “science specialist or coach, ” it is difficult to determine whether or not they

are full

- or part

- time. One administrator further explained that none of the response options fit his school’s situation because the resource teacher is responsible for all high schools in the county and the 12 th grade teachers in his school rarely ask for support fro m this individual.

2. A variety different terms were often used in schools for describing individuals with the responsibilities of a “science specialist or coach, ” such as “resource teacher, ” “science department chairperson, ” and “science coordinator. ”

3. Overall, administrators felt that the response option did not fit their preferences

. 4. All administrators felt the sub

- items in the following matrix item covered the majority of responsibilities they would consider as parts of the role of “ science specialist or coach. ”

Rationale:

1. ETS posits that the intention of the item should be whether there is personnel available in school performing the function of “science specialist or coach”, not whether the particular title of “science specialist or coach” available in school. Since most

administrators felt that the sub

- items in the following matrix item covered the majority of the responsibility of the personnel, the content of the sub

- items would provide better context for administrators to comprehend the question than providing abstract definitions of “science specialist or coach”. Therefore, ETS recommends collecting the information through revising the following item in which the majority of the responsibilities of a “science specialist or coach” have been provid ed in detail.

2. It appears to be difficult for some school administrators to choose between “full

- time” and “part

- time” even though there are personnel available fulfilling the role of a “science specialist or coach”, because not all personnel who function a s a “science specialist or coach” are from the school. They could be also from the school district or the county (not a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the New (Revised based on VE015623)

Is there a science specialist or coach available (full

- or part

- time) to twelfth

- grade teachers at your school?

A. Yes, available full

- time at my school

. B. Yes, available part

- time at my school

. C. No81 revised version of the following item, the avail ability of a “science specialist or coach” will not be differentiated between “part

- time” and “full

- time”.

Recommendation :
1. Remove the item from the 2014 pilot administration82 [ Grade 12 Science

– School #2 ]

Cognitive lab results:

1. Overall, the administrators share a common understanding of the term “ science specialist

or coach

. ”

2. All administrators felt that the sub

- items covered the majority of the responsibilities they would consider part of the role of a “ science specialist or coach. ”

Rationale for recommendation:

1. ETS posits that the intention of the item should be whether there ar e personnel available

for fourth

- grade science instruction functioning as a “science specialist or coach”, not whether the particular title of “science specialist or coach” exists in school. ETS recommends adding “other instructional support personnel” to the item stem. Since all New (Revised stem based on VE015624 )

To what extent are any of the following a responsibility of the science specialist or coach , or other instructional support personnel available to twelf th

- grade science instruction teachers at your school?

Fill in one oval on each line.

Not available at my school Not
at all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about

science content or the teaching of science A B C C E VE015625

b. Conduct professional development for groups of

teachers about science content or the teaching of

science A B C C E VE015626

c. Provide science instruction to students on various topics A B C C E VE015627

d. Interpret and incorporate

Translate science standards

and current research into classroom practice A B C C E Revised

sub

- items

based on

VE015628

e. Provide science enrichment to some student groups A B C C E VE01562983 administrators felt that the sub

- items covered the majority of the responsibility of a “science specialist or coach”, these sub

- items would provide administrators better context to comprehend the question than providing abstract de finitions of “science specialist or coach”. Therefore, ETS recommends combining the matrix item with the previous skip pattern item and adding “ Not available at my school ” to the option list to capture the information originally collected through the previ ous skip pattern item ( VE015623 ).

2. For sub

- item b, the word “translate” is vague (used in an analogous sense). ETS recommends revising the word to “interpret and incorporate” to clearly reflect the kind of service that a “science specialist or coach” provid es. Recommendation:

1. Replace the trend item (VE015624) with this revised version of the item in the 2014 pilot

administration.84 [ Grade 12 Science

– School #3 ]

Cognitive lab results:

1. All administrators interpreted the item as asking about science

- related opportunities that the school provides beyond traditional classroom instruction.

2. Three of the five administrators agreed that there is a distinction between a “science fair” and a “science competition. ”

3. It appears that grade 12 administrators further differentiate “science museums

, ” including traditional museums, such as a natural history museums, and locations like aquariums or

other similar sites.

4. Most administrators (3 of 5) expressed a preference for having response options with frequencies, such as certain times a week/a semester/a school year. One administrator suggested adding ranges to quantify what was meant by a small, moderate and large extent. One administrator indicated that the best option would be to base answers on a school year.

Rationale:

1. Standing Committee (06/19/2012) has suggested developing more items asking about

students’ science activities outside of schoo l. ADC (07/05/2012) specifically suggested

out

- of

- school activities could include science fairs, national competitions, robotics competitions, etc. To address these suggestions, ETS developed relevant items for all
student, teacher, and school administrat or questionnaires and tested these items in cognitive lab.

2. Cognitive lab results reveals that school administrators differentiate “science fair” and “science competition”, ETS recommends keeping them as two separate sub

- items. New
In the last school year, t o what extent did does you r school provide students with the following opportunities ? Fill in one oval on each line.

Not at all Small

e xtent

(1

– 3 times) Moderate

e xtent (4

–

7 times) Large

e xtent (8 times

or more)

e. Science fairs A B C D New f. Science clubs A B C D New
g. Science competitions A B C D New h. Trips to science museums (including traditional museums and other similar sites) A B C D New85 3. Phrase “in the last school ye ar” and range of the option “small extent”, “moderate extent”, and “large extent” haven been added to the item stem and options to further quantify the answers.

Recommendation:

1. A dd to the 2014 pilot administration.2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

86 

# Cognitive Laboratories Report and Findings

(Eureka Facts)

## Cognitive Interviews of Background Question naire for

NAEP 2012

Science Assessment

Final Report

January 22 , 2013

### Contents

1. Introduction

................................ ................................ ................................ ......................... 1

1.1 Sample

................................ ................................ ................................ ......................... 1

1.2 Recruitment and Interview Procedures

................................ ................................ ......... 3

2. Cognitive Interview Results

................................ ................................ ................................ . 4

2.1 Student Background Questionnaire

................................ ................................ .............. 4

2.1.1. Grade 4 Students

................................ ................................ ................................ .. 4

2.1.2. Grade 8 Students

................................ ................................ ................................ . 24

2.1.3. Grade 12 Students

................................ ................................ ............................... 41

2.2 Teacher Background Questionnaire

................................ ................................ ............ 58

2.2.1. Grade 4 Teachers

................................ ................................ ................................ 58

2.2.2. Grade 8 Teachers

................................ ................................ ................................ 68

2.3 Administrator Background Questionnaire

................................ ................................ .... 77

2.3.1. Grade 4 Administrators

................................ ................................ ........................ 77

2.3.2. Grade 8 Administrators

................................ ................................ ........................ 82

2.3.3. Grade 12 Administrators

................................ ................................ ...................... 87Prepared by EurekaFacts, LLC 1 1



### Cognitive Interviews Report for BQ CogLab
Date of Report: January 22 , 2013

Interview Dates: 11 / 27 /2012

– 01 / 04/2013



### 1. Introduction

1.1 Sample

Cognitive interviews designed to test survey items developed for the National Assessment of Educational Progress (NAEP) Science Assessment were conducted with 15 school administrators, 10 teachers

, and 30 students. Each group of respondents was further div ided in to sub

- groups based on grade level: school

administrators for fourth, eighth, and twelfth

- grade students ; teachers of fourth and eighth

- grade students ; and students in fourth, eighth, and twelfth grade s

.

The administrators, teachers and students wh o participated in the cognitive interview study were recruited
from the greater Washington D.C. , and Baltimore, Maryland metropolitan area s

.
Administrator Sample

Five school administrators of fourth

- grade students , five school administrators of eighth

- grade students , and
five school administrators of twelfth

- grade students participate d in the cognitive interviews. D emographics of participating administrators and their schools are presented in Table 1

.

Table 1 : Participant and School Demographics for Administrator Respondent Group

Respondent Group Participant Demographics School Demographics
Total

Gender Race/Ethnicity SES School Size

4 th grade Administrator s Female: 5 White: 2 Black: 2 Low: 3

High: 2 Large: 3

Small: 2 5

8 th grade Administrator s Male: 3

Female: 2 White: 2

Black: 1 Other: 2 Low: 2

High: 3 Large: 3

Medium: 2 5

12 th grade Administrator s Male: 3

Female: 2 White: 3

Black: 2 Low: 2

High: 3 Large: 5 5

Total Male: 6

Female: 9 White: 9

Black: 5 Other: 1 Low: 7

High: 8 Large: 11

Medium: 2 Small: 2 15

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 2 2 Teacher Sample

Five fourth

- grade teachers and five eighth

- grade teachers who teach science during the current school year participated in the cognitive interviews

. D emographics of participating teacher s and their schools are presented in Table 2.

Table 2 : Participant and School Demographics for Teacher Respondent Group

Student Sample

Ten fourth

- grade students , ten eighth

- grade students , and ten twelfth

- grade students participated in cognitive

interviews

. The demographics of participating students are presented in Table 3.

Table 3 : Participant Demographics for Student Respondent Group

Respondent Group SES Gender Race/Ethnicity Total

4 th grade Student s Low: 4 High: 6 Male: 4

Female: 6 White: 5

Hispanic: 2 Black: 3 10

8 th grade Student s Low: 5 High: 5 Male: 4

Female: 6 White: 4

Hispanic: 2 Black: 4 10

12 th grade Student s Low: 5 High: 5 Male: 7

Female: 3 White: 8

Black: 1 Asian: 1 10

Total Low: 14 High: 16 Male: 15

Female: 15 White: 17

Black: 8 Hispanic: 4 Asian: 1 30 Respondent Group Participant Demographics School Demographics Total Gender Race/Ethnicity SES School Size

4 th grade Teachers Male: 2 Female: 3 White: 4

Other: 1 Low: 4

High: 1 Large: 2

Medium: 2 Small: 1 5

8 th grade Teachers Female: 5 White: 3 Black: 2 Low: 2

High: 3 Large: 1

Medium: 2 Small: 2 5

Total Male: 2 Female: 8 White: 7

Black: 2 Other: 1 Low: 6

High: 4 Large: 3

Medium: 4 Small: 3 10

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 3 3 

### 1.2 Recruitment and Interview Procedures

C ognitive interviews began on November 27, 2012 and were completed on January 4, 2013.

EurekaFacts recruited participants through email and telephone outreach. Trained interviewers screened candidates who responded to the outreach using the screening questionnaire approved by OMB (Appendix A). Parents of student candidates were informed that they would be compensated with two $25 gift cards, one for the student participant and one for the parent of the student. Administrators and teachers were informed that they would be compensated with a $40 gift card.

The cognitive interview process in thi s study followed the OMB approved script and interview protocol as presented in A ppendices A and B. Written consent was obtained from participants or the parents/legal guardians of participants before interviews were conducted. Participants were informed about the purpose of the study and reassured that their participation was voluntary and that their answers would be used only for

research purposes and w ould not be disclosed, or used, in identifiable form for any other purpose e xcept as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

I nterview sessions lasted for one hour. During the sessions , interviewers explained the think

- aloud process, conducted a think aloud practice question exercise, and then ask ed a series of verbal questions. Interviewers used several different cognitive interviewing techniques, including general think

- aloud, question

specific probes, and interviewers’ observations.

At the end of the session

, participants were briefly interview ed about their overall impressions of the survey

items. P articipants were thanked, remunerated, and asked to sign a receipt for their incentive payment.

The interview procedure for the fourth

- grade student group included pairing of a sub

- set of the survey items to reduce possible order effect . Items 2, 3, 4, 5, and 6 were presented in paired order, following the design presented in Table 4. Based on this paired order, six versions of questionnaire were developed; these

versions w ere assigned to participants from the fourth

- grade student group according to a predefined order.

Table 4 : Paired Design for Fourth

- grade Student Questionnaire

Questionnaire

version Question Order Participants

1 2 3 4 5 6 1 st and 7 th

2 2 3 4 6 5 2 nd and 8 th

3 2 3 5 4 6 3 rd

4 3 2 5 6 4 4 th and 9 th

5 3 2 6 4 5 5 th and 10 th

6 3 2 6 5 4 6 th

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 4 4 

### 2. Cognitive Interview Results

2. 1 Student Background Questionnaire

2.1 .1. Grade 4 Students

Grade 4 Science

– Student

— Item 1

Reading

Interviewer notes report the following observations:

 Three students had trouble with the word “circuits:” two mispronounced the word, while the third took an unusually long pause at first, and then read the word properly.

Comprehension

 Two students understood the item as asking about projects both at school and at home.

 When asked to define “activities or projects,” eight of the ten students described this as “things” or “stuff” to do. Three of these eight furth er described them as “fun,” while two described them as “work.” One student only provided examples of activities; one student could not define the phrase

.

 When asked what “circuits and energy” meant to them, three students said that this meant “ light ” or

“ power, ” while two could not define the phrase . Four students did not define the phrase, but rather provided an example (e.g., “wires”).

Response Process

 Four students reported that they would have responded to the item differently if the term “batteries and light” were used in place of “circuits and energy.” Two students explicitly said they would change responses because “batteries and light” is clearer to them, and would have changed their responses from “rarely” to “sometimes.”

Recommendations for Impr ovement

 Although only two students reported not understanding the term “circuits and energy,” nearly half (4 of 10) would have changed their response if the more concrete example “batteries and light” were used. In addition, there was much variation in how students understood the phrase “circuits and energy.” Thus, New (Revised based on VC519362)

In this school year, how often have you done activities or projects to learn about electricity (for example, circuits and energy)?

A Never

B Rarely
C Sometimes

D Often

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 5 5 we suggest that the tested alternative phrase, “batteries and light,” would be a more relevant example in

this item.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 6 6 Grade 4 Science

– Student

— Item 2

Response Process

 When asked how they would respond to this item if they had used chemicals only at certain times during the school year, students provided a variety of responses. Three students would have chosen “never or hardly ever,” two would have chosen “about once a week,” and two would have chosen “two or three times a week.” The remaining response options (“once every few weeks” and “every day or almost every day”) would have been chosen by one student each . One stud ent did not understand the probe.

 Only three students reported the time when they had carried out any activities, providing such time frames as “10 weeks into the year

, ” “more towards the beginning [of the year],” and “2 weeks ago.” Only two students provi ded specific examples of the activities, such as:

o Watching videos in class about what happens when you touch or drink unknown chemicals.

o Looking at and smelling different liquids.

 More than half of students (6 of 10) preferred the response options in item two to those of item three. Three students stated that they liked item two because the choices were more concrete than those of

item three.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315219

In this school year, how often have you done activities or projects to learn about chemicals (for example, mixing sugar or salt in water)?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 7 7 Grade 4 Science

– Student

— Item 3

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item.

 When asked to define “rarely,” “sometimes,” and “often,” the students used different approaches:

o Four students defined the answer choices using subjective, vague terminology: rarely means “hardly ever,” sometimes means “once in a while,” and often means “a lot” or “all the time.”

o Two students defined the response op tions in terms of comparisons to the other categories: "rarely” means between “often” and “never;” “sometimes” is very similar to “rarely;” “often means more than sometimes, and is between sometimes and always.”

o Two students defined response options in ter ms of frequencies: “sometimes” as once or twice a week; “often” as either every day or every other day.

o Two students used a mix of approaches described above.

Response Process

 When asked how they would respond if they had used chemicals only at certain t imes during the school year, three students chose “rarely,” three chose “sometimes,” and three chose “often.” The remaining student misunderstood the probe.

 Four students preferred the response options in item three to those in item two. One student said t hat these options were easier to read; another said they were “less to read” and therefore made more

sense.

Recommendations for Improvement

 Six of the ten students preferred the response options in item two to those in item three. Only four

students preferred the response options provided in item three. It appears that the students had difficulty differentiating between “rarely,” “sometimes,” and “often,” and also with matching their internally generated answers to the more abstract response options i n item three. Based on these difficulties with the response options in item three, we suggest that the response options used in item two are more appropriate for fourth grade students. New (Revised based on VC315219)

In this school year, how often have you done activities or projects to learn about chemicals (for example, mixing sugar or salt in water)?

A Never

B Rarely
C Sometimes

D Often

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 8 8 Grade 4 Science

– Student

— Item 4

Comprehension

Interviewer notes report the following observations:

 When asked to paraphrase the item, the students displayed a good understanding of the item.

 When asked to define “scientific tools,” students had difficu lty explaining this term, and instead provided examples of such tools.

o Four students identified “magnifying glass” as an example of a scientific tool; two students listed “pH testing strip;” two students mentioned “dropper.”

Response Process

 When asked ho w they would respond if they had only used tools at certain times of the year, three students would have responded “once every few weeks,” two would have answered “two or three times a week,” one would have answered “about once a week,” two would have answ ered “never or hardly ever,” and one did not understand the probe.

 Of the three students who viewed item four last , after items five and six, only one preferred the response options in item four, while two preferred item six. These two students explained t hat the response options in item four were confusing, and one added that option A and B in item four seemed to be the

same.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New (Revised based on VC315222, VC3 15227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 9 9 Grade 4 Science

– Student

— Item 5

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item and

instructions.

 When asked to define “scientific tools,” the students had difficulty explaining this term and instead provided examples of such tools.

o Students reported the following examples of scientific tools: “magnifying glass,” “magnet,” and “dropper.”

 When asked to define “rarely,” “sometimes,” and “often,” students defined the response options either using subjective, vague terminology (rarely means “hardly ever,” sometimes means “once in a while,” and often means “a lot” or “all the time”), or in term s of frequencies (“often” as either every day or every other day; “sometimes” as once or twice a week; “often” as either every day or every other day).

Response Process

 When asked how they would answer if they had only used tools at certain times of the year, four students would have answered “rarely,” three would have answered “sometimes,” and one would have
answered “never.” One student was not asked this probe, and one student could not provide an answer.

Recommendations for Improvement

 There are no re commendations for improvement for this item based on the evidence collected from the

interviews. New (Revised based on VC315222, VC315227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A Never

B Rarely
C Sometimes

D Often

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 10 10 Grade 4 Science

– Student

— Item 6

Response Process

 When asked how they would answer if they had only used tools at certain times of the year, half of the students (5 of 10) would have answered “yes, a little ,” two would have answered “no,” and two would have answered “yes, a lot.” One student did not understand the probe.

 Between items four, five, and six, half of the students (5 of 10) preferred item six, three students preferred item four, and two preferred item five. Three students indicated that they did not prefer item five because the words of the response options were hard for them to understand. Students typically preferred item six because it was simpler than item four, and easier to understand than i tem five.

Recommendations for Improvement

 Student verbal reports and responses to the probing questions show that item six was the preferred version of this question, and easiest for fourth grade students to understand. New (Revised based on VC315222, VC315227, and VC315229)

Have you used scientific tools in school this year?

A No

B Yes, a little

C Yes, a lot

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 11 11 Grade 4 Science

– Student

— Item 7

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item. However, they did not mention the time period included in the item stem

. Response Process

 When asked if they had i ncluded material that they read outside of school in their answers, five students indicated that they had not, three indicated that they did include readings outside of school, and two

students did not read science books at all. Half of the students (5 of 10) also indicated that they did not
read materials about science outside of school.

 Of the five students who responded that they had read a book or magazine about science in this school year, all indicated that they read the most often in the classroom.

 Only four students stated that they read books about science that are not for school work.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315239

In this school year, how often have you read a book or magazine about science?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 12 12 Grade 4 Science

– Student

— Item 8

Reading

Interviewer notes report the following issues with reading:

 Two students showed hesitation in pronouncing the word “presented.”

Comprehension

 When asked to paraphrase the item, the students displayed a good unde rstanding of the item. However, they did not mention the time reference included in the item.

 When asked to define “presented,” students varied in their responses. Half of the students (5 of 10) said that it involved “showing” something, while two used the word “share” instead. Six of the students stated that it involved showing something in front of others, and five explicitly referred to the classroom.

 When asked if “to show what you have learned” was different from “to present what you have learned,”

six students said that these were the same, while one student said that they differed. Two students did not answer, and one student did not understand what “show what you’ve learned” meant.

Response Process

 Six of the ten students indicated that they have pre sented something to their class. Of these six, four presented information that they learned, and the other two presented a project that they worked on.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315259

In this school year, how often have you presented what you learned about science to your class?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 13 13 Grade 4 Science

– Student

— Item 9

Comprehension

 When asked to paraphrase the item, three students stated that it asked if they had used measurements, and did not note that the item asks whether they have talked about them.

 When asked to define “talked ab out,” the majority of students (8 of 10) answered in a similar manner, stating that it meant to “say something” or “talk to someone.” Only one student did not know how to respond.

 When asked to define the term “measurements or results,” the students showed good understanding of these terms, and explained them by referring to results, the end

- effect or product of experiments, and solutions to problems. Two students used examples to help illustrate what they were trying to say.

Response Process

 Seven of the ten students stated that they “never or hardly ever” talked about measurements or results. However, when probed, two of these students reported talking with others: one with a teacher, and the

other with classmates.

 The remaining three reported that they talked about measurements or results with a teacher, classmates, or the whole class.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315288

In this school year, how often have you talked about measurements or results from your science activities or projects?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 14 14 Grade 4 Science

– Student

— Item 10

Reading

 Interviewer notes report no major problems with reading in this item.

Comprehension

 Interviewer notes report no major problems with comprehension in this item.

 When asked to paraphrase the item, the students di splayed a good understanding of the item. However, they did not mention the time period referenced in the item stem

.

 All students but one defined “report” in a similar manner, as writing or sharing information from measurements or science projects with oth er people.

Response Process

 Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this school year; two responded “once every few weeks;” three said “two or three times a week.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315265

In this school year, how often have you written a report on your science activities or projects?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 15 15 Grade 4 Science

– Student

— Item 11

Comprehension

 All students had a similar understanding of the phrase “science journal,” describing it as something you write in to keep track of information you have for science topics.

 The descriptions that students pro vided for the phrase “lab write

- up” were less consistent.

o Six students either did not know what the phrase meant or defined it incorrectly; another student had never heard of it, but had an intuitive grasp of its meaning.

o Of the students that did define the term, three described it more as a report, where they write the results of a project or test, and one described it more as a journal, where general science notes and information go.

Response Process

 Four of the ten students would call the written work they do for a science activity or project a “science journal” (these same four students were unable to define a “lab write

- up”). Two students would call the work a “report,” while two students would call it something else (a “writing piece” and “results”), and one

was unsure what to call it. Only one student said he/she would call the work a “lab write

- up,” but did not know why.

 Half of the students (5 of 10) reported having “never or hardly ever” been asked to write about science activities or projects. Ho wever, only two students who answered “never or hardly ever” to the previous item (item ten) responded the same way to this item (item eleven).

Recommendations for Improvement

 Verbal reports and student responses to item

- specific probes show that over half of the students did not understand the phrase “lab write

- up,” but all students had at least a rudimentary grasp of the phrase “science journal.”

 Although the interviewer notes for the previous item (item ten) show that all but one student had a good

unde rstanding of the term “report,” the more generic formulation of item eleven (“asked to write about your science activities or projects”) produced a different answer pattern among students, suggesting that item ten did not capture all possible information f rom students on this topic. We suggest that it might be helpful using the question formulation from item eleven, with the addition of specific examples of written deliverables, such as reports, science journals, and lab write

- ups. New (Revised based on VC315265)

In this school year, how often have you been asked to write about your science activities or projects?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 16 16 Grade 4 Science

– Student

— Item 12

Reading

Interviewer notes report the following issues with reading:

 In the phrase “library resources for science,” two students had trouble pronouncing “resources,” and another skipped reading the phrase.

Comprehension

 When asked, five students interpreted the item as asking if they had ever used library resources for

science.

 Two students could not define the phrase “library resources.” Of the remaining eight students, all gave books as an example of a library resource. Three also gave the example of computers, and two others gave the example of magazines.

 Only after being probed did students acknowledge the internet as being a library resource. After being asked, six students said that internet was a library resource, w hile three said that it was not. When probed, two students also acknowledged computers as library resources; another student was unsure, because computers in his/her school were not for fourth graders.

Response Process

 Nine students were asked whether they thought of their school library, the public library, or both. Of the nine, seven thought only of the school library, one thought of both, and one answered that they “do not go to the library.”

 When explicitly asked about individual resources, three studen ts included encyclope dias, and three

included videos as library resources.

Recommendations for Improvement

 Only two students were unable to explain the meaning of the term “library resources.” However, verbal reports and student paraphrasing of item twelve show that a majority of students thought only of books as library resources. We suggest including in the item, in parentheses, examples of different types of library resources (such as magazines, computers, internet, and videos), in order to aid understan ding of this term and remind students of library resources other than books. VC315266

In this school year, how often have you used library resources for science?

A Never or hardly ever

B Once every few weeks
C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 17 17 Grade 4 Science

– Student

— Item 13

Comprehension

 On the whole, the students understood the phrase “science activities” in a similar manner: eight of ten described it as activities or projects that one does, that are related to a science topic. Only two did not explicitly refer to a science topic.

 Students demonstrated a good and consistent understanding of the phrase “not for schoolwork : ” five students said it meant activities done at home, and three said it meant being “out of school.”

Response Process

 When asked to provide examples of activities they had done that were not for schoolwork, two students gave technology/engineering examples (e.g., fixing a TV), two gave natural science examples (e.g., studying bugs), and two gave chemical examples (e.g., mixing substances together). Two students who answered “never or hardly ever” could not remember any examples of activities they had done that were

not for schoolwork.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315410

How often do you do science activities that are not for schoolwork?

A Never or hardly ever

B Sometimes
C Often

D Always or almost always

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 18 18 Grade 4 Science

– Student

— Item 14

Comprehension

 When asked to define a “science fair,” six students described it as related to making experiments . O f these six, two also stated that “science fair” involved showing their work to other people. One student defined it as showing something, but did not mention experiments.

 Two students defined science fairs as having a competition element; one said first place gets a prize, and the other said science fair was synonymous with science competition.

 Two students d id not understand what a “science fair” was; they knew only that it had something to do

with science.

Response Process

 When asked to provide an example of a science fair, there were few prevailing trends. Six students stated that they had not participated in one; two referred to “event nights” at their schools where science activities took place, but they did not present their own experiments.

Recommendations for Improvement

 Verbal reports and answers to the probing questions show that more than half of students had not participated in a science fair and had a vague understanding of the term. Thus, we suggest that “science fair” might not be an appropriate term to use. New

In this school year, have you participated in a science fair?

A Yes

B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 19 19 Grade 4 Science

– Student

— Item 15

Comprehension

 When asked to define a “science club,” all students provided responses indicating that science clubs had to do with people meeting together to learn more about science or do science activities. Additionally , four students explicitly indicated they were after

- school activities.

Response Process

 When asked to provide an example of a science club, four students provided concise examples of specific clubs, three could only give examples of activities they had done in the clubs, and one student had never participated in a club. Two students could not recall any examples of science clubs. E xamples provided were:

o STEM club, robotics club, Hands

- On Science, and Mad Science.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you participated in a science club?

A Yes B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 20 20 Grade 4 Science

– Student

— Item 16

Comprehension

 When asked to define a “science competition,” five students gave responses indicating that this activity involves “prizes” or “first place, second place, etc.” Six students said it involved either creating or developing their own project, or that it was like a science fair.

Response Process

 When asked to provide an example of a science competition, three students either gave “science fair” as their example, or gave their same example from item fourteen (which asked for an example of a science fair). One student said fairs and competitions were similar, but not the same.

 Of the five students that were asked, all had heard of a robotics competition, but only one indicated that

th ey had participated in a robotics competition.

 Students were asked whether or not science fairs, science clubs, and science competitions were different things. Over half (6 of 10) said that they were all different things; three said they were all the

same; one said that fairs and competitions were the same, but clubs were different. All three of the students who indicated that they were all the same gave the reason that they all dealt with science.

Recommendations for Improvement

 Verbal reports and answers to the probing questions show that while a majority of the students had a very vague understanding of the term “science fair,” a majority had quite a good understanding of the term ”science competition.” Thus, it is likely that fourth graders do not have m uch experience with science fairs, so “science competition” may be a more appropriate term to use. New

In this school year, have you participated in a science competition?

A Yes B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 21 21 Grade 4 Science

– Student

— Item 17

Comprehension

 Because “science museum” is a relatively vague term, there was some variance in the definitions that students provided. However, most (7 of 10) indicated that it either involved scientific activities, lesso ns, and/or technology, or gave appropriate examples. Two students were unsure of the meaning, and one

student seemed to include non

- scientific museums in their definition.

 Nine students were asked what specific museums they considered to be science and non

- science museums. Two students could not remember any science museums; two students could not remember

any non

- science museums.

 When asked to give an example of a science museum that they had been to, three students could not remember any museums.

Response Process

 When students were asked what museums they thought of as science museums, examples included:

o The Museum of Natural History, the Space Museum, and an animal museum called Amazing

Animals.

 When students were asked what museums they thought of as non

- science museums, examples

included:

o The National History Museum, the Wax Museum, and the Native American Museum.

 When asked to give an example of a science museum they had been to, three students stated that they had not been to any museums. One student p rovided an example that was not a museum, and another provided three examples, one of which was a non

- science museum. Examples of relevant museums

included:

o Museum of Natural History, Air and Space Museum, and the Animal Museum.

 Only one student was asked who he/she had visited a science museum with, and responded, “the entire fourth grade.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 22 22 Grade 4 Science

– Studen t

— Item 18

Comprehension

 When asked to define “on a school trip,” three students stated it was a trip involving their class and/or grade; three students mentioned riding on a school bus; three students said it was a field trip; two students simply re

- worded the phrase (e.g., “a trip for school”).

 Of the seven students asked to provide an example of a science museum that they visited on a school trip, two could not think of any examples.

 Of the five students who provided examples, tw o referenced museums, and two could only discuss the activities they did at the museums (one simply referenced a field trip to the Chesapeake Bay). The
referenced museums were:

o The Animal Museum and the Kid’s Museum in Baltimore Harbor.

 Only one student had been to a science museum on a school trip this year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum on a school trip?

A Yes B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 23 23 Grade 4 Science

– Student

— Item 19

Comprehension

 When asked to define “not on a school trip,” six of the ten students stated it was a trip that did not include school individuals or classmates, while five students described it as involving family and/or

f riends.

 Even though five students gave examples of trips they went on that were not for school, four of them were unable to provide specific names of museums. Of these four students, two described what the museum was like, and the other two could only say that they had been to a museum.

 Five students provided the following examples of trips that were not for school:

o The Air and Space Museum, the Natural History Museum, and the American Indian Museum.

 Only one student indicated that they had visited a science museum this yea r that was not on a school trip

. Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum that was not on a school trip?

A Yes B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 24 24 

### 2. 1 .2. Grade 8 Students

Grade 8 Science

– Student

— Item 1

Reading

Interviewer notes report the following observations:

 Three students misread “barometer” in sub

- item “f.”

 Three students skipped over the examples in parentheses for all sub

- items.

Comprehension

 Four students did not understand the reference period of the item to be “this school year;” instead, their responses reflected prior school years as well.

 When asked what the phrase “hands

- on activities or projects” meant to them, half of the students (5 of 10) indicated that this meant working on assigned tasks. Two students understood this to mean working VC304986

In your science class this year, have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Yes No

a. Living things (for example, plants, animals, bacteria) A B VC304988

b. Electricity (for example, circuits and energy) A B Revised sub

- item

based on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B VC304991

d. Rocks or minerals (for example, identifying types) A B VC305007

e. Magnifying glass or microscope (for looking at small things) A B VC305008

f. Thermometer or barometer (for making measurements) A B VC305009

g. Simple machines (for example, pulleys and levers) A B VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 25 25 with their peers on an activity. The remaining three students provided limited, vague definitions for the phrase.

 When asked the meaning of the phrase “circuits and energy,” students’ responses varied greatly. While two students understood a circuit as something consisting of wires, another two understood it as something that electricity moves through. Two students understood energy to be synonymous with the term “power.” Four students provided a limited and vague definition for the phrase, showing a partial understanding that “circuits and energy” referred to electricity. Two of the students were unable to provide a definition.

Response Process

 Of the f our students wh o answered “yes” to sub

- item “b ” (“electricity”), three recalled that the activity they thought of when formulating their answer was related to using wires to light a light bulb. The remaining respondent recalled learning about electricity, but could not provide examples of specific activities related to this topic.

 When asked how they would respond to sub

- item “b” if the example had been “batteries and light” instead of “circuits and energy,” half (5 of 10) responded that they would have cha nged their answers. Four of these students would have changed their answers from “no” to “yes,” while the fifth would have changed from “yes” to “no.”

Recommendations for Improvement

 There was significant variation in how students understood the phrase “ci rcuits and energy.” In terms of their recollections of activities relating to electricity, students predominantly recalled activities where they had used batteries, wires, and/or light bulbs. In addition, probing revealed that if sub

- item “b” included

the example of “batteries and light,” instead of “circuits and energy,” twice as many students would have answered “yes” to this sub

- item. Thus, we recommend that the tested alternative phrase, “batteries and light,” would be more a relevant example to include in this item.

 A large proportion of the students did not appear to understand that the item referred only to “this year,” and included in their answers activities from previous school years. The term “this year” can be interpreted as “this school year,” “this calendar year,” or “at any time between now and last year at the same time.” If it is important that this measurement reflect a specific time period, we suggest providing a more specifically defined time reference in the item’s stem.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 26 26 Grade 8 Scie nce

– Student

— Item 2

Reading

Interviewer notes report the following observations:

 Three students misread “barometer” in sub

- item “f.”

Response Process

When asked how they would respond if they had only done activities or projects at certain times during the school year (e.g., doing an activity every day for three weeks, but not since then), students provided a variety of responses.

 Four students said th ey would respond in the lower half of the scale. Of these four, three students said they would answer “once every few weeks,” while the fourth would answer “never or hardly ever.”

 Two students said they would choose the middle response option, “about once a week.” New (Revised based on VC304986)

In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never or hardly

ever Once

every few

weeks About

once a week Two or

three
times a

week Every

day or

almost

every day

a. Living things (for example, plants, animals, bacteria) A B C D E VC304988

b. Electricity (for example, circuits and energy) A B C D E Revised sub

item based on
VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D E VC304991

d. Rocks or minerals (for example, identifying types) A B C D E VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D E VC305008

f. Thermometer or barometer (for making measurements) A B C D E VC305009

g. Simple machines (for example, pulleys and levers) A B C D E VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 27 27  Two students said they would answer in the upper half of the scale. One indicated that they would answer “two or three times a week;” the other would choose “every day or almost every day.”

 One student said they would arbitrarily choose either the first or last response option on the scale, because the item’s reference period lacked specificity. This student suggested that the item should ask how often they did each of these things in “each science unit,” rather than the entire school year.

Reco mmendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 28 28 Grade 8 Science

– Student

— Item 3

Reading

Interviewer notes report the following observations:

 Three students skipped over the examples in parentheses for all sub

- items.

Comprehension

 Two students did not understand the reference period of the item to be “this school year;” instead, their responses reflected prior school years as well.

Response Process

 Six students defined the response options using more subjective and vague terminology, with “rarely” meaning “not much,” “sometimes” meaning “once in a w hile,” and “often” meaning “a lot.” Of these six students, four defined “rarely” in terms of comparisons to the other categories (“not often,” “less than sometimes”). The definitions for “sometimes” and “often” varied among these six students, but their

de finitions showed that they understood the concept in all cases.

 Four students defined the response options according to more objective descriptions. Among these respondents, the term “rarely” was defined by all four students as either “once” or “once a mon th;” the term “sometimes” had varying definitions; the term “often” was defined by all four students as once or multiple times a week. New (Revised based on VC304986)

In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never Rarely Sometimes Often

a. Living things (for example, plants, animals, bacteria) A B C D VC304988

b. Electricity (for example, circuits and energy) A B C D Revised

sub

- item

based on
VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D VC304991

d. Rocks or minerals (for example, identifying types) A B C D VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D VC305008

f. Thermometer or barometer (for making measurements) A B C D VC305009

g. Simple machines (for example, pulleys and levers) A B C D VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 29 29  When asked how they would respond if they had only done activities or projects at certain times during the school year, a majority of students (7 of 10) said that they would respond “sometimes.”

 When asked which version of the response options they preferred, eight of the ten students said they preferred the response options used in this item. Many implied that answering the item with the dichotomous response options used in item one was easy, but that these options were too broad to accurately assess their internally generated answers. Moreover, the students explained that it was difficult to answer the item with frequencies , as used in item two, because the response options were too specific.

Recommendations for Improvement

 Verbal reports and answers to probing questions show that the students strongly preferred the response option scale used in item three, as compared to i tem two. It appears that the students could more easily fit their internally generated answers into the response options provided in item three; in addition, these response options placed a lower overall cognitive burden upon students. When asked how they would respond if they had only done activities or projects at certain times during the school year, student

responses showed lower variability in item three, as compared to item two, which may indicate that the response scale in item three is able to produ ce responses with higher reliability.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 30 30 Grade 8 Science

– Student

— Item 4

Comprehension

 When asked to state the item in their own words, the students displayed a good understanding of the item or instructions. However, further probing revealed that four students missed, or did not take into account, that their answer should focus on activities performed for sci ence class, and they consequently considered media that was not for school work when answering this item.

 When asked what the phrase “science topics” meant, five students implied that this meant subjects studied in science class, two students gave discrete examples of specific science topics, and two answered that this phrase meant “anything” that has to do with science.

Response Process

 When asked if they would respond differently if the item used the term “science” instead of “science topics,” seven said they would answer differently. The most common reason given for this change of response among students (4 of 7) was that the term “science” was broader than “science topics,” and would allow them to consider more types of media as related to “science” in their responses.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or hardly

ever Once

every few

weeks About

once a week Two or

three

times a week Every

day or

almost

every day

a. Read a science textbook, in

class or at home A B C D E VC546510

b. Read a book or magazine about science topics A B C D E Revised sub

item based

on VC305295

c. Use the Internet to learn about science topics A B C D E Revised sub

item based

on VC720562

d. Watch a movie, video, or DVD about science topics A B C D E Revised sub

item based

on VC305307

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 31 31 Grade 8 Science

– Student

— Item 5

Comprehension

 When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub

- items “c” and “d” implied summarizing what they did or learned in class. The other five students defined the phrase as generally talking with people (students or teach ers) in their class.

 When asked what is meant by the term “lab write

- up,” half of the students (5 of 10) described it as something akin to a report, describing a science lab that they had completed. Three students described it as another type of class assignment unrelated to a science lab. Two students were unable to provide a
definition. VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or hardly

ever Once

every few

weeks About

once a week Two or

three
times a week Every

day or

almost

every day

a. Identify questions that can be addressed through science experiments A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements you took for your science project or activity A B C D E VC720635

d. Talk about the results of your science project or activity A B C D E VC720636

e. Watch your teacher do a science experiment or activity A B C D E VC720638

f. Make graphs or charts of the results from your science project or activity A B C D E VC720639

g. Write a report on your science project or activity A B C D E VC720641

h. Write a lab write

- up on your science project or activity A B C D E New

i. Write a science journal A B C D E New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 32 32  When asked the meaning of the term “science journal,” responses varied greatly, and included the following: where they write their “warm

- up” exercise (2 students); whe re to write notes (2 students); where to write about experiments (2 students); something that is used to describe what happened in class (2 students); where one can write about science in general (2 students).

Recommendations for Improvement

 Interview not es revealed considerable variability in students’ understanding of the terms “lab write

- up” and “science journal.” In order to avoid any potential confusion in the future and to ensure consistent interpretation among students, we suggest providing a standa rdized definition of these terms.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 33 33 Grade 8 Science

– Student

— Item 6

Comprehension

 Three students indicated during their think

- aloud that they found the item and the term “library resources” to be vague.

Response Process

 When probed, seven students stated that they were thinking about their school’s library when answering this item and three stated that they were thinking of both their school’s library and a public library.

 In regards to what students thought of as library resources, the top library resource identified by students was books (9 of 10). Other resources that students mentioned, unaided, were computers (4 students), internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1 student), and magazines (1 student).

 When asked directly about each resource, eight students said they thought of computers as resources while answering the item, seven students thought of encyclopedias, five thought of the internet, five of magazines, and four thought of videos.

R ecommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC315266

In this school year, how often have you used library resources for science?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 34 34 Grade 8 Science

– Student

— Item 7

Comprehension

 Interviewer notes do not indicate any consistent problems in students’ understanding of the item or

instructions.

 When asked what was meant by the phrase “science

- related activities” in sub

- item “a,” the students either provided general, broad descriptions related to activities that had “anything to do with science,” or offered some examples such as measurements, experiments, and research.

 When asked what was meant by the p hrase “not for schoolwork,” five students emphasized in their responses that these activities were done outside of school, or were not assigned schoolwork. Four students emphasized in their explanations that these activities were done based on personal int erests.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in

one oval on each line.

Strongly disagree Disagree Agree Strongly

agree

a. I do science

- related activities that are
not for schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help
me in the future. A B C D VC305353

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 35 35 Grade 8 Science

– Student

— Item 8

Reading

Interviewer notes report the following observations:

 Three students had trouble reading the word “ecology” in sub

- item “a.”

Comprehension

 When asked what was meant by the term “covered” in the item stem, seven students provided a general, broad definition for the term ( e.g., “gone over,” “what we learned”). In their definitions, three of the students included a term that described the extent of the learning (“we learned about it completely,” “we know it thorough ly”).

 Students that answered “yes” to any of the sub

- items were asked to describe how the material was “covered” in their class. A majority of the students (8 of 9) answered this by providing the names of the topics that were covered. All of the topics me ntioned by students fit within the existing sub

- item’s category. One student described the types of assignments and materials that were used to learn each topic.

Response Process

 When probed, seven students said that they would prefer a larger scale that asked to what extent the topics were covered, as opposed to “yes” and “no” response options. The main reason for this preference, according to five of these seven students, was the ability to more accurately assess the extent to which these topics were cov ered. Of the three students who preferred the yes/no format, two had previously included a term that described the extent of learning in their description of the term
“covered.”

Recommendations for Improvement

 Based on the students’ verbal reports and answ ers to probes, we suggest that a change in item scale would be relevant, from a yes/no scale to a Likert scale that measures the extent to which a topic was covered. Overall, the students preferred the Likert scale

- based response options and seem capable o f accurately assessing sub

- items on this scale. VC304978

In your science class this year, which of the following topics have been covered? Fill in one oval on each

line.

Yes No

a. Life science (for example, biology, the human body, or ecology) A B VC304982

b. Physical science (for example, energy, physics, or chemistry) A B VC304983

c. Earth and space science (for example, geology or astronomy) A B VC304984

d. Engineering and technology (for example, designing solutions to problems) A B VC304985

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 36 36 Grade 8 Science

– Student

— Item 9

Comprehension

 When asked what was meant by the term “science fair” in sub

- item “a,” all ten students provided explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven of the ten explanations also indicated that these projects wer e shown or talked about to others.

 When asked what was meant by the term “science club” in sub

- item “b,” students had varying answers. Five students indicated that these clubs occurred after school, one student said that they occur annually,

and the oth er four students did not discuss when clubs occurred. Four students included in their definition that the purpose of the club was to do science activities, and three noted that the purpose of

the club was to discuss or talk about science.

 When asked what was meant by the term “science competition” in sub

- item “c,” nine students provided a definition that described it as similar to a science fair, where one makes a project, invention, or experiment and competes against others. One student mentioned that peo ple also compete by answering questions. One student was unable to define the term.

Response Process

 Eight students were able to provide an example of a science fair that either they or a classmate had participated in.

o Three students cited examples from elementary school, one referred to a science expo, one
student referred to the school STEM fair.

 Five students were able to provide an example of a science club that either they or a classmate had participated in. Examples included:

o Technology club, life s cience club, a science club involving robotics, and Project Change.

 Only one student was able to provide an actual example of a science competition, which they described as a “science club competition, where they had to build things to compete in certain c hallenges.” One student answered that a geography bee was a science competition. The other eight students were unable to retrieve a cogent example.

 All ten students answered “no” to sub

- items “a” and “c,” and nine answered “no” to sub

- item “b.” New

In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 37 37 Recomme ndations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 38 38 Grade 8 Science

– Student

— Item 10

Comprehension

 All ten students interpreted the item as asking about trips to museums that involve science.

 When asked what was meant by the term “science museum,” more than half of students (6 of 10) gave general and vag ue definitions for the term (e.g., “a museum that has to do with science of any kind” and “a museum about science”). The other four students provided examples of what might be found in a science museum (e.g., “a collection of experiments” and “rocks and di nosaurs”).

 When asked what they think of as science museums, eight students were able to provide examples of actual science museums, including: Air and Space, Smithsonian, and Port Discovery. Only one student provided an example that was not an actual s cience museum.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 39 39 Grade 8 Science

– Student

— Item 11

Comprehension

 The students interpreted the item as asking if they have visited a museum on a school trip, or under the supervision of the school.

 When asked what was meant by the phrase “on a school trip,” all ten studen ts either found the term synonymous with the term “field trip,” or described traveling to a location with people from their school.

 When asked if they had been on a school trip to a science museum, six students described a situation where they had been o n a school trip. Of those six, three provided descriptions of the specific trip, including traveling to a science museum with classmates, the class, or the teacher on a bus.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum on a school trip?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 40 40 Grade 8 Science

– Student

— Item 12

Comprehension

 When asked what was meant by the phrase “not on a school trip,” seven of the students stated that it meant visiting the museum without others from their school, and three said it meant visiting in the presence of family or friends.

 Six students described a time when they visited a science museum that was not on a school trip, and four students said they could not remember a visit to a science museum that was not on a school trip.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

In this school year, have you visited a science museum that was not on a school trip?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 41 41 

### 2.1 .3. Grade 12 Students

Grade 12 Science

– Student

— Item 1

Comprehension

 Half of the students (5 of 10) described “hands on activities” as actively interacting with what they were learning. The other half of students (5 of 10) thought the term “hands on activities” referred to labs and

research.

 When the students were asked wh at they thought was meant by the term “circuits and energy,” half the students (5 of 10) attempted to explain the concept of how circuits and energy are related (“circuits carry electrical energy from one place to another”). The other half of students name d tangible components of circuits such as wires, batteries, and light bulbs.

 Of the three students that answered “yes” to sub

- item “b,” two recalled activities in class related to electricity or circuits. The other described a chemistry lab where react ions were measured as a function of heat applied by an electric heater. VC304986

In your science class this year, have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Yes No

a. Living things (for example, plants, animals, bacteria) A B VC304988

b. Electricity (for example, circuits and energy) A B Revised sub

- item based
on VC304989

c. Chemicals (for example, mixing or dissolving

sugar or salt in water) A B VC304991

d. Rocks or minerals (for example, identifying types) A B VC305007

e. Magnifying glass or microscope (for looking at small things) A B VC305008

f. Thermometer or barometer (for making measurements) A B VC305009

g. Simple machines (for example, pulleys and levers) A B VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 42 42 Response Process

 When probed, eight of the ten students would not change their response to sub

- item “b” if instead of “circuits and light” the example was “batteries and light.” Five students noted they believed that asking the item this way would have changed its intent. Three of these five thought that “light” did not fit with the concept of electricity.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 43 43 Grade 12 Science

– Student

— Item 2

Response Process

When asked how they would respond if they had only done activities or projects at certain times during the school year (e.g., if they had done an activity every day for three weeks, but they have not don e the activity since then) seven of ten students provided a clear indication of the response options they would choose.

 Six of these seven students selected options on the lower end of the scale such as “never or hardly ever” (2 of 6) or “once every few w eeks” (4 of 6).

 Of the five students who would answer in the lower end of the scale, three explained their choices by describing a process similar to averaging the frequency of the activity over the course of a school year.

The other two students explained that the response option “about once a week” or higher would be a poor way of describing this situation. New (Revised based on VC304986)

In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never or hardly

ever Once

every few

weeks About

once a week Two or

three

times a

week Every day

or almost every day

a. Living things (for example, plants, animals, bacteria) A B C D E VC304988

b. Electricity (for example, circuits and energy) A B C D E Revised sub

item based

on VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D E VC304991

d. Rocks or minerals (for example, identifying types) A B C D E VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D E VC305008

f. Thermometer or barometer (for making measurements) A B C D E VC305009

g. Simple machines (for example, pulleys and levers) A B C D E VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 44 44  One student chose “every day or almost every day” explaining that they would consider the period of time that they were working on that proje ct rather than the entire year.

 The remaining three students did not specify the response option they would choose and explained that their answer would depend on how often they studied that topic or how well they would remember it.

Recommendations for I mprovement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 45 45 Grade 12 Science

– Student

— Item 3

Response Process

 The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to them. Five of the students provided objective, quantifiable answers, and five students provided subjective answers.

o For the term “rarely

, ” three of the five who answered objectively indicated that the term means “once,” and two described it as “once every few weeks.” Of the five who answered subjectively, three compared the term to the other response categories (“between never and sometimes”).

o For the term “someti mes,” three of the five who answered objectively indicated that it meant more than once during the week, and two described it as more than once in a month. Of the five that answered subjectively, three compared it to the other response categories (”between rarely and often”).

o For the term “often,” three of the five who answered objectively indicated that it meant every day, and two described it as once or more in a week. Of those who answered subjectively, all used a term to describe a large amount of use such as “very frequently.”

 When asked how they would respond if they had only done activities or projects at certain times during the school year, six of the ten students provided a clear indication of which of the response options they

would choose

. New (Revised based on VC304986)

In your science class this year, how often have you done hands

- on activities or projects with any of the following? Fill in one oval on each line.

Never Rarely Sometimes Often

a. Living things (for example, plants, animals, bacteria) A B C D VC304988

b. Electricity (for example, circuits and energy) A B C D Revised sub

item based on
VC304989

c. Chemicals (for example, mixing or dissolving sugar or salt in water) A B C D VC304991

d. Rocks or minerals (for example, identifying types) A B C D VC305007

e. Magnifying glass or microscope (for looking at small things) A B C D VC305008

f. Thermometer or barometer (for making measurements) A B C D VC305009

g. Simple machines (for example, pulleys and levers) A B C D VC305012

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 46 46 o All six students provided answers at the higher end of the scale by choosing either “sometimes” (4 of 6) or “often” (2 of 6).

o Two of these students explained that they based their answers on the amount of time that they were working on that project rat her than the entire school year. Two explained that if they had spent three weeks on a project they would have learned a lot, and one stated that “rarely” would not be applicable in the situation described in the probe.

 When asked which of the first thre e item formats they preferred, six students preferred item three, two preferred item two, and two preferred item one.

o Of the six students that preferred item three, five stated they believed these response options better matched their answers compared to the other items.

o The two students that preferred item two stated that they liked the specificity of the response options.

o The two students that preferred item one stated that they liked that the item was straight forward and easy to answer.

Recommendatio ns for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 47 47 Grade 12 Science

– Student

— Item 4

Comprehension

 When asked what the phrase “science topics” meant, six students explained that it means academic subjects such as physics, chemistry, and biology. Four students answered that this phrase means anything that has to do with science.

Response Process

 Five st udents indicated they read books or magazines about science topics that are not required for

schoolwork.

 Four of the five students who indicated they read books or magazines that were not for schoolwork did not include these books or magazines when answeri ng sub

- item “b.” The reason provided for not including these books or magazines was that they were not being read for their science classes.

 When asked if they would respond differently if the item used the term “science” instead of “science topics,” four of the ten students responded that they would answer the item differently. One student explained that they would have included activities don e at home in addition to those completed in class. The explanations of the other two students suggest that the phrase “science topics” was interpreted as referring to more narrow and strictly defined science subjects, while the term ”science” would include a wider range of subjects.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or hardly

ever Once

every few

weeks About

once a week Two or

three

times a week Every

day or

almost

every day

a. Read a science textbook, in

class or at home A B C D E VC546510

b. Read a book or magazine about science topics A B C D E Revised sub

item based on
VC305295

c. Use the Internet to learn about science topics A B C D E Revised sub

item based on
VC720562

d. Watch a movie, video, or DVD about science topics A B C D E Revised sub

item based on
VC305307

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 48 48 Grade 12 Science

– Student

— Item 5

Comprehension

 When asked what the phrase “talk about” meant in sub

- item “c” and “d,” all ten students described it as “discussing” with the entire class, classmates, or teachers.

 When probed, most students (7 of 10) provided similar definitions of the term “lab write

- up” in sub

- item
“h.” VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never

or hardly

ever Once

every few

weeks About

once a week Two or

three
times a week Every

day or

almost

every day

a. Identify questions that can be addressed through science experiments A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements you took for your science project or activity A B C D E VC720635

d. Talk about the results of your science project or activity A B C D E VC720636

e. Watch your teacher do a science experiment or activity A B C D E VC720638

f. Make graphs or charts of the results from your science project or activity A B C D E VC720639

g. Write a report on your science project or activity A B C D E VC720641

h. Write a lab write

- up on your science project or activity A B C D E New

i. Write a science journal A B C D E New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 49 49 o These seven students provided explanations that were centered on the idea that it was a written piece that summarized a science lab. However, thr ee of these students expressed some confusion about the meaning of the term “lab write

- up.”

o Two students believed that the term “lab write

- up” meant answering questions.

o One student was unable provide a definition.

 When initially answering, three student s expressed some confusion about the meaning of the term “science journal” in sub

- item “i.” When asked directly what the term meant, the students had varying responses. While providing a definition, six of the students expressed that they were unsure what the term meant; of these six, two were unable to guess a definition. Two students believed the term could mean a long science report. Two other students believed a science journal may be where students keep their notes for science. Two students thought the term might mean a short daily class or homework assignment. One student thought a science journal might be where short brief constructed response items were answered. Another student thought that it might be used to provide a summary of an experiment for publication.

Response Process

 Nine students answered that they had at some point “talked about” measurements or results in sub

items “c” or “d.” Of these nine, five gave examples of discussing their results with a teacher and six students provided an e xample of talking with classmates.

 Of the eight students reporting that they had done a lab write

- up, six gave examples of specific labs where they had completed this activity by summarizing the results of the lab. One student described doing a lab write

- up as answering questions about the lab and one student could not remember any specifics, but remembered the students had to write a hypothesis.

 One student stated that he had written a science journal. He described a middle school assignment for

which he was asked to write a nightly entry about something related to science. He specifically explained providing a nightly description of the moon.

Recommendation for Improvement

 There was considerable variation in how students understood the phrase “science jo urnal.” More than

one half of the students interviewed (6 of 10) expressed that they were unsure of the meaning of the term. Only one student was able to provide an example of having written a science journal and this example was drawn from an experience in middle school. We s uggest providing a standardized definition of the term “science journal,” in order to avoid any potential misunderstandings in the future.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 50 50 Grade 12 Science

– Student

— Item 6

Reading

 Overall, students did not exhibit consistent reading prob lems in item six.

 One student skipped reading the response options and proceeded to answer the item.

Comprehension

 Overall, students did not exhibit consistent comprehension problems. A majority of students (8 of 10) stated the item was asking how often information from the library

- books, an online resource, or a

librarian

- was used for science class. Only one student mentioned that this took place this school year.

 When answering the probe, “did you think about your school’s library, a public li brary, or both,” there was an even split among students.

o Five students thought only of the school library resources when answering.

o Five students thought of both the school and public libraries and their resources.

 When asked to provide examples of libr ary resources that they have used for science class, all students were able to provide at least one example. Some of these included:

o Biographies, websites, online journals, internet databases, magazines, autobiographies, encyclopedia Britannica, and textbo oks.

 A majority of students (9 of 10) considered computers and the internet as library resources.

o While thinking about his answer, one student was not sure whether the probe was asking about the use of computers in general, or solely as a means to access t he internet.

 A majority of students (9 of 10) did not think of videos as a library resource. One of these students responded to the probe by indicating that while not mentioning videos as a library resource originally, he had used the library computers to watch online videos.

Recommendations for Improvement

 The item does not provide instructions regarding the type of library it referring to, resulting in inconsistent responses among the students. We suggest that specifying the type of library (school, publ ic or both) might improve the validity of responses captured by the item. VC315266 In this school year, how often have you used library resources for science?

A Never or hardly ever
B Once every few weeks C About once a week
D Two or three times a week
E Every day or almost every day

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 51 51 Grade 12 Science

– Student

— Item 7

Comprehension

 Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10) stated that the item asked if they agreed or disagreed with these statements about science. Three students stated that the item is asking about their opinions and feelings about science.

 When probed what the phrase “science

- related activities” meant to them, a majority of students (9 of 10) reported that it meant doing anything related to science such as watchi ng a movie or conducting an experiment.

 When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10) stated that it was something that was not required by the school (something extracurricular).

 When asked to provide an e xample of “science

- related activities that you or your classmates have done or might do that are not for schoolwork,” more than half of the students (6 of 10) were able to provide an example, such as:

o Using a telescope, working at NIH, reading Stephen Hawk ing books, nature treks, and science
fairs.

o More than half of the students (6 of 10) stated that these activities were not for schoolwork because a teacher did not instruct them to perform these activities, they would not be graded,

and the activities woul d not be discussed in class.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in

one oval on each line.

Strongly

D isagree Disagree Agree Strongly

A gree

a. I do science

- related activities that are not
for schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help
me in the future. A B C D VC305353

f. When I graduate from high school, I would like to have a job related to

science. A B C D VC720596

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 53 53 Grade 12 Science

– Student

— Item 8

Comprehension

 A majority of students (7 of 10) indicated that the item asked whether they had done the listed activities during the current school year. Three students thought it referred to all extracurricular activiti es.

 A majority of students (8 of 10) stated that they did not participate i n any of the listed activities. Of the

tw o that had participated in the listed activities, one had participated in a science club and a science competition ; t he other student had pa rticipated in a science fair. Those who reported participating in the listed activities also specified that they participated in the activities during the current school year.

 When asked what the phrase “science fair” means to them, most students (7 of 10) explained that it refers to creating a research project and presenting it to other people.

o Two students did not differentiate the terms “science fair” and “science competition. ”

o One student had never participated in a science fair but had attended one.

 O ver half of students (6 of 10) were able to provide an example of a science fair, but only two students mentioned science fairs that were taking place this year. Four students provided examples of science fairs ranging chronologically from elementary schoo l to their first year of high school. Four students were unable to provide specific examples of science fairs.

 When asked what the phrase “science club” meant to them, students had varied ideas about whether the club was a school sponsored, extracurricular , or an activity separate from the school altogether. All

students stated that a science club was a club where students meet and discuss science

- related topics.

 When asked to provide an example of a science club that they or their classmates have participa ted in or might participate in, students provided the following examples:

o Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club, biology, physics, or chemistry clubs, and an environmental club.

o Three students were unable to pro vide an example, or stated that a science club was the same as a science fair or competition.

 Most students (7 of 10) stated that a science competition includes conducting an experiment and being judged against other students, explaining that it was differ ent from a fair. Two students considered a science competition to be the same as a science fair or club. New

In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 54 54  When asked to provide an example of a science competition that they or their classmates have participated in or might participate in, some examples inc luded:

o Science bowl, Siemens competition, NIH competition, and county competitions.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 55 55 Grade 12 Science

– Student

— Item 9

Comprehension

 Overall, students did not exhibit any consistent comprehension problems.

 A majority of students (9 of 10) stated that they were being asked if they had been to a museum that focuses on science. Only two s tudents indicated that the visit had to take place during the current school

year.

 When asked what the phrase “science museum” meant to them, all students stated that it is a museum that focuses on any field of science.

Response Process

 When asked to provide examples of science museums, all students responded and listed:

o The National Air and Space Museum, the Natural History Museum, and the Cryptologic

Museum.

 When asked to provide examples of museums that are not for science, all students responded an d

listed:

o The National Gallery of Art, Spy Museum, Native American Museum and Holocaust Museum.

 When asked to provide an example of a science museum that they or their classmates have visited or might visit, all students responded and provided these exampl es:

o The Baltimore Science Museum, the Natural History Museum, the National Air and Space Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but was

unable to name it.

 Of the four students who were asked who they went to the science museum with this school year, two went with their class, one went with friends, and one went with family.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews

. New In this school year, have you visited a science museum?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 56 56 Grade 12 Science

– Student

— Item 10

Comprehension

 Overall, students did not exhibit any consistent comprehension problems.

 All students understood that the item was asking about field trips for science class.

 All students stated that “on a school trip” means a trip that is sponsored by the school with school supervision. Most students (8 of 10) were able to report an example of a school trip. They included:

o The Medical Museum in Walter Reed, the Natural History Museum, the National Aquarium, and

the Air and Space Museum.

 None of the students had visited a museum with their school during the current school year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New In this school year, have you visited a science museum on a school trip?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 57 57 Grade 12 Science

– Student

— Item 11

Comprehension

 Overall, students did not exhibit any consistent comprehension problems.

 A majority of students (9 of 10) were able to identify that the item was asking if they have visited a science museum on a trip that was not sponsored by the school.

 All students stated that “not on a school trip” meant that the visit was not sponsored by the school, and

done on the students own time.

Response Process

 More than half (6 of 10) of students gave examples of museum trips with their families. Some of the

museums mentioned were:

o The National Air and Space Museum, and the Smithsonian.

 A majority of students (9 of 10) reported that they have not visited a museum this year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New In this school year, have you visited a science museum that was not on a school trip?

A Yes
B No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 58 58 

### 2.2 Teacher Background Questionnaire

2. 2 .1. Grade 4 Teachers

Grade 4 Science

– Teacher

— Item 1

Comprehension

 The teachers interpreted this item as asking about how students were placed in smaller groups to work during class and how this related to students’ learning.

 All teachers interpreted the phrase “students’ learning preferences” in sub

- item “b” to refer to students’ learning styles; one teacher noted that she initially thought it meant “what the student wanted to learn about,” but changed her interpretation after re

- reading.

 Teachers interpreted “s tudents’ readiness needs” in sub

- item “c” to mean students’ ability levels related to reading or content comprehension. One teacher noted some confusion with the term, calling it “weird wording,” and wondered if this referred to how English learners were g rouped with other students.

Response Process

 Most teachers (3 of 5) noted during probing that they did not group their students during science lessons, or that they only occasionally had students work in pairs that were not based on any of the

criteria me ntioned in this item. However, this did not affect their comprehension of the item.

 When asked if it would be efficient for the item to ask “Do you create groups within this class for science instruction on the basis of ability,” solely with “yes” or “no” response options, all five teachers responded that they preferred the way the item was currently asked, noting that this would allow them to better explain how they taught.

 One teacher mentioned that he would have preferred to have the ability to provide an open

- ended

response. New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Groupings based on students’ interest in science/science

- related topics A B C D New

b. Groupings based on students’ learning preferences A B C D New

c. Groupings based on students’ readiness
needs A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 59 59 Recommendations for Improvement

 Although only one teacher had difficulty interpreting the terms “students’ learning preferences” and “students’ readiness needs,” we suggest including definitions of these two terms within each sub

- item, to avoid any potential misunderstanding in the future and ensure consistent interpretation of the item among teachers.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 60 60 Grade 4 Science

– Teacher

— Item 2

Comprehension

 One teacher did not understand the meaning of the phrase “inquiry skills” in sub

- item “e.”

 One teacher described the phrase “develop skills in lab techniques” in sub

- item “g” as “weird wording.”

 One teach er had some difficulty with the phrase “scientific writing skills” in sub

- item “k.” She interpreted it as referring to “lab write

- up skills” which, she noted, would apply only to older students. She therefore
found this sub

- item to be irrelevant to element ary school teachers and did not appear to comprehend the

sub

- item’s intent.

Response Process

 One teacher called the term “objectives” in the item problematic, or not relevant to him, because teachers had to follow rigid county

- approved objectives in their teaching.

 One teacher stated that sub

- item “f ” (“d evelop problem

- solving (design) skills ” ) was asking two different questions at the same time. VC970876

To what extent do you emphasize each of the following objectives in teaching science to your fourth

- grade
class? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Increase students’ interest in science A B C D VC970917

b. Teach scientific facts and principles A B C D VC970919

c. Teach scientific methods A B C D VC970920

d. Equip students with the knowledge and skills needed for studying science in upper grade levels A B C D Revised sub

item based on
VC970922

e. Develop inquiry skills A B C D VC970923

f. Develop problem

- solving (design) skills A B C D VC970925

g. Develop skills in lab techniques A B C D VC970926

h. Increase awareness of the importance of science in daily life A B C D VC970928

i. Develop systematic observation skills A B C D VC970929

j. Learn about applications of science to

environmental issues A B C D VC970930

k. Develop scientific writing skills A B C D VC970931

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 61 61  A majority of teachers (4 of 5) noted that they interpreted “upper grade levels” in sub

- item “d” to refer to

a ny grade above fourth grade, including middle school; one teacher indicated that it meant seventh grade or above.

 All teachers mentioned referring to school, county, or standards and curricula to understand what knowledge and skills students would need f or studying science in upper grade levels.

 Most teachers (3 of 5) stated that they would have answered sub

- item “d” the same way if it had stated “prepare students for further study” instead of “equip students with the knowledge and skills needed.”
One te acher felt that the phrase “knowledge and skills” was necessary for clarification, and another

noted that if the sub

- item had only said “prepare students for further study,” she would have also included “study skills that will transfer to the next grade le vels,” and would consequently have selected a higher frequency response option.

Recommendations for Improvement

 Although only one teacher had difficulty interpreting “d evelop problem

- solving (design) skills” in sub

- item “f,” we suggest removing the word “ design” or breaking this item into two parts, to avoid any potential
confusion. If separated into two parts, we suggest one sub

- item about problem solving skills and another about design skills. We also suggest keeping the current formulation of sub

- item “ d,” “Equip students with the knowledge and skills needed for studying science in upper grade levels.”

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 62 62 Grade 4 Science

– Teacher

— Item 3

Comprehension

 One teacher found sub

- item “j” and the term “audiovisual” to be “vague.”

 There were two occasions where teachers appeared somewhat confused, likely due to forgetting what the item stem was originally asking. One teacher stated that “computerized science labs for classroom

use” in sub

- item “i” was confusing because she could not tell whether it referred strictly to science or to overall teaching. Another teacher had difficulties answering sub

- item “j,” stating that she had audiovisual materials, but was not provided these resources by her school system. VC970932

How much of the following instructional materials and other resources does your school system provide you with to teach science to your fourth

- grade class? Fill in one oval on each line.

None Little Some A L ot

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for

classroom use A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC970964

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 63 63 Response Process

 When describing what the item was asking, most teachers (3 of 5) said it asked what the “school system” provided. One teacher specifi cally noted that this was different from the “school.” Another teacher, however, said the item asked what the school provided. One teacher did not indicate whether the question referred to the school system or the school, and merely stated that the questio n asked “what kind of resources are available for teaching.”

 One teacher found it difficult to choose between response options “B” ( “ little ” ) and “C” ( “ some ” ) for sub

items “c” and “d” (supplies or equipment for science demonstrations, or labs). He indica ted that these

sub

- items were “hard” because the county (the school system) will no longer pay for “consumables,” meaning that they do not have all the material they need and he obtains some materials on his own. This teacher appeared to understand the sub

- item, but had difficulty recalling the necessary information to confidently provide an accurate response.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 64 64 Grade 4 Science

– Teacher

— Item 4

Comprehension

 Two teachers indicated that the different wording of this item, as compared to the previous item, did not change how they answered this item. However, two other teachers stated that their answers for this item differed; both noted that they saw a distinction between “school system” (used in previous item) and “school” (used in the current item), and the role of each entity in providing them with materials. Another teacher said he had answered a couple of su b

- items differently than before, but could not explain why.

Response Process

 All teachers stated that they found the options “not at all,” “small extent,” “moderate extent,” and “large extent” appropriate and sufficient response options.

 When asked what the response options “small extent,” “moderate extent,” and “large extent” meant, all teachers indicated that “small extent” meant there were not enough resources, “moderate extent” meant New (Revised based on VC970932)

To what extent does your school provide the following to you? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for

classroom use A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC970964

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 65 65 there was a sufficient amount for class work with students, and “lar ge extent” meant there were more than enough. Although teachers used different ways of quantifying the amount of resources that they considered sufficient or insufficient for their needs, they shared a similar interpretation of the response options.

 When asked which version of the response options they preferred, between the third and fourth items, a majority of teachers (4 of 5) did not seem to understand this probe and selected one particular response option (such as “small extent”) as their reply. The s ingle teacher who understood the probe stated she would choose the item with the response options “small extent, etc.” because “it sounds more professional.”

 When asked if they could suggest other response options that would provide more specific informati on about resources, one teacher suggested adding the response option “Not Applicable.” Another teacher had several suggestions: ask separately about science magazines and science textbooks, ask about a separate lab classroom devoted to science, and ask abo ut safety equipment.

Recommendations for Improvement

 The previous item (item three) includes the term “school system,” while item four includes the term
“school.” Because of this difference, two teachers provided different answers for item three and item four. They inferred that item three was asking about resources provided by the county or school district,
while item four asked about resources from the school. Their responses suggest that they tried to differentiate between resources provided by the scho ol system (county or school district) and those provided by the school itself. Teachers’ verbal reports and answers to items three and four show that the formulation of item three, with the phrase “school system,” may cause confusion among teachers. We

sug gest that the final item version provide a more clear and unequivocal formulation of the question.

 Interviewer notes suggest teachers had a consistent interpretation of the response options used in item four, and stated that they are appropriate and suffic ient for describing the extent of resources provided by the school.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 66 66 Grade 4 Science

– Teacher

— Item 5

Comprehension

 One teacher said that the meaning of the word “activities” in sub

- item “d” was not clear.

Response Process

 All teachers understood that the item was asking about different instructional methods they use with their

students.

 All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to whether students could learn at a faster or slower pace based on how quickly they understood the material or

lesson.

 All teachers stated that they would have answered sub

- item “c” the same way if it had instead read “pace my teaching differently for some students.” Only one teacher said she would prefer the question use this alternate wording, because it “probably makes more sense.”

Recomm endations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC767811

When you teach science to your fourth

- grade class, do you do any of the following? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Use a different set of methods in teaching some students A B C D VC767814

b. Supplement the regular course curriculum with additional material for some students A B C D VC767820

c. Vary the pace of instruction for some students A B C D Revised sub

- tem
based on
VC767821

d. Have some students engage in different classroom activities A B C D VC767823

e. Set different achievement standards for
some students A B C D VC767824

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 67 67 Grade 4 Science

– Teacher

— Item 6

Reading

 Overall, teachers did not exhibit consistent reading problems or any difficulty understanding the words in

the item.

Comprehension

 A majority of teachers (4 of 5) said they made a distinction between “science fair” and “science competition.” Three seemed to feel that science fairs were more informal and involved a larger number of students exhibiting their work. One teacher said that he had not previously thought there was a distinction, but now did; however, he did not provide further explanation. One teacher stated that, after thinking about it, she was confused by what “science competition” meant, and how it differed from a
“science fair.” She suggested the questionnaire provide examples of each.

Response P rocess

 Two teachers specifically mentioned that the item asked what outside opportunities teachers were providing, while others indicated that it asked what the teachers or the school provided. However, all teachers answered in regard to activities or oppo rtunities the school organized, rather than what the teacher personally provided.

 All teachers felt that the response options provided were appropriate, though one teacher commented that it would be better to ask teachers to quantify the frequency of eve nts per school year. When asked
the follow

- up probe of whether it would better to use frequencies such as certain times a week, a semester, or a school year, a majority of teachers (4 of 5) said it would be better.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

To what extent do you provide students with the following learning experiences? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 68 68 

### 2.2.2. Grade 8 Teachers

Grade 8 Science

– Teacher

— Item 1

Comprehension

 Three of the teachers interpreted this item as asking about how they group students in the classroom, and how these groupings are related to students’ learning. Two interpreted this item as asking how they identify student groupings based on their abilities, learning preferences and interests. However, one teacher also stated that it is not clear if the item is asking about interests, learning preferences, or
academic factors.

 All teachers interpreted the t erm “student groupings” to mean separating the students into work groups to collaborate on assignments.

 The teachers defined the phrase “students’ learning preferences” in two different ways

. Three teachers explained that the phrase refers to learning st yle typology, such as auditory, visual or kinetic learners, while the remaining two teachers interpreted the phrase as referring to more generally understood interests and preferences.

 Three teachers expressed confusion regarding the phrase “students’ re adiness needs” in sub

- item “c

. ”
One noted that it could mean "two different things: the desire and willingness to learn," or whether students "have a background without holes in things like math and reading abilities, and their conceptual knowledge is on p ar.” Two stated that the phrase meant students’ academic preparedness to understand the curriculum content and complete tasks. Two other teachers equated it with either content accessibility or students’ abilities.

Response Process

 A majority of teachers (4 of 5) stated that they use all three grouping methods in their teaching day, but that they also take into consideration student behaviors to avoid a volatile study environment. Two stated that this item could be restructured as a yes or no question, ask ing only whether they use groupings by student ability in their science class instruction. Three indicated that because the item is not, in one
teacher’s words, “black and white,” it was in fact necessary to ask about various grouping methods.

 All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” are appropriate for describing the extent of instructional grouping strategies. New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Groupings based on students’ interest in science/science

- related topics A B C D New

b. Groupings based on students’ learning preferences A B C D New

c. Groupings based on students’ readiness
needs A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 69 69 Recommendations for Improvement

 The teachers offered two different in terpretations of the phrases “students’ readiness needs” and “students’ learning preferences.” Therefore, we suggest including standardized definitions of these phrases within sub

- item “b” and “c,” to ensure consistent interpretation of the item among resp ondents and avoid any potential confusion in the future.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 70 70 Grade 8 Science

– Teacher

— Item 2

Comprehension

 One teacher had difficulty interpreting a number of sub

- items, asking how sub

- item “e” differed from asking about the teaching of scientific methods in sub

- item “c.” For sub

- ite m “h” ( “ increase awareness of the importance of science in daily life ” ) she noted that it was very similar to what was asked in sub

- item “a” ( “ increase students’ interest in science ” ).

 A majority of teachers (4 of 5) stated that the item asks how much they focus on different areas and topics when teaching science.

 When asked what the phrase “upper grade levels” means to them, a majority of teachers (4 of 5) stated that it meant high school level material. One teacher stated that it meant honors biology, che mistry, and physics, as well as AP high school and college material.

 A majority of teachers (4 of 5) stated that the phrase “studying science” meant taking physics, chemistry, and biology in high school. VC976013

To what extent do you emphasize each of the following objectives in teaching science to your eighth

- grade
class? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Increase students’ interest in science A B C D VC976015

b. Teach scientific facts and principles A B C D VC976017

c. Teach scientific methods A B C D VC976018

d. Equip students with the knowledge and

skills needed for studying science in upper grade levels A B C D Revised sub

item based on
VC976019

e. Develop inquiry skills A B C D VC976020

f. Develop problem

- solving (design) skills A B C D VC976021

g. Develop skills in lab techniques A B C D VC976022

h. Increase awareness of the importance of science in daily life A B C D VC976023

i. Develop systematic observation skills A B C D VC97625

j. Learn about applications of science to

environmental issues A B C D VC970926

k. Develop scientific writing skills A B C D VC970927

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 71 71  When asked what the phrase “knowledge and skills nee ded for studying science in upper grade levels” meant, all teachers explained that it meant the principles and techniques students need to be successful in high school science.

Response Process

 A majority of teachers (4 of 5) said they determine what knowl edge and skills will be needed beyond 8th grade by following state curriculum standards. One stated that he uses a tie

- in with current events, such as conservation and streams and the Chesapeake Bay.

 Most teachers (3 of 5) reported that if the item had bee n phrased “prepare students for further study in science,” their answer would change slightly but remain similar. They also stated that “prepare students for further study in science” means to prepare students for scientific studies in the future.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 72 72 Grade 8 Science

– Teacher

— Item 3

Comprehension

 The teachers interviewed displayed inconsistent interpretations of this item. Two interpreted the item as asking about the resources they have available to carry out all the tasks required for their students to learn throughout the school year. Three teachers interpreted the item as asking what resources were provided by their school system, with two teachers specifically mentioning the county or school district.

 One teacher was unsure about the meani ng of sub

- item “j” (Audiovisual materials), and whether it meant particular software or the actual technology.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC976028

How much of the following instructional materials and other resources does your school system provide you with to teach science to your eighth

- grade class? Fill in one oval on each line.

None Little Some A L ot

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for

classroom use A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC976045

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 73 73 Grade 8 Science

– Teacher

— Item 4

Comprehension

 Three teachers stated that item four is asking the same question as item three. However, one of these respondents paraphrased the item using the term “ school district.”

 Two teachers were not entirely clear if the item was asking about their school or their school system. Both stated that item three and item four were making the distinction between materials provided by the school district and those provided by the school itself. One of these respondent s stated that item three asks about resources provided by the county, while item four asks about in

- house resources and not those from the county science office. Another noted that the term "school" can have multiple interpretations in the context of this item. New (Revised based on VC976028)

To what extent does your school provide the following to you? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for

classroom use A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments (e.g., telescopes, microscopes, thermometers, or weighing scales) A B C D VC976045

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 74 74 Response Process

 Three teachers reported that the way the item four was phrased compared to the way item three was phrased would not make a difference in how they would answer any of the sub

- items listed.

 A majority of teachers (4 of 5) stated t hat the response options “not at all,” “small extent,” “moderate extent,” and “large extent” are appropriate and sufficient for describing the extent to which resources are provided by the school.

 When asked what the phrases “small extent,” “moderate exten t,” and “large extent” meant, a majority of teachers (4 of 5) said that “small extent” meant to have some resources but not enough, “moderate extent” meant an adequate amount of resources, and “large extent” meant more than enough resources.

 When asked whi ch version of the response options they preferred between the third and fourth items, all teachers misinterpreted the probe and selected one particular response option (such as “small extent”) as their reply. A majority of teachers (4 out of 5) stated that they would answer moderate to large extent.

 Two teachers gave suggestions for other answer choices that would provide more specific information about the resources provided by the school. One suggested that the item should be asked in terms of

usage. An other suggested the scale should be none, some, etc.

Recommendations for Improvement

 The previous item (item three) includes the term “school system

, ” while this item includes the term
“school.” Because of this difference two teachers considered item three and item four as two entirely different questions. They deduced that item three asks about the resources provided by the county,
which is different from item four which asks about available “in

- house” school resources. While answering, they tried to diffe rentiate between resources provided by the school system (county or school district), and those provided by the school itself. Respondents’ verbal reports and answers to item four and item three show that the formulation of the item three, which utilizes t he phrase “school system,” may cause confusion among respondents. We suggest that item four provide s a more clear and unequivocal formulation of the question.

 Interviewer notes provided no evidence suggesting a preferential type of response scale between i tems
three and four for teachers. Three of the five teachers indicated they would answer item four the same as item three, and four of the five stated that the response scales are appropriate and sufficient for describing the extent of resources provided b y the school.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 75 75 Grade 8 Science

– Teacher

— Item 5

Comprehension

 A majority of teachers (4 of 5) stated that item five is asking about different instructional methods, and how they plan and change lesson presentation methods to accommodate students with varying levels o f understanding.

 A majority of teachers (4 of 5) also stated that the phrase “vary the pace of instruction for some students” meant providing students with different amounts of time to complete tasks, based on their different performance levels. One indica ted that the phrase meant more time with computer

- based activities.

 One teacher did not like the wording “set different achievement standards for some students,” describing it as a hard question because its wording would suggest that she holds some student s to lower standards; she felt this was untrue, because some students would still master the content, but in a different way than others.

 Three teachers stated that they would have answered sub

- item “c” (“vary the pace of instruction for some students”) th e same way if it had instead said “pace my teaching differently for some students.” Two stated that their answers would be different because they interpreted the first version to be more

student

- specific, instead of referring to their overall teaching appr oach.

 All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some students.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. VC976085

When you teach science to your eighth

- grade class, do you do any of the following? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Use a different set of methods in teaching some students A B C D VC976086

b. Supplement the regular course curriculum with additional material for some students A B C D VC976088

c. Vary the pace of instruction for some students A B C D Revised sub

item based on
VC976091

d. Have some students engage in different classroom activities A B C D VC976092

e. Set different achievement standards for
some students A B C D VC976094

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 76 76 Grade 8 Science

– Teacher

— Item 6

Comprehension

 A majority of teachers (4 of 5) stated that item six asks about science

- related activities outside of the

classroom.

 When asked if they make a distinction between “science fair” and “science competition,” three teachers described the judging component as the difference between a science fair and competition. Two

teachers did not find the two to be different.

 When asked of different activities offered by the school, teachers provided the following examples:

o Robotics club, STEM club, and Recycling club

o Visits to the Koshland Science Museum, USA Science and Engineering Festival, and the Smithsonian Air and Space Museum.

 When asked what types of science museums they take student s to, teachers provided the following examples:

o Children’s Museum, the National Aquarium, and the Smithsonian Air and Space Museum.

 Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” wer e sufficient to describe the extent of the learning experience provided to the students. One thought the choices were sufficient but felt that the term “science museum” was too limiting and suggested to add science field trips such as taking kids to Chesap eake Bay and testing compounds.

 In regards to whether frequencies, such as certain times a week/a semester/a school year, would be better than the response options provided for this item, opinions were divided. While two teachers rejected the idea that fre quencies would be more relevant response options, two noted that they would be preferred. One respondent specifically suggested the following options: per quarter, per semester, or

per year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews. New

To what extent do you provide students with the following learning experiences? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 77 77 

### 2 .3 Administrator Background Questionnaire

2.3 .1. Grade 4 Administrators

Grade 4 Science

– School

— Item 1

Comprehension

 Administrators were not unified in their understanding of what the role and responsibilities would be for a “science specialist or coach.” One understood “specialist” and “coach” to have different meanings, explaining that a specialist works with students to teach them concepts directly, while a coach works with teachers to improve teaching practices and planning.

Response Process

 Although a majority of administrators (4 of 5) indicated that they did not ha ve a part

- time or full

- time science specialist or coach in their school, they emphasized that the responsibilities they associated with the role were taken on by classroom teachers and/or other types of specialists (e.g. instructional support teacher).

 T he one administrator who indicated that she did have a STEM (Science, Technology, Engineering, and Mathematics) teacher on staff indicated that it was a full

- time staff member, and explained that the STEM teacher was responsible for helping teachers plan t heir lessons and demonstrations.

 Of the administrators who indicated that they did not have a science specialist or coach on staff, a majority (3 of 4) indicated that the responsibilities were divided among classroom teachers at their schools. One administrator indicated that she does have an “instructional support teacher” in the fields of literacy and science that she would consider as fulfill ing the role of a science specialist or coach.

Recommendations for Improvement

 As interpretations of the term “science specialist or coach” seemed to vary among administrators, it may be helpful to include a short definition. It was also important to the administrators to qualify a “no” response to this item by explaining that although they did not have a designated full or part

- time science specialist or coach, they did have individuals who provided the services and responsibilities they

associate with th is role. In order to capture this information, we recommend expanding the range of the response options to include additional answer choices, such as “Yes, available on an as needed basis,” and/or “No, but other individuals within the school provide the se rvices of a science specialist or coach

. ” New (Revised based on VE013369) Is there a science specialist or coach available (full

- or part

- time) to fourth

- grade teachers at your
school?

A Yes, available full

- time at my school.

B Yes, available part

- time at my school.
C No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 78 78 Grade 4 Science

– School

— Item 2

Comprehension

 All administrators interpreted sub

- item “d” (“Translate science standards and current research into classroom practice”) as meaning the ability to : understand and internalize science standards, use that ability to create learning plans and objectives easily understood by students, and use appropriate vocabulary and teaching methods.

 When asked what sub

- item “e” (“Provide science enrichment to some student groups”) meant to them,

all administrators i ndicated that it meant some student groups would receive more attention or different activities than the rest of the class. A majority of administrators (4 of 5) indicated that the enrichment activities would apply to students who wanted to go beyond what was being taught in the classroom. One understood “some student groups” as students who may be struggling with the curriculum (such as those enrolled in ESOL or remedial classes), and who need additional activities to help them catch up to their peers.

Response Process

 All administrators felt that the sub

- items covered the majority of the responsibilities they would consider part of the role of a science specialist or coach. Additional responsibilities mentioned by administrators included the incorporati on of literacy skills into science instruction, and motivating trends within the

school to assess how well students understood the core science standards.

 As mentioned earlier, only one out of five administrators reported having a full

- or part

- time scien ce specialist or coach. This resulted in a majority of administrators (4 of 5) feeling unsure of how to provide appropriate responses to the sub

- items. However, when probed about the response options, all administrators felt that they were appropriate. New (Revised stem based on VE013372)

To what extent are any of the following a responsibility of the science specialist or coach available to fourth

grade teachers at your school? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about science content or the teaching of science A B C D VE013373

b. Conduct professional development for groups of teachers about science content or the teaching of science A B C D VE013382

c. Provide science instruction to students on various topics A B C D VE013383

d. Translate science standards and current research into classroom practice A B C D Revised sub

- item

based on
VE013384

e. Provide science enrichment to some student groups A B C D VE013385

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 79 79 o Mos t administrators (3 of 5) who did not have a science specialist or coach on staff answered

the sub

- items as if they referred to the staff that performed each responsibility (teachers, administrators, etc.).

o One administrator mentioned that she would have r eplied “not applicable” for each of the sub

items if the response option had been available, since she did not have a science specialist or
coach on staff.

Recommendations for Improvement

 Depending on the item’s intent, there are two ways to address admini strators’ response process challenges. First, if the intent of the item is to measure to what extent these responsibilities are performed at the school, we recommend modifying item so that it does not specifically refer to a “science specialist or coach.” Second, if the intent of the item is to ask specifically about the activities of a “science specialist or coach,” we recommend an additional “not applicable” response category for those

administrators who do not have one on staff.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 80 80 Grade 4 Science

– School

— Item 3

Comprehension

 All administrators interpreted this item as asking about science

- related opportunities the school provides beyond the traditional classroom instruction.

 Although most administrators (3 of 5) agreed that there is a distinction between a “science fair” and a “science competition,” definitions of each term differed.

o Two characterized a “science fair” as being open to all students in the school, and indicated that students pursued individual interests.

o Two ch aracterized a “science competition” as being more structured, with students receiving a certain question or task and competing against one another.

o One did not differentiate between the two terms, but guessed that a “science competition” involved asking st udents questions about science. Another administrator did not understand the term “science competition,” but guessed that it would be similar to a spelling or geography bee,

and that students would be asked questions related to science.

 When asked for exam ples of how the school provided opportunities for students, administrators mentioned included school and county

- based science fairs, gardening and recycling clubs, and other

environmental clubs.

 When asked specifically about trips to science museums, admin istrators mentioned the National Aquarium, the Smithsonian Natural History Museum, the Baltimore Science Center, NASA, and the Smithsonian Air and Space Museum.

Response Process

 A majority of administrators (4 of 5) preferred the response options of “large extent, moderate extent, and small extent,” versus frequencies such as certain times a week/semester/school years.

 One administrator suggested merging the two categories of res ponse options, changing the question to ask “To what extent does your school provide students with the following opportunities per school year ?” She also suggested adding ranges to qualify what was meant by a small, moderate and large extent New

To what extent does your school provide students with the following opportunities? Fill in one oval on

each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 81 81 (e.g. “small”, 0

- 1 times per school year, “moderate,” 2

- 3 times per school year, “large,” 4 or more times per school year).

Recommendations for Improvement

 As the definitions for “science fair” and “science competition” differed among administrators, it is

recommended t hat descriptions or definitions be provided for these terms to ensure consistent understanding of this term across administrators.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 82 82 

### 2.3 .2. Grade 8 Administrators

Grade 8 Science

– School

— Item 1

Response process

 All administrators selected option “C,” because their schools do not have a science specialist or coach available to eighth

- grade teachers. Three administrators noted that a teacher or administrator woul d perform some responsibilities of a science coach at certain times; however, they did not have a specialist or coach dedicated solely to science.

 One administrator displayed a strong discomfort stemming from the fact that his school does not have a

scienc e specialist or coach position, and defended and explained the school’s situation.

 When asked to describe the position or role of a “science specialist or coach,” administrators identified responsibilities such as providing support and help with developmen t of curricula, lesson plans, and meeting standards for science instruction.

 When asked whether a different term is used in their school to describe individuals with the responsibilities of a science specialist, most administrators (3 of 5) answered that a similar type of position existed in other subject areas (e.g., math, social studies), and was described as either an “instructional lead teacher,” “instructional coach,” or “mentor teacher.” Two administrators noted that the responsibilities associated wi th a “science specialist” were performed by the principal or the assistant principal.

Recommendations for Improvement

 Although all administrators responded that their schools do not have a science specialist or coach
available full

- or part

- time to eighth

- grade teachers, further probing revealed that the responsibilities of this role are fulfilled by teachers or administrators who do not hold an official position as a science specialist or coach. In order to improve the collection of this information, we su ggest expanding the range of response options to include “Yes, available on an as

- needed basis.” New (Revised based on VE013964) Is there a science specialist or coach available (full

- or part

- time) to eighth

- grade teachers at your
school?

A Yes, available full

- time at my school.

B Yes, available part

- time at my school.
C No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 83 83 Grade 8 Science

– School

— Item 2

Comprehension

 Two administrators exhibited some confusion or difficulty understanding this item. One stated that sub

item “a” is “too wordy” and needed to reread it in order to understand it. The use of the word “translate”

in sub

- item “d” was also confusing to this administrator. The other administrator was unsure about the intent of the item, and tried to clarify whether it was referring to what his school currently offers or what

he envision s a science specialist would do if there were such a role at his school.

Response Process

 Given his school’s lack of an available science specialist for eighth

- grade teachers, one administrator

answered the item based on the extent to which the services described in each sub

- item are available to teachers. For the same reason, another administrator based his answers on the role of the science department chair; this individual serves as an instructional coach but does not fit the administrator’s

definition of a science coach.

 When probed about the phrase “translate science standards and current research into classroom practice,” administrators stated that it means explaining or understanding “what the standard is asking us [teachers, students] to do,” expl aining what the standards “mean for day to day instruction,” and how “to help map these standards into daily objectives,” or measureable objectives.

 A majority of administrators (4 of 5) had consistent and similar interpretations of “provide science

enrich ment to some student groups,” as additional activities that go beyond the standards or requirements set in curriculum. These activities may be provided to a small group of students needing additional practice, or to a group that has already mastered the cu rricula standards and requires more New (Revised stem based on VE013966)

To what extent are any of the following a responsibility of the science specialist or coach available to eighth

grade teachers at your school? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about science content or the teaching of science A B C D VE013967

b. Conduct professional development for groups of teachers about science content or the teaching of science A B C D VE013968

c. Provide science instruction to students on various topics A B C D VE013969

d. Translate science standards and current research into classroom practice A B C D Revised sub

item based on
VE013970

e. Provide science enrichment to some student groups A B C D VE013971

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 84 84 differentiated materials. As examples, administrators provided such activities as “dissection, fun experiments,” and scientific investigations.

 Most (3 of 5) administrators agreed that the activities presented in the sub

- items cover all main responsibilities of a science specialist or coach. Two stated that sub

- items “a” and “c” are not very relevant to the situation in their schools, and are specific to the person with training in science.

 Two of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” are sufficient for describing the extent of a science specialist’s or coach’s responsibilities in their school. One administrator stated a preference for a numeric scale, and another suggested a three

- point scale that would combine small and moderate extent response options, because he had a problem deciding which option is more relevant to his school. The last administrator misinterpreted the probe.

R ecommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 85 85 Grade 8 Science

– School

— Item 3

Comprehension

 Overall, administrators did not exhibit any consistent problems with this item.

 The administrators interpreted the item as asking if the school provides opportunities to participate in

science related activities beyond the regular classroom and outside of school hours.

Response P rocess

 All administrators made a clear distinction between “science fair” and “science competition,” and defined the differences between these two concepts as follows:

o The scien ce fair has a larger scale and extent, and gives all students the opportunity to participate.

o The competition is limited to the science clubs and some specific topics or areas of specialization.

 Administrators reported the following examples of opportunit ies provided by the school:

o Trips: outdoor education at Camp Schmidt with hydroponic gardens, Howard Owens Science Center for Astronomy.

o Science competitions: robotics tournaments, Science Olympia competition.

o Science clubs: robotic s club, science club, en vironmental club.

 Administrators reported the following examples of field trips to science museums organized by the

school:

o The Owens Science Center, Bladensburg Waterfront Park, Maryland Science Museum in Baltimore, Smithsonian museums in Washington DC (e .g., Air and Space Museum, Museum of Natural History).

 Four of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large extent,” are sufficient for describing the extent of the opportunities t heir school provides; the last administrator misinterpreted the probe. New

To what extent does your school provide students with the following opportunities? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 86 86  Administrators had diverse opinions about the use of frequencies, such as certain times a week/ semester/school year, and whether these would be better response options for this item. W hile one administrator noted that it would be appropriate for the science fair and clubs, another stated it would not be applicable for science fairs, because they happen only once a year. Two administrators indicated a preference for frequency

- based respo nse options, accompanied by a numeric scale.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 87 87 

### 2.3 .3. Grade 12 Administrators

Grade 12 Science

– School

— Item 1

Comprehension

 Overall, the administrators did not exhibit any consistent problems with this item. One administrator indicated that he did not understand what the term “science specialist or coach” was meant to convey

. H owever, when asked to describe the role of a “science specialist or coach,” he indicated that it was a “per son that helps with the content: an expert with a toolbox of resources to develop lesson plans, field trips, etc., and who observes classes to coach teachers on better practices.”

 All administrators gave similar definitions of a “science specialist or coach,” indicating that an individual in this role would be a subject matter expert who helps to coach and provide instructional support to

teac hers.

Response Process

 Two administrators answered “no,” explaining that although they did have an individual fulfilling the role of a “science specialist or coach,” they did not consider this person as being employed either full

- or part

time. One of thes e administrators indicated that the “science specialist or coach” was available on an as needed basis, and the other explained that the “science specialist or coach” was not dedicated solely to

12 th grade.

 Two administrators selected the response option, “ yes, available part

- time.” One explained that the individual taught class in the morning and was available as a “science specialist or coach” in the afternoon. The other administrator explained that the role of the “science specialist or coach” was fulfill ed by two individuals

– one who works with more than one school, and another with a full teaching load, who also took on the responsibilities of a “science specialist or coach.”

 One administrator chose not to select any of the response options, indicating that none of these options fit his school’s situation. He further explained that although a resource teacher is available, this teacher is responsible for all high schools in the county, and that the 12 th grade teachers in his school rarely ask for support from this individual.

 When asked if there is a different term used in their school for describing individuals with the responsibilities of a “science specialist or coach,” administrators noted a “resource teacher,” “science department chairperson,” and “s cience coordinator.” New (Revised based on VE015623) Is there a science specialist or coach available (full

- or part

- time) to twelfth

- grade teachers at your school?

A Yes, available full

- time at my school.

B Yes, available part

- time at my school.
C No

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 88 88 Recommendations for Improvement

 Although administrators were fairly unified in their definitions of the roles and responsibilities of a “science specialist or coach,” it may be helpful to provide a specific definition or additional descriptive

terms.

 Overall, administrators felt that the response options did not fit their internally generated answer. The addition of a response option such as “yes, on an as

- needed basis” may help to alleviate this difficulty.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 89 89 Grade 12 Science

– Schoo l

— Item 2

Comprehension

 Overall, administrators were able to comprehend the item and sub

- items. One administrator expressed

confusion over the term “science enrichment” in sub

- item “e

. ” H owever, when probed, his definition

matched those of the other administrators.

 All administrators interpreted “translate science standards and current research into classroom practice” as meaning the ability to understand and internalize science standards, and using that knowledge to

create learnin g plans and objectives easily understood by students, and include appropriate vocabulary and teaching methods. When asked for terms that might be more meaningful for school administrators, one administrator suggested “curriculum review or standards based i nstructional development.”

 When asked what “provide science enrichment to some student groups” meant to them, all administrators indicated that it meant that some student groups would receive more attention or different
activities than the rest of the cla ss, particularly opportunities that may go beyond the classroom. Most administrators (3 of 5) indicated that the enrichment activities would apply to students who excelled in the subject matter and wanted to expand their knowledge base, while two administr ators also explained that “some student groups” could also be interpreted as students who may be struggling with the curriculum (such as those enrolled in remedial classes) and are in need of additional activities to help them catch up to their peers.

R esponse Process

 All administrators felt that the sub

- items covered the majority of responsibilities they would consider as parts of the role of science specialist or coach. Additional responsibilities mentioned by administrators included facilitating train ing regarding new technology, determining the best resource materials (textbooks, secondary documents, etc.) to use, and facilitating hands

- on experiences and field trips. New (Revised stem based on VE015624)

To what extent are any of the following a responsibility of the science specialist or coach available to
twelfth

- grade teachers at your school? Fill in one oval on each line.

Not at

all Small

extent Moderate

extent Large

extent

a. Provide technical assistance/support to

individual teachers about science content or the teaching of science A B C D VE015625

b. Conduct professional development for groups of teachers about science content or the teaching of science A B C D VE015626

c. Provide science instruction to students on various topics A B C D VE015627

d. Translate science standards and current research into classroom practice A B C D Revised

sub

- items

based on
VE015628

e. Provide science enrichment to some student groups A B C D VE015629

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 90 90  The two administrators who reported that they did not have a “science specialist or coach” seemed unsure about appropriate responses to the sub

- items. However, when probed about the response options (not at all, small extent, etc.), all administrators felt that they were appropriate.

o One administrator mentioned that he would have replied “not applicable” for each of the sub

items if that response option had been available, since he did not have a science specialist or
coach on staff.

o One administrator suggested using numbers (1

- 4) to represent the response options, rather than letters (“A”

- “D”).

Recommendations for Improvement

 Depending on the item’s intent, two alternatives could address administrators’ response process concerns. If the intent of the item is to measure to what extent these responsibilities are performed at the school, we r ecommend changing item so that it does not specifically identify a “science specialist or coach.” If the intent of the item is to ask specifically about the role of the science specialist or coach, we recommend a “not applicable” response option for those administrators who do not have one on staff.

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 91 91 Grade 12 Science

– School

— Item 3

Comprehension

 All administrators interpreted this item as asking about science

- related opportunities that the school provides beyond traditional classroom instruction.

 Two administrators indicated that the term “science museums” is unclear, since it could incorporate both traditional museums, such as a natural history museum, and locations like aquariums or other similar sites. Both of these administrators suggested that the sub

- item should be clarified.

 Most administrato rs (3 of 5) agreed that there is a distinction between a “science fair” and a “science competition.”

o Two administrators characterized a “science fair” as an on

- site exhibition where students demonstrate a science project or experiment to others.

o All admini strators characterized a “science competition” as where students compete against each other; two of the five described it as more structured, where students may be given a certain question or task.

o One administrator did not differentiate between the two te rms, explaining that in her school the “science fair” is a competition where students can win first place, second place, and honorable

mentions.

Response Process

 When asked for examples of how the school provided opportunities for students, administrators mentioned robotics competitions, natural history museums, aquariums, environmental science clubs, annual science fairs, and electronic science fairs.

 When asked specifically about trips to science museums, administrators mentioned the National Aquarium, th e Chesapeake Bay Foundation, the National Electronics Museum, the Smithsonian Natural History Museum, and the National Institutes of Health.

 Most administrators (3 of 5) expressed a preference for having response options with frequencies, such

as certain t imes a week/a semester/a school year, explaining that it would be clearer and easier to New

To what extent does your school provide students with the following opportunities? Fill in one oval on each

line.

Not at

all Small

extent Moderate

extent Large

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New

## ETS Cog Lab Report: BQ Science 201 3

Prepared by EurekaFacts, LLC 92 92 quantify their answers. One administrator indicated that the best option would be to base answers on a school year, since it would be harder to quantify in weeks.

 One a dministrator suggested adding ranges to qualify what was meant by a small, moderate and large

extent. He suggested small extent would mean one to three times per year, moderate extent would mean four to seven times per year, and large extent would be eight or more.

Recommendations for Improvement

 Verbal reports and answers to probing questions show that the sub

- items do not provide administrators with a chance to report all types of trips used for science classes. It may be helpful to specify how narrowly o r widely the term “science museum” should be interpreted within the context of this item, and

whether to include non

- traditional museums (e.g., aquariums and arboretums). Alternately, additional

sub

- items could be provided to capture other types of trips t hat can be used for science instruction. Lastly, most administrators seemed to have difficulty using the response options of “small, moderate and large extent,” explaining that the terms were subjective. Overall, administrators preferred frequencies in

the response options instead.

## This page was intentionally left blank

.

## This page was intentionally left blank

.2015 Science Survey Questionnaire Cognitive Interview Study

, January 2013

183 

# Appendices184 

### Contents

Interview Protocol: Generic Probes and Think

- Aloud Hints for the Interviewer

............................... 185

Science Student Questionnaire Interview Protocal: Items and Probes

................................ .............. 187

Recruitment and Operational Forms, Letters, and Scripts

................................ ................................ . 188

Parent or Legal Guardian Consent Form

................................ ................................ ..................... 188

Student (age 18 or older) Consent Form

................................ ................................ ..................... 189

Teacher and School Administrator Consent Form

................................ ................................ ...... 190

Phone Script Recruiting Parents or Legal Guardians of Student Participants

............................ 191

E

- mail Recruiting Parents or Legal Guardians of Student Participants

................................ ....... 194

Letter Recruit ing Parents or Legal Guardians of Student Participants

................................ ....... 195

Phone Script Recruiting Teachers and School Administrators

................................ .................... 196

E

- mail Recruiting Teachers and School Administrators

................................ .............................. 199

Letter Recruiting Teachers and School Administrators

................................ ............................... 200

Email/Letter Confirmation to Parent or Legal Guardia n of Student (under age 18)
Participants

................................ ................................ ................................ ................................ .. 201

Email/Letter Confirmation to Student (age 18 or older) Participants

................................ ........ 202

Thank You Letter to Parent or Legal Guardian of Student (under age 18) Participants

............. 203

Thank You Letter to Student (age 18 or older) Participants

................................ ....................... 204

Thank You Letter to Teachers and School Administrators

................................ .......................... 205185 

## Interview Protocol **: Generic Probes and Think** 

**- Aloud Hints for** 
**the Interviewer**

If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging the participant to read the que stion aloud and say his/her thoughts, or to help him/her elaborate the responses.

**•** Please read the question aloud and tell me what you are thinking.

**•** You mentioned __________, tell me more about that.

**•** You mentioned __________, what was that like for you?

We are interested in capturing all the participant’s mental processes while answering these questions. Your goal is to have the participant speak aloud all his or her thoughts while answering the questions. By asking follow

- up probes after each item, you w ill ensure that the data collected are as complete as possible. Provide positive feedback by telling the participant when he or she has given helpful information. The participant may provide more detailed responses if given positive feedback such as:

**•** Thi s information is very helpful.

**•** The example you just gave is very useful.

**•** Thank you for providing such a thorough response.

If a participant is continually providing short responses or not answering, use “continuers” to encourage the participant to be m ore descriptive. The goal is to get participants to verbalize their thoughts without “putting words in their mouth.” Do not lead the participant’s response by asking a question that implies or contains its own answer such as:

**•** Leading Question: Do you disl ike science class like most students?

**•** Neutral Question: How do you feel about science class?

You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such

as:

**•** What are you thinking now?

**•** Any other thoughts?

**•** Tell me how you came to pick that answer.

Use your best judgment. If a participant exhibits difficulty reading or understanding an item, probe the participant without biasing the response.

**•** Can you tell me in your own words what that question was asking? [comprehension and interpretation probe]

**•** What does the word [term] mean to you as it’s used in this question? [comprehension and interpretation probe]186 Use your best judgment. If a participant appears to have guessed or seems less than certain of the
answer he/she has given, probe the participant.

**•** How much would you say you know about [topic]? [confidence probe]

**•** How sure are you of your answer? [confidence probe]

**•** How did you come up with that answer? [recall strategy and bias probe]

**•** Was this easy or hard to answer? [comprehension and recall probe]

**•** How easy or difficult is it to remember [topic]? [recall probe]

**•** Did you find the list of answer choices easy or hard? [response probe]

**•** How easy or hard was it to choose an answer from that list of choices? [response probe]

Use your best judgment. If a participant appears uncomfortable with a word or answering a question, probe the participant.

**•** The question uses the word [term]. Does that sound OK to you, or would you choose something different? [sensitivity probe]

**•** In gene ral, how do you feel about this question? [sensitivity probe]

Use your best judgment. After testing an item, using the above generic probes to understand why the question as written is not clear, you may reword the question in a way that the participant

w ill understand based on shared interviewer feedback or previous interview experiences. If this is done, be sure to document precisely the reworded question that was tested and test the item using relevant generic probes.

In addition to generic probes, int erviewer protocols will include item

- specific probes. Not all
items have an item

- specific probe. Item

- specific probes are provided only if the researchers deemed it important to learn more about a cognitive process or issue that might not be raised
natural ly from the think

- aloud process or generic probing. If the issue to be inquired through the
item

- specific probe was addressed, or if time is running short, the item

- specific probe does not need to be asked. Use your best judgment.187 

## Science **Student Questio nnaire Interview Protocal**  **: Items and**
**Probes**

Interviewer Note : Proceed in the following manner for each of the survey items in this section:

**•** Direct the participant to the item and ask him/her to read the question (and any preceding or following instructions) out loud

**•** Instruct the participant to answer the question and to think aloud while doing so

**•** Record the participant’s comments

**•** Ask generic and item

- specific probes, as appropriate, before going on to the next item

Note: Some items are grade sp ecific and will be asked only of participants for that grade. The grade, subject, and respondent associated with the items are identified in brackets (e.g., [Grade 4

Science

– Student #1]) preceding the item.

Note: Each item to be tested is presented on its own page.188 

## Recruitment and Operational Forms, Letters, and Scripts

**Parent or Legal Guardian Consent Form**

Dear Parent or Legal Guardian:
<DATE>

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service and EurekaFacts are conducting voluntary research
interviews with <fourth

- /eighth

- /twelfth

- > grade students about survey questions for the National Assessment of Educational Progress (NAEP). Each student interview wil l take no more than one hour. The purpose of these interviews is not to test or grade students. NCES wants to hear their thoughts to improve the survey questions and make the questions easier to understand.

The interviewer will meet one

- on

- one with each s tudent to ask for his or her input. These sessions will be recorded for the researchers on this study to review. No one else will see or hear the recording. Your child’s answers will be used only for statistical purposes and may not be disclosed or used, i n identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. The comments students provide help NCES improve the survey questions to make the best survey possible. Your child will be given a $25 gift card as thanks for his or her time. In addition, you will be given a $25 gift card for your time to take the participating student to and from the
inte rview site.

We hope that you will give your consent for your son or daughter to participate in the voluntary NAEP interview by signing this form. Without your consent, your child will not be able to participate in the interview.

Respectfully,

Laurie Policastro NAEP Survey Development Educational Testing Service

I hereby give my permission for _____________________________________________ to (Student’s Name) participate in the NAEP survey interview.

____________________________________ ________________________ Signature Date189 

### Student (age 18 or older) Consent Form

Dear Student: <DATE>

On behalf of The National Center for Education Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service and EurekaFacts are conducting voluntary research interviews with students about survey questions for the National Assessment of Educational Progress (NAEP). Each student interview will take no more than one hour.

NCES needs to know if the survey questions are clear. Your feedback will help NCES improve the survey so students like you understand all the questions. You will be given questions that are very similar to the actual questions that will be asked during the national testing. The interview er will pay very close attention to your opinions and feelings about all aspects of the survey. He or she will also ask what you think about the questions after you complete them.

You will not be judged on how well you answer the questions, and you will not receive a grade. Other students like you will also participate in one

- on

- one sessions just like this. This session will be recorded for the researchers on this study to review. No one else will see or hear the recording. Your answers will be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP sur vey development process. Your comments, along with the comments of other students, will help NCES improve the survey. You will be given a $25 gift card in compensation for your time and effort. In addition, NCES is offering a gift card of $25 for a parent or legal guardian to remunerate him or her for the time involved and to help offset the travel/transportation costs of taking you to and from the EurekaFacts cognitive laboratory site. We thank you for your voluntary participation and ask you to please si gn the form below.

Thank you,

Laurie Policastro NAEP Survey Questionnaires Educational Testing Service

I hereby accept the terms described above and agree to participate voluntarily in the NAEP survey interview.

__________________________________ Print Name

____________________________________ ________________________ Signature Date190 

### Teacher and School Administrator Consent Form

Dear Educator: <DATE>

On behalf of The National Center for Educati on Statistics (NCES), part of the U.S. Department of Education, Educational Testing Service and EurekaFacts are conducting voluntary research interviews with teachers and school administrators about survey questions for the National
Assessment of Education al Progress (NAEP). Each teacher and school administrator interview
will take no more than one hour.

NCES needs to know if the survey questions are clear. Your feedback will help NCES improve the survey so educators like you understand all the questions. You will be given questions that are very similar to the actual questions that will be asked of teachers and school administrators whose students participate in future NAEP assessments. The interviewer will pay very close attention to your opinions and fee lings about all aspects of the survey. He or she will also ask what you think about the questions after you complete them.

Other educators like you will also participate in one

- on

- one sessions just like this. This session
will be recorded for the research ers on this study to review. No one else will hear the recording, or see your responses or comments. Your answers will be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. Your comments, along with the comments of other educators, will help NCES improve the survey. In
ad dition, you will be given a $40 gift card in compensation for your time and effort. We thank you for your voluntary participation and ask you to please sign the form below.

Thank you,

Laurie Policastro NAEP Survey Questionnaires Educational Testing Ser vice

I hereby accept the terms described above and agree to participate voluntarily in the NAEP survey interview.

__________________________________ Print Name

____________________________________ ________________________ Sig nature Date191 

### Phone Script Recruiting Parents or Legal Guardians of Student Participants

Good morning/afternoon. May I please speak to <NAME>?

Good morning/afternoon. My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the U.S. Department of Education’s National Center for Education Statistics (NCES).

We are recruiting students to participate in voluntary research i nterviews supporting the development of survey questions for the National Assessment of Educational Progress (NAEP). The survey will be administered to students throughout the United States. The research interviews will explore how students work through th e survey questions to ensure that they are clear and understandable to a broad range of students. The students who qualify for and participate in the study will be given a $25 gift card as thanks for their time. In addition, you will be given a $25 gift ca rd for your time to take the participating student to and from the interview
site.

The interviews will take place at the EurekaFacts cognitive laboratory site in Rockville, Maryland and will last no more than one hour. During the interview, your child wil l be asked to respond to several survey questions. NCES simply wants to know how he or she would answer the questions if they were included on an actual survey given to students after taking a NAEP

test.

These interviews are an important stage of the NAE P survey development process. The comments students provide help NCES improve the survey questions in order to create the best survey possible.

Would your child be interested in participating in this research project?

[If No] Thank you for taking the tim e to speak with us today about this important project. Have a wonderful day/evening.

[If Yes] Great, thank you very much. To ensure that we interview a broad mix of students, I have a few brief screening questions to ask about you and your child. It shoul d only take a few minutes of your time. First, I want to collect some additional information about your child to be certain he or she is a good fit for participating.

Screener

[Interviewer: Determine recruitment needs in advance. Record all responses. End screener at whatever point the parent or legal guardian gives a response that confirms that their group has already been sampled adequately. End screener script: Based on the r equirements of this study, we are not able to include your child in the interviews at this time. ]192 1. What is your child’s name?

2. Is <STUDENT NAME> a boy or a girl?

3. What grade is <STUDENT NAME> currently in? ( If the student is in twelfth

- grade, ask
if he or she is age 18 or over. This information will be used later in the script .)

4. What is your child’s race or ethnicity? [Interviewer: Only read list if parent doesn’t immediately reply] Are you...?

a. American Indian or Alaska Native
b. Asian

c. Black or African American
d. Hispanic or Latino

e. Native Hawaiian or other Pacific Islander f. White or Caucasian

5. What is the highest level of education that you have completed? [Interviewer: Do not read this list].

a. Less than high school degree
b. High school degree or GED

c. Some coll ege d. Trade school degree

e. Associate’s or Bachelor’s degree, or f. Post graduate degree?

6. Just to be sure we’re interviewing a broad range of students for this study, what is your
annual household income?

If the respondent indicates they prefer not to disclose this information, the interviewer should say “That’s fine”, and move on to the next question.

7. What is <STUDENT NAME’s> current zip code?

8. And has <STUDENT NAME> said <SHE, HE> would be interested in participating in
the interview session which will take n o more than one hour?

Eligible Student Participant Script

Thank you for taking the time to answer the screening questions. Your child is eligible to participate in the research interviews.

In the session, we will be asking <STUDENT NAME> to respond to several survey questions. NCES needs to make sure that students will understand what the survey is asking them to do.

Participation is entirely voluntary, and your child’s answers will only be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as193 required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]

. Interviewer note: this reference is for informational purposes only. If asked what law is be ing referenced, please provide the information contained within this bracketed text.]

Your child’s input will help NCES improve the survey that will be given nationwide. To thank <STUDENT NAME>, <SHE, HE> will receive a $25 gift card. In addition, you wil l receive a $25 gift card for your time to take the <STUDENT NAME> to and from the interview site.

Again, the interview will last no more than one hour and will be held at the EurekaFacts cognitive laboratory site in Rockville, Maryland. Do you think <STU DENT NAME> would like to participate?

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes (and the student is under age 18)] And would you be willing to submit a signed form on

site, indicating your approval for <STUDENT NAME> to participate? [If No] Thank you for taking the time to speak with us today about this important project.

[If Yes] Are you and your child available f or the interview at <TIME> on <DATE>?

[If the student is age 18 or over)] Is <STUDENT NAME> available for the interview at <TIME>
on <DATE>?

[Interviewer: if necessary, provide alternate times/dates to find an available time/date for
interview and then r ecord time/date of appointment]

Thank you. We will send you a confirmation e

- mail containing the address, directions and consent form. Please bring the signed consent form to the interview. What e

- mail and mailing address would you prefer us to use to sen d you this information? We may need to reach you by telephone. Is the number I called the best number to use?

[If the student is age 18 or over)] Thank you. We will send <STUDENT NAME> a confirmation

e

- mail containing the address, directions, and consent form that <HE/SHE> must be sign. What email address would be best to send this contact information to <STUDENT NAME> at? We may need to reach <STUDENT NAME> by telephone. Is the number I called the best number to
use?

Do you have any questions at this tim e?

[Interviewer: Answer any questions.]

If you or <STUDENT NAME> have any questions before the interview scheduled for <DATE/TIME>, please call <PHONE NUMBER>.

We look forward to having <STUDENT NAME> participate in this very important research project.194 

### E

**- mail Recruiting Parents or Legal Guardians of Student Participants** 

Subject: Research Study for Students in Fourth

- Grade, Eighth

- Grade, or Twelfth

- Grade

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the U.S. Department of Education’s National Center for Education Statistics (NCES).

We are recruiting <fourth

- /eighth

- /twelfth

- > grade students to participate in voluntary research interviews supporting the development of survey questions for the National Assessment of Educational Progress (NAEP) assessment. The interviews will explore how students work through the survey to ensure that the questi ons are clear and understandable to a broad range of
students. Your child will be given a $25 gift card as thanks for his or her time. In addition, you will be given a $25 gift card for your time to take the participating student to and from the
interview site. Students age 18 or older have the option to attend the interview without being accompanied by a parent or legal guardian.

The interview for your child will take place at the EurekaFacts cognitive laboratory site in Rockville, Maryland and will las t no more than one hour. The interview can be scheduled at the convenience of you and your child. During the interview, your child will be asked to respond to several survey questions. NCES simply wants to know how he or she would answer the questions if they were to be included in the NAEP assessment. Your child’s responses will be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 200 2 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. By sharing their feedback with us, students can directly impact the future of this national assessment by helping NCES evaluate the survey t o create the best questions possible.

If your child is interested in participating in this research study, please call <CONTACT
PHONE> or e

- mail <CONTACT E

- MAIL>. We look forward to speaking with you soon!

Thank you,

<NAME>
EurekaFacts
<CONTACT PHONE >195 

### Letter Recruiting Parents or Legal Guardians of Student Participants

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We are recruiting <fourth

- /eighth

- /twelfth

- > grade students to participate in voluntary research interviews supporting the development of survey questions for the National Assessment of
Educatio nal Progress (NAEP) assessment. Before giving the survey to all students, NCES needs to know if the survey questions are clear. Your input will help NCES improve the survey so students like your child can better understand the survey questions.

The inter views will take place at the EurekaFacts cognitive laboratory site in Rockville, Maryland and will last no more than one hour. Your child will be asked to respond to several survey questions. NCES simply wants to know how he or she would answer the questio ns if it were an actual survey that was given to students after taking a NAEP test. Your child’s answers will be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Edu cation Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. The comments students provide help NCES improve the survey questions to make the best survey possible. Your child will be given a $25 gift card as thanks for his or her time. In addition, you will be given a $25 gift card for your time to take the participating student to and from the interview site. Students age 18 or older have the option to attend the intervi ew without being accompanied by a parent or legal guardian.

If your child is interested in participating in this research project, please call [phone number] or

e

- mail [e

- mail address]. We look forward to speaking with you soon!

Thank you,

<NAME>
EurekaFacts
<CONTACT PHONE>196 

### Phone Script Recruiting Teachers and School Administrators

Good morning/afternoon. May I please speak to <NAME>?

Good morning/afternoon. My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the U.S. Department of Education’s National Center for Education Statistics (NCES). The National Assessment of Educational Progress (NAEP) has develope d survey questions for teachers and school administrators. Before giving the survey to all teachers and school administrators whose students participate in a NAEP assessment, NCES needs to know if the survey questions are clear. Your input will help NCES i mprove the survey so educators like you understand all the questions and the things you are being asked to do. To thank participants for their time, each participant will receive a $40 gift card.

The voluntary research interviews will take place at <LOCAT ION> and will last no more than one hour. You will be asked about the survey questions and to respond to several questions. NCES simply wants to know how you would answer them if it were an actual survey. NCES wants to hear your thoughts to improve the sur vey questions and make the questions easier to
understand. Your responses will be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573] . Interviewer note: this reference is for informational purposes only. If asked what law is being referenced, please provide the information contained within this bracketed text.]

These interviews are an important stag e of the NAEP survey development process. Your comments, along with the comments of other teachers and school administrators, will help NCES improve the survey.

Are you interested in participating in this research project?

If no: Thank you for your tim e. Have a wonderful day/evening.

If yes: Great, thank you very much. To ensure that we interview a broad mix of educators, I have a few brief screening questions to ask you. It should only take a few minutes of your time.

The interview will last no more than one hour . It will take place at <LOCATION>

.

Screener

[Interviewer: Determine recruitment needs in advance. Record all responses. End screener at whatever point the teacher or administrator gives a response that confirms that their group has197 already been sampled adequately. End screener script: Based on the r equirements of this study, we are not able to include you in the interviews at this time. ] What is your job title?

a. Administrator (GO TO Part A)
b. Teacher (GO TO Part B)

Part A.

1. At what school are you an administrator?

2. Are you at a school with a population that includes:

a. Fourth

- grade students (recruit ~5)
b. Eighth

- grade students (recruit ~5)

c. Twelfth

- grade students (recruit ~5)

3. What is the size of the student population at your school?

a. 299 students or less (recruit ~5)
b. 30 0

- 599 students (recruit ~5)

c. 600 students or more (recruit ~5)

Part B.

1. At what school are you a teacher?

2. What grade level do you teach?

a. Fourth grade (recruit ~5)
b. Eighth grade (recruit ~5)

c. Other  END SCREENER (Based on the requirements of this study, we are not able to include you in the interviews at this time. We greatly appreciate your help. Thank you for your time.)

3. Do you currently (or within this school year) teach science to eighth/fourth graders?

a. Yes  CONTINUE
b. No  END SCREENER (Based on the requirements of this study, we are not able to include you in the interviews at this time. We greatly appreciate your help. Thank you for your time.)

4. What is the size of the student population at your school?

a. 299 students or less (recruit ~5)
b. 300

- 599 students (recruit ~5)

c. 600 students or more (recruit ~5)198 Eligible Participant Script

Thank you for taking the time to answer the screening questions. You are eligible to participate in the research interviews. Again, the interview will last about 1 hou r and will be held at <LOCATION>. Are you available for the interview at <TIME> on <DATE>?

[Interviewer: if necessary, provide alternate times/dates to find an available time/date for interview and then record time/date of appointment]

Thank you.

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[Interviewer: Answer any questions.]

If you do have any questions before your interview scheduled for <DATE/TIME>, please call

< PHONE NUMBER>.

Thank you again for agreeing to be part of this very important research project.199 

### E

**- mail Recruiting Teachers and School Administrators** 

Subject: Research Study for School Administrators

Dear <NAME>:

My name is <CONTACT NAME> and I am co ntacting you from EurekaFacts, a local research company. We are conducting an important study for the U.S. Department of Education’s National Center for Education Statistics (NCES).

We are recruiting teachers and school administrators to participate in v oluntary research interviews supporting the development of survey questions for the National Assessment of Educational Progress (NAEP) assessment.

The interviews will explore how teachers and school administrators work through the survey to
ensure that t he questions are clear and understandable to a broad range of educators. To thank participants for their time, each participant will receive a $40 gift card.

The interviews will take place at <LOCATION> and will last no more than one hour. The
interview can be scheduled at your convenience. Our interviewer will arrange to meet with you
in

- person. During the interview, you will be asked to respond to sample survey questions. NCES wants to hear your thoughts to improve the survey questions and make the que stions easier to understand. Your responses will be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9 573].

These research interviews are an important stage of the NAEP survey development process. Your comments, along with the comments of other educators, will help NCES improve the

survey.

If you are interested in participating in this research study, please call <CONTACT PHONE> or

e

- mail <CONTACT E

- MAIL>. We look forward to speaking with you soon!

Thank you,

<NAME>
EurekaFacts
<CONTACT PHONE>200 

### Letter Recruiting Teachers and School Administrators

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research company. We are conducting an important study for the National Center for Education Statistics (NCES), part of the U.S. Department of Education.

We are conducting voluntary research interviews with te achers and school administrators about survey questions for the National Assessment of Educational Progress (NAEP). Before administering the survey questions, NCES needs to know if the survey questions are clear. Your input will help NCES improve the surve y so educators like you will understand all the questions.

The interviews will take place at <LOCATION> and will last no more than one hour. You will be asked to respond to survey questions. NCES simply wants to know how you would answer
them if it were an actual survey. NCES wants to hear your thoughts to improve the survey questions and make the questions easier to understand. Your responses will be used only for statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. To thank you for your time, you will receive a $40 gift card.

These interviews are an important stage of the NAEP survey development process. Your

co mments, along with the comments of other educators, will help NCES improve the survey.

If you are interested in participating in this research project, please call [phone number] or e

- mail [e

- mail address].

Thank you,

<NAME>
EurekaFacts
<CONTACT PHON E>201 

### Email/Letter Confirmation to Parent or Legal Guardian of Student (under

**age 18) Participants** 

IF SENT AS E

- MAIL

- Subject: Confirmation of NAEP Research Interview
<DATE>

Dear <PARENT NAME>,

Thank you for agreeing to allow your child, <STUDENT NAME> , to participate in this voluntary research interview to provide feedback on survey questions for the National Assessment of Educational Progress (NAEP). All the information your child provides will be used only for statistical purposes and may not be disc losed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

Your child has been scheduled for a research interview at <TIME> on <DATE>. The interview
will last no more than one hour. To thank your child for participating, <he/she> will receive a $25 gift card. In addition, you will receive a $25 gift card to thank you for your time in bringing your child to and from the interview site.

The interview will take place at the E urekaFacts cognitive laboratory site at:

<ADDRESS>
<DIRECTIONS>

We have <attached/included> a consent form <to/with> this confirmation. Please sign, date, and return the form when you bring your child to the interview, granting consent for your child t o participate in the NAEP research. Without your consent, your child will not be able to participate
in the interview.

We look forward to having <STUDENT NAME> participate in this very important research project. If you or your child has any questions, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>
EurekaFacts
<CONTACT PHONE>202 

### Email/Letter Confirmation to Student (age 18 or older) Participants

IF SENT AS E

- MAIL

- Subject: Confirmation of NAEP Research Interview
<DATE>

Dear <STUDENT NAME>,

Thank you for agreeing to participate in this voluntary research interview to provide feedback on survey questions for the National Assessment of Educational Progress (NAEP). All the information you provide will be used only for statistical purposes and m ay not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

You have been scheduled for a research interview at <TIME> on <DATE>. The interview will
last no m ore than one hour. To thank you for participating, you will receive a $25 gift card. In addition, your parent or legal guardian will receive a $25 gift card to remunerate him or her for the time involved and to help offset the travel/transportation costs o f taking you to and from the EurekaFacts cognitive laboratory site.

The interview will take place at the EurekaFacts cognitive laboratory site at:

<ADDRESS>
<DIRECTIONS>

We have <attached/included> a consent form <to/with> this confirmation. Please sign, date, and return the form when you come to the interview, agreeing to participate in the voluntary NAEP
research.

We look forward to having you participate in this very important research project. If you have any questions, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>
EurekaFacts
<CONTACT PHONE>203 

### Thank You Letter to Parent or Legal Guardian of Student (under age 18) **Participants**

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, I would like to thank you for allowing your child to participate in our survey research study. Your child’s input will help us develop the best poss ible survey questionnaires for the National Assessment of Educational Progress (NAEP).

The assistance provided by your child helped us to better understand how students approach the survey questions. Your child’s input, combined with input from other stu dents, is incredibly
useful and informative.

Again, thank you for your assistance.

Sincerely,

<NAME>
EurekaFacts
<CONTACT PHONE>204 

### Thank You Letter to Student (age 18 or older) Participants

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, I would like to thank you for your participation in our survey research study. Thanks to you, we are better able to develop the best possible survey questionnaires for the National Assessment of Educational Progress (NAEP).

Your assistance helped us better understand how students approach the survey questions. Your open, candid, and insightful comments are very helpful.

Again, thank you for your as sistance.

Sincerely,

<NAME>
EurekaFacts
<CONTACT PHONE>205 

### Thank You Letter to Teachers and School Administrators

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, I would like to thank you for your participation in our research study. Thanks to you, we are better able to develop the best possible survey questi onnaires for the National Assessment of Educational Progress (NAEP).

Your assistance helped us better understand how <teachers/school administrators> approach the survey questions. Your open, candid, and insightful comments were very valuable. Your input

, combined with input from other <teachers/school administrators> like you, is incredibly useful
and informative.

Again, thank you for your assistance.

Sincerely,

<NAME>
EurekaFacts
<CONTACT PHONE>Appendix C

2. List of New or Revised Items

Questionnaire Grade New Pilot Items

Science Student
K1SXB1 4 3, 6, 16, 19, 27, 28, 29

Science Student
K2SXB1 8 2 a

- g , 3b

- d, 5g, 8, 11 a

- c , 12, 13

Science Student
K3SXB1 12 7 a

- g , 8b

- d, 10g, 13, 15 a

- c , 16, 17

Science Teacher K1TQ

- COISX 4 2, 5 a

- c , 9f, 9j, 10 a

- l , 13c, 16 a

- c

Science Teacher K2TQ

- COISX 8 2, 5 a

- c , 9f, 9j, 10 a

- l , 14c, 17 a

- c

Science School K1SQ

- SX 4 1, 2 a

- d , 3, 4 a

- c , 9a

- b, 11 a

- c , 12 a

- c

Science School K2SQ

- SX 8 1, 2, 3, 4 a

- c , 9a

- b, 11 a

- c , 12 a

- c

Science School K3SQ

- SX 12 4, 5 a

- d , 6, 7 a

- c , 9a

- b, 11 a

- c , 12 a

- c

Core Teacher BET K1TQ

- BETCGHS 4 13 a

- l

Core Teacher BET K2TQ

- BETS 8 4 Note: Revised items are considered “new” items because they have not been administered in previous pilot or operational questionnaires.