Appendix C

NAEP Science Cognitive Interview Reports

and

List of New Questions

1. Science survey question cognitive interview study results (January 2013)

2. List of new/revised questions for Wave 1 materials
Listening.

Learning.

Leading.

NAEP Item Development (ID)

2015 Science Survey Questionnaires

2012–2013 Cognitive Interview Study

Grades 4, 8, and 12

Students, Teachers, & School Administrators

Deliverable in response to ID Task 3.1.3

Submitted: January 23, 2013
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

1

Table of Contents

Overview ......................................................................................................................................... 2

Project Objectives........................................................................................................................... 3

Study Design and Implementation ................................................................................................. 4

Data Collection and Analytic Methods........................................................................................... 6

ETS Proposed Recommendations................................................................................................... 8

Student................................................................................................................................... 9

Teacher ................................................................................................................................ 58

School Administrator ........................................................................................................... 72

Cognitive Laboratories Report and Findings (Eureka Facts)......................................................... 86

Appendices.................................................................................................................................. 183
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

2

Overview

As required by the National Assessment Governing Board (Governing Board), the National Center for 

Education Statistics (NCES) of the U.S. Department of Education will conduct nationwide assessments at 

grades 4, 8, and 12 for the National Assessment of Educational Progress (NAEP). In addition to assessing 

subject-area achievement, NAEP collects questionnaire data to provide context for the reporting and 

interpretation of assessment results. Questionnaire data comes from one of three respondent types: 

students, teachers, and school administrators. NAEP questionnaires serve to fulfill reporting 

requirements of federal legislation and to provide a context for reporting student performance.

This document presents findings and ETS recommendations based on results from a recent cognitive 

interview study of the 2015 Science survey questionnaire items. The report presents project objectives, 

study design and implementation, research design, cognitive interview findings, and recommendations 

for new or revised items. The report concludes with an appendix, including interview protocols, scripts, 

and forms.

As part of the item development process, items undergo several forms of pretesting including expert 

panel reviews and cognitive interviews (or cog labs). Cognitive interviews are a form of in-depth, oneon-one pretesting used to assess whether items are being understood as intended and to identify any 

problems in respondent comprehension. A goal of cog labs is to identify any problems that warrant 

adding, revising, or dropping of items prior to the pilot assessment that will be administered in 2014. 

These survey items will be included in all science administrations: paper-and pencil, Hands-on Tasks 

(HOTs), and Interactive Computer Tasks (ICTs).

Note that proposed item revisions in the section “ETS Proposed Item Recommendations” are 

recommendations for changes to the versions that were administered during the cognitive laboratories. 

Deleted text is indicated by strikethrough and new text is highlighted yellow.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

3

Project Objectives

This cognitive interview study investigates the cognitive processes that respondents use to answer 

survey questions. The ability of the respondent to comprehend the questions and provide valid 

responses is helpful in identifying the problems and limitations associated with the questions prior to 

the 2014 pilot. Early identification of such concerns, prior to administration to a large number of 

respondents, will increase the quality of the information collected from the questionnaires by reducing 

potentially confusing language and/or improving response categories.

In cognitive interviews, an interviewer uses a structured protocol in a one-on-one interview using two 

methods: think-aloud interviewing and verbal probing techniques. With think-aloud interviewing, 

respondents are explicitly instructed to “think aloud” (i.e., describe what they are thinking) as they 

determine their answers to the survey questions. The interviewer reads each question to the 

respondent and then records the cognitive processes that the respondent uses in arriving at an answer 

to the question. With verbal probing techniques, the interviewer asks probing questions, as necessary, 

to clarify points that are not evident from the “think aloud” process. These probes might include, for 

example, asking the respondents to rephrase the question in their own words or assess whether the

response categories are relevant. The specific protocols used in this study are presented later in this 

document.

Cognitive interview studies are largely observational. The data collected are predominantly qualitative 

verbal reports given in response to probes, think-aloud tasks, and volunteered comments. The objective 

is to identify and correct problems of ambiguity or misunderstanding, or other difficulties respondents 

have answering questions. The result should be questionnaires that are easier to understand and 

therefore less burdensome for respondents while also yielding more accurate information.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

4

Study Design and Implementation

Recruitment process

EurekaFacts, a subcontractor to ETS, recruited student, teacher, and school administrator participants

for this study within the greater Washington, DC and Baltimore, MD metropolitan areas through email 

and telephone outreach. Using an approved screener script via telephone, trained EurekaFacts 

interviewers screened parents of student candidates who responded to the outreach. 

During the telephone screening, interviewers confirmed the interview date, time, and location with 

parents. Student interviews took place at the EurekaFacts cognitive interview laboratory in Rockville, 

MD. Legal guardians of the participating students completed consent forms in advance of the cognitive 

interviews (see Appendix for recruitment scripts, consent forms, etc.) Students age 18 or older signed 

their own consent forms. Teacher and school administrator interviews were conducted at the 

respondents’ school or via telephone.

A monetary incentive aimed at ensuring participation and motivation of the participants was provided. 

Each participating student earned a $25 Visa gift card in compensation for time and effort. In addition, a 

Visa gift card of $25 for the students’ parent(s) was provided to remunerate parents for their time and 

to help offset the travel/transportation costs of bringing the participating student to and from the 

EurekaFacts cognitive laboratory site. Teachers and school administrators were provided with a $40 Visa 

gift card.

Study Sample and Population

Existing research and practice have failed to offer a methodological or practical consensus regarding the 

minimum or optimal sample size necessary to provide valid results for background question 

development.1 Nonetheless, a sample size of five to fifteen individuals has become the standard. Several 

researchers have confirmed the standard of five as the minimum number of participants per subgroup 

for analysis (i.e., cell) for the purposes of exploratory cognitive interviewing for question development.2

Other researchers have indicated that although a sample size of five per cell will likely identify major 

problems with a question, more is better, up to approximately fifteen per cell.3

Interview Protocols and Processes

Participants were first welcomed, introduced to the interviewer and the observer, and told they were 

there to help answer questions about how people answer survey questions. Participants were reassured 

that their participation was voluntary and that their answers may be used only for research purposes 

 

1

Almond, P. J., Cameto, R., Johnstone, C. J., Laitusis, C., Lazarus, S., Nagle, K., Parker, C. E., Roach, A. T., & Sato, E. 

(2009). White paper: Cognitive interview methods in reading test design and development for alternate 

assessments based on modified academic achievement standards (AA-MAS). Dover, NH: Measured Progress and 

Menlo Park, CA: SRI International.

2

Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to 

modeling cognitive processes. San Diego, CA: Academic Press.

3 Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

5

and may not be disclosed, or used, in identifiable form for any other purpose except as required by law 

[Education Sciences Reform Act of 2002, 20 U.S.C §9573]. 

Interviewers explained the cognitive interviewing process, conducted a practice question, and then 

participants answered questions verbally. 

Interviewers used several different cognitive interviewing techniques, including general think-aloud and 

question-specific probes, observation, and debriefing questions. 

The interview protocols were developed to facilitate capturing respondents’ thought processes while 

answering the questionnaire items. The interviewers encouraged participants to speak aloud all their 

thoughts while answering the questions and asked follow-up probes after each item. The probes were 

designed to help participants elaborate their responses.

If a participant did not appear to fully think aloud their response, interviewers used “continuers” to 

encourage the participant to be more descriptive and verbalize their thoughts without “putting words in 

their mouth.” Interviewers were instructed to remain as objective and unbiased as possible, but to offer 

a verbal “nudge,” such as:

 What are you thinking now?

 Any other thoughts?

 Tell me how you came to pick that answer.

If a participant exhibited difficulty reading or understanding an item, interviewers used probes similar to 

the examples below:

 Can you tell me in your own words what that question was asking? [comprehension and 

interpretation probe]

 What does the word [term] mean to you as it is used in this question? [comprehension 

and interpretation probe]

 How much would you say you know about [topic]? [confidence probe]

 How sure are you of your answer? [confidence probe]

 How did you come up with that answer? [recall strategy and bias probe]

 Was this easy or hard to answer?” [comprehension and recall probe]

 How easy or difficult is it to remember [topic]? [recall probe]

 Did you find the list of answer choices easy or hard? [response probe]

 How easy or hard was it to choose an answer from that list of choices? [response probe]

 The question uses the word [term]. Does that sound OK to you, or would you choose 

something different? [sensitivity probe]

 In general, how do you feel about this question? [sensitivity probe]

The generic probes were designed to understand why the question as written was not clear, and 

interviewers were instructed to reword the question in a way that the participant would understand 

based on shared interviewer feedback or previous interview experiences. Any instances of reworded 

questions that were tested have been documented in the “Findings” section of this report.

In addition to generic probes, interviewer protocols included item-specific probes. Not all items had an 

item-specific probe. Item-specific probes were provided only if the researchers deemed it important to 
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

6

learn more about a cognitive process or issue that might not be raised naturally from the think-aloud 

process or generic probing.

The interview protocols can be found in the Appendix.

Data Collection and Analytic Methods

The primary objective of the analysis was to identify items with problems and understand the nature of 

those problems. In the course of cognitive interviewing, some type of response difficulty will likely occur 

for almost every question. Our analytical approach is designed to identify the items that posed 

difficulties to respondents in ways that (1) appeared to reduce the validity of the response, (2) could be 

predicted to occur at a statistically significant rate in a larger population, and (3) exhibited a logical basis 

for why the problem occurred.

Interview teams recorded their notes into a Microsoft Excel data entry spreadsheet. The spreadsheet is 

based on a template developed by ETS researcher Zhitong Yang. The spreadsheet is designed to 

facilitate the collection and analysis of cognitive interview notes based on Roger Tourangeau’s fourstage task-analytical cognitive model, as elaborated and adapted by Gordon Willis,4

described below. 

There are four primary questions regarding respondents’ survey response processes that are explored 

through cognitive interviewing:

1. Comprehension: What does the respondent believe the question to be asking?

2. Memory: How does the respondent recall the information needed to answer the question?

3. Decision: How does the respondent approach (e.g., effort, sentiment) answering the question 

accurately and truthfully?

4. Response: Does the respondent’s answer match the given or planned response categories?

To collect information to identify problems with respondents’ survey response processes, we asked 

interviewers to explicitly code the occurrence of six different threats to obtaining valid and reliable 

survey responses.

1. Did the respondent skip over, misread, or reread any word(s)?

2. Did the respondent exhibit difficulty reading an item?

3. Did the respondent exhibit difficulty understanding the question?

4. Did the respondent exhibit difficulty understanding the meaning of particular words or 

concepts?

5. Did the respondent exhibit difficulty remembering the question or answer choices?

6. Did the respondent exhibit difficulty or discomfort answering the question?

In addition to the explicit coding to identify problems, interviewer data entry sheets also include 

descriptions of identified problems, summarizations of respondent verbal reports, and respondent 

responses to item-specific probes.

For the purposes of this report, the analysts focused on providing and presenting evidence to better 

assist in the item-level adjudication process. Consequently, most of the findings reported in this report 

 

4 Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

7

are centered on potential problems with items and discussion of the revisions to items that could best 

correct for the identified problems.

A four-stage process was used to identify problems with items. In the first stage, explicit codes related 

to the six aforementioned threats to obtaining valid and reliable survey responses were evaluated to 

highlight problematic items. In the second stage, the descriptions of identified threats were then 

reviewed to determine the nature of the problem. In the third stage, respondents’ verbal reports and 

respondents’ responses to item-specific probes were then reviewed to locate logical bases for the 

problems. In the fourth stage, analysts and item developers met to discuss the findings to identify 

potential approaches to item revision. These data collection and analytical methods are the basis for the 

results presented in the following findings section.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

8

ETS Proposed Item Recommendations
9

Student Questionnaire–Grade 4

[Grade 4 Science–Student #1]

Cognitive lab results:

1. There was much variation in how students understood the phrase “circuits and energy,” and two of 

the ten students reported not knowing what the phrase “circuits and energy” meant. They reported 

that “batteries and light” is clearer to them. 

2. Four students reported that they would have responded to the item differently if the phrase

“batteries and light” was used in place of “circuits and energy.”

Rationale:

1. Cognitive lab results show that grade 4 students’ understanding of the phrase “circuits and energy” 

varies, and some students do not know what the phrase meant.

2. Cognitive lab results also show that grade 4 students shared a common understanding of the 

phrase “batteries and light” and correctly linked the example to learning about electricity.

Recommendation:

1. ETS recommends using the trend phrase “batteries and light” instead of revised phrase “circuits 

and energy.”

New (Revised based on VC519362)

In this school year, how often have you done activities or projects to learn about

electricity (for example, circuits and energy)? 

A. Never

B. Rarely

C. Sometimes

D. Often
10

[Grade 4 Science–Student #2]

Cognitive lab results:

1. When asked how they would respond to the item if they had done activities with chemicals only at 

certain times during the school year, the methods used by grade 4 students to calculate the rate 

vary. Some provided the average rates; some others provided a frequency based on the particular 

period of time when they had the activities. Other grade 4 students did not understand the probe.

2. Compared with the revised version of this item (using “Never,” “Rarely,” “Sometimes,” and 

“Often” as options), more than half of students (6 of 10) preferred the response options in this 

item, and three stated they liked these response options because they are more concrete.

Rationale:

1. Cognitive lab shows that grade 4 students recall the information based on two major methods: 

frequency within the particular period of time when they have the activities, and averaging the 

frequency across the whole school year. Reporting frequency within a particular period is more 

common among grade 4 students.

2. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”

and “Often”) vary and are vague.

Recommendation:

1. Keep the trend item in the 2014 pilot administration.

VC315219

In this school year, how often have you done activities or projects to learn about 

chemicals (for example, mixing sugar or salt in water)?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
11

[Grade 4 Science–Student #3]

Cognitive lab results:

1. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”

and “Often”) vary and are vague.

2. When asked how they would respond to the item if they had done activities with chemicals only at 

certain times during the school year, most students chose the more frequent response choice, 

implying that they tend to report the frequency based on the particular period time when they had 

the activities.

Rationale:

1. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”

and “Often”) vary and are vague.

2. Fewer students preferred these response options in this item.

Recommendation:

1. Drop the revised version of the item in the 2014 pilot administration.

New (Revised based on VC315219)

In this school year, how often have you done activities or projects to learn about

chemicals (for example, mixing sugar or salt in water)?

A. Never

B. Rarely

C. Sometimes

D. Often
12

[Grade 4 Science–Student #4]

Cognitive lab results:

1. All students displayed a good understanding of the item.

2. Most students provided examples of “scientific tools.” The “magnifying glass” was the most cited 

tool; others included “pH testing strip” and “dropper.”

3. When being questioned which of the three versions of the answer choices made the most sense, 

the opinions were divided. Three students said the answer choices in this item made more sense to 

them because it provide more details; two students preferred response options “Never,” “Rarely,”

“Sometimes,” and “Often” stating they are easier; five students reported that the response options 

“No,” “Yes, a little,” and “Yes, a lot” made more sense because they are simple and fit their 

spontaneously generated answers.

4. When asked how they would respond if they had only used tools at certain times of the year, 

students typically used two major methods to recalculate the frequency of use 1) recall the 

frequency within the particular period, and, 2) average the frequency across the whole school year.

5. The cognitive lab results show that most students answered the question based on the frequency 

within the particular period of time when they have science classes.

Rationale:

1. Since all students displayed a good understanding of the item, the revised item stem works well.

2. Most students’ spontaneous answers were based on the frequency of scientific tool use within the 

particular period of time when they had the science activities.

3. Students’ are divided among which of the three sets of response options they prefer. Therefore, the 

current response options should be used to maintain trend.

Recommendation:

1. Replace current items VC315222, VC315227, and VC315229 with this item in the 2014 pilot 

administration.

New (Revised based on VC315222, VC315227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
13

[Grade 4 Science–Student #5]

Cognitive lab results:

1. All students displayed a good understanding of the item.

2. When asked to define “rarely,” “sometimes,” and “often,” students either used subjective, vague 

terms, or referred to a frequency similar to the trend response options (e.g., “once very few 

weeks”) to explain.

Rationale:

1. Although a few students preferred the response options in this item, when asked to define the 

options, they either used vague terms or used a frequency similar to the trend options.

Recommendation:

1. Do not use this item in the 2014 pilot administration.

New (Revised based on VC315222, VC315227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A. Never

B. Rarely

C. Sometimes

D. Often
14

[Grade 4 Science–Student #6]

Cognitive lab results:

1. All students displayed a good understanding of the item.

2. When asked to define “Yes, a little,” and “Yes, a lot,” students either used vague terms, or referred 

to the frequency response similar to the trend response options (e.g., “once very few weeks”) to 

explain.

3. Half of the students (5 of 10) preferred the response options in this item because they are easier.

Rationale:

1. Although half of the students (5 of 10) preferred the response options in this item, when being 

asked to define the options, they either used vague terms or used the frequency similar to the trend 

options to provide explanation.

Recommendation:

1. Do not use this item in the 2014 pilot administration.

New (Revised based on VC315222, VC315227, and VC315229)

Have you used scientific tools in school this year?

A. No

B. Yes, a little

C. Yes, a lot
15

[Grade 4 Science–Student #7]

Cognitive lab results:

1. All students displayed a good understanding of the item.

2. Out of 4 students who stated that they read books about science that are not for school work, only 

one said that he/she included it in selecting the choices.

3. Of the five students who responded that they had read a book or magazine about science in this 

school year, all indicated that they read most often in the classroom.

Rationale:

1. The cognitive lab results show that grade 4 students read books or a magazine about science most 

often in school. 

2. There is no need to specify whether grade 4 students read books or magazines that are for school 

or not for school.

Recommendation:

1. Keep the trend item in the 2014 pilot administration.

VC315239

In this school year, how often have you read a book or magazine about science?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
16

[Grade 4 Science–Student #8]

Cognitive lab results:

1. Two students showed hesitation in pronouncing the word “presented.”

2. All students displayed a good understanding of the item.

3. When asked if “to show what you have learned” was different from “to present what you have 

learned,” most students (6 of 10) said that these were the same. Only one student said that they 

differ, and one student did not understand what “show what you have learned,” meant.

4. Six of the ten students indicated that they have presented something to their class. Of these six, 

four presented information that they learned, and the other two presented a project that they 

worked on.

Rationale:

1. The cognitive lab results show that grade 4 students understand the word “presented,” and 

interpreted the word similar to the word “showed.”

Recommendation:

1. Keep the trend item in the 2014 pilot administration.

VC315259

In this school year, how often have you presented what you learned about science to 

your class?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
17

[Grade 4 Science–Student #9]

Cognitive lab results:

1. When asked to define “talked about,” the majority of students (8 of 10) answered in a similar 

manner, stating that it meant to “say something” or “talk to someone.” Only one student did not 

know how to respond.

2. When asked to define the term “measurements or results,” the students showed good 

understanding of these terms, and explained by referring to results, the end-effect, or product of 

experiments, and solutions to problems. Two students used examples to help illustrate what they 

were trying to say.

3. Seven of the ten students stated that they “never or hardly ever” talked about measurements or 

results. The remaining three reported that they talked about measurements or results with a 

teacher, classmates, or the whole class.

Rationale:

1. The cognitive lab results show that grade 4 students understand the phrase “talked about.”

Recommendation:

1. Keep the trend item in the 2014 pilot administration.

VC315288

In this school year, how often have you talked about measurements or results from your 

science activities or projects?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
18

[Grade 4 Science–Student #10]

Cognitive lab results:

1. Interviewer notes report no major problems with comprehension in this item. When asked to 

paraphrase the item, the students displayed a good understanding of the item.

2. All students but one defined “report” in a similar manner as writing or sharing information from 

measurements or science projects with other people.

3. Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this 

school year; two responded “once every few weeks,” and three said, “two or three times a week.”

Rationale:

1. The cognitive lab results show that grade 4 students understand the word “report.”

2. The revised version of the item (item 11) produced response that is more thorough.

Recommendation:

1. Replace the item with the revised version of the item (item 11) in the 2014 pilot administration.

VC315265

In this school year, how often have you written a report on your science activities or 

projects?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
19

[Grade 4 Science–Student #11]

Cognitive lab results:

1. Most students (9 of 10) displayed a good understanding of the item.

2. All students had a similar understanding of the phrase “science journal,” describing it as 

something you write in to keep track of information you have for science topics.

3. Four of the ten students would call the written work they do for a science activity or project a 

“science journal” (these same four students were unable to define a “lab write-up”). Two students 

would call the work a “report,” while two students would call it something else (a “writing piece” 

and “results”), and one was unsure what to call it. Only one student said he/she would call the 

work a “lab write-up,” but did not know why.

4. The descriptions that students provided for the phrase “lab write-up” were less consistent.

Rationale:

1. The cognitive lab results show that, for grade 4 students, this item captured more comprehensive

information than the trend item in which only the frequency of report writing was asked. Although 

the interviewer notes for the previous item (item ten) show that all but one student had a good 

understanding of the term “report,” the more generic formulation of item eleven (“asked to write 

about your science activities or projects”) produced a different answer pattern among students.

This suggests that the item ten did not capture all possible information from students on this topic. 

2. To further clarify the intent of the item, ETS suggests adding common examples (i.e., “such as 

reports, science journals, or lab write-ups”) to the item stem.

Recommendation:

1. Replace the trend item with this revised version in the 2014 pilot administration.

New (Revised based on VC315265)

In this school year, how often have you been asked to write about your science activities 

or projects (such as reports, science journals, or lab write-ups)?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
20

[Grade 4 Science–Student #12]

Cognitive lab results:

1. Two students could not define the phrase “library resources.” Of the remaining eight students, all 

gave books as an example of a library resource. Three also gave the example of computers, and 

two others gave the example of magazines.

2. Only after being probed, did students acknowledge the internet as being a library resource. After 

being asked, six students said that internet was a library resource, while three said that it was not. 

When probed, two students also acknowledged computers as library resources; another student 

was unsure, because computers in his/her school were not for fourth graders.

3. Nine students were asked whether they thought of their school library, the public library, or both

in selecting the options. Of the nine, seven thought only of the school library, one thought of both, 

and one answered that they “do not go to the library.”

4. When explicitly asked about individual resources, three students included encyclopedias, and three 

included videos, as library resources.

Rationale:

1. The cognitive lab results show that a majority of students thought only of books as library 

resources.

2. To capture all the other library resources used by student, ETS suggests adding common examples 

to the item stem (i.e., “such as books, magazines, computers, and audio-video materials”).

Recommendation:

1. Revise and keep the item in the 2014 pilot administration.

VC315266

In this school year, how often have you used your school library resources for science

(such as books, magazines, computers, and audio-video materials)?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
21

[Grade 4 Science–Student #13]

Cognitive lab results:

1. Overall, the students understood the phrase “science activities” in a similar manner: eight of the

ten students described it as activities or projects that one does, that are related to a science topic. 

Only two did not explicitly refer to a science topic.

2. Interviewer notes demonstrated a good and consistent understanding of the phrase “not for 

schoolwork” among the students: five students said it meant activities done at home, and three 

said it meant being “out of school.”

3. When asked to provide examples of activities they had done that were not for schoolwork, two 

students gave technology/engineering examples (e.g., fixing a TV), two gave natural science 

examples (e.g., studying bugs), and two gave chemical examples (e.g., mixing substances 

together). Two students who answered “never or hardly ever” could not remember any examples 

of activities they had done that were not for schoolwork.

Rationale:

1. The cognitive lab results show that a majority of students have a good understanding of the item as 

intended.

Recommendation:

1. Keep the trend item in the 2014 pilot administration.

VC315410

How often do you do science activities that are not for schoolwork?

A. Never or hardly ever

B. Sometimes

C. Often

D. Always or almost always
22

[Grade 4 Science–Student #14]

Cognitive lab results:

1. When asked to define a “science fair,” six students described it as related to making experiments; 

of these six, two also stated that “science fair” involved showing their work to other people. One

student defined it as showing something, but did not mention experiments.

2. Two students defined science fairs as having a competitive element; one said first place gets a 

prize, and the other said science fair was synonymous with science competition.

3. Two students did not understand what a “science fair” was; they knew only that it had something 

to do with science.

4. When asked to provide an example of a science fair, there were few prevailing trends. Six students 

stated that they had not participated in one; two referred to “event nights” at their schools where 

science activities took place, but they did not present their own experiments.

Rationale:

1. Cognitive lab results show that grade 4 students do not differentiate between the three scientific 

activities (i.e., “science fair,” “science competition,” and “science club”).

2. ETS suggests combining the three scientific activities in one question for grade 4 questionnaire.

Recommendation:

1. Revise the item combining the three scientific activities and add the item to the 2014 pilot 

administration.

New

In this school year, have you participated in a science club, a science fair, or a science

competition?

A. Yes

B. No
23

[Grade 4 Science–Student #15]

Cognitive lab results:

1. When asked to define a “science club,” all students provided responses indicating that science 

clubs had to do with people meeting together to learn more about science or do science activities. 

Additionally, four students explicitly said they were after-school activities.

2. When asked to provide an example of a science club, four students provided concise examples of 

specific clubs, three could only give examples of activities they had done in the clubs, and one 

student had never participated in a club. Two students could not recall any examples of science 

clubs.

Rationale:

1. Cognitive lab results show that grade 4 students do not differentiate between the three scientific 

activities (i.e., “science fair,” “science competition,” and “science club”).

2. Suggest combining the three scientific activities in one question for grade 4 questionnaire.

Recommendation:

1. Revise the item combining the three scientific activities and add the item to the 2014 pilot 

administration.

New

In this school year, have you participated in a science club?

A. Yes

B. No
24

[Grade 4 Science–Student #16]

Cognitive lab results:

1. When asked to define a “science competition,” five students gave responses indicating that this 

activity involves “prizes” or “first place, second place, etc.” Six students said it involved either 

doing your own project/doing science, or that it was like a science fair.

2. When asked to provide an example of a science competition, three students either gave “science 

fair” as their example, or gave their same example from item fourteen (which asked for an 

example of a science fair). One student said fairs and competitions were similar, but not the same.

3. Students were asked whether science fairs, science clubs, and science competitions were different 

things. Over half (6 of 10) said that they were all different things; three said they were all the 

same; one said that fairs and competitions were the same, but clubs were different. All three of the 

students who indicated that they were all the same gave the reason that they all dealt with science.

Rationale:

1. Cognitive lab results show that some grade 4 students do not differentiate between the three 

scientific activities (i.e., “science fair,” “science competition,” and “science club”).

2. Suggest combining the three scientific activities in one question for grade 4 questionnaire.

Recommendation:

1. Revise the item combining the three scientific activities and add the item to the 2014 pilot 

administration.

New

In this school year, have you participated in a science competition?

A. Yes

B. No
25

[Grade 4 Science–Student #17]

Cognitive lab results:

1. The cognitive results show that there were some variations in the definitions of “science museum”

among grade 4 students. However, most (7 of 10) indicated that it involved scientific activities, 

lessons, and/or technology, or gave appropriate examples. Two students were unsure of the 

meaning, and one student seemed to include non-scientific museums in their definition.

2. Nine students were asked what specific museums they considered science and non-science 

museums. Two students could not remember any science museums; two students could not 

remember any non-science museums.

3. When students were asked what museums they thought of as science museums, examples 

included: The Museum of Natural History, the Space Museum, and an animal museum called 

Amazing Animals.

4. When students were asked what museums they thought of as non-science museums, examples 

included: The National History Museum, the Wax Museum, and the Native American Museum.

Rationale:

1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science 

museum” with some variations. 

2. Since the intent of the item is to measure students’ “out of school” science activities, ETS suggests

not using the “not on a school trip” version of this item.

Recommendation:

1. Do not use this item in the 2014 pilot administration.

New

In this school year, have you visited a science museum?

A. Yes

B. No
26

[Grade 4 Science–Student #18]

Cognitive lab results:

1. When asked to define “on a school trip,” three students stated it was a trip involving their class 

and/or grade; three students mentioned riding on a school bus; three students said it was a field 

trip; two students simply re-worded the phrase (e.g., “a trip for school”).

2. Of the five students who provided examples, two referenced museums, and two could only discuss 

the activities they did at the museums, (one simply referenced a field trip to the Chesapeake Bay). 

The referenced museums were The Animal Museum and the Kid’s Museum in Baltimore Harbor.

3. Only one student had been to a science museum on a school trip this year.

Rationale:

1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science 

museum” with some variations. 

2. Since the intention of the item is to measure students’ “out of school” science activities, ETS 

suggests omitting the “not on a school trip” version.

Recommendation:

1. Do not use this item in the 2014 pilot administration.

New

In this school year, have you visited a science museum on a school trip?

A. Yes

B. No
27

[Grade 4 Science–Student #19]

Cognitive lab results:

1. Seven of the ten students displayed good understanding of the item.

2. When asked to define “not on a school trip,” seven of the ten students stated it was a trip that did 

not include school individuals or classmates, while five students described it as involving family 

and/or friends.

3. Only one student indicated that they had visited a science museum this year that was not on a 

school trip they went once.

4. Five students provided the following examples of trips that were not for school: The Air and 

Space Museum, the Natural History Museum, and the American Indian Museum.

Rationale:

1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science 

museum” with some variations. 

2. Most of the students (7 of 10) displayed good understanding of the phrase “not on a school trip” as 

the phrase intended.

3. Since the intention of the item is to measure students’ “out of school” science activities, ETS 

suggests using this version of the item.

Recommendation:

1. Add this item to the 2014 pilot administration.

New

In this school year, have you visited a science museum that was not on a school trip?

A. Yes

B. No
28

Student Questionnaire–Grade 8

[Grade 8 Science–Student #1]

Cognitive lab results:

1. In regards to the meaning of the phrase “circuits and energy,” students’ responses varied greatly. 

While two students understood a circuit as something consisting of wires, another two understood 

it as something that electricity moves through. Two students understood energy to be synonymous 

with the term “power.” Four students provided a limited and vague definition for the phrase, 

showing a partial understanding that “circuits and energy” referred to electricity. Two of the 

students were unable to provide a definition. 

2. Of the four students who answered “yes” to sub-item b, three recalled that the activity they 

thought of when formulating their answer was related to using wires to light a light bulb. The 

remaining student recalled learning about electricity, but could not provide examples of specific 

activities related to this topic.

3. When asked how they would respond to sub-item b if the example had been “batteries and light” 

instead of “circuits and energy,” half (5 of 10) responded that they would have changed their 

answers. Four of these students would have changed their answers from “no” to “yes,” while the 

fifth would have changed from “yes” to “no.”

VC304986

In your science class this year, have you done hands-on activities or projects with any of 

the following? Fill in one oval on each line. 

Yes No

a. Living things (for example, 

plants, animals, bacteria) A B VC304988

b. Electricity (for example, 

circuits and energy) A B Revised sub-item 

based on VC304989

c. Chemicals (for example, 

mixing or dissolving sugar or 

salt in water)

A B VC304991

d. Rocks or minerals (for 

example, identifying types) A B VC305007

e. Magnifying glass or 

microscope (for looking at 

small things)

A B VC305008

f. Thermometer or barometer (for 

making measurements) A B VC305009

g. Simple machines (for example, 

pulleys and levers) A B VC305012
29

Rationale:

1. There was significant variation in how students understood the phrase “circuits and energy.” In 

terms of their recollections of activities relating to electricity, students predominantly recalled 

activities where they had used batteries, wires, and/or light bulbs. In addition, probing revealed 

that if sub-item b included the example of “batteries and light,” instead of “circuits and energy,” 

twice as many students would have answered “yes” to this sub-item. Thus, “batteries and light” 

appears to be the most relevant examples for grade 8 students.

2. Although grade 8 students comprehend the item, there could be better distinction between

response options as shown in the following two items.

Recommendation:

1. For sub-item b, ETS recommends using the trend version examples “batteries and light” for the 

grade 8 student questionnaire in the 2014 pilot administration.

2. ETS recommends using the revised version of the item as shown in the student item #3 (with 

response options “Never,” “Rarely,” “Sometimes,” and “Often”) in the 2014 pilot administration.
30

[Grade 8 Science–Student #2]

Cognitive lab results:

1. All students displayed a good comprehension of the item.

2. When asked how they would respond if they had only done activities or projects at certain times 

during the school year (e.g., doing an activity every day for three weeks, but not since then), 

students provided a variety of responses. 

3. Cognitive lab results show that grade 8 students typically use two methods to respond to the 

hypothetical question if they had only done activities or projects at certain times during the school 

year: recall an experience within a particular period, or averaging the frequency across the entire 

school year.

4. Four of the 10 students expressed that the response options are hard to follow and do not align 

well with their science course schedule.

Rationale:

1. Although grade 8 students comprehend the item, the response options are difficult to follow.

Recommendation:

1. ETS recommends not using this item in the 2014 pilot administration.

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never or 

hardly ever

Once every 

few weeks

About once 

a week

Two or 

three times 

a week

Every day 

or almost 

every day

a. Living things (for 

example, plants, 

animals, bacteria)

A B C D E VC304988

b. Electricity (for example, 

circuits and energy) A B C D E

Revised subitem based on 

VC304989

c. Chemicals (for example, 

mixing or dissolving 

sugar or salt in water)

A B C D E VC304991

d. Rocks or minerals (for 

example, identifying 

types)

A B C D E VC305007

e. Magnifying glass or 

microscope (for looking 

at small things)

A B C D E VC305008

f. Thermometer or 

barometer (for making 

measurements)

A B C D E VC305009

g. Simple machines (for 

example, pulleys and 

levers)

A B C D E VC305012
31

[Grade 8 Science–Student #3]

Cognitive lab results:

1. Six students defined the response options using more subjective and vague terminology, with 

“rarely” meaning “not much,” “sometimes” meaning “once in a while,” and “often” meaning “a 

lot.” Of these six students, four defined “rarely” in terms of comparisons to the other categories 

(“not often,” or “less than sometimes”). The definitions for “sometimes” and “often” varied 

among these six students, but their definitions showed that they understood the concept in all 

cases.

2. Four students defined the response options according to more objective descriptions. Among these 

respondents, the term “rarely” was defined by all four students as either “once,” or “once a 

month”; the term “sometimes” had varying definitions; the term “often” was defined by all four 

students as once or multiple times a week.

3. When asked which version of the response options they preferred, nine of the ten students said 

they preferred the response options used in this item. Many implied that answering the item with 

the dichotomous response options used in item one was easy and these options were too broad to 

assess their internally generated answers. Moreover, the students explained that it was difficult to 

answer the item with frequencies, as used in item two, because the response options were too 

specific. 

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never Rarely Sometimes Often

a. Living things (for example, 

plants, animals, bacteria) A B C D VC304988

b. Electricity (for example, 

circuits and energy) A B C D Revised sub-item 

based on VC304989

c. Chemicals (for example, 

mixing or dissolving sugar or 

salt in water)

A B C D VC304991

d. Rocks or minerals (for 

example, identifying types) A B C D VC305007

e. Magnifying glass or 

microscope (for looking at

small things)

A B C D VC305008

f. Thermometer or barometer (for 

making measurements) A B C D VC305009

g. Simple machines (for example, 

pulleys and levers) A B C D VC305012
32

Rationale:

1. Verbal reports and answers to probing questions show that the students strongly preferred the 

response option scale used in item three, as compared to items one and two. It appears that the 

students could more easily fit their internally generated answers into the response options provided 

in item three; in addition, these response options placed a lower overall cognitive burden upon 

students. 

Recommendation:

1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item

in the 2014 pilot administration.

2. For sub-item b, ETS recommends using the trend wording of the sub-item (using examples 

“batteries and light”) in the 2014 pilot administration.
33

[Grade 8 Science–Student #4]

Cognitive lab results:

1. When asked to rephrase the item in their own words, the students displayed a good understanding 

of the item or instructions. 

2. When asked what the phrase “science topics” meant, five students implied that this meant subjects 

studied in science class, two students gave discrete examples of specific science topics, and two 

answered that this phrase meant “anything” that has to do with science. 

3. When asked if they would respond differently if the item used the term “science” instead of 

“science topics,” seven said they would answer differently. The most common reason given for 

this change of response among students (4 of 7) was that the term “science” was broader than 

“science topics,” and would allow them to consider more types of media as related to “science” in 

their responses. 

Rationale:

1. The item is classified under the issue “Organization of Instruction” and sub-issue “Instructional 

Strategies.” Adding the word “topics” to the relevant sub-items will help clarify the intent of the 

item.

Recommendation:

1. ETS recommends using the revised version of the three sub-items in the 2014 pilot administration.

VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few weeks

About 

once

a week

Two or

three 

times

a week

Every day 

or

almost

every day

a. Read a science

textbook, in class or at home A B C D E VC546510

b. Read a book or

magazine about science topics A B C D E Revised subitem based on 

VC305295

c. Use the Internet to learn about 

science topics A B C D E Revised subitem based on 

VC720562

d. Watch a movie, video, or DVD 

about science topics A B C D E

Revised subitem based on 

VC305307
34

[Grade 8 Science–Student #5]

Cognitive lab results:

1. When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub-items c and 

d implied summarizing what they did or learned in class. The other five students defined the 

phrase as generally talking with people (students or teachers) in their class. 

2. When asked what is meant by the term “lab write-up,” half of the students (5 of 10) described it as 

something akin to a report, describing a science lab that they had completed. Three students 

described it as another type of class assignment unrelated to a science lab. Two students were 

unable to provide a definition.

3. In regards to the meaning of the term “science journal,” responses varied greatly, and included the 

following: where they write their “warm-up” exercise (2 students); where to write notes (2 

students); where to write about experiments (2 students); something that is used to describe what 

happened in class (2 students); where one can write about science in general (2 students). 

VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few weeks

About 

once

a week

Two or

three 

times

a week

Every day 

or

almost

every day

a. Identify questions that can be 

addressed through science 

experiments

A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements you 

took for your science project or 

activity

A B C D E VC720635

d. Talk about the results of your 

science project or activity A B C D E VC720636

e. Watch your teacher do a 

science experiment or activity A B C D E VC720638

f. Make graphs or charts of the 

results from your science 

project or activity

A B C D E VC720639

g. Write a report on your science 

project or activity (such as 

reports, science journals, or lab 

write-ups)

A B C D E VC720641

h. Write about your science 

activities or projects (such as 

reports, science journals, or lab 

write-ups)

A B C D E New

h. Write a lab write-up on your 

science project or activity A B C D E New

i. Write a science journal A B C D E New
35

Rationale:

1. Interview notes revealed considerable variation in how students understood the phrases “lab writeup” and “science journal.” Considerable overlap between the two phrases exists in some students’ 

interpretations. To avoid confusion and loss of data, ETS suggests revising the two sub-items by 

using the more generic phrase to describe the activities, i.e., “write about your science activities or 

projects (such as reports, science journals, or lab write-ups).”

Recommendation:

1. ETS recommends using the revised sub-item h in the 2014 pilot administration.
36

[Grade 8 Science–Student #6]

Cognitive lab results:

1. When probed, seven students stated that they were thinking about their school’s library when 

answering this item and three stated that they were thinking of both their school’s library and a 

public library.

2. In regards to what students thought of as library resources, the top library resource identified by 

students was books (9 of 10). Other resources that students mentioned, unaided, were computers 

(4 students), internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1 

student), and magazines (1 student). 

3. When asked directly about each resource, eight students said they thought of computers as 

resources while answering the item, seven students thought of encyclopedias, five thought of the 

internet, five of magazines, and four thought of videos.

Rationale:

1. Not all students’ spontaneous responses included library resources other than books. ETS suggests 

adding specific examples at the end of the item to be consistent with the item used in grade 4 and 

grade 12 questionnaires. 

2. Since the item is under issue “Availability and Use of Instructional Recourses” and sub-issue 

“Products,” ETS suggests clarifying that the question as asking about “school library” uses.

Recommendation:

1. Revise and keep the item in the 2014 pilot administration.

VC315266

In this school year, how often have you used your school library resources for science

(such as books, magazines, computers, and audio-video materials)?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
37

[Grade 8 Science–Student #7]

Cognitive lab results:

1. Interviewer notes do not indicate any consistent problems in students’ understanding of the item or 

instructions.

2. When asked what was meant by the phrase “science-related activities” in sub-item a, the students 

either provided general, broad descriptions related to activities that had “anything to do with 

science,” or offered some examples such as measurements, experiments, and research.

3. When asked what was meant by the phrase “not for schoolwork,” five students emphasized in their 

responses that these activities were done outside of school, or were not assigned schoolwork. Four 

students emphasized in their explanations that these activities were done based on personal 

interests.

Rationale:

1. Cognitive lab results show that all students understood and interpreted the item as intended. 

Recommendation:

1. ETS recommends keeping the trend item in the 2014 pilot administration.

VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill 

in one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. I do science-related activities that are not for 

schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help me in 

the future. A B C D VC305353
38

[Grade 8 Science–Student #8]

Cognitive lab results:

1. When asked what was meant by the term “covered” in the item stem, seven students provided a 

general, broad definition for the term (“gone over,” “what we learned”). In their definitions, three 

of the students included a term that described the extent of the learning (“we learned about it 

completely,” “we know it thoroughly”). 

2. Students that answered “yes” to any of the sub-items were asked to describe how the material was 

“covered” in their class. A majority of the students (8 of 9) answered this by providing the names 

of the topics that were covered. All of the topics mentioned by students fit within the existing subitem’s category. One student described the types of assignments and materials that were used to 

learn each topic. 

3. When probed, seven students said that they would prefer a larger scale that asked to what extent 

the topics were covered, as opposed to “yes” and “no” response options. The main reason for this 

preference, according to five of these seven students, was the ability to more accurately assess the 

extent to which these topics were covered. Of the three students who preferred the “yes/no”

format, two had previously included a term that described the extent of learning in their 

description of the term “covered.”

Rationale:

1. Cognitive lab results show that all students displayed a good understanding of the item and the 

word “covered.”

Recommendation:

1. ETS recommends keeping the trend item in the 2014 pilot administration.

VC304978

In your science class this year, which of the following topics have been covered? Fill in one oval on each 

line.

Yes No

a. Life science (for example, biology, the human body, or ecology) A B VC304982

b. Physical science (for example, energy, physics, or chemistry) A B VC304983

c. Earth and space science (for example, geology or astronomy) A B VC304984

d. Engineering and technology (for example, designing solutions to 

problems) A B VC304985
39

[Grade 8 Science–Student #9]

Cognitive lab results:

1. When asked what was meant by the term “science fair” in sub-item a, all ten students provided 

explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven 

of the ten explanations also indicated that these projects were shown or talked about to others.

a. Eight students were able to provide an example of a science fair in which they or a 

classmate had participated.

2. When asked what was meant by the term “science club” in sub-item b, students had varying 

answers. Five students indicated that these clubs occurred after school; one student said that they 

occur annually, and the other four students did not discuss when clubs occurred. Four students 

included in their definition that the purpose of the club was to do science activities, and three 

noted that the purpose of the club was to discuss or talk about science. 

a. Five students were able to provide an example of a science club in which they or a 

classmate had participated.

3. When asked what was meant by the term “science competition” in sub-item c, nine students 

provided a definition that described it as similar to a science fair, where one makes a project, 

invention, or experiment and competes against others. One student mentioned that people also 

compete by answering questions. One student was unable to define the term.

a. Only one student was able to provide an actual example of a science competition, which 

they described as a “science club competition, where they had to build things to compete in 

certain challenges.” One student answered that a geography bee was a science competition. 

The other eight students were unable to retrieve a cogent example. 

4. All ten students answered “no” to sub-items a and c, and nine answered “no” to sub-item b. 

Rationale:

1. Most students comprehend the item as intended.

2. Although almost all students indicated that they have not participated in any of the activities, ETS 

suggests that the item could be still used to measure the growth of out-of-school science activities.

Recommendation:

1. ETS recommends adding the item to the 2014 pilot administration.

New

In this school year, have you participated in any of the following activities? Fill in one

oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B
40

[Grade 8 Science–Student #10]

Cognitive lab results:

1. All ten students interpreted the item as asking about trips to museums that involve science.

2. When asked what was meant by the term “science museum,” a majority of students (6 of 10) gave 

general and vague definitions for the term: “a museum that has to do with science of any kind” and 

“a museum about science.” The other four students provided examples of what might be found in 

a science museum (“a collection of experiments” and “rocks and dinosaurs”). 

3. When asked what they think of as science museums, eight students were able to provide examples 

of actual science museums, including Air and Space, Smithsonian, and Port Discovery. Only one 

student provided an example that was not an actual science museum. 

Rationale:

1. Cognitive lab results show that grade 8 students comprehend the item as intended. 

2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school 

trip” and “not on a school trip,” as shown in the following two items, ETS suggests using the “not 

on a school trip” version of the item to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends not using the item in the 2014 pilot administration.

New

In this school year, have you visited a science museum?

A. Yes

B. No
41

[Grade 8 Science–Student #11]

Cognitive lab results:

1. The students interpreted the item as asking if they have visited a museum on a school trip, or 

under the supervision of the school.

2. When asked what was meant by the phrase “on a school trip,” all ten students either found the 

term synonymous with the term “field trip,” or described traveling to a location with people from 

their school. 

3. When asked if they had been on a school trip to a science museum, six students described a 

situation where they had been on a school trip. Of those six, three provided descriptions of the 

specific trip, including traveling to a science museum with classmates, the class, or the teacher on 

a bus. 

Rationale:

1. Cognitive lab results show that grade 8 students understand the item as intended. 

2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school 

trip” and “not on a school trip” (as compared to the following item), ETS suggests using the “not 

on a school trip” version of the item to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends not using the item in the 2014 pilot administration.

New

In this school year, have you visited a science museum on a school trip?

A. Yes

B. No
42

[Grade 8 Science–Student #12]

Cognitive lab results:

1. When asked what was meant by the phrase “not on a school trip,” seven of the students stated that 

it meant visiting the museum without others from their school, and three said it meant visiting in 

the presence of family or friends. 

2. Six students described a time when they visited a science museum that was not on a school trip, 

and four students said they could not remember a visit to a science museum that was not on a 

school trip.

Rationale:

1. Cognitive lab results show that grade 8 students understand the item as intended. 

2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school 

trip” and “not on a school trip," ETS suggests using the “not on a school trip” version of the item 

to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends adding this item to the 2014 pilot administration.

New

In this school year, have you visited a science museum that was not on a school trip?

A. Yes

B. No
43

Student Questionnaire–Grade 12

[Grade 12 Science–Student #1]

Cognitive lab results:

1. When the students were asked what they thought was meant by the term “circuits and energy,” 

half the students (5 of 10) attempted to explain the concept of how circuits and energy are related 

(“circuits carry electrical energy from one place to another”). The other half of students named 

tangible components of circuits such as wires, batteries, and light bulbs. 

2. Of the three students that answered “yes” to sub-item b, two recalled activities in class related to 

electricity or circuits. The other described a chemistry lab where reactions were measured as a 

function of heat applied by an electric heater.

3. When probed, eight of the ten students would not change their answer if they would change their 

response to sub-item b if instead of “circuits and energy” the example was “batteries and light.” 

Five students noted they believed that asking the item this way would have changed its intent. 

Three of these five thought that “light” did not fit with the concept of electricity. 

Rationale:

1. Cognitive lab results show that, for grade 12 students, the phrase “circuits and energy” appears to 

be a better example because this phrase includes “batteries and light,” in its overall meaning.

Recommendation:

1. For sub-item b, ETS recommends using the revised examples “circuits and energy” for the grade 

12 student questionnaire instead of using “batteries and light” in the 2014 pilot administration.

2. ETS recommends using the revised version of the item as shown in the student item #3 (with 

response options “Never,” Rarely,” Sometimes,” and “Often”) in the 2014 pilot administration.

VC304986

In your science class this year, have you done hands-on activities or projects with any of 

the following? Fill in one oval on each line. 

Yes No

a. Living things (for example, 

plants, animals, bacteria) A B VC304988

b. Electricity (for example, 

circuits and energy) A B Revised sub-item 

based on VC304989

c. Chemicals (for example, 

mixing or dissolving sugar or 

salt in water)

A B VC304991

d. Rocks or minerals (for 

example, identifying types) A B VC305007

e. Magnifying glass or 

microscope (for looking at 

small things)

A B VC305008

f. Thermometer or barometer (for 

making measurements) A B VC305009

g. Simple machines (for example, 

pulleys and levers) A B VC305012
44

[Grade 12 Science–Student #2]

Cognitive lab results:

1. All students displayed a good comprehension of the item.

2. When asked how they would respond if they had only done activities or projects at certain times 

during the school year (e.g., if they had done an activity every day for three weeks, but they have 

not done the activity since then), students typically use two methods to respond: recall experience 

within a particular period of time, and averaging the frequency across the whole school year. 

a. Of the five students who would answer on the lower end of the scale, three explained their 

choices by describing a process similar to averaging the frequency of the activity over the 

course of a school year. 

b. The other two students explained that the response option “about once a week” or higher 

would be a poor way of describing this situation. 

c. One student chose “every day or almost every day” explaining that they would consider the 

period of time that they were working on that project rather than the entire year. 

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never or 

hardly ever

Once every 

few weeks

About once 

a week

Two or 

three times 

a week

Every day 

or almost 

every day

a. Living things (for 

example, plants, 

animals, bacteria)

A B C D E VC304988

b. Electricity (for example, 

circuits and energy) A B C D E

Revised subitem based on 

VC304989

c. Chemicals (for example, 

mixing or dissolving 

sugar or salt in water)

A B C D E VC304991

d. Rocks or minerals (for 

example, identifying 

types)

A B C D E VC305007

e. Magnifying glass or 

microscope (for looking 

at small things)

A B C D E VC305008

f. Thermometer or 

barometer (for making 

measurements)

A B C D E VC305009

g. Simple machines (for 

example, pulleys and 

levers)

A B C D E VC305012
45

d. The remaining three students did not specify the response option they would choose and 

explained that their answer would depend on how often they studied that topic or how well 

they would remember it.

Rationale:

1. Cognitive lab results show that grade 12 students used two methods to respond to the item: recall 

experiences within a particular period of time and averaging the frequency across the whole school 

year. 

2. Two students explicitly stated that the response option “about once a week” or higher would be a 

poor way of describing the situation.

Recommendation:

1. ETS recommends not using this item in the 2014 pilot administration.
46

[Grade 12 Science–Student #3]

Cognitive lab results:

1. The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to 

them. Five of the students provided objective, quantifiable answers, and five students provided 

subjective answers. 

a. For the term “rarely,” three of the five who answered objectively indicated that the term 

means “once,” and two described it as “once every few weeks.” Of the five who answered 

subjectively, three compared the term to the other response categories (“between never and 

sometimes”).

b. For the term “sometimes,” three of the five who answered objectively indicated that it 

meant more than once during the week, and two described it as more than once in a month. 

Of the five that answered subjectively, three compared it to the other response categories 

(between “rarely” and “often”). 

c. For the term “often,” three of the five who answered objectively indicated that it meant 

every day, and two described it as once or more in a week. Of those who answered 

subjectively, all used a term to describe a large amount of use as “very frequently.” 

2. When asked how they would respond if they had only done activities or projects at certain times 

during the school year, six of the ten students provided a clear indication of which of the response 

options they would choose. 

a. All six students provided answers at the higher end of the scale by choosing either 

“sometimes” (4 of 6) or “often” (2 of 6). 

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never Rarely Sometimes Often

a. Living things (for example, 

plants, animals, bacteria) A B C D VC304988

b. Electricity (for example, 

circuits and energy) A B C D Revised sub-item 

based on VC304989

c. Chemicals (for example, 

mixing or dissolving sugar or 

salt in water)

A B C D VC304991

d. Rocks or minerals (for 

example, identifying types) A B C D VC305007

e. Magnifying glass or 

microscope (for looking at 

small things)

A B C D VC305008

f. Thermometer or barometer (for 

making measurements) A B C D VC305009

g. Simple machines (for example, 

pulleys and levers) A B C D VC305012
47

b. Two of these students explained that they based their answers on the amount of time that 

they were working on that project rather than the entire school year. Two explained that if 

they had spent three weeks on a project they would have learned a lot, and one stated that 

“rarely” would not be applicable in the situation described in the probe. 

3. When asked which of the first three item formats they preferred, six students preferred item three, 

two preferred item two, one preferred item one and one said that all are easy to understand. 

a. Of the six students that preferred item three, five stated they believed these response 

options better matched their answers compared to the other items. 

b. The two students that preferred item two stated that they liked the specificity of the 

response options.

Rationale:

1. Verbal reports and answers to probing questions show that the students strongly preferred the 

response option scale used in item three, as compared to items one and two. It appears that the 

students could more easily fit their internally generated answers into the response options provided 

in item three; in addition, these response options placed a lower overall cognitive burden upon 

students. 

Recommendation:

1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item 

in the 2014 pilot administration.

2. For sub-item b, ETS recommends using the revised wording of the sub-item (using examples 

“circuits and energy”) in the 2014 pilot administration.
48

[Grade 12 Science–Student #4]

Cognitive lab results:

1. When asked what the phrase “science topics” meant, six students explained that it means academic 

subjects such as physics, chemistry, and biology. Four students answered that this phrase means 

anything that has to do with science. 

2. Four of the five students who indicated they read books or magazines that were not for 

schoolwork did not include these books or magazines when answering sub-item b. The reason 

provided for not including these books or magazines was that they were not being read for their 

science classes.

3. When asked if they would respond differently if the item used the term “science” instead of 

“science topics,” four of the ten students responded that they would answer the item differently. 

One student explained that he/she would have included activities done at home in addition to those 

completed in class. The explanations of the other two students suggest that the phrase “science 

topics” was interpreted as referring to more narrow and strictly defined science subjects, while the 

term ”science” would include a wider range of subjects. 

Rationale:

1. The item is classified under issue “Organization of Instruction” and sub-issue “Instructional 

Strategies.” Adding the word “topics” to the relevant sub-items, helps further clarify the intent of 

the item. 

Recommendation:

1. ETS recommends using the revised version of the three sub-items in the 2014 pilot administration.

VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few weeks

About 

once

a week

Two or

three 

times

a week

Every day 

or

almost

every day

a. Read a science

textbook, in class or at home A B C D E VC546510

b. Read a book or

magazine about science topics A B C D E Revised subitem based on 

VC305295

c. Use the Internet to learn about 

science topics A B C D E Revised subitem based on 

VC720562

d. Watch a movie, video, or DVD 

about science topics A B C D E

Revised subitem based on 

VC305307
49

[Grade 12 Science–Student #5]

Cognitive lab results:

1. When asked what the phrase “talk about” meant in sub-item c and d, all ten students, described it 

as “discussing” with the entire class, classmates, or teachers. 

2. When probed, most students (7 of 10) provided similar definitions of the term “lab write-up” in 

sub-item h. 

a. These seven students provided explanations that were centered on the idea that it was a 

written piece that summarized a science lab. However, three of these students expressed 

some confusion about the meaning of the term “lab write-up.” 

b. Two students believed that the term “lab write-up” meant answering questions. 

c. One student was unable provide a definition.

3. When initially answering, three students expressed some confusion about the meaning of the term 

“science journal” in sub-item i. When asked directly what the term meant, the students had 

varying responses. While providing a definition, six of the students expressed that they were 

unsure what the term meant; of these six, two were unable to guess a definition. Two students 

believed the term could mean a long science report. Two other students believed a science journal 

might be where students keep their notes for science. Two students thought the term might mean a 

short daily class or homework assignment. One student thought a science journal might be where 

VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few weeks

About 

once

a week

Two or

three 

times

a week

Every day 

or

almost

every day

a. Identify questions that can be 

addressed through science 

experiments

A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements you 

took for your science project or 

activity

A B C D E VC720635

d. Talk about the results of your 

science project or activity A B C D E VC720636

e. Watch your teacher do a 

science experiment or activity A B C D E VC720638

f. Make graphs or charts of the 

results from your science 

project or activity

A B C D E VC720639

g. Write a report on your science 

project or activity (such as 

reports, science journals, or lab 

write-ups)

A B C D E VC720641

h. Write a lab write-up on your 

science project or activity A B C D E New

i. Write a science journal A B C D E New
50

short brief constructed response items were answered. Another student thought that it might be 

used to provide a summary of an experiment for publication. 

Rationale:

1. Interview notes revealed considerable variation in how students understood the phrases “lab writeup” and “science journal.” Considerable overlap between the two phrases exists in some students’ 

interpretations. To avoid confusion and loss of data, ETS suggests revising the two sub-items by 

using the more generic phrase to describe the activities: “write about your science activities or 

projects (such as reports, science journals, or lab write-ups).”

Recommendation:

1. ETS recommends using the revised sub-item in the 2014 pilot administration.
51

[Grade 12 Science–Student #6]

Cognitive lab results:

1. When answering the probe, “did you think about your school’s library, a public library, or both,” 

there was an even split among students. 

a. Five students thought only of the school library resources when answering.

b. Five students thought of both the school and public libraries and their resources. 

2. When asked to provide examples of library resources that they have used for science class, all 

students were able to provide at least one example. Some of these included:

a. Biographies, websites, online journals, internet databases, magazines, autobiographies, 

encyclopedia Britannica, and textbooks.

3. A majority of students (9 of 10) considered computers and the internet as library resources.

a. While thinking about his answer, one student was not sure whether the probe was asking 

about the use of computers in general, or solely as a means to access the internet. 

4. A majority of students (9 of 10) did not think of videos as a library resource. One of these students 

responded to the probe by indicating that while not mentioning videos as a library resource 

originally, he had used the library computers to watch online videos.

Rationale:

1. Not all students’ spontaneous responses included library resources other than books. ETS suggests 

adding specific examples at the end of the item to be consistent with the item used in grade 4 and 

grade 8 questionnaires. 

2. Since the item is under issue “Availability and Use of Instructional Recourses” and sub-issue 

“Products,” ETS suggests clarifying the question as asking about “school library” use.

Recommendation:

1. Revise and keep the item in the 2014 pilot administration.

VC315266

In this school year, how often have you used your school library resources for science

(such as books, magazines, computers, and audio-video materials)?

A. Never or hardly ever

B. Once every few weeks

C. About once a week

D. Two or three times a week

E. Every day or almost every day
52

[Grade 12 Science–Student #7]

Cognitive lab results:

1. Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10) 

stated that the item asked if they agreed or disagreed with these statements about science. Three 

students stated that the item is asking about their opinions and feelings about science.

2. When probed what the phrase “science-related activities” meant to them, a majority of students (9 

of 10) reported that it meant doing anything related to science such as watching a movie or 

conducting an experiment.

3. When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10) 

stated that it was something that was not required by the school (something extracurricular).

4. When asked to provide an example of “science-related activities that you or your classmates have 

done or might do that are not for schoolwork,” more than half of the students (6 of 10) were able 

to provide an example, such as:

a. Using a telescope, working at NIH, reading Stephen Hawking books, nature treks, and 

science fairs.

b. More than half of the students (6 of 10) stated that these activities were not for schoolwork 

because a teacher did not instruct them to perform these activities, they would not be 

graded, and the activities would not be discussed in class.

Rationale:

1. Cognitive lab results show that all students understood and interpreted the item as intended. 

Recommendation:

1. ETS recommends keeping the trend item in the 2014 pilot administration.

VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill 

in one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. I do science-related activities that are not for 

schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help me in 

the future. A B C D VC305353

f. When I graduate from high school, I would 

like to have a job related to science. A B C D VC720596
53

[Grade 12 Science–Student #8]

Cognitive lab results:

1. A majority of students (7 of 10) indicated that the item asked whether they had done the listed 

activities during the current school year. Three students thought it referred to all extracurricular 

activities.

2. A majority of students (8 of 10) stated that they did not participate in any of the listed activities. 

Two students had participated in the listed activities; one had participated in a science club and a 

science competition. The other student had participated in a science fair. Those who reported 

participating in the listed activities also specified that they participated in the activities during the 

current school year.

3. When asked what the phrase “science fair” means to them, most students (7 of 10) explained that 

it refers to creating a research project and presenting it to other people. 

a. Two students did not differentiate the terms “science fair” and “science competition.”

b. One student had never participated in a science fair but had attended one.

c. Over half of students (6 of 10) were able to provide an example of a science fair, but only 

two students mentioned science fairs that were taking place this year. Four students 

provided examples of science fairs ranging chronologically from elementary school to their 

first year of high school. Four students were unable to provide specific examples of science 

fairs.

4. When asked what the phrase “science club” meant to them, students had varied ideas about 

whether the club was a school sponsored, extracurricular, or an activity separate from the school 

altogether. All students stated that a science club was a club where students meet and discuss 

science-related topics. When asked to provide an example of a science club that they or their 

classmates have participated in or might participate in, students provided the following examples:

a. Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club, 

biology, physics, or chemistry clubs, and an environmental club.

b. Three students were unable to provide an example, or stated that a science club was the 

same as a science fair or competition.

5. Most students (7 of 10) stated that a science competition includes conducting an experiment and 

being judged against other students, explaining that it was different from a fair. Two students 

considered a science competition to be the same as a science fair or club. When asked to provide 

an example of a science competition that they or their classmates have participated in or might 

participate in, some examples included:

a. Science bowl, Siemens competition, NIH competition, and county competitions.

New

In this school year, have you participated in any of the following activities? Fill in one

oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B
54

Rationale:

1. Most students comprehend the item as intended.

2. Although almost all students indicated that they have not participated in any of the activities, ETS 

suggests that item could be still used to measure the growth of out-of-school science activities.

Recommendation:

1. ETS recommends adding the item to the 2014 pilot administration.
55

[Grade 12 Science–Student #9]

Cognitive lab results:

1. Overall, students did not exhibit any consistent comprehension problems. 

2. A majority of students (9 of 10) stated that they were being asked if they had been to a museum 

that focuses on science. Only two students indicated that the visit had to take place during the 

current school year.

3. When asked what the phrase “science museum” meant to them, all students stated that it is a 

museum that focuses on any field of science.

4. When asked to provide examples of science museums, all students responded and listed:

a. The National Air and Space Museum, the Natural History Museum, and the Cryptologic 

Museum.

5. When asked to provide examples of museums that are not for science, all students responded and 

listed:

a. The National Gallery of Art, Spy Museum, Native American Museum, and Holocaust 

Museum.

6. When asked to provide an example of a science museum that they or their classmates have visited 

or might visit, all students responded and provided these examples:

a. The Baltimore Science Museum, the Natural History Museum, the National Air and Space 

Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but 

was unable to name it.

7. Of the four students who were asked who they went to the science museum with this school year, 

two went with their class, one went with friends, and one went with family.

Rationale:

1. Cognitive lab results show that grade 12 students comprehend the item as intended. 

2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip” 

as shown in the following two items, ETS suggests using the “not on a school trip” version of the 

item to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends not using the item in the 2014 pilot administration.

New

In this school year, have you visited a science museum?

A. Yes

B. No
56

[Grade 12 Science–Student #10]

Cognitive lab results:

1. Overall, students did not exhibit any consistent comprehension problems. 

2. All students understood that the item was asking about field trips for science class.

3. All students stated that “on a school trip” means a trip that is sponsored by the school with school 

supervision. Most students (8 of 10) were able to report an example of a school trip. They 

included:

a. The Medical Museum in Walter Reed, the Natural History Museum, the National 

Aquarium, and the Air and Space Museum.

4. None of the students had visited a museum with their school during the current school year.

Rationale:

1. Cognitive lab results show that grade 12 students comprehend the item as intended. 

2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip” 

as compared to the following item, ETS suggests using the “not on a school trip” version of the 

item to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends not using the item in the 2014 pilot administration.

New

In this school year, have you visited a science museum on a school trip?

A. Yes

B. No
57

[Grade 12 Science–Student #11]

Cognitive lab results:

1. Overall, students did not exhibit any consistent comprehension problems. 

2. A majority of students (9 of 10) were able to identify that the item was asking if they have visited 

a science museum on a trip that was not sponsored by the school.

3. All students stated that “not on a school trip” meant that the visit was not sponsored by the school, 

and done on the students own time.

4. More than half (6 of 10) of students gave examples of museum trips with their families. Some of 

the museums mentioned were:

a. The National Air and Space Museum, and the Smithsonian.

5. A majority of students (9 of 10) reported that they have not visited a museum this year. 

Rationale:

1. Cognitive lab results show that grade 12 students comprehend the item as intended. 

2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip” 

as compared to the previous item, ETS suggests using the “not on a school trip” version of the 

item to measure students’ out-of-school science-related activities. 

Recommendation:

1. ETS recommends adding this item to the 2014 pilot administration.

New

In this school year, have you visited a science museum that was not on a school trip?

A. Yes

B. No
58

Teacher Questionnaire–Grade 4

[Grade 4 Science–Teacher #1]

Cognitive lab results:

1. The teachers shared a common understanding of the item as asking about how students were 

placed in smaller groups to work during class and how this related to students’ learning.

2. All teachers interpreted the phrase “students’ learning preferences” in sub-item b to refer to 

students’ learning style.

3. Teachers interpreted “students’ readiness needs” in sub-item b to mean students’ ability levels 

related to reading or content comprehension, though one teacher noted some confusion with the 

term “students’ readiness needs,” calling it “weird wording,” and wondered if this referred to how 

English learners are grouped with other students.

4. Compared to the original version of the item (i.e., “Do you create groups within this class for 

science instruction on the basis of ability,” solely with “yes/no” options), all five teachers 

responded that they preferred the way the item was currently asked, noting that this would allow 

them to better explain how they teach.

Rationale: 

1. The cognitive lab report shows that all five teachers interpreted the items as intended, and 

compared to the trend version of the item, they preferred the cognitive lab tested version.

2. To address the confusion regarding the wording “learning preferences” and “readiness needs,” and 

further clarify the intention of the two sub-items, ETS recommends revising the two phrases to 

“learning preferences or styles” and “readiness or ability levels” respectively.

Recommendation:

1. Replace the trend version of the item (VC305014) with the cognitive lab tested and further revised 

version.

New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your 

classroom? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Groupings based on students’ interest 

in science/science-related topics A B C D New

b. Groupings based on students’ learning 

preferences or styles A B C D New

c. Groupings based on students’ 

readiness needs or ability levels A B C D New
59

[Grade 4 Science–Teacher #2]

Cognitive lab results:

1. Some phrases, such as “inquiry skills,” “develop skills in lab techniques,” and “scientific writing 

skills,” were difficult to comprehend.

2. One teacher expressed that sub-item f “developing problem-solving (design) skills” is asking two 

different questions at the same time.

3. A majority of teachers (4 of 5) interpreted the phrase “upper grade levels” in sub-item d as 

intended, including any grade above fourth grade and middle school.

4. All teachers referred to school, county, or science standards and curricula to explain what 

knowledge and skills that student would need for studying science in upper grade levels.

5. One teacher felt that the phrase “knowledge and skills” was necessary for clarification.

VC970876

To what extent do you emphasize each of the following objectives in teaching science to your fourthgrade class? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Increase students’ interest in 

science A B C D VC970917

b. Increase awareness of the 

importance of science in daily life A B C D VC970928

c. Learn about applications of science 

to environmental issues A B C D VC970930

d. Teach scientific facts and principles A B C D VC970919

e. Teach scientific methods A B C D VC970920

f. Equip students with the knowledge 

and skills needed for studying 

science in upper grade levels

A B C D Revised sub-item 

based on 

VC970922

g. Develop systematic observation 

skills A B C D VC970929

h. Develop inquiry skills A B C D VC970923

i. Develop skills in lab techniques A B C D VC970926

j. Develop problem-solving (design)

skills A B C D VC970925

k. Develop scientific writing skills A B C D VC970931
60

Rationale: 

1. The difficulties that teachers have with understanding some phrases might result from not 

grouping the sub-items together categorically. If these sub-items were grouped together in a 

logical order, it would potentially clarify the issue. 

2. Indeed, sub-item f “develop problem-solving (design) skills” is confusing in that it implies that 

problem-solving skills equate experimental design skills. Removing the word “design” and the 

parenthesis would clarify the issue. Since experimental design skills should have been included in 

“inquiry skills,” there is no need to develop a sub-item to address it.

Recommendation:

1. Re-order the sub-items (as shown above) and remove the word “design” and the parenthesis from 

sub-item f “develop problem-solving (design) skills.”
61

[Grade 4 Science–Teacher #3]

Cognitive lab results:

1. One teacher found sub-item j and the term “audiovisual” to be vague.

2. There are two occasions where teachers appeared confused, likely due to forgetting what the item 

stem was originally asking. 

3. Teachers differentiated between the “school” and the “school system.” They interpreted “school 

system” as including the school district and the county school office.

4. One teacher found it difficult to choose between options B (little) and C (some) for sub-items c 

(supplies or equipment for science demonstrations) and d (supplies or equipment for science labs).

Rationale: 

1. Cognitive lab results reveal that teachers have a consistent understanding on the revised version of 

the item (see next item) and preferred the response options in the revised version of the item.

Recommendation:

1. Replace the item with the revised version of the item (Teacher, #4).

VC970932

How much of the following instructional materials and other resources does your school system provide 

you with to teach science to your fourth-grade class? Fill in one oval on each line.

None Little Some A lot

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science 

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for classroom use A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments (e.g., 

telescopes, microscopes, thermometers, or 

weighing scales)

A B C D VC970964
62

[Grade 4 Science–Teacher #4]

Cognitive lab results:

1. Teachers differentiated between “school” and “school system” and the role of each entity in 

providing materials. Teachers would respond to the sub-items differently between this item (the 

revised version) and previous item (the original version).

2. All teachers indicated that the options in this item (i.e., “not at all,” “small extent,” “moderate 

extent,” “large extent”) appropriate and sufficient for describing the resources available, and had a

similar interpretation of each option.

3. Most teachers implied that they preferred this item (the revised version) compared to the previous 

item (the original item).

Rationale: 

1. Cognitive lab results show that this item (the revised version) performed better than the previous 

item (the original item).

2. To address the potential of losing information regarding resources provided by the county and 

school districts caused by revising the item stem from “school system” to “school,” ETS 

recommends revising the item stem from “school” to “school system (including your school, 

county office, and school district).”

Recommendation: 

1. Replace the original item (VC970932) with this item (the revised version).

New (Revised based on VC970932)

To what extent does your school system (including your school, county office, and school district)

provide the following to you? Fill in one oval on each line.

Not at all

Small 

extent

Moderate 

extent

Large 

extent

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science 

demonstrations A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for classroom 

use

A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments (e.g., 

telescopes, microscopes, thermometers, or 

weighing scales)

A B C D VC970964
63

[Grade 4 Science–Teacher #5]

Cognitive lab results:

1. All teachers understood the item was asking about different instructional methods they use with 

their students.

2. All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to 

whether students could learn at a faster or slower pace based on how quickly they understood the 

material or lesson. 

3. All teachers stated that they would have answered sub-item c “vary the pace of instruction for 

some students” the same way if it had instead read “pace my teaching differently for some 

students.” One teacher, however, said she would prefer the question asked in the alternate 

wording.

Rationale: 

1. Cognitive lab results show that teachers consider the alternate wording of the sub-item c as asking 

the same question, slightly preferring the alternate wording to (i.e., the original wording “pace my 

teaching differently for some students.”

2. ETS recommends replacing the original wording for sub-item c with this one (the revised 

wording) to be consistent with grade 8 questionnaire.

Recommendation: 

1. Replace the original wording of the sub-item c with this one (the revised wording).

VC767811

When you teach science to your fourth-grade class, do you do any of the following? Fill in one oval on 

each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Use a different set of methods 

in teaching some students A B C D VC767814

b. Supplement the regular course 

curriculum with additional 

material for some students

A B C D VC767820

c. Vary the pace of instruction for 

some students A B C D Revised sub-tem based on 

VC767821

d. Have some students engage in 

different classroom activities A B C D VC767823

e. Set different achievement 

standards for some students A B C D VC767824
64

[Grade 4 Science–Teacher #6]

Cognitive lab results:

1. Two teachers interpreted the item as asking what outside opportunities teachers are providing, 

while others indicated that it asked what the teachers or the school provided.

2. All teachers answered the questions in regards to activities or opportunities that the school 

organized, rather than what the teacher personally provided.

3. Teachers made a distinction between “science fair” and “science competition.”

4. All teachers felt the response options provided were appropriate, although one teacher commented 

that it would be better to ask teachers to quantify the frequency of events per school year. When 

asked the follow-up probe of whether it would better to use frequencies such as certain times a 

week, a semester, or a school year, a majority of teachers (4 of 5) said this would be better. 

Rationale: 

1. To address a majority of teachers’ (4 of 5) preference on frequency options, and to be consistent 

with the school questionnaires, ETS recommends revising the options to “1–3 time per year,” “4–7 

times per year,” and “8 or more times per year.”

2. ETS recommends revising the wording of sub-item c to be consistent with school questionnaire.

Recommendation: 

1. Add this item to the 2014 pilot questionnaire.

New

To what extent do you provide students with the following learning experiences? Fill in one oval on each 

line.

Not at all

Small 

Extent

1–3 times 

per year

Moderate

Extent

4–7 times 

per year

Large 

Extent

8 or 

more 

times 

per year

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums

(including traditional museums 

and other similar sites)

A B C D New
65

Teacher Questionnaire–Grade 8

[Grade 8 Science–Teacher #1]

Cognitive lab results:

1. All teachers interpreted the term “student groupings” to mean separating the students into work 

groups to collaborate on assignments.

2. Three of the five teachers interpreted the phrase “students’ learning preferences” as referring to 

learning style, while the remaining two teachers interpreted the phrase as referring generally to 

interests and preferences.

3. Three of the five teachers expressed confusion regarding the phrase ‘students’ readiness needs” in 

sub-item c. One teacher noted that "students' readiness needs" could mean “the desire and 

willingness to learn.”

4. Four of the five teachers stated that they use all three grouping methods in their teaching day.

5. All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” 

and “large extent,” are appropriate for describing the extent of instructional grouping strategies.

Rationale: 

1. The cognitive lab report shows that all five teachers interpreted the items as intended. Three 

teachers further indicated that it is necessary to ask about various grouping methods.

2. To address the confusion regarding the wording “learning preferences” and “readiness needs,” and 

further clarify the intention of the two sub-items, ETS recommends revising the two phrases to 

“learning preferences or styles” and “readiness or ability levels” respectively.

Recommendation:

1. Replace the trend version of the item (VC305014) with this cognitive lab tested and further 

revised version.

New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your 

classroom? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Groupings based on students’ interest 

in science/science-related topics A B C D New

b. Groupings based on students’ learning 

preferences or styles A B C D New

c. Groupings based on students’ 

readiness needs or ability levels A B C D New
66

[Grade 8 Science–Teacher #2]

Cognitive lab results:

1. A majority teachers (4 of 5) stated that the item asks about how much they focus on different areas 

and topics when teacher science. 

2. All teachers interpreted the phrase “upper grade levels” as high school, and one of them included 

honors courses, AP courses, and college. 

3. All teachers explained the phrase “knowledge and skills needed for study science in upper grade 

levels” as the principles and techniques that students need to be successful in high school science.

4. A majority of teachers (4 of 5) referred to state curriculum standards to determine what knowledge 

and skills will be needed beyond 8th grade.

5. Most teachers (3 of 5) reported that their answer would change slightly but remain similar if the 

item had been phrased “prepare students for further study in science."

Rationale: 

1. The cognitive lab report on this matrix item did not reveal any difficulty among teachers with the 

wording of the item.

2. Since issues have been identified in grade 4 version of the item and recommended revision has 

been made, ETS recommends revising the item to be consistent with the grade 4 version.

VC976013

To what extent do you emphasize each of the following objectives in teaching science to your eighthgrade class? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Increase students’ interest in 

science A B C D VC976015

b. Increase awareness of the 

importance of science in daily life A B C D VC976023

c. Learn about applications of science 

to environmental issues A B C D VC970926

d. Teach scientific facts and principles A B C D VC976017

e. Teach scientific methods A B C D VC976018

f. Equip students with the knowledge 

and skills needed for studying 

science in upper grade levels

A B C D Revised sub-item 

based on VC976019

g. Develop systematic observation 

skills A B C D VC97625

h. Develop inquiry skills A B C D VC976020

i. Develop skills in lab techniques A B C D VC976022

j. Develop problem-solving (design) 

skills A B C D VC976021

k. Develop scientific writing skills A B C D VC970927
67

Recommendation:

1. Re-order the sub-items (as shown above) and remove the word “design” and the parenthesis from 

sub-item f “develop problem-solving (design) skills.”
68

[Grade 8 Science–Teacher #3]

Cognitive lab results:

1. The teachers displayed inconsistent interpretations of the item. Two interpreted the item as asking 

about the resources they have available; three interpreted the item as asking about what resources 

were provided by their school system, including county or school district.

Rationale: 

1. Cognitive lab results reveal that grade 8 teachers do not have a preference between the current

revised versions of the item stem.

2. A majority of teachers (4 of 5) indicated that the response options in the revised version of the 

item (i.e., “not at all,” “small,” “moderate extent,” “large extent”) are appropriate and sufficient 

for describing the extent to which resources are provided by the school.

Recommendation: 

1. Replace the item with the revised version of the item to be consistent with grade 4 teacher 

questionnaire.

VC976028

How much of the following instructional materials and other resources does your school system provide 

you with to teach science to your eighth-grade class? Fill in one oval on each line.

None Little Some A lot

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science 

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for classroom use A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments (e.g., 

telescopes, microscopes, thermometers, or 

weighing scales)

A B C D VC976045
69

[Grade 8 Science–Teacher #4]

Cognitive lab results:

1. Teachers differentiated between “school” and “school system” and the role of each entity in 

providing materials. 

2. Three teachers reported that the way this item is phrased (the revised version) and the previous 

item (the original version) would not make a difference in how they would answer any of the subitems.

3. A majority of teachers (4 of 5) stated that the options in this item (i.e., “not at all,” “small extent,”

“moderate extent,” “large extent”) are appropriate and sufficient for describing the extent to which 

resources are provided by the school.

4. All teachers shared a consistent understanding of the options and implied that they prefer these 

options compared to the ones in the previous item (the original item). 

Rationale: 

1. Cognitive lab results show that this item (the revised version) performed better than the previous 

item (the original item).

2. To address the potential loss of information regarding resources provided by county and school 

district caused by revising item from “school system” to “school,” ETS recommends revising the

item stem from “school” to “school system (including your school, county office, and school 

district).”

Recommendation: 

1. Replace the original item with this item (the revised version).

New (Revised based on VC976028)

To what extent does your school system (including your school, county office, and school district)

provide the following to you? Fill in one oval on each line.

Not at all

Small 

extent

Moderate 

extent

Large 

extent

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science 

demonstrations A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for classroom use A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments (e.g., 

telescopes, microscopes, thermometers, or 

weighing scales)

A B C D VC976045
70

[Grade 8 Science–Teacher #5]

Cognitive lab results:

1. A majority of teachers (4 of 5) stated that the question is asking about different instructional 

methods, and how they plan and change lesson presentation methods to accommodate students 

with varying levels of understanding.

2. A majority of teachers (4 of 5) stated that the phrase “vary the pace of instruction for some 

students” meant providing students with different amounts of time to complete tasks, based on 

their different performance levels.

3. Three teachers stated that they would have answered sub-item c “vary the pace of instruction for 

some students” the same way if it had instead said “pace my teaching differently for some 

students.” Two stated that their answers would be different because they interpreted the first 

version to be more student-specific, instead of referring to their overall teaching approach.

4. All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some 

students.”

Rationale: 

1. Two teachers stated that their answer on sub-item c would be different if it had instead asking 

“pace my teaching differently for some students.”

2. All teachers stated that they prefer the current phrasing, “vary the pace of instruction for some 

students.”

Recommendation: 

1. Replace the original wording of the sub-item c with this one (the revised wording).

VC976085

When you teach science to your eighth-grade class, do you do any of the following? Fill in one oval on 

each line. 

Not at all Small 

extent

Moderate 

extent

Large 

extent

a. Use a different set of methods in 

teaching some students A B C D VC976086

b. Supplement the regular course 

curriculum with additional 

material for some students

A B C D VC976088

c. Vary the pace of instruction for 

some students A B C D Revised sub-item 

based on VC976091

d. Have some students engage in 

different classroom activities A B C D VC976092

e. Set different achievement 

standards for some students A B C D VC976094
71

[Grade 8 Science–Teacher #6]

Cognitive lab results:

1. A majority of teachers (4 of 5) stated that the item asks about science-related activities outside of 

the classroom.

2. Three teachers made a distinction between “science fair” and “science competition,” while the 

other two did not.

3. Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” 

and “large extent,” were sufficient to describe the extent of the learning experience provided to the 

students. 

4. One thought the choices were sufficient but felt that the term “science museum” was too limiting 

and suggested to add science field trips.

5. In regards to whether frequencies, such as certain times a week/a semester/a school year, would be 

better than the response options provided for this item, opinions were divided. While two teachers 

rejected the idea that frequencies would be more relevant response options, two noted that they 

would be preferred. One respondent specifically suggested the following options: per quarter, per 

semester, or per year.

Rationale: 

1. To address some teachers’ preference on frequency options and to be consistent with grade 4 

questionnaires, ETS recommends revising the options to “1–3 times per year,” “4–7 times per 

year,” and “8 or more times per year.” 

2. ETS recommends revising the wording of sub-item c to be consistent with school questionnaire.

Recommendation: 

1. Add this item to the 2014 pilot questionnaire.

New

To what extent do you provide students with the following learning experiences? Fill in one oval on each 

line.

Not at all

Small 

Extent

1–3 times 

per year

Moderate

Extent

4–7 times 

per year

Large 

Extent

8 or 

more 

times 

per year

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums

(including traditional museums 

and other similar sites)

A B C D New
72

School, Grade 4:

[Grade 4 Science–School #1]

Cognitive lab results:

1. School administrators were not unified in their understanding of what the role and 

responsibilities would be for a “science specialist or coach.”

2. Although a majority of administrators (4 of 5) indicated that they did not have a part-time 

or full-time science specialist or coach in their school, they emphasized that the 

responsibilities they associated with the role were taken on by classroom teachers and/or 

other types of specialists (e.g., instructional support teacher). 

3. All administrators felt that the sub-items in the following matrix item covered the 

majority of the responsibilities they would consider part of the role of a science specialist 

or coach.

Rationale: 

1. ETS posits that the intention of the item should be whether or not there are personnel 

available in school performing the function of a “science specialist or coach,” not 

whether the particular title of “science specialist or coach” is available in the school. 

Since all administrators felt that the sub-items in the following matrix item (item 2) 

covered the majority of the responsibilities of science personnel, the content of the subitems provide a better context for administrators to comprehend the question than 

providing an abstract definition of “science specialist or coach.” Therefore, ETS 

recommends collecting the information through revising the following item in which the 

majority of the responsibilities of a “science specialist or coach” have been explained in 

more detail.

2. It appears to be difficult for some school administrators to choose between the options of 

“full-time” and “part-time” since the role may be filled by school, district, or county staff; 

or by someone who is not a school employee. Therefore, in the revised version of the 

following item, the availability of a “science specialist or coach” will not differentiate 

between “part-time” and “full-time.”

Recommendation:

1. Remove item from the 2014 pilot administration.

New (Revised based on VE013369)

Is there a science specialist or coach available (full- or part-time) to fourth-grade 

teachers at your school? 

A. Yes, available full-time at my school.

B. Yes, available part-time at my school.

C. No 
73

[Grade 4 Science–School #2]

Cognitive lab results:

1. Although a majority of administrators (4 of 5) indicated that they did not have a part-time 

or full-time science specialist or coach in their school, they emphasized that the 

responsibilities they associated with the role were taken on by classroom teachers and/or 

other types of specialists (e.g., instructional support teacher). 

2. All administrators felt that the sub-items covered the majority of the responsibilities they 

would consider part of the role of a science specialist or coach.

New (Revised stem based on VE013372)

To what extent are any of the following a responsibility of the science specialist or coach, or other 

instructional support personnel, available to fourth-grade science instruction teachers at your school? 

Fill in one oval on each line. 

Not available 

at my school

Not 

at all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical 

assistance/support to 

individual teachers about 

science content or the 

teaching of science

A B C C E VE013373

b. Conduct professional 

development for groups of 

teachers about science 

content or the teaching of 

science

A B C C E VE013382

c. Provide science instruction 

to students on various topics A B C C E VE013383

d. Interpret and incorporate

Translate science standards

and current research into 

classroom practice

A B C C E Revised 

sub-item 

based on 

VE013384

e. Provide science enrichment 

to some student groups A B C C E VE013385
74

Rationale:

1. ETS posits that the intention of the item should be whether or not there are personnel 

available for fourth-grade science instruction functioning as a “science specialist or 

coach”, not whether the particular title of “science specialist or coach” exists in school. 

ETS recommends adding “other instructional support personnel” to the item stem. Since 

all administrators felt that the sub-items covered the majority of the responsibility of a 

“science specialist or coach” these sub-items would provide administrators better context 

to comprehend the question than providing abstract definitions of “science specialist or 

coach.” Therefore, ETS recommends combining the matrix item with the previous skip 

pattern item and adding “Not available at my school” to the option list to capture the 

information originally collected through the previous skip pattern item (VE013369).

2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS 

recommends revising the word “translate” to the phrase “interpret and incorporate” to 

clearly reflect the kind of service and support that a “science specialist or coach” 

provides.

Recommendation: 

1. Replace the trend item (VE013372) with this revised version of the item in the 2014 pilot 

administration.
75

[Grade 4 Science–School #3]

Cognitive lab results:

1. The definition for “science fair” and “science competition” differed among 

administrators.

2. A majority of administrators (4 of 5) preferred the response options of “large extent,”

“moderate extent,” and “small extent,” versus frequencies (i.e., certain times a 

week/semester/school years). 

Rationale:

1. The Science Subject-Area Standing Committee (06/19/2012) has suggested developing 

more items asking about students’ science activities outside of school. The Assessment 

Development Committee (ADC) (07/05/2012) specifically suggested that out-of-school 

activities could include science fairs, national competitions, robotics competitions, etc. 

To address these suggestions, ETS developed relevant items for all student, teacher, and 

school administrator questionnaires and tested these items in cognitive laboratories.

2. Since cognitive lab results revealed that school administrators differentiate between the 

terms “science fair” and “science competition,” ETS recommends keeping them as two 

separate sub-items.

3. To be consistent with grade 12 questionnaire, the phrase “in the last school year” and 

frequency ranges have been added to the item stem and response options to further 

quantify the answers.

Recommendation: 

1. Add to the 2014 pilot administration

New

In the last school year, to what extent did does your school provide students with the following 

opportunities? Fill in one oval on each line.

Not at all

Small 

extent

(1–3 

times)

Moderate 

extent (4–

7 times)

Large 

extent

(8 times 

or 

more)

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums

(including traditional museums 

and other similar sites)

A B C D New
76

School, Grade 8:

[Grade 8 Science–School #1]

Cognitive lab results:

1. Three of the five administrators indicated that they did not have a part-time or full-time 

science specialist or coach in their school, and also emphasized the responsibilities they 

associated with the role taken on by classroom teachers and/or other types of specialists 

such as “instructional lead teacher,” “instructional coach,” or “mentor teacher.” 

2. Three of the five administrators agreed that the sub-items in the following matrix item 

covered all main responsibilities of a science specialist or coach. 

Rationale: 

1. ETS posits that the intention of the item should be whether there are personnel within 

school performing the function of a “science specialist or coach”, not whether personnel 

exists within the school with the specific title of “science specialist or coach”. Since most 

administrators felt that the sub-items in the following matrix item covered the majority of 

the responsibility of the personnel, the content of the sub-items would provide better 

context for administrators to comprehend the question than providing abstract definitions 

of “science specialist or coach”. Therefore, ETS recommends collecting the information 

through revising the following item in which the majority of the responsibilities of a 

“science specialist or coach” have been provided in detail.

2. It appears to be difficult for some school administrators to choose between “full-time” 

and “part-time” even though there are personnel available fulfilling the role of a “science 

specialist or coach”, because not all personnel who function as a “science specialist or 

coach” are from the school. They could be also from the school district or the county (not 

a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the 

revised version of the following item, the availability of a “science specialist or coach” 

will not be differentiated between “part-time” and “full-time”.

Recommendation:

1. Remove the item from the 2014 pilot administration

New (Revised based on VE013964)

Is there a science specialist or coach available (full- or part-time) to eighth-grade 

teachers at your school? 

A. Yes, available full-time at my school.

B. Yes, available part-time at my school.

C. No 
77

[Grade 8 Science–School #2]

Cognitive lab results:

1. The use of the word “translate” in sub-item “d” is confusing to one administrator.

2. Although two administrators indicated that they did not have a part-time or full-time 

science specialist or coach in their school, they were able to answer the question based on 

other available personnel performing the function of a “science specialist or coach.” 

3. Most (3 of 5) administrators felt that the sub-items covered the majority of the 

responsibilities they would consider part of the role of a science specialist or coach.

4. Two of the five administrators agreed that the four answer choices are sufficient for 

describing the extent of a science specialist or coach’s responsibilities in their school.

New (Revised stem based on VE013966)

To what extent are any of the following a responsibility of the science specialist or coach, or other 

instructional support personnel available to eighth-grade science instruction teachers at your school? 

Fill in one oval on each line. 

Not available 

at my school

Not 

at all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical 

assistance/support to 

individual teachers about 

science content or the 

teaching of science

A B C C E VE013967

b. Conduct professional 

development for groups of 

teachers about science 

content or the teaching of 

science

A B C C E VE013968

c. Provide science instruction 

to students on various topics A B C C E VE013969

d. Interpret and incorporate

Translate science standards 

and current research into 

classroom practice

A B C C E

Revised 

sub-item 

based on 

VE013970

e. Provide science enrichment 

to some student groups A B C C E VE013971
78

Rationale:

1. ETS posits that the intention of the item should be whether there are personnel available 

for fourth-grade science instruction functioning as a “science specialist or coach”, not 

whether the particular title of “science specialist or coach” exists in school. ETS 

recommends adding “other instructional support personnel” to the item stem. Since all 

administrators felt that the sub-items covered the majority of the responsibility of a 

“science specialist or coach”, these sub-items would provide administrators better context 

to comprehend the question than providing abstract definitions of “science specialist or 

coach”. Therefore, ETS recommends combining the matrix item with the previous skip 

pattern item and adding “Not available at my school” to the option list to capture the 

information originally collected through the previous skip pattern item (VE013964).

2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS 

recommends revising the word “translate” to the phrase “interpret and incorporate” to 

clearly reflect the kind of service that a “science specialist or coach” provides.

Recommendation:

1. Replace the trend item (VE013966) with this revised version of the item in the 2014 pilot 

administration.
79

[Grade 8 Science–School #3]

Cognitive lab results:

1. All administrators made a clear distinction between “science fair” and “science 

competition.”

2. A majority of administrators (4 of 5) agreed that the response options of “not at all,”

“large extent,” “moderate extent,” and “small extent,” are sufficient for describing the 

extent of the opportunities their school provides. 

Rationale:

1. Standing Committee (06/19/2012) has suggested developing more items asking about 

students’ science activities outside of school. ADC (07/05/2012) specifically suggested 

out-of-school activities could include science fairs, national competitions, robotics 

competitions, etc. To address these suggestions, ETS developed relevant items for all 

student, teacher, and school administrator questionnaires and tested these items in 

cognitive lab.

2. Cognitive lab results reveals that school administrators differentiate “science fair” and 

“science competition”, ETS recommends keeping them as two separate sub-items.

3. To be consistent with grade 12 questionnaire, phrase “in the last school year” and range 

of the option “small extent,” “moderate extent,” and “large extent” haven been added to 

the item stem and options to further quantify the answers.

Recommendation: 

1. Add to the 2014 pilot administration.

New

In the last school year, to what extent did does your school provide students with the following 

opportunities? Fill in one oval on each line.

Not at all

Small 

extent

(1–3 

times)

Moderate 

extent (4–

7 times)

Large 

extent

(8 times 

or 

more)

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums

(including traditional museums 

and other similar sites)

A B C D New
80

School, Grade 12:

[Grade 12 Science–School #1]

Cognitive lab results:

1. Two of the five administrators stated that although they did have an individual fulfilling 

the role of a “science specialist or coach,” it is difficult to determine whether or not they 

are full- or part-time. One administrator further explained that none of the response 

options fit his school’s situation because the resource teacher is responsible for all high 

schools in the county and the 12th grade teachers in his school rarely ask for support from 

this individual.

2. A variety different terms were often used in schools for describing individuals with the 

responsibilities of a “science specialist or coach,” such as “resource teacher,” “science 

department chairperson,” and “science coordinator.”

3. Overall, administrators felt that the response option did not fit their preferences.

4. All administrators felt the sub-items in the following matrix item covered the majority of 

responsibilities they would consider as parts of the role of “science specialist or coach.”

Rationale: 

1. ETS posits that the intention of the item should be whether there is personnel available in 

school performing the function of “science specialist or coach”, not whether the 

particular title of “science specialist or coach” available in school. Since most 

administrators felt that the sub-items in the following matrix item covered the majority of 

the responsibility of the personnel, the content of the sub-items would provide better 

context for administrators to comprehend the question than providing abstract definitions 

of “science specialist or coach”. Therefore, ETS recommends collecting the information 

through revising the following item in which the majority of the responsibilities of a 

“science specialist or coach” have been provided in detail.

2. It appears to be difficult for some school administrators to choose between “full-time” 

and “part-time” even though there are personnel available fulfilling the role of a “science 

specialist or coach”, because not all personnel who function as a “science specialist or 

coach” are from the school. They could be also from the school district or the county (not 

a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the 

New (Revised based on VE015623)

Is there a science specialist or coach available (full- or part-time) to twelfth-grade 

teachers at your school? 

A. Yes, available full-time at my school.

B. Yes, available part-time at my school.

C. No 
81

revised version of the following item, the availability of a “science specialist or coach” 

will not be differentiated between “part-time” and “full-time”.

Recommendation:

1. Remove the item from the 2014 pilot administration
82

[Grade 12 Science–School #2]

Cognitive lab results:

1. Overall, the administrators share a common understanding of the term “science specialist 

or coach.”

2. All administrators felt that the sub-items covered the majority of the responsibilities they 

would consider part of the role of a “science specialist or coach.”

Rationale for recommendation:

1. ETS posits that the intention of the item should be whether there are personnel available 

for fourth-grade science instruction functioning as a “science specialist or coach”, not 

whether the particular title of “science specialist or coach” exists in school. ETS 

recommends adding “other instructional support personnel” to the item stem. Since all 

New (Revised stem based on VE015624)

To what extent are any of the following a responsibility of the science specialist or coach, or other 

instructional support personnel available to twelfth-grade science instruction teachers at your school? 

Fill in one oval on each line. 

Not available 

at my school

Not 

at all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical 

assistance/support to 

individual teachers about 

science content or the 

teaching of science

A B C C E VE015625

b. Conduct professional 

development for groups of 

teachers about science 

content or the teaching of 

science

A B C C E VE015626

c. Provide science instruction 

to students on various topics A B C C E VE015627

d. Interpret and incorporate

Translate science standards 

and current research into 

classroom practice

A B C C E

Revised 

sub-items 

based on 

VE015628

e. Provide science enrichment 

to some student groups A B C C E VE015629
83

administrators felt that the sub-items covered the majority of the responsibility of a 

“science specialist or coach”, these sub-items would provide administrators better context 

to comprehend the question than providing abstract definitions of “science specialist or 

coach”. Therefore, ETS recommends combining the matrix item with the previous skip 

pattern item and adding “Not available at my school” to the option list to capture the 

information originally collected through the previous skip pattern item (VE015623).

2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS 

recommends revising the word to “interpret and incorporate” to clearly reflect the kind of 

service that a “science specialist or coach” provides.

Recommendation:

1. Replace the trend item (VE015624) with this revised version of the item in the 2014 pilot 

administration.
84

[Grade 12 Science–School #3]

Cognitive lab results:

1. All administrators interpreted the item as asking about science-related opportunities that 

the school provides beyond traditional classroom instruction.

2. Three of the five administrators agreed that there is a distinction between a “science fair” 

and a “science competition.”

3. It appears that grade 12 administrators further differentiate “science museums,” including 

traditional museums, such as a natural history museums, and locations like aquariums or

other similar sites.

4. Most administrators (3 of 5) expressed a preference for having response options with 

frequencies, such as certain times a week/a semester/a school year. One administrator 

suggested adding ranges to quantify what was meant by a small, moderate and large 

extent. One administrator indicated that the best option would be to base answers on a 

school year.

Rationale:

1. Standing Committee (06/19/2012) has suggested developing more items asking about 

students’ science activities outside of school. ADC (07/05/2012) specifically suggested 

out-of-school activities could include science fairs, national competitions, robotics 

competitions, etc. To address these suggestions, ETS developed relevant items for all 

student, teacher, and school administrator questionnaires and tested these items in 

cognitive lab.

2. Cognitive lab results reveals that school administrators differentiate “science fair” and 

“science competition”, ETS recommends keeping them as two separate sub-items.

New

In the last school year, to what extent did does your school provide students with the following 

opportunities? Fill in one oval on each line.

Not at all

Small 

extent

(1–3 

times)

Moderate 

extent (4–

7 times)

Large 

extent

(8 times 

or 

more)

e. Science fairs A B C D New

f. Science clubs A B C D New

g. Science competitions A B C D New

h. Trips to science museums

(including traditional museums 

and other similar sites)

A B C D New
85

3. Phrase “in the last school year” and range of the option “small extent”, “moderate 

extent”, and “large extent” haven been added to the item stem and options to further 

quantify the answers.

Recommendation: 

1. Add to the 2014 pilot administration.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

86

Cognitive Laboratories Report and Findings

(Eureka Facts)
Cognitive Interviews of Background Questionnaire for

NAEP 2012

Science Assessment 

Final Report

January 22, 2013
Contents

1. Introduction......................................................................................................................... 1

1.1 Sample .........................................................................................................................1

1.2 Recruitment and Interview Procedures .........................................................................3

2. Cognitive Interview Results................................................................................................. 4

2.1 Student Background Questionnaire ..............................................................................4

2.1.1. Grade 4 Students.................................................................................................. 4

2.1.2. Grade 8 Students.................................................................................................24

2.1.3. Grade 12 Students...............................................................................................41

2.2 Teacher Background Questionnaire............................................................................58

2.2.1. Grade 4 Teachers................................................................................................58

2.2.2. Grade 8 Teachers................................................................................................68

2.3 Administrator Background Questionnaire....................................................................77

2.3.1. Grade 4 Administrators ........................................................................................77

2.3.2. Grade 8 Administrators ........................................................................................82

2.3.3. Grade 12 Administrators ......................................................................................87
Prepared by EurekaFacts, LLC 

1

Cognitive Interviews Report for BQ CogLab

Date of Report: January 22, 2013 

Interview Dates: 11/27/2012 – 01/04/2013

1. Introduction

1.1 Sample

Cognitive interviews designed to test survey items developed for the National Assessment of Educational 

Progress (NAEP) Science Assessment were conducted with 15 school administrators, 10 teachers, and 30 

students. Each group of respondents was further divided into sub-groups based on grade level: school 

administrators for fourth, eighth, and twelfth-grade students; teachers of fourth and eighth-grade students; 

and students in fourth, eighth, and twelfth grades.

The administrators, teachers and students who participated in the cognitive interview study were recruited 

from the greater Washington D.C., and Baltimore, Maryland metropolitan areas.

Administrator Sample

Five school administrators of fourth-grade students, five school administrators of eighth-grade students, and 

five school administrators of twelfth-grade students participated in the cognitive interviews. Demographics of 

participating administrators and their schools are presented in Table 1.

Table 1: Participant and School Demographics for Administrator Respondent Group

Respondent 

Group

Participant Demographics School Demographics

Total

Gender Race/Ethnicity SES School Size

4

th grade 

Administrators

Female: 5 White: 2

Black: 2

Low: 3

High: 2

Large: 3

Small: 2

5

8

th grade 

Administrators

Male: 3

Female: 2

White: 2

Black: 1

Other: 2

Low: 2

High: 3

Large: 3

Medium: 2 5

12th grade 

Administrators

Male: 3

Female: 2

White: 3

Black: 2

Low: 2

High: 3 Large: 5 5

Total Male: 6

Female: 9

White: 9

Black: 5

Other: 1

Low: 7

High: 8

Large: 11

Medium: 2

Small: 2

15
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 2

Teacher Sample

Five fourth-grade teachers and five eighth-grade teachers who teach science during the current school year 

participated in the cognitive interviews. Demographics of participating teachers and their schools are 

presented in Table 2.

Table 2: Participant and School Demographics for Teacher Respondent Group

Student Sample

Ten fourth-grade students, ten eighth-grade students, and ten twelfth-grade students participated in cognitive 

interviews. The demographics of participating students are presented in Table 3.

Table 3: Participant Demographics for Student Respondent Group

Respondent Group SES Gender Race/Ethnicity Total

4

th grade Students

Low: 4

High: 6

Male: 4

Female: 6

White: 5

Hispanic: 2

Black: 3

10

8

th grade Students

Low: 5

High: 5

Male: 4

Female: 6

White: 4

Hispanic: 2

Black: 4

10

12th grade Students

Low: 5

High: 5

Male: 7

Female: 3

White: 8

Black: 1

Asian: 1

10

Total Low: 14

High: 16

Male: 15

Female: 15

White: 17

Black: 8

Hispanic: 4

Asian: 1

30

Respondent Group Participant Demographics School Demographics Total

Gender Race/Ethnicity SES School Size

4

th grade Teachers Male: 2

Female: 3

White: 4

Other: 1

Low: 4

High: 1

Large: 2

Medium: 2

Small: 1

5

8

th grade Teachers Female: 5 White: 3

Black: 2

Low: 2

High: 3

Large: 1

Medium: 2

Small: 2

5

Total Male: 2

Female: 8

White: 7

Black: 2

Other: 1

Low: 6

High: 4

Large: 3

Medium: 4

Small: 3

10
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 3

1.2 Recruitment and Interview Procedures

Cognitive interviews began on November 27, 2012 and were completed on January 4, 2013. 

EurekaFacts recruited participants through email and telephone outreach. Trained interviewers screened 

candidates who responded to the outreach using the screening questionnaire approved by OMB (Appendix 

A). Parents of student candidates were informed that they would be compensated with two $25 gift cards, 

one for the student participant and one for the parent of the student. Administrators and teachers were 

informed that they would be compensated with a $40 gift card.

The cognitive interview process in this study followed the OMB approved script and interview protocol as 

presented in Appendices A and B. Written consent was obtained from participants or the parents/legal 

guardians of participants before interviews were conducted. Participants were informed about the purpose of 

the study and reassured that their participation was voluntary and that their answers would be used only for 

research purposes and would not be disclosed, or used, in identifiable form for any other purpose except as 

required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. 

Interview sessions lasted for one hour. During the sessions, interviewers explained the think-aloud process, 

conducted a think aloud practice question exercise, and then asked a series of verbal questions. 

Interviewers used several different cognitive interviewing techniques, including general think-aloud, questionspecific probes, and interviewers’ observations. 

At the end of the session, participants were briefly interviewed about their overall impressions of the survey 

items. Participants were thanked, remunerated, and asked to sign a receipt for their incentive payment.

The interview procedure for the fourth-grade student group included pairing of a sub-set of the survey items

to reduce possible order effect. Items 2, 3, 4, 5, and 6 were presented in paired order, following the design 

presented in Table 4. Based on this paired order, six versions of questionnaire were developed; these 

versions were assigned to participants from the fourth-grade student group according to a predefined order.

Table 4: Paired Design for Fourth-grade Student Questionnaire

Questionnaire

version Question Order Participants

1 2 3 4 5 6 1

st and 7th

2 2 3 4 6 5 2

nd and 8th

3 2 3 5 4 6 3

rd

4 3 2 5 6 4 4

th and 9th

5 3 2 6 4 5 5

th and 10th

6 3 2 6 5 4 6

th
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 4

2. Cognitive Interview Results 

2.1 Student Background Questionnaire

2.1.1. Grade 4 Students

Grade 4 Science–Student—Item 1

Reading

Interviewer notes report the following observations:

 Three students had trouble with the word “circuits:” two mispronounced the word, while the third took an 

unusually long pause at first, and then read the word properly.

Comprehension

 Two students understood the item as asking about projects both at school and at home.

 When asked to define “activities or projects,” eight of the ten students described this as “things” or “stuff” 

to do. Three of these eight further described them as “fun,” while two described them as “work.” One 

student only provided examples of activities; one student could not define the phrase.

 When asked what “circuits and energy” meant to them, three students said that this meant “light” or 

“power,” while two could not define the phrase. Four students did not define the phrase, but rather 

provided an example (e.g., “wires”).

Response Process

 Four students reported that they would have responded to the item differently if the term “batteries and 

light” were used in place of “circuits and energy.” Two students explicitly said they would change 

responses because “batteries and light” is clearer to them, and would have changed their responses 

from “rarely” to “sometimes.”

Recommendations for Improvement

 Although only two students reported not understanding the term “circuits and energy,” nearly half (4 of 

10) would have changed their response if the more concrete example “batteries and light” were used. In 

addition, there was much variation in how students understood the phrase “circuits and energy.” Thus, 

New (Revised based on VC519362)

In this school year, how often have you done activities or projects to learn about electricity (for 

example, circuits and energy)? 

A Never

B Rarely

C Sometimes

D Often
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 5

we suggest that the tested alternative phrase, “batteries and light,” would be a more relevant example in 

this item. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 6

Grade 4 Science–Student—Item 2

Response Process

 When asked how they would respond to this item if they had used chemicals only at certain times during 

the school year, students provided a variety of responses. Three students would have chosen “never or 

hardly ever,” two would have chosen “about once a week,” and two would have chosen “two or three 

times a week.” The remaining response options (“once every few weeks” and “every day or almost every 

day”) would have been chosen by one student each. One student did not understand the probe.

 Only three students reported the time when they had carried out any activities, providing such time 

frames as “10 weeks into the year,” “more towards the beginning [of the year],” and “2 weeks ago.” Only 

two students provided specific examples of the activities, such as: 

o Watching videos in class about what happens when you touch or drink unknown chemicals.

o Looking at and smelling different liquids.

 More than half of students (6 of 10) preferred the response options in item two to those of item three.

Three students stated that they liked item two because the choices were more concrete than those of 

item three. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315219

In this school year, how often have you done activities or projects to learn about chemicals (for 

example, mixing sugar or salt in water)?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 7

Grade 4 Science–Student—Item 3

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item.

 When asked to define “rarely,” “sometimes,” and “often,” the students used different approaches:

o Four students defined the answer choices using subjective, vague terminology: rarely means 

“hardly ever,” sometimes means “once in a while,” and often means “a lot” or “all the time.”

o Two students defined the response options in terms of comparisons to the other categories: 

"rarely” means between “often” and “never;” “sometimes” is very similar to “rarely;” “often means 

more than sometimes, and is between sometimes and always.”

o Two students defined response options in terms of frequencies: “sometimes” as once or twice a 

week; “often” as either every day or every other day. 

o Two students used a mix of approaches described above.

Response Process

 When asked how they would respond if they had used chemicals only at certain times during the school 

year, three students chose “rarely,” three chose “sometimes,” and three chose “often.” The remaining 

student misunderstood the probe.

 Four students preferred the response options in item three to those in item two. One student said that 

these options were easier to read; another said they were “less to read” and therefore made more 

sense.

Recommendations for Improvement

 Six of the ten students preferred the response options in item two to those in item three. Only four 

students preferred the response options provided in item three. It appears that the students had difficulty 

differentiating between “rarely,” “sometimes,” and “often,” and also with matching their internally 

generated answers to the more abstract response options in item three. Based on these difficulties with 

the response options in item three, we suggest that the response options used in item two are more 

appropriate for fourth grade students.

New (Revised based on VC315219)

In this school year, how often have you done activities or projects to learn about chemicals (for 

example, mixing sugar or salt in water)?

A Never

B Rarely

C Sometimes

D Often
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 8

Grade 4 Science–Student—Item 4

Comprehension

Interviewer notes report the following observations:

 When asked to paraphrase the item, the students displayed a good understanding of the item.

 When asked to define “scientific tools,” students had difficulty explaining this term, and instead provided 

examples of such tools.

o Four students identified “magnifying glass” as an example of a scientific tool; two students listed 

“pH testing strip;” two students mentioned “dropper.” 

Response Process

 When asked how they would respond if they had only used tools at certain times of the year, three 

students would have responded “once every few weeks,” two would have answered “two or three times 

a week,” one would have answered “about once a week,” two would have answered “never or hardly 

ever,” and one did not understand the probe.

 Of the three students who viewed item four last, after items five and six, only one preferred the response 

options in item four, while two preferred item six. These two students explained that the response 

options in item four were confusing, and one added that option A and B in item four seemed to be the 

same. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the

interviews.

New (Revised based on VC315222, VC315227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 9

Grade 4 Science–Student—Item 5

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item and 

instructions.

 When asked to define “scientific tools,” the students had difficulty explaining this term and instead 

provided examples of such tools.

o Students reported the following examples of scientific tools: “magnifying glass,” “magnet,” and 

“dropper.” 

 When asked to define “rarely,” “sometimes,” and “often,” students defined the response options either 

using subjective, vague terminology (rarely means “hardly ever,” sometimes means “once in a while,” 

and often means “a lot” or “all the time”), or in terms of frequencies (“often” as either every day or every 

other day; “sometimes” as once or twice a week; “often” as either every day or every other day). 

Response Process

 When asked how they would answer if they had only used tools at certain times of the year, four 

students would have answered “rarely,” three would have answered “sometimes,” and one would have 

answered “never.” One student was not asked this probe, and one student could not provide an answer.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New (Revised based on VC315222, VC315227, and VC315229)

In this school year, how often have you done science activities using scientific tools?

A Never

B Rarely

C Sometimes

D Often
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 10

Grade 4 Science–Student—Item 6

Response Process

 When asked how they would answer if they had only used tools at certain times of the year, half of the 

students (5 of 10) would have answered “yes, a little,” two would have answered “no,” and two would 

have answered “yes, a lot.” One student did not understand the probe.

 Between items four, five, and six, half of the students (5 of 10) preferred item six, three students 

preferred item four, and two preferred item five. Three students indicated that they did not prefer item 

five because the words of the response options were hard for them to understand. Students typically 

preferred item six because it was simpler than item four, and easier to understand than item five.

Recommendations for Improvement

 Student verbal reports and responses to the probing questions show that item six was the preferred 

version of this question, and easiest for fourth grade students to understand.

New (Revised based on VC315222, VC315227, and VC315229)

Have you used scientific tools in school this year?

A No

B Yes, a little

C Yes, a lot
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 11

Grade 4 Science–Student—Item 7

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item. However, 

they did not mention the time period included in the item stem.

Response Process

 When asked if they had included material that they read outside of school in their answers, five students 

indicated that they had not, three indicated that they did include readings outside of school, and two 

students did not read science books at all. Half of the students (5 of 10) also indicated that they did not 

read materials about science outside of school.

 Of the five students who responded that they had read a book or magazine about science in this school 

year, all indicated that they read the most often in the classroom.

 Only four students stated that they read books about science that are not for school work.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315239

In this school year, how often have you read a book or magazine about science?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 12

Grade 4 Science–Student—Item 8

Reading 

Interviewer notes report the following issues with reading:

 Two students showed hesitation in pronouncing the word “presented.”

Comprehension

 When asked to paraphrase the item, the students displayed a good understanding of the item. However, 

they did not mention the time reference included in the item.

 When asked to define “presented,” students varied in their responses. Half of the students (5 of 10) said 

that it involved “showing” something, while two used the word “share” instead. Six of the students stated 

that it involved showing something in front of others, and five explicitly referred to the classroom.

 When asked if “to show what you have learned” was different from “to present what you have learned,” 

six students said that these were the same, while one student said that they differed. Two students did 

not answer, and one student did not understand what “show what you’ve learned” meant.

Response Process

 Six of the ten students indicated that they have presented something to their class. Of these six, four 

presented information that they learned, and the other two presented a project that they worked on.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315259

In this school year, how often have you presented what you learned about science to your class?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 13

Grade 4 Science–Student—Item 9

Comprehension

 When asked to paraphrase the item, three students stated that it asked if they had used measurements, 

and did not note that the item asks whether they have talked about them.

 When asked to define “talked about,” the majority of students (8 of 10) answered in a similar manner, 

stating that it meant to “say something” or “talk to someone.” Only one student did not know how to 

respond.

 When asked to define the term “measurements or results,” the students showed good understanding of 

these terms, and explained them by referring to results, the end-effect or product of experiments, and 

solutions to problems. Two students used examples to help illustrate what they were trying to say.

Response Process

 Seven of the ten students stated that they “never or hardly ever” talked about measurements or results.

However, when probed, two of these students reported talking with others: one with a teacher, and the 

other with classmates.

 The remaining three reported that they talked about measurements or results with a teacher, 

classmates, or the whole class.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315288

In this school year, how often have you talked about measurements or results from your science 

activities or projects?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 14

Grade 4 Science–Student—Item 10

Reading

 Interviewer notes report no major problems with reading in this item.

Comprehension

 Interviewer notes report no major problems with comprehension in this item.

 When asked to paraphrase the item, the students displayed a good understanding of the item. However, 

they did not mention the time period referenced in the item stem.

 All students but one defined “report” in a similar manner, as writing or sharing information from 

measurements or science projects with other people.

Response Process

 Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this school 

year; two responded “once every few weeks;” three said “two or three times a week.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315265

In this school year, how often have you written a report on your science activities or projects?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 15

Grade 4 Science–Student—Item 11

Comprehension

 All students had a similar understanding of the phrase “science journal,” describing it as something you 

write in to keep track of information you have for science topics.

 The descriptions that students provided for the phrase “lab write-up” were less consistent.

o Six students either did not know what the phrase meant or defined it incorrectly; another student 

had never heard of it, but had an intuitive grasp of its meaning.

o Of the students that did define the term, three described it more as a report, where they write the 

results of a project or test, and one described it more as a journal, where general science notes 

and information go.

Response Process

 Four of the ten students would call the written work they do for a science activity or project a “science 

journal” (these same four students were unable to define a “lab write-up”). Two students would call the 

work a “report,” while two students would call it something else (a “writing piece” and “results”), and one 

was unsure what to call it. Only one student said he/she would call the work a “lab write-up,” but did not 

know why.

 Half of the students (5 of 10) reported having “never or hardly ever” been asked to write about science 

activities or projects. However, only two students who answered “never or hardly ever” to the previous 

item (item ten) responded the same way to this item (item eleven).

Recommendations for Improvement

 Verbal reports and student responses to item-specific probes show that over half of the students did not 

understand the phrase “lab write-up,” but all students had at least a rudimentary grasp of the phrase 

“science journal.” 

 Although the interviewer notes for the previous item (item ten) show that all but one student had a good 

understanding of the term “report,” the more generic formulation of item eleven (“asked to write about 

your science activities or projects”) produced a different answer pattern among students, suggesting that 

item ten did not capture all possible information from students on this topic. We suggest that it might be 

helpful using the question formulation from item eleven, with the addition of specific examples of written 

deliverables, such as reports, science journals, and lab write-ups. 

New (Revised based on VC315265)

In this school year, how often have you been asked to write about your science activities or projects?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 16

Grade 4 Science–Student—Item 12

Reading 

Interviewer notes report the following issues with reading:

 In the phrase “library resources for science,” two students had trouble pronouncing “resources,” and 

another skipped reading the phrase.

Comprehension

 When asked, five students interpreted the item as asking if they had ever used library resources for 

science.

 Two students could not define the phrase “library resources.” Of the remaining eight students, all gave 

books as an example of a library resource. Three also gave the example of computers, and two others 

gave the example of magazines.

 Only after being probed did students acknowledge the internet as being a library resource. After being 

asked, six students said that internet was a library resource, while three said that it was not. When 

probed, two students also acknowledged computers as library resources; another student was unsure, 

because computers in his/her school were not for fourth graders.

Response Process

 Nine students were asked whether they thought of their school library, the public library, or both. Of the 

nine, seven thought only of the school library, one thought of both, and one answered that they “do not 

go to the library.”

 When explicitly asked about individual resources, three students included encyclopedias, and three 

included videos as library resources.

Recommendations for Improvement

 Only two students were unable to explain the meaning of the term “library resources.” However, verbal 

reports and student paraphrasing of item twelve show that a majority of students thought only of books 

as library resources. We suggest including in the item, in parentheses, examples of different types of 

library resources (such as magazines, computers, internet, and videos), in order to aid understanding of 

this term and remind students of library resources other than books. 

VC315266

In this school year, how often have you used library resources for science?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 17

Grade 4 Science–Student—Item 13

Comprehension

 On the whole, the students understood the phrase “science activities” in a similar manner: eight of ten 

described it as activities or projects that one does, that are related to a science topic. Only two did not 

explicitly refer to a science topic.

 Students demonstrated a good and consistent understanding of the phrase “not for schoolwork:” five 

students said it meant activities done at home, and three said it meant being “out of school.”

Response Process

 When asked to provide examples of activities they had done that were not for schoolwork, two students 

gave technology/engineering examples (e.g., fixing a TV), two gave natural science examples (e.g., 

studying bugs), and two gave chemical examples (e.g., mixing substances together). Two students who 

answered “never or hardly ever” could not remember any examples of activities they had done that were 

not for schoolwork.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315410

How often do you do science activities that are not for schoolwork?

A Never or hardly ever

B Sometimes

C Often

D Always or almost always
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 18

Grade 4 Science–Student—Item 14

Comprehension

 When asked to define a “science fair,” six students described it as related to making experiments. Of 

these six, two also stated that “science fair” involved showing their work to other people. One student 

defined it as showing something, but did not mention experiments.

 Two students defined science fairs as having a competition element; one said first place gets a prize, 

and the other said science fair was synonymous with science competition.

 Two students did not understand what a “science fair” was; they knew only that it had something to do 

with science.

Response Process

 When asked to provide an example of a science fair, there were few prevailing trends. Six students 

stated that they had not participated in one; two referred to “event nights” at their schools where science 

activities took place, but they did not present their own experiments.

Recommendations for Improvement

 Verbal reports and answers to the probing questions show that more than half of students had not 

participated in a science fair and had a vague understanding of the term. Thus, we suggest that “science 

fair” might not be an appropriate term to use.

New

In this school year, have you participated in a science fair?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 19

Grade 4 Science–Student—Item 15

Comprehension

 When asked to define a “science club,” all students provided responses indicating that science clubs had 

to do with people meeting together to learn more about science or do science activities. Additionally, four 

students explicitly indicated they were after-school activities.

Response Process

 When asked to provide an example of a science club, four students provided concise examples of 

specific clubs, three could only give examples of activities they had done in the clubs, and one student 

had never participated in a club. Two students could not recall any examples of science clubs. Examples 

provided were:

o STEM club, robotics club, Hands-On Science, and Mad Science.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you participated in a science club?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 20

Grade 4 Science–Student—Item 16

Comprehension

 When asked to define a “science competition,” five students gave responses indicating that this activity 

involves “prizes” or “first place, second place, etc.” Six students said it involved either creating or 

developing their own project, or that it was like a science fair.

Response Process

 When asked to provide an example of a science competition, three students either gave “science fair” as 

their example, or gave their same example from item fourteen (which asked for an example of a science 

fair). One student said fairs and competitions were similar, but not the same.

 Of the five students that were asked, all had heard of a robotics competition, but only one indicated that 

they had participated in a robotics competition.

 Students were asked whether or not science fairs, science clubs, and science competitions were 

different things. Over half (6 of 10) said that they were all different things; three said they were all the 

same; one said that fairs and competitions were the same, but clubs were different. All three of the 

students who indicated that they were all the same gave the reason that they all dealt with science.

Recommendations for Improvement

 Verbal reports and answers to the probing questions show that while a majority of the students had a 

very vague understanding of the term “science fair,” a majority had quite a good understanding of the 

term ”science competition.” Thus, it is likely that fourth graders do not have much experience with 

science fairs, so “science competition” may be a more appropriate term to use.

New

In this school year, have you participated in a science competition?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 21

Grade 4 Science–Student—Item 17

Comprehension

 Because “science museum” is a relatively vague term, there was some variance in the definitions that 

students provided. However, most (7 of 10) indicated that it either involved scientific activities, lessons, 

and/or technology, or gave appropriate examples. Two students were unsure of the meaning, and one 

student seemed to include non-scientific museums in their definition.

 Nine students were asked what specific museums they considered to be science and non-science 

museums. Two students could not remember any science museums; two students could not remember 

any non-science museums.

 When asked to give an example of a science museum that they had been to, three students could not 

remember any museums.

Response Process

 When students were asked what museums they thought of as science museums, examples included:

o The Museum of Natural History, the Space Museum, and an animal museum called Amazing 

Animals.

 When students were asked what museums they thought of as non-science museums, examples 

included:

o The National History Museum, the Wax Museum, and the Native American Museum.

 When asked to give an example of a science museum they had been to, three students stated that they 

had not been to any museums. One student provided an example that was not a museum, and another 

provided three examples, one of which was a non-science museum. Examples of relevant museums 

included:

o Museum of Natural History, Air and Space Museum, and the Animal Museum.

 Only one student was asked who he/she had visited a science museum with, and responded, “the entire 

fourth grade.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 22

Grade 4 Science–Student—Item 18

Comprehension

 When asked to define “on a school trip,” three students stated it was a trip involving their class and/or 

grade; three students mentioned riding on a school bus; three students said it was a field trip; two

students simply re-worded the phrase (e.g., “a trip for school”).

 Of the seven students asked to provide an example of a science museum that they visited on a school 

trip, two could not think of any examples.

 Of the five students who provided examples, two referenced museums, and two could only discuss the 

activities they did at the museums (one simply referenced a field trip to the Chesapeake Bay). The 

referenced museums were:

o The Animal Museum and the Kid’s Museum in Baltimore Harbor.

 Only one student had been to a science museum on a school trip this year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 23

Grade 4 Science–Student—Item 19

Comprehension

 When asked to define “not on a school trip,” six of the ten students stated it was a trip that did not 

include school individuals or classmates, while five students described it as involving family and/or 

friends.

 Even though five students gave examples of trips they went on that were not for school, four of them 

were unable to provide specific names of museums. Of these four students, two described what the 

museum was like, and the other two could only say that they had been to a museum.

 Five students provided the following examples of trips that were not for school: 

o The Air and Space Museum, the Natural History Museum, and the American Indian Museum.

 Only one student indicated that they had visited a science museum this year that was not on a school 

trip.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum that was not on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 24

2.1.2. Grade 8 Students

Grade 8 Science–Student—Item 1

Reading 

Interviewer notes report the following observations:

 Three students misread “barometer” in sub-item “f.”

 Three students skipped over the examples in parentheses for all sub-items.

Comprehension 

 Four students did not understand the reference period of the item to be “this school year;” instead, their 

responses reflected prior school years as well. 

 When asked what the phrase “hands-on activities or projects” meant to them, half of the students (5 of 

10) indicated that this meant working on assigned tasks. Two students understood this to mean working 

VC304986 

In your science class this year, have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Yes No

a. Living things (for example, plants, animals, 

bacteria) A B

VC304988

b. Electricity (for example, circuits and energy) A B

Revised sub-item 

based on VC304989

c. Chemicals (for example, mixing or dissolving 

sugar or salt in water) A B

VC304991

d. Rocks or minerals (for example, identifying 

types) A B

VC305007

e. Magnifying glass or microscope (for looking 

at small things) A B

VC305008

f. Thermometer or barometer (for making 

measurements) A B

VC305009

g. Simple machines (for example, pulleys and 

levers) A B

VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 25

with their peers on an activity. The remaining three students provided limited, vague definitions for the 

phrase.

 When asked the meaning of the phrase “circuits and energy,” students’ responses varied greatly. While 

two students understood a circuit as something consisting of wires, another two understood it as 

something that electricity moves through. Two students understood energy to be synonymous with the 

term “power.” Four students provided a limited and vague definition for the phrase, showing a partial 

understanding that “circuits and energy” referred to electricity. Two of the students were unable to 

provide a definition. 

Response Process

 Of the four students who answered “yes” to sub-item “b” (“electricity”), three recalled that the activity they 

thought of when formulating their answer was related to using wires to light a light bulb. The remaining 

respondent recalled learning about electricity, but could not provide examples of specific activities 

related to this topic.

 When asked how they would respond to sub-item “b” if the example had been “batteries and light” 

instead of “circuits and energy,” half (5 of 10) responded that they would have changed their answers. 

Four of these students would have changed their answers from “no” to “yes,” while the fifth would have 

changed from “yes” to “no.”

Recommendations for Improvement

 There was significant variation in how students understood the phrase “circuits and energy.” In terms of 

their recollections of activities relating to electricity, students predominantly recalled activities where they 

had used batteries, wires, and/or light bulbs. In addition, probing revealed that if sub-item “b” included 

the example of “batteries and light,” instead of “circuits and energy,” twice as many students would have 

answered “yes” to this sub-item. Thus, we recommend that the tested alternative phrase, “batteries and 

light,” would be more a relevant example to include in this item. 

 A large proportion of the students did not appear to understand that the item referred only to “this year,” 

and included in their answers activities from previous school years. The term “this year” can be 

interpreted as “this school year,” “this calendar year,” or “at any time between now and last year at the 

same time.” If it is important that this measurement reflect a specific time period, we suggest providing a 

more specifically defined time reference in the item’s stem. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 26

Grade 8 Science–Student—Item 2

Reading 

Interviewer notes report the following observations:

 Three students misread “barometer” in sub-item “f.”

Response Process

When asked how they would respond if they had only done activities or projects at certain times during the 

school year (e.g., doing an activity every day for three weeks, but not since then), students provided a variety 

of responses. 

 Four students said they would respond in the lower half of the scale. Of these four, three students said 

they would answer “once every few weeks,” while the fourth would answer “never or hardly ever.” 

 Two students said they would choose the middle response option, “about once a week.” 

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never or 

hardly 

ever

Once 

every 

few 

weeks

About 

once a 

week

Two or 

three 

times a 

week

Every 

day or 

almost 

every 

day

a. Living things (for example, 

plants, animals, bacteria) A B C D E

VC304988

b. Electricity (for example, 

circuits and energy) A B C D E

Revised subitem based on 

VC304989

c. Chemicals (for example, 

mixing or dissolving sugar 

or salt in water)

A B C D E

VC304991

d. Rocks or minerals (for 

example, identifying types)

A B C D E VC305007

e. Magnifying glass or 

microscope (for looking at 

small things)

A B C D E VC305008

f. Thermometer or barometer 

(for making 

measurements)

A B C D E VC305009

g. Simple machines (for 

example, pulleys and 

levers)

A B C D E VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 27

 Two students said they would answer in the upper half of the scale. One indicated that they would 

answer “two or three times a week;” the other would choose “every day or almost every day.” 

 One student said they would arbitrarily choose either the first or last response option on the scale, 

because the item’s reference period lacked specificity. This student suggested that the item should ask 

how often they did each of these things in “each science unit,” rather than the entire school year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 28

Grade 8 Science–Student—Item 3

Reading 

Interviewer notes report the following observations:

 Three students skipped over the examples in parentheses for all sub-items.

Comprehension 

 Two students did not understand the reference period of the item to be “this school year;” instead, their 

responses reflected prior school years as well.

Response Process

 Six students defined the response options using more subjective and vague terminology, with “rarely” 

meaning “not much,” “sometimes” meaning “once in a while,” and “often” meaning “a lot.” Of these six 

students, four defined “rarely” in terms of comparisons to the other categories (“not often,” “less than 

sometimes”). The definitions for “sometimes” and “often” varied among these six students, but their 

definitions showed that they understood the concept in all cases.

 Four students defined the response options according to more objective descriptions. Among these 

respondents, the term “rarely” was defined by all four students as either “once” or “once a month;” the 

term “sometimes” had varying definitions; the term “often” was defined by all four students as once or 

multiple times a week.

New (Revised based on VC304986) 

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never Rarely Sometimes Often

a. Living things (for example, plants, 

animals, bacteria) A B C D

VC304988

b. Electricity (for example, circuits 

and energy) A B C D

Revised 

sub-item 

based on 

VC304989

c. Chemicals (for example, mixing or 

dissolving sugar or salt in water)

A B C D

VC304991

d. Rocks or minerals (for example, 

identifying types)

A B C D VC305007

e. Magnifying glass or microscope 

(for looking at small things)

A B C D VC305008

f. Thermometer or barometer (for 

making measurements)

A B C D VC305009

g. Simple machines (for example, 

pulleys and levers)

A B C D VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 29

 When asked how they would respond if they had only done activities or projects at certain times during 

the school year, a majority of students (7 of 10) said that they would respond “sometimes.”

 When asked which version of the response options they preferred, eight of the ten students said they 

preferred the response options used in this item. Many implied that answering the item with the 

dichotomous response options used in item one was easy, but that these options were too broad to 

accurately assess their internally generated answers. Moreover, the students explained that it was 

difficult to answer the item with frequencies, as used in item two, because the response options were too 

specific. 

Recommendations for Improvement

 Verbal reports and answers to probing questions show that the students strongly preferred the response 

option scale used in item three, as compared to item two. It appears that the students could more easily 

fit their internally generated answers into the response options provided in item three; in addition, these 

response options placed a lower overall cognitive burden upon students. When asked how they would 

respond if they had only done activities or projects at certain times during the school year, student 

responses showed lower variability in item three, as compared to item two, which may indicate that the 

response scale in item three is able to produce responses with higher reliability.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 30

Grade 8 Science–Student—Item 4

Comprehension 

 When asked to state the item in their own words, the students displayed a good understanding of the 

item or instructions. However, further probing revealed that four students missed, or did not take into 

account, that their answer should focus on activities performed for science class, and they consequently 

considered media that was not for school work when answering this item.

 When asked what the phrase “science topics” meant, five students implied that this meant subjects 

studied in science class, two students gave discrete examples of specific science topics, and two 

answered that this phrase meant “anything” that has to do with science. 

Response Process

 When asked if they would respond differently if the item used the term “science” instead of “science 

topics,” seven said they would answer differently. The most common reason given for this change of 

response among students (4 of 7) was that the term “science” was broader than “science topics,” and 

would allow them to consider more types of media as related to “science” in their responses. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few 

weeks

About 

once a 

week

Two or

three 

times a 

week

Every 

day or

almost

every 

day

a. Read a science textbook, in 

class or at home A B C D E VC546510

b. Read a book or magazine 

about science topics A B C D E

Revised subitem based 

on 

VC305295

c. Use the Internet to learn 

about science topics A B C D E

Revised subitem based 

on 

VC720562

d. Watch a movie, video, or 

DVD about science topics A B C D E

Revised subitem based 

on 

VC305307
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 31

Grade 8 Science–Student—Item 5

Comprehension 

 When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub-items “c” and “d” 

implied summarizing what they did or learned in class. The other five students defined the phrase as 

generally talking with people (students or teachers) in their class. 

 When asked what is meant by the term “lab write-up,” half of the students (5 of 10) described it as 

something akin to a report, describing a science lab that they had completed. Three students described 

it as another type of class assignment unrelated to a science lab. Two students were unable to provide a 

definition.

VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few 

weeks

About 

once a 

week

Two or

three 

times a 

week

Every 

day or

almost

every 

day

a. Identify questions that can be 

addressed through science 

experiments

A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements 

you took for your science 

project or activity

A B C D E VC720635

d. Talk about the results of your 

science project or activity A B C D E VC720636

e. Watch your teacher do a 

science experiment or 

activity

A B C D E VC720638

f. Make graphs or charts of the 

results from your science 

project or activity

A B C D E VC720639

g. Write a report on your 

science project or activity A B C D E VC720641

h. Write a lab write-up on your 

science project or activity A B C D E New

i. Write a science journal A B C D E New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 32

 When asked the meaning of the term “science journal,” responses varied greatly, and included the 

following: where they write their “warm-up” exercise (2 students); where to write notes (2 students); 

where to write about experiments (2 students); something that is used to describe what happened in 

class (2 students); where one can write about science in general (2 students). 

Recommendations for Improvement

 Interview notes revealed considerable variability in students’ understanding of the terms “lab write-up” 

and “science journal.” In order to avoid any potential confusion in the future and to ensure consistent 

interpretation among students, we suggest providing a standardized definition of these terms. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 33

Grade 8 Science–Student—Item 6

Comprehension 

 Three students indicated during their think-aloud that they found the item and the term “library 

resources” to be vague.

Response Process

 When probed, seven students stated that they were thinking about their school’s library when answering 

this item and three stated that they were thinking of both their school’s library and a public library.

 In regards to what students thought of as library resources, the top library resource identified by students 

was books (9 of 10). Other resources that students mentioned, unaided, were computers (4 students), 

internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1 student), and 

magazines (1 student). 

 When asked directly about each resource, eight students said they thought of computers as resources 

while answering the item, seven students thought of encyclopedias, five thought of the internet, five of 

magazines, and four thought of videos.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC315266

In this school year, how often have you used library resources for science?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 34

Grade 8 Science–Student—Item 7

Comprehension 

 Interviewer notes do not indicate any consistent problems in students’ understanding of the item or 

instructions.

 When asked what was meant by the phrase “science-related activities” in sub-item “a,” the students 

either provided general, broad descriptions related to activities that had “anything to do with science,” or 

offered some examples such as measurements, experiments, and research.

 When asked what was meant by the phrase “not for schoolwork,” five students emphasized in their 

responses that these activities were done outside of school, or were not assigned schoolwork. Four 

students emphasized in their explanations that these activities were done based on personal interests.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in 

one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. I do science-related activities that are 

not for schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help 

me in the future. A B C D VC305353
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 35

Grade 8 Science–Student—Item 8

Reading 

Interviewer notes report the following observations:

 Three students had trouble reading the word “ecology” in sub-item “a.”

Comprehension

 When asked what was meant by the term “covered” in the item stem, seven students provided a general, 

broad definition for the term (e.g., “gone over,” “what we learned”). In their definitions, three of the 

students included a term that described the extent of the learning (“we learned about it completely,” “we 

know it thoroughly”). 

 Students that answered “yes” to any of the sub-items were asked to describe how the material was 

“covered” in their class. A majority of the students (8 of 9) answered this by providing the names of the 

topics that were covered. All of the topics mentioned by students fit within the existing sub-item’s 

category. One student described the types of assignments and materials that were used to learn each 

topic. 

Response Process

 When probed, seven students said that they would prefer a larger scale that asked to what extent the 

topics were covered, as opposed to “yes” and “no” response options. The main reason for this 

preference, according to five of these seven students, was the ability to more accurately assess the 

extent to which these topics were covered. Of the three students who preferred the yes/no format, two 

had previously included a term that described the extent of learning in their description of the term 

“covered.”

Recommendations for Improvement

 Based on the students’ verbal reports and answers to probes, we suggest that a change in item scale 

would be relevant, from a yes/no scale to a Likert scale that measures the extent to which a topic was 

covered. Overall, the students preferred the Likert scale-based response options and seem capable of 

accurately assessing sub-items on this scale. 

VC304978

In your science class this year, which of the following topics have been covered? Fill in one oval on each 

line.

Yes No

a. Life science (for example, biology, the human body, or ecology) A B VC304982

b. Physical science (for example, energy, physics, or chemistry) A B VC304983

c. Earth and space science (for example, geology or astronomy) A B VC304984

d. Engineering and technology (for example, designing solutions to 

problems)

A B VC304985
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 36

Grade 8 Science–Student—Item 9

Comprehension

 When asked what was meant by the term “science fair” in sub-item “a,” all ten students provided 

explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven of the 

ten explanations also indicated that these projects were shown or talked about to others. 

 When asked what was meant by the term “science club” in sub-item “b,” students had varying answers. 

Five students indicated that these clubs occurred after school, one student said that they occur annually, 

and the other four students did not discuss when clubs occurred. Four students included in their 

definition that the purpose of the club was to do science activities, and three noted that the purpose of 

the club was to discuss or talk about science. 

 When asked what was meant by the term “science competition” in sub-item “c,” nine students provided a 

definition that described it as similar to a science fair, where one makes a project, invention, or 

experiment and competes against others. One student mentioned that people also compete by 

answering questions. One student was unable to define the term.

Response Process

 Eight students were able to provide an example of a science fair that either they or a classmate had 

participated in. 

o Three students cited examples from elementary school, one referred to a science expo, one 

student referred to the school STEM fair.

 Five students were able to provide an example of a science club that either they or a classmate had 

participated in. Examples included:

o Technology club, life science club, a science club involving robotics, and Project Change.

 Only one student was able to provide an actual example of a science competition, which they described 

as a “science club competition, where they had to build things to compete in certain challenges.” One 

student answered that a geography bee was a science competition. The other eight students were 

unable to retrieve a cogent example. 

 All ten students answered “no” to sub-items “a” and “c,” and nine answered “no” to sub-item “b.” 

New

In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 37

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 38

Grade 8 Science–Student—Item 10

Comprehension

 All ten students interpreted the item as asking about trips to museums that involve science.

 When asked what was meant by the term “science museum,” more than half of students (6 of 10) gave 

general and vague definitions for the term (e.g., “a museum that has to do with science of any kind” and 

“a museum about science”). The other four students provided examples of what might be found in a 

science museum (e.g., “a collection of experiments” and “rocks and dinosaurs”). 

 When asked what they think of as science museums, eight students were able to provide examples of 

actual science museums, including: Air and Space, Smithsonian, and Port Discovery. Only one student 

provided an example that was not an actual science museum. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 39

Grade 8 Science–Student—Item 11

Comprehension

 The students interpreted the item as asking if they have visited a museum on a school trip, or under the 

supervision of the school.

 When asked what was meant by the phrase “on a school trip,” all ten students either found the term 

synonymous with the term “field trip,” or described traveling to a location with people from their school. 

 When asked if they had been on a school trip to a science museum, six students described a situation 

where they had been on a school trip. Of those six, three provided descriptions of the specific trip, 

including traveling to a science museum with classmates, the class, or the teacher on a bus. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 40

Grade 8 Science–Student—Item 12

Comprehension

 When asked what was meant by the phrase “not on a school trip,” seven of the students stated that it 

meant visiting the museum without others from their school, and three said it meant visiting in the 

presence of family or friends. 

 Six students described a time when they visited a science museum that was not on a school trip, and 

four students said they could not remember a visit to a science museum that was not on a school trip.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum that was not on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 41

2.1.3. Grade 12 Students

Grade 12 Science–Student—Item 1

Comprehension

 Half of the students (5 of 10) described “hands on activities” as actively interacting with what they were 

learning. The other half of students (5 of 10) thought the term “hands on activities” referred to labs and 

research. 

 When the students were asked what they thought was meant by the term “circuits and energy,” half the 

students (5 of 10) attempted to explain the concept of how circuits and energy are related (“circuits carry 

electrical energy from one place to another”). The other half of students named tangible components of 

circuits such as wires, batteries, and light bulbs. 

 Of the three students that answered “yes” to sub-item “b,” two recalled activities in class related to 

electricity or circuits. The other described a chemistry lab where reactions were measured as a function 

of heat applied by an electric heater.

VC304986 

In your science class this year, have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Yes No

a. Living things (for example, plants, animals, 

bacteria) A B

VC304988

b. Electricity (for example, circuits and energy) A B

Revised sub-item based 

on VC304989

c. Chemicals (for example, mixing or dissolving 

sugar or salt in water) A B

VC304991

d. Rocks or minerals (for example, identifying 

types) A B VC305007

e. Magnifying glass or microscope (for looking 

at small things) A B VC305008

f. Thermometer or barometer (for making 

measurements) A B VC305009

g. Simple machines (for example, pulleys and 

levers) A B VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 42

Response Process

 When probed, eight of the ten students would not change their response to sub-item “b” if instead of 

“circuits and light” the example was “batteries and light.” Five students noted they believed that asking 

the item this way would have changed its intent. Three of these five thought that “light” did not fit with the 

concept of electricity. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 43

Grade 12 Science–Student—Item 2

Response Process

When asked how they would respond if they had only done activities or projects at certain times during the 

school year (e.g., if they had done an activity every day for three weeks, but they have not done the activity 

since then) seven of ten students provided a clear indication of the response options they would choose. 

 Six of these seven students selected options on the lower end of the scale such as “never or hardly 

ever” (2 of 6) or “once every few weeks” (4 of 6). 

 Of the five students who would answer in the lower end of the scale, three explained their choices by 

describing a process similar to averaging the frequency of the activity over the course of a school year. 

The other two students explained that the response option “about once a week” or higher would be a 

poor way of describing this situation.

New (Revised based on VC304986)

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never or 

hardly 

ever

Once 

every 

few 

weeks

About 

once a 

week

Two or 

three 

times a 

week

Every day 

or almost 

every day

a. Living things (for example, 

plants, animals, bacteria) A B C D E

VC304988

b. Electricity (for example, 

circuits and energy) A B C D E

Revised subitem based 

on 

VC304989

c. Chemicals (for example, 

mixing or dissolving sugar 

or salt in water)

A B C D E

VC304991

d. Rocks or minerals (for 

example, identifying types)

A B C D E VC305007

e. Magnifying glass or 

microscope (for looking at 

small things)

A B C D E VC305008

f. Thermometer or barometer 

(for making 

measurements)

A B C D E VC305009

g. Simple machines (for 

example, pulleys and 

levers)

A B C D E VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 44

 One student chose “every day or almost every day” explaining that they would consider the period of 

time that they were working on that project rather than the entire year. 

 The remaining three students did not specify the response option they would choose and explained that 

their answer would depend on how often they studied that topic or how well they would remember it. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 45

Grade 12 Science–Student—Item 3

Response Process

 The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to them. Five 

of the students provided objective, quantifiable answers, and five students provided subjective answers. 

o For the term “rarely,” three of the five who answered objectively indicated that the term means 

“once,” and two described it as “once every few weeks.” Of the five who answered subjectively, 

three compared the term to the other response categories (“between never and sometimes”).

o For the term “sometimes,” three of the five who answered objectively indicated that it meant 

more than once during the week, and two described it as more than once in a month. Of the five 

that answered subjectively, three compared it to the other response categories (”between rarely 

and often”). 

o For the term “often,” three of the five who answered objectively indicated that it meant every day, 

and two described it as once or more in a week. Of those who answered subjectively, all used a 

term to describe a large amount of use such as “very frequently.” 

 When asked how they would respond if they had only done activities or projects at certain times during 

the school year, six of the ten students provided a clear indication of which of the response options they 

would choose. 

New (Revised based on VC304986) 

In your science class this year, how often have you done hands-on activities or projects with any of the 

following? Fill in one oval on each line. 

Never Rarely Sometimes Often

a. Living things (for example, plants, 

animals, bacteria) A B C D

VC304988

b. Electricity (for example, circuits 

and energy) A B C D

Revised subitem based on 

VC304989

c. Chemicals (for example, mixing or 

dissolving sugar or salt in water)

A B C D

VC304991

d. Rocks or minerals (for example, 

identifying types)

A B C D VC305007

e. Magnifying glass or microscope 

(for looking at small things)

A B C D VC305008

f. Thermometer or barometer (for 

making measurements)

A B C D VC305009

g. Simple machines (for example, 

pulleys and levers)

A B C D VC305012
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 46

o All six students provided answers at the higher end of the scale by choosing either “sometimes” 

(4 of 6) or “often” (2 of 6). 

o Two of these students explained that they based their answers on the amount of time that they 

were working on that project rather than the entire school year. Two explained that if they had 

spent three weeks on a project they would have learned a lot, and one stated that “rarely” would 

not be applicable in the situation described in the probe. 

 When asked which of the first three item formats they preferred, six students preferred item three, two 

preferred item two, and two preferred item one. 

o Of the six students that preferred item three, five stated they believed these response options 

better matched their answers compared to the other items. 

o The two students that preferred item two stated that they liked the specificity of the response 

options.

o The two students that preferred item one stated that they liked that the item was straight forward 

and easy to answer. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 47

Grade 12 Science–Student—Item 4

Comprehension 

 When asked what the phrase “science topics” meant, six students explained that it means academic 

subjects such as physics, chemistry, and biology. Four students answered that this phrase means 

anything that has to do with science. 

Response Process

 Five students indicated they read books or magazines about science topics that are not required for 

schoolwork.

 Four of the five students who indicated they read books or magazines that were not for schoolwork did 

not include these books or magazines when answering sub-item “b.” The reason provided for not 

including these books or magazines was that they were not being read for their science classes.

 When asked if they would respond differently if the item used the term “science” instead of “science 

topics,” four of the ten students responded that they would answer the item differently. One student 

explained that they would have included activities done at home in addition to those completed in class. 

The explanations of the other two students suggest that the phrase “science topics” was interpreted as 

referring to more narrow and strictly defined science subjects, while the term ”science” would include a 

wider range of subjects. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC305292

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never or

hardly 

ever

Once 

every

few 

weeks

About 

once a 

week

Two or

three 

times a 

week

Every 

day or

almost

every 

day

a. Read a science textbook, in

class or at home A B C D E VC546510

b. Read a book or magazine 

about science topics A B C D E

Revised subitem based on 

VC305295

c. Use the Internet to learn 

about science topics A B C D E

Revised subitem based on 

VC720562

d. Watch a movie, video, or

DVD about science topics A B C D E

Revised subitem based on 

VC305307
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 48

Grade 12 Science–Student—Item 5

Comprehension 

 When asked what the phrase “talk about” meant in sub-item “c” and “d,” all ten students described it as 

“discussing” with the entire class, classmates, or teachers. 

 When probed, most students (7 of 10) provided similar definitions of the term “lab write-up” in sub-item 

“h.” 

VC720622

In your science class this year, how often do you do each of the following? Fill in one oval on each line.

Never 

or

hardly 

ever

Once 

every

few 

weeks

About 

once a 

week

Two or

three 

times a 

week

Every 

day or

almost

every 

day

a. Identify questions that can be 

addressed through science 

experiments

A B C D E VC720623

b. Design a science experiment A B C D E VC720634

c. Talk about measurements you 

took for your science project 

or activity

A B C D E VC720635

d. Talk about the results of your 

science project or activity A B C D E VC720636

e. Watch your teacher do a 

science experiment or activity A B C D E VC720638

f. Make graphs or charts of the 

results from your science 

project or activity

A B C D E VC720639

g. Write a report on your science 

project or activity A B C D E VC720641

h. Write a lab write-up on your 

science project or activity A B C D E New

i. Write a science journal A B C D E New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 49

o These seven students provided explanations that were centered on the idea that it was a written 

piece that summarized a science lab. However, three of these students expressed some 

confusion about the meaning of the term “lab write-up.” 

o Two students believed that the term “lab write-up” meant answering questions. 

o One student was unable provide a definition.

 When initially answering, three students expressed some confusion about the meaning of the term 

“science journal” in sub-item “i.” When asked directly what the term meant, the students had varying 

responses. While providing a definition, six of the students expressed that they were unsure what the 

term meant; of these six, two were unable to guess a definition. Two students believed the term could 

mean a long science report. Two other students believed a science journal may be where students keep 

their notes for science. Two students thought the term might mean a short daily class or homework 

assignment. One student thought a science journal might be where short brief constructed response 

items were answered. Another student thought that it might be used to provide a summary of an 

experiment for publication. 

Response Process 

 Nine students answered that they had at some point “talked about” measurements or results in subitems “c” or “d.” Of these nine, five gave examples of discussing their results with a teacher and six 

students provided an example of talking with classmates. 

 Of the eight students reporting that they had done a lab write-up, six gave examples of specific labs 

where they had completed this activity by summarizing the results of the lab. One student described 

doing a lab write-up as answering questions about the lab and one student could not remember any 

specifics, but remembered the students had to write a hypothesis.

 One student stated that he had written a science journal. He described a middle school assignment for 

which he was asked to write a nightly entry about something related to science. He specifically explained 

providing a nightly description of the moon. 

Recommendation for Improvement

 There was considerable variation in how students understood the phrase “science journal.” More than 

one half of the students interviewed (6 of 10) expressed that they were unsure of the meaning of the 

term. Only one student was able to provide an example of having written a science journal and this 

example was drawn from an experience in middle school. We suggest providing a standardized 

definition of the term “science journal,” in order to avoid any potential misunderstandings in the future.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 50

Grade 12 Science–Student—Item 6

Reading 

 Overall, students did not exhibit consistent reading problems in item six. 

 One student skipped reading the response options and proceeded to answer the item.

Comprehension 

 Overall, students did not exhibit consistent comprehension problems. A majority of students (8 of 10) 

stated the item was asking how often information from the library - books, an online resource, or a 

librarian - was used for science class. Only one student mentioned that this took place this school year.

 When answering the probe, “did you think about your school’s library, a public library, or both,” there was 

an even split among students. 

o Five students thought only of the school library resources when answering.

o Five students thought of both the school and public libraries and their resources. 

 When asked to provide examples of library resources that they have used for science class, all students 

were able to provide at least one example. Some of these included:

o Biographies, websites, online journals, internet databases, magazines, autobiographies, 

encyclopedia Britannica, and textbooks.

 A majority of students (9 of 10) considered computers and the internet as library resources.

o While thinking about his answer, one student was not sure whether the probe was asking about 

the use of computers in general, or solely as a means to access the internet. 

 A majority of students (9 of 10) did not think of videos as a library resource. One of these students 

responded to the probe by indicating that while not mentioning videos as a library resource originally, he 

had used the library computers to watch online videos.

Recommendations for Improvement

 The item does not provide instructions regarding the type of library it referring to, resulting in inconsistent 

responses among the students. We suggest that specifying the type of library (school, public or both) 

might improve the validity of responses captured by the item. 

VC315266

In this school year, how often have you used library resources for science?

A Never or hardly ever

B Once every few weeks

C About once a week

D Two or three times a week

E Every day or almost every day
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 51

Grade 12 Science–Student—Item 7

Comprehension 

 Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10) stated that 

the item asked if they agreed or disagreed with these statements about science. Three students stated 

that the item is asking about their opinions and feelings about science.

 When probed what the phrase “science-related activities” meant to them, a majority of students (9 of 10) 

reported that it meant doing anything related to science such as watching a movie or conducting an 

experiment.

 When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10) stated 

that it was something that was not required by the school (something extracurricular).

 When asked to provide an example of “science-related activities that you or your classmates have done 

or might do that are not for schoolwork,” more than half of the students (6 of 10) were able to provide an 

example, such as:

o Using a telescope, working at NIH, reading Stephen Hawking books, nature treks, and science 

fairs.

o More than half of the students (6 of 10) stated that these activities were not for schoolwork 

because a teacher did not instruct them to perform these activities, they would not be graded, 

and the activities would not be discussed in class.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC305330

Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in 

one oval on each line.

Strongly 

Disagree Disagree Agree

Strongly 

Agree

a. I do science-related activities that are not 

for schoolwork. A B C D VC305348

b. I like science. A B C D VC305350

c. Science is one of my favorite subjects. A B C D VC305351

d. I take science only because I have to. A B C D VC305352

e. I take science only because it will help 

me in the future. A B C D VC305353

f. When I graduate from high school, I 

would like to have a job related to 

science.

A B C D VC720596
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 53

Grade 12 Science–Student—Item 8

Comprehension 

 A majority of students (7 of 10) indicated that the item asked whether they had done the listed activities 

during the current school year. Three students thought it referred to all extracurricular activities.

 A majority of students (8 of 10) stated that they did not participate in any of the listed activities. Of the 

two that had participated in the listed activities, one had participated in a science club and a science 

competition; the other student had participated in a science fair. Those who reported participating in the 

listed activities also specified that they participated in the activities during the current school year.

 When asked what the phrase “science fair” means to them, most students (7 of 10) explained that it 

refers to creating a research project and presenting it to other people. 

o Two students did not differentiate the terms “science fair” and “science competition.”

o One student had never participated in a science fair but had attended one.

 Over half of students (6 of 10) were able to provide an example of a science fair, but only two students 

mentioned science fairs that were taking place this year. Four students provided examples of science 

fairs ranging chronologically from elementary school to their first year of high school. Four students were 

unable to provide specific examples of science fairs.

 When asked what the phrase “science club” meant to them, students had varied ideas about whether the 

club was a school sponsored, extracurricular, or an activity separate from the school altogether. All 

students stated that a science club was a club where students meet and discuss science-related topics.

 When asked to provide an example of a science club that they or their classmates have participated in or 

might participate in, students provided the following examples:

o Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club, biology, 

physics, or chemistry clubs, and an environmental club.

o Three students were unable to provide an example, or stated that a science club was the same 

as a science fair or competition.

 Most students (7 of 10) stated that a science competition includes conducting an experiment and being 

judged against other students, explaining that it was different from a fair. Two students considered a 

science competition to be the same as a science fair or club.

New

In this school year, have you participated in any of the following activities? Fill in one oval on each line.

Yes No

a. Science fair A B

b. Science club A B

c. Science competition A B
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 54

 When asked to provide an example of a science competition that they or their classmates have 

participated in or might participate in, some examples included:

o Science bowl, Siemens competition, NIH competition, and county competitions.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 55

Grade 12 Science–Student—Item 9

Comprehension 

 Overall, students did not exhibit any consistent comprehension problems. 

 A majority of students (9 of 10) stated that they were being asked if they had been to a museum that 

focuses on science. Only two students indicated that the visit had to take place during the current school 

year.

 When asked what the phrase “science museum” meant to them, all students stated that it is a museum 

that focuses on any field of science.

Response Process

 When asked to provide examples of science museums, all students responded and listed:

o The National Air and Space Museum, the Natural History Museum, and the Cryptologic 

Museum.

 When asked to provide examples of museums that are not for science, all students responded and 

listed:

o The National Gallery of Art, Spy Museum, Native American Museum and Holocaust Museum.

 When asked to provide an example of a science museum that they or their classmates have visited or 

might visit, all students responded and provided these examples:

o The Baltimore Science Museum, the Natural History Museum, the National Air and Space 

Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but was 

unable to name it.

 Of the four students who were asked who they went to the science museum with this school year, two 

went with their class, one went with friends, and one went with family.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 56

Grade 12 Science–Student—Item 10

Comprehension 

 Overall, students did not exhibit any consistent comprehension problems. 

 All students understood that the item was asking about field trips for science class.

 All students stated that “on a school trip” means a trip that is sponsored by the school with school 

supervision. Most students (8 of 10) were able to report an example of a school trip. They included:

o The Medical Museum in Walter Reed, the Natural History Museum, the National Aquarium, and 

the Air and Space Museum.

 None of the students had visited a museum with their school during the current school year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 57

Grade 12 Science–Student—Item 11

Comprehension 

 Overall, students did not exhibit any consistent comprehension problems. 

 A majority of students (9 of 10) were able to identify that the item was asking if they have visited a 

science museum on a trip that was not sponsored by the school.

 All students stated that “not on a school trip” meant that the visit was not sponsored by the school, and 

done on the students own time.

Response Process

 More than half (6 of 10) of students gave examples of museum trips with their families. Some of the 

museums mentioned were:

o The National Air and Space Museum, and the Smithsonian.

 A majority of students (9 of 10) reported that they have not visited a museum this year. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

In this school year, have you visited a science museum that was not on a school trip?

A Yes

B No
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 58

2.2 Teacher Background Questionnaire

2.2.1. Grade 4 Teachers

Grade 4 Science–Teacher—Item 1

Comprehension 

 The teachers interpreted this item as asking about how students were placed in smaller groups to work 

during class and how this related to students’ learning. 

 All teachers interpreted the phrase “students’ learning preferences” in sub-item “b” to refer to students’ 

learning styles; one teacher noted that she initially thought it meant “what the student wanted to learn 

about,” but changed her interpretation after re-reading.

 Teachers interpreted “students’ readiness needs” in sub-item “c” to mean students’ ability levels related 

to reading or content comprehension. One teacher noted some confusion with the term, calling it “weird 

wording,” and wondered if this referred to how English learners were grouped with other students. 

Response Process

 Most teachers (3 of 5) noted during probing that they did not group their students during science 

lessons, or that they only occasionally had students work in pairs that were not based on any of the 

criteria mentioned in this item. However, this did not affect their comprehension of the item.

 When asked if it would be efficient for the item to ask “Do you create groups within this class for science 

instruction on the basis of ability,” solely with “yes” or “no” response options, all five teachers responded 

that they preferred the way the item was currently asked, noting that this would allow them to better 

explain how they taught. 

 One teacher mentioned that he would have preferred to have the ability to provide an open-ended 

response.

New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your 

classroom? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Groupings based on students’ interest in 

science/science-related topics

A B C D New

b. Groupings based on students’ learning 

preferences

A B C D New

c. Groupings based on students’ readiness 

needs

A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 59

Recommendations for Improvement

 Although only one teacher had difficulty interpreting the terms “students’ learning preferences” and 

“students’ readiness needs,” we suggest including definitions of these two terms within each sub-item, to 

avoid any potential misunderstanding in the future and ensure consistent interpretation of the item 

among teachers. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 60

Grade 4 Science–Teacher— Item 2

Comprehension 

 One teacher did not understand the meaning of the phrase “inquiry skills” in sub-item “e.”

 One teacher described the phrase “develop skills in lab techniques” in sub-item “g” as “weird wording.” 

 One teacher had some difficulty with the phrase “scientific writing skills” in sub-item “k.” She interpreted it 

as referring to “lab write-up skills” which, she noted, would apply only to older students. She therefore 

found this sub-item to be irrelevant to elementary school teachers and did not appear to comprehend the 

sub-item’s intent. 

Response Process

 One teacher called the term “objectives” in the item problematic, or not relevant to him, because 

teachers had to follow rigid county-approved objectives in their teaching. 

 One teacher stated that sub-item “f” (“develop problem-solving (design) skills”) was asking two different 

questions at the same time.

VC970876

To what extent do you emphasize each of the following objectives in teaching science to your fourth-grade 

class? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Increase students’ interest in science A B C D VC970917

b. Teach scientific facts and principles A B C D VC970919

c. Teach scientific methods A B C D VC970920

d. Equip students with the knowledge and 

skills needed for studying science in 

upper grade levels

A B C D

Revised subitem based on 

VC970922

e. Develop inquiry skills A B C D VC970923

f. Develop problem-solving (design) skills A B C D VC970925

g. Develop skills in lab techniques A B C D VC970926

h. Increase awareness of the importance 

of science in daily life

A B C D VC970928

i. Develop systematic observation skills A B C D VC970929

j. Learn about applications of science to 

environmental issues

A B C D VC970930

k. Develop scientific writing skills A B C D VC970931
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 61

 A majority of teachers (4 of 5) noted that they interpreted “upper grade levels” in sub-item “d” to refer to 

any grade above fourth grade, including middle school; one teacher indicated that it meant seventh 

grade or above. 

 All teachers mentioned referring to school, county, or standards and curricula to understand what 

knowledge and skills students would need for studying science in upper grade levels. 

 Most teachers (3 of 5) stated that they would have answered sub-item “d” the same way if it had stated 

“prepare students for further study” instead of “equip students with the knowledge and skills needed.”

One teacher felt that the phrase “knowledge and skills” was necessary for clarification, and another 

noted that if the sub-item had only said “prepare students for further study,” she would have also 

included “study skills that will transfer to the next grade levels,” and would consequently have selected a 

higher frequency response option. 

Recommendations for Improvement

 Although only one teacher had difficulty interpreting “develop problem-solving (design) skills” in sub-item 

“f,” we suggest removing the word “design” or breaking this item into two parts, to avoid any potential 

confusion. If separated into two parts, we suggest one sub-item about problem solving skills and another 

about design skills. We also suggest keeping the current formulation of sub-item “d,” “Equip students 

with the knowledge and skills needed for studying science in upper grade levels.”
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 62

Grade 4 Science–Teacher— Item 3

Comprehension

 One teacher found sub-item “j” and the term “audiovisual” to be “vague.”

 There were two occasions where teachers appeared somewhat confused, likely due to forgetting what 

the item stem was originally asking. One teacher stated that “computerized science labs for classroom 

use” in sub-item “i” was confusing because she could not tell whether it referred strictly to science or to 

overall teaching. Another teacher had difficulties answering sub-item “j,” stating that she had audiovisual 

materials, but was not provided these resources by her school system. 

VC970932

How much of the following instructional materials and other resources does your school system provide you 

with to teach science to your fourth-grade class? Fill in one oval on each line.

None Little Some A Lot

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science 

demonstrations

A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for 

classroom use

A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments 

(e.g., telescopes, microscopes, 

thermometers, or weighing scales)

A B C D VC970964
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 63

Response Process

 When describing what the item was asking, most teachers (3 of 5) said it asked what the “school 

system” provided. One teacher specifically noted that this was different from the “school.” Another 

teacher, however, said the item asked what the school provided. One teacher did not indicate whether 

the question referred to the school system or the school, and merely stated that the question asked 

“what kind of resources are available for teaching.” 

 One teacher found it difficult to choose between response options “B” (“little”) and “C” (“some”) for subitems “c” and “d” (supplies or equipment for science demonstrations, or labs). He indicated that these 

sub-items were “hard” because the county (the school system) will no longer pay for “consumables,” 

meaning that they do not have all the material they need and he obtains some materials on his own. This 

teacher appeared to understand the sub-item, but had difficulty recalling the necessary information to 

confidently provide an accurate response. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 64

Grade 4 Science–Teacher— Item 4

Comprehension 

 Two teachers indicated that the different wording of this item, as compared to the previous item, did not 

change how they answered this item. However, two other teachers stated that their answers for this item 

differed; both noted that they saw a distinction between “school system” (used in previous item) and 

“school” (used in the current item), and the role of each entity in providing them with materials. Another 

teacher said he had answered a couple of sub-items differently than before, but could not explain why. 

Response Process

 All teachers stated that they found the options “not at all,” “small extent,” “moderate extent,” and “large 

extent” appropriate and sufficient response options. 

 When asked what the response options “small extent,” “moderate extent,” and “large extent” meant, all 

teachers indicated that “small extent” meant there were not enough resources, “moderate extent” meant 

New (Revised based on VC970932)

To what extent does your school provide the following to you? Fill in one oval on each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science textbooks A B C D VC970953

b. Science magazines and books A B C D VC970954

c. Supplies or equipment for science 

demonstrations

A B C D VC970955

d. Supplies or equipment for science labs A B C D VC970956

e. Space to conduct science labs A B C D VC970957

f. Computers for students’ use in class A B C D VC970958

g. Computer labs A B C D VC970959

h. Computers for teachers’ use A B C D VC970960

i. Computerized science labs for 

classroom use

A B C D VC970961

j. Audiovisual materials A B C D VC970962

k. Science kits A B C D VC970963

l. Scientific measurement instruments 

(e.g., telescopes, microscopes, 

thermometers, or weighing scales)

A B C D VC970964
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 65

there was a sufficient amount for class work with students, and “large extent” meant there were more 

than enough. Although teachers used different ways of quantifying the amount of resources that they 

considered sufficient or insufficient for their needs, they shared a similar interpretation of the response 

options. 

 When asked which version of the response options they preferred, between the third and fourth items, a 

majority of teachers (4 of 5) did not seem to understand this probe and selected one particular response 

option (such as “small extent”) as their reply. The single teacher who understood the probe stated she 

would choose the item with the response options “small extent, etc.” because “it sounds more 

professional.”

 When asked if they could suggest other response options that would provide more specific information 

about resources, one teacher suggested adding the response option “Not Applicable.” Another teacher 

had several suggestions: ask separately about science magazines and science textbooks, ask about a 

separate lab classroom devoted to science, and ask about safety equipment. 

Recommendations for Improvement

 The previous item (item three) includes the term “school system,” while item four includes the term 

“school.” Because of this difference, two teachers provided different answers for item three and item 

four. They inferred that item three was asking about resources provided by the county or school district, 

while item four asked about resources from the school. Their responses suggest that they tried to 

differentiate between resources provided by the school system (county or school district) and those 

provided by the school itself. Teachers’ verbal reports and answers to items three and four show that the 

formulation of item three, with the phrase “school system,” may cause confusion among teachers. We 

suggest that the final item version provide a more clear and unequivocal formulation of the question.

 Interviewer notes suggest teachers had a consistent interpretation of the response options used in item 

four, and stated that they are appropriate and sufficient for describing the extent of resources provided 

by the school. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 66

Grade 4 Science–Teacher— Item 5

Comprehension 

 One teacher said that the meaning of the word “activities” in sub-item “d” was not clear. 

Response Process

 All teachers understood that the item was asking about different instructional methods they use with their 

students.

 All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to whether 

students could learn at a faster or slower pace based on how quickly they understood the material or 

lesson. 

 All teachers stated that they would have answered sub-item “c” the same way if it had instead read 

“pace my teaching differently for some students.” Only one teacher said she would prefer the question 

use this alternate wording, because it “probably makes more sense.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC767811

When you teach science to your fourth-grade class, do you do any of the following? Fill in one oval on each 

line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Use a different set of methods in 

teaching some students

A B C D VC767814

b. Supplement the regular course 

curriculum with additional material for 

some students

A B C D VC767820

c. Vary the pace of instruction for some 

students A B C D

Revised sub-tem 

based on 

VC767821

d. Have some students engage in 

different classroom activities

A B C D VC767823

e. Set different achievement standards for 

some students

A B C D VC767824
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 67

Grade 4 Science–Teacher— Item 6

Reading 

 Overall, teachers did not exhibit consistent reading problems or any difficulty understanding the words in 

the item. 

Comprehension 

 A majority of teachers (4 of 5) said they made a distinction between “science fair” and “science 

competition.” Three seemed to feel that science fairs were more informal and involved a larger number 

of students exhibiting their work. One teacher said that he had not previously thought there was a 

distinction, but now did; however, he did not provide further explanation. One teacher stated that, after 

thinking about it, she was confused by what “science competition” meant, and how it differed from a 

“science fair.” She suggested the questionnaire provide examples of each. 

Response Process

 Two teachers specifically mentioned that the item asked what outside opportunities teachers were 

providing, while others indicated that it asked what the teachers or the school provided. However, all 

teachers answered in regard to activities or opportunities the school organized, rather than what the 

teacher personally provided. 

 All teachers felt that the response options provided were appropriate, though one teacher commented 

that it would be better to ask teachers to quantify the frequency of events per school year. When asked 

the follow-up probe of whether it would better to use frequencies such as certain times a week, a 

semester, or a school year, a majority of teachers (4 of 5) said it would be better. 

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

To what extent do you provide students with the following learning experiences? Fill in one oval on each 

line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 68

2.2.2. Grade 8 Teachers

Grade 8 Science–Teacher—Item 1

Comprehension 

 Three of the teachers interpreted this item as asking about how they group students in the classroom, 

and how these groupings are related to students’ learning. Two interpreted this item as asking how they

identify student groupings based on their abilities, learning preferences and interests. However, one 

teacher also stated that it is not clear if the item is asking about interests, learning preferences, or 

academic factors.

 All teachers interpreted the term “student groupings” to mean separating the students into work groups 

to collaborate on assignments. 

 The teachers defined the phrase “students’ learning preferences” in two different ways. Three teachers 

explained that the phrase refers to learning style typology, such as auditory, visual or kinetic learners, 

while the remaining two teachers interpreted the phrase as referring to more generally understood 

interests and preferences. 

 Three teachers expressed confusion regarding the phrase “students’ readiness needs” in sub-item “c.” 

One noted that it could mean "two different things: the desire and willingness to learn," or whether 

students "have a background without holes in things like math and reading abilities, and their conceptual 

knowledge is on par.” Two stated that the phrase meant students’ academic preparedness to understand 

the curriculum content and complete tasks. Two other teachers equated it with either content 

accessibility or students’ abilities. 

Response Process

 A majority of teachers (4 of 5) stated that they use all three grouping methods in their teaching day, but 

that they also take into consideration student behaviors to avoid a volatile study environment. Two stated 

that this item could be restructured as a yes or no question, asking only whether they use groupings by 

student ability in their science class instruction. Three indicated that because the item is not, in one 

teacher’s words, “black and white,” it was in fact necessary to ask about various grouping methods.

 All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large 

extent,” are appropriate for describing the extent of instructional grouping strategies.

New (Revised based onVC305014)

To what extent do you use each of the following student groupings for science instruction in your 

classroom? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Groupings based on students’ interest in 

science/science-related topics

A B C D New

b. Groupings based on students’ learning 

preferences

A B C D New

c. Groupings based on students’ readiness 

needs

A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 69

Recommendations for Improvement

 The teachers offered two different interpretations of the phrases “students’ readiness needs” and 

“students’ learning preferences.” Therefore, we suggest including standardized definitions of these 

phrases within sub-item “b” and “c,” to ensure consistent interpretation of the item among respondents 

and avoid any potential confusion in the future.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 70

Grade 8 Science–Teacher—Item 2

Comprehension 

 One teacher had difficulty interpreting a number of sub-items, asking how sub-item “e” differed from 

asking about the teaching of scientific methods in sub-item “c.” For sub-item “h” (“increase awareness of 

the importance of science in daily life”) she noted that it was very similar to what was asked in sub-item 

“a” (“increase students’ interest in science”).

 A majority of teachers (4 of 5) stated that the item asks how much they focus on different areas and 

topics when teaching science.

 When asked what the phrase “upper grade levels” means to them, a majority of teachers (4 of 5) stated 

that it meant high school level material. One teacher stated that it meant honors biology, chemistry, and 

physics, as well as AP high school and college material.

 A majority of teachers (4 of 5) stated that the phrase “studying science” meant taking physics, chemistry, 

and biology in high school.

VC976013

To what extent do you emphasize each of the following objectives in teaching science to your eighth-grade 

class? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Increase students’ interest in science A B C D VC976015

b. Teach scientific facts and principles A B C D VC976017

c. Teach scientific methods A B C D VC976018

d. Equip students with the knowledge and 

skills needed for studying science in 

upper grade levels

A B C D

Revised subitem based on 

VC976019

e. Develop inquiry skills A B C D VC976020

f. Develop problem-solving (design) skills A B C D VC976021

g. Develop skills in lab techniques A B C D VC976022

h. Increase awareness of the importance 

of science in daily life

A B C D VC976023

i. Develop systematic observation skills A B C D VC97625

j. Learn about applications of science to 

environmental issues

A B C D VC970926

k. Develop scientific writing skills A B C D VC970927
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 71

 When asked what the phrase “knowledge and skills needed for studying science in upper grade levels” 

meant, all teachers explained that it meant the principles and techniques students need to be successful 

in high school science.

Response Process

 A majority of teachers (4 of 5) said they determine what knowledge and skills will be needed beyond 8th 

grade by following state curriculum standards. One stated that he uses a tie-in with current events, such 

as conservation and streams and the Chesapeake Bay.

 Most teachers (3 of 5) reported that if the item had been phrased “prepare students for further study in 

science,” their answer would change slightly but remain similar. They also stated that “prepare students 

for further study in science” means to prepare students for scientific studies in the future.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 72

Grade 8 Science–Teacher—Item 3

Comprehension 

 The teachers interviewed displayed inconsistent interpretations of this item. Two interpreted the item as 

asking about the resources they have available to carry out all the tasks required for their students to 

learn throughout the school year. Three teachers interpreted the item as asking what resources were 

provided by their school system, with two teachers specifically mentioning the county or school district.

 One teacher was unsure about the meaning of sub-item “j” (Audiovisual materials), and whether it meant 

particular software or the actual technology.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC976028

How much of the following instructional materials and other resources does your school system provide you 

with to teach science to your eighth-grade class? Fill in one oval on each line.

None Little Some A Lot

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science 

demonstrations

A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for 

classroom use

A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments 

(e.g., telescopes, microscopes, 

thermometers, or weighing scales)

A B C D VC976045
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 73

Grade 8 Science–Teacher—Item 4

Comprehension 

 Three teachers stated that item four is asking the same question as item three. However, one of these 

respondents paraphrased the item using the term “school district.” 

 Two teachers were not entirely clear if the item was asking about their school or their school system. 

Both stated that item three and item four were making the distinction between materials provided by the 

school district and those provided by the school itself. One of these respondents stated that item three 

asks about resources provided by the county, while item four asks about in-house resources and not 

those from the county science office. Another noted that the term "school" can have multiple 

interpretations in the context of this item.

New (Revised based on VC976028)

To what extent does your school provide the following to you? Fill in one oval on each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science textbooks A B C D VC976031

b. Science magazines and books A B C D VC976032

c. Supplies or equipment for science 

demonstrations

A B C D VC976034

d. Supplies or equipment for science labs A B C D VC976035

e. Space to conduct science labs A B C D VC976036

f. Computers for students’ use in class A B C D VC976037

g. Computer labs A B C D VC976039

h. Computers for teachers’ use A B C D VC976040

i. Computerized science labs for 

classroom use

A B C D VC976041

j. Audiovisual materials A B C D VC976042

k. Science kits A B C D VC976043

l. Scientific measurement instruments 

(e.g., telescopes, microscopes, 

thermometers, or weighing scales)

A B C D VC976045
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 74

Response Process

 Three teachers reported that the way the item four was phrased compared to the way item three was 

phrased would not make a difference in how they would answer any of the sub-items listed. 

 A majority of teachers (4 of 5) stated that the response options “not at all,” “small extent,” “moderate 

extent,” and “large extent” are appropriate and sufficient for describing the extent to which resources are 

provided by the school.

 When asked what the phrases “small extent,” “moderate extent,” and “large extent” meant, a majority of 

teachers (4 of 5) said that “small extent” meant to have some resources but not enough, “moderate 

extent” meant an adequate amount of resources, and “large extent” meant more than enough resources.

 When asked which version of the response options they preferred between the third and fourth items, all 

teachers misinterpreted the probe and selected one particular response option (such as “small extent”) 

as their reply. A majority of teachers (4 out of 5) stated that they would answer moderate to large extent. 

 Two teachers gave suggestions for other answer choices that would provide more specific information 

about the resources provided by the school. One suggested that the item should be asked in terms of 

usage. Another suggested the scale should be none, some, etc.

Recommendations for Improvement

 The previous item (item three) includes the term “school system,” while this item includes the term 

“school.” Because of this difference two teachers considered item three and item four as two entirely 

different questions. They deduced that item three asks about the resources provided by the county, 

which is different from item four which asks about available “in-house” school resources. While 

answering, they tried to differentiate between resources provided by the school system (county or school 

district), and those provided by the school itself. Respondents’ verbal reports and answers to item four 

and item three show that the formulation of the item three, which utilizes the phrase “school system,” 

may cause confusion among respondents. We suggest that item four provides a more clear and 

unequivocal formulation of the question.

 Interviewer notes provided no evidence suggesting a preferential type of response scale between items 

three and four for teachers. Three of the five teachers indicated they would answer item four the same 

as item three, and four of the five stated that the response scales are appropriate and sufficient for 

describing the extent of resources provided by the school. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 75

Grade 8 Science–Teacher—Item 5

Comprehension 

 A majority of teachers (4 of 5) stated that item five is asking about different instructional methods, and 

how they plan and change lesson presentation methods to accommodate students with varying levels of 

understanding.

 A majority of teachers (4 of 5) also stated that the phrase “vary the pace of instruction for some students” 

meant providing students with different amounts of time to complete tasks, based on their different 

performance levels. One indicated that the phrase meant more time with computer-based activities.

 One teacher did not like the wording “set different achievement standards for some students,” describing

it as a hard question because its wording would suggest that she holds some students to lower 

standards; she felt this was untrue, because some students would still master the content, but in a 

different way than others.

 Three teachers stated that they would have answered sub-item “c” (“vary the pace of instruction for 

some students”) the same way if it had instead said “pace my teaching differently for some students.”

Two stated that their answers would be different because they interpreted the first version to be more 

student-specific, instead of referring to their overall teaching approach.

 All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some students.”

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

VC976085

When you teach science to your eighth-grade class, do you do any of the following? Fill in one oval on each 

line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Use a different set of methods in 

teaching some students

A B C D VC976086

b. Supplement the regular course 

curriculum with additional material for 

some students

A B C D VC976088

c. Vary the pace of instruction for some 

students A B C D

Revised subitem based on 

VC976091

d. Have some students engage in different 

classroom activities

A B C D VC976092

e. Set different achievement standards for 

some students

A B C D VC976094
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 76

Grade 8 Science–Teacher—Item 6

Comprehension 

 A majority of teachers (4 of 5) stated that item six asks about science-related activities outside of the 

classroom.

 When asked if they make a distinction between “science fair” and “science competition,” three teachers 

described the judging component as the difference between a science fair and competition. Two 

teachers did not find the two to be different.

 When asked of different activities offered by the school, teachers provided the following examples:

o Robotics club, STEM club, and Recycling club

o Visits to the Koshland Science Museum, USA Science and Engineering Festival, and the 

Smithsonian Air and Space Museum.

 When asked what types of science museums they take students to, teachers provided the following 

examples:

o Children’s Museum, the National Aquarium, and the Smithsonian Air and Space Museum.

 Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and 

“large extent,” were sufficient to describe the extent of the learning experience provided to the students.

One thought the choices were sufficient but felt that the term “science museum” was too limiting and 

suggested to add science field trips such as taking kids to Chesapeake Bay and testing compounds.

 In regards to whether frequencies, such as certain times a week/a semester/a school year, would be 

better than the response options provided for this item, opinions were divided. While two teachers 

rejected the idea that frequencies would be more relevant response options, two noted that they would 

be preferred. One respondent specifically suggested the following options: per quarter, per semester, or 

per year.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.

New

To what extent do you provide students with the following learning experiences? Fill in one oval on each 

line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 77

2.3 Administrator Background Questionnaire

2.3.1. Grade 4 Administrators

Grade 4 Science–School—Item 1

Comprehension

 Administrators were not unified in their understanding of what the role and responsibilities would be for a 

“science specialist or coach.” One understood “specialist” and “coach” to have different meanings,

explaining that a specialist works with students to teach them concepts directly, while a coach works with 

teachers to improve teaching practices and planning. 

Response Process

 Although a majority of administrators (4 of 5) indicated that they did not have a part-time or full-time 

science specialist or coach in their school, they emphasized that the responsibilities they associated with 

the role were taken on by classroom teachers and/or other types of specialists (e.g. instructional support 

teacher). 

 The one administrator who indicated that she did have a STEM (Science, Technology, Engineering, and 

Mathematics) teacher on staff indicated that it was a full-time staff member, and explained that the 

STEM teacher was responsible for helping teachers plan their lessons and demonstrations.

 Of the administrators who indicated that they did not have a science specialist or coach on staff, a 

majority (3 of 4) indicated that the responsibilities were divided among classroom teachers at their 

schools. One administrator indicated that she does have an “instructional support teacher” in the fields of 

literacy and science that she would consider as fulfilling the role of a science specialist or coach. 

Recommendations for Improvement

 As interpretations of the term “science specialist or coach” seemed to vary among administrators, it may 

be helpful to include a short definition. It was also important to the administrators to qualify a “no” 

response to this item by explaining that although they did not have a designated full or part-time science 

specialist or coach, they did have individuals who provided the services and responsibilities they 

associate with this role. In order to capture this information, we recommend expanding the range of the 

response options to include additional answer choices, such as “Yes, available on an as needed basis,” 

and/or “No, but other individuals within the school provide the services of a science specialist or coach.”

New (Revised based on VE013369)

Is there a science specialist or coach available (full- or part-time) to fourth-grade teachers at your 

school? 

A Yes, available full-time at my school. 

B Yes, available part-time at my school. 

C No 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 78

Grade 4 Science–School—Item 2

Comprehension

 All administrators interpreted sub-item “d” (“Translate science standards and current research into 

classroom practice”) as meaning the ability to: understand and internalize science standards, use that 

ability to create learning plans and objectives easily understood by students, and use appropriate 

vocabulary and teaching methods.

 When asked what sub-item “e” (“Provide science enrichment to some student groups”) meant to them, 

all administrators indicated that it meant some student groups would receive more attention or different 

activities than the rest of the class. A majority of administrators (4 of 5) indicated that the enrichment 

activities would apply to students who wanted to go beyond what was being taught in the classroom. 

One understood “some student groups” as students who may be struggling with the curriculum (such as 

those enrolled in ESOL or remedial classes), and who need additional activities to help them catch up to 

their peers. 

Response Process

 All administrators felt that the sub-items covered the majority of the responsibilities they would consider 

part of the role of a science specialist or coach. Additional responsibilities mentioned by administrators 

included the incorporation of literacy skills into science instruction, and motivating trends within the 

school to assess how well students understood the core science standards. 

 As mentioned earlier, only one out of five administrators reported having a full- or part-time science 

specialist or coach. This resulted in a majority of administrators (4 of 5) feeling unsure of how to provide 

appropriate responses to the sub-items. However, when probed about the response options, all 

administrators felt that they were appropriate.

New (Revised stem based on VE013372)

To what extent are any of the following a responsibility of the science specialist or coach available to fourthgrade teachers at your school? Fill in one oval on each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical assistance/support to 

individual teachers about science content 

or the teaching of science

A B C D VE013373

b. Conduct professional development for 

groups of teachers about science content 

or the teaching of science

A B C D VE013382

c. Provide science instruction to students 

on various topics

A B C D VE013383

d. Translate science standards and current 

research into classroom practice A B C D

Revised sub-item 

based on 

VE013384

e. Provide science enrichment to some 

student groups

A B C D VE013385
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 79

o Most administrators (3 of 5) who did not have a science specialist or coach on staff answered 

the sub-items as if they referred to the staff that performed each responsibility (teachers, 

administrators, etc.).

o One administrator mentioned that she would have replied “not applicable” for each of the subitems if the response option had been available, since she did not have a science specialist or 

coach on staff.

Recommendations for Improvement

 Depending on the item’s intent, there are two ways to address administrators’ response process 

challenges. First, if the intent of the item is to measure to what extent these responsibilities are 

performed at the school, we recommend modifying item so that it does not specifically refer to a “science 

specialist or coach.” Second, if the intent of the item is to ask specifically about the activities of a 

“science specialist or coach,” we recommend an additional “not applicable” response category for those 

administrators who do not have one on staff. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 80

Grade 4 Science–School—Item 3

Comprehension

 All administrators interpreted this item as asking about science-related opportunities the school provides 

beyond the traditional classroom instruction.

 Although most administrators (3 of 5) agreed that there is a distinction between a “science fair” and a 

“science competition,” definitions of each term differed.

o Two characterized a “science fair” as being open to all students in the school, and indicated that 

students pursued individual interests.

o Two characterized a “science competition” as being more structured, with students receiving a 

certain question or task and competing against one another.

o One did not differentiate between the two terms, but guessed that a “science competition” 

involved asking students questions about science. Another administrator did not understand the 

term “science competition,” but guessed that it would be similar to a spelling or geography bee, 

and that students would be asked questions related to science.

 When asked for examples of how the school provided opportunities for students, administrators 

mentioned included school and county-based science fairs, gardening and recycling clubs, and other 

environmental clubs.

 When asked specifically about trips to science museums, administrators mentioned the National 

Aquarium, the Smithsonian Natural History Museum, the Baltimore Science Center, NASA, and the 

Smithsonian Air and Space Museum.

Response Process

 A majority of administrators (4 of 5) preferred the response options of “large extent, moderate extent, 

and small extent,” versus frequencies such as certain times a week/semester/school years. 

 One administrator suggested merging the two categories of response options, changing the question to 

ask “To what extent does your school provide students with the following opportunities per school year?” 

She also suggested adding ranges to qualify what was meant by a small, moderate and large extent 

New

To what extent does your school provide students with the following opportunities? Fill in one oval on 

each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 81

(e.g. “small”, 0-1 times per school year, “moderate,” 2-3 times per school year, “large,” 4 or more times 

per school year).

Recommendations for Improvement

 As the definitions for “science fair” and “science competition” differed among administrators, it is 

recommended that descriptions or definitions be provided for these terms to ensure consistent 

understanding of this term across administrators. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 82

2.3.2. Grade 8 Administrators

Grade 8 Science–School—Item 1

Response process

 All administrators selected option “C,” because their schools do not have a science specialist or coach 

available to eighth-grade teachers. Three administrators noted that a teacher or administrator would 

perform some responsibilities of a science coach at certain times; however, they did not have a 

specialist or coach dedicated solely to science.

 One administrator displayed a strong discomfort stemming from the fact that his school does not have a 

science specialist or coach position, and defended and explained the school’s situation.

 When asked to describe the position or role of a “science specialist or coach,” administrators identified 

responsibilities such as providing support and help with development of curricula, lesson plans, and 

meeting standards for science instruction.

 When asked whether a different term is used in their school to describe individuals with the 

responsibilities of a science specialist, most administrators (3 of 5) answered that a similar type of 

position existed in other subject areas (e.g., math, social studies), and was described as either an 

“instructional lead teacher,” “instructional coach,” or “mentor teacher.” Two administrators noted that the 

responsibilities associated with a “science specialist” were performed by the principal or the assistant 

principal.

Recommendations for Improvement

 Although all administrators responded that their schools do not have a science specialist or coach 

available full- or part-time to eighth-grade teachers, further probing revealed that the responsibilities of 

this role are fulfilled by teachers or administrators who do not hold an official position as a science 

specialist or coach. In order to improve the collection of this information, we suggest expanding the 

range of response options to include “Yes, available on an as-needed basis.”

New (Revised based on VE013964)

Is there a science specialist or coach available (full- or part-time) to eighth-grade teachers at your 

school? 

A Yes, available full-time at my school. 

B Yes, available part-time at my school. 

C No 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 83

Grade 8 Science–School—Item 2

Comprehension

 Two administrators exhibited some confusion or difficulty understanding this item. One stated that subitem “a” is “too wordy” and needed to reread it in order to understand it. The use of the word “translate” 

in sub-item “d” was also confusing to this administrator. The other administrator was unsure about the 

intent of the item, and tried to clarify whether it was referring to what his school currently offers or what 

he envisions a science specialist would do if there were such a role at his school. 

Response Process

 Given his school’s lack of an available science specialist for eighth-grade teachers, one administrator 

answered the item based on the extent to which the services described in each sub-item are available to 

teachers. For the same reason, another administrator based his answers on the role of the science 

department chair; this individual serves as an instructional coach but does not fit the administrator’s 

definition of a science coach. 

 When probed about the phrase “translate science standards and current research into classroom 

practice,” administrators stated that it means explaining or understanding “what the standard is asking us 

[teachers, students] to do,” explaining what the standards “mean for day to day instruction,” and how “to 

help map these standards into daily objectives,” or measureable objectives.

 A majority of administrators (4 of 5) had consistent and similar interpretations of “provide science 

enrichment to some student groups,” as additional activities that go beyond the standards or 

requirements set in curriculum. These activities may be provided to a small group of students needing 

additional practice, or to a group that has already mastered the curricula standards and requires more 

New (Revised stem based on VE013966)

To what extent are any of the following a responsibility of the science specialist or coach available to eighthgrade teachers at your school? Fill in one oval on each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical assistance/support to 

individual teachers about science 

content or the teaching of science

A B C D VE013967

b. Conduct professional development for 

groups of teachers about science 

content or the teaching of science

A B C D VE013968

c. Provide science instruction to students 

on various topics

A B C D VE013969

d. Translate science standards and current 

research into classroom practice A B C D

Revised subitem based on 

VE013970

e. Provide science enrichment to some 

student groups

A B C D VE013971
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 84

differentiated materials. As examples, administrators provided such activities as “dissection, fun 

experiments,” and scientific investigations.

 Most (3 of 5) administrators agreed that the activities presented in the sub-items cover all main 

responsibilities of a science specialist or coach. Two stated that sub-items “a” and “c” are not very 

relevant to the situation in their schools, and are specific to the person with training in science. 

 Two of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate 

extent,” and “large extent,” are sufficient for describing the extent of a science specialist’s or coach’s 

responsibilities in their school. One administrator stated a preference for a numeric scale, and another 

suggested a three-point scale that would combine small and moderate extent response options, because 

he had a problem deciding which option is more relevant to his school. The last administrator 

misinterpreted the probe.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 85

Grade 8 Science–School—Item 3

Comprehension

 Overall, administrators did not exhibit any consistent problems with this item. 

 The administrators interpreted the item as asking if the school provides opportunities to participate in 

science related activities beyond the regular classroom and outside of school hours.

Response Process

 All administrators made a clear distinction between “science fair” and “science competition,” and defined 

the differences between these two concepts as follows:

o The science fair has a larger scale and extent, and gives all students the opportunity to 

participate.

o The competition is limited to the science clubs and some specific topics or areas of 

specialization. 

 Administrators reported the following examples of opportunities provided by the school:

o Trips: outdoor education at Camp Schmidt with hydroponic gardens, Howard Owens Science 

Center for Astronomy.

o Science competitions: robotics tournaments, Science Olympia competition.

o Science clubs: robotics club, science club, environmental club.

 Administrators reported the following examples of field trips to science museums organized by the 

school:

o The Owens Science Center, Bladensburg Waterfront Park, Maryland Science Museum in 

Baltimore, Smithsonian museums in Washington DC (e.g., Air and Space Museum, Museum of 

Natural History).

 Four of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate 

extent,” and “large extent,” are sufficient for describing the extent of the opportunities their school 

provides; the last administrator misinterpreted the probe.

New

To what extent does your school provide students with the following opportunities? Fill in one oval on each 

line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 86

 Administrators had diverse opinions about the use of frequencies, such as certain times a week/ 

semester/school year, and whether these would be better response options for this item. While one 

administrator noted that it would be appropriate for the science fair and clubs, another stated it would not 

be applicable for science fairs, because they happen only once a year. Two administrators indicated a 

preference for frequency-based response options, accompanied by a numeric scale.

Recommendations for Improvement

 There are no recommendations for improvement for this item based on the evidence collected from the 

interviews.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 87

2.3.3. Grade 12 Administrators

Grade 12 Science–School—Item 1

Comprehension

 Overall, the administrators did not exhibit any consistent problems with this item. One administrator 

indicated that he did not understand what the term “science specialist or coach” was meant to convey.

However, when asked to describe the role of a “science specialist or coach,” he indicated that it was a 

“person that helps with the content: an expert with a toolbox of resources to develop lesson plans, field 

trips, etc., and who observes classes to coach teachers on better practices.”

 All administrators gave similar definitions of a “science specialist or coach,” indicating that an individual 

in this role would be a subject matter expert who helps to coach and provide instructional support to 

teachers.

Response Process

 Two administrators answered “no,” explaining that although they did have an individual fulfilling the role 

of a “science specialist or coach,” they did not consider this person as being employed either full- or parttime. One of these administrators indicated that the “science specialist or coach” was available on an as 

needed basis, and the other explained that the “science specialist or coach” was not dedicated solely to 

12th grade.

 Two administrators selected the response option, “yes, available part-time.” One explained that the 

individual taught class in the morning and was available as a “science specialist or coach” in the 

afternoon. The other administrator explained that the role of the “science specialist or coach” was fulfilled 

by two individuals – one who works with more than one school, and another with a full teaching load, 

who also took on the responsibilities of a “science specialist or coach.”

 One administrator chose not to select any of the response options, indicating that none of these options 

fit his school’s situation. He further explained that although a resource teacher is available, this teacher 

is responsible for all high schools in the county, and that the 12th grade teachers in his school rarely ask 

for support from this individual.

 When asked if there is a different term used in their school for describing individuals with the 

responsibilities of a “science specialist or coach,” administrators noted a “resource teacher,” “science 

department chairperson,” and “science coordinator.”

New (Revised based on VE015623)

Is there a science specialist or coach available (full- or part-time) to twelfth-grade teachers at 

your school? 

A Yes, available full-time at my school. 

B Yes, available part-time at my school. 

C No 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 88

Recommendations for Improvement

 Although administrators were fairly unified in their definitions of the roles and responsibilities of a 

“science specialist or coach,” it may be helpful to provide a specific definition or additional descriptive 

terms.

 Overall, administrators felt that the response options did not fit their internally generated answer. The 

addition of a response option such as “yes, on an as-needed basis” may help to alleviate this difficulty.
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 89

Grade 12 Science–School—Item 2

Comprehension

 Overall, administrators were able to comprehend the item and sub-items. One administrator expressed 

confusion over the term “science enrichment” in sub-item “e.” However, when probed, his definition 

matched those of the other administrators.

 All administrators interpreted “translate science standards and current research into classroom practice”

as meaning the ability to understand and internalize science standards, and using that knowledge to 

create learning plans and objectives easily understood by students, and include appropriate vocabulary 

and teaching methods. When asked for terms that might be more meaningful for school administrators, 

one administrator suggested “curriculum review or standards based instructional development.” 

 When asked what “provide science enrichment to some student groups” meant to them, all 

administrators indicated that it meant that some student groups would receive more attention or different 

activities than the rest of the class, particularly opportunities that may go beyond the classroom. Most 

administrators (3 of 5) indicated that the enrichment activities would apply to students who excelled in 

the subject matter and wanted to expand their knowledge base, while two administrators also explained 

that “some student groups” could also be interpreted as students who may be struggling with the 

curriculum (such as those enrolled in remedial classes) and are in need of additional activities to help 

them catch up to their peers. 

Response Process

 All administrators felt that the sub-items covered the majority of responsibilities they would consider as 

parts of the role of science specialist or coach. Additional responsibilities mentioned by administrators 

included facilitating training regarding new technology, determining the best resource materials 

(textbooks, secondary documents, etc.) to use, and facilitating hands-on experiences and field trips.

New (Revised stem based on VE015624)

To what extent are any of the following a responsibility of the science specialist or coach available to 

twelfth-grade teachers at your school? Fill in one oval on each line. 

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Provide technical assistance/support to 

individual teachers about science content 

or the teaching of science

A B C D VE015625

b. Conduct professional development for 

groups of teachers about science content 

or the teaching of science

A B C D VE015626

c. Provide science instruction to students on 

various topics

A B C D VE015627

d. Translate science standards and current 

research into classroom practice A B C D

Revised 

sub-items 

based on 

VE015628

e. Provide science enrichment to some 

student groups

A B C D VE015629
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 90

 The two administrators who reported that they did not have a “science specialist or coach” seemed 

unsure about appropriate responses to the sub-items. However, when probed about the response 

options (not at all, small extent, etc.), all administrators felt that they were appropriate.

o One administrator mentioned that he would have replied “not applicable” for each of the subitems if that response option had been available, since he did not have a science specialist or 

coach on staff.

o One administrator suggested using numbers (1-4) to represent the response options, rather than 

letters (“A”-“D”).

Recommendations for Improvement

 Depending on the item’s intent, two alternatives could address administrators’ response process 

concerns. If the intent of the item is to measure to what extent these responsibilities are performed at the 

school, we recommend changing item so that it does not specifically identify a “science specialist or 

coach.” If the intent of the item is to ask specifically about the role of the science specialist or coach, we 

recommend a “not applicable” response option for those administrators who do not have one on staff. 
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 91

Grade 12 Science–School—Item 3

Comprehension

 All administrators interpreted this item as asking about science-related opportunities that the school 

provides beyond traditional classroom instruction.

 Two administrators indicated that the term “science museums” is unclear, since it could incorporate both 

traditional museums, such as a natural history museum, and locations like aquariums or other similar 

sites. Both of these administrators suggested that the sub-item should be clarified.

 Most administrators (3 of 5) agreed that there is a distinction between a “science fair” and a “science 

competition.”

o Two administrators characterized a “science fair” as an on-site exhibition where students 

demonstrate a science project or experiment to others.

o All administrators characterized a “science competition” as where students compete against 

each other; two of the five described it as more structured, where students may be given a 

certain question or task.

o One administrator did not differentiate between the two terms, explaining that in her school the 

“science fair” is a competition where students can win first place, second place, and honorable 

mentions.

Response Process

 When asked for examples of how the school provided opportunities for students, administrators 

mentioned robotics competitions, natural history museums, aquariums, environmental science clubs, 

annual science fairs, and electronic science fairs.

 When asked specifically about trips to science museums, administrators mentioned the National 

Aquarium, the Chesapeake Bay Foundation, the National Electronics Museum, the Smithsonian Natural 

History Museum, and the National Institutes of Health.

 Most administrators (3 of 5) expressed a preference for having response options with frequencies, such 

as certain times a week/a semester/a school year, explaining that it would be clearer and easier to 

New

To what extent does your school provide students with the following opportunities? Fill in one oval on each 

line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Science fairs A B C D New

b. Science clubs A B C D New

c. Science competitions A B C D New

d. Trips to science museums A B C D New
ETS CogLab Report: BQ Science 2013

Prepared by EurekaFacts, LLC 92

quantify their answers. One administrator indicated that the best option would be to base answers on a 

school year, since it would be harder to quantify in weeks.

 One administrator suggested adding ranges to qualify what was meant by a small, moderate and large 

extent. He suggested small extent would mean one to three times per year, moderate extent would 

mean four to seven times per year, and large extent would be eight or more.

Recommendations for Improvement

 Verbal reports and answers to probing questions show that the sub-items do not provide administrators 

with a chance to report all types of trips used for science classes. It may be helpful to specify how 

narrowly or widely the term “science museum” should be interpreted within the context of this item, and 

whether to include non-traditional museums (e.g., aquariums and arboretums). Alternately, additional 

sub-items could be provided to capture other types of trips that can be used for science instruction.

Lastly, most administrators seemed to have difficulty using the response options of “small, moderate and 

large extent,” explaining that the terms were subjective. Overall, administrators preferred frequencies in 

the response options instead. 
This page was intentionally left blank.
This page was intentionally left blank.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013

183

Appendices
184

Contents

Interview Protocol: Generic Probes and Think-Aloud Hints for the Interviewer...............................185

Science Student Questionnaire Interview Protocal: Items and Probes..............................................187

Recruitment and Operational Forms, Letters, and Scripts.................................................................188

Parent or Legal Guardian Consent Form.....................................................................................188

Student (age 18 or older) Consent Form.....................................................................................189

Teacher and School Administrator Consent Form......................................................................190

Phone Script Recruiting Parents or Legal Guardians of Student Participants ............................191

E-mail Recruiting Parents or Legal Guardians of Student Participants.......................................194

Letter Recruiting Parents or Legal Guardians of Student Participants.......................................195

Phone Script Recruiting Teachers and School Administrators....................................................196

E-mail Recruiting Teachers and School Administrators..............................................................199

Letter Recruiting Teachers and School Administrators...............................................................200

Email/Letter Confirmation to Parent or Legal Guardian of Student (under age 18) 

Participants..................................................................................................................................201

Email/Letter Confirmation to Student (age 18 or older) Participants........................................202

Thank You Letter to Parent or Legal Guardian of Student (under age 18) Participants.............203

Thank You Letter to Student (age 18 or older) Participants.......................................................204

Thank You Letter to Teachers and School Administrators..........................................................205
185

Interview Protocol: Generic Probes and Think-Aloud Hints for 

the Interviewer

If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging 

the participant to read the question aloud and say his/her thoughts, or to help him/her elaborate 

the responses.

• Please read the question aloud and tell me what you are thinking. 

• You mentioned __________, tell me more about that.

• You mentioned __________, what was that like for you?

We are interested in capturing all the participant’s mental processes while answering these 

questions. Your goal is to have the participant speak aloud all his or her thoughts while 

answering the questions. By asking follow-up probes after each item, you will ensure that the 

data collected are as complete as possible. Provide positive feedback by telling the participant 

when he or she has given helpful information. The participant may provide more detailed 

responses if given positive feedback such as: 

• This information is very helpful.

• The example you just gave is very useful.

• Thank you for providing such a thorough response. 

If a participant is continually providing short responses or not answering, use “continuers” to 

encourage the participant to be more descriptive. The goal is to get participants to verbalize their 

thoughts without “putting words in their mouth.” Do not lead the participant’s response by 

asking a question that implies or contains its own answer such as:

• Leading Question: Do you dislike science class like most students?

• Neutral Question: How do you feel about science class?

You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such 

as:

• What are you thinking now?

• Any other thoughts?

• Tell me how you came to pick that answer.

Use your best judgment. If a participant exhibits difficulty reading or understanding an item, 

probe the participant without biasing the response.

• Can you tell me in your own words what that question was asking?

[comprehension and interpretation probe]

• What does the word [term] mean to you as it’s used in this question?

[comprehension and interpretation probe]
186

Use your best judgment. If a participant appears to have guessed or seems less than certain of the 

answer he/she has given, probe the participant.

• How much would you say you know about [topic]? [confidence probe]

• How sure are you of your answer? [confidence probe]

• How did you come up with that answer? [recall strategy and bias probe]

• Was this easy or hard to answer? [comprehension and recall probe]

• How easy or difficult is it to remember [topic]? [recall probe]

• Did you find the list of answer choices easy or hard? [response probe]

• How easy or hard was it to choose an answer from that list of choices? [response 

probe]

Use your best judgment. If a participant appears uncomfortable with a word or answering a 

question, probe the participant.

• The question uses the word [term]. Does that sound OK to you, or would you 

choose something different? [sensitivity probe]

• In general, how do you feel about this question? [sensitivity probe]

Use your best judgment. After testing an item, using the above generic probes to understand why 

the question as written is not clear, you may reword the question in a way that the participant 

will understand based on shared interviewer feedback or previous interview experiences. If this 

is done, be sure to document precisely the reworded question that was tested and test the item 

using relevant generic probes.

In addition to generic probes, interviewer protocols will include item-specific probes. Not all 

items have an item-specific probe. Item-specific probes are provided only if the researchers 

deemed it important to learn more about a cognitive process or issue that might not be raised 

naturally from the think-aloud process or generic probing. If the issue to be inquired through the 

item-specific probe was addressed, or if time is running short, the item-specific probe does not 

need to be asked. Use your best judgment.
187

Science Student Questionnaire Interview Protocal: Items and 

Probes 

Interviewer Note: Proceed in the following manner for each of the survey items in this section:

• Direct the participant to the item and ask him/her to read the question (and any preceding 

or following instructions) out loud

• Instruct the participant to answer the question and to think aloud while doing so

• Record the participant’s comments

• Ask generic and item-specific probes, as appropriate, before going on to the next item

Note: Some items are grade specific and will be asked only of participants for that grade. The 

grade, subject, and respondent associated with the items are identified in brackets (e.g., [Grade 4 

Science–Student #1]) preceding the item.

Note: Each item to be tested is presented on its own page. 
188

Recruitment and Operational Forms, Letters, and Scripts

Parent or Legal Guardian Consent Form

Dear Parent or Legal Guardian: 

<DATE>

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of 

Education, Educational Testing Service and EurekaFacts are conducting voluntary research 

interviews with <fourth-/eighth-/twelfth-> grade students about survey questions for the National 

Assessment of Educational Progress (NAEP). Each student interview will take no more than one 

hour. The purpose of these interviews is not to test or grade students. NCES wants to hear their 

thoughts to improve the survey questions and make the questions easier to understand.

The interviewer will meet one-on-one with each student to ask for his or her input. These 

sessions will be recorded for the researchers on this study to review. No one else will see or hear 

the recording. Your child’s answers will be used only for statistical purposes and may not be 

disclosed or used, in identifiable form, for any other purpose except as required by law 

[Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. 

These interviews are an important stage of the NAEP survey development process. The 

comments students provide help NCES improve the survey questions to make the best survey 

possible. Your child will be given a $25 gift card as thanks for his or her time. In addition, you 

will be given a $25 gift card for your time to take the participating student to and from the 

interview site.

We hope that you will give your consent for your son or daughter to participate in the voluntary 

NAEP interview by signing this form. Without your consent, your child will not be able to 

participate in the interview.

Respectfully,

Laurie Policastro

NAEP Survey Development

Educational Testing Service

I hereby give my permission for _____________________________________________ to 

(Student’s Name)

participate in the NAEP survey interview.

____________________________________ ________________________

Signature Date
189

Student (age 18 or older) Consent Form

Dear Student: <DATE>

On behalf of The National Center for Education Statistics (NCES), part of the U.S. Department 

of Education, Educational Testing Service and EurekaFacts are conducting voluntary research 

interviews with students about survey questions for the National Assessment of Educational 

Progress (NAEP). Each student interview will take no more than one hour.

NCES needs to know if the survey questions are clear. Your feedback will help NCES improve 

the survey so students like you understand all the questions. You will be given questions that are 

very similar to the actual questions that will be asked during the national testing. The interviewer 

will pay very close attention to your opinions and feelings about all aspects of the survey. He or 

she will also ask what you think about the questions after you complete them.

You will not be judged on how well you answer the questions, and you will not receive a grade. 

Other students like you will also participate in one-on-one sessions just like this. This session 

will be recorded for the researchers on this study to review. No one else will see or hear the 

recording. Your answers will be used only for statistical purposes and may not be disclosed or 

used, in identifiable form, for any other purpose except as required by law [Education Sciences 

Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. Your 

comments, along with the comments of other students, will help NCES improve the survey. You 

will be given a $25 gift card in compensation for your time and effort. In addition, NCES is 

offering a gift card of $25 for a parent or legal guardian to remunerate him or her for the time 

involved and to help offset the travel/transportation costs of taking you to and from the 

EurekaFacts cognitive laboratory site. We thank you for your voluntary participation and ask 

you to please sign the form below. 

Thank you,

Laurie Policastro

NAEP Survey Questionnaires

Educational Testing Service

I hereby accept the terms described above and agree to participate voluntarily in the NAEP 

survey interview.

__________________________________

Print Name

____________________________________ ________________________

Signature Date
190

Teacher and School Administrator Consent Form

Dear Educator: <DATE>

On behalf of The National Center for Education Statistics (NCES), part of the U.S. Department 

of Education, Educational Testing Service and EurekaFacts are conducting voluntary research 

interviews with teachers and school administrators about survey questions for the National 

Assessment of Educational Progress (NAEP). Each teacher and school administrator interview 

will take no more than one hour.

NCES needs to know if the survey questions are clear. Your feedback will help NCES improve 

the survey so educators like you understand all the questions. You will be given questions that 

are very similar to the actual questions that will be asked of teachers and school administrators 

whose students participate in future NAEP assessments. The interviewer will pay very close 

attention to your opinions and feelings about all aspects of the survey. He or she will also ask 

what you think about the questions after you complete them.

Other educators like you will also participate in one-on-one sessions just like this. This session 

will be recorded for the researchers on this study to review. No one else will hear the recording, 

or see your responses or comments. Your answers will be used only for statistical purposes and 

may not be disclosed or used, in identifiable form, for any other purpose except as required by

law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. Your 

comments, along with the comments of other educators, will help NCES improve the survey. In 

addition, you will be given a $40 gift card in compensation for your time and effort. We thank 

you for your voluntary participation and ask you to please sign the form below.

Thank you,

Laurie Policastro

NAEP Survey Questionnaires

Educational Testing Service

I hereby accept the terms described above and agree to participate voluntarily in the NAEP 

survey interview.

__________________________________

Print Name

____________________________________ ________________________

Signature Date
191

Phone Script Recruiting Parents or Legal Guardians of Student Participants

Good morning/afternoon. May I please speak to <NAME>?

Good morning/afternoon. My name is <CONTACT NAME> and I am contacting you from 

EurekaFacts, a local research company. We are conducting an important study for the U.S. 

Department of Education’s National Center for Education Statistics (NCES). 

We are recruiting students to participate in voluntary research interviews supporting the 

development of survey questions for the National Assessment of Educational Progress (NAEP). 

The survey will be administered to students throughout the United States. The research 

interviews will explore how students work through the survey questions to ensure that they are 

clear and understandable to a broad range of students. The students who qualify for and 

participate in the study will be given a $25 gift card as thanks for their time. In addition, you will 

be given a $25 gift card for your time to take the participating student to and from the interview 

site.

The interviews will take place at the EurekaFacts cognitive laboratory site in Rockville, 

Maryland and will last no more than one hour. During the interview, your child will be asked to 

respond to several survey questions. NCES simply wants to know how he or she would answer 

the questions if they were included on an actual survey given to students after taking a NAEP 

test. 

These interviews are an important stage of the NAEP survey development process. The 

comments students provide help NCES improve the survey questions in order to create the best 

survey possible.

Would your child be interested in participating in this research project?

[If No] Thank you for taking the time to speak with us today about this important project. Have a 

wonderful day/evening.

[If Yes] Great, thank you very much. To ensure that we interview a broad mix of students, I have 

a few brief screening questions to ask about you and your child. It should only take a few 

minutes of your time. First, I want to collect some additional information about your child to be 

certain he or she is a good fit for participating. 

Screener

[Interviewer: Determine recruitment needs in advance. Record all responses. End screener at 

whatever point the parent or legal guardian gives a response that confirms that their group has 

already been sampled adequately. End screener script: Based on the requirements of this study, 

we are not able to include your child in the interviews at this time.]
192

1. What is your child’s name?

2. Is <STUDENT NAME> a boy or a girl?

3. What grade is <STUDENT NAME> currently in? (If the student is in twelfth-grade, ask 

if he or she is age 18 or over. This information will be used later in the script.)

4. What is your child’s race or ethnicity? [Interviewer: Only read list if parent doesn’t 

immediately reply] Are you...?

a. American Indian or Alaska Native

b. Asian

c. Black or African American

d. Hispanic or Latino

e. Native Hawaiian or other Pacific Islander

f. White or Caucasian

5. What is the highest level of education that you have completed? [Interviewer: Do not read 

this list].

a. Less than high school degree

b. High school degree or GED

c. Some college

d. Trade school degree

e. Associate’s or Bachelor’s degree, or

f. Post graduate degree?

6. Just to be sure we’re interviewing a broad range of students for this study, what is your 

annual household income?

If the respondent indicates they prefer not to disclose this information, the interviewer 

should say “That’s fine”, and move on to the next question.

7. What is <STUDENT NAME’s> current zip code?

8. And has <STUDENT NAME> said <SHE, HE> would be interested in participating in 

the interview session which will take no more than one hour?

Eligible Student Participant Script

Thank you for taking the time to answer the screening questions. Your child is eligible to 

participate in the research interviews. 

In the session, we will be asking <STUDENT NAME> to respond to several survey questions. 

NCES needs to make sure that students will understand what the survey is asking them to do. 

Participation is entirely voluntary, and your child’s answers will only be used only for statistical 

purposes and may not be disclosed, or used, in identifiable form for any other purpose except as 
193

required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. 

Interviewer note: this reference is for informational purposes only. If asked what law is being 

referenced, please provide the information contained within this bracketed text.]

Your child’s input will help NCES improve the survey that will be given nationwide. To thank 

<STUDENT NAME>, <SHE, HE> will receive a $25 gift card. In addition, you will receive a 

$25 gift card for your time to take the <STUDENT NAME> to and from the interview site.

Again, the interview will last no more than one hour and will be held at the EurekaFacts 

cognitive laboratory site in Rockville, Maryland. Do you think <STUDENT NAME> would like 

to participate?

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes (and the student is under age 18)] And would you be willing to submit a signed form onsite, indicating your approval for <STUDENT NAME> to participate? 

[If No] Thank you for taking the time to speak with us today about this important project.

[If Yes] Are you and your child available for the interview at <TIME> on <DATE>?

[If the student is age 18 or over)] Is <STUDENT NAME> available for the interview at <TIME> 

on <DATE>?

[Interviewer: if necessary, provide alternate times/dates to find an available time/date for 

interview and then record time/date of appointment]

Thank you. We will send you a confirmation e-mail containing the address, directions and 

consent form. Please bring the signed consent form to the interview. What e-mail and mailing 

address would you prefer us to use to send you this information? We may need to reach you by 

telephone. Is the number I called the best number to use?

[If the student is age 18 or over)] Thank you. We will send <STUDENT NAME> a confirmation 

e-mail containing the address, directions, and consent form that <HE/SHE> must be sign. What 

email address would be best to send this contact information to <STUDENT NAME> at? We 

may need to reach <STUDENT NAME> by telephone. Is the number I called the best number to 

use?

Do you have any questions at this time?

[Interviewer: Answer any questions.]

If you or <STUDENT NAME> have any questions before the interview scheduled for 

<DATE/TIME>, please call <PHONE NUMBER>.

We look forward to having <STUDENT NAME> participate in this very important research 

project.
194

E-mail Recruiting Parents or Legal Guardians of Student Participants

Subject: Research Study for Students in Fourth-Grade, Eighth-Grade, or Twelfth-Grade

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research

company. We are conducting an important study for the U.S. Department of Education’s 

National Center for Education Statistics (NCES). 

We are recruiting <fourth-/eighth-/twelfth-> grade students to participate in voluntary research 

interviews supporting the development of survey questions for the National Assessment of 

Educational Progress (NAEP) assessment. The interviews will explore how students work 

through the survey to ensure that the questions are clear and understandable to a broad range of 

students. Your child will be given a $25 gift card as thanks for his or her time. In addition, you 

will be given a $25 gift card for your time to take the participating student to and from the 

interview site. Students age 18 or older have the option to attend the interview without being 

accompanied by a parent or legal guardian. 

The interview for your child will take place at the EurekaFacts cognitive laboratory site in 

Rockville, Maryland and will last no more than one hour. The interview can be scheduled at the 

convenience of you and your child. During the interview, your child will be asked to respond to 

several survey questions. NCES simply wants to know how he or she would answer the 

questions if they were to be included in the NAEP assessment. Your child’s responses will be 

used only for statistical purposes and may not be disclosed, or used, in identifiable form for any 

other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 

20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. By sharing 

their feedback with us, students can directly impact the future of this national assessment by 

helping NCES evaluate the survey to create the best questions possible. 

If your child is interested in participating in this research study, please call <CONTACT 

PHONE> or e-mail <CONTACT E-MAIL>. We look forward to speaking with you soon!

Thank you,

<NAME>

EurekaFacts

<CONTACT PHONE>
195

Letter Recruiting Parents or Legal Guardians of Student Participants

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research 

company. We are conducting an important study for the National Center for Education Statistics 

(NCES), part of the U.S. Department of Education.

We are recruiting <fourth-/eighth-/twelfth-> grade students to participate in voluntary research 

interviews supporting the development of survey questions for the National Assessment of 

Educational Progress (NAEP) assessment. Before giving the survey to all students, NCES needs 

to know if the survey questions are clear. Your input will help NCES improve the survey so 

students like your child can better understand the survey questions. 

The interviews will take place at the EurekaFacts cognitive laboratory site in Rockville, 

Maryland and will last no more than one hour. Your child will be asked to respond to several 

survey questions. NCES simply wants to know how he or she would answer the questions if it 

were an actual survey that was given to students after taking a NAEP test. Your child’s answers 

will be used only for statistical purposes and may not be disclosed, or used, in identifiable form 

for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 

2002) 20 U.S.C., § 9573].

These interviews are an important stage of the NAEP survey development process. The 

comments students provide help NCES improve the survey questions to make the best survey 

possible. Your child will be given a $25 gift card as thanks for his or her time. In addition, you 

will be given a $25 gift card for your time to take the participating student to and from the 

interview site. Students age 18 or older have the option to attend the interview without being 

accompanied by a parent or legal guardian.

If your child is interested in participating in this research project, please call [phone number] or 

e-mail [e-mail address]. We look forward to speaking with you soon!

Thank you,

<NAME>

EurekaFacts

<CONTACT PHONE>
196

Phone Script Recruiting Teachers and School Administrators

Good morning/afternoon. May I please speak to <NAME>?

Good morning/afternoon. My name is <CONTACT NAME> and I am contacting you from 

EurekaFacts, a local research company. We are conducting an important study for the U.S. 

Department of Education’s National Center for Education Statistics (NCES). The National 

Assessment of Educational Progress (NAEP) has developed survey questions for teachers and 

school administrators. Before giving the survey to all teachers and school administrators whose 

students participate in a NAEP assessment, NCES needs to know if the survey questions are 

clear. Your input will help NCES improve the survey so educators like you understand all the 

questions and the things you are being asked to do. To thank participants for their time, each 

participant will receive a $40 gift card.

The voluntary research interviews will take place at <LOCATION> and will last no more than 

one hour. You will be asked about the survey questions and to respond to several questions. 

NCES simply wants to know how you would answer them if it were an actual survey. NCES 

wants to hear your thoughts to improve the survey questions and make the questions easier to 

understand. Your responses will be used only for statistical purposes and may not be disclosed 

or used, in identifiable form, for any other purpose except as required by law [Education 

Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573]. Interviewer note: this reference is 

for informational purposes only. If asked what law is being referenced, please provide the 

information contained within this bracketed text.] 

These interviews are an important stage of the NAEP survey development process. Your 

comments, along with the comments of other teachers and school administrators, will help NCES 

improve the survey. 

Are you interested in participating in this research project? 

If no: Thank you for your time. Have a wonderful day/evening.

If yes: Great, thank you very much. To ensure that we interview a broad mix of educators, I have 

a few brief screening questions to ask you. It should only take a few minutes of your time.

The interview will last no more than one hour. It will take place at <LOCATION>.

Screener

[Interviewer: Determine recruitment needs in advance. Record all responses. End screener at 

whatever point the teacher or administrator gives a response that confirms that their group has 
197

already been sampled adequately. End screener script: Based on the requirements of this study, 

we are not able to include you in the interviews at this time.]

What is your job title?

a. Administrator (GO TO Part A)

b. Teacher (GO TO Part B)

Part A.

1. At what school are you an administrator?

2. Are you at a school with a population that includes: 

a. Fourth-grade students (recruit ~5)

b. Eighth-grade students (recruit ~5)

c. Twelfth-grade students (recruit ~5)

3. What is the size of the student population at your school?

a. 299 students or less (recruit ~5)

b. 300-599 students (recruit ~5)

c. 600 students or more (recruit ~5)

Part B.

1. At what school are you a teacher?

2. What grade level do you teach?

a. Fourth grade (recruit ~5)

b. Eighth grade (recruit ~5)

c. Other END SCREENER (Based on the requirements of this study, we are not 

able to include you in the interviews at this time. We greatly appreciate your help. 

Thank you for your time.)

3. Do you currently (or within this school year) teach science to eighth/fourth graders?

a. Yes  CONTINUE

b. No END SCREENER (Based on the requirements of this study, we are not 

able to include you in the interviews at this time. We greatly appreciate your help. 

Thank you for your time.)

4. What is the size of the student population at your school?

a. 299 students or less (recruit ~5)

b. 300-599 students (recruit ~5)

c. 600 students or more (recruit ~5)
198

Eligible Participant Script

Thank you for taking the time to answer the screening questions. You are eligible to participate 

in the research interviews. Again, the interview will last about 1 hour and will be held at 

<LOCATION>. Are you available for the interview at <TIME> on <DATE>?

[Interviewer: if necessary, provide alternate times/dates to find an available time/date for 

interview and then record time/date of appointment]

Thank you. 

We may need to reach you by telephone. Is the number I called the best number to use?

Do you have any questions at this time?

[Interviewer: Answer any questions.]

If you do have any questions before your interview scheduled for <DATE/TIME>, please call 

<PHONE NUMBER>.

Thank you again for agreeing to be part of this very important research project.
199

E-mail Recruiting Teachers and School Administrators

Subject: Research Study for School Administrators 

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research 

company. We are conducting an important study for the U.S. Department of Education’s 

National Center for Education Statistics (NCES). 

We are recruiting teachers and school administrators to participate in voluntary research 

interviews supporting the development of survey questions for the National Assessment of 

Educational Progress (NAEP) assessment. 

The interviews will explore how teachers and school administrators work through the survey to 

ensure that the questions are clear and understandable to a broad range of educators. To thank 

participants for their time, each participant will receive a $40 gift card. 

The interviews will take place at <LOCATION> and will last no more than one hour. The 

interview can be scheduled at your convenience. Our interviewer will arrange to meet with you 

in-person. During the interview, you will be asked to respond to sample survey questions. NCES 

wants to hear your thoughts to improve the survey questions and make the questions easier to 

understand. Your responses will be used only for statistical purposes and may not be disclosed, 

or used, in identifiable form for any other purpose except as required by law [Education Sciences 

Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

These research interviews are an important stage of the NAEP survey development process. 

Your comments, along with the comments of other educators, will help NCES improve the 

survey. 

If you are interested in participating in this research study, please call <CONTACT PHONE> or 

e-mail <CONTACT E-MAIL>. We look forward to speaking with you soon!

Thank you,

<NAME>

EurekaFacts

<CONTACT PHONE>
200

Letter Recruiting Teachers and School Administrators

Dear <NAME>:

My name is <CONTACT NAME> and I am contacting you from EurekaFacts, a local research 

company. We are conducting an important study for the National Center for Education Statistics 

(NCES), part of the U.S. Department of Education.

We are conducting voluntary research interviews with teachers and school administrators about 

survey questions for the National Assessment of Educational Progress (NAEP). Before 

administering the survey questions, NCES needs to know if the survey questions are clear. Your 

input will help NCES improve the survey so educators like you will understand all the questions. 

The interviews will take place at <LOCATION> and will last no more than one hour. You will 

be asked to respond to survey questions. NCES simply wants to know how you would answer 

them if it were an actual survey. NCES wants to hear your thoughts to improve the survey 

questions and make the questions easier to understand. Your responses will be used only for 

statistical purposes and may not be disclosed or used, in identifiable form, for any other purpose 

except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 

9573]. To thank you for your time, you will receive a $40 gift card.

These interviews are an important stage of the NAEP survey development process. Your 

comments, along with the comments of other educators, will help NCES improve the survey. 

If you are interested in participating in this research project, please call [phone number] or e-mail 

[e-mail address]. 

Thank you,

<NAME>

EurekaFacts

<CONTACT PHONE>
201

Email/Letter Confirmation to Parent or Legal Guardian of Student (under 

age 18) Participants

IF SENT AS E-MAIL - Subject: Confirmation of NAEP Research Interview 

<DATE>

Dear <PARENT NAME>,

Thank you for agreeing to allow your child, <STUDENT NAME>, to participate in this 

voluntary research interview to provide feedback on survey questions for the National 

Assessment of Educational Progress (NAEP). All the information your child provides will be 

used only for statistical purposes and may not be disclosed, or used, in identifiable form for any 

other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C 

§9573]. 

Your child has been scheduled for a research interview at <TIME> on <DATE>. The interview 

will last no more than one hour. To thank your child for participating, <he/she> will receive a 

$25 gift card. In addition, you will receive a $25 gift card to thank you for your time in bringing 

your child to and from the interview site.

The interview will take place at the EurekaFacts cognitive laboratory site at: 

<ADDRESS>

<DIRECTIONS>

We have <attached/included> a consent form <to/with> this confirmation. Please sign, date, and 

return the form when you bring your child to the interview, granting consent for your child to 

participate in the NAEP research. Without your consent, your child will not be able to participate 

in the interview.

We look forward to having <STUDENT NAME> participate in this very important research 

project.

If you or your child has any questions, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>

EurekaFacts

<CONTACT PHONE>
202

Email/Letter Confirmation to Student (age 18 or older) Participants

IF SENT AS E-MAIL - Subject: Confirmation of NAEP Research Interview 

<DATE>

Dear <STUDENT NAME>,

Thank you for agreeing to participate in this voluntary research interview to provide feedback on 

survey questions for the National Assessment of Educational Progress (NAEP). All the 

information you provide will be used only for statistical purposes and may not be disclosed, or 

used, in identifiable form for any other purpose except as required by law [Education Sciences 

Reform Act of 2002, 20 U.S.C §9573]. 

You have been scheduled for a research interview at <TIME> on <DATE>. The interview will 

last no more than one hour. To thank you for participating, you will receive a $25 gift card. In 

addition, your parent or legal guardian will receive a $25 gift card to remunerate him or her for 

the time involved and to help offset the travel/transportation costs of taking you to and from the 

EurekaFacts cognitive laboratory site. 

The interview will take place at the EurekaFacts cognitive laboratory site at: 

<ADDRESS>

<DIRECTIONS>

We have <attached/included> a consent form <to/with> this confirmation. Please sign, date, and 

return the form when you come to the interview, agreeing to participate in the voluntary NAEP 

research. 

We look forward to having you participate in this very important research project. If you have 

any questions, please contact <CONTACT INFORMATION>.

Sincerely,

<NAME>

EurekaFacts

<CONTACT PHONE>
203

Thank You Letter to Parent or Legal Guardian of Student (under age 18) 

Participants

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of 

Education, I would like to thank you for allowing your child to participate in our survey research 

study. Your child’s input will help us develop the best possible survey questionnaires for the 

National Assessment of Educational Progress (NAEP). 

The assistance provided by your child helped us to better understand how students approach the 

survey questions. Your child’s input, combined with input from other students, is incredibly 

useful and informative. 

Again, thank you for your assistance. 

Sincerely,

<NAME>

EurekaFacts

<CONTACT PHONE>
204

Thank You Letter to Student (age 18 or older) Participants

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of 

Education, I would like to thank you for your participation in our survey research study. Thanks 

to you, we are better able to develop the best possible survey questionnaires for the National 

Assessment of Educational Progress (NAEP). 

Your assistance helped us better understand how students approach the survey questions. Your 

open, candid, and insightful comments are very helpful.

Again, thank you for your assistance. 

Sincerely,

<NAME>

EurekaFacts

<CONTACT PHONE>
205

Thank You Letter to Teachers and School Administrators

<DATE>

Dear <Name>:

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of 

Education, I would like to thank you for your participation in our research study. Thanks to you, 

we are better able to develop the best possible survey questionnaires for the National Assessment 

of Educational Progress (NAEP). 

Your assistance helped us better understand how <teachers/school administrators> approach the 

survey questions. Your open, candid, and insightful comments were very valuable. Your input, 

combined with input from other <teachers/school administrators> like you, is incredibly useful 

and informative. 

Again, thank you for your assistance. 

Sincerely,

<NAME>

EurekaFacts

<CONTACT PHONE>
Appendix C

2. List of New or Revised Items

Questionnaire Grade New Pilot Items

Science Student 

K1SXB1 4 3, 6, 16, 19, 27, 28, 29 

Science Student 

K2SXB1 8 2a-g, 3b-d, 5g, 8, 11a-c, 12, 13

Science Student 

K3SXB1 12 7a-g, 8b-d, 10g, 13, 15a-c, 16, 17

Science Teacher 

K1TQ-COISX 4 2, 5a-c, 9f, 9j, 10a-l, 13c, 16a-c

Science Teacher 

K2TQ-COISX 8 2, 5a-c, 9f, 9j, 10a-l, 14c, 17a-c

Science School 

K1SQ-SX 4 1, 2a-d, 3, 4a-c, 9a-b, 11a-c, 12a-c

Science School 

K2SQ-SX 8 1, 2, 3, 4a-c, 9a-b, 11a-c, 12a-c

Science School 

K3SQ-SX 12 4, 5a-d, 6, 7a-c, 9a-b, 11a-c, 12a-c

Core Teacher BET 

K1TQ-BETCGHS 4 13a-l

Core Teacher BET 

K2TQ-BETS 8 4

Note: Revised items are considered “new” items because they have not been administered in 

previous pilot or operational questionnaires.