Appendix F

NIES Cognitive Interview Reports
Listening.

Learning.

Leading.

NAEP Item Development (ID)

Survey Questionnaires

Cognitive Interview Study Report and 

Recommendations for Operational

2015 National Indian Education Study

Student, Teacher, and School Questionnaires

Grades 4 and 8

Deliverable in response to ID Task 3.2.3

Submitted: May 2, 2013
Prepared by Kauffman & Associates, Inc.

Table of Contents

Cognitive Interviews Report for NIES CogLab...............................................................................................3

Sample...........................................................................................................................................................3

Procedures....................................................................................................................................................3

Analytic Approach.........................................................................................................................................4

Survey Instrument: Student Grade 4/Sequence I.........................................................................................5

Survey Instrument: Student Grade 4/Sequence II......................................................................................33

Survey Instrument: Student Grade 8/Sequence I.......................................................................................57

Survey Instrument: Student Grade 8/Sequence II......................................................................................83

Survey Instrument: Teacher Grade 4/Sequence I.....................................................................................110

Survey Instrument: Teacher Grade 8/Sequence I.....................................................................................133

Survey Instrument: Administrator Grade 4/Sequence I ...........................................................................165

Survey Instrument: Administrator Grade 8/Sequence I ...........................................................................187

Note: This report was prepared by Kauffman & Associates, Inc. (submitted to ETS on April 28, 2014). For 

each item, an ETS recommendation was added to the report based on an evaluation of the findings 

summarized by Kauffman. ETS recommendations also consider overall student burden, in addition to 

item performance in cognitive interviews. ETS recommendations are highlighted in yellow. For more 

detailed information on suggested item wordings and overall item counts, please see RoD.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 3

Cognitive Interviews Report for NIES CogLab

Date of Report: April 28, 2014

Interview Dates: December 11, 2013 – March 18, 2014

Location of Interviews: Arizona, California, Idaho, Maryland, Minnesota, Mississippi, Nevada, New 

Mexico, North Carolina, Oklahoma, Oregon, South Dakota, and Washington.

Sample

Kauffman & Associates, Inc. (KAI) recruited a total of 159 individuals for participation in cognitive 

interviews during which participants completed the background questions for the 2015 National Indian 

Education Study (NIES). The demographics of the participants were as follows:

Table 1: Sample Population Demographics

Respondent Category Total

4

th Grade Students 49

8

th Grade Students 44

4

th Grade Teachers 22

8

th Grade Teachers 20

4

th Grade Administrators 12

8

th Grade Administrators 12

Procedures

As part of the cognitive interview recruitment process, KAI attended five conferences with 

representation from across the United States from October to December 2013. KAI’s outreach strategy 

included sending emails to conferences attendees who indicated an interest in participating in, or 

knowing more about, the cognitive labs after OMB approval had been granted and working through 

these initial contacts to assess potential participants. 

In addition to leveraging contacts made during conferences, KAI also engaged in direct outreach to 

directors of tribal education departments and Title VII Directors around the United States. Through 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 4

these contacts, KAI secured the interest of schools who were asked a series of eligibility questions using 

the sampling guidelines agreed upon between the National Center for Education Statistics, KAI, and ETS. 

Schools were additionally asked to have consent forms signed by parents or guardians prior to KAI 

interviewers arriving at the school. 

At the end of the sessions, KAI staff asked participants to reflect on their overall impressions of the 

cognitive interview. KAI interviewers then thanked and remunerated the participant who was then 

asked to sign a receipt for their incentive payment.

The cognitive interviews were scheduled for a duration of 60 minutes. The items for both 4th and 8th

grade student questionnaires were split into two sequences per grade level and administered to at least 

20 students per sequence. All 4th grade teachers received the same sequence and all 8th grade teachers 

received the same sequence. Further, all 4th grade administrators responded to the same sequence and 

all 8th administrators responded to the same sequence. 

Analytic Approach

This report presents findings directly associated with the 159 cognitive interviews conducted by KAI. The 

analysis is based upon both the responses given par interview participants as well as from the 

observances from KAI interview staff regarding the comfort and demeanor of participants throughout 

the interview process. 

The analysis of the data collected was qualitative in nature and therefore based on drawing out themes 

from participant responses and counting frequency of response occurrence as well as reporting out on 

anomalous data. Due to the nature of qualitative data sometimes response rates equal more than the 

number of respondents, particularly if multiple themes were present in a single participant’s response. 

Further, several probe questions in the interview instruments were conditional so not all respondents 

answered; some respondents chose not to respond to certain questions; and there are inevitable 

skipped questions at times where interviewers missed asking a question. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 5

Survey Instrument: Student Grade 4/Sequence I

Item 1

Reading: Two respondents had some trouble reading the item. One misread “please specify” as “please 

speak clearly,” and another had trouble with the words “who,” “representatives,” and “specify.”

Understanding: Two respondents seemed to have trouble with “Alaska”; one said they didn’t 

understand what Alaska was, another said they didn’t understand what an Alaska Native was, and a 

third kept referring to Africa though they seemed to be referring to Alaska. A fourth respondent seemed 

confused by what “AI/AN history” meant.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Four students out of 23 respondents seemed to misunderstand the question based on answers ranging 

from “I don’t know” to non sequitur responses to the question; the other 19 clearly restated the 

question. 

Probe 3: What does the phrase “American Indian or Alaska Native history” mean to you? 

Can you give me examples of “American Indian or Alaska Native history” that you have learned?

There was confusion for the 22 respondents around Probe 3 with one respondent speaking to issues in 

Africa (ostensibly meaning Alaska), and another asked what Alaska and history were. Answers that 

seemed out of place were from two respondents who answered “I don’t know,” one who responded 

“pretty much nothing,” and two others who made references to language as AI/AN history. The other 15 

answers demonstrated a clear understanding of the question. 

Probe 4: In answer choice B, what does the phrase “Family members” mean to you?

Of the 22 respondents for this question, most noted specific family members; several spoke to broader 

relationships of interdependence, for instance saying that the tribe was their family.

Student #1 New/VH039733 

Who taught you most of what you know about American Indian or Alaska Native history?

A. No one has taught me about American Indian or Alaska Native history.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders 

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 6

Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? 

Can you give me examples of “Tribal representatives or elders”?

“Elders” was relatively easy for the 22 respondents to pinpoint and identify as people older than 

themselves or with more knowledge. “Tribal representatives” presented a bit of a challenge for most 

respondents who either ignored the question or said that they did not know what a tribal representative 

was. Many respondents also conflated the terms of “tribal representative” and “elder,” perhaps due to 

the fact that they were presented in one question together.

Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

The 10 responses to this question were inconsistent and questionable with regard to their validity. One 

respondent felt that their father was a tribal representative as he worked at the college. Another noted 

that their grandfather was very involved in powwows, though his position was never clarified. Of the six 

respondents who replied in the affirmative to this question, only the two aforementioned respondents 

gave specific reasons as to why they considered their family members tribal representatives. Two

indicated that they did not have family members who were tribal representatives. 

Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

Only one respondent chose answer choice F and named their parents as the “someone else” who had 

taught them most of what they knew about AI/AN history.

Probe 8: What does the phrase “most of what you know” mean to you?

Out of 23 respondents, one said “nothing,” one responded “I don’t know,” and another said, “I don’t 

understand this question.” All other (21) responses were appropriate and valid. 

Probe 9: Which answer choice would you select if multiple people had taught you about American 

Indian or Alaska Native history? How would you decide which answer choice to select?

The validity of the answers proffered a lot of variety among the 18 respondents. Seven suggested they 

would choose a different answer entirely. None appropriately responded to being asked how they 

would decide the best answer in a hypothetical situation, though five gave explanations as to why they 

chose their original answer.

Solutions to Improve Item

Respondent: NA

Interviewer: Interviewers consistently noted that hypothetical questions (such as Probe 9) posed a 

problem for many respondents, and this was particularly true for the 4

th grade interviews.

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally seemed to understand and be comfortable with answering the question.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 7

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 8

Item 3

Reading: One respondent misread “please specify” as “please speak clearly”; another skipped the word 

“Native” in Alaska Native.

Understanding: Only one student said that they did not know what the question was asking.

Changing Answers: Of 23 respondents, two chose multiple answer choices for the item.

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Only one student out of 22 respondents reported that they did not know what the item was trying to 

ask. 

Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you? 

Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have 

learned?

Twenty of the 23 students gave valid responses to the questions. One student said that they did not 

know, another said that this question was the same as the last, and a third student referenced 

storytelling—which could be considered an art form but might also indicate some conflagration with the 

previous question about history. 

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

This probe seemed to be confusing for the five students who responded to this probe. Answers ranged 

from simply stating one family member to mentioning several. One student recounted, “I don’t really 

know because when my grandma still has to take me to my other family members too.”

Student #3 New/VH040030

Who taught you most of what you know about American Indian or Alaska Native arts and 

crafts? 

A. No one has taught me about American Indian or Alaska Native arts and crafts.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders 

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 9

Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents seemed to understand the item question and answer it 

appropriately.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 10

Item 8

Reading: NA

Understanding: Given the inconsistency of responses across the various probes, it would seem that 

most respondents did not truly understand what this question was asking as there was no clear 

explanation for what respondents interpreted “systems of counting” to mean.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Two out of 23 respondents interpreted the question as having to do with whether or not one could 

count in an AI/AN language. Fourteen students essentially repeated the question, though one thought 

that perhaps the question had to do with the invention of computers. Seven respondents essentially 

reported that they did not know what the question was asking; one of the seven did not even choose an 

answer for the item due to not understanding. 

Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you? 

Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have 

learned?

Eight students of 22 respondents indicated that they did not know what AI/AN “systems of counting”

meant. Four thought it had to do with whether or not one could count in a Native language. Three felt it 

had to do with how AI/AN people kept track of their populations. And four students essentially repeated 

the words of the question. Other responses varied in their guesses to the meaning. 

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

Of the 13 respondents who answered, five students said they did not know, two thought it had to do 

with language, one thought it had to do with populations, one referenced basket weaving, at least two 

answers were confusing regarding what the respondent was trying to express, one respondent spoke

about technology 50 years ago, and one respondent talked about their teacher having them count on 

their hands. 

Student #8 New/VH040093

How much do you know about American Indian or Alaska Native systems of counting?

A. Nothing

B. A little

C. Some

D. A lot
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 11

Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

All 20 respondents who provided answers found the choices clear. One student noted that although 

they found the answers clear, A was the exception, saying “I don’t know that if anybody knows or not.”

Solutions to Improve Item

Respondent: NA

Interviewer: Through team discussions, this question came up often as one that was confusing both to 

students as well as to interview staff. No one seemed to be clear about what AI/AN systems of counting 

was referring to, something that should be clarified if this question is to be used in the 2015 NIES study.

Initial KAI Recommendation (Keep/Change/Discard)

Discard: If there is not a clear definition of what systems of counting entails, then the recommendation 

is to drop this question.

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS 

recommends revising the phrase to "ways of counting".

This recommendation is aligned with the similar grade 4 item (VH040103).
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 12

Item 9

Reading: NA

Understanding: It is hard to say how well students truly understood the question as many of them 

repeated the question. Some claimed they did not know what the question was trying to ask, some 

thought it had to do with language, others thought it was referencing AI/AN history or culture, others 

thought it referred to general teaching (not specific to a particular topic), and one thought it was the 

same as the previous question.

Changing Answers: No one changed their answers, but one student chose two answers.

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Two out of 23 responding students said that they did not know what the question was asking, 11 

essentially parroted back the question to the interviewer, two students felt that it had to do with being 

able to count in a Native language, four thought the question was referring to AI/AN culture or history, 

three seemed to think that the question had to do with who had taught them the most (did not seem 

specific to AI/ANs), and one student asked if this was a repeat of the previous question. 

Probe 3: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you 

first think of them as your family members or as tribal representatives?

Only one student was eligible to answer this conditional probe and they named their grandmother and 

uncle as tribal representatives but did not offer what their positions in the tribe were. 

Probe 4: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

Student #9 New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of 

counting?

A. No one has taught me about American Indian or Alaska Native systems of counting.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 13

Only two students were eligible to answer this conditional probe. The first said “the people that come 

to our school and teach us about it. I have no idea what they’re called. Most of them are dancers.” The 

other replied that the “someone else” they were thinking of was their grandma.

Solutions to Improve Item

Respondent: NA

Interviewer: Again, a specification about what is meant by “systems of counting” would be critical in 

making the meaning of this question clearer.

Initial KAI Recommendation (Keep/Change/Discard)

Discard: Given the current confusion about the meaning of the phrase, it is the suggestion of the 

interviewers to discard the question.

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS 

recommends revising the phrase to "ways of counting".

This recommendation is aligned with the similar grade 4 item (VH040093).
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 14

Item 10

Reading: One student misread “systems” as “single.”

Understanding: Six respondents were clearly confused about what the question was trying to ask and 

either said that they did not know, or gave responses that did not make sense in the context of the 

question.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Responses varied among the 23 respondents, one said they did not know, one said “no” and 21 others 

repeated sections of the question but did not seem to grasp what the question was trying to elicit. 

Probe 3: What does the phrase “American Indian or Alaska Native symbols and designs” mean to you?

Answers from the 23 respondents varied greatly to this question. Only one respondent mentioned math 

in their response. Other answers ranged from punctuation to arts and crafts. Five students expressed 

not knowing what the question was trying to elicit, and the rest gave answers that have very little 

validity as they were widely disparate in their interpretation of what was meant by “symbols and 

designs,” and as noted before, only one student tied their answer back to math.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of “American Indian or Alaska Native symbols and designs” that 

you know about?

Only three of the 13 respondents referenced math, one noting that they were unsure how symbols and 

designs were used in math but did point out how they may be used in decoration or communication; 

and the other giving the following answer: “American Indian, basically my whole class, we just know that 

when you write the symbols in math that if you’re from somewhere else, it’s a new kid you don’t know 

and they could be from anywhere. If they don’t know how to do their Indian math that there would be a 

whole class and they would have to decide how would you write the math symbols down.” Six gave 

examples referring to arts, crafts, and modes of communication, one thought the question had to do 

with punctuation, one referenced animals, and four claimed that they did not know.

Student #10 New/VH040098

How much do you know about American Indian or Alaska Native symbols and designs used in 

math?

A. Nothing

B. A little

C. Some

D. A lot
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 15

Probe 5: Do you think that answer choices A through D are clear? If not, can you give me examples of 

other answer choices that would be more clear?

Eighteen of 19 respondents felt the answers were clear; one felt that they weren’t and gave the 

following explanation: “No. If they make them like, the X for times, the plus for plus or add, and

subtract, a minus. A minus with two on top and on bottom of it for division. That's how we do it in 

English.”

Solutions to Improve Item

Respondent: One respondent made the following suggestion: “I think they could’ve asked it more 

clearer because it’s not all just used in math, it’s used in different stuff, so maybe they could have just 

said, ‘How much do you know about American Indian or Alaska Native symbols and designs used in 

anything?’ maybe.”

Interviewer: Perhaps the category of symbols and designs could be expanded beyond math as most 

students seemed to think about symbols and designs as decoration or communication.

Initial KAI Recommendation (Keep/Change/Discard)

Change: Given how most students interpreted the question having to do with arts, crafts, or forms of 

communication, perhaps the question could ask about that. The relevance to math made little sense to 

respondents and if this is the only purpose of the question, it is the suggestion of the interviewer team 

that it be removed from the study.

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS agrees with KAI that this item should be dropped as the specific relevance to math made 

little sense to respondents. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 16

Item 11

Reading: NA

Understanding: Three respondents were noted as having difficulty understanding the item due to their 

responses; one seemed to be equating symbols and designs with punctuation, one thought the question 

referred to history, another felt the question had to do with whether or not American Indians or Alaska 

Natives had taught them more about symbols and designs in math.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One student out of 22 respondents referenced using math in a Native language, one said they did not 

know, and 17 essentially repeated the question in slight different terms. Three gave unusual answers 

ranging from symbols and designs having to do with AI/AN history, or whether or not American Indians 

or Alaska Natives had taught them more, and another student made a reference to gestures.

Probe 3: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

Only one student responded to this question on this item and specified their mother as a tribal 

representative in the following: “There's my mom. I don't know what she is. She likes to file papers and 

she's getting people's license or something like that. She's always busy all the time. Sometimes I can 

barely even see her.”

Probe 4: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

NA

Student #11 New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and 

designs used in math?

A. No one has taught me about American Indian or Alaska Native symbols and designs 

used in math.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 17

Solutions to Improve Item

Respondent: NA

Interviewer: Again, knowing the purpose and/or what this question is trying to ask would be very 

helpful.

Initial KAI Recommendation (Keep/Change/Discard)

Discard: Given how the question currently stands, the recommendation is to discard it.

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting 

the phrase "symbols and designs", and the specific reference to math caused additional confusion. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 18

Item 12

Reading: Two respondents had trouble with the word “assemblies”; of these two, one had trouble with 

the word “traditions,” and the other had difficulty with the word “elders.”

Understanding: Two respondents were reported by interviewers as having trouble understanding the 

question due to either difficulty with words in the question.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Sixteen of 22 respondents repeated some variation of the item question, four respondents gave 

somewhat confusing answers such as “Have you ever met them?” and “Have I done everything I was 

told to do?” and an explanation about “stories from a long long time ago.” Additionally there was one “I 

don’t know” and one non-response.

Probe 3: What time frame were you thinking about when you answered this question? 

Very little agreement or consistency in the 20 respondents’ answers to this question. Answers rarely 

overlapped and included responses such as, “one to two times a year,” “two to three days,” 

“December,” “four years,” “all school year,” “all years in school,” “five months,” “5 minutes,” “a week,”

and “a semester.” The most confounding was “When I was a baby. Yeah. It’s so dark.”

Probe 4: What does the phrase “assemblies or presentations” mean to you? Can you give me 

examples of “assemblies or presentations”?

Half (11) of the 22 respondents gave broad definitions of what programs, presentations, and assemblies 

consisted of, and about six of the students cited specific examples of assemblies or presentations they 

had been to as a way to explain what assemblies and presentations meant to them. Only two of the 

respondents said that they did not know, and seemingly non sequitur answers were limited to three:

“I don't think that much, I don't know that much. [Interviewer prompts] About learning traditions and 

cultures and the way we live in right now and how did they create it before the date of 2014 out of 

1614.”

“Like programs. [Confusion over what we were trying to ask, mentioned singing in Chapel Hill and having 

activities at Grandma's house.]”

Student #12 Revised/VH040119

During 4th grade, have you attended assemblies or presentations in your school given by 

elders or leaders from an American Indian or Alaska Native tribe or group about traditions and 

cultures (ways of life, customs)? 

A. Yes, once or twice

B. Yes, three or more times

C. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 19

“It means that it's kind of telling me a story.”

Probe 5: What does the phrase “elders or leaders from an American Indian or Alaska Native tribe or 

group” mean to you? Can you give me examples of “elders or leaders”?

The 21 responses to this probe yielded the following kinds of answers. About 10 respondents gave 

either specific examples or provided explanations about what they thought made an elder or leader, 

four respondents mentioned specific family members, three spoke about people coming to give 

presentations, three said that they did not know, and two gave answers difficult to qualify.

Probe 6: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up 

question]: How many times have you attended these assemblies or presentations in your school?

Given the responses to Probe 4 it is difficult to say the precise time frame that respondents were 

thinking about for this question. The 15 who provided responses broke into two broad categories. Eight 

students mentioned attending assemblies or presentations three times or more, and four said that they 

had attended either one or two times. 

Probe 7: [Conditional probe: If the participant selects answer choice C, then ask the follow-up 

questions]: Is the reason you have not attended assemblies or presentations during 4th grade because 

they are not available? If not, please explain why you have not attended assemblies or presentations 

during 4th grade.

Of the seven students responding, three said that assemblies were not available at their school or 

would be occurring later on in their schooling, two respondents said they had been to assemblies but 

there had not been any elders or leaders present, and two respondents gave non sequitur answers 

indicating that they did not fully understand the question. 

Probe 8: Do you think that answer choices A and B are clear? If not, can you give me examples of 

other answer choices that would be more clear?

Fourteen of 17 respondents felt that the answer choices were clear, one said that they did not know, 

and two said that the answer choices were not clear.

Solutions to Improve Item

Respondent: One respondent felt that there should be another answer choice to reflect whether or not 

someone had attended more than three times. Another felt that more specific kinds of answer choices 

would be better.

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: There seemed to be a lot of variance in answers to the probes related to this item. Perhaps 

bolding “4th grade” would help to ensure that students are answering for that time period. From 

responses to Probe 4 there seemed to be some agreement that assemblies required people coming 

together to see a presentation that someone else is giving, whereas “presentation” seemed to be 

interpreted as something that students might do in a classroom setting. This may indicate that using the 

term “presentation” may be confusing if the item seeks to parse out whether or not students are 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 20

attending assemblies (which would likely involve dancers, etc.) or presentations given by leaders/elders. 

Another point is the use of “leaders/elders” as they may not be the ones leading presentations or 

assemblies about AI/AN cultures (for instance, given the example of dancers coming to the schools). 

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS agrees with KAI that this item should be dropped given the large degree of variance in 

the probe responses. Also, there seems to be a cognitive overload with this item for a 4th grader.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 21

Item 17

Reading: One student misread the word “culture.”

Understanding: One student seems to be interpreting AI and AN as two separate issues or questions 

within the item.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Seventeen of 22 respondents repeated some version of the item question, two interpreted the question 

as asking them what they knew, two simply mentioned AI/AN culture, and one gave the following 

response, “You know what you're doing about yourself that you know is explaining about you.”

Probe 3: What does the phrase “when my teacher talks about” mean to you?

Sixteen of 22 respondents made reference to their teachers talking or teaching; within this group the 

topics were either not specified or were seen as revolving around AI/ANs (five students mentioned this), 

history (two students mentioned this), art (one student), or bullying (one student). Three students said 

that they did not know. 

Probe 4: Which answer choice would you select if your teacher had not talked about American Indian 

or Alaska Native history or culture? Why would you select this answer choice?

Fifteen of 22 respondents felt that they would choose a different answer if their teacher had not talked 

about AI/AN history or culture in class, many of them citing that if their teacher had not talked about it 

then they probably would not know about the topic. Seven students felt that their answer would stay 

the same.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents overall seemed to understand what the item was asking.

Student #17 Revised/VH040212

When my teacher talks about American Indian or Alaska Native history or culture, I try to read 

more about it. 

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 22

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 23

Item 19

Reading: NA

Understanding: Respondents seemed to have a firm grasp on what this question was trying to ask.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eighteen out of 21 respondents provided responses that were in keeping with what the question was 

asking with some variation on what “being the best” meant to them—from getting good grades to being 

the best reader to being well-behaved. Three respondents interestingly gave responses with a negative 

take on being one of the best in the class, referencing greed, showing off, and making others feel bad 

about themselves. 

Probe 3: What does the phrase “one of the best students” mean to you?

This probe elicited interesting responses from the 22 respondents who provided answers. Eight 

referenced having good behavior, six students used the word “best” to talk about what being “one of 

the best students” meant to them, six mentioned knowing more than others or being smart, five talked 

about grades, three simply noted being “better” than others, two discussed popularity, one said it 

meant being fast with schoolwork, and one said that they did not know.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Question seemed easily understood and was being interpreted as intended by the majority of 

students.

ETS Recommendation (05/02/2014)

Keep this item. 

Student #19 New/VH040276

I want to be one of the best students in my class.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 24

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 25

Item 20

Reading: NA

Understanding: All students seemed to understand the gist of the question, though two seemed to 

interpret “challenge” with negative connotations.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Answers to this question seemed fairly consistent. Ten of 21 respondents essentially repeated the 

question, two mentioned that “being challenged” meant answering hard questions, two noted that 

challenge meant not doing easy things, two specifically mentioned math, one student pointed to doing 

independent work, and one noted learning new things. Only two responses seemed a little out of the 

norm; one student stated “It’s telling me they like challenges,” and the other referenced staring 

contests. 

Probe 3: What does the phrase “being challenged” mean to you? Can you give me examples of you 

being challenged in your classes?

Thirteen of 22 respondents noted that “being challenged” to them had to do with being pushed to do 

harder work, usually by their teachers. Three specifically mentioned math in their responses, two talked 

about tests, two referenced competitions (one of these respondents was the one who spoke of staring 

contests), and two talked about tests.

Solutions to Improve Item

Respondent: NA

Interviewer: One of the interviewers felt that there could be more emphasis on this question being 

related to schoolwork—they had the interview with the student who used staring contests as an 

example of challenges.

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents understood the question and interpreted as intended.

Student #20 New/VH040379

I enjoy being challenged in my classes.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 26

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 27

Item 22

Reading: NA

Understanding: Most respondents seemed to understand that the item wanted to know if they were 

getting help on their schoolwork; the piece about it being “someone else who lives in your community 

or is a friend of your family” was not necessarily interpreted as it should have been. Students seemed to 

latch onto different words in the bolded part of the statement like friends, family, and community, but 

did not necessarily tie them together. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Respondents generally understood that the question had to do with receiving help on one’s homework 

with 12 out of 22 respondents referencing that piece of the question specifically. Nine mentioned family 

providing help, five referred to friends helping, five mentioned community members in their responses, 

one spoke of a tutor, and one referenced a friend of the family. Three answers were somewhat 

confusing; one student mentioned their teacher providing examples, another seemed to think the 

question had to do with them helping their friends as well as being helped by their own family, and one 

student interpreted the question as having to do with how much time they spent on homework. 

Probe 3: What does the phrase “someone else who lives in your community or is a friend of your 

family” mean to you?

Nine of 22 respondents spoke about friends of the family as their contemporaries in age, rather than as 

an adult who was a friend of the family. Additionally nine responses referenced family specifically from 

mothers/fathers, brothers/sisters, aunts/uncles, grandparents, and cousins. At times the cousins were 

noted also as friends. Of the seven responses directly mentioning the idea of friends of the family, 

friends were often noted as of contemporary age with the respondent and cousins were also often 

mentioned. Three students said that they did not know what was meant by “someone else who lives in 

your community or is a friend of the family.”

Student #22 Trend/VC759291

How often does someone else who lives in your community or is a friend of your family help 

you with your schoolwork? For example, they might help you to study for a test, help you with 

a school project, or go over your homework with you.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a week

D. Every day or almost every day
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 28

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of how these people have helped you with your schoolwork? 

Who is this person?

Eleven out of 20 responding students mentioned family members from parents to grandparents and 

everyone in between, and six students spoke of friends helping them—most indicated that these friends 

were of the same age, though some were slightly ambiguous. Two students mentioned tutors, and three 

gave unclassifiable answers.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: The wording of the bolded section seems to be causing confusion for many respondents who 

do not seem to be interpreting the question as intended. We suggest revising the question to ask 

specifically about adults, friends, or tutors in the community, or asking whether or not family is involved. 

Friend of the family/someone else who lives in your community seems to involve a nuanced 

understanding that is not being exhibited by the majority of the respondents to this question.

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS recommends dropping this item as it seems to overlap with the previous three trend 

items (VC759288, VC759289, VC75929). Having four sequential items about who is helping the student 

with his/her schoolwork is excessive; therefore this item can be dropped to make room for a new 

item(s). Asking about "someone else who lives in your community/friend of your family" seems to 

involve a nuanced understanding that is not being exhibited by the majority of the respondents to this 

question. 

The similar grade 8 item VC996640 is also proposed to be dropped.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 29

Item 23

Reading: NA

Understanding: The grand majority of students seemed to understand what this item was asking.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eighteen out of 22 respondents interpreted the question correctly, one student specifically mentioned 

math homework, and another mentioned helping their siblings with school work. Only one of the 

respondents seemed to be a bit confused about the angle of the question and seemed to think it was 

asking them about whether other students were helping them with course work.

Probe 3: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

questions]: Can you give me examples of how you have helped other students with their schoolwork? 

Are the other students that you have helped with their schoolwork in 4th grade? If not, what grade 

are the students that you have helped with their schoolwork in?

Ten of 19 respondents mentioned helping “friends” with homework, nine noted that those they were 

helping were in the 4th grade, six respondents pointed to helping family members such as siblings and 

cousins with homework, and five respondents pointed to working with students in other grades. Two 

respondents gave C or D as answers, but when asked for examples, they responded “no.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Student #23 New/VH040389

How often do you help other students from your school with their schoolwork? For example, 

you might help other students to study for a test, help them with a school project, or go over 

their homework with them.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a week

D. Every day or almost every day
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 30

Initial KAI Recommendation (Keep/Change/Discard)

Keep/Change: It seems that there may be some room for confusion on this item depending on what 

information is being sought. If the question is simply to capture whether or not students are helping 

others with classwork, it is fine as is. If the question specifically wants to know about “other students 

from your school,” that may need to be highlighted. Generally, the question is fine as is with students 

responding consistently about helping others.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 31

Item 25

Reading: NA

Understanding: The majority of respondents seemed to grasp the question being posed to them.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eleven of 21 responding students noted the importance of education and mentioned post-elementary 

education in their explanation of what the question was asking, seven students essentially repeated the 

question using slightly different words, one noted hard work, one mentioned money, and one said that 

they did not know what the question was asking. 

Probe 3: What does the phrase “trying hard in school” mean to you? Can you give me examples of 

what “trying hard in school” would be?

Fifteen of 22 responding students noted that “trying hard in school” meant studying or practicing or 

doing your best. Seven mentioned doing well on tests in their responses, four mentioned getting good 

grades, three mentioned college, and one mentioned getting help when you need it.

Probe 4: What does the phrase “a good life” mean to you? Can you give me examples of what “a good 

life” would look like?

Answers to this probe varied in terms of what themes presented themselves. For instance, nine out of 

22 responding students mentioned jobs or “good jobs” when discussing what it meant to have a good 

life, five mentioned having a family or being able to help family, four mentioned having money, four 

noted having knowledge or knowing a lot, four talked about having a house, three mentioned good 

health (ranging from eating well to being drug free), two talked about happiness in general terms 

(though one was specific about that having to do with ponies in a mansion), one said they thought it had 

to do with having good manners, and one said they did not know. 

Probe 5: What does the phrase “when I grow up” mean to you? Can you give me examples of what 

being “grown up” means to you?

Student #25 New/VH040394

Trying hard in school will help me live a good life when I grow up.

A. Strongly disagree

B. Disagree

C. Agree

D. Strongly agree
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 32

Responses to this probe varied greatly in terms of what students saw as being “grown up.” Seven of 22 

responding students mentioned having a job, five said they thought they would be grown up when they 

were in their 20s, five referenced education (often referring to college), five said it would be when they 

were independent (often referencing not needing parents any more), three said when they were bigger 

or taller, three mentioned having a house, two said it would be in junior high, two said when they had a 

family of their own, one mentioned being clean, one mentioned having a car, one spoke of good 

penmanship, and one said they did not know. One of the respondents summed up being a grown up 

with the following, “They go to work every day. They go to dinner. They go to bed. They wake up, and 

then they go back to work. That’s it.” 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: All respondents seemed to understand the question as it was intended.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 33

Survey Instrument: Student Grade 4/Sequence II

Item 2

Reading: One of the students who did segments of both sequence 1 and sequence 2 initially thought 

they had already read this question. 

Understanding: Only one student was reported as not understanding the item and having trouble with 

the probes as well. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-six out of 30 responding students essentially repeated the question; four indicated that they did 

not understand or did not know what the question was asking.

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions” 

mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you 

have learned?

All of the 31 responding students asked this question gave a response, 10 of them mentioned traditions 

in their answers ranging from smudging and tobacco use, and one student noted Christmas as a 

tradition. Seven students talked about powwows or dancing, four mentioned crafts, three expressed 

that they did not know what “Alaska Natives” meant, and other answers involved food, housing, history, 

language, and hunting. Six students indicated that they did not know what the question was trying to 

ask.

Probe 4: In answer choice B, what does the phrase “Family members” mean to you?

Of the 30 respondents to this question (one was inaudible), 18 students named different members of 

their family (for instance, parents or siblings or grandparents). Seven students simply said “family” when 

Student #2 New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions 

(way of life, customs)? 

A. No one has taught me about American Indian or Alaska Native traditions.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders 

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 34

asked what “family members” meant to them, four students spoke about relationships of teaching and 

learning when talking about what it meant to them, and two mentioned being loved or cheered up. 

Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? 

Can you give me examples of “Tribal representatives or elders”?

The 30 responses to this question varied a good bit as it asks about both tribal representatives and 

elders. Thirteen students named someone either specifically or in terms of who in their family they

considered to be an elder or tribal representative. These answers ranged from grandparents to teachers 

to family friends. Nine students noted that tribal representatives or elders were “old people,” six 

mentioned teachers—either specific teachers or just people who teach younger people about their 

culture. Five students said that they were people who represented the tribe, four said they did not 

know, two simply responded “no,” and two respondents had difficult-to-categorize answers—“that 

means they help me get yeast and stuff and all that.”

Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

The validity of responses to this question vary greatly given that very few students seemed particularly 

sure of what a tribal representative was. Of the 16 respondents who provided answers, eight indicated 

that they did have a family member who was a tribal representative (often giving answers like “family 

members”), four indicated they did not, two said maybe, and one said they did not know, and another 

student gave a hard-to-characterize answer—one said “elders.” 

Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: Separate out elders and tribal representatives in probes to better determine how students 

understand each of these categories. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It would seem that the majority of the students asked this question understood what the question 

wanted to know about who had taught them what they knew about AI/AN traditions. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 35

Item 4

Reading: No difficulties reading the item. 

Understanding: A few of the respondents seemed to have some issues with fully understanding the item 

without some redirection from interviewers. One student seemed very confused by the term “Alaska 

Native,” two others seemed to have a hard time understanding the broader question and got bogged 

down in the idea of specific vocabulary or attendance overall. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 31 responding students mentioned language in their restatement of the question, five 

students did not mention language but rather seemed to interpret the question as having to do with 

AI/AN culture, five seemed to be confused by the question—one stating, “What time do I arrive at 

school”—and one student simply noted that they do not learn anything about AI/ANs in their school as 

they are one of five students of AI/AN heritage in the entire school.

Probe 3: What does the phrase “American Indian or Alaska Native language” mean to you? Can you 

give me an example of “American Indian or Alaska Native language” that you can speak or know 

about?

Nineteen of 31 students responding to this probe referenced themes having to do with how people 

spoke in the past; some of these students even referenced language being spoken by elders in their 

community. Eight students mentioned specific AI/AN languages, three talked about being taught or 

learning other languages, two students said “no,” and one gave an answer having to do with their 

grandmother that strayed from language and into horseback riding. 

Student #4 New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska 

Native language?

A. My school does not offer courses that are taught in an American Indian or Alaska 

Native language.

B. Never or hardly ever

C. Once or twice a month

D. Once or twice a week

E. Every day or almost every day
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 36

Probe 4: How many American Indian or Alaska Native languages do you speak?

Eleven of 31 responding students reported speaking at least one AI/AN language, nine said they spoke a 

little of one or two languages, nine said they did not speak any AI/AN languages, one student said they 

spoke two languages, and one said they did not know. 

Probe 5: How would you answer this question if you speak more than one American Indian or Alaska 

Native language?

This probe drew varied answers from the 27 respondents. Five students indicated that they would 

change their answer to D. Five gave answers that were difficult to characterize from “Talk like America” 

to “If I could speak more than one American Indian languages, I would go lots of places” to “I speak 

Spanish, Chinese—I say hello in Chinese. I can speak Italian.” Four said they did not know, and four gave 

the answer B, but at least two of these indicated they thought that meant “once or twice a week.” It is 

hard to say if this is a misunderstanding or misreading on the part of the students or if perhaps B was 

heard instead of D by the transcriptionist. Three students would change their answer to E, two would 

keep their answer as is, and one would change their answer to A (interestingly this participant gave D for 

their original answer and indicated that they do in fact speak Lakota).

Probe 6: What is the difference between answer choices A and B?

Fourteen out of 31 responding students gave responses that indicated that they had interpreted the 

differences between A and B correctly. Six students gave responses that were difficult to characterize in 

that they were unfinished thoughts, or seemed to confuse what the question was asking. For instance, 

one student said, “One talks about the Alaska Native language and the other one is for attending class.” 

Another said “That usually A, they usually don't teach you much about A. You are very like the entire

language and what you eat and B is probably like you are never really learned anything about your 

culture,” and another student had the categories backwards and said “Because B you don’t learn it and 

A you do.” Four students said they didn’t know the difference between the two answers and four other 

students felt that the two answer were the same. Two students suggested that A referred to learning in 

school and B could be outside of school. 

Probe 7: If there were no courses offered in your school that are taught in an American Indian or 

Alaska Native language, which answer choice would you select?

Thirteen of 31 responding students said they would choose A; two indicated they were choosing this 

because their school did not currently have classes in an AI/AN language. Nine said they would choose B. 

Four said they would choose D, one said they would choose C, one said they would choose E, and two 

indicated they did not know. It seems clear that many of the students did not understand what this 

question was trying to ask.

Solutions to Improve Item

Respondent: NA

Interviewer: Hypothetical questions seemed to confuse many students. One actually said “Again, that 

seems like another confusing question. I would think that I would answer the same thing but I don't. I 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 37

just feel like I don't want my answer to be the same.” This led the interview team to question how often 

this might have been happening with other probes.

Initial KAI Recommendation (Keep/Change/Discard)

Change: This question generally seemed to cause some confusion for respondents. The item seemed to 

make sense to students, but subsequent probes revealed that there was some confusion around the 

difference between answers A and B. Some of this may be attributable to the difference in wording of 

the two questions, and the description for A is much more detailed than that of the other answer 

choices. Our recommendation is to change to be similar to B through E and to highlight in the item the 

fact that the question specifically is asking about classes at school (keeping in mind that many students 

who may take classes may do so outside of school). 

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about 

classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer 

choice “A”, ETS recommends changing the word “courses” to the word “classes” to be consistent with 

the item stem. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 38

Item 5

Reading: NA

Understanding: Some respondents seemed to think that the question had to do with teaching or 

learning the language, and even of those who restated the question in a way that seemed to capture the 

intent, there seemed to be a wide interpretation of what the item was actually trying to capture. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-one of the 31 respondents provided a response that was on track with what the question was 

trying to ask, but the variations within these answers was interesting. Some students essentially 

restated the question with some explanation, and others said things like “Can you speak American 

Indian language.” Three students seemed to think that the question had to do with how much they learn 

or “how you teach it.” Another three gave answers that were difficult to categorize, ranging from “It 

means how do you talk when you try to talk like them” (ostensibly referring to AI/ANs) to “There’s three 

different ones and then you pick which one you think you’re at” (likely referring to the fact that they are 

being asked to choose between three different answer choices). 

Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?

Twenty-five students out of 30 respondents gave answers indicating that they had correctly interpreted 

the phrase “I can speak a few words or phrases” as intended. Three students said that they did not 

know, and one students said, “It means you can speak a few words in English and stuff. Alaska.”

Probe 4: In answer choice C, what does the phrase “I can speak well” mean to you?

Of the 29 students who responded to this question, 26 seemed to understand the gist behind what “I 

can speak well” was trying to elicit, though this occurred in different ways with some students 

interpreting it as a generic speaking well or fluently whereas others were more nuanced in their 

descriptions, drawing on breadth of vocabulary and pronunciation. Only one student indicated that they 

did not know what the phrase meant. 

Probe 5: Do you think that additional answer choices should be added between B and C?If yes, can 

you give me examples of what those answer choices would be?

Student #5 New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?

A. I cannot speak an American Indian or Alaska Native language.

B. I can speak a few words or phrases.

C. I can speak well.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 39

Sixteen out of 30 responding students said that they did not think any additional answer choices should 

be added. One said that they did not know, and 12 said that they did think that there was a need for 

more answer choices. Of these 12, the answers generally focused on different nuances that students felt 

were missing, though one suggested both B and C should be the same, indicating that they did not truly 

understand the probe. Additionally, some of the “yes” responses restated the options available in 

different words, suggesting that they may have been confused by what the probe was trying to capture. 

A few examples of the other “yes” answers are as follows:

“I think they could put a little more in it, like, ‘I can speak medium well,’ like in the middle of little and 

well. And like, ’I can barely do some words.’"

“I can speak mostly a couple of words. I can speak them very well, but sometimes I forget how they 

sound.”

“Yeah. If you could only speak few words you can’t speak a lot, but you can speak a few, but if you can 

speak well you can speak a lot. You can speak a few, but well.”

Solutions to Improve Item

Respondent: Might consider some of the answers given by students who thought that other answer 

choices were needed between B and C, though many of the examples do not seem to indicate a viable 

additional choice. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that generally speaking, the majority of students responding understood what the 

question was trying to ask. If there were any change to be made, it might be to stay consistent regarding 

phrasing in the different answer options. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 40

Item 6

Reading: NA

Understanding: Some of the respondents seemed to have difficulty understanding the item based on 

responses to Probe 1, and at least two respondents felt initially that this was the same question as Item 

5. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-five of 29 respondents provided a rephrasing of the question that indicated that they 

understood the meaning of the question. One said they did not know, one referenced English (indicating 

that they may have been confused about the item or probe), and one asked if the question was referring 

to one’s first language or a Native language. 

Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: Can you give me an example of “American Indian or Alaska Native language” that you can 

read?

Thirteen out of 19 respondents mentioned a specific language in their answers. In at least three of these 

cases, the interviewer or data entry person noted what language the student was familiar with was 

based on answers to other questions if the student did not mention it themselves. One student said they 

did not know, and four gave difficult-to-categorize answers such as, “I can read not that much but I can 

read some” and “I can probably read a sentence or maybe a couple words that's written like on a board 

or on a book” and “I could tell you who I am”—this last one may be referencing being able to introduce 

oneself in a Native language, but it is hard to tell.

Probe 4: In answer choice B, what does the phrase “I can read a few words or phrases” mean to you?

Twenty-two of 27 respondents (one was inaudible) responding felt that that answer choice B was 

referring to knowing a little bit or at least some words in a Native language. Three said they did not 

know, and two gave the following answers: “That you kind of see something, and that you’re doing good 

on something, that something is really hard to do” and “B is saying that I can only read most of the 

words in Choctaw or in Indian.”

Student #6 New/VH040067

How do you rate yourself in reading in an American Indian or Alaska Native language? 

A. I cannot read in an American Indian or Alaska Native language.

B. I can read a few words or phrases.

C. I can read well.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 41

Probe 5: In answer choice C, what does the phrase “I can read well” mean to you?

Of the 27 students who responded, 23 seemed to adequately understand what answer choice C was 

referring to, three said they did not know, and one said, “That you see something really big because 

sometimes it's really hard to read.”

Probe 6: Do you think that additional answer choices should be added between B and C? If yes, can 

you give me examples of what those answer choices would be?

Fourteen of 22 respondents indicated that they thought the answer choices were fine as they were and 

eight felt there was a need for additional answer choices. Of the remaining respondents, nine answers 

varied greatly, some of which are included in the following:

“Like, if you can't, if you don't know if you can read American language, they should have another thing 

that says you don't know if you can read it.”

“I can read half of the words I know, but I can read a little better than that. Like, I almost can read really 

well.”

“I think that there should be, I could read a little words, like maybe like 10 or five words because phrases 

and words don’t go together really well sometimes.”

“Probably, like it should go, ‘I can read a few phrases,’ and then between them, it should say, ‘I can read 

two or three paragraphs,’ and then, ‘I can read well.’"

Solutions to Improve Item

Respondent: Some of the suggestions for adding additional answer choices could be considered. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Most of the students given this question seemed to understand the item as it was. One suggestion 

may be to keep phrasing consistent across answer choices. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 42

Item 7

Reading: NA

Understanding: Some respondents thought that this was a repeat question when they read it. 

Changing Answers: Not within the question, but a respondent who previously had noted that they did 

not know what a tribal representative was indicated during this item that they had family members who 

were tribal representatives. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-four of 31 respondents provided answers consistent with what the item was asking. Two 

students did not mention language but instead seemed to think the question was referring to AI/ANs in 

general, and four gave answers that were difficult to characterize:

“If the teachers are doing good in the school and which they are, they're doing an excellent job I think.” 

“B, C, and E. For the first one.”

“I think it’s the same as number one.”

“Someone who you know, like a friend or a teacher, or your mom or sister, for example they could all 

teach me but one that teaches me the best and the most. They teach me important stuff and good stuff 

about them.”

Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?

All 29 responses to this probe were too varied to categorize, with the exception of “I don’t know,” which 

occurred five times. Some students thought the question had to do with whether or not they could 

speak a language, who taught them, how well they spoke it, etc. There was very little validity with 

regard to answers. 

Student #7 New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?

A. No one has taught me about an American Indian or Alaska Native language.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): _____________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 43

Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska 

Native language”?

Nine out of 27 respondents indicated that they did not know, five mentioned specific languages, five 

said that they could not give an example, three gave examples indicating they were thinking about 

vocabulary, three indicated that they considered counting in an AI/AN language as an example, and two 

suggested that being able to speak it was “part of an American Indian or Alaska Native language.” As 

with Probe 3, the phrasing of this question leaves a lot up for interpretation and the meaning seems not 

to be clear to respondents.

Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

There were nine respondents to this question. Eight indicated that they did have family members who 

were also tribal representatives, and one said they did not. This question was skipped for several 

respondents as the question had been asked and answered in an earlier probe. 

Probe 6: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: Revise or remove Probes 3 and 4 as they seem to confuse students, and answers given 

have questionable validity. Additionally, the question about tribal representatives seems in general to 

draw unreliable responses from students who do not seem to be sure of what a tribal representative is 

or what such positions consist of. We recommend either changing the probe to more deeply explore 

what students define as a tribal representative or remove the probe. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students seemed to easily understand what the question was asking. We 

recommend keeping phrasing consistent throughout answer options. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 44

Item 13

Reading: NA

Understanding: Some students did not interpret the item only in terms of school activities but also 

brought in times that they may have gone on field trips with non-school organizations or families. One 

student interpreted the question as having to do with funding. Additionally, several students seemed to 

have a hard time only answering for what they had done during the 4

th grade. 

Changing Answers: Two students changed their answers; one could not explain why they chose to 

change the answer, and the other switched their answer from B to C after some consideration. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-seven of 30 respondents answered the question correctly in the sense that they appropriately 

identified the main components of the item, though some bordered on essentially repeating the 

question. Two students gave difficult-to-categorize answers such as, “Have you seen or they put up in 

museums what it does that they think,” and, “If you knew a little bit more about American Indian or 

Alaska Native people.” One student said they did not know and another seemed to think the question 

had to do with funding for field trips.

Probe 3: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

question]: What types of places have you attended to learn about American Indian or Alaska Native 

people?

Seven students out of 13 respondents noted that they had visited museums; nine mentioned field trips 

to historic sites, powwows, hatcheries, or missions. There may be some issues regarding the validity of 

these answers as one student noted that the field trips they had been on had not been “traditional” per 

se, and there may be some question as to the time frame from which students were drawing upon to 

answer the question.

Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: How do you know that other 4th-grade students attended, or that activities were not 

offered to 4th-grade students?

Seven out of 17 students responding said that field trips were either not offered or that they had not 

been told about them, three responded that all the 4

th graders go on field trips together, one mentioned 

Student #13 Trend/VC759216

During 4th grade, have you attended school field trips to museums, traditional villages, or 

other places to learn about American Indian or Alaska Native people? 

A. Yes

B. No, but other 4th-grade students did.

C. Not offered to 4th-grade students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 45

that they needed a permission slip—indicating that perhaps they did not fully understand the question 

as it was asked. There were four responses that were hard to make sense of as they were non sequitur 

answers ranging from having to do with getting sick on Halloween, an Ojibwe reading group, and not 

wanting to go on a field trip. Some of the confusion related to answering the question may be related to 

having two questions presented as one as part of this probe.

Probe 5: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

question]: Do you know whether other 4th-grade students had the activities or the activities were not 

offered to 4th-grade students? 

Five of 10 respondents said that all of the 4

th graders go to the field trips together, three said they did 

not know, and two said “no.”

Solutions to Improve Item

Respondent: NA

Interviewer: The probes in this question seem to cause some confusion, particularly Probe 4 where 

there are perhaps too many components to the question for students to easily make sense of, and the 

same may be true of Probe 4 as there are two distinct components to that probe as well. Additionally, 

there seemed to be some issues with ensuring that respondents were choosing answers that only 

applied to students’ time in the 4

th grade. Perhaps this point could be further emphasized in the wording 

of the item.

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of the students seemed to understand the question as written, but there are some 

questions regarding the validity of the time frame students were thinking of in their responses. Our 

recommendation is to further emphasize that the question is referring to students’ time in the 4

th grade,

perhaps by using the term “this year” either in the item or the answer choices. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 46

Item 14

Reading: NA

Understanding: It seemed that some respondents did not interpret the question solely in terms of 

school resources. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-seven out of 30 responding students seemed to interpret the item correctly. There were varying 

degrees of detail in respondents’ answers, but the gist of the question appeared to be intact. One 

student seemed a little confused and stuck on the in school portion of the question and did not really 

engage the idea of materials, and two students offered non sequitur responses such as, “How would you 

treat people that went in war or something who built the Statue of Liberty or something” and, “Have 

you found books or videos that teaches you languages and ... Do you know what a translator is on your 

phone or something? That is something you could use. You could put a word down and it could translate 

that. So like that. Or YouTube, you could look Indian words up and stuff.”

Probe 3: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up 

question]: How many times have you used these materials in school? 

Answers among the 27 respondents to Probe 3 varied somewhat widely with six students saying they 

had used these materials two times, five indicating 10 or more times, four saying three to four times, 

three saying only once, and other answers with one response were five times, “no,” “a lot,” “two times a 

week,” and “I don’t know.” Additionally there were three difficult-to-categorize answers:

“During Lakota class we watch videos of how they speak and we watch legends like dinosaurs from the 

end of the world and stuff.”

“Kind of a lot in Idaho history. We had to make dioramas about Native American tribes. I did Nez Perce. 

We went on-line to go find stuff about it. I think that's all.”

“On my old school, they had spears way back when the days and then they had the sharp rocks on them. 

They sharpen them and one time, they brought ... we had a science experiment where we’re learning 

about this and then they brought this one across like that and then it had one that goes through it and 

Student #14 Revised/VH040200

During 4th grade, have you used books, videos, or other materials (including online resources) 

about American Indian or Alaska Native people in school? 

A. Yes, once or twice

B. Yes, three or more times

C. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 47

it’s really [inaudible 00:19:11] and we looked at it and see ... one had signs on it like paint writing on it 

but it’s really tiny like this little thing when you look at them.”

Probe 4: What does the phrase “online resources” mean to you? Can you give me examples of “online 

resources”?

In general, all 31 responding students seemed to understand this question across the board, although 

the particulars of their answers varied quite a bit with regard to the components they chose to discuss. 

Eight students mentioned computers, seven mentioned websites (general, not specific ones), seven 

reiterated “online,” six mentioned Google specifically, four said something about the Internet, two 

mentioned YouTube, two mentioned Wikipedia specifically, and one said something about MP3s and 

listening to Native music. One said “I don’t know” and another gave a difficult-to-characterize answer 

with the following: “Electricity and pictures that move.”

Probe 5: What does the phrase “in school” mean to you?

Twenty-six of 31 respondents seemed to understand what “in school” was referring to, one student said 

they did not understand, and four gave difficult-to-categorize answers only peripherally related to 

school, but still related to the item—having to do with school or online resources, just answers out of 

place given the probe asked.

Probe 6: Do you think that answer choices A and B are clear? If not, can you give me examples of 

other answer choices that would be more clear?

All 28 responding students felt that answer choices A and B were clear. One said they were not but did 

not give an alternate to improve the choices, and one said, “C, I don’t really like because no because if I 

never knew it, I’ll probably put C but I know it so I put B but A and B are good answers but C, it ain’t.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students asked this question seemed to grasp the meaning and intent of the item. 

The phrase “online resources” could possibly be replaced with the terms “Internet” or “websites” as 

they may be more recognizable to students. 

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources” 

with the phrase "internet resources" which is likely more recognizable to students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 48

Item 15

Reading: NA

Understanding: A few of the respondents seemed to think this was the same question as Item 14 at 

first. 

Changing Answers: Three respondents changed their answers—two from B to A and one from A to B—

and one other answered A, but later said they had used materials “three or more times.”

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 31 respondents seemed to grasp the main components of this question regarding having used 

materials outside of school. Again, there was some variation within answers, many specifically referring 

to AI/ANs and others simply talking about using materials not on school grounds, but all seemed to 

understand underlying concept of item. 

Probe 3: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up 

question]: How many times have you used these materials outside of school? 

As with the previous item, all 18 responses to this question varied quite a bit. Six said one to two times, 

three said three or more times, two said a “few” times, two said 10 times, and other answers occurring 

only once were “at camp,” “not a lot,” “once a week,” or “eight or nine times.”

Probe 4: What does the phrase “outside of school” mean to you?

All 30 respondents to this question correctly identified what was meant by “out of school” with varying 

details. Most indicated that they were thinking of home or at the library or at a friend or family 

member’s home. 

Probe 5: Do you think that answer choices A and B are clear? If not, can you give me examples of 

other answer choices that would be more clear?

All 27 responses indicated that they thought the answer choices made sense. One said “no,” offering no 

explanation; one had this to say after answering affirmatively that the answers made sense: “I don't 

think they make sense. Because they're both the same,” and another student who answered in the 

affirmative offered this caveat: “Yeah. Only B does though. Because why would you learn it if you’re not 

learning much looking at it once or twice? It shouldn't be there. Should be three times or more.”

Student #15 Revised/VH040205

During 4th grade, have you used books, videos, or other materials (including online resources) 

about American Indian or Alaska Native people outside of school? 

A. Yes, once or twice

B. Yes, three or more times

C. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 49

Solutions to Improve Item

Respondent: Differentiate answer choices—opinion was not shared by the majority of respondents.

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents seemed to understand this item with ease and interpret it as it was 

meant. As with the previous question, the only change might be to consider replacing the phrase “online

resources” with the terms “Internet” or “websites” as they may be more recognizable to students.

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources” 

with the phrase "internet resources" which is likely more recognizable to students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 50

Item 16

Reading: NA

Understanding: Some issues that arose with the interpretation of the item came from confusion around 

the term “media center” and students not interpreting the question solely in terms of school—also 

taking community resources into account. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-six students out of 28 respondents seemed to generally understand the question as it related to 

having a place to go to get information such as a library, and some of these students directly related it to 

AI/ANs. Not all students seemed to interpret this question as being located in a school per se.

Probe 3: What does the phrase “school library, media center, or resource center” mean to you? 

This probe drew a number of different answers from the 28 respondents, with students referencing 

different pieces of the phrase “school library, media center, or resource center.” Ten students said 

something about computer labs, nine mentioned libraries or books, five felt all of the different entities 

were essentially the same thing, five mentioned resources, one said something about it being in school, 

one said it would be elsewhere, and one said they did not know. The term “media center” seemed to 

throw some students off, as did “resource center” (though to a lesser extent). 

Prove 4: What does the phrase “have access” mean to you? 

Twenty-three out of 31 respondents felt that “have access” was related to being able to go somewhere 

or having permission to go somewhere—sometimes this was seen as a physical place, or the ability to go 

to certain sites on the Internet. Three students indicated that they did not understand what “access” 

meant, one thought the question was related to having the time to go somewhere, and another thought 

it had to do with being physically close to a place. Four gave difficult-to-categorize answers such as, 

“Access more food and more computers” and, “Really enjoy and really into learning about Native 

American people.”

Solutions to Improve Item

Respondent: NA

Student #16 Revised/VH040207

Do you have access to a school library, media center, or resource center with books, videos, 

or other materials (including online resources) about American Indian or Alaska Native 

people? 

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 51

Interviewer: Perhaps look at rephrasing the item to use easier-to-recognize words and phrases. For 

instance, perhaps look at alternatives for “have access” and “media center.”

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The item seemed to be generally understandable by the majority of respondents. One 

recommendation may be to simplify the language with regard to “access” and “media center.” 

Additionally, if the aim is to specifically ask about these resources at school, that should be highlighted, 

perhaps by only bolding the word “school” and leaving the other words without bolding, or alternatively 

using the word “school” before every component of the question such as: “school library” “school media 

center” or “school resource center.”

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically 

asking about these resources in school by adding the bolded words "In school" to the beginning of the 

item. For consistency with the proposed revision to the grade 4 student items VH040200 and VH040205, 

ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is 

likely more recognizable to students. The proposed revised item wording is consistent with other NAEP 

student questionnaires, such as the 2014 TEL Probe questionnaire.

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 52

Item 18

Reading: NA

Understanding: The grand majority of students were able to understand the question. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-six out of 31 respondents indicated that putting in effort equated to trying hard in one way or 

another. Only one student said that they did not know, and one student mentioned teachers. Two 

students gave hard-to-characterize answers: “I really think about what the question is trying to ask me” 

and, “People effort into the schoolwork by learning about them.” There are potentially typos from the 

transcription process. 

Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: Can you give me examples of how you put a lot of effort into your schoolwork?

Out of 29 respondents, 19 gave examples of trying hard to show how they put effort into their 

schoolwork. Four mentioned the importance of listening to teachers, two mentioned getting good 

grades, and two mentioned studying. The following responses were mentioned once: doing work 

quickly, doing homework, getting help when needed, showing one’s work, and taking time to do work.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

Student #18 New/VH040240

I put a lot of effort into my schoolwork.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 53

Item 21

Reading: One student misread the answer choices. 

Understanding: One student seemed to have trouble grasping the overall concept of the question. 

Changing Answers: One student changed their answer from C to A after the interviewer prompted them 

to re-read the item’s answer choices (as they had been misread). 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 31 of the respondents grasped the meaning behind this item with some variation. Many felt that 

they were being asked if they felt safe at school, or if they enjoyed school, or if they wanted to be at 

school. 

Probe 3: What does the phrase “belong at school” mean to you? 

All but one of the 31 respondents seemed to understand the probe and all gave varying answers about 

what the phrase “belong at school” meant to them. Six said that it had to do with being at school, seven 

said it had to do with learning, five indicated that it meant you wanted to stay at school, four mentioned 

college aspirations, two said that it felt good to be at school, one said that it meant being part of 

something, one felt it meant wanting to stay at your school rather than going to another, and one did 

not know. One gave a difficult-to-categorize answer: “I think it means did your parents put you in [art 

00:28:43] school or did they have to put you in a private school because maybe the kids aren’t so good 

at school to you. They treat you so mean but you treat them really nice.”

Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: Can you give me examples of when you feel you “belong at school”?

Of 27 respondents, eight mentioned friends, eight mentioned learning, six brought up feeling good at 

school, two mentioned college, two mentioned simply being at school, and one said they were not sure. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Student #21 New/VH040382

I feel that I belong at school.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 54

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The grand majority of students understood the meaning and wording of this question and probes. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 55

Item 24

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-eight of 30 respondents answered this probe in keeping with the meaning of the item, two said 

that they did not know (which may be related to interview fatigue—particularly with what could be 

construed as repetitive questions). 

Probe 3: What does the phrase “put in enough effort” mean to you? Can you give me examples of 

how you could put in enough effort?

Twenty-three of 31 responding students referenced trying hard, three mentioned getting good grades, 

two said something about going to college, two mentioned listening to teachers and directions, one 

mentioned homework, one said staying in school, and one said they did not know. 

Probe 4: What does the phrase “succeed in school” mean to you? Can you give me examples of what 

“succeed in school” would look like?

Answers among 31 respondents were varied for this question. Six referenced trying hard, six said 

something about passing grades and moving from one grade to the next, six mentioned “graduation,” 

six mentioned getting good grades, three referenced listening to teachers and directions, three said they 

did not know and indicated the question was somewhat confusing, two spoke of college degrees, one 

simply said “no,” and one mentioned homework. 

Solutions to Improve Item

Respondent: NA

Interviewer: The question regarding putting in enough effort question was a little repetitive given Item 

18. 

Student #24 New/VH040391

If I put in enough effort, I will succeed in school.

A. Strongly disagree

B. Disagree

C. Agree

D. Strongly agree
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 56

Initial KAI Recommendation (Keep/Change/Discard)

Keep: All students seemed to grasp the meaning and wording of this item. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 57

Survey Instrument: Student Grade 8/Sequence I

Item 3

Reading: NA

Understanding: Most of the students understood the question without trouble; two had to be 

redirected by the interviewer for a more appropriate interpretation. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 23 respondents seemed to grasp the meaning of the item, two said they did not know, and 

one did not answer.

Probe 3: In answer choice B, what does the phrase “Family members” mean to you?

Thirteen of 23 respondents named different types of family members that the question brought to 

mind, four mentioned that family members are people related to them, three mentioned that it would 

be other tribal members as they are all family to some degree or another, and two mentioned love as a 

component of the relationship. One gave a difficult-to-characterize response: “Which one of your family 

members taught you the most of your people?”

Probe 4: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? 

Can you give me examples of “Tribal representatives or elders”?

Seven students (out of 21 respondents) mentioned that tribal representatives were people who 

represented the tribe, five students gave specific names or relationships of people who were 

representatives or elders, four mentioned people who had more knowledge, four said it had to do with 

Student #3 New/VH039931

Who taught you most of what you know about issues today that are important to American 

Indian or Alaska Native people?

A. No one has taught me about issues today that are important to American Indian or 

Alaska Native people.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 58

being older, three mentioned grandparents, one said that they were people higher up in the tribe, one 

said they did not know, one said people on the council, and two gave uncharacterizable answers.

Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

Eleven of 17 respondents indicated yes, three said no, and three said maybe. Most who indicated that 

they did have a family member who was also a tribal representative noted that they thought of the 

person as family first (one respondent was the exception).

Solutions to Improve Item

Respondent: NA

Interviewer: In the future, it might be helpful to separate out Probe 4 into two different questions, same 

for Probe 5. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that the overwhelming majority of respondents understood the question as it was asked. 

One suggestion for potential change may be to keep the structure of the answers the same. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 59

Item 4

Reading: NA

Understanding: Most of the respondents seemed to understand the item as it was, though some 

students misconstrued the question and thought it had to do with learning about culture rather than 

language. 

Changing Answers: One student changed their answer from B to A as a result of answering Probe 6.

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Seventeen of 23 responding students seemed to interpret the item correctly, five thought it had to do 

with culture or tradition, and one student was confused. 

Probe 3: What does the phrase “American Indian or Alaska Native language” mean to you? Can you 

give me an example of “American Indian or Alaska Native language” that you can speak or know 

about?

Twelve of 23 respondents gave examples of specific languages, three said they did not know, three 

referred to “the language they speak,” two mentioned taking classes, two were confused by the 

question, and one would not give an example. 

Probe 4: How many American Indian or Alaska Native languages do you speak?

Fourteen out of 23 respondents said they spoke at least one language—or had some sort of knowledge 

of one language. A few mentioned being able to understand but not speak. Five mentioned non-AI/AN 

languages they could speak, four said they did not speak any AI/AN languages, and one student said 

they spoke two AI/AN languages.

Probe 5: How would you answer this question if you speak more than one American Indian or Alaska 

Native language?

Answers to this probe were varied among the 16 respondents, leading interviewers to posit that 

hypothetical questions are difficult for many respondents to answer. Three students gave D as their new 

Student #4 New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska 

Native language?

A. My school does not offer courses that are taught in an American Indian or Alaska 

Native language.

B. Never or hardly ever

C. Once or twice a month

D. Once or twice a week

E. Every day or almost every day
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 60

answer, three were confused by the question, three said they would answer E, three stated that they do 

not speak more than one language, two said their answer would stay the same, one said they would 

speak more often (but did not specify how much more often), and one mentioned a non-AI/AN 

language. 

Probe 6: What is the difference between answer choices A and B?

Twelve students of the 23 respondents who responded to this probe responded in a seemingly 

appropriate manner; however, there seemed to be a fair bit of confusion related to the fact that answer 

choice A mentions school in the answer options but answer choice B does not. Five respondents gave 

responses that indicated that they were confused by the question, two mentioned that family would 

teach them, one said there were too many tribes at the school to offer language classes, one said there 

was no difference between the two answer choices, and one felt they were the same.

Probe 7: If there were no courses offered in your school that are taught in an American Indian or 

Alaska Native language, which answer choice would you select?

Fourteen of 22 students responding answered A, five answered B, and three answered D. 

Solutions to Improve Item

Respondent: NA

Interviewer: Make answer choices consistent to cut back on confusion. Students seem to interpret only 

answer choice A as being specifically related to school. They seem to interpret the other answer choices 

as having more to do with use, and not necessarily related to classes or school. 

Initial KAI Recommendation (Keep/Change/Discard)

Change: The item seems to get distorted by the way the answer choices are phrased. We recommend 

making wording consistent and potentially highlighting the phrase “in school” if the interest is in 

whether or not students learn a language at school and not at home, informally, or through a tribal or 

cultural center. 

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about 

classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer 

choice “A”, ETS recommends changing the word “courses” to the word “classes” to be consistent with 

the item stem. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 61

Item 5

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 23 respondents appropriately rephrased the question. All understood that the item was asking them 

to determine how well they could speak in an AI/AN language. 

Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?

Of the 23 respondents, 22 essentially reiterated the probe and said that they thought the question, “I 

can speak a few words or phrases,” meant being able to speak a few words. One student said they did 

not know. 

Probe 4: In answer choice C, what does the phrase “I can speak well” mean to you?

All 21 students responding said that “I can speak well” meant speaking a language with fluency—the 

amount of fluency ranged from one paragraph to being a fully fluent speaker. 

Probe 5: Do you think that additional answer choices should be added between B and C? If yes, can 

you give me examples of what those answer choices would be?

Out of the 23 respondents, 14 students did not feel that there was a need for additional answer choices. 

Nine students indicated that there could be additional answer choices. One of the suggestions that came 

up three times was the notion of being able to understand but not speak a language, or something 

between speaking a few words and phrases and speaking a language well.

Solutions to Improve Item

Respondent: Add an additional answer choice between B and C. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider adding an additional answer choice between B and C. 

Student #5 New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?

A. I cannot speak an American Indian or Alaska Native language.

B. I can speak a few words or phrases.

C. I can speak well.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 62

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 63

Item 6

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents interpreted the item as it was intended and understood that it was asking them to 

determine how well they could read in an AI/AN language.

Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: Can you give me an example of “American Indian or Alaska Native language” that you can 

read?

Eleven of 12 respondents mentioned specific languages, and one mentioned basic literacy in a language 

but did not specify which one. 

Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up 

question]: Can you give me an example of “American Indian or Alaska Native language” that you can 

read?

Eighteen of 20 respondents indicated that they felt that the question meant that you could read some 

of a language but not all of it, or that you could make sense of some works and that you could 

pronounce things but not necessarily understand it. Two students indicated that they did not know. 

Probe 5: In answer choice C, what does the phrase “I can read well” mean to you?

Fourteen of 18 students responding said they thought it meant that you could read something with 

ease (such as picking up a book and being able to read it). One mentioned comfort with reading, while 

another pointed to the idea of ease of reading. Two students answered that they did not know.

Probe 6: Do you think that additional answer choices should be added between B and C? If yes, can 

you give me examples of what those answer choices would be?

Fifteen of 21 respondents said that the answers were fine as they were. Six students felt that additional 

answer options could be added—at least three of these students heralded back to their suggestions 

from the last item having to do with speaking. Their suggestions were to add something about being 

Student #6 New/VH040067

How do you rate yourself in reading in an American Indian or Alaska Native language? 

A. I cannot read in an American Indian or Alaska Native language.

B. I can read a few words or phrases.

C. I can read well.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 64

able to read but still stumbling over words, something between B and C, fluent reading, and sounding

out words but not making sense of them.

Solutions to Improve Item

Respondent: There were a few suggestions for additional response options between B and C.

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally seemed comfortable with the question. Consider making answer choices 

uniform in phrasing. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 65

Item 7

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 22 students responding essentially restated the item using slightly different words. 

Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?

There were varied answers among the 20 respondents. It seemed that many of the students did not

really understand what the probe was trying to get at. Twelve indicated that they were confused or did 

not understand the probe, two said it had to do with knowing the language, two said it had to do with 

Native languages, one said they did not know any Alaska Native languages, one restated the probe, one 

said it had to do with the “old ways,” one said it meant you could write and speak a language, and one 

thought it had to do with culture.

Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska 

Native language”?

Again, there was a lot of variation among the 20 respondents to the answers, and the probe seemed to 

not make sense to a lot of the respondents. Eight were confused by the question, three mentioned 

specific words in an AI/AN language, two mentioned specific languages, two said it had to do with the 

tribe, three thought it might refer to words for animals, shapes, prayers, or songs. One student 

mentioned tongue positioning for pronunciation, one mentioned pronunciation, and one student said 

they did not have any examples.

Student #7 New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?

A. No one has taught me about an American Indian or Alaska Native language.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): _____________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 66

Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

Of the eight students answering the question, five said that they did have family members who were 

tribal members, and all said they thought of them as family first. Two students said no, and one said 

they were not sure. 

Probe 6:

NA

Solutions to Improve Item

Respondent: NA

Interviewer: Probes 3 and 4 were confusing to students as they could be considered somewhat vague 

questions. We recommend asking more specific probe questions in the future. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The item seems to be fine as is. Reconsider keeping answer choices phrased similarly. Probes 

could use some re-tooling if they are to be used in the future. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 67

Student #10

Item 10

Reading: NA

Understanding: One student reportedly had trouble making sense of answer choices. 

Changing Answers: One student changed their answer several times while working through options for 

the A portion of the question. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Thirteen of 21 students responding essentially repeated the question with varying degrees of sameness, 

referring to whether or not they had learned about Native culture through different means, such as field 

trips or presentations. Four students thought that the question was referring only to things that 

happened within the school, two thought the question was only referring to what they had learned 

about AI/AN cultural aspects (no mention of school), one student thought the probe was referring to 

their thoughts on the item structure, one student interpreted the question as having to do with what 

they had learned in class, and one student thought the question was referring to what presentations 

they had given at school. 

Probe 3: How do you know that other 8th-grade students attended, or that activities were not offered 

to 8th-grade students?

A variety of answers were given among the 17 respondents for this probe, indicating some validity 

issues with understanding the probe. Six students said that other students were there (indicating that 

that was how they knew that other students attended), five were confused by the question, three said 

that they were there and that was how they knew, two students said it was difficult to answer because 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 68

they were unsure about what was happening at other schools, one said they did not know, one said no 

one was told, and another mentioned that they had gone in another grade.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

questions]: Do you know whether other 8th-grade students had the activities? Again, there was a 

variety of answers among the 18 respondents suggesting that students may not have fully grasped the 

probe. Thirteen said “yes,” four said they were there, three indicated that they did not understand the 

question, one said that none of the students had activities, one simply said “no,” and another simply 

said that they did know.

Probe 5: For questions a. through d., how would you answer this question if you did not attend these 

activities and did not know whether other 8th graders attended these activities? 

Again, there was wide variety among the 19 responses. Four students indicated that they were confused 

by the question, two said B, seven said C, two said that these activities were not offered, one said “yes,” 

one indicated they would still give the same answer, one student was not sure, and one student 

quipped, “how would I know.”

Solutions to Improve Item

Respondent: One student suggested that the questions would be difficult for a new student not familiar 

with activities prior to their arrival to answer. 

Interviewer: Item and probes should be reconsidered for this question as they seem to confuse 

respondents and the validity of the data is questionable. 

Initial KAI Recommendation (Keep/Change/Discard)

Change: Perhaps it would make more sense to ask this question in a more simplified format as it was for 

4

th graders, or just make answer choices “yes” or “no.” Additionally, it may be helpful to determine the 

most salient pieces of information to be gathered from asking the question and tailor the phrasing 

better to best capture that—perhaps bold “through your school” and consider bolding “During the 8th

grade.” 

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that the format of the item should be simplified by collapsing the answer 

choices to “Yes” and “No”.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 69

Student #14 Revised/VC996624

Here are some sentences about reading. Fill in one oval on each line to show whether the sentence 

describes a person like you.

This is not 

like me.

This is a 

little like 

me.

This is a lot 

like me.

a. When my teacher talks about American 

Indian or Alaska Native history or culture, I 

try to read more about it. 

A B C

Revised/ 

VH043704

b. I enjoy reading about American Indian or 

Alaska Native people. A B C

Trend/ 

VC996627

c. I enjoy reading about people who have 

different traditions and cultures (ways of 

life, customs) than I have.

A B C

Trend/ 

VC996628

Item 14

Reading: One student misread “sentence” as “sometimes.”

Understanding: Students had varied interpretations of the item that were all peripherally related to the 

question at hand but were dynamic in the different pieces that students picked up on—outlined further 

in Probe 1. This variation in response may be due to the fact that there are three different questions to 

summarize in Item 14.

Changing Answers: One student changed their answer after responding to Probe 4 as they realized they 

had confused or misread the answer choices for A and C. 

Probes

Probe 1: Can you tell me, in your own words, what question a. is asking?

The 22 responses to the probe were varied with students picking up on different pieces of the overall 

item questions. For example, eight students felt the question was asking them about learning more or 

researching on their own, two interpreted the question as having to do with enjoying learning, two 

commented on the structure of the item questions and answers, and two gave difficult-to-characterize 

answers such as, “It is basically like a little test to see if you are that person or not” and, “Sounds like 

they’re talking about what you would describe your friend as.” Other responses had to do with interest, 

whether or not someone studied, reading, paying attention, if the teacher covered AI/AN 

issues/cultures, if students thought of their tribe when teachers were talking about AI/ANs, and whether 

or not students were interested in the topic. Again, the wide variety is likely due to the fact that there 

are three questions and students were essentially being asked to sum them up into one.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 70

Probe 3: In question a. what does the phrase “when my teacher talks about” mean to you?

All 23 respondents felt that the phrase “when my teacher talks about” referred to when their teacher 

was teaching—either reading something to them, lecturing, or as part of a class activity. 

Prove 4: Which answer choice would you select if your teacher had not talked about American Indian 

or Alaska Native history or culture? Why would you select this answer choice?

Ten of 22 respondents said they would give A as an answer, six said they would choose B, and two said 

they would choose C. Two students said they would “do more” but didn’t elaborate on whether or not 

that meant they would change their answer to B or C, and two students said they would keep their 

answer the same. One student was confused by the question. Interestingly, and primarily among the 

students who answered A, five students indicated that if their teacher was not telling them about AI/AN 

history or culture, then how would they know to go look anything up?

Solutions to Improve Item

Respondent: Possibly put answers in boxes rather than at top of column with corresponding letters.

Interviewer: Consider dropping Probe 3 as the concept is easily understood. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It is hard to tell if anything should be changed with this item given the variety of responses to 

Probe 1. Again, it is very likely that the wide variety is due to students trying to describe all three 

questions from this item at once rather than the item itself being confusing. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 71

Item 16

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 23 respondents mentioned helping others with school work. Four specifically mentioned 

helping classmates or peers with school work, one specifically mentioned friends, and one spoke about 

helping younger students.

Probe 3: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of how you have helped other students with their schoolwork? 

Are the other students that you have helped with their schoolwork in 8th grade? If not, what grade

are the students that you have helped with their schoolwork in?

Of the 19 respondents, 12 students mentioned helping other 8

th graders, five said they helped students 

in other grades, five mentioned siblings or family members. With regard to areas where help was 

provided, five said they just worked through and explained things to other students, four mentioned 

math specifically, four mentioned language or reading, three mentioned helping on projects, three 

pointed to helping with homework, and one mentioned science.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

Student #16 New/ VH040389

How often do you help other students from your school with their schoolwork? For example, 

you might help other students to study for a test, help them with a school project, or go over 

their homework with them.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a week

D. Every day or almost every day
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 72

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 73

Student #17 Trend/VC996635

How often do any of the following people help you with your schoolwork? For example, they might 

help you to study for a test, help you with a school project, or go over your homework with you. Fill in 

one oval on each line.

Never 

or 

hardly 

ever

Once 

or 

twice a 

month

Once 

or 

twice a 

week

Every 

day or 

almost 

every 

day

a. A parent or someone else from my 

family A B C D

Trend/ 

VC996636

b. A teacher or another adult from my 

school A B C D

Trend/ 

VC996637

c. Another student

A B C D

Trend/ 

VC996639

d. A tribal representative or elder A B C D New/VH018268

e. Someone else who lives in my 

community or is a friend of my family A B C D

Trend/ 

VC996640

Item 17

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 23 students responding felt the question was asking them about who helps them with their 

homework, five thought the question was focused on how often others helped them with their 

homework, one student made a comment about how the item is structured (rather than focusing on 

content), and one student was focused on the question being about someone else who lived in the 

community.

Probe 2: Can you tell me, in your own words, what question d. is asking?

Seventeen students out of 23 respondents answered the question in a manner that was in keeping with 

the meaning of the item though most focused on tribal representatives over elders, likely because the 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 74

two are posed in a question together. Six students seemed confused by the question and gave answers 

such as, “I try an elder that has been in, that has known a lot about schoolwork and stuff and knows a lot 

about the tribal stuff,” or, “Do you have any... Do you have any - I don't know,” or, “Not really. Usually a 

tribal representative or something or elder. I'm not really sure what it means ... like elder, or is that like 

an older person?”

Probe 3: Can you tell me, in your own words, what question e. is asking?

Seventeen of 23 respondents seemed to understand Probe 3 with ease. Two mentioned elders, two 

mentioned family members, and one mentioned tribal representatives. One pointed to the fact that 

they lived far from their community so they were unsure how to answer and one thought the question 

was focused on how often you were helped by the community member or family friend rather than 

what the phrase meant. Two seemed to not fully grasp what the question was asking and gave varied 

responses such as, “Say you're given a month and you had problems within the month, you see how 

many ... which one of your parent, teacher, other student, tribal representative, or someone else helps 

you,” and, “D, my ... one of the elders, Nana Grace has helped me doing a project on Native Americans. 

She was really helpful. [Nana Grace is family].”

Probe 5: In question e., what does the phrase “someone else who lives in your community or is a 

friend of your family” mean to you?

Sixteen out of 22 students responding answered the question in keeping with what the question was 

asking, and a few mentioned that the question was very similar to Probe 3. Six mentioned neighbors, six 

mentioned family friends, two spoke of family, two said something about random people in the 

neighborhood, one noted their tutor, and one mentioned that they did not live in a community. 

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

questions]: Can you give me examples of these people who have helped you with your schoolwork? 

Can you give me examples of how these people have helped you?

Ten of 20 respondents mentioned family—family included mothers (primarily), parents, grandparents, 

aunts, and uncles. Seven noted their siblings or cousins, eight referenced teachers, four spoke of friends 

helping, and two mentioned family friends. Two gave difficult-to-classify answers such as, “Like, for a 

tournament... If I'm in a basketball tournament, I'll go to my friend Braden. His uncle is like.... He's like, a 

coach for us, when tournaments come, so I'll ask him to help me. Like, "How should I do this?" Or, "How 

should I play in the game?" And he'll help me do it.” Or, “If I have certain problems with a subject, they 

make sure I focus on that one subject so I get it easier.”

Solutions to Improve Item

Respondent: One student wanted the additional answer choice “whenever they can.”

Interviewer: It would be helpful to ask about elders and tribal representatives separately as students 

choose to respond about one and not the other. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It appears that the item is easily understood by most students. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 75

ETS Recommendation (05/02/2014)

Keep this matrix item, but drop sub-item “e” VC996640.

Rationale: ETS recommends dropping this item (VC996640) to reduce student burden. Having five subitems about who is helping the student with his/her schoolwork is excessive. Asking about "someone 

else who lives in your community/friend of your family" seems to involve a nuanced understanding that 

is not being exhibited by some of the respondents to this question. 

The similar grade 4 item VC759291 is also proposed to be dropped.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 76

Student #19 Trend/ VC996644

During 8th grade, how many times have you talked to each of the following people about the classes 

you should take in high school or about what you want to do after high school? Fill in one oval on 

each line.

Never

One 

time

Two or 

three 

times

Four or 

more 

times

a. A family member A B C D Trend/VC996645

b. A teacher A B C D Trend/VC996646

c. A school counselor A B C D Trend/VC996647

d. Another student A B C D Trend/VC996648

e. A tribal representative or elder A B C D New/VH026209

f. Someone else who lives in my 

community or is a friend of my 

family

A B C D

Trend/VH026211

Item 19

Reading: One student misread the answer scale. 

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The majority of the 23 students providing answers seemed to understand the gist of the item but took 

away different pieces from it, likely based on the number of answer options and trying to rephrase them 

into a cohesive restatement of the question. Eight felt the question was focused on talking to someone 

about future plans, five felt it had to do with how often, four interpreted the question as having to do 

with talking to family members, two mentioned classes, two felt that the focus was who they had 

spoken with, one talked about friends, and one seemed to think the question was tied to success. 

Probe 2: Can you tell me, in your own words, what question e. is asking?

This question had a variety of answers among the 18 respondents with 10 students referencing tribal 

representatives and seven mentioning elders. One student mentioned wisdom, one said something 

about speaking with a school president, one mentioned family friends, one thought it had to do with 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 77

people who came to the school to talk about such topics, one mentioned their sister’s boyfriend, and 

one student thought the question had to do with how many tribal representatives or elders they had 

spoken with. The combination of elders and tribal representatives often seems to prevent students from 

parsing out the differences between them. However, the overlap is often very great because most 

representatives will be their elders, but not all elders are representatives. 

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

Of the two respondents, one student said “yes” and that they thought of them as family first, and one 

student said “no.”

Solutions to Improve Item

Respondent: NA

Interviewer: Potentially consider ways to split out elders and tribal representatives as different 

categories or just be aware of the issues of overlap and what that may mean for data interpretation. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It appears that the question is easily understood by students.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 78

Student #20 New/VH040157

How much do you agree with each of the following statements? Fill in one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. If I put in enough effort I will succeed 

in school. A B C D New/VH040158

b. If I put in enough effort in school I 

will get into college. A B C D New/VH040160

c. If I put in enough effort in school I 

will get a good job. A B C D New/VH040161

Item 20

Reading: NA

Understanding: NA

Changing Answers: One student changed their answer at Probe 5 from “strongly agree” to “agree” due 

to the idea that some people do not have enough money to go to college. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fourteen of 20 respondents interpreted the question as having to do with the correlation between 

effort and success. Six students spoke to the item structure.

Probe 3:

Nineteen students of the 23 responding spoke about trying hard and working hard in school, four 

mentioned doing homework, two pointed to paying attention, one mentioned passing class, and one 

mentioned grades.

Probe 4:

Fourteen of 23 respondents referred to getting good grades, 11 referenced graduation, three pointed to 

good behavior, one mentioned getting an education, and one spoke about their future.

Probe 5:

All 23 respondents seemed to struggle a bit with this probe because it was so straightforward a 

question. Twenty-one said it meant getting into college; of these 21, three mentioned scholarships, and 

two mentioned going to a college of choice. Other answers were that the question had to do with 

money (one student), further education (three students), being smart (one student), having a career 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 79

(one student), and how big an achievement that would be for someone from a small town (one 

student). 

Probe 6:

Nineteen of 23 respondents interpreted this question as having to do with being able to get the job that 

you want—your job of choice. Specific jobs mentioned varied among the 23 responses such as lawyers,

doctors, and teachers were mentioned as was having good hours and pay and not working at 

McDonalds. Other professions mentioned were architecture, being a judge, photographer, business 

owner, and river runner among others.

Solutions to Improve Item

Respondent: NA

Interviewer: Probe 5 is very straightforward and thus actually a little confusing to answer for some 

students, as if they suspect that it may be a trick question. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 80

Student #21 New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student tardiness A B C D New/VH040217

b. Student health problems A B C D New/VH040219

c. Student misbehavior in class A B C D New/VH040221

d. Physical conflicts among students A B C D New/VH040223

e. Bullying A B C D New/VH040226

f. Low student motivation A B C D New/VH040227

g. Low teacher expectations A B C D New/VH040231

Item 21

Reading: One student stumbled over a few words but reading did not seem generally to be a problem.

Understanding: Two students seemed to be interpreting the item in terms of how it related to them 

personally, as if the focus of the question were on them rather than on the school they attended. 

Changing Answers: There were no changing answers, but there was some difficulty choosing answers. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Sixteen students out of the 19 respondents felt the item and subsequent questions wanted them to 

reflect on problems at their school, one student seemed to interpret the probe on a personal level, one 

said they did not know what the question was asking, and one thought it had to do with expectations.

Probe 3: What does the phrase "student tardiness” mean to you?

Twenty of the 23 students responding felt the question had to do with lateness—both to school and to 

class. Five students felt it also had to do with absence from school, and one said, “If they actually want 

to go to school and just learn new things.”

Probe 4: In question a., what does the phrase "student health problems" mean to you?

Thirteen students out of 23 respondents felt the question was asking about students getting ill or sick, 

four mentioned obesity, three brought up asthma, one said cancer, one noted broken limbs, one said 

autism, one mentioned drugs or alcohol, one talked about scoliosis, and one mentioned disabilities in 

general.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 81

Probe 5: In question c., what does the phrase “student misbehavior in class” mean to you? 

Out of all 23 respondents, 12 students felt that the question had to do with students’ disruptive 

behavior or “acting up” in class, eight mentioned talking in class or talking back to the teacher, five said 

it included not listening to the teacher, and four said it meant not doing work. 

Probe 6: In question d., what does the phrase “physical conflicts among students” mean to you?

Nineteen of 23 respondents indicated fighting, four mentioned teasing or bullying, and two interpreted 

the question as also being related to verbal conflicts like teasing. 

Probe 7: In question e., what does the phrase “bullying” mean to you?

Fourteen of 23 respondents talked about verbal conflicts such as teasing, picking on others, and talking 

behind someone’s back. Six felt that bullying could also be physical, and five spoke of other aspects such 

as the jealousy of bullies, the troubles with being “different,” and students sparring with one another.

Probe 8: In question f., what does the phrase “low student motivation” mean to you?

This probe and the following one seemed to cause a little bit of confusion to all 23 respondents as they 

had a harder time explaining what they thought the concepts meant without generally restating the 

phrase they were being asked about. For instance, 14 students felt that “low student motivation” meant 

that students had no motivation. Some explained a bit more by saying that some students saw no point 

in trying hard at school, or were not being encouraged to do better by friends or teachers. Nine 

mentioned that low motivation meant not trying hard, and one said it meant sleeping in class. 

Probe 9: In question g., what does the phrase “low teacher expectations” mean to you? 

As with the previous probe, many of the 23 respondents’ answers fell into the same category as many 

students tended to rephrase the phrase they were being asked to interpret. To that end, 18 students felt 

that “low teacher expectations” meant that teachers did not expect much from their students, that they 

did not care and were not motivating students. Four students said that they did not know what the 

question was trying to ask, and one interpreted the question as having to do with whether or not people 

had high expectations of their teachers. 

Solutions to Improve Item

Respondent: NA

Interviewer: It is hard to say if students were answering for their 8th grade class or their school overall. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Perhaps consider what is included when asking about the school in general versus just asking 

about the 8th grade. 

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students 

as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 82

the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; 

therefore the items should reference "AI/AN" when necessary in order to capture the most useful data. 

Drop sub-item “g” VH040231.

Rationale: ETS recommends dropping this item (VH040231) to reduce student burden. This sub-item 

caused much more confusion for respondents than any of the other sub-items within this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 83

Survey Instrument: Student Grade 8/Sequence II

Item 1

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fourteen of 25 responding students felt the item was asking them to think about who had taught them 

the most of what they knew about AI/AN history. Four said they did not know, three mentioned history 

and culture, three felt the question was asking if they knew about AI/AN history, and one student felt 

the question had to do with tribal elders. 

Probe 3: What does the phrase “American Indian or Alaska Native history” mean to you? Can you give 

me examples of “American Indian or Alaska Native history” that you have learned?

Eight out of 23 responding students simply reiterated that the question was asking about history, four 

spoke about culture, four mentioned specific tribes, and three felt it had to do with AI/AN lifestyles of 

the past. Four respondents mentioned drumming, art, dance, or storytelling. Two mentioned language, 

two mentioned religion, and one student brought up ancestors.

Probe 4: In answer choice B, what does the phrase “Family members” mean to you?

Twelve students out of 24 respondents mentioned specific family members such as grandparents, 

parents, aunts and uncles, siblings, and cousins. Five said that “family members” were people who were 

“related,” five reiterated the term “family,” three mentioned that they considered the reservation or 

part of their family. One student mentioned friends, and one talked about love. 

Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? 

Can you give me examples of “Tribal representatives or elders”?

Student #1 New/VH039733

Who taught you most of what you know about American Indian or Alaska Native history?

A. No one has taught me about American Indian or Alaska Native history.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders 

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 84

Eleven of 25 respondents spoke of family, six talked about tribal councils or chair people, three 

mentioned being old and/or wise, two students mentioned specific people, six mentioned the tribe. The 

responses to this question were fairly dispersed due to the fact that the question covers two groups of 

people who may or not may not overlap in terms of their roles as elders or tribal representatives.

Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

Seven out of 12 students responding indicated that they did have a family member who was a tribal 

representative. Of these seven, four said they thought of this person as a family member first, two did 

not specify, and one said they thought of the person as a tribal representative first. Five students 

reported that they did not think that they had a family member who was also a tribal representative. 

Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

Only one student chose F as their answer for the item and this student gave a specific name. 

Probe 8: What does the phrase “most of what you know” mean to you?

This question seemed to confuse students somewhat as it is a straightforward question, which at times 

leads to students either repeating almost verbatim what the question was, or trying to come up with an 

answer that is different. Of the 17 students responding, six felt the question was asking them about the 

most or the majority of what they knew, five thought it meant what they had learned, three felt it had 

to do with language, two thought it was asking about everything that they knew, and two said they did 

not know.

Probe 9: Which answer choice would you select if multiple people had taught you about American 

Indian or Alaska Native history? How would you decide which answer choice to select?

As with previous hypothetical probes, this one did not generate the same kind of reasoning that may 

have been needed to answer the question completely. Of the 17 students responding, seven said that 

they would choose B, five said they would choose D, two said E, one said F, one said that they did not 

know, and one said they would not choose more than one answer. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of the students seemed to grasp the meaning of the item overall. Consider keeping 

all answer choices phrased in a similar fashion, and consider if there is a need to separate out tribal 

representatives from elders or to keep the question as is. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 85

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 86

Item 2

Reading: One student misread the answer for E.

Understanding: One student initially thought this was a repeat of Item 1. 

Changing Answers: One student initially chose answer A but then quickly changed to B. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fourteen of 23 responding students interpreted this question as having to do with who had taught them 

the most about what they knew about AI/AN traditions, two mentioned how much they knew, one 

thought the question was the same as the last, one simply said “no.” Five gave difficult-to-classify 

answers ranging from, “About American Indian Native tribes. It is about our tradition,” to, “What they 

did or what they do, the people, American Indians” or, “What do they do? We have Christmas, 

Thanksgiving, what do they do? ["They?"] The tribes.”

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions” 

mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you 

have learned?

Eight of 25 responding students mentioned dancing and drumming, six brought up festivals and 

powwows, three spoke of religion, three talked about food ways and hunting, two brought up specific 

tribes, two mentioned language, one discussed leaders, one brought up clothes, and yet another 

mentioned ceremonies.

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives?

Four of six students responding to this question said that they did not have family members who were 

tribal representatives, and two said they did and thought of them as family first. Five had already 

answered the question previously.

Student #2 New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions 

(way of life, customs)? 

A. No one has taught me about American Indian or Alaska Native traditions.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders 

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 87

Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students appear to understand the question. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 88

Item 8

Reading: NA

Understanding: Twelve students expressed confusion over the term “systems of counting” and did not 

know what it referred to. This calls into question the validity of the responses to this question. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twelve out of 25 responding students indicated that they did not know what the question was trying to 

ask or what systems of counting were. Though several students came up with answers, it seemed that 

there was a lot of guessing as to what “systems of counting” meant. Six focused on the part about who 

taught them about systems of counting, four guessed that the question meant counting in an AI/AN 

language, three thought it might have to do with tribes being able to keep tabs on their population 

counts, one thought it might have to do with money.

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native systems of 

counting” mean to you? Can you give me examples of “American Indian or Alaska Native systems of 

counting” that you know about?

Eleven of 22 respondents said they did not know, eight said that they thought it had to do with 

language, two brought up population counts, and one mentioned money.

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

Only two students responded to this probe and they thought of these people as family first. One 

student had already been asked the question and had already answered it. 

Student #8 New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of 

counting?

A. No one has taught me about American Indian or Alaska Native systems of counting.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 89

Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: 

Who is the person?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: Define what is meant by “systems of counting.”

Initial KAI Recommendation (Keep/Change/Discard)

Discard: Students clearly did not grasp what the question was trying to ask but would often provide 

answers B-E, thus calling the validity of such responses into question.

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS 

recommends revising the phrase to "ways of counting".

This recommendation is aligned with the similar grade 4 item (VH040103).
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 90

Item 9

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The 25 responses given for this probe varied greatly as students were not sure what the question was 

asking or what “symbols and designs used in mathematics” meant. Five students focused on the part of 

the question regarding who taught them what they know, three mentioned math, four said they did not 

know, one said shapes, and one referenced algebra. Nine gave answers that were difficult to categorize. 

Some of these are shared below: 

“About the symbols they used to tell each other what to do and yeah...”

“What does it mean like they’re never direct like Egyptians people really have their [inaudible 00:07:07] 

so they want to know what it is, probably like that.”

“I guess it’s the way they do stuff, like making a teepee. They would have to do all this counting and 

stuff.”

“In mathematics, I'm pretty sure I know that symbols and designs, but I'm not really sure what it is doing 

in mathematics.”

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native symbols and 

designs” mean to you? Can you give me examples of “American Indian or Alaska Native symbols and 

designs” that you know about?

Eleven of 24 responding students felt that “symbols and designs” had to do with decorative symbols, 

usually in art. Six suggested that perhaps they were variations on plus, minus, greater than, division, and 

Student #9 New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and 

designs used in mathematics?

A. No one has taught me about American Indian or Alaska Native symbols and designs 

used in mathematics.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 91

multiplication symbols in math. Five said they did not know, and two said that they did not think that 

symbols and designs were used in math.

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up 

questions]: Do you have any family members who are also tribal representatives? If yes, do you first 

think of them as your family members or as tribal representatives? 

Only one student was asked the question; they answered “no.”

Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: Who is the person?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: This item should either be clarified or removed. 

Initial KAI Recommendation (Keep/Change/Discard)

Discard: Remove the question as there is a lack of clear understanding about what the question is 

asking. The lack of understanding calls into question the validity of the answers given by students. 

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting 

the phrase "symbols and designs", and the specific reference to math caused additional confusion. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 92

Student #11 Revised/VH039972

During 8th grade, how often have you used any of the following materials in school? Fill in one oval 

on each line.

Never 

or 

hardly 

ever

Once or 

twice a 

month

Once 

or 

twice a 

week

Every 

day or 

almost 

every 

day

a. Books, videos, or other materials 

(including online resources) 

about American Indian or Alaska 

Native history, traditions, or 

cultures (way of life, customs)

A B C D Revised/VH039974

b. Books, videos, or other materials 

(including online resources) not 

related to American Indian or 

Alaska Native people, traditions, 

or cultures

A B C D Revised/VH040079

c. Books, videos, or other materials 

(including online resources) 

about current issues that are 

important to American Indian or 

Alaska Native people

A B C D Revised/VH040083

Item 11

Reading: One student misread question row B, one misread the question as “today” rather than “8th

grade.”

Understanding: Five respondents expressed some confusion over how to make sense of the item and 

the different questions posed within it. 

Changing Answers: One student changed their answer to row B from A to B after a discussion of the 

item’s meaning. Another student expressed some difficulty choosing an answer for row A and row C. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The 25 responses to this question were varied in part perhaps because respondents were being asked to 

restate three different questions as one. Four students thought it had to do with how often resources 

were used, four felt it was asking if they had ever used them, two restated the question, one thought it 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 93

had to do with how often they used their AI/AN culture in their everyday life, and one thought it had to 

do with learning traditions. Two felt that the question was asking about where they learned about AI/AN 

culture and traditions, two said they did not know, and two restated all three portions of the question in 

their own words. Six students gave difficult-to-categorize answers such as: 

“The bad stuff that's happening, not the bad stuff but the important stuff. [Happening to what?] 

Something that’s important to them. [Who?] To the American Indians and Alaska Natives.”

“Like what's going on right now, in that culture or whatever.”

“Traditions they used, or how they used to work their life.”

Probe 3: In question b., what does the phrase “books, videos, or other materials not related to 

American Indian or Alaska Native people, traditions, or cultures” mean to you? Can you give me 

examples of these books, videos, or other materials?

There were a variety of answers to this question from the 25 respondents. Twelve students referenced 

doing research or project work on topics not having to do with AI/ANs, three talked about history books, 

four mentioned going online to do research about anything (or looking at Facebook or viral videos), one 

student thought the question still had to do with AI/ANs, three said they did not know, and two 

mentioned research in general. 

Probe 4: In question a., what does the phrase “online resources” mean to you? Can you give me 

examples of “online resources”?

Twelve of 25 respondents specifically mentioned going to websites to learn more about their AI/AN 

culture, nine mentioned Google, four talked about Bing, two mentioned YouTube, two mentioned 

Facebook, one said Twitter, one mentioned the media, and one said anything on the Internet.

Probe 5: What does the phrase “in school” mean to you?

As this question was very straightforward, it produced some interesting answers among the 22 

respondents where students seemed to be unsure of the answers because the answer seemed so 

obvious. Sixteen said that “in school” meant in school, two said they did not know, one thought it had to 

do with education broadly, one said it had to do with what they were learning, and five gave difficult-toclassify answers:

“That means like spent in at max.”

“Having someone come in and talk to us about it or lessons with video clips about it.”

“Using books, reading, talking about it socially.”

“Boring. [Interviewer prompts.] Prison, learning, get educated, getting you ready for high school.”

Probe 6: Have you used any of the books, videos, or materials listed in questions a. through c. outside 

of school, but not for “in school” class work or assignments? If yes, do you consider those activities to 

be “in school” class work or assignments?

Ten of the 21 students responding said that they had used materials outside of school and five of those 

considered that work to be school-related. Five said they did not consider that work school-related. Ten 

students said that they did not use materials outside of school, and one said probably. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 94

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It is difficult to make a recommendation on this particular item given the lack of validity in how 

students interpreted the item, which may be due in large part to having to summarize three questions 

into one. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 95

Student #12 Revised/VH040104

During 8th grade, how often have you used any of the following materials outside of school? Fill in 

one oval on each line.

Never 

or 

hardly 

ever

Once or 

twice a 

month

Once or 

twice a 

week

Every 

day or 

almost 

every 

day

a. Books, videos, or other 

materials (including online 

resources) about American 

Indian or Alaska Native history, 

traditions, or cultures (way of 

life, customs)

A B C D Revised/VH040106

b. Books, videos, or other 

materials (including online 

resources) not related to 

American Indian or Alaska 

Native history, traditions, or 

cultures

A B C D Revised/VH040109

c. Books, videos, or other 

materials (including online

resources) about current issues 

that are important to American 

Indian or Alaska Native people

A B C D Revised/VH040111

Item 12

Reading: NA

Understanding: Two students had to be redirected about the question referring to materials “outside of 

school.”

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fifteen of 24 respondents said that the question was asking about what they did outside of school, two 

restated the question, and one said they did not understand. Six gave hard-to-qualify answers such as:
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 96

“It's asking you how often you would go and look up stuff about Native Americans or the way of life or 

what's going on with them in oppose to how much you don't look up about them while you're still on 

the Internet.”

“It's talking about outside and we just came back from outside. I get to play basketball, kickball, 

stickball.”

“Things you don't do outside of school.”

“What you do the most or what you like to do.”

Probe 3: What does the phrase “outside of school” mean to you? Do you consider “outside of school” 

to include all activities that are for schoolwork and not for schoolwork?” 

Out of 25 respondents, 15 said that “outside of school” meant outside of school. This question seemed 

to make students uncomfortable because of how easy it was. Three said doing things on their own time, 

two said it had to do with hanging out with friends, two said they did not know, one said being at home,

and two gave hard-to-categorize answers: 

“Like at home or at people's houses or powwows or something. [All activities included?] Like kind of for 

school.”

“Mmhmm. Sometimes, but whenever it gets kind of fun for friends and play. I'll start reading just like-“

Probe 4: In question b., what does the phrase “books, videos, or other materials not related to 

American Indian or Alaska Native people, traditions, or cultures” mean to you?

Sixteen of 24 respondents said that the phrase in question B just meant researching different topics that 

did not have to do with AI/ANs. Four thought it had to do with their or another AI/AN tribe, and four 

said they did not know. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that most respondents seem to understand this question based on responses to the 

item. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 97

Item 13

Reading: NA

Understanding: One student said they were unsure what “media center” meant. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 22 students responding to the question mentioned having access to different materials;

only six mentioned in school. One student said they did not know and two thought the question had to 

do with where you would go if you did not have access. 

Probe 3: What does the phrase “school library, media center, or resource center” mean to you?

Nineteen of 24 respondents essentially defined the idea of “school library, media center, or resource 

center” as a place to go to find materials. However, it became clear that many students were unsure of 

what a resource or media center were (10), though they often hazarded guesses, and it seemed that 

many students interpreted “resource center” as a place outside of school. This is perhaps because 

“library” is the place they are most familiar with, and also it is the only term with “school” before it.

Probe 4: What does the phrase “have access” mean to you?

Twenty-three of 24 students responding felt that “have access” meant being allowed, having 

permission; one said that “have access” did not mean anything to them. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider changing “school library, media center, or resource center” to terms more familiar to 

students. School library seemed universally known, whereas the other two options were either 

unknown to students or they thought they were not school-based. 

Student #13 Revised/VH040207

Do you have access to a school library, media center, or resource center with books, videos, 

or other materials (including online resources) about American Indian or Alaska Native 

people? 

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 98

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically 

asking about these resources in school by adding the bolded words "In school" to the beginning of the 

item. For consistency with the proposed revision to the grade 4 student items VH040200 and VH040205, 

ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is 

likely more recognizable to students. The proposed revised item wording is consistent with other NAEP 

student questionnaires, such as the 2014 TEL Probe questionnaire.

This recommendation is the same at grades 4 and 8.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 99

Student #15 New/VH040130

Here are some sentences about your school. Fill in one oval on each line to show whether the 

sentence describes a person like you.

This is 

not like 

me.

This is a 

little like 

me.

This is a 

lot like 

me.

a. I put a lot of effort into my schoolwork.

A B C

New/ 

VH040131

b. I want to be one of the best students in my 

class. A B C

New/ 

VH040134

c. I enjoy being challenged in my classes.

A B C

New/ 

VH040137

d. I feel that I belong at school.

A B C

New/ 

VH040138

Item 15

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what question a. is asking?

Fourteen of 21 responding students said that the question was getting at whether or not they tried hard 

at school. Three commented on the structure of the item, three mentioned liking school, and one 

restated the overall question with all four parts. 

Probe 2: [Conditional probe: For question a., if the participant selects answer choice B or C, then ask 

the follow-up question]: Can you give me examples of how you put a lot of effort into your 

schoolwork?

Twenty of 23 respondents said the question was about trying hard in school such as paying attention or 

taking one’s time to do good work. One student mentioned getting help from others to succeed, and 

two gave difficult-to-classify answers:

“Okay. Monday and Tuesday, I told you, we were in Greensboro. Tuesday night, whenever we came

back, we had PTA that night. We got our report cards. Before we got report cards, I went to all of my 

teachers and I asked them to make sure if I'd missed anything, what it was, or if I could get it now, or if I 

could get it tomorrow at school or whenever. I don't want to miss out of anything, and I don't want to 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 100

be behind whenever we get back in class. Everybody's like, "I know, I know," and I'm sitting there like "I 

don't really know". If that's the case, I should have just stayed at home. I shouldn't have come to school 

[inaudible 00:22:30].”

“Well me, people try to split me up from my best friend but then we go right back to each other so like I 

feel like when I am with her I get my work done and stuff but people don't think that because we goof 

off a lot but then we get our work done.”

Probe 3: Can you tell me, in your own words, what question b. is asking?

Eleven of 24 respondents mentioned being the best or the top of their class, six mentioned getting good 

grades, four said it was about being the smartest, four indicated that it was about being the teacher’s 

pet, three said it had to do with being better than others, three mentioned good behavior as a 

component, and two spoke about being popular.

Probe 4: Can you tell me, in your own words, what question c. is asking?

Twelve out of 24 responding students indicated that enjoying a challenge had to do with harder 

material, nine said that it had to do with liking a challenge, five mentioned competition with others, and 

one said they did not know.

Probe 5: Can you tell me, in your own words, what question d. is asking?

Ten of 24 respondents said that belonging had to do with feeling like you should be there, six said it was 

about feeling safe, four mentioned it had to do with liking school, one mentioned popularity, one said it 

was about one’s future, and one indicated that they did not know. One gave a difficult-to-categorize 

answer: “Going against other people in my classroom.”

Probe 7: In question b., what does the phrase “one of the best students” mean to you?

Twelve out of 24 students responding said that “one of the best students” meant being one of the top 

students, six felt it had to do with grades, six brought up the topic of good behavior, one felt it had to do 

with being the fastest, and one indicated that they did not know what it meant.

Probe 8: In question c., what does the phrase “being challenged” mean to you? Can you give me 

examples of you “being challenged in your classes”?

When asked about being challenged, 13 of 24 responding students mentioned having harder material to 

work with, five said it had to do with putting in more effort and pushing oneself, four talked about 

competing with others, one said it had to do with getting help when needed, and one said they did not

know. 

Probe 9: In question d., what does the phrase “belong at school” mean to you? 

Eight of 23 respondents said that they felt they belonged at school because being there felt like they fit 

in, four said they felt safe, four said you should be at school, four indicated that it was because you were 

good at school, and three said to learn. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 101

Probe 10: [Conditional probe: For question d. if the participant selects answer choice B or C, then ask 

the follow-up question]: Can you give me examples of when you feel you “belong at school”?

Of the 23 respondents to the question, 11 said that they felt good at school, eight replied that it had to 

do with friends, two said that they felt they belonged at school when they were learning, one said that 

they had to be at school, and one simply replied “no.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. All respondents seemed to understand the item as written. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 102

Student #18 Revised/VH040140 

During 8th grade, have you participated in any of the following activities organized by your school? 

Fill in one oval on each line.

Yes No

a. Sports team A B Revised/VH040143

b. Drama club A B Revised/VH040146

c. Music club, orchestra, or band A B Revised/VH040147

d. Cultural dances or other American Indian or Alaska 

Native activities (for example, drum groups) A B Revised/VH040150

e. Academic club (for example, math club) A B Revised/VH040152

Item 18

Reading: One student had a hard time reading “orchestra.”

Understanding: Four students had some difficulty defining some of the item’s different components. 

Changing Answers:

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 21 responding students essentially restated the question in general terms regarding 

whether or not they had taken part in any of the after-school activities mentioned. Two students talked 

about interest rather than participation. 

Probe 3: In question a., what does the phrase “sports team” mean to you?

Twenty-three respondents gave a multitude of examples of what they considered sports teams. Nine 

mentioned basketball, seven brought up football, four mentioned track or cross-country, three talked 

about baseball, three mentioned soccer, three said volleyball, two said tennis, two said softball, one said 

golf, and one said wrestling.

Probe 4: In question b., what does the phrase “drama club” mean to you?

Eleven of 22 responding students felt that “drama club” entailed theatre or acting in plays, seven said 

they did not know, three said they did not have a drama club at their school, one said they’d never done 

it, and one thought it had to do with dance. One interpreted the question as students creating drama 

and referenced groups of girls who created drama.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 103

Probe 5: In question c., what does the phrase “music club, orchestra, or band” mean to you? 

Eighteen of 21 responding students interpreted “music club, orchestra, or band” as being related to 

playing instruments, although nine students also mentioned choir or singing. Three brought up dance as 

an example. 

Probe 6: In question d., what does the phrase “cultural dances or other American Indian or Alaska 

Native activities” mean to you? Do you think the example “drum groups” is an example of cultural 

dances? If not, can you give me examples of what “cultural dances” would be?

When asked about cultural dances, 14 of 22 respondents talked about cultural dances, and five students 

mentioned powwows. Something interesting about other responses to this question is that most of the 

examples were not school-based but rather from outside of school. Additionally there were several 

students who, when prompted, said that drum groups were a good example; but there was also a 

contingency that pointed out that drum groups were not a dance.

Probe 7: In question e., what does the phrase “academic club” mean to you?

Fourteen of 22 respondents used math as an example of an academic club, three felt that it had to do 

with smart kids, two mentioned spelling bees, two brought up science, one mentioned history, one said 

they did not know, and another student said that they did not have academic club at their school. 

Probe 8: In addition to the activities listed in questions a. through e., do you think any additional 

activities should be included? If yes, can you give me examples of additional activities?

Fourteen of 23 responding students felt that no additional activities needed to be added to the list. The 

following categories had one student each suggesting that it be added: AI/AN-specific organizations, 

photography, choir, tutoring club, chess, science, art, and gardening.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider putting word “school” in front of each activity to ensure that students remember that 

the question is contextually based on activities offered by the school—if indeed that is the key 

component being measured.

ETS Recommendation (05/02/2014)

Keep this matrix item, but drop sub-item “b” VH040146.

Rationale: ETS recommends dropping this item (VH040146) to reduce student burden. This sub-item 

caused much more confusion for respondents than any of the other sub-items within this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 104
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 105

Student #20 New/VH040157

How much do you agree with each of the following statements? Fill in one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. If I put in enough effort I will succeed 

in school. A B C D New/VH040158

b. If I put in enough effort in school I 

will get into college. A B C D New/VH040160

c. If I put in enough effort in school I 

will get a good job. A B C D New/VH040161

Item 20

Reading: One respondent skipped a word when reading the item. 

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 21 respondents felt that the question was asking about trying hard to do well in school. One 

student restated the question, one thought it was a repeat of an earlier question, and one said they did 

not know. 

Probe 3: What does the phrase “put in enough effort” mean to you?

All 23 respondents to this probe said that “put in enough effort” meant trying hard. 

Probe 4: In question a., what does the phrase “succeed in school” mean to you? Can you give me 

examples of what “succeed in school” would look like?

Seventeen of the 23 students responding said that “succeed in school” meant passing or graduating. 

Eight felt it had to do with getting good grades, three said it was related to being the best, and two 

mentioned good behavior.

Probe 5: In question b., what does the phrase “get into college” mean to you? 

Twenty of 23 respondents felt that “get into college” meant going to college. One mentioned 

scholarships, two had difficult-to-categorize answers: “That you did succeed in school and you did what 

you are doing and you did it really good” and “It means, I don't know the word for it ... it gets harder, it-“
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 106

Probe 6: In question c., what does the phrase “get a good job” mean to you? Can you give me 

examples of what “a good job” would be?

Fourteen for 23 respondents mentioned that “get a good job” meant having a job of one’s choice. The 

options listed at least once by students were: doctor, lawyer, teacher, astronomer, sports player, desk 

job, restaurant owner, physicist, engineer business owner, and others. Five students indicated that a 

good job meant having enough money, and two said that it was not working at McDonalds.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 107

Student #21 New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student tardiness A B C D New/VH040217

b. Student health problems A B C D New/VH040219

c. Student misbehavior in class A B C D New/VH040221

d. Physical conflicts among students A B C D New/VH040223

e. Bullying A B C D New/VH040226

f. Low student motivation A B C D New/VH040227

g. Low teacher expectations A B C D New/VH040231

Item 21

Reading: One student misread “moderate.”

Understanding: A few students had some trouble with some of the vocabulary used in the item, one had 

trouble understanding “extent,” another had a problem with “physical conflict,” and some grappled with 

adequately understanding “low student motivation” and “low teacher expectations.”

Changing Answers: One student changed their answer in row A from C to B after the word “extent” was 

explained to them. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eighteen of 20 students responding felt the question was asking them about problems in their school. 

One person felt the question was asking if any of the problems were personally experienced; one 

respondent’s answer was dropped due to losing audio.

Probe 3: What does the phrase “student tardiness” mean to you?

All 22 respondents said that “student tardiness” meant being late—there was a combination of 

respondents who thought this was just for class or for school overall. Additionally three students 

thought that it meant being absent from school entirely. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 108

Probe 4: In question a., what does the phrase “student health problems” mean to you?

Of the 22 respondents, 10 students felt that “student health problems” meant being sick or ill. Five 

mentioned asthma, two brought up diabetes, one mentioned obesity, one brought up cancer, one said 

broken limb, one student mentioned “disease” in general, and one talked about nutrition.

Probe 5: In question c., what does the phrase “student misbehavior in class” mean to you? 

Eleven students out of 22 respondents felt that the probe about misbehavior could be characterized by 

not paying attention or listening, seven said it had to do with being disruptive in class, three said it was 

about being rude, one said talking in class, and one mentioned students acting up.

Probe 6: In question d., what does the phrase ““physical conflicts among students” mean to you? 

Of the 22 respondents, 17 respondents felt that “physical conflicts” consisted of fights. Interestingly, 

three students interpreted the question as having to do with physical activity such as sports and 

exercise, and two said it meant bothering others.

Probe 7: In question d., what does the phrase "bullying" mean to you?

Fifteen of 23 students responding said that bullying had to do with picking on others (often those 

younger or smaller), two mentioned teasing, two said it was being rude or mean, two mentioned talking 

badly about someone, one student felt that it involved physical conflicts, and another student felt it had 

to do with problem students. 

Probe 8: In question f., what does the phrase “low student motivation” mean to you? 

Of the 22 respondents, 13 indicated that “low student motivation” had to do with not trying in school, 

three said it was students who did not want to be at school, two felt that they were students who 

needed help, one indicated that they were kids with problems either at home or at school, one student 

said they did not know, and two gave hard-to-characterize answers in the following: 

“Sad, guilty, or it's nothing.”

“No, I think that's a large extent. Our teachers are always, even if a student doesn't think they can do it, 

our teacher helps us, "You can do it. You can do it. Don't give up." I'm also in student council, so if 

there's ever anybody that's feeling like they can't go down, because I'm the president, they will come 

and talk to me. They'll ask for my advice. I also feel like I'm a guidance counselor. I guess they feel more 

comfortable talking to me about it than they would an adult. I guess I would understand more. Low 

student motivation, no, because we're always trying to help in any way that we can.”

Probe 9: In question g., what does the phrase “low teacher expectations” mean to you? 

Students struggled a bit with “low teacher expectations.” Ten of the 22 respondents said that it was 

teachers who expect little and do not push their students. Three respondents said it was teachers who 

were not doing their jobs, two referenced mean teachers, one said it was teachers who did not 

discipline, and four students said they did not know. There were two difficult-to-categorize answers:

“When the teacher would tell you to do something you wouldn't do it and you would like tell her no or 

yeah...”

“Like teachers they need to go on the right path, trying to help them out, do everything good to them. 

Just try to help them out on math, reading, science if a student won't know and a teacher will tell him 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 109

how to do it, give them the rules and everything. Like Rhonda Fry she helped us out on our rules 

learning from mathematics.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider rephrasing “physical conflicts among students” to “physical fights among students” and 

consider easier phrasing for “low student motivation” and “low teacher expectations.”

ETS Recommendation (05/02/2014)

Revise this matrix item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students 

as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of 

the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; 

therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

Drop sub-item “g” VH040231.

Rationale: ETS recommends dropping this item (VH040231) to reduce student burden. This sub-item 

caused much more confusion for respondents than any of the other sub-items within this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 110

Survey Instrument: Teacher Grade 4/Sequence I

Item 1

Reading: NA

Understanding: Two respondents had a difficult time determining what “culturally specific instructional 

practices for AI/AN students” meant. There was some question as to whether or not that meant 

teaching about the culture or teaching in a culturally appropriate manner. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fifteen teachers (out of 23 respondents) said that they felt the question was asking them to reflect on 

whether they use cultural topics when teaching that are applicable to AI/AN students. Eight respondents 

thought it might have to do with culturally appropriate practices of engaging and meeting the needs of 

AI/AN students. 

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me 

examples of what a reasonable time frame would be?

Twenty-one of the 24 respondents felt that the time frame of two years was appropriate. Of the three 

who did not, reasons had to do with either being very new to teaching, or (for long-tenured teachers) 

that five years might be a better measure. 

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the 

last 24 months? If not, can you give me examples of what “the last two years” would be?

Of the 23 respondents, 20 teachers thought of “the last two years” as the last two school years, while 

three thought of it as the last 24 months. 

Teacher #1 New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional 

practices for American Indian or Alaska Native students in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 111

Probe 5: What does the phrase “culturally specific instructional practices for American Indian or 

Alaska Native students” mean to you?

Twenty-four teachers responded to this item. Of these, 12 felt that “culturally specific instructional 

practices” had to do with how one might make their teaching more culturally appropriate or responsive 

with regard to being aware of and honoring cultural differences among their AI/AN students (i.e., 

adapting their teaching methods to meet the cultural needs of their students). Eleven mentioned 

teaching AI/AN topics, three mentioned working to incorporate AI/AN vocabulary into their teaching, 

and two said they made an effort to invite outside speakers to present on different AI/AN-specific 

topics. 

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me an example for culturally specific instructional practices for American or 

Alaska Native students that you have implemented in your classroom?

Seven teachers (out of 18 respondents) reported making lesson plans more interactive and hands-on so 

as to better engage their AI/AN students, seven said they chose topics of interest for AI/AN students, 

and four mentioned being mindful of cultural practices (e.g., not touching children, limited eye contact). 

One mentioned having little leeway in the school-prescribed curriculum to offer other topics, and one 

teacher again mentioned bringing in speakers to share different topics with the class. 

Solutions to Improve Item

Respondent: NA

Interviewer: The number and type of probes for this question were reasonable. The question about 

participants’ answers should come before asking them to restate the question as that seems to be the 

natural flow of the interaction.

Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems like the question could be clarified to better define for teachers if they are reflecting 

on topics or instructional practice as the difference between the two. The rather in-depth analysis many 

teachers seemed to give the question while trying to determine what it was actually asking suggests that

there is some lacking consensus on the actual subject of the question. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 112

Item 2

Reading: One participant misread “culturally” as “cultural.”

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-one of 22 respondents interpreted the question appropriately. One answer was a non-sequitur 

response.

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me 

examples of what a reasonable time frame would be?

Fifteen teachers (of the 19 respondents) said that they did think that two years was a reasonable time 

frame, three said no (again, usually calling for either less or more time in the form of one or five years), 

and one respondent said it depended on what the question was asking. 

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the 

last 24 months? If not, can you give me examples of what “the last two years” would be?

Of the 17 who responded, 11 teachers said they thought of the last two years in school years, six said 

they were thinking of 24 consecutive months. 

Probe 5: What does the phrase “professional or community-based development programs” mean to 

you?

Fourteen teachers out of 21 respondents felt that professional programs would have to do with 

receiving training by professionals or the district on specific topics or in other areas to help them 

maintain their teaching certification. Six felt that community programs would entail some kind of 

training given specifically by the community to address community needs, and six additional 

respondents said that community programs might be programs or workshops help by tribes. Two 

mentioned that maybe this would have to do with culturally specific training. 

Teacher #2 Revised/VH040275

During the last two years, how many times have you attended professional or communitybased development programs (such as in-service classes and workshops, including online 

classes) aimed at developing culturally specific instructional practices for American Indian or 

Alaska Native students?

A. Never Skip Pattern.

B. 1 or 2 times

C. 3 or 4 times

D. 5 or more times
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 113

Probe 6: What does the phrase “in-service classes” mean to you?

The answers varied greatly in specifics for this question, but all teachers (24 respondents in total) were 

aware of what in-service classes were and what they meant for them and their certification. 

Descriptions used to answer this question included that in-service classes were required, offered by the 

school, offered by the district, a workshop, focused on academic topics, for credit, served as knowledge 

enrichment, done in-school when students were not in school, were a class with other teachers, were 

part of their job, a professional development day, and required for licensure. 

Probe 7: When you answered this question, did you consider activities that were not in-service classes 

and workshops or online classes? If yes, can you give me examples of what these other activities are?

Twelve teachers (out of 22 respondents) said “yes” and nine said “no.” Examples of other activities 

mostly related to school and teacher training; six had to do with culturally specific learning related to 

AI/AN students.

Probe 8a: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: a. What development programs have you attended? 

Of the 14 respondents, six teachers said that they had attended development programs sponsored by 

the school, three mentioned conferences, three mentioned presentations [Note: it was difficult to 

determine if these were through the school, community, conference, or other], one mentioned a local 

historical center and one said s/he still took classes. 

Probe 8b: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: b. Were any of the programs you attended taught as online classes?

Eight teachers (out of 13 respondents) said “no” to online classes, three said “yes,” and three indicated 

that they had done some online activities though these ranged from looking things up on the Internet to 

attending a webinar. 

Probe 8c: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

questions]: c. Have you attended development programs that are not through in-service classes?

Seven of 13 respondents said “yes,” three said “no,” and three indicated that they possibly had. 

Examples were a conference, a class, and learning through peers. 

Solutions to Improve Item

Respondent: NA

Interviewer: Some of the probes have multiple question components imbedded into one probe, making 

the answers somewhat messy. 

Initial KAI Recommendation (Keep/Change/Discard)

Change: The question is generally fine as is, but may be helpful to highlight the fact that the question is 

specifically geared to learning more about AI/AN students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 114

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 115

Item 3

Reading: NA

Understanding: There was one question about whether or not the time frame for the question was still 

two years. 

Changing Answers: One teacher changed their response from C to D when considering their response.

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 17 respondents, 15 teachers appropriately restated the Item 3 question as having to do with 

whether or not they were using gained knowledge in their classrooms. Two gave difficult-to-categorize 

answers with the following: 

“Have you taught a lesson that was geared on a topic maybe that your community had a need for in 

your classroom?”

“As a part of that class that I took when I first came here, I had to teach a lesson in class and I also had to 

come back to my specific class and teach a lesson.”

Probe 3: What does the phrase “lessons learned” from these programs mean to you?

Twelve of the 16 respondents felt that “lessons learned” essentially meant that they had gained 

knowledge that they could put into practice. Three felt the question specifically related to new 

knowledge gained. One gave a difficult-to-qualify answer:

“Hopefully there’s a positive outcome and that you want your lesson to be... That whoever is attending 

get something that’s an eye-opener for them for their community, help them develop whether it’s their 

students or the teachers. Make them more aware. I think awareness maybe overall.”

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of how you have implemented lessons learned from these 

professional or community-based development programs in your classroom? 

Answers to this probe varied in terms of topics noted as implementing lessons learned and new 

knowledge. Of the 17 respondents, four mentioned that particular teaching practices they had learned 

Teacher #3 New/VH040281

To what extent have you implemented lessons learned from these professional or community-based 

development programs in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 116

and made use of in their classrooms. Other responses had to do with using AI/AN stories (three), or 

cultural practices into their teaching (two). The following responses were only mentioned by one 

teacher for each topic: Singapore math, cultural awareness, common core, history, AI/AN language, 

social justice, 3D, and using projects. 

Solutions to Improve Item

Respondent: NA

Interviewer: Skip pattern not obvious at first.

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 117

Item 4

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 16 respondents, 11 teachers felt that the question was trying to get at who sponsored or paid for 

the programs in which they had taken part over the last two years. Three felt the question had to do 

with where they were trained. Two gave difficult-to-classify answers:

“I interpret it what avenues are you pursuing to get professional development. Is it all just coming from 

your school, is it all just coming from the state, or are you personally going out and looking for other 

opportunities to get that development.”

“Well, I kind of think the point of it is to find out how effective...If you have a program and it was from a 

certain place, how effective was it. Or what are the resources that are out there that you can take 

advantage of, like are they coming from the state, are they coming from new district, from tribal, from 

Indian education, or are they coming from colleges or universities? So that’s kind of what the question 

means to me.”

Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: What is this funding source?

When asked about the funding source, the eight responses for this probe pointed to the BIE (two), the 

tribe (two), Teach for America (two), STEP (one), PACER (one), NSF (one), and one “I don’t know.”

Solutions to Improve Item

Respondent: NA

Teacher #4 Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the 

last two years? Fill in all ovals that apply.

A. State

B. District

C. Tribal education department

D. Indian education professional associations

E. College or university

F. Other (please specify): ________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 118

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 119

Teacher #5 Trend/VE012668

To what extent do you use the following to assess student progress? Fill in one oval on each line.

Not 

at all

Small 

extent

Moderate 

extent

Large 

extent

a. State assessments A B C D Trend/VE012670

b. District assessments A B C D Trend/VE012672

c. Assessments developed by American 

Indian or Alaska Native organizations A B C D Trend/VE012673

d. Tests supplied by textbook publishers 

(for example, end of unit or chapter 

tests)

A B C D Trend/VE012674

e. Teacher-made tests or quizzes A B C D Trend/VE012675

f. Performance-based assessments A B C D Trend/VE012676

g. Group projects A B C D Trend/VE012678

h. Oral responses of students during class 

discussions A B C D Trend/VE012681

i. Assessments to evaluate English 

language proficiency A B C D New/VH040300

Item 5

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eight teachers (out of 21 respondents) thought the question was asking them what assessments they 

used, seven thought the question referred to how they assess their students, two thought the question 

was asking how often they assessed students, and one felt the question had to do with how well s/he

assessed students. Four respondents gave difficult-to-qualify answers [Note: one is too long to include in 

this report]:
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 120

“Well, because one of the things...Going back to the gentleman who gave me the same, basically 

American Indian students learned differently. If you're going to assess their performance, I would argue 

it's not just American Indian community that I know may sound differently. If you are only using things 

like state assessments and districts assessment, you're really not getting at the whole picture and I 

wholeheartedly agree with that especially since a lot of those things I see have definite [inaudible 

00:21:40].”

“I only think this because part B surprised me so much because I haven't actually seen any assessments 

developed by American Indian or Alaska Native organizations, so in my mind it is just to cue my 

awareness, are you using assessments that are going to be fair and specific and incorporating the culture 

into the assessment? So often that can be a determining factor in success.”

“I think it's demonstrating more than asking that there's a lack of input from...In assessments from 

American Indian and Alaska Native organizations. It could be how important I would state the 

assessments in my own teaching. Yeah, I mean look at that stuff. Usually the teacher made it tough, the 

day to day quizzes and I have districts to place our tests for things like Math. I supposed that helps guide 

us to see clearly what the students need more help in kind of thing with their stem of understanding.”

Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency” 

mean to you?

This question seemed to confuse many respondents: none of the 23 respondents reported using an 

assessment to evaluate English language proficiency. All 23 teachers who responded had a notion of 

what English language proficiency testing consisted of, but it seemed from their answers that this was a 

confusing question. It is hard to say whether or not this is because most seemed to associate language 

proficiency assessment specifically with students of foreign descent. 

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask 

the follow-up question]: Can you give me examples of how you have used “assessments to evaluate 

English language proficiency” to assess student progress?

Answers to this probe varied greatly, suggesting some validity issues with teacher response choices to 

the item. Respondents lacked the ability to explain what they were doing in their own classrooms to 

assess English language proficiency. Of the 15 respondents, four teachers used school-based tests as 

their answer (i.e., tests given to students across the board at the school level), three mentioned that 

they might better understand a student’s English proficiency through the use of math problems, one 

used team meetings to go over test results (bit did not specify a type of test), one said s/he simply 

listened, one said reports, one said the language proficiency assessment, and one said s/he did not 

know. There were three answers difficult to categorize:

“Well, the assessments to evaluate English language, they're only used with students who speak two 

languages. [Inaudible 00:18:39] students when English isn't necessarily their first language, yeah.”

“I know at first for me, I was like, I don't teach English, I teach Ojibwe. So I have to try and teach the 

structure and the makeup of the language and how it's set up different than English. But I don't test 

English so for me, that's opposite of what I do. Then after I thought, when you told me to focus in on it 

then I thought oh, maybe it's for first speakers who are in maybe an ELL program.”
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 121

“Maybe we’re practicing for our NMSBA tests that are coming up. From the very beginning, kids will 

come in and I would ask them a question, they would say no, yes or this and that. Then we have to tell 

them, you need to answer me in complete sentence, complete thought. We have to practice like this so 

by the time the tests come around, hopefully, they’ll be able to answer in complete sentences and not 

just by one or two words.”

Probe 5: In question c., what specific organizations did you consider when answering this question?

Of the 21 respondents, nine teachers said they did not know of any specific organizations, three said 

that they did not consider any specific organization, two teachers mentioned the BIA, two mentioned 

Indian Education departments, and two said they thought of tribal organizations. One reported using the 

local text, one thought of AI/AN organizations, and one specifically mentioned NIEA. 

Probe 6: When answering this question did you consider NIES information about student progress? If 

yes, in which question specifically did you consider NIES information about student progress?

All 21 respondents said that they had not considered the NIES when answering the question, with many 

noting that they had never heard of the NIES. 

Solutions to Improve Item

Respondent: NA

Interviewer: Is this question meant to be specific to AI/AN students? If so, that should probably be 

explicit in the question asked. 

Initial KAI Recommendation (Keep/Change/Discard)

Change: If this item is geared toward AI/AN students, consider highlighting the specification. Also, 

determine whether or not the item is referring specifically to assessments used by teachers or by the 

school. And last but not least, is there a time frame to be associated with this question?

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking 

about AI/AN students.

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 122

Item 6

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of the 20 respondents seemed to understand the meaning of the question. One response 

recorded consisted of “No, I didn’t.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

Teacher #6 New/VH043923

Do you teach reading/language arts to grade 4 students?

A. Yes  Go to Question 7

B. No  Go to Question 8
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 123

Teacher #7 Revised/VH040381

How often do you integrate materials about the following topics into your reading/language arts

lessons? Fill in one oval on each line.

Never

At least 

once a 

year

At least 

once a 

month

At least 

once a 

week

Every day 

or almost 

every day

a. American Indian or Alaska 

Native culture or history A B C D E

Revised/VH0

40385

b. Current issues affecting 

American Indian or Alaska 

Native people or 

communities

A B C D E

Revised/VH0

40386

Item 7

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answer for question A to C from D, and for question 

B from D to C. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 19 respondents interpreted the item as it was intended. 

Probe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history” 

mean to you?

Twenty out of 20 respondents either worked to explain or gave examples of the meaning of “AI/AN 

culture or history.” Some respondents felt the question was straightforward, thus leading to redundant 

answers. It would be better in the future to simply ask for examples to determine understanding of the 

concept.

Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska 

Native people or communities” mean to you?

Eighteen teachers (out of 18 respondents) gave answers indicating that they did indeed have an 

understanding of the phrase “current issues affecting AI/ANs.” Examples given in the course of 

answering included: treaties, land rights, sovereignty, historical trauma, economies, casinos, lifestyle, 

health, and poverty. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 124

Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., 

or to both, then ask the follow-up question]: Can you give me examples of how you have integrated 

materials about the topics into your reading/language arts lessons?

Five of the 16 respondents said they used state history to integrate topics into their lessons. Five said 

they used stories, traditional and otherwise. Three mentioned crafts from creating regalia to basket 

weaving, one mentioned language, and one mentioned dance. Issues of how much leeway teachers had 

with their state or district curriculum was again brought up by one respondent.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 125

Item 8

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty teachers (out of 20 respondents) seemed to easily understand what the item was asking. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

Teacher #8 New/VH043941

Do you teach mathematics to grade 4 students?

A. Yes  Go to Question 9

B. No  Go to Question 10
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 126

Teacher #9 Revised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in 

one oval on each line.

Never

At least 

once a 

year

At least 

once a 

month

At least 

once a 

week

Every day 

or almost 

every day

a. American Indian or Alaska 

Native culture or history A B C D E

Revised/VH

040390

b. Current issues affecting 

American Indian or Alaska 

Native people or communities

A B C D E

Revised/VH

040392

Item 9

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 18 respondents, all 18 teachers understood the item.

Probe 3: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., 

or to both, then ask the follow-up question]: Can you give me examples of how you have integrated 

materials about the topics into your mathematics lessons?

Only 11 respondents provided answers for Probe 3. These ranged from teachers using questions about 

ancient or current populations, measurements on how to build things (such as teepees or drums), 

counting in an AI/AN language, culturally specific word problems, data from current issues such as water 

rights, and mapping and scaling using math.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 127

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 128

Teacher #10 New/VH043953

How much do you agree with each of the following statements about the materials available in your 

school library, media center, or resource center? Fill in one oval on each line.

Strongly 

disagree Disagree Agree

Strongly 

agree

a. The number of books and materials 

available for 4th grade students is 

sufficient.

A B C D New/VH043957

b. The quality of the books and 

materials available for 4th grade 

students is satisfactory.

A B C D New/VH043959

c. The number of books and materials 

specific to American Indian and 

Alaska Native culture available for 

4th grade students is sufficient.

A B C D New/VH043964

d. The quality of the books and 

materials specific to American 

Indian and Alaska Native culture 

available for 4th grade students is 

satisfactory.

A B C D New/VH043966

Item 10

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen teachers (out of 21 respondents) accurately explained that the question had to do with 

whether or not teachers thought there were enough satisfactory books for 4th graders. Only 10 

specifically mentioned AI/AN-specific books. 

Probe 3: In questions a. and c., what does the phrase “is sufficient” mean to you? 

All 21 respondents mentioned amounts of books in their response to what “sufficient” meant to them. 

One felt that the question had to do with quality, one said access, and one felt that it had to do with 

averages (i.e., sufficiency serving as another word for average, as does satisfactory). 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 129

Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you? 

All 24 respondents provided an answer to the probe. Nine teachers felt that the question about what 

“satisfactory” means talked about the idea of averages. Five felt the question was talking about quality, 

four mentioned if the books were contemporary, four said if books were appropriate for the reading 

age, three mentioned the condition of the books, two said it meant if there were enough books, and one 

felt it had to do with whether or not books were relevant. 

Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean 

to you?

Fourteen of the 24 respondents said that the quality of books and materials had to do with content (i.e., 

whether it was appropriate), seven mentioned the condition of the books, seven mentioned whether or 

not the books were contemporary, and four felt it had to do with whether or not the books were for the 

appropriate reading age. 

Probe 6: In questions c. and d., what does the phrase “specific to American Indian and Alaska Native 

cultures” mean to you?

Of the 23 respondents, 11 teachers said that they felt that materials specific to AI/AN cultures should be 

historically accurate or accurate in general. Ten mentioned the word “specific” either in terms of 

explaining what they thought of the phrase posed to them, or sometimes having to do with the notion 

that materials should be about specific tribes or groups. Three teachers mentioned that materials should 

be authored by AI/AN writers. 

Solutions to Improve Item

Respondent: There should be an “I don’t know” category. 

Interviewer: The way the probes were phrased generated quite a deal of reflection on the part of 

teachers, but it is notable that the vast majority of teachers focused only on books as “materials.”

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 130

Teacher #11 New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student absenteeism A B C D New/VH040413

b. Student tardiness A B C D New/VH040414

c. Student health problems A B C D New/VH040416

d. Student misbehavior in class A B C D New/VH040422

e. Physical conflicts among students A B C D New/VH040424

f. Bullying A B C D New/VH040425

g. Low student aspirations A B C D New/VH040427

h. Low teacher expectations A B C D New/VH040428

i. Low family involvement A B C D New/VH040430

Item 11

Reading: NA

Understanding: One teacher wanted to know if the timeline for this question was still two years. 

Another seemed to not understand the question but answered all subsequent probes appropriately. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty-two teachers (out of 22 respondents) seemed to appropriately understand what the question 

was trying to invoke. 

Probe 3: In question a., what does the phrase “student absenteeism” mean to you? 

Nineteen of the 24 respondents felt “absenteeism” meant students not at school; seven felt it had to do 

with the rate at which students were missing school. 

Probe 4: In question b., what does the phrase “student tardiness” mean to you? 

Of the 24 respondents, 21 teachers said that “tardiness” meant being late; five teachers mentioned the 

rate at which students were late. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 131

Probe 5: In question c., what does the phrase “student health problems” mean to you? 

Thirteen of the 24 teachers who responded to the probe felt that the question regarding student health 

problems involved students being sick. Five mentioned asthma, four talked about diabetes, three 

brought up allergies, two mentioned diet and nutrition, two noted disabilities (including learning 

disabilities), two mentioned physical problems, two noted mental problems, and one mentioned dental 

and one vision problems. Other issues mentioned were epilepsy and ADD. 

Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you? 

Eleven teachers (out of 24 respondents) mentioned disruptive or distracting behavior as constituting 

student misbehavior. Nine brought up students not following the rules, and five mentioned students 

being disrespectful or talking in class. 

Probe 7: In question e., what does the phrase “physical conflicts among students” mean to you? 

Of the 24 respondents, 18 teachers felt that physical conflicts among students constituted fighting. Six 

teachers mentioned students putting their hands on each other, five mentioned hitting, three brought 

up kicking, and two said pushing.

Probe 8: In question f., what does the phrase “bullying” mean to you? 

Eighteen of the 24 respondents said that bullying constituted repeated behavior such as picking on 

others. Ten teachers mentioned verbal bullying such as name calling, while cyber bullying was also 

brought up by three teachers. Nine teachers felt that bullying could also be physical. 

Probe 9: In question g., what does the phrase “low student aspirations” mean to you? 

Fourteen teachers (out of 23 respondents) mentioned motivation of students as an issue in low student 

aspirations. Nine brought up students not feeling like they had much to look forward to with regard to 

their futures, and seven mentioned a lack of self-esteem. 

Probe 10: In question h., what does the phrase “low teacher expectations” mean to you? 

All 24 teachers responded to the probe. Of these, 13 teachers felt that low teacher expectations were 

akin to teachers not pushing their students, and 12 mentioned teachers with low standards for their 

students’ achievements. 

Probe 11: In question i., what does the phrase “low family involvement” mean to you?

All 24 teacher responses were consistent on ideas of what might constitute family involvement, but not 

all teachers answered the question in terms of what they thought low family involvement entailed. 

Different aspects discussed included whether or not parents volunteered at the school, sent notes to the 

teacher, were responsive to teacher communication, attended parent-teacher conferences, attended 

school events, etc. 

Solutions to Improve Item

Respondent: One respondent asked about whether or not they should be reflecting on the class level, 

grade level, or school level. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 132

Interviewer: Is this item meant to elicit reflection on the entire school (which may be K-12 sometimes) 

or just for the 4th grade?

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Be more specific about the level on which teachers should be focus when answering the item. 

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students 

as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of 

the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; 

therefore the items should reference "AI/AN" when necessary in order to capture the most useful data. 

This recommendation is the same at grades 4 and 8.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 133

Survey Instrument: Teacher Grade 8/Sequence I

Item 1

Reading: NA

Understanding: Two respondents had trouble with the term “culturally specific instructional practices” 

and what that meant. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Validity of this item is questionable: 16 out of 23 respondents felt that “implementing culturally specific 

instructional practices” had to do with covering topics related to AI/ANs, four felt like it had to do with 

actual teaching practices, one said that s/he did not know, and two expressed that they were not sure 

what the term referred to. As one of the unsure noted: 

“I think the thing that catches me about it is what they might mean by an instructional practice because 

I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s 

why I would go with a small extent. I actually started a class called Engineering from a Native 

Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best 

practices for all students.”

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me 

examples of what a reasonable time frame would be?

Of the 23 respondents, 19 expressed that they thought two years was a good time frame. One felt that 

four to five years was a better time frame, two said that one year was a better time frame, and one felt 

that six months was a more appropriate time frame. One response was hard to characterize: 

“I have done it throughout my career and I have taught many different grade levels. I think that we 

always have Native students in classes. We also have Hispanic and Non-Natives in other schools I have 

taught in and I think that everybody needs to know how rich the Native culture is. Sometimes we are 

Teacher #1 New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional 

practices for American Indian or Alaska Native students in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 134

marginalized to Native American Day. The Hispanics are Cinco de Mayo. That just makes me angry. I 

think that we need to be integrated and respected for the rich history that we have got.”

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the 

last 24 months? If not, can you give me examples of what “the last two years” would be?

Twenty-one of 22 respondents felt that “the last two years” referred to school years. One said 24 

months. 

Probe 5: What does the phrase “culturally specific instructional practices for American Indian or 

Alaska Native students” mean to you?

There was quite a bit of variance in this question; 15 of 23 respondents felt that “culturally specific 

instructional practices” referred to incorporating cultural material into lessons, six felt it had to do with 

culturally sensitive teaching practices, three said language, three said they were unsure what the phrase 

meant, and one gave a difficult-to-characterize answer:

“To me, the Lakota culture is an oral culture, and there's a lot more listening and talking in my room, in 

part because I'm a big talker, but also that's just traditionally how Native parents would have educated 

their children. I think it becomes vague because there are so many, I'm sure this is exactly what you 

want on this recording, I think it becomes vague because there are so many generations of Native 

parents that have led to the students that are in my classroom today that were not raised in the 

traditional household, traditionally by the Native parents.”

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me an example for culturally specific instructional practices for American or 

Alaska Native students that you have implemented in your classroom?

There were 19 respondents for this question, and the answers varied: six respondents indicated that 

they implemented culturally specific practices through language, four mentioned science and nature, 

three said arts and crafts, two mentioned Native American History Month, one reported doing so 

through a school bulletin, one mentioned ELL practices, one said history, one reported not 

implementing practices, one described being mindful of eye contact, one said honoring shyness, and 

one described pairing students. 

Solutions to Improve Item

Respondent: One respondent suggested that an example of what is meant by “culturally specific 

instructional practices” would be helpful. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems that this question is an engaging one; however, as one of the respondents suggested, 

it would be very helpful to be specific about what is meant by “culturally specific instructional 

practices,” whether that refers to teaching methods or incorporating culturally appropriate material. 

And if both are intended, then perhaps this could be reconfigured as a matrix question.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 135

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 136

Item 2

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 21 respondents, 20 adequately noted that the question was asking about attending other 

training opportunities. Only 12 specifically mentioned AI/ANs in their restatement of the answer, one 

respondent thought the question had to do with implementing what was learned from the trainings, 

and one thought it had to do with language. 

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me 

examples of what a reasonable time frame would be?

Fifteen of 19 respondents said that the last two years was a reasonable amount of time. Three said no, 

that it should be a longer time frame (again, five years was a common suggestion), and one said it 

should be less time. 

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the 

last 24 months? If not, can you give me examples of what “the last two years” would be?

Of 17 respondents, 10 said that they thought of the last two school years as school years, and five said 

they thought of it as 24 months. One said “year round,” and another said they were thinking of this year. 

Probe 5: What does the phrase “professional or community-based development programs” mean to 

you?

Responses from 20 respondents were somewhat varied to this probe. Ten felt that the professional 

development programs were provided by the school or district. Answers about community-based 

programs ranged with eight respondents essentially saying that they thought this meant that the 

community was involved—though it varied by respondent as to whether they thought it was hosted by 

Teacher #2 Revised/VH040275

During the last two years, how many times have you attended professional or communitybased development programs (such as in-service classes and workshops, including online 

classes) aimed at developing culturally specific instructional practices for American Indian or 

Alaska Native students?

A. Never Skip Pattern.

B. 1 or 2 times

C. 3 or 4 times

D. 5 or more times
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 137

the community or just involved the community. Five specifically mentioned AI/AN communities, four felt 

that the programs were offered by an outside source. Other comments were that it had to do with 

cultural integration or that it was something you could get credit for, and one person mentioned 

conferences. 

Probe 6: What does the phrase “in-service classes” mean to you?

Ten of 20 respondents said that in-service classes were offered by the district, seven said they were held 

in school, three said they involved professional development, and two respondents seemed to think that 

it had to do with shadowing someone else (like a student teacher). 

Probe 7: When you answered this question, did you consider activities that were not in-service classes 

and workshops or online classes? If yes, can you give me examples of what these other activities are?

Eight of 22 respondents said “no,” and seven said that other activities they were thinking of were 

community events. Two mentioned assemblies, one said they went to see a tribal council meeting, one 

mentioned conferences, one said they were thinking of a partnership with a local organization, one said 

they taught online classes and learned from that experience, and one said they did not know.

Probe 8a: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: a. What development programs have you attended? 

Five of 13 teachers responding mentioned outside speakers when asked about other development 

programs they had attended. Five mentioned AI/AN groups, four noted cultural training, three brought 

up nearby universities as resources, one mentioned community organizations, one said Teach for 

America, one said GLAD (guided language acquisition design), and another said they did not remember. 

Probe 8b: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: b. Were any of the programs you attended taught as online classes?

Eleven of 13 respondents said that they had not attended any online classes. Two said that they had but 

that they were not Native-specific. 

Probe 8c: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

questions]: c. Have you attended development programs that are not through in-service classes?

Seven of 14 responding teachers said that they had attended development programs that were not 

through in-service classes and six said they had not. One said they did not remember. Of those who said 

yes, conferences, community events, presentations, and workshops were mentioned.

Solutions to Improve Item

Respondent: One respondent suggested a larger time frame than two years for this item. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: We recommend highlighting American Indian/Alaska Native students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 138

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 139

Item 3

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answer from B to D in response to Probe 3. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eleven of 13 respondents to this question felt that the question had to do with whether or not they had 

implemented lessons learned. Two gave hard-to-classify answers in the following: 

“I think this is trying to ask that, I’m saying that the professional and community-based development 

programs in my classroom has happened according to the language that I teach and the history that I 

bring. However, somebody has to teach some others like me so that all of us teachers can do what I do.”

“I think the thing that catches me about it is what they might mean by an instructional practice because 

I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s 

why I would go with a small extent. I actually started a class called Engineering from a Native 

Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best 

practices for all students.”

Probe 3: What does the phrase “lessons learned” from these programs mean to you?

Eight out of 15 respondents felt the question had to do with what implementation they had done of 

lessons learned. Seven indicated that it had to do with what they had taken away from trainings. Two 

responses were hard to categorize: 

“I’m 65 years old. I think some of these lessons learned, I think I need to know some something about 

the Southwest, you know, for me to make inference of what they do and make a direct connection to 

maybe the Ojibwe way of doing that also. Let’s say, they do maple sugar right now here in 

Minnesota...During this time, what are they doing in New Mexico? What are they doing in the Pacific 

Northwest? What do they do up in Six Nations Iroquois Confederacy area? What are they doing over 

here about the Blue Corn Festival? What’s going on with the drum? These kind of things. People do not 

know anything about that and that’s what I think is real. We have a living, truly beautiful, living culture, 

Teacher #3 New/VH040281

To what extent have you implemented lessons learned from these professional or community-based 

development programs in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 140

but it’s all in books and these kids are going to read and make their worksheets and they’re going to be 

done and get graded, move on.”

“Well, I was hoping that'd be the in-service lessons that we learned. Well, as an educator, it's just 

something that you learned in the classroom situation or something like that, not necessarily learned on 

the street (which could be ‘lessons learned,’ but I meant it literally).”

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of how you have implemented lessons learned from these 

professional or community-based development programs in your classroom?

Six of the 14 respondents gave examples of including specific topics into their teaching (not all of these 

answers were specific to AI/ANs), six responded that they had made sure to be more culturally sensitive 

in their teaching methods, two brought up professional development in general, and one mentioned the 

GLAD program. 

Solutions to Improve Item

Respondent: One respondent noted that they found the item too Euro-centric. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Is this question meant to invoke whether or not teachers implement lessons learned in general or 

whether they are implementing lessons learned specifically targeting AI/AN students? If the latter, it 

should be noted in the question and highlighted. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 141

Item 4

Reading: NA

Understanding: One subject indicated that they were a little confused about answer option C regarding 

the tribal education department and whether that referred specifically to a department on their 

reservation or if it was meant more generally. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twelve of 14 respondents felt that question was asking who had sponsored any community-based 

development programs they may have attended. One respondent thought the question was asking if 

they had gone, and two responses were hard to categorize:

“I'm a little confused about whether that means a department specifically within our tribe that we have 

on our reservation or if that's just more general. I know CCSD puts on trainings. I don't know whether 

the ones that I attended were sponsored by CCSD. I don't have the impression that they were. I have the 

impression that it was more like our individual schools reach out to the tribal reservation and set it up 

that way, but I do know that the district offers trainings.”

“Professional development, which we do it every day, or we have it every other week. We do it every 

other week, because I'm the head of the math department. We have some kind of peer share, we pull 

videos up from the web site teaching channel and I implement it a large extent. We have a lady coming 

in also, doing professional development, she comes about every three months and she asked us to read 

a book and to implement it into our teaching.”

Teacher #4 Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the 

last two years? Fill in all ovals that apply.

A. State

B. District

C. Tribal education department

D. Indian education professional associations

E. College or university

F. Other (please specify): ________________
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 142

Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow-up 

question]: What is this funding source?

Of the five teachers who responded, one said they did not know, one said Teach for America, one said 

ELO (unsure of what this stands for), one said the school where they taught, and one said the Minnesota 

Literary Council. 

Solutions to Improve Item

Respondent: One respondent said, “To me it automatically makes me think of the school district, which 

out here, like for our school, is kind of meaningless, because we're the only school in our school district. 

Tribal Education Department, Indian Education Professional Associations, college university, for me, I 

guess the people that actually sponsored mine were a college or a university, Indian Education 

Professional Associations, because I feel like the Tribal Education Department is nested within that, and 

then the Tribal Education Department and the district.”

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 143

Teacher #5Trend/VE012668

To what extent do you use the following to assess student progress? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. State assessments A B C D Trend/VE012670

b. District assessments A B C D Trend/VE012672

c. Assessments developed by American 

Indian or Alaska Native organizations A B C D Trend/VE012673

d. Tests supplied by textbook 

publishers (for example, end of unit 

or chapter tests)

A B C D Trend/VE012674

e. Teacher-made tests or quizzes A B C D Trend/VE012675

f. Performance-based assessments A B C D Trend/VE012676

g. Group projects A B C D Trend/VE012678

h. Oral responses of students during 

class discussions A B C D Trend/VE012681

i. Assessments to evaluate English 

language proficiency A B C D New/VH040300

Item 5

Reading: NA

Understanding: One respondent noted that they did not think of grades as a measure of student 

progress and further indicated that they did not know what “assessments developed by AI/AN 

organizations” meant. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 23 respondents, 22 thought the question had to do with how they assessed their students. One 

respondent thought it had to do with whether or not AI/AN students could understand questions. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 144

Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency” 

mean to you?

Out of a total of 22 respondents to this question, 12 referenced either ELL, ESL, or ELP as assessments 

for English language. Many of these respondents additionally commented that they were unsure if 

AI/AN students received these assessments, mentioned that the counselor administered these tests, or 

noted that all of their students were English speakers. Eight teachers responding felt that the question 

was simply asking about testing English, and two indicated that they did not know.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask 

the follow-up question]: Can you give me examples of how you have used “assessments to evaluate 

English language proficiency” to assess student progress?

There were 12 respondents for this question and three said they looked at vocabulary, two mentioned 

writing assignments, and one said ESL. One brought up that they were a math teacher and another said 

they were a science teacher—both indicated that they did not do English assessments. One teacher said 

they used oral assessments and another said they used the ACCESS test but said this was used when 

they taught in the Caribbean. Two gave difficult-to-classify answers:

“That’s the only thing that I’m thinking about when I think of assessments to evaluate English. Specific 

involves what is in that test or other examples?”

“The only thing I can think of is the language program that OLC has and then Red Cloud has their own 

language program too so they develop their own curriculum, all of that stuff.”

Probe 5: In question C, what specific organizations did you consider when answering this question?

Eight of 21 respondents said that they had not considered any specific organizations when answering 

questions, seven mentioned some specific AI/AN organizations or groups including tribes. One said the 

state, four said they did not know, and one said the Northwest Regional Lab.

Probe 6: When answering this question did you consider NIES information about student progress? If 

yes, in which question specifically did you consider NIES information about student progress?

Out of 20 respondents, 17 indicated that they had not considered the NIES, many noting they had never 

heard of it before. One said yes and two gave difficult-to-classify answers: 

“Yeah. When I read it, I actually froze up because I am anti-assessments or anti-standardized 

assessments. When I read that, that's all I felt right away and it makes me defensive because I do not 

follow this prescription of those types of assessments. That being said though, my students, they have 

high success rates. They're high achievers. The way I assess them is at a different level. No. It's just a 

feeling. I just have a general stereotypical feeling, I guess.”

“I guess when I read it, I was really thinking about how I assess my students in class and where I get that 

from.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 145

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider clarifying the question by adding “in your classroom” if the question is intended to 

capture what teachers themselves are doing rather than all assessments used at school. Otherwise, this 

item seems generally to be understood. 

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking 

about AI/AN students.

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 146

Item 6

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Interestingly the 21 responses given varied more than expected, likely because the question is 

straightforward, causing some respondents to overly analyze the question. Seventeen thought the 

question was asking them if they taught reading or language arts. Seven of these specifically mentioned 

the 8th grade, and one said middle school. One teacher simply said “no,” and three teachers gave 

difficult-to-categorize answers:

“Teaching them strategies on how to understand what they're reading and what they're writing, how to 

write sentences.”

“I think specifically if you have a reading language arts license, I would have answered yes.”

“I think that’s a departmentalized kind of a thing. We have reading teachers. We have language arts 

teachers. As an elementary teacher, this would’ve referred to me living on a reservation. [I either 

00:32:34] do everything.”

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

Teacher #6 Trend/VE012771

Do you teach reading/language arts to grade 8 students?

A. Yes  Go to Question 7

B. No  Go to Question 8
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 147
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 148

Teacher #7 Revised/VH040381

How often do you integrate materials about the following topics into your reading/language arts

lessons? Fill in one oval on each line.

Never

At least 

once a 

year

At least 

once a 

month

At least 

once a 

week

Every day 

or almost 

every day

a. American Indian or Alaska 

Native culture or history A B C D E

Revised/ 

VH0403

85

b. Current issues affecting 

American Indian or Alaska 

Native people or communities

A B C D E

Revised/ 

VH0403

86

Item 7

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 10 respondents, six mentioned AI/ANs specifically when answering the question about 

integrating materials. Four simply mentioned implementing culturally specific material. 

Probe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history” 

mean to you?

Answers from the 10 respondents to this question were all over the map. Three teachers mentioned 

stories, four mentioned history, one pointed to tradition, one said identity, and one said that they did 

not know.

Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska 

Native people or communities” mean to you?

All nine respondents pointed to current issues (such as the Keystone pipeline); other themes mentioned 

by the remaining respondents were treaties, mascots, suicide, government, issues with the state, 

alcohol, land rights, and suffering.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 149

Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., 

or to both, then ask the follow-up question]: Can you give me examples of how you have integrated 

materials about the topics into your reading/language arts lessons?

The 10 responses received for this question were very varied. Five respondents pointed to covering 

issues in their classes, two said they talked about history, one teacher said something about MLK Day 

(the rest of the answer had to do with having an African American president and needing an AI/AN 

president), one said they had students read Native authors, and one pointed to Native American History 

Month. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. Respondents seem comfortable with the item. We suggest asking for examples in the 

probes rather than asking what phrases mean to respondents as the simplicity of the question often 

leads to unnecessarily complicated answers. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 150

Item 8

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 20 respondents, 17 felt the question was asking them if they were math teachers; of these 17, 

eight teachers mentioned 8th graders. Three teachers stated that the question was trying to get at what 

one’s subject area was.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

Teacher #8 Trend/VE012783

Do you teach mathematics to grade 8 students?

A. Yes  Go to Question 9

B. No  Go to Question 10
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 151

Teacher #9 Revised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in 

one oval on each line.

Never

At least 

once a 

year

At least 

once a 

month

At least 

once a 

week

Every day 

or almost 

every day

a. American Indian or Alaska 

Native culture or history A B C D E

Revised/ 

VH0403

90

b. Current issues affecting 

American Indian or Alaska 

Native people or communities

A B C D E

Revised/ 

VH0403

92

Item 9

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All six respondents understood the question to be asking how they implemented AI/AN-specific culture 

or current issues into their teaching. 

Probe 3: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., 

or to both, then ask the follow-up question]: Can you give me examples of how you have integrated 

materials about the topics into your mathematics lessons?

Out of eight respondents, six spoke to using local context—whether that be local animals or fruits or 

history. One mentioned using the local language and one made a statement about Native American 

History Month but did not elaborate on what that looked like in their math class. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 152

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 153

Teacher #10 New/VH040395

How much do you agree with each of the following statements about the materials available in your 

school library, media center, or resource center? Fill in one oval on each line.

Strongly

disagree Disagree Agree

Strongly 

agree

a. The number of books and materials 

available for 8th grade students is 

sufficient.

A B C D New/VH040399

b. The quality of the books and 

materials available for 8th grade 

students is satisfactory.

A B C D New/VH040401

c. The number of books and materials 

specific to American Indian and 

Alaska Native culture available for 

8th grade students is sufficient.

A B C D New/VH040402

d. The quality of the books and 

materials specific to American 

Indian and Alaska Native culture 

available for 8th grade students is 

satisfactory.

A B C D New/VH040404

Item 10

Reading: NA

Understanding: Two respondents assumed that rows A and B also referred to AI/AN materials. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents felt that the question was asking about the types of materials available to their 

students. Ten teachers specifically mentioned the 8th grade and eight mentioned AI/ANs specifically. 

Probe 3: In questions a. and c., what does the phrase “is sufficient” mean to you? 

Answers from 22 respondents to this question varied greatly with seven saying they thought that 

“sufficient” meant enough, five said it meant minimum or adequate, four said it was a subjective term, 

four said it referred to quality, four felt it meant availability, one said variety, and one gave a hard-toqualify answer: “It's looking at resources in the building so that if we do want to work in ... I'm thinking 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 154

especially in terms of history and language arts. I'm doing things in ... Type into it to Native American 

culture and kind of geared toward that, do we have the resources in the building.”

Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you? 

Again, answers from 22 respondents were very varied to this question. Thirteen said that “satisfactory” 

meant meets minimum quality, two said the term was the same as “sufficient.” One said it had to do 

with reading level, one said it was needed to be current, one said it was a subjective term, one felt it had 

to do with the condition of books, one said it meant “enough,” and another said it meant “needs work.” 

Two gave hard-to-qualify answers:

“Let’s have a Native-American section here, don’t I? Look in the library yourself. Check it out. Look 

around. Where’s the Native-American side? You’ll find out. Go to [Harlem 00:38:02]. Go over there. You 

know what, these 2 schools are American Indian studies [sites 00:38:10].”

“Sufficient means enough, enough that we can ... whatever it is that we’re doing in our classroom. Do 

we have enough books that we can use in order to get across to our students what it is that we’re 

teaching, you know, that objective? Do we have not only specific things, but do we have a variety of 

things to where the kids can do a lot of research? Do they have a lot of places to go to find materials 

from different tribes or different languages and other customs? We just don’t have that.”

Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean 

to you?

Eight of 22 respondents said that the question had to do with the content of books, for instance if it was 

up-to-date and accurate. Six said it meant if the quality was good, four said it was the condition of 

materials, two said it was a subjective question, one said it should be by Native authors, one said it 

should be at appropriate reading level for grade, one mentioned variety, and one said it should be 

chosen by Native teachers. 

Probe 6: In questions c. and d., what does the phrase “specific to American Indian and Alaska Native 

cultures” mean to you?

Nineteen out of 21 respondents said it had to do with AI/ANs, and four mentioned that it had to do with 

being authored by AI/ANs. 

Solutions to Improve Item

Respondent: Add an “I don’t know” option. 

Interviewer: Consider specifying in rows A and B that the question refers to school books. 

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally seemed to grasp what the question was asking. 

ETS Recommendation (05/02/2014)

Keep this item. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 155

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 156

Item 11

Reading: NA

Understanding: It was hard for some teachers who taught multiple grades to specify about their 8th

grade students. Additionally, in responding to Probe 1, many respondents did not mention their AI/AN 

students specifically, but this does not necessarily have bearing on understanding. We recommend 

highlighting the term “American Indian or Alaska Native” to ensure that is the focus for respondents. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 21 respondents said they were being asked to predict how many of their students would 

pass the 8th grade, and five mentioned AI/AN students specifically.

Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what 

extent does having a high or low proportion of American Indian or Alaska Native students influence 

the answer choice you selected?

There were 22 responses to this question, and given the incredibly wide variety of answers, there was a 

total lack of validity for this probe. The term “proportion” in the question led respondents to give actual 

numbers, percentages, or numbers of their AI/AN students compared to the rest of the class. And their 

answers about whether or not the proportions had an influence were often not answered, though they 

usually fell into three categories: it makes no difference, makes it easier to answer because the 

proportion is small, or makes it easier to answer because the proportion is high (usually for schools 

where AI/AN population is 100%).

Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the 

follow-up question]: Please explain why you selected the answer choice. In other words, how did you 

figure out the answer?

Teacher #11 New/VH040406

What proportion of your American Indian or Alaska Native students will complete the 8th grade?

A. None

B. A few

C. Some

D. Most

E. All

F. I don’t know.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 157

There were 21 responses to this question, and again, the answers varied greatly. Some teachers gave

answers that heralded back to the previous probe having to do with whether or not they had a high or 

low proportion (thus making it easy to calculate), while other respondents made blanket statements 

about expecting most of their students to pass. It is hard to say if the wide variety and vague answers 

given to this probe are a result of the way the probe is written or, more likely, that teachers are sensitive 

about the subject of the question.

Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

When asked if it was difficult or easy to answer, 15 of the 21 respondents said it was easy, though 

several of these respondents noted that they thought this was because they were at a smaller school 

and had smaller classes, whereas teachers at larger schools may have trouble. Three said it was difficult 

because they didn’t know who their AI/AN students were, and another said they did not like to admit to 

the numbers of where students were. Lastly there were three difficult-to-categorize answers, one of 

which was the following: “To answer totally honestly, it could have been written better and I could have 

answered it easily. But since it was, "A proportion," is a math-bound, and there's no proportions listed, 

in my view, anyhow, that made it difficult. Not the question ... or, the concept itself was not difficult, 

just the way it was worded.”

Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of 

other answer choices that would be more clear?

Out of 20 responses to this question, 18 respondents felt that the answers were clear, while two felt 

that the question was a little difficult as it was vague. As one stated, “Well, you need to know who 

they're comparing with. Who are you comparing with? You don't have a comparison, so how can you 

make a proportion if you don't have a comparison. To answer those, if I was to do it, this is how I'd do it, 

I'd be thinking okay, whites 60-40. We had 54 last year, we have 60 this year. It's more than some, I'd 

say most, because I think of the kids that left, I'd be thinking of the kids that left school. One expelled, 

went home at Christmas. Our turnover rate is about 45% here.”

Solutions to Improve Item

Respondent: One respondent said, “I think the proportion part should be taken out, because honestly if 

somebody is taking that survey and they haven't like recently looked at the, I don't go around thinking 

I've got this many of these students. I don't think people would do that. So, I just think the question 

maybe should ask, how many of your Native American students will complete the 8

th grade, or do you 

think a big amount would? But the proportional part, I don't like that.”

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: Given the wide variety of answers and the push back from some of the respondents, consider 

alternate wordings for this item. Also consider bolding “American Indian or Alaska Native students.”

ETS Recommendation (05/02/2014)

Revise this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 158

Rationale: ETS recommends revising this item to simplify the phrase "What proportion", which caused 

confusion for the respondents, by changing it to the phrase "About how many".
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 159

Item 12

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents to this question felt that the question was referring to how many of their students 

would be prepared for high school. AI/AN students were only referenced by a few respondents, and it 

should be kept in mind that some of the schools where interviews took place were 100% AI/AN, though 

the majority were not. 

Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what 

extent does having a high or low proportion of American Indian or Alaska Native students influence 

the answer choice you selected?

There were 17 responses to this question. Six teachers said that they had a small proportion of students; 

four said they had a large proportion. Seven said that the proportion had no effect, three said that it did. 

One teacher said they did not know. Two teachers had difficult-to-categorize responses: 

“Most because it has to be a pair with what I answer for 11 if I believe that most of my 8th grader were 

going to pass then most of them will be ready for high school as well.”

“Just like the last one. I think that the answer choices can be more specific. What percentages or 

different words to use. But I think that I had a good sense of what most meant related to some or all. So 

I think it was okay.”

Teacher #12 New/VH040408

What proportion of your American Indian or Alaska Native students will be prepared for high school?

A. None

B. A few

C. Some

D. Most

E. All

F. I don’t know.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 160

Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the 

follow-up question]: Please explain why you selected the answer choice. In other words, how did you 

figure out the answer?

The 18 respondents for this question gave very varied answers that ranged from counting the number 

of students to answers about how students pass no matter what, to pointing out that if the question 

actually used the term “8th graders,” that would change how they answered, to saying things about how 

they know their students pretty well and that is what they based their answers on.

Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

Twelve of the 19 respondents said that the question was easy to answer, six said it was hard, and one 

said “in the middle.”

Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of 

other answer choices that would be more clear?

Fifteen of the 17 respondents said that the answer choices were clear, and two said they were not—

largely because they were seen as vague. 

Solutions to Improve Item

Respondent: One teacher felt that “proportion” was the wrong term and should be changed to 

"portion." Another suggested that the word “prepared” should be clarified as having to do with 

academic preparedness or social preparedness. 

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider changing the term “proportion” and specifying what is meant by “prepared.” 

Additionally, specify if teachers should be reflecting on their 8th grade students. 

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to simplify the phrase "What proportion", which caused 

confusion for the respondents, by changing it to the phrase "About how many".
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 161

Teacher #13 New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student absenteeism A B C D New/VH040413

b. Student tardiness A B C D New/VH040414

c. Student health problems A B C D New/VH040416

d. Teen pregnancies A B C D New/VH040417

e. Drug or alcohol use by students A B C D New/VH040421

f. Student misbehavior in class A B C D New/VH040422

g. Physical conflicts among students A B C D New/VH040424

h. Bullying A B C D New/VH040425

i. Low student aspirations A B C D New/VH040427

j. Low teacher expectations A B C D New/VH040428

k. Low family involvement A B C D New/VH040430

Item 13

Reading: NA

Understanding: NA 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 21 respondents understood what the item was asking. One respondent provided a difficultto-quantify answer: “Behavior, I guess involvement from students, teachers, and the community to 

parents.”

Probe 3: In question a., what does the phrase “student absenteeism” mean to you? 

All 21 respondents felt that “student absenteeism” meant students missing school. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 162

Probe 4: In question b., what does the phrase “student tardiness” mean to you? 

All 21 respondents said that “student tardiness” meant students being late. Most of the respondents 

interpreted this as being late to school, others as late to class, and still others as late to both school and 

class. 

Probe 5: In question c., what does the phrase “student health problems” mean to you? 

Answers varied for this question from the 21 teachers providing answers. Nine mentioned being sick, 

four said it could be emotional, three brought up asthma, two said lice, two said diabetes, two 

mentioned ADHD, two said it would be a physical issue, two said allergies, one mentioned obesity, two 

brought up hygiene, and one said broken bones. 

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then ask 

the follow-up question]: Can you give me examples of how teen pregnancies are a problem in your 

school?

There were only seven respondents to this question about teen pregnancies. Two said bullying of the 

girls was a problem, two said that missing school was an issue, and four gave difficult-to-qualify 

responses:

“After hours there's not enough supervision. We had three or four pregnant. That got pregnant the last 

year that were 8

th graders. Some of them got pregnant at home, at Christmas and came back. So it all 

comes down to talking to your kids, interacting with your kids, and basically talking to them. But then 

that doesn't help.”

“I had a pregnant 8th grader last year. I feel that’s a problem, not that I know her. She’s great but I feel 

when you’re pregnant and you’re in 8th grade, that’s a problem. I think we have four, not 8th graders. 

We had one 8th grader and then I think three or four high school girls that were pregnant last year. We 

have 50 students in our high school. We have 60 in our middle school. Proportionality wise, that’s a 

good number.”

Probe 7: In question e., what does the phrase “drug or alcohol use by students” mean to you? 

All 21 respondents felt that “drug or alcohol use among students” meant whether or not students were 

using drugs. Most respondents expanded to explain that they thought that most of this was occurring 

outside of school, though some said it could happen in school. 

Probe 8: In question f., what does the phrase “student misbehavior in class” mean to you? 

There were 20 responses to this question. Seven said that student misbehavior had to do with students 

not following rules, five mentioned students disrupting class, four said it was about students acting up, 

one said it was students being rude, another felt it was about students making bad choices, another said 

insubordination, and yet another said it had to do with students not listening. 

Probe 9: In question g., what does the phrase “physical conflicts among students” mean to you? 

Sixteen out of 21 respondents used the word “fight” to describe what they felt “physical conflicts among 

students” meant to them. Three used the word “physical” and two mentioned pushing. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 163

Probe 10: In question h., what does the phrase “bullying” mean to you? 

Out of 21 respondents, eight said it was systematic or repeated behavior, seven felt that “bullying” was 

verbal in nature, six mentioned cyber bullying, four felt it was physical in nature, and four brought up 

students being picked on, harassed, or intimidated. 

Probe 11: In question i., what does the phrase “low student aspirations” mean to you? 

When asked about “low student aspirations,” the 21 respondents had a variety of answers. Thirteen felt 

that it had to do with a lack of initiative or lack of vision for the future, six said it had to with low selfesteem, one said “yes,” and another said that they did not know. 

Probe 12: In question j., what does the phrase “low teacher expectations” mean to you? 

Of the 20 responses received for this question, six said that they thought “low teacher expectations” 

had to do with teachers underestimating students, five indicated that it was about teachers who did not 

set that bar high, three mentioned teachers who do not care, two said it was about teachers who had 

given up, one said that low expectations led to bad behavior, one said it had to do with the obstacles 

teachers face, one mentioned little administrative support, and one simply disagreed that it was a 

problem. 

Probe 13: In question k., what does the phrase “low family involvement” mean to you?

There were 20 respondents on the issue of “low family involvement.” Seven of these mentioned parents 

missing parent-teacher conferences, seven mentioned low investment or engagement with children’s 

education, six said it was about parents not attending school activities (such as opportunities to 

volunteer, or going to games or concerts), five said it was about not communicating with teachers, one 

pointed to the fact that they were at a boarding school, and another mentioned that by the 8th grade, 

many parents see their children as adults and so are more hands-off. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: All respondents seem to understand the item and what it is asking, although one point may be 

that the question does not specify a grade level or a time frame. The time frame issue did not come up 

during participants’ answers, but it became clear that some teachers were answering for the entire 

school (as stated in the question) while others were responding only for 8th grade.

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students 

as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 164

the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; 

therefore the items should reference "AI/AN" when necessary in order to capture the most useful data. 

This recommendation is the same at grades 4 and 8.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 165

Survey Instrument: Administrator Grade 4/Sequence I

Item 1

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 11 respondents, six felt the question was asking about who funded or who had authority over 

their school, and five saw the question asking what kind of school it was.

Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain 

why.

Eight of the 12 respondents said that they did not think they needed to supply more than one answer, 

and four said that they could choose more than one answer due to the way the funding of their school 

was set up. 

Probe 4: In answer choice E, what does the phrase “other nonpublic school” mean to you? Can you 

give me examples of “other nonpublic schools?”

Six of the 11 respondents mentioned parochial schools as an example, five brought up private schools, 

three mentioned homeschooling, one said tribal schools, and one said alternative schools. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

School #1 Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.

A. Regular public school

B. Charter public school

C. Bureau of Indian Education contracted or grant school

D. Bureau of Indian Education operated school

E. Other nonpublic school
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 166

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 167

Item 2

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 11 respondents appropriately responded to the item and said that a boarding school was one where 

students lived at the school. 

Probe 3: What does the phrase “boarding school” mean to you?

Of the 11 teachers responding, 10 indicated that it meant living on campus and two mentioned dorms. 

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

questions]: Are there any students at your school that do not board? If yes, what proportion of your 

students do not board? To what extent does having a high or low proportion of students who do not 

board influence the answer choice you selected?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.

School #2 New/VH040433

Is your school a boarding school?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 168

Item

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 respondents, eight essentially said that the question was asking if their school was on a 

reservation, one said “tribal land,” and two gave difficult-to-categorize answers: 

“This one is a double-sided question because to the [tribe], we are on tribal land, all of California but, 

no, we are not on the reservation. We know. Whether the site that that school is on happens to coincide 

with the reservation that's been established by the Federal Government and it doesn't. But for the tribal 

land, because it's supposed to all tribal land, they still hold on to view that gap. We are on tribal land 

whenever or anywhere in this region.”

“That the property that is around us is tribal property, and that obviously we have a lot of tribal 

members living in this area, it doesn’t mean to me that the school is sitting on tribal land, but 

everywhere around us is and I don’t know if it is tribal land right here, I’m not sure. It’s all around us, I

know that.”

Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?

When asked what “located on a reservation/tribal land” meant, nine of the 12 respondents said that it 

meant being on a federal reservation, three felt that reservations and tribal land were the same thing, 

and one said that it meant self-governance. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

School #3 New/VH040437

Is your school located on a reservation/on tribal land?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 169

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 170

Item

Reading: NA

Understanding: One respondent was confused by what “immersion school” meant. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of the 11 respondents for this question, nine said it meant teaching in another language, one said 

they did not know what the phrase meant, and one said, “I think it might be getting at trying to see 

whether to track whether languages that might become extinct are still being taught. It might be looking 

also at curriculum.”

Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school” 

mean to you?

Of the 12 respondents for this question, one restated that they did not know what an immersion school 

was, nine said it meant teaching in an AI/AN language all day, one mentioned teaching in another 

language, and one talked about how their Native language was gone. 

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

question]: What proportion of students in your school participates in your schools’ language 

immersion program?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

School #4 New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 171

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 172

School #5 Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the 

following ways? Fill in one oval on each line.

Yes No

a. Academic club (for example, math club) A B Revised/VH040576

b. Arts club A B Revised/VH040577

c. Cultural dances or activities (for example, drum groups) A B Revised/VH040578

d. Drama club A B Revised/VH040580

e. Making school curriculum decisions A B Revised/VH040581

f. Open houses or back-to-school nights A B Revised/VH040584

g. Parent-teacher conferences A B Revised/VH040588

h. Parent-teacher organizations A B Revised/VH040589

i. School sports A B Revised/VH040592

j. Volunteer programs A B Revised/VH040596

k. Other (please specify): _____________________________ A B Revised/VH040597

Item

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answers once realizing that the answers were 

supposed to relate to families. Another respondent changed their answers once they reached the 

probes specific to each question section. 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 12 respondents to this question, 10 felt it was asking about parent involvement in the school, one 

thought it had to do with the tribe, and one felt it had to do with what the school was involved in. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 173

Probe 3: Are your answer choices for questions a. through k. specific to the families of your grade 4 

American Indian or Alaska Native students, or do they apply to the families of all American Indian or 

Alaska Native students in your school?

Eight of the 11 respondents said they considered the question as relating to all of the school’s families, 

two said just the AI/AN families, one said all families K-4, one said all 3-5 grades, and another said all K5. In several of these schools, it was noted that the population of AI/AN students and families was very 

high. 

Probe 4: In question a., what does the phrase “academic club” mean to you?

Out of 12 respondents, nine brought up math, six said science, five said reading/language, two said 

social studies, one said history, one said spelling, one said to enhance subject material, and one said it 

was after school. 

Probe 5: In question b., what does the phrase “arts club” mean to you?

Ten out of the 11 respondents said that “arts club” meant that it was fine art, three just said their school 

did not have such an activity, and one mentioned music. 

Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you? 

Seven of the 12 responding administrators said that they felt that “cultural dances or activities” were 

AI/AN-related. One mentioned drums, three said powwows, and one brought up ORF (did not explain 

what this stands for).

Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the 

follow-up question]: Can you give me examples of cultural dances or activities? Do you consider 

“drum groups” a good example of “cultural dances or activities”?

Four out of 10 responding administrators said that drum groups were a good example, two mentioned 

powwows, three mentioned different kinds of crafts, one said pit house activities, one mentioned sweat 

lodges, one brought up hand games, another said that for their tribe it was rattles and not drum groups,

one mentioned songs, and one had the following to say: 

“To me it means either assemblies or students doing these things. They’re either taking lessons or 

showing off with what they’ve learned where they is some theme to what they’re engaged in. [Drum 

groups good example?] AC/DC plays drum too so that’s a little weird to have it be the one thing that 

somehow the drums are ethnic or somehow. I don't know. I think because this questionnaire is in its 

infancy. Someone is just writing down some random stuff. That’s probably an example that needs to be 

refined with a couple of more activities. That would be clear if you decided to lift this as extracurricular 

or parent involvement. It would depend on what category they’re putting that in.”

Probe 8: In question d., what does the phrase “drama club” mean to you?

Nine of the 11 responses were that plays or theater constituted a drama club, four mentioned the 

performing arts in general, poetry was mentioned twice, and music, dance, and art were each 

mentioned once. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 174

Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?

When asked about making school curriculum decisions, seven of the 11 responding administrators said 

that it meant having curriculum input from parents, and five mentioned parent advisory committees. 

Probe 10: In question f., what does the phrase “open houses or back-to-school nights” mean to you?

All 11 respondents appropriately characterized back-to-school nights as open houses or opportunities to 

involve families in familiarizing themselves with the school. 

Probe 11: In question g., what does the phrase “Parent-teacher conferences” mean to you?

Again, all 11 respondents were in agreement that “parent-teacher conferences” meant parents and 

teachers meeting to discuss children’s progress. 

Probe 12: In question h., what does the phrase “Parent-teacher organizations” mean to you?

All 11 respondents saw parent-teacher organizations as an opportunity for parents and teachers to work 

together for the improvement of the school. 

Probe 13: In question i., what does the phrase “school sports” mean to you?

All 11 respondents spoke about sports in terms of identifying what sports constituted a school sport 

(and to that end basketball, baseball, soccer, volleyball, track, and softball were mentioned). And some 

administrators pointed to the fact that their elementary schools did not have school-sponsored sports 

but rather that was something managed by the wider community or to be found in middle or high 

school. 

Probe 14: In question j., what does the phrase “Volunteer programs” mean to you?

Volunteer programs were seen by all 10 respondents to correspond with non-paid work on the part of 

community or family members. 

Probe 15: What proportion of your students are American Indian or Alaska Native? To what extent 

does having a high or low proportion of American Indian or Alaska Native students in your school 

influence the answer choice you selected?

The 10 answers to this question on the proportion ranged from 2% to 33% to 100%. Seven said that the 

proportion amount had no effect on their answers, one said they did not know, one said it made 

answering easier, and one said this was very influential in how they chose to answer. 

Probe 16: [Conditional probe: If the participant selects answer choice A for question k, then ask the 

follow-up question]: What are the other ways in which families of your American Indian or Alaska 

Native students are involved with your school?

Among the nine answers given to this question, the following themes arose: Bible study, festivals, 

budget decisions, leadership groups, grad school programs, reading to kids (mentioned twice), fund 

raising, and career day. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 175

Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider bolding “families of AI/AN students.”

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 176

School #6 Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions 

and culture are offered at your school each year? Fill in one oval on each line.

None 1 2 3

4 or 

more

a. Year-long course A B C D E Trend/VC962954

b. Semester-long course A B C D E Trend/VC962956

c. Workshop or unit A B C D E Trend/VC962957

d. Clubs A B C D E Trend/VC962958

e. Other programs (for example, study 

groups before or after regular class 

periods)

A B C D E Revised/VH018332

Item

Reading: NA

Understanding: Three respondents were confused about some of the terms used in this item such as 

“year-long” vs. “semester long” or “other courses,” and another felt that the question was better geared 

toward middle or high schools. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 11 respondents felt that the question was asking whether or not their schools were teaching about 

AI/AN culture. 

Probe 3: What does the phrase “study groups before or after regular class periods” mean to you?

All 11 respondents said that “study groups before or after regular class periods” meant before or afterschool programs either as activities or additional help for students. 

Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E to question e., then 

ask the follow-up question]: Can you give me examples of other programs offered at your school? 

Answers from the seven respondents were varied and included youth development, AI/AN-specific 

activities such as field trips, summer camp, tutoring, study groups, and, “The counselor could possibly 

have say maybe one. The counselor could pull a group of kids and work with them specifically. There 

could be one there. Again, there's just ...”
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 177

Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of 

other answer choices that would be more clear?

All 12 administrators responded to this question and five felt that the answer choices (or at least part of 

the answer choices) were clear, but nine of the responses indicated that there was some confusion over 

the answer choices. One mentioned that the chart was unnecessary, one pointed to the possible need 

for a “daily basis” category, one felt that E was not clear, three felt that the answer choices were in 

keeping with high school or middle school rather than elementary, another felt that clubs was repetitive 

of the other questions. Points made about the structure and content of questions are below: 

“Like I said, instead of how many of the following courses or programs are offered each year, I would 

maybe word it like is American Indian or Alaska Native traditions and culture taught in your school in any 

of the following methods maybe. Not methods, but ... How is the American Indian or Alaska Native 

traditions and culture taught in the curriculum, or offered in the curriculum at your school? Again, we 

don't have a year-long course, and we don't have a semester-long course, but the teachers do teach 

units that would address that.”

“It’s just kind of silly to have CBB but whatever. That’s a problem with this. You got to come up of some 

other way to be like answer to 6B is B. You know what I mean? There’s a different way to do that 

somehow. I don't know. [Number is not clear?] Right. If the answer to 6B is B then you haven’t clarified. 

I would do it like 6.1, 6.2, 6.3, 6.4, the answer is B so that you get a little bit clearer there. A, B, C, D, E, 

there’s no 6 associated with it. You should associate this as 6A, 6B, 6C, 6D, 6E not just to the .1.”

“Yeah, this is ambiguous...Yes, because it is a different groups, we have a couple different workshops 

and then we have one that goes every week, two times a week, so it's a same answer, two for if you 

meet two times or if it is two times a week. See what I'm saying?”

Solutions to Improve Item

Respondent: Some respondents felt that the chart was unnecessary. Course structure is different in the 

4

th grade and so the structure of answers is inappropriate or inapplicable. Row D is repetitive of Item 5. 

Change language from “courses” to “class.” Find a different numbering system for question rows or 

answers—don’t use letters, just use numbers. “Courses” does not allow for summer programs or afterschool activities. 

Interviewer: Consider including “language” along with traditions and culture in the question. 

Initial KAI Recommendation (Keep/Change/Discard)

Change: Given the respondent feedback, particularly to Probe 5, this question should be changed to 

better reflect the realities of elementary school. Additionally, may consider highlighting if this question 

is only to refer to the 4th grade, as it currently does not. 

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS recommends revising this item to a more simplified format by collapsing the answer 

choices to “Yes” and “No”. The answer choices for this existing item caused confusion among 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 178

respondents. Also, response frequencies indicate that the strong majority of the responses fall under 

the following two out of the five categories: “None” or “1”. As a result of changing this item to a 

“Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs 

about American Indian or Alaska Native traditions and culture offered at your school each year?”.

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 179

School #7 Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your 

school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student absenteeism A B C D Trend/VC963030

b. Student tardiness A B C D Trend/VC963031

c. Student health problems A B C D Trend/VC963032

d. Student misbehavior in class A B C D Trend/VC963034

e. Physical conflicts among students A B C D Trend/VC963035

f. Bullying A B C D Trend/VC963037

g. Low student aspirations A B C D Trend/VC963038

h. Low teacher expectations A B C D Trend/VC963039

i. Low family involvement A B C D Trend/VC963040

j. Inadequate transportation for 

students A B C D New/VH040475

Item

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 12 respondents, 10 felt that the question was referring to problems at the school. One person 

specifically mentioned problems for AI/AN students, one person felt it had to do with problems for 

students, staff, and family, and another mentioned that they were thinking of all students. 

Probe 3: In question a., what does the phrase “student absenteeism” mean to you? 

All 12 respondents felt that this question was asking about students who were missing from school or 

class or both. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 180

Probe 4: In question b., what does the phrase “student tardiness” mean to you? 

All 12 respondents thought this question was about asking about students who were late to school or 

class or both. 

Probe 5: In question c., what does the phrase “student health problems” mean to you? 

Of the 12 respondents providing an answer to this question, six mentioned children being sick or ill, four 

brought up diabetes, three said asthma, two mentioned mental or emotional problems, one said dental, 

one said sleep deprivation, one mentioned physical issues, one brought up allergies, one noted obesity, 

and one mentioned lice. 

Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you? 

There were 12 respondents for this question and four mentioned disruptive behavior as student 

misbehavior in class, three said it was students not listening, two said that it had to do with poor 

classroom management, two noted rule breaking and two brought up interrupting and disrespect. 

Probe 7: In question e., what does the phrase “physical conflicts among students” mean to you? 

When asked about physical conflicts, out of 12 responses, four mentioned pushing, three said shoving, 

three used the word “fight,” three said it had to do with students putting their hands on one another, 

two said hitting, one said bullying, and one said it had to do with classroom management. 

Probe 8: In question f., what does the phrase “bullying” mean to you? 

When discussing bullying, seven of the 12 respondents said that the behavior had to be consistent and 

repetitive, five said it was generally verbal, three mentioned physical aspects, one brought up cyber 

bullying, one said that it had an effect on children’s self-esteem, and one said it was related to poor 

classroom management/poorly organized adults. Interestingly it seemed that many of the 

administrators did not think that many of the complaints about bullying at their schools were actual 

bullying, but rather just kids being kids. 

Probe 9: In question g., what does the phrase “low student aspirations” mean to you? 

Out of the 12 respondents, six intimated that low student aspirations were due to the low bar that 

teachers set, four mentioned motivation on the part of students, one said it had to do with students not 

trying hard, one brought up disabilities, and one said that some children did not like coming to school 

because they were bullied. 

Probe 10: In question h., what does the phrase “low teacher expectations” mean to you? 

With regard to low teacher expectations, five of the 12 respondents said that it had to do with teachers 

setting a low bar, two said it was because teachers were overwhelmed or had given up, one noted poor 

organization among teachers, one said it was teachers who did not bother to push their students, and 

three gave hard-to-qualify answers:

“A lot of them coming up with that problem. Is there a problem? But I don’t see. Teachers aren’t coming 

to us and saying that there’s a problem.”

“That says to me how much the administration expects of them as far as putting effort into educating 

our children.”
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 181

“I would say ... Overall, here the teachers actually have really high expectations. I still answered that 

small extent, because I think sometimes ... One of the areas we struggle with as a school is we have a 

high level of proficiency. About 80% of our kids are proficient, but there's still that other 20%. We've got 

to focus on that. Even the students that are already proficient, we don't show a lot of growth. It's how 

are we challenging the kids that are already proficient, because proficient isn't really good enough.”

Probe 11: In question i., what does the phrase “low family involvement” mean to you?

Of the 12 respondents to this question, nine felt that low family involvement was related to families not 

physically coming to events or activities, and four mentioned parents being non-responsive to teacher 

communication. 

Probe 12: In question j., what does the phrase “inadequate transportation” mean to you?

There were 12 responses to this question and of those, eight felt that “inadequate transportation” 

meant not having a way to get to and from somewhere, one interpreted it as students having dangerous 

transportation options, and three gave hard-to-categorize answers:

“Somebody is trying to figure out. That’s a kind of different category. That’s like a jacked up district.”

“That means they're asking if the ability to transport our students hurts their education.”

“There’s none. I don’t see any inadequate transportation.”

Solutions to Improve Item

Respondent: Consider rephrasing "considering entire student body" or eliminating it completely; also 

consider increasing the scale to five or adding proportion/percentage. One respondent did not care for

the term "extent." Another suggested answer categories: rarely, sometimes, usually, always.

Interviewer: Is this question meant to apply to the entire school (which may be K-8 or higher) or just to 

the 4th grade?

Initial KAI Recommendation (Keep/Change/Discard)

Change: This item seems generally understandable. Consider making it specific to 4th grade. Additionally, 

should administrators be reflecting on all students or just AI/AN students?

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students 

as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of 

the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; 

therefore the items should reference "AI/AN" when necessary in order to capture the most useful data. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 182

School #8 Trend/VC963041

Which of the following types of student and family services are offered at your school on a regular 

basis? Fill in one oval on each line.

Offered Not 

offered

a. Health services for students such as a school nurse A B Trend/VC963042

b. Health services for families such as a community clinic A B Trend/VC963043

c. Social or counseling services for students A B Trend/VC963045

d. Social or counseling services for families A B Trend/VC963047

e. Adult education programs for all ages A B Trend/VC963048

f. Tutoring services for students A B New/VH017944

g. Career counseling services for students A B New/VH017945

Item

Reading: NA

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 responses given, nine concentrated on providing or offering services to families, and two 

focused on providing services to students. 

Probe 3: What does the phrase “a regular basis” mean to you?

Answers from the 12 respondents were varied in how they determined what “a regular basis” meant. 

Five said it meant regularly occurring, other answers were that it depended, would be most of the week, 

weekly, daily, three times a week, not once a year, all the time, or two times a month. 

Probe 4: In question f., what does the phrase “tutoring service for students” mean to you? 

There were 12 responses to this question. Ten were that “tutoring services” meant extra help for 

students, one person said it was to help students elevate their scores, and another simply said “yes.”
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 183

Probe 5: [Conditional probe: If the participant selects answer choice A to question f., then ask the 

follow-up question]: Can you give me examples of the tutoring services offered at your school?

Eleven out of 12 respondents said that their school offers tutoring before, after, and during classes. Only 

one respondent said that their school did not have such offerings. 

Probe 6: In question g., what does the phrase “career counseling services for students” mean to you? 

With regard to career counseling services, it is difficult to determine the group that the 12 respondents

were thinking about when giving their answers. At least four mentioned that they thought the question 

was more appropriate for secondary-level students, two mentioned job fairs generally, and seven said 

that their schools offered career counseling—something it is hard to imagine that 4th graders would be 

involved with given their age. 

Probe 7: [Conditional probe: If the participant selects answer choice A to question g., then ask the 

follow-up question]: Can you give me examples of the career counseling services offered at your 

school?

There were only seven responses to this question. Three mentioned job fairs, three brought up outside 

speakers, and two mentioned teachers as people providing career information to students. Again, it is 

difficult to say if the focus here was purely on 4th grade students. 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep/Change: Consider bolding if the question is focused on the whole school or only 4th graders.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 184

Item

Reading: NA

Understanding: One respondent found the item difficult to answer but could not clearly convey why, 

and also noted that they were from Guatemala originally and so had a different definition for what a 

pueblo was. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 12 responses given to this question, seven mentioned preparedness, and six thought that the 

question meant incorporating the culture.

Probe 3: What does the phrase “preparing American Indian or Alaska Native students for life in their 

community, tribe, group, pueblo, or village” mean to you?

Out of 12 responses, five mentioned the tribe with regard to preparing AI/AN students, one said they 

did not know, and three mentioned the community. Several respondents gave hard-to-qualify answers: 

“That’s really a touchy question there, because I am not Native American and some of this is allowing 

them to have excuses why they are late for school and my opinion is that I think we ought to treat 

everybody the same. And they want to be treated different and they think the teachers need to learn 

the culture of the Native American and accept that they’re late all the time or they’re not at school and 

it’s hard for me to deal with this one. (Chuckle.)”

“It it's specific to American Indian or Alaska Native in living like ... the dynamics on a reservation or 

something like that, I don't know the answer to that.” 

“That means that how much emphasis is being put in and to get the kids ready to utilize their way of 

living and resources to be better citizens in the future. However, when we say, “To what extent do your 

school curriculum emphasize,” if I just put the 4th grade as a block of learning time, then I’m talking 

about the curriculum that we, all the schools have, the buy curriculum. But if our school has learning 

every day, that our school is an active school with activities related to their life, I think we are doing a 

better, a very good job in that.”

School #9 New/VH040496

To what extent do your school’s 4th grade curricula emphasize preparing American Indian or 

Alaska Native students for life in their community, tribe, group, pueblo, or village? 

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 185

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up 

question]: Can you give me examples of how your school’s 4th grade curricula emphasize preparing 

American Indian or Alaska Native students for life in their community, tribe, group, pueblo or village?

There were eight responses to this question. Three administrators stated that they did not think that 

their 4th grade curricula did emphasize preparing AI/AN students for life in the different communities 

listed. Three said that their schools offered culturally relevant programming, and two brought up 

encouraging common core educational components. 

Probe 5: What does the word “pueblo” mean to you?

Of the 11 responses given, eight felt that “pueblos” were related to American Indian people in the 

Southwest, one person (from Guatemala) said that it meant city, one said “house drum,” and another 

said that it did not really have any meaning to them. 

Probe 6: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

When asked if the item was easy or difficult to answer, four of the 12 respondents said it was easy, and 

eight felt that it was difficult. Of those indicating that it was difficult, the reasons had to do with the fact 

that several respondents did not see “community” and “tribe” (among the other phrases) as being the 

same thing. For instance, many felt that the community referred to being among the wider community, 

whereas being with the tribe was a separate kind of community. Additionally, many respondents 

mentioned that the item’s emphasis on the 4th grade curricula was problematic because they often do 

not emphasize cultural learning that may happen at the school level. This is important as many curricula 

are approved at the district or state level.

Solutions to Improve Item

Respondent: Expand examples on what is meant by “preparing youth for their community.”

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems that there was some confusion over the multiple terms to describe what were seen as 

different kinds of communities. The language around “curriculum” caused many respondents confusion 

over how they should answer the question as it is currently worded. If the wording could be changed to 

be more inclusive around the curriculum piece then it should be kept. Consider removing the word 

“community” unless rephrasing it to “tribal community.”

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS recommends dropping this item as respondents indicated that life in their community is 

not necessarily the same as life in their tribes. Respondents indicated that the school curricula may 

succeed in preparing students for life in their community, for instance, but not necessarily life in their 

tribe. In addition, not all students may have a “tribal life” and may live in a community without other 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 186

AI/AN families. Furthermore, sub-items "b" and "c" may be more applicable to higher grades, as 

expressed by some of the respondents, therefore they may not be appropriate for grade 4.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 187

Survey Instrument: Administrator Grade 8/Sequence I

Item

Reading: NA 

Understanding: NA

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 10 respondents, six administrators felt the question item asked them to identify school type, 

three felt it asked about the school’s funding source, and one felt the question was trying to determine 

whether or not the school was effective. 

Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain 

why.

All 11 administrators who responded reported needing only one answer choice.

Probe 4: In question e., what does the phrase “other nonpublic school” mean to you? Can you give me 

examples of “other nonpublic schools?”

Of the 11 responses to this probe, six pointed to religious or parochial schools, five focused on private 

schools or institutions, and two highlighted charter schools as types of “other nonpublic schools.” One 

administrator cited alliance or alternative schools.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

School #1 Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.

A. Regular public school

B. Charter public school

C. Bureau of Indian Education contracted or grant school

D. Bureau of Indian Education operated school

E. Other nonpublic school
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 188

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 189

Item

Reading: NA

Understanding: None of the respondents demonstrated any difficulty understanding the question; 

however, one administrator pointed out that some lay/regular survey participants may not be familiar 

with the term “boarding school.” 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 respondents, 10 administrators felt that the question asked whether the school boarded 

students/students lived at the school, while one felt that the question was looking at the type of school 

operated, specifically. 

Probe 3: What does the phrase “boarding school” mean to you?

Seven administrators (out of 12 respondents) defined “boarding school” in terms of providing students 

with housing or students living on campus or in dorms. Two referred to students living on campus or at 

the school “24/7,” while one felt that students in boarding schools lived independently while on 

campus. Three respondents made it a point to reference the historical context of boarding schools in 

AI/AN communities. 

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

questions]: Are there any students at your school that do not board? If yes, what proportion of your 

students do not board? To what extent does having a high or low proportion of students who do not 

board influence the answer choice you selected?

Two administrators responded to this probe. The first reported a 60% boarding population, while the 

other noted that 33 of the 460 enrolled students boarded. Neither respondent felt that this proportion 

influenced their answer choice selection.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

School #2 New/VH040433

Is your school a boarding school?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 190

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 191

Item

Reading: NA

Understanding: One respondent admitted to experiencing some difficulty determining the meaning of 

the phrase “on a reservation/on tribal land.” 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the nine respondents, six felt that the question was asking if the school was located on land owned 

by the tribe or within the boundaries of the reservation, while one felt that it was asking about which 

group or entity had jurisdiction over the provision of educational services. Two administrators 

referenced special or complex circumstances in their responses:

“Yes, it was a reservation land originally here in Oklahoma, but because of the Treaty it still is identified 

with us, but it’s not reservation land. It’s not set aside specifically anymore, just for the tribe its set aside 

for individuals. But you have to know that kind of in your head, because some people will say, ‘Yeah, I’m 

on a reservation land.’ But in Oklahoma we are not.

“Up here, this could be a little confusing but I'm interpreting it whether it's on reservation or tribal land, 

which the answer would be no. The uniqueness of Wellpinit is we're on a reservation but we're on 

publics land within the reservation.”

Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?

Nine administrators (out of 12 respondents) felt that “reservation or tribal lands” referred to 

reservations, Native-owned land, or federal land set aside for tribes. One felt that it referred to land 

where taxes were not paid on the land, while one other administrator felt that the term was nested 

within a historical definition. One admitted to being unsure of the definition of the phrase:

“That’s where I’m struggling. I know what a reservation is but I thought it was the same thing. Me being 

a non-Native maybe that’s why I don’t know.” 

Solutions to Improve Item

Respondent: NA

Interviewer: NA

School #3 New/VH040437

Is your school located on a reservation/on tribal land?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 192

Initial KAI Recommendation (Keep/Change/Discard)

Change: While none of the respondents demonstrated difficulty responding to the question, a few 

noted situations where the presence of trust lands could make it more challenging to answer. Consider 

adding “trust land” to the list of tribal land options within the text of the question. 

Another respondent noted that nontribal members or individuals unfamiliar with the concept or 

definition of tribal lands/reservations may experience confusion or difficulty when answering the 

question.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 193

Item

Reading: NA

Understanding: One respondent thought the question was asking about American Indian and Alaska 

Native languages separately. A second administrator demonstrated some difficulty interpreting the 

phrase “immersion program.”

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 respondents, eight felt that the question was trying to ask if an AI/AN language was used all 

day or included in the general school curriculum. Two administrators felt that the question only asked if 

the school ran an immersion program, while one felt that the question wanted to determine if staff and 

students spoke a Native language instead of simply teaching it. 

Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school” 

mean to you?

Seven administrators (out of 12 respondents) felt that an immersion program incorporated an AI/AN 

language throughout the day and throughout course curriculum. One felt that the term meant that the 

school had certified teachers to provide appropriate language immersion instruction. Five of the 

administrators discussed current or past plans to implement an immersion program in their responses.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up 

question]: What proportion of students in your school participates in your schools’ language 

immersion program?

NA

Solutions to Improve Item

Respondent: NA

Interviewer: NA

School #4 

New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

A. Yes

B. No
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 194

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 195

School #5 Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the 

following ways? Fill in one oval on each line.

Yes No

a. Academic club (for example, math club) A B Revised/VH040576

b. Arts club A B Revised/VH040577

c. Cultural dances or activities (for example, drum groups) A B Revised/VH040578

d. Drama club A B Revised/VH040580

e. Making school curriculum decisions A B Revised/VH040581

f. Open houses or back-to-school nights A B Revised/VH040584

g. Parent-teacher conferences A B Revised/VH040588

h. Parent-teacher organizations A B Revised/VH040589

i. School sports A B Revised/VH040592

j. Volunteer programs A B Revised/VH040596

k. Other (please specify): _____________________________ A B Revised/VH040597

Item

Reading: NA

Understanding: One respondent appeared to equate references to clubs and activities with actual 

school course activities. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 10 respondents, eight felt that the question was asking about types and levels of family 

involvement, while two felt that the question asked about the opportunities for involvement offered to 

families. 

Probe 3: Are your answer choices for questions a. through k. specific to the families of your grade 8 

American Indian or Alaska Native students, or do they apply to the families of all American Indian or 

Alaska Native students in your school?

Six administrators (out of nine respondents) referenced all families when answering. Two referenced 

only the families of 8th grade students, and one limited their answer to AI/AN families. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 196

Probe 4: In question a., what does the phrase “academic club” mean to you?

When asked about academic clubs, eight out of 10 respondents felt that the term referred to academic 

or course-specific groups; one felt that it referred to a group for gifted or talented students; and one felt 

that it referred to a type of peer-based mentoring. 

Probe 5: In question b., what does the phrase “arts club” mean to you?

Six of the 10 administrators who responded felt that the item referred to any visual or performance art. 

Three administrators, rather than defining the term, described past or present examples of arts clubs at 

the school, while one simply noted that the school did not offer such an activity.

Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you? 

Ten administrators responded to the probe. Of these, two made general references to powwows and 

drum groups, while four cited specific examples of cultural activities offered at or by the school. Three 

administrators discussed the traditional and cultural value of such activities in their responses. One 

respondent defined the probe item as visual or performance art. 

Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the 

follow-up question]: Can you give me examples of cultural dances or activities? Do you consider 

“drum groups” a good example of “cultural dances or activities”?

Of the nine respondents, five described powwows and drum groups offered by or at the school. Two 

respondents described specific types of dance taught to students, while one made a more general 

reference to teaching students traditional types of dance. One administrator failed to provide an 

example, simply noting that drum groups are a good example of traditional activity. Three other 

administrators similarly noted that drum groups are, in fact, a good example. 

Probe 8: In question d., what does the phrase “drama club” mean to you?

All but one administrator (out of 10 respondents) described “drama club” in terms of acting, plays, 

stage, or theater. The 10th administrator’s response described the school’s plans to develop a student 

drama group. 

Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?

Of the 11 respondents, six described the probe item as parental involvement or input into curriculum or 

course decisions. Three administrators felt it referred to a specific council or committee that included 

parents. One administrator’s response defined it simply as parental feedback on school curriculum, 

while one other administrator noted that only the state drove curriculum-based decisions for the school. 

Probe 10: In question f., what does the phrase “open houses or back-to-school nights” mean to you?

Ten of the 11 respondents to this probe defined this probe item in terms of a specific school event 

where parents met or interacted with teachers and school staff. One administrator referred to it as a 

type of public relations activity. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 197

Probe 11: In question g., what does the phrase “Parent-teacher conferences” mean to you?

All 10 responses described “parent-teacher conferences” as a meeting between parents and teachers 

(both formal and informal in nature). 

Probe 12: In question h., what does the phrase “Parent-teacher organizations” mean to you?

Three administrators (out of 10 respondents) specifically referred to PTAs or PTOs in their responses, 

while five made more general references to some type of parent-teacher committee. One respondent 

described a past example of this type of organization, while two others noted that no such organization 

currently existed. 

Probe 13: In question i., what does the phrase “school sports” mean to you?

Of the 10 responses describing “school sports,” six referred to a specific sport (e.g., football, basketball, 

and track) and four referred to sports as a collective, general category. 

Probe 14: In question j., what does the phrase “Volunteer programs” mean to you?

Six administrators (out of 10 respondents) described “volunteer programs” as occasions when parents 

or family members provided assistance to the school. Two administrators described specific programs or 

examples in their responses, while two others noted that no such programs currently exist.

Probe 15: What proportion of your students are American Indian or Alaska Native? To what extent 

does having a high or low proportion of American Indian or Alaska Native students in your school 

influence the answer choice you selected?

Of the nine total responses given, one administrator reported a proportion of 2%, one a proportion of 

20%, and one a proportion of 33%. Five administrators reported a proportion of 95% or greater. Two 

respondents noted that these proportions had no influence over their answer selection. One respondent 

found this probe very confusing and needed to reread it before responding, while three other 

respondents found the probe too confusing to answer.

Probe 16: [Conditional probe: If the participant selects answer choice A for question k, then ask the 

follow-up question]: What are the other ways in which families of your American Indian or Alaska 

Native students are involved with your school?

Of the seven responses provided, two described alumni activities, one described families serving as 

chaperones or assisting with supervision during field trips, and four described more miscellaneous 

activities (e.g., cheerleading, band, and archery contests). 

Solutions to Improve Item

Respondent: One respondent noted that not all question row items were available or offered at the 

school. 

Interviewer: NA
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 198

Initial KAI Recommendation (Keep/Change/Discard)

Change: Several respondents noted that some of the question rows listed did not apply as they were not 

available at the school. Consider adjusting the answer choices to include an option for “N/A” or “Do not 

offer/Not available.” Also specify and emphasize if the question focuses on all AI/AN families or just the 

families of 8th grade students. 

ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 199

School #6 Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions and 

culture are offered at your school each year? Fill in one oval on each line.

None 1 2 3

4 or 

more

a. Year-long course A B C D E Trend/VC962954

b. Semester-long course A B C D E Trend/VC962956

c. Workshop or unit A B C D E Trend/VC962957

d. Clubs A B C D E Trend/VC962958

e. Other programs (for example, study 

groups before or after regular class 

periods)

A B C D E Revised/VH018332

Item

Reading: NA

Understanding: Two respondents expressed some confusion when reading and answering the question 

item, one because the answer choices are not specifically offered at the respondent’s school, while the 

other could not understand the question’s meaning. 

Changing Answers: NA 

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One respondent (out of 12 respondents) was not sure what the question was trying to ask. Seven 

administrators felt the question was asking about AI/AN culture-specific classes and courses. Two 

administrators felt that the question explored the types of cultural programming offered at the school, 

outside of a traditional curriculum, and one thought the question wanted to know how culture was 

taught at the school. One respondent thought the question was asking about their own level of 

knowledge of Native culture. 

Probe 3: What does the phrase “study groups before or after regular class periods” mean to you?

One of the 11 respondents was unsure of the definition of the probe item. Four defined the phrase in 

terms of activities taking place outside of the classroom or regular school day. Three respondents 

described it as a formal group or tutoring program. Three administrators described the phrase as topicrelated discussion groups (one), independent study (one), or follow-up study or instruction (one). 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 200

Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E for question e., then 

ask the follow-up question]: Can you give me examples of other programs offered at your school? 

The four responses to the probe referred to the Century 21 program (one), the MEFE (one), a program 

involving NASA (one), and a professional performance group (one). 

Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of 

other answer choices that would be more clear?

All 11 administrators who responded noted that the answer choices seemed clear. 

Solutions to Improve Item

Respondent: One administrator suggested rephrasing some of the question rows to read more clearly:

"That just meant a year-long course ... or and programs so it's really, is there a specific course, is there a 

specific program that they're talking about?"

Another respondent suggested separating out questions, as the current format can be confusing and 

overly complicated to answer.

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider providing examples of or definitions for the question row items. This should help 

alleviate some of the respondent confusion. Also specify if these courses are school-wide or 8th gradespecific. 

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS recommends revising this item to a more simplified format by collapsing the answer 

choices to “Yes” and “No”. The answer choices for this existing item caused confusion among 

respondents. Also, response frequencies indicate that the strong majority of the responses fall under 

the following two out of the five categories: “None” or “1”. As a result of changing this item to a 

“Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs 

about American Indian or Alaska Native traditions and culture offered at your school each year?”.

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 201

School #7 Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your 

school? Fill in one oval on each line.

Not at 

all

Small 

extent

Moderate 

extent

Large 

extent

a. Student absenteeism A B C D Trend/VC963030

b. Student tardiness A B C D Trend/VC963031

c. Student health problems A B C D Trend/VC963032

d. Teen pregnancies A B C D New/VH040470

e. Drug or alcohol use by students A B C D Trend/VC963033

f. Student misbehavior in class A B C D Trend/VC963034

g. Physical conflicts among students A B C D Trend/VC963035

h. Bullying A B C D Trend/VC963037

i. Low student aspirations A B C D Trend/VC963038

j. Low teacher expectations A B C D Trend/VC963039

k. Low family involvement A B C D Trend/VC963040

l. Inadequate transportation for 

students A B C D New/VH040475

Item

Reading: NA

Understanding: One respondent was unsure of the time frame covered by the question. This same 

individual later interpreted the question as specific to AI/ANs rather than to the entire school. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One administrator (out of eight respondents) answered in reference to specific answer rows rather than 

the entire question. Five administrators felt the question was asking about the types of problems or 

issues experienced at the school. Two administrators thought the question focused on issues that 

specifically inhibited students’ ability to learn or succeed in school. One respondent felt that the 

question was trying to explore the school’s management abilities. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 202

Probe 3: In question a., what does the phrase “student absenteeism” mean to you? 

Of the 12 respondents, 11 defined the probe phrase in terms of the student being away from or out of 

school without an excuse. One respondent described it as a lack of parental support.

Probe 4: In question b., what does the phrase “student tardiness” mean to you? 

Ten administrators (out of 11 respondents) described “tardiness” as being late for school or class. One 

administrator described a circumstance where the student’s physical distance from the school could 

impact his or her ability to come to school on time.

Probe 5: In question c., what does the phrase “student health problems” mean to you? 

All 11 respondents referred to the phrase in terms of a health or medical condition. Among these 

responses, two specifically referred to diabetes, one referred to lice, one to mental health issues, one to 

the flu, and one to any condition requiring medication. 

Probe 6: In question e., what does the phrase “drug or alcohol use by students” mean to you? 

Twelve administrators responded to this probe. Of these, seven defined the phrase as the use or abuse 

of drugs, alcohol, or other illegal substances. Four administrators described the prevalence of the issue 

in their school setting; one described a specific instance of addressing the issue with students. One 

administrator’s response described the school’s response to the issue in general. 

Probe 7: In question f., what does the phrase “student misbehavior in class” mean to you? 

Of the 11 respondents, 10 described the phrase as some kind of problem behavior or misconduct. One 

administrator defined it more specifically as behavior resulting in some kind of formal discipline. 

Probe 8: In question g., what does the phrase “physical conflicts among students” mean to you? 

Ten of the 12 respondents described the phrase as physical fighting or harm done to other students, 

with one other respondent detailing a specific instance of physical conflict that took place in the school. 

One administrator’s response was that the item was not an issue.

Probe 9: In question h., what does the phrase “bullying” mean to you? 

The 12 respondents to this probe had varying definitions of bullying. Five described it as harassing 

another student or causing another student stress or harm. Four described specific examples of bullying, 

with one other noting that bullying is something that happens more than once (i.e., the behavior is 

repetitive). Three defined it in terms of one student overpowering another, while two respondents 

noted that the term itself is difficult to define: 

“I hear this all the time, large extent. I think it may be due to a definition problem. They just don't 

understand the definition of it. So any type of verbal words looking at.”

“This one is so hard. Everyone got their own definition for it. I would say people lump a lot of things into 

bullying that may not be...”
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 203

Probe 10: In question i., what does the phrase “low student aspirations” mean to you? 

Eleven of the 12 respondents defined the probe item as students lacking goals, direction, or motivation. 

One respondent’s answer described a specific example of low student aspirations.

Probe 11: In question j., what does the phrase “low teacher expectations” mean to you? 

Seven administrator responses (out of 12 respondents) consisted of general descriptions of teacher 

expectations. Four administrator responses described the high levels of teacher expectations present at 

their schools. One response was difficult to code: 

“Well, these kids just come from the reservation and they haven’t been in schools, so they’ll probably 

not going to help my scores in any ... I mean instead of ... I’m a firm believer that kids will do what you 

expect them to do and I always think of the same few teachers that all year, they always came to mind 

and just ... I don’t know, I’m just kind of almost borderline, just don’t care.” 

Probe 12: In question k., what does the phrase “low family involvement” mean to you?

Of the 12 responses to the probe, nine discussed low family involvement in general terms, and two 

described how the school currently struggled with low involvement. One administrator had difficulty 

responding to the probe: 

“For some reason when they get out of elementary we lose them at the middle school level usually, 

because in 8

th grade they are involved in the sports and the other activities. So I don’t get a lot of family 

involvement, but again you need to say, ‘Is that involvement in the school or is it involvement in our 

program?’ So that might be... You might want to say something about you know are they involved in the 

school or... Because they may be involved in our program, but not in the school. You see what I mean? 

They might come to my meetings, but they won’t go to the parent-teacher meetings at the school. So I 

don’t know. That question was just a little... You might want to be a little more specific on that one.”

Probe 13: In question l., what does the phrase “inadequate transportation” mean to you?

Six administrators (out of 10 respondents) described the probe items as a lack of transportation to or 

from the school. Two administrators noted that inadequate transportation was not an issue for their 

school, while two others described it as an issue involving students’ parents. 

Probe 14: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then 

ask the follow-up question]: Can you give me examples of how teen pregnancies are a problem in 

your school?

Of the five administrators who responded, two noted that it is not a current problem but went on to 

describe past experiences with teen pregnancy. Three other administrators described how the issue 

impacted students’ ability to succeed in school.

Solutions to Improve Item

Respondent: One respondent requested more clarification on the time frame of the question (e.g., this 

school year? multiple school years?)
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 204

Interviewer: Consider rephrasing the question about teen pregnancies to focus on how they impact 

schools rather than how they are a "problem.”

Initial KAI Recommendation (Keep/Change/Discard)

Keep: Question seems generally understandable; however, potential changes could include specifying 

the time frame to which respondents should refer when answering, and rephrasing “teen pregnancy” to 

seem less stigmatic. Additionally, consider whether or not answers are wanted for the whole school, 

which could include up to 12th grade, or just for 8th grade students.

ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN 

students. 

This recommendation is the same at grades 4 and 8. 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 205

School #8 Trend/VC963041

Which types of student and family services are offered at your school on a regular basis? Fill in one oval on 

each line.

Offered Not 

Offered

Health services for students such as a school nurse A B Trend/VC963042

Health services for families such as a community clinic A B Trend/VC963043

Social or counseling services for students A B Trend/VC963045

Social or counseling services for families A B Trend/VC963047

Adult education programs for all ages A B Trend/VC963048

Tutoring services for students A B New/VH017944

Career counseling services for students A B New/VH017945

Item

Reading: NA

Understanding: One respondent found the language in question row B confusing. 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eight of the 10 respondents felt that the question tried to identify the types of services or programs 

available to students and families. Two administrators limited their responses to row A specifically. 

Probe 3: What does the phrase “a regular basis” mean to you?

Of the 10 respondents, nine administrators referred to a recurring schedule or regular time. One 

administrator simply referred to things taking place “on average.” 

Probe 4: In question f., what does the phrase “tutoring service for students” mean to you? 

Of the 11 administrators who responded, five defined tutoring services as the provision of academic 

support, five specifically described such services as taking place outside of regular school hours and 

three mentioned specific examples of types of tutoring services currently provided.

Probe 5: [Conditional probe: If the participant selects answer choice A for question f., then ask the 

follow-up question]: Can you give me examples of the tutoring services offered at your school?
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 206

Seven of the 11 respondents referred to generic tutoring services that provide students with additional 

academic support, and three specifically mentioned the Century 21/21st Century tutoring program. 

Probe 6: In question g., what does the phrase “career counseling services for students” mean to you? 

Six administrators (out of 11 respondents) referred to guidance or decision-making assistance about 

students’ futures, and one spoke of how it informs students of options outside the reservation. Four 

administrators described specific programs or services provided by the school. 

Probe 7: [Conditional probe: If the participant selects answer choice A for question g., then ask the 

follow-up question]: Can you give me examples of the career counseling services offered at your 

school?

Of the five responses provided, career counseling was described as career days (four), staff counselors 

(four), college visits or partnerships (three), and bringing in outside speakers or programs (four). 

Solutions to Improve Item

Respondent: One respondent suggested adjusting the question language to accommodate alternative 

terms, such as "school nurse."

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. 

ETS Recommendation (05/02/2014)

Keep this item. 

This item performed well in cognitive interviews. No issues were identified that would warrant revising 

or dropping this item.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 207

Item

Reading: NA

Understanding: One administrator expressed some confusion over the meaning of the phrase “dropping 

out” (i.e., Does it refer to a student leaving school or leaving the entire school system?). Another 

administrator questioned the time frame referenced by “last school year” (i.e., Does it refer to a 

calendar year or the student’s last year in school?). 

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One administrator (out of nine respondents) noted that they lacked sufficient data to accurately answer 

the question. Five administrators felt that the question was asking about dropout rates for the school. 

Two thought the question was asking them to define the problem, while one administrator felt that the 

question was asking for reasons why students drop out. 

Probe 3: Please explain why you selected the answer choice. In other words, how did you calculate 

the answer?

While four of the 11 respondents simply stated that student dropouts were not an issue in their schools 

(thus, the answer selection would be A), four other respondents conducted a general count of students 

still in school. Three administrators, though, noted that they lacked the information to come up with an 

accurate calculation. 

Probe 4: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

Of the 11 responses to the probe, five respondents found the question easy to answer, five found it 

difficult, and one found it in between easy and hard to answer. When describing why, three respondents 

School #9 Revised/VH040486

What percentage of your 8th grade students dropped out of school during the last school year? 

A. 0 – 2 %

B. 3 – 5 %

C. 6 – 10 %

D. 11 – 20 %

E. 21 – 30 %

F. 31 – 40 %

G. 41 – 50 %

H. More than 50 %
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 208

cited the phrasing of the question, while two noted they lacked the information to calculate an answer. 

One respondent explained that the question was easy to answer because student dropouts were not an 

issue at the school.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Keep (with reservations): Question generally seems fine as is. However, given the reservations 

expressed by multiple respondents regarding the lack of information (affecting their ability to accurately 

calculate a figure for their response), there could be concern regarding the reliability of respondents’ 

answers and the validity of the question item. If the question is meant to assess how many AI/AN 

students dropped out, consider adding that in and highlighting it.

ETS Recommendation (05/02/2014)

Revise this item. 

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking 

about AI/AN students.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 209

School #10 New/VH040507

To what extent do your school’s 8th grade curricula emphasize preparing American Indian or Alaska 

Native students for the following? Fill in one oval on each line.

Not 

at all

Small 

extent

Moderate 

extent

Large 

extent

a. Prepare students for life in their community, 

tribe, group, pueblo, or village A B C D

New/VH0

40511

b. Prepare students for the workforce A B C D

New/VH0

40518

c. Prepare students for high school and college A B C D

New/VH0

40522

Item

Reading: NA

Understanding: One respondent needed to reread the question item a few times in order to understand 

the phrasing.

Changing Answers: NA

Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the nine respondents, six felt the question was asking about how the school generally prepared 

students. One respondent felt that the question was asking about teacher capabilities, while another 

respondent thought it was asking if the school had high expectations for its students. One respondent 

was confused by the question and thought it could be interpreted in multiple ways: 

“I don’t know what they mean by “I prepare students for life.” I mean, if we’re preparing students for a 

community that has 80% unemployment and no job opportunities, then we’re doing a good job because 

a lot of our students don’t have opportunities when they graduate high school. But are we preparing 

them to be active participants in their community’s culture? That’s different then ... I guess without any 

qualifying awards or statements, every single kid is being prepared to be part of their community, but 

it’s like, what kind of preparation are they coming with? Are they prepared ... Does that make sense? I 

would interpret it ...”

Probe 3: In question a., what does the phrase “prepare students for life in their community, tribe, 

group, pueblo, or village” mean to you?

Three of the 11 respondents felt that the students were prepared for community life, but not tribal life. 

Ten administrators felt that the probe item referred to general preparation for the work and behaving 

appropriately in the community.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 210

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question a., then ask 

the follow-up question]: Can you give me examples of how your school’s 8th grade curricula 

emphasize preparing American Indian or Alaska Native students for life in their community, tribe, 

group, pueblo or village?

Four administrators (out of 11 respondents) noted that their school curricula did not prepare students 

for life in their tribe. Six administrators felt that general school curricula and teacher interactions 

prepared students, while two administrators discussed how students take part in community service 

activities or otherwise engaged with the community. One administrator’s response proved difficult to 

code: 

“We're going to Reno for about two weeks in March to a summit that I've gone to for about the last five 

years. When I read this though, I think in like history or in language arts, are they getting stories that 

belong to them or to the American Indian culture? I still think that has a long way to go. That's my goal, 

is that why wouldn't we take time to talk about this specific tribe in this area and their history when that 

would make them have ownership, be a part of ... validate them.”

Probe 5: What does the word “pueblo” mean to you?

Six administrators (out of nine respondents) defined pueblos in terms of geographic location, 

particularly the southwest. Four administrators noted that the term means “village” or “community,” 

and three highlighted that the term was specifically Native in origin and reference.

Probe 6: In question b., what does the phrase “prepare students for the workforce” mean to you?

Of the 12 respondents, four defined the probe item as general preparation for the work place, two 

provided specific examples of how the school prepared students, and five discussed how student 

preparation included development of math, reading, speaking, reasoning, and social skills. One 

respondent’s response was difficult to code: 

“Well, not you're getting into personal beliefs of mine where I think, I'll just go back to begin with the 

end in mind ... I'm getting philosophical in here, but a lot of times we're not thinking about the end 

game and what it is that we want them to know and be able to do and I really dislike how sometimes we 

don't make anything apply to different situations. I don't know if you saw the visuals out there, did you 

see any of those out there?”

Probe 7: [Conditional probe: If the participant selects answer choice B, C, or D for question b., then 

ask the follow-up question]: Can you give me examples of how your school’s 8th grade curricula 

emphasize preparing American Indian or Alaska Native students for workforce?

Six of the 10 respondents reported that school courses and curricula prepared students for the 

workforce. Three respondents described the use of team work or a general discussion of student 

preparation when answering. One administrator’s response was difficult to code: 

“I work with Fredina Drye-Romero and she is the American Indian Consultant there in Reno. Her mom 

actually lives on the reservation. I went to a thing in Vegas the other day and I've gone on five different 

American Indian college tours in different states from Clark County with kids checking out colleges. But 

as far as this school, this is my first year here and already, like I said in there, it's probably more so than 
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 211

your average principle, we've had the student being in boarding school presentation and he came here 

taking the 7th grade US Nevada history teacher.”

Probe 8: In question c., what does the phrase “prepare students for high school and college” mean to 

you?

All 11 administrators who responded said that the probe item referred to the academic preparation of 

students for graduation. 

Probe 9: [Conditional probe: If the participant selects answer choice B, C, or D for question c., then ask 

the follow-up question]: Can you give me examples of how your school’s 8th grade curricula 

emphasizes preparing American Indian or Alaska Native students for high school and college?

Ten administrators responded to the probe item. Of these, six cited college preparation coursework or 

curricula, three cited the use of outside speakers or college prep programs, and one described general 

discussions that staff had with students about preparing for high school and college.

Probe 10: Which answer choice would you select for questions a. through c. if the curricula is focused 

on high school only?

Two of the 10 respondents found the question confusing. Three administrators reported that their 

answer would not change, while four stated that their answer would. One administrator’s response 

proved difficult to code: 

“I mean, you really have to let them know that you do have high expectations and they've got to do 

better than what they're doing in high school.”

Probe 11: Do you think the question is easy or difficult to answer? Why is it easy or difficult to 

answer?

Ten administrators responded to the probe item. Of these, five found the question easy to answer, 

while four found it difficult. One administrator responded that the question was in between (i.e., both 

easy and difficult to answer.) Four respondents explained that the question’s phrasing is specific to high 

school rather than middle or elementary school, while one commented on the complex and interrelated 

nature of the various elements within the question.

Solutions to Improve Item

Respondent: NA

Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)

Change: Several respondents noted that school curricula succeeds in preparing students for life in the 

community but fails in preparing them for life in their tribes. Consider separating this into two different 

questions, as it examines two different (though admittedly intertwined) aspects of a student life.
ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations Page 212

ETS Recommendation (05/02/2014)

Drop this item. 

Rationale: ETS recommends dropping this item as respondents indicated that life in their community is 

not necessarily the same as life in their tribes. Respondents indicated that the school curricula may 

succeed in preparing students for life in their community, for instance, but not necessarily life in their 

tribe. In addition, not all students may have a “tribal life” and may live in a community without other 

AI/AN families. Furthermore, sub-items B and C may be more applicable to higher grades, as expressed 

by some of the respondents, therefore they may not be appropriate for grade 8.