# Appendix F

NIES Cognitive Interview ReportsListening

.

Learning

. Leading

. 

# NAEP Item Development (ID)

Survey Questionnaires

Cognitive Interview Study Report and Recommendations for Operational

2015 National Indian Education Study
Student, Teacher, and School Questionnaires

Grades 4 and 8

Deliverable in response to ID Task 3.2.3

Submitted: May 2 , 2013Prepared by Kauffman & Associates, Inc. 

# Table of Contents

Cognitive Interviews Report for NIES CogLab

................................ ................................ ............................... 3

Sample

................................ ................................ ................................ ................................ ........................... 3

Procedures

................................ ................................ ................................ ................................ .................... 3

Analytic Approach

................................ ................................ ................................ ................................ ......... 4
Survey Instrument: Student Grade 4/Sequence I

................................ ................................ ......................... 5
Survey Instrument: Student Grade 4/Sequence II

................................ ................................ ...................... 33
Survey Instrument: Student Grade 8/Sequence I

................................ ................................ ....................... 57
Survey Instrument: Student Grade 8/Sequence II

................................ ................................ ...................... 83
Survey Instrument: Teacher Grade 4/Sequence I

................................ ................................ ..................... 110
Survey Instrument: Teacher Grade 8/Sequence I

................................ ................................ ..................... 133
Survey Instrument: Administrator Grade 4/Sequence I

................................ ................................ ........... 165
Survey Instrument: Administrator Grade 8/Sequence I

................................ ................................ ........... 187

Note: This report was prepared by Kauffman & Associates, Inc. (submitted to ETS on April 28, 2014). For each item, an ETS recommendation was added to the report based on an evaluation of the findings summarized by Kauffman. ETS recommendations also consider overall student burden, in addition to item performance in cognitive interviews. ETS recommendations are highlighted in yellow. For more detailed information on suggested item wordings and overall item counts, please see R oD.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 3 

# Cognitive Interviews Report for NIES CogLab

Date of Report: April 28, 2014

Interview Dates: December 11, 2013

– March 18, 2014

Location of Interviews: Arizona, California, Idaho, Maryland, Minnesota, Mississippi, Nevada, New Mexico, North Carolina , Oklahoma, Oregon, South Dakota, and Washington. 

# Sample

Kauffman & Associates, Inc. (KAI) recruited a total of 159 individuals for participation in cognitive interviews during which participants completed the background questions for the 2015 National Ind ian Education Study (NIES). The demographics of the participants were as follows:

Table 1 : Sample Population Demographics

Respondent Category Total

4 th Grade Students 49

8 th Grade Students 44

4 th Grade Teachers 22

8 th Grade Teachers 20

4 th Grade Administrators 12

8 th Grade Administrators 12



# Procedures

As part of the cog nitive interview recruitment process, KAI attended five conferences with representation from across the United States from October to December 2013. KAI’s outreach strategy included sending emails to conferences attendees who indicated an interest in participating in, or knowing more about, the cognitive labs after OMB approval had been granted and working through
these i nitial contacts to assess potential participants.

In addition to leveraging contacts made during conferences, KAI also engaged in direct outreach to directors of tribal education departments and Title VII Directors around the United States . Through

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 4 these contacts
, KAI secured the interest of schools who were asked a series of eligibility questions using the sampling guidelines agreed upon between the National Center for Education Statistics , KAI, and ETS. Schools were additionally asked to have consent for ms signed by parents or guardians prior to KAI interviewers arriving at the school.

At the end of the sessions

, KAI staff asked participants to reflect on their overall impressions of the cognitive interview

. KAI interviewers then thanked and remunerated the participant w ho w as then asked to sign a receipt for their incentive payment.

The cognitive interviews were scheduled for a duration of 60 minutes. The items for both 4 th and 8 th grade student questionnaire s were split into two sequences per grade leve l and administered to at least

20 students per sequence

. All 4 th grade teachers received the same sequence and all 8 th grade teachers received the same sequence. Further, all 4 th grade administrators responded to the same sequence and
all 8 th administrator s responded to the same sequence. 

# Analytic A pproach

This report presents findings directly associated with the 159 cognitive interviews conducted by KAI. The analysis is based upon both the responses given par interview participants as well as from the
ob servances from KAI interview staff regarding the comfort and demeanor of participants throughout the interview process.

The analysis of the data collected wa s qualitative in nature and therefore based on drawing out themes
from participant responses and c ounting frequency of response occurrence as well as reporting out on anomalous data. Due to the nature of qualitative data sometimes response rates equal more than the number of respondents, particularly if multiple themes were present in a single particip ant’s response. Further, several probe questions in the interview instruments were conditional so not all respondents answered; some respondents chose not to respond to certain question s ; and there are inevitable skipped questions at times where interviewe rs missed asking a question.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 5 

# Survey Instrument: Student Grade 4/Sequence I

Item 1

Reading: Two respondents had some trouble read ing the item. One misread “please specify” as “please speak clearly

, ” and another had trouble with the words “who

, ” “representatives

, ” and “specify.”

Understanding: Two respondents seemed to have trouble with “Alaska” ; one said they didn’t
understand what Alaska was, another said they didn’t understand what an Alaska Native was, and a third kept referring to Africa though they seemed to be referring to Alaska. A fourth respondent seemed confused by what “AI/AN history” meant.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Four student s out of 23 respondents seemed to misunderstand the question based on answers ranging from “I don’t know” to non sequitur responses to the question ; the other 19 clearly restated the question.

Probe 3: What does the phrase “American Indian or Alaska Native history” mean to you? Can you give me examples of “American Indian or Alaska Native history” that you have learned?

There was confusion for the 2 2 respondents around P robe 3 with one respondent speaking to issues in Africa (ostensibly meaning Alaska), and another asked what Alaska and history were. Answers that seemed out of place were from two respondents who answered “I don’t know

, ” one who respo nded “pretty much nothing

, ” and two others who made references to language as AI/AN history. The other 15 answers demonstrated a clear understanding of the question.

Probe 4: In answer choice B, what does the phrase “Family members” mean to you?

Of the 22 respondents for this question, most noted specific family members; several spoke to broader relationships of interdependence, for instance saying that the tribe was their family. Student #1 New/VH039733

Who taught you most of what you know about American Indian or Alaska Native history?

A. No one has taught me about American Indian or Alaska Native history.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 6 Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?

“ Elders ” was relatively easy for the 22 respondents to pinpoint and identify as people older than themselves or with more knowledge. “ Tribal representatives ” presented a bit of a challenge for most respondents who either ignored the question or said that they did not know what a tribal representative was. Many respondents also conflated the terms of “ tribal representative ” and “ elder, ” perhaps due to the fact that they were presented in one question together.

Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?

The 10 responses to this question were inconsistent and questionable with regard to their validity. One respondent felt that their father was a tribal representative as he worked at the college. Another noted
that t he i r grandfather was very involved in powwows, though his position was never clarified. Of the six respondents who replied in the affirmative to this question, only the two aforementioned respondents gave specifi c reasons as to why they considered their family members tribal representatives. Two indicated that they did not have family members who were tribal representatives.

Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the fol low

- up question]: Who is the person?
Only one respondent chose answer choice F and named their parents as the “someone else” who had taught them most of what they knew about AI/AN history.

Probe 8: What does the phrase “most of what you know” mean to you?

Out of 23 respondents , one said “nothing

, ” one responded “I don’t know,” and another said, “I don’t understand this question. ” All other (21) responses were appropriate and valid.

Probe 9: Which answer choice would you select if multiple people had taught you about American Indian or Alaska Native history? How would you decide which answer choice to select?
The validity of the answers proffered a lot of variety among the 18 respondents . Seven suggested they would choose a different answer entirely. None ap propriately responded to being asked how they would decide the best answer in a hypothetical situation, though five gave explanations as to why they chose their original answer.



## Solutions to Improve Item

Respondent: NA

Interviewer: Interviewers consistently noted that hypothetical questions (such as Probe 9) posed a problem for many respondents, and this was particularly true for the 4 th grade interviews.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally s eemed to understand and be comfortab le with answering the question.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 7 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 8 

## Item 3

Reading: One respondent misread “please specify” as “please speak clearly” ; another skipped the word

“ N ative” in Al aska Native.

Understanding: Only one student said that they did not know what the question was asking.

Changing Answers: Of 23 respondents, two chose multiple answer choices for the item.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Only one student out of 22 respondents reported that they did not know what the item was trying to
ask.

Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you? Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have learned?

Twenty of the 23 students gave valid responses to the questions. One student said that they did not know, another said that this question was the same as the last, and a third studen t referenced storytelling

— which could be considered an art form but might also indicate some conflagration with the previous question about history.

Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow

- up questions] : Do you have any family members who are also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?
This probe s eemed to be confusing for the five students who responded to this probe. Answers range d from simply stating one family member to mentioning several. One student recounted, “I don’t really know because when my grandma still has to take me to my other family members too.” Student #3 New/VH040030

Who taught you most of what you know about American Indian or Alaska Native arts and
crafts?

A. No one has taught me about American Indian or Alaska Native arts and crafts.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): ___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 9 Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who are also tribal represen tatives? If yes, do you first think of them as your family members or as tribal representatives?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents seemed to understand the item question and answer it appropriately.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No iss ues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 10 

## Item 8

Reading: NA

Understanding: Given the inconsistency of responses across the various probes, it would seem that most respondents did not truly understand what this question was asking as there was no clear explanati on for what respondents interpreted “systems of counting” to mean.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Two out of 23 respondents interpreted the question as having to do with whether or not one could count in an AI/AN language. Fourteen students essentially repeated the question, though one thought that perhaps the question had to do with the invention of computers. Seven responden ts essentially reported that they did not know what the question was asking ; one of the seven did not even choose an answer for the item due to not understanding.

Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you ? Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have learned?
Eight students of 22 respondents indicated that they did not know what AI/AN “ systems of counting ” meant. Four thought it had to do with whether or not one could count in a N ative language. Three felt it had to do with how AI/AN people kept track of their populations. And four students essentially repeated the words of the question. Other responses varied in their guesses to the meaning.

Probe 4: [Cond itional probe: If the participant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who are also tribal rep resentatives? If yes, do you first think of them as your family members or as tribal representatives?

Of the 13 respondents who answered, five students said they did not know, two thought it had to do with language, one thought it had to do with populations, one referenced basket weaving, at least two answers were confusing regarding what the respondent was tryi ng to express, one respondent spoke about technology 50 years ago, and one respondent talked about their teacher having them count on
their hands. Student #8 New/VH040093

How much do you know about American Indian or Alaska Native systems of counting?

A. Nothing

B. A little

C. Some

D. A lot

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 11 Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow

- up question]: W ho is the person?

All 20 respondents who provided answers found the choices clear. One student noted that although they found the answers clear, A was the exception, saying “ I don’t know that if anybody knows or not.”



## Solutions to Improve Item

Respondent: NA

Interviewer: Through team discussions, this question came up often as one that was confusing both to students as well as to interview staff. No one seemed to be clear about what AI/AN systems of counting was referring to, something that should be clarified if this question is to be used in the 2015 NIES study.



## Initial KAI Recommendation (Keep/Change/Discard)

Discard: If there is not a clear definition of what systems of counting entails

, then the recommendation is to drop this question.



## ETS Recomme ndation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS recommends revising the phrase to "ways of counting".

This recommendation is aligned with the similar grade 4 item (VH040103).

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 12 

## Item 9

Reading: NA

Understanding: It is hard to say how well students truly understood the quest ion as many of them repeated the question. Some claimed they did not know what the question was trying to ask, some thought it had to do with language, others thought it was referencing AI/AN history or culture, others thought it referred to general teachi ng (not specific to a particular topic), and one thought it was the same as the previous question.

Changing Answers: No one changed their answers, but one student chose two answers.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Two out of 23 responding students said that they did not know what the question was asking, 11 essentially parroted back the question to the interviewer, two students felt that it had to do with being
able to count in a N ative language, four though t the question was referring to AI/AN culture or history, three seemed to think that the question had to do with who had taught them the most (did not seem specific to AI/ANs), and one student asked if this was a repeat of the previous question.

Probe 3: [Conditional probe: If the participant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?
Only one student was eligible to answer this conditional probe and they named their grandmother and uncle as tribal representatives but did not offer what their positions in the tribe were.

Probe 4: [Conditional probe: If the participant selects answer ch oice F, then ask the follow

- up question]: Who is the person? Student #9 New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of counting?

A. No one has taught me about American Indian or Alaska Native systems of counting.

B. Family members

C. Friends

D. Teacher s

E. Tribal representatives or elders

F. Someone else (please specify):

___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 13 Only two students were eligible to answer this conditional probe. The first said “the people that come to our school and teach us about it. I have no idea what they’re called. Most of them are da ncers.” The other replied that the “someone else” they were thinking of was their grandma.



## Solutions to Improve Item

Respondent : NA

Interviewer: Again, a specification about what is meant by “systems of counting” would be critical in making the meaning of this question clearer.



## Initial KAI Recommendation (Keep/Change/Discard)

Discard: G iven the current confusion about the meaning of the phrase, it is the suggestion of the interviewers to discard the question.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS recommends revising the phrase to "ways of c ounting".

This recommendation is aligned with the similar grade 4 item (VH040093).

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 14 

## Item 10

Reading: One student misread “systems” as “single.”

Understanding: Six respondents were clearly confused about what the question was trying to ask and either said that they did not kno w, or gave responses that did not make sense in the context of the question.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Responses varied among the 23 respondents , one said they did not know, one sa id “no” and 21 others repeated sections of the question but did not seem to grasp what the question was trying to elicit.

Probe 3: What does the phrase “American Indian or Alaska Native symbols and designs” mean to you?

Answers from the 23 respondents var ied greatly to this question. Only one respondent mentioned math in their response. Other answers ranged from punctuation to arts and crafts. Five students expressed not knowing what the question was trying to elicit, and the rest gave answers that have ve ry little validity as they were widely disparate in their interpretation of what was meant by “ symbols and designs, ” and as noted before, only one student tied their answer back to math.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, the n ask the follow

- up question]: Can you give me examples of “American Indian or Alaska Native symbols and designs” that you know about?

Only three of the 13 respondents referenced math, one noting that they were unsure how symbols and designs were used in math but did point out how they may be used in decoration or communication; and the other giving the following answer : “American Indian, basically my whole class, we just know that when you write the symbols in math that if you’re from somewhere else, it’s a new kid you don’t know and they could be from anywhere. If they don’t know how to do their Indian math that there would be a whole class and they would have to decide how would you write the math symbols down.” Six gave examples ref erring to arts, crafts, and modes of communication, one thought the question had to do with punctuation, one reference d animals, and four claim ed that they did not know. Student #10 New/VH040098

How much do you know about American Indian or Alaska Native symbols and designs used in
math?

A. Nothing

B. A little

C. Some

D. A lot

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 15 Probe 5: Do you think that answer choices A through D are clear? If not, can you give me examples of
other answer choices that would be more clear?

Eighteen of 19 responden ts felt the answers were clear ; one felt that they weren’t and gave the following explanation : “No. If they make them like, the X for times, the plus for plus or add, and subtract, a minus. A minus with two on top and on bottom of it for division. That's how we do it in English.”



## Solutions to Improve Item

Respondent: One respondent made the following suggestion: “ I think they could’ve asked it more
clearer because it’s not all just used in math, it’s used in different stuff, so maybe they could have just said, ‘ How much do you know about American Indian or Alaska N ative symbols and designs used in anything ?’ maybe

. ”

Interviewer: Perhaps the category of symbols and designs c ould be expanded beyond math as most students seemed to think about symbols and designs as decoration or communication.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: Given how most students interpreted the question having to do with arts, crafts, or forms of communication, perhaps the question could ask about that. The relevance to math made little sense to respondents and if this is the only purpose of the question, it is the suggestion of the interviewer team that it be removed from the study.



## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS agrees with KAI that this item should be dropped as the specific relevance to math made little sense to responde nts.
This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 16 

## Item 11

Reading: NA

Understanding: Three respondents were noted as having difficulty understanding the item due to their responses; one seemed to be equating symbols and designs with punctuation, one thought the question referred to history, another felt the question had to d o with whether or not American Indians or Alaska Natives had taught them more about symbols and designs in math.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One student out of 22 respondents referenced using math in a N ative language, one said they did not know, and 17 essentially repeated the question in slight different terms. Three gave unusual answers ranging from symbols and designs having to do with AI/AN history, or whether or not Ameri can Indians or Alaska Natives had taught them more, and another student made a reference to gestures.

Probe 3: [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tr ibal representatives? If yes, do you first think of them as your family members or as tribal representatives?

Only one student responded to this question on this item and specified their mother as a tribal representative in the following : “There's my mom. I don't know what she is. She likes to file papers and she's getting people's license or something like that. She's always busy all the time. Sometimes I can barely even see her.”

Probe 4: [Conditional probe: If the participant selects answer choice F, the n ask the follow

- up question]: Who is the person?

NA Student #11 New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and designs used in math?

A. No one has taught me about American Indian or Alas ka Native symbols and designs
used in math.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify):

___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 17 

## Solutions to Improve Item

Respondent: NA

Interviewer: Again, knowing the purpose and/or what this question is trying to ask would be very helpful.



## Initial KAI Recommendation (Keep/Change/Discard)

Discard : Given how the question currently stands, the recommendation is to discard it.



## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting the phrase "symbols an d designs", and the specific reference to math caused additional confusion.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 18 

## Item 12

Reading: Two respondents had trouble with the word “assemblies” ; of these two, one had trouble with
the word “traditions

, ” and the other had difficulty with the word “elders.”

Understa nding: Two respondents were reported by interviewers as having trouble understanding the question due to either difficulty with words in the question.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Six teen of 22 respondents repeated some variation of the item question, four respondents gave somewhat confusing answers such as “ H ave you ever met them ? ” and “Have I done everything I was told to do?” and an explanation about “stories from a long long time ago.” Additionally there was one “I
don’t know” and one non

- response.

Probe 3: What time frame were you thinking about when you answered this question?

Very little agreement or consistency in the 20 respondents’ answers to this question. Answers rarely overlapped and included responses such as, “one to two times a year,” “two to three days,”

“ December, ” “ four years, ” “ all school year, ” “ all years in school, ” “ five months, ” “ 5 minutes, ” “ a week, ”
and “ a semester .” T he most confounding was “When I was a ba by. Yeah. It’s so dark.”

Probe 4: What does the phrase “assemblies or presentations” mean to you? Can you give me examples of “assemblies or presentations”?

Half (11) of the 22 respon dents gave broad definitions of what programs, presentations

, and assemblies
consisted of

, and about six of the students cited specific examples of assemblies or presentations they had been to as a way to explain what assemblies and presentations meant to them. Only two of the respondents said that they did not know , and seemingly non sequitur answers were limited to three:

“I don't think that much, I don't know that much. [Interviewer prompts] About learning traditions and cultures and the way we live in right now and how did they create it before the date of 2014 out of
1614.”

“ Like programs. [Confusion over what we were trying to ask, mentioned singing in Chapel Hill and having activities at Grandma's house.] ” Student #12 Revised/VH040119

During 4 th grade, have you attended assemblies or presentations in your school given by elders or leaders from an American Indian or Alaska Native tribe or group about traditions and cultures (ways of life, customs)?

A. Yes, once or twice

B. Yes, three or more times

C. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 19 “ It means that it's kind of telling me a story. ”

Probe 5: What does the phrase “elders or leaders from an American Indian or Alaska Native tribe or group” mean to you? Can you give me examples of “elders or leaders”?

The 21 responses to this probe yielded the following kinds of answers. About 10 respondents gave either specific examples or provided explanatio ns about what they thought made an elder or leader, four respondents mentioned specific family members, three spoke about people coming to give presentations, three said that they did not know, and two gave answers difficult to qualify.

Probe 6: [Condition al probe: If the participant selects answer choice A or B, then ask the follow

- up question]: How many times have you attended these assemblies or presentations in your school?

Given the responses to Probe 4 it is difficult to say the precise time frame tha t respondents were thinking about for this question. The 15 who provided responses broke into two broad categories. Eight students mentioned attending assemblies or presentations three times or more, and four said that they
had attended either one or two t imes.

Probe 7: [Conditional probe: If the participant selects answer choice C, then ask the follow

- up questions]: Is the reason you have not attended assemblies or presentations during 4th grade because they are not available? If not, please explain why y ou have not attended assemblies or presentations during 4th grade.

Of the seven students responding , three said that assemblies were not available at their school or would be occurring later on in their schooling, two respondents said they had been to asse mblies but there had not been any elders or leaders present, and two respondents gave non sequitur answers indicating that they did not fully understand the question.

Probe 8: Do you think that answer choices A and B are clear? If not, can you give me exa mples of
other answer choices that would be more clear?

Fourteen of 17 respondents felt that the answer choices were clear, one said that they did not know,
and two said that the answer choices were not clear.



## Solutions to Improve Item

Respondent: One resp ondent felt that there should be another answer choice to reflect whether or not someone had attended more than three times. Another felt that more specific kinds of answer choices
would be better.

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Change: There seemed to be a lot of variance in answers to the probes related to this item. Perhaps bolding “4th grade” would help to ensure that students are answering for that time period. From responses to Probe 4 there seemed to be some agreement that assembl ies required people coming together to see a presentation that someone else is giving

, whereas “ presentation ” seemed to be interpreted as something that students might do in a classroom setting. This may indicate that using the term “presentation” may be c onfusing if the item seeks to parse out whether or not students are

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 20 attending assemblies (which would likely involve dancers, etc

. ) or presentations given by leaders/elders. Another point is the use of “ leaders/elders ” as they may not be the ones leading p resentations or assemblies about AI/AN cultures ( for instance

, given the example of dancers coming to the schools )

. 

## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS agrees with KAI that this item should be dropped given the large degree of variance in the probe responses. Also, there seems to be a cognitive overload with this item for a 4th grader.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 21 

## Item 17

Reading: One student misread the word “culture.”

Understanding: One student seems to be interpreting AI and AN as two separate issues or questions
within the item.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Seventeen of 22 respondents repeated some version of the item question, two interpreted the question as asking them what they knew, two simply mentioned AI/AN culture, and one gave the following response, “You know what you're doing about yourself that you know is explaining about you.”

Probe 3: What does the phrase “when my teacher talks about” mean to you?

Sixteen of 22 respon dents made reference to their teachers talking or teaching; within this group the topics were either not specified or were seen as revolving around AI/ANs (five students mentioned this), history (two students mentioned this ), art (one student), or bullying (one student). Three students said that they did not know.

Probe 4: Which answer choice would you select if your teacher had not talked about American Indian or Alaska Native history or culture? Why would you select this answer choice?

Fifteen of 22 respondents felt that they would choose a different answer if their teacher had not talked about AI/AN history or culture in class, many of them citing that if their teacher had not talked about it then they probably would not know about the topic. Seven students fel t that their answer would stay
the same.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents overall seemed to understand what the item was asking. Student #17 Revised/VH040212

When my teacher talks about American Indian or Alaska Native history or culture, I try to read
more abou t it.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 22 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 23 

## Item 19

Reading: NA

Understanding: Respondents seemed to have a firm grasp on what this question was trying to ask.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Ei ghteen out of 21 respondents provided responses that were in keeping with what the question was asking with some variation on what “ being the best ” meant to them

— from getting good grades to being the best reader to being well

- behaved. Three respondents int erestingly gave responses with a negative take on being one of the best in the class

, referencing greed, showing off, and making others feel bad
about themselves.

Probe 3: What does the phrase “one of the best students” mean to you?

This probe elicited in teresting responses from the 22 respon dents who provided answers. Eight referenced having good behavior, six students used the word “best” to talk about what being “ one of
the best students ” meant to them, six mentioned knowing more than others or being sm art, five talked about grades, three simply noted being “better” than others, two discussed popularity, one said it meant being fast with schoolwork, and one said that they did not know.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Question seem ed easily understood and wa s being interpreted as intended by the majority of
students.



## ETS Recommendation (05/02/2014)

Keep this item. Student #19 New/VH040276

I want to be one of the best students in my class.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 24 This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 25 

## Item 20

Reading: NA

Understanding: All students seemed to understand the gist of the question, though two seemed to interpret “challenge” with negative connotations.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own wo rds, what the question is asking?

Answers to this question seemed fairly consistent. Ten of 21 respondents essentially repeated the question, two mentioned that “ being challenged ” meant answering hard questions, two noted that challenge meant not doing easy things, two specifically mentioned math, one student pointed to doing independent work, and one noted learning new things. Only two responses seemed a little out of the

norm ; on e student stated “It’s telling me they like challenges

, ” and the other referenced staring

contests.

Probe 3: What does the phrase “being challenged” mean to you? Can you give me examples of you being challenged in your classes?

Thirteen of 22 respondents noted that “being challenged” to them had to do with being pushed to do harder work, usually by their teachers. Three specifically mentioned math in their responses, two talked about tests, two referenced competitions (one of these respondents was the one who spoke of staring contests), and two talked about tests.



## Solutions to Improve Item

Respondent: NA

Interviewer: One of the interviewers felt that there could be more emphasis on this question being
related to schoolwork

— they had the interview with the s tudent who used staring contests as an example of challenges.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents understood the question and interpreted as intended. Student #20 New/VH040379

I enjoy being challenged in my classes.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 26 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 27 

## Item 22

Reading: NA

Understanding: Most respondents seemed to understand that the item wanted to know if they were getting help on their schoolwork ; the piece about it being “someone else who lives in your community or is a friend of your family” was not necessarily interpreted as it should have been. Students seemed to latch onto different words in the bolded part of the statement like friends, famil y, and community, but did not necessarily tie them together.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Respondents generally understood that the question had to do with receiving help on one’s homework
with 12 out of 22 respond ents referencing that piece of the question specifically. Nine mentioned family providing help, five referred to friends helping, fi ve mentioned community members in their responses, one spoke of a tutor

, and one referenced a friend of the family. Three answers were somewhat confusing; one student mentioned their teacher providing examples, another seemed to think the question had to d o with them helping their friends as well as being helped by their own family, and one student interpreted the question as having to do with how much time they spent on homework.

Probe 3: What does the phrase “someone else who lives in your community or i s a friend of your family” mean to you?

Nine of 22 respondents spoke about friends of the family as their contemporaries in age, rather than as
an adult who was a friend of the family. Additionally nine responses referenced family specifically from
mothers /fathers, brothers/sisters, aunts/uncles, grandparents

, and cousins. At times the cousins were noted also as friends. Of the seven responses directly mentioning the idea of friends of the family, friends were often noted as of contemporary age with the res pondent and cousins were also often mentioned. Three students said that they did not know what was meant by “someone else who lives in your community or is a friend of the family.” Student #22 Trend/VC759291

How often does someone else who lives in your community or is a friend of your family help you with your schoolwork? For example, they might help you to study for a test, help you with a school project, or go over your homework with you.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a week

D. Every day or almost every day

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 28 Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can you give me examples of how these people have helped you with your schoolwork? Who is this person?

Eleven out of 20 responding students mentioned family members from parents to grandparents and everyone in betwe en, and six students spoke of friends helping them

— most indicated that these friends were of the same age, though some were slightly ambiguous. Two students mentioned tutors, and three gave unclassifiable answers.



## Solutions to Improve Item

Respondent: NA

I nterviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Change: The wording of the bolded section seems to be causing confusion for many respondents who do not seem to be interpreting the question as intended. We s uggest revising the question to as k specifically about adults , friends, or tutors in the community, or asking whether or not family is involved. Friend of the family/someone else who lives in your community seems to involve a nuanced understanding that is not being exhibited by the majorit y of the respondents to this question.



## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS recommends dropping this item as it seems to overlap with the previous three trend items (VC759288, VC759289, VC75929). Having four sequential items about who is helping the student with his/her schoolwork is excessive; therefore this item can be dropp ed to make room for a new item(s). Asking about "someone else who lives in your community/friend of your family" seems to involve a nuanced understanding that is not being exhibited by the majority of the respondents to this question.

The similar grade 8 item VC996640 is also proposed to be dropped.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 29 

## Item 23

Reading: NA

Understanding: The grand majority of students seemed to understand what this item was asking.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eighteen out of 22 respondents interpreted the question correctly, one stu dent specifically mentioned
math homework

, and another mentioned helping their siblings with school work. Only one of the respondents seemed to be a bit confused about the angle of the question and seemed to think it was asking them about whether other stu dents were helping them with course work.

Probe 3: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up questions]: Can you give me examples of how you have helped other students with their schoolwork?
Are the ot her students that you have helped with their schoolwork in 4th grade? If not, what grade are the students that you have helped with their schoolwork in?

Ten of 19 respondents mentioned helping “friends” with homework, nine noted that those they were helping were in the 4th grade, six respondents pointed to helping family members such as siblings and cousins with homework, and five respondents pointed to working with students i n other grades. Two respondents gave C or D as answers, but when asked for examples

, they responded “no.”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA Student #23 New/VH040389

How often do you help other students from your school with their schoolwork? For example, you might help other students to study for a test, help them with a school project , or go over their homework with them.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a week

D. Every day or almost every day

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 30 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep/Change: It seems that there may be some room for confusion on this item depending on what information is being sought. If the question is simply to capture whether or not students are helping others with classwork, it is fine as is. If the question sp ecifically wants to know about “ other studen ts from your school

, ” that may need to be highlighted. Generally, the question is fine as is with students responding consistently about helping others.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No i ssues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 31 

## Item 25

Reading: NA

Understanding: The majority of respondents seemed to grasp the question being posed to them.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eleven of 21 responding students noted the importance of education and mentioned post

- elementary education in their explanation of what the question was asking, seven students essentially repeated the question using slightly different words, one noted hard work, one mentioned money, and one said that they did not know what the question was asking.

Probe 3 : What does the phrase “tryin g hard in school” mean to you? Can you give me examples of what “trying hard in school” would be?

Fifteen of 22 responding st udents noted that “trying hard in school” meant studying or practicing or doing your best. Seven mentioned doing well on tests in their responses, four mentioned getting good grades, three mentioned college, and one mentioned getting help when you need it.

Probe 4: What does the phr ase “a good life” mean to you? Can you give me examples of what “a good
life” would look like?

Answers to this probe varied in terms of what themes presented themselves. For instance, nine out of 22 responding students mentione d jobs or “good jobs” when discussing what it meant to have a good life, five mentioned having a family or being able to help family, four mentioned having money, four noted having knowledge or knowing a lot, four talked about having a house, three mention ed good health (ranging from eating well to being drug free), two talked about happiness in general terms (though one was specific about that having to do with ponies in a mansion), one said they thought it had to do with having good manners, and one said they did not know.

Probe 5: What does the phrase “when I grow up” mean to you? Can you give me examples of what being “grown up” means to you? Student #25 New/VH040394

Trying hard in school will help me live a good life when I grow up.

A. Strongly disagree

B. Disagree

C. Agree

D. Strongly agree

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 32 Responses to this probe varied greatly in terms of what students saw as being “grown up.” Seven of 22 respondin g students mentioned having a job, five said they thought they would be grown up when they were in their 20s, five referenced education (often referring to college), five said it would be when they were independent (often referencing not needing parents an y more), three said when they were bigger or taller, three mentioned having a house, two said it would be in junior high, two said when they had a family of their own, one mentioned being clean, one mentioned having a car, one spoke of good penmanship

, and one said they did not know. One of the respondents summed up being a grown up with the following, “They go to work every day. They go to dinner. They go to bed. They wake up, and then they go back to work. That’s it.”



## Solutions to Improve Item

Responde nt: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: All respondents seemed to understand the question as it was intended.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 33 

# Survey Instrument: Student Grade 4/Sequence II

Item 2

Reading: One of the students who did segments of both sequence 1 and sequence 2 initially thought

t he y had already read this question.

Understanding: Only one student was reported as not understanding the item and having trouble with the probes as well.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- six out of 30 responding students essenti ally repeated the question ; four indicated that they did not understand or did not know what the question was asking.

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions” mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you
have learned?

All of the 31 responding students asked this question gave a response, 10 of them mentioned traditions in their answers ranging from smudging and tobacco use

, and one student noted Chris tmas as a tradition. Seven students talked about powwows or dancing, four mentioned crafts, three expressed that they did not know what “Alaska Natives” meant, and other answers involved food, housing, history, language

, and hunting. Six students indicated that they did not know what the question was trying to
ask.

Probe 4 : In answer choice B, what does the phrase “Family members” mean to you?

Of the 30 respondents to this question (one was inaudible)

, 18 students named different members of their family ( for instance

, parents or siblings or grandparents ) . Seven students simply said “family” when Student #2 New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions (way of life, customs)?

A. No one has taught me about American Indian or Alaska Native traditions.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please spec ify): ___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 34 asked what “family members” meant to them, four students spoke about relationships of teaching and learning when talking about what it meant to them

, and two menti oned being loved or cheered up.

Probe 5: In answer choice E, what does the phrase “Tribal represent atives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?

T he 30 responses to this question varied a good bit as it as ks about both tribal representatives and elders. Thirteen students named someone either specifically or in terms of who in their family the y considered to be an elder or tribal representative. These answers ranged from grandparents to teachers to family friends. Nine students noted that tribal representatives or elders were “old people,” six
mention ed teachers

— either specific teachers or just people who teach younger people about their culture. Five students said that they were people who represented the tribe, four said they did not know, two simply responded “no,” and two respondents had difficult

- to

- categorize answers

— “that means they help m e get yeast and stuff and all that.”

Probe 6: [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who are also tribal repre sentatives? If yes, do you first think of them as your family members or as tribal representatives?

The validity of responses to this question vary greatly given that very few students seemed particularly sure of what a tribal representative was. Of the 16 respondents who provided answers, eight indicated
th at they did have a family member who was a tribal representative (often giving answers like “family members”), four indicated they did not, two said maybe, and one said they did not know, and an other
student gave a hard

- to

- characterize answer

— one said “eld ers.”

Probe 7: [Conditional probe: If the participant selects answer choice F, the n ask the follow

- up question]: Who is the person?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: Separate out elders and tribal representatives in probes to better determine how students understand each of these categories.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It would seem that the majority of the students asked this question unde rstood what the question wanted to know about who had taught them what they knew about AI / AN traditions.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 35 

## Item 4

Reading: No difficulties reading the item.

Understanding: A few of the res pondents seemed to have some issues with fully understanding the item without some redirection from interviewers. One student seemed very confused by the term “ Alaska Native, ” two others seemed to have a hard time understanding the broader question and got bogged down in the idea of specific vocabulary or attendance overall.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 31 responding students mention ed language in their restatement of the question, five students did not mention language but rather seemed to interpret the question as having to do with AI/AN culture, five seemed to be confused by the question

— one stating, “What time do I arrive at
school”

— and one student simply noted that they do not learn anything about AI/ANs in their school as they are one of five students of AI/AN heritage in the entire school.

Probe 3 : What does the phrase “American Indian or Alaska Native language” mean to you? Can you give me an example of “American Indian or Alaska Native language” that you can speak or know
about?

Nineteen of 31 students responding to this probe referenced themes having to do with how people spoke in the past ; some of these students even reference d language being spoken by elders in their community. Eight students mentioned specific AI/AN languages, three talked about being taught or learning other languages, two students said “no,” and one gave an answer having to do with their grandmother that st rayed from language and into horseback riding. Student #4 New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska Native language?

A. My school does not offer courses that are taught in an American Indian or Alaska Native language.

B. Never or hardly ever

C. Once or twice a month

D. Once or twice a week

E. Every day or almost every day

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 36 Probe 4: How many American Indian or Alaska Native languages do you speak?

Eleven of 31 responding students reported speaking at least one AI/AN language, nine said they spoke a little of one or two langu ages, nine said they did not speak any AI/AN languages, one student said they spoke two languages, and one said they did not know.

Probe 5: How would you answer this question if you speak more than one American Indian or Alaska Native language?

This probe drew varied answers from the 27 respondents . Five students indicated that they would change their answer to D . Five gave answers that were difficult to characterize from “Talk like America” to “If I could speak more than one American Indian languages, I w ould go lots of places” to “I speak Spanish, Chinese

— I say hello in Chinese. I can speak Italian.” Four said they did not know, and four gave
the answer B

, but at least two of these indicated they thought that meant “once or twice a week

. ” I t is
hard to sa y if this is a misunderstanding or misreading on the part of the students or if perhaps B was heard instead of D by the transcriptionist. Three students would change their answer to E , two would keep their answer as is, and one would change their answer to A (interestingly this participant gave D for their original answer and indicated that they do in fact speak Lakota).

Probe 6: What is the difference between answer choices A and B?

Fourteen out of 31 responding students gave responses that indicated that they had interpreted the
differences between A and B correctly. Six students gave responses that were difficult to characterize in that they were unfinished thoughts, or seemed to co nfuse what the question was asking. For instance, one student said, “One talks about the Alaska Native language and the other one is for attending class

. ”

A nother said “ That usually A , they usually don't teach you much about A . You are very like the entire language and what you eat and B is probably like you are never really learned anything about your
culture

, ” and another student had the categories backwards and said “Because B you don’t learn it and

A you do.” Four students said they didn’t know the diff erence between the two answers and four other students felt that the two answer were the same. Two students suggested that A referred to learning in
school and B could be outside of school.

Probe 7: If there were no courses offered in your school that are taught in an American Indian or Alaska Native language, which answer choice would you select?

Thirteen of 31 responding students said they would choose A ; two indicated they were choosing this because their school did not currently have classes in an AI/ AN language. Nine said they would choose B

. Four said they would choose D , one said they would choose C , one said they would choose E

, and two indicated they did not know. It seems clear that many of the students did not understand what this question was t rying to ask.



## Solutions to Improve Item

Respondent: NA

Interviewer: Hypothetical questions seemed to confuse many students. One actually said “ Again, that seems like another confusing question. I would think that I would answer the same thing but I don't. I

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 37 just feel like I don't want my answer to be the same. ” This led the interview team to question how often this might have been happening with other probes.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: This question generally seemed to cause some confusion for respondents. The item seemed to make sense to students, but subsequent probes revealed that there was some confusion around th e difference between answers A and B . Some of this may be attributable to the difference in wording of the two questions

, and the description for A is much more detailed than that of the other answer
choices. Our r ecommendation is to change to be similar t o B through E and to highlight in the item the fact that the question specifically is asking about classes at school (keeping in mind that many students who may take classes may do so outside of school).



## ETS Recommendation (05/02/2014)

Revise this item.

R ationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer choice “A”, ETS recommends changing the word “courses ” to the word “classes” to be consistent with
the item stem.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 38 

## Item 5

Reading: NA

Understanding: Some respondents seemed to think that the question had to do with teaching or learning the language, and even of those who restated the question in a way that seemed to capture the intent, there seemed to be a wide interpretation of what the item was actu ally trying to capture.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- one of the 31 respondents provided a response that was on track with what the question was trying to ask, but the variati ons within these answers was interesting . S ome students essentially

restat ed the question with some explanation

, and others said things like “Can you speak American Indian language.” Three students seemed to think that the question had to do with how much they learn or “how you teach it.” Another three gave answers that were difficult to categorize, ranging from “It means how do you talk when you try to talk like them” (ostensibly referring to AI/ANs) to “There’s three different ones and then you pick which one you think you’re at” (likely referring to the fact that they are being asked to choose between three different answer choices).

Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?

Twenty

- five students out of 30 respondents gave answers indicating that they had correctly interpreted the phrase “I can speak a few words or phrases” as intended. Three students said that they did not know, and one students said, “It means you can speak a few words in English and stuff. Alaska.”

Probe 4 : In answer choice C, what does the phrase “I can speak well” mean to you?

Of the 29 students who responded to this question, 26 seemed to understand the gist behind what “I can speak well” was trying to elicit, though this occurred in different ways with some students interpreting it as a generic speaking well or fluently whereas others were more nuanced in th eir descriptions

, drawing on breadth of vocabulary and pronunciation. Only one student indicated that they did not know what the phrase meant.

Probe 5: Do you think that additional answer choices s hould be added between B and C? If yes, can you give me exa mples of what those answer choices would be? Student #5 New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?

A. I cannot speak an American Indian or Alaska Native language.

B. I can speak a few words or phrases.

C. I can speak well.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 39 Sixteen out of 30 responding students said that they did not think any additional answer choices should be added. One said that they did not know, and 12 said that they did think that there was a need for

more a nswer choices. Of these 12

, the answers generally focused on different nuances that students felt were missing, though one suggested both B and C should be the same, indicating that they did not truly understand the probe. Additionally, some of the “yes” r esponses restated the options available in different words, suggesting that they may have been confused by what the probe was trying to capture. A few examples of the other “yes” answers are as follows:

“ I think they could put a little more in it, like, ‘ I can speak medium well, ’ like in the middle of little and well. And like, ’ I can barely do some words. ’ "

“ I can speak mostly a couple of words. I can speak them very well, but sometimes I forget how they
sound. ”

“ Yeah. If you could only speak few words yo u can’t speak a lot, but you can speak a few, but if you can speak well you can speak a lot. You can speak a few, but well. ”



## Solutions to Improve Item

Respondent: Might consider some of the answers given by students who thought that other answer
choices we re needed between B and C , though many of the examples do not seem to indicate a viable
additional choice.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that generally speaking, the majority of students responding understood what the question was trying to ask. If there were any change to be made

, it might be to stay consistent regarding phrasing in the different answer options.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 40 

## Item 6

Reading: NA

Understanding: Some o f the respondents seemed to have difficulty understanding the item based on
responses to P robe 1 , and at least two respondents felt initially that this was the same question as Item

5.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own word s, what the question is asking?

Twenty

- five of 29 respondents provided a rephrasing of the question that indicated that they understood the meaning of the question. One said they did not know, one referenced English (indicating that they may have been con fused about the item or probe), and one asked if the question was referring to one’s first language or a Native language.

Probe 3: [Conditional probe: If the participant selects answer choice B or C, the n ask the follow

- up question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?

Thirteen out of 19 respondents mentioned a specific language in their answers

. I n at least three of these

cases , the interviewer or data entry person noted what language the student was familiar with was based on answers to other questions if the student did not mention it themselves. One student said they did not know, and four gave difficult

- to

- categorize answers such as, “I can read not that much but I can read some” and “I can pr obably read a sentence or maybe a couple words that's written like on a board or on a book” and “I could tell you who I am”

— this last one may be referencing being able to introduce
oneself in a N ative language, but it is hard to tell.

Probe 4: In answer c hoice B, what does the phrase “I can read a few words or phrases” mean to you?

Twenty

- two of 27 respon dents (one was inaudible) responding felt that that answer choice B was referring to knowing a little bit or at least some words in a N ative language. Th ree said they did not know, and two gave the following answers: “ That you kind of see something, and that you’re doing good on something, that something is really hard to do” and “ B is saying that I can only read most of the
words in Choctaw or in Indian. ” Student #6 New/VH040067

How do you rate yourself in reading in an Americ an Indian or Alaska Native language?

A. I cannot read in an American Indian or Alaska Native language.

B. I can read a few words or phrases.

C. I can read well.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 41 Probe 5: In answer choice C, what does the phrase “I can read well” mean to you?

Of the 27 students who responded , 23 seemed to adequately understand what answer choice C was referring to, three said they did not know

, and one said, “That you see something really big because sometimes it's really hard to read.”

Probe 6: Do you think that additional answer choices should be added betwee n B and C? If yes, can you give me examples of what those answer choices would be?

Fo urteen of 22 respondents indicated that they thought the answer choices were fine as they were and eight felt there was a need for additional answer choices. Of the remaining respondents, nine answers varied greatly, some of which are included in the follo wing:

“ Like, if you can't, if you don't know if you can read American language, they should have another thing that says you don't know if you can read it. ”

“ I can read half of the words I know, but I can read a little better than that. Like, I almost can read really
well. ”

“ I think that there should be, I could read a little words, like maybe like 10 or five words because phrases and words don’t go together really well sometimes. ”

“ Probably, like it should go, ‘ I can read a few phrases, ’ and then between them, it should say, ‘ I can read two or three paragraphs, ’ and then, ‘ I can read well. ’ "



## Solutions to Improve Item

Respondent: Some of the suggestions for adding additional answer choices could be considered.

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: M ost of the students given this question seemed to understand the item as it was. One suggestion may be to keep phrasing consistent across answer choices.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed wel l in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 42 

## Item 7

Reading: NA

Understanding: Some respondents thought that this was a repeat question when they read it.

Changing Answers: Not within the question, but a respondent who previously had noted that they did
not know w hat a tribal representative was indicated during this item that they had family members who were tribal representatives.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- four of 31 respondents provided answers consistent with what the item was asking. Two students did not mention language but instead seemed to think the question was referring to AI/ANs in general, and four gave answers that were difficult to characterize:

“If the teachers are d oing good in the school and which they are, they're doing an excellent job I think.”

“ B, C

, and E . For the first one.”

“I think it’s the same as number one.”

“ Someone who you know, like a friend or a teacher, or your mom or sister, for example they could all teach me but one that teaches me the best and the most. They teach me important stuff and good stuff
about them. ”

Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?

All 29 responses to this probe were too va ried to categorize, with the exception of “I don’t know

, ” which occurred five times. Some students thought the question had to do with whether or not they could speak a language, who taught them, how well they spoke it, etc

. There was v ery little validity with regard to answers. Student #7 New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?

A. No one has taught m e about an American Indian or Alaska Native language.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): _____________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 43 Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska Native language”?

Nine out of 27 respondents indicated that they did not know, five mentioned specific languages, five said that they could not give an example, three gave examples indicating they were thinking about vocabulary, three indicated that they considered counting in an AI/AN language as an example, and two suggested that being able to speak it was “part of an American Indian or Alaska Native language.” As with Probe 3, the phrasing of this question leaves a lot up for interpretation and the meaning seems not to be clear to respondents.

Probe 5: [Conditional probe: If the partic ipant selects answer choice B, then ask the follo w - up questions]: Do you have any family members who are also tribal represe ntatives? If yes, do you first think of them as your family members or as tribal representatives?

There were nine respondents to this question. Eight indicated that they did have family members who were also tribal representatives, and one said they did not. This question was skipped for several respondents as the question had been asked and answered in an earlier probe.

Probe 6: [Conditional probe: If the participant selects answer choice F, the n ask the follow

- up question]: Who is the person?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: Revise or remove P robes 3 and 4 as they seem to confuse students

, and answers given have questionable validity. Additionally, the question about tribal representatives seems in general to draw unreliable responses from students who do not seem to be sure of what a tribal representative is or what such positions consist of. We r ecommend either changing the probe to more deeply explore what students define as a tribal representative or remove the probe.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students seemed to easily understand what the que stion was asking. We

r ecommend keeping phrasing consistent throughout answer options.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 44 

## Item 13

Reading: NA

Understanding: Some students did not interpret the item only in terms of school activities but also brought in times that they may have gone on field trips with non

- school organizations or families

. O ne student interpreted the question as having to do with funding. Additionally, several students seemed to have a hard time only answering for what they had done during the 4 th grade.

Changing Answers: Two students changed their answers ; one could not ex plain why they chose to change the answer

, and the other switched their answer from B to C after some consideration.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- seven of 30 respondents answered the question corr ectly in the sense that they appropriately identified the main components of the item, though some bordered on essentially repeating the question. Two students gave difficult

- to

- categorize answers such as

, “Have you seen or they put up in
museums what it d oes that they think,” and

, “If you knew a little bit more about American Indian or Alaska Native people.” One student said they did not know and another seemed to think the question had to do with funding for field trips.

Probe 3: [Conditional probe: If the participant selects answer choice A, the n ask the follow

- up question]: What types of places have you attended to learn about American Indian or Alaska Native people?

Seven students out of 13 respondents noted that they had visited museums; nine mentio ned field trips
to historic sites

, powwows , hatcheries

, or missions. There may be some issues regarding the validity of
these answers as one student noted that the field trips they had been on had not been “traditional” per se, and there may be some questi on as to the time frame from which students were drawing upon to answer the question.

Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow

- up question]: How do you know that other 4th

- grade students attended, or that activities were not
offered to 4th

- grade students?

Seven out of 17 students responding said that field trips were either not offered or that they had not been told about them, three responded that all the 4 th graders go on field trips together, one mentioned Student #13 Trend/VC759216

During 4th grade, have you attended school field trips to museums, traditional villages, or other places to learn about American Indian or Alaska Native people?

A. Yes

B. No, but other 4th

- grade students did.

C. Not of fered to 4th

- grade students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 45 that they needed a permission slip

— indicating that perhaps they did not fully understand the question as it was asked. There were four responses that were hard to make sense of as they were non sequitur answers ranging from having to do with gett ing sick on Halloween, an Ojibwe reading group, and not wanting to go on a field trip. Some of the confusion related to answering the question may be related to having two questions presented as one as part of this probe.

Probe 5: [Conditional probe: If t he participant selects answer choice A, the n ask the follow

- up question]: Do you know whether other 4th

- grade students had the activities or the activities were not
offered to 4th

- grade students?

Five of 10 respondents said that all of the 4 th graders go to the field trips together, three said they did

no t know, and two said “no.”



## Solutions to Improve Item

Respondent: NA

Interviewer: The probes in this question seem to cause some confusion, particularly P robe 4 where there are perhaps too many components to the question for students to easily make sense of, and the same may be true of P robe 4 as there are two distinct components to that probe as well. Additionally,
there seemed to be some issues with ensuring that respondents were choosing answers that onl y applied to students’ time in the 4 th grade. Perhaps this point could be further emphasized in the wording
of the item.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of the students seemed to understand the question as written, but t here are some questions regarding the validity of the time frame students were thinking of in their responses. Our

r ecommendation is to further emphasize that the question is referring to students’ time in the 4 th grade

, p erhaps by using the term “this yea r” either in the item or the answer choices.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 46 

## Item 14

Reading: NA

Understanding: It s eemed that some respondents did not interpret the question solely in terms of
school resources.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- seven out of 30 responding students seemed to interpret the item correctly. There were varying degrees of detail in respondents’ answers, but the gist of the question appeared to be intact. One
student seemed a little confused an d stuck on the in school portion of the question and did not really engage the idea of materials, and two students offered non sequitur responses such as, “ How would you treat people that went in war or something who built the Statue of Liberty or s omethin g” and

, “ Have you found books or videos that teaches you languages and ... Do you know what a translator is on your phone or something? That is something you could use. You could put a word down and it could translate that. So like that. Or YouTube, you co uld look Indian words up and stuff. ”

Probe 3: [Conditional probe: If the participant selects answer choice A or B, the n ask the follow

- up question]: How many times have you used these materials in school?

Answers among the 27 respondents to P robe 3 varie d somewhat widely with six students saying they had used these materials two times, five indicating 10 or more times, four saying three to four times, three saying only once, and other answers with one response were five times, “no

, ” “ a lot, ” “ two times a week, ” and “I don’t know.” Additionally there were three difficult

- to

- categorize answers:

“ During Lakota class we watch videos of how they speak and we watch legends like dinosaurs from the
end of the world and stuff. ”

“ Kind of a lot in Idaho history. We h ad to make dioramas about Native American tribes. I did Nez Perce.

We went on

- line to go find stuff about it. I think that's all. ”

“ On my old school, they had spears way back when the days and then they had the sharp rocks on them. They sharpen them and on e time, they brought ... we had a science experiment where we’re learning about this and then they brought this one across like that and then it had one that goes through it and Student #14 Revised /VH040200

During 4th grade, have you used books, videos, or other materials (including online resources) about American Indian or Alaska Native people in school ?

A. Yes, once or twice

B. Yes, three or more times

C. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 47 it’s really [inaudible 00:19:11] and we looked at it and see ... one had signs on it like paint writing on it but it’s really tiny like this little thing when you look at them. ”

Probe 4: What does the phrase “ online resources” mean to you? Can you give me examples of “online
resources”?

In general, all 31 responding students seemed to understand this question across the board, although the particulars of their answers varied quite a bit with regard to the components they chose to discuss. Eight students mentioned computers, seven mention ed websites (general, not specific ones), seven

re iterated “online,” six mentioned Google specifically, four said something about the I nternet, two mentioned YouTube, two mentioned Wikipedia specifically, and one said something about MP3s and listening to N ative music. One said “I don’t know” and another gave a difficult

- to

- characterize answer with the following: “Electricity and pictures that move.”

Probe 5: What does the phrase “in school” mean to you?

Twenty

- six of 31 respondents seemed to understand what “in school” was referring to, one student said they did not understand, and four gave difficult

- to

- categorize answers only peripherally related to school, but still related to the item

— having to do with school or online resources, just answers out of place given the probe asked.

Probe 6: Do you think that ans wer choices A and B are clear? If not, can you give me examples of
other answer choices that would be more clear?

All 28 responding students felt that answer choices A and B were clear. One said they were not but did not give an alternate to improve the choices, and one said, “ C , I don’t really like because no because if I never knew it, I’ll probably put C but I know it so I put B but A and B are good answers but C , it ain’t

. ”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students asked this question seemed to grasp the meaning and intent of the item. The phrase “online resources” could possibly be replaced with the term s “ I nternet” or “websites” as they may be more recognizable to students.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources” with the phrase "internet resources" which is likely more recognizable to students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 48 

## Item 15

Reading: NA

Understanding: A few of the respondents seemed to think this was the same question as I tem 14 at
first.

Changing Answers: Three respondents changed their answers

— two from B to A and one from A to B

—

a nd one other answered A , but later said they had used materials “three or more times.”



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 31 respondents seemed to grasp the main components of this question regarding having used materials outside of school. Again, there was some variation within answers, many specifically referring to AI/ANs and others simply talking about using materials not on school grounds, but all seemed to understand underlying concept of item.

Probe 3: [Conditional probe: If the participant selects answer choice A or B, the n ask the follow

- up question]: How many times have you used these materials outside of school?

As with the previous item, all 18 responses to this question varied quite a bit. Six said one to two times, three said three or more times, two said a “few” times, two said 10 times

, and other answers occurring only once were “at camp,” “not a lot,” “ once a week, ” or “ eight or nine times. ”

Probe 4: What does the phrase “outside of school” mean to you?

All 30 respondents to this question correctly identified what was meant by “out of school” with varying details. Most indicated that they were thinking of home or at the library or at a friend or family
member’s home.

Prob e 5: Do you think that ans wer choices A and B are clear? If not, can you give me examples of
other answer choices that would be more clear?

All 27 responses indicated that they thought the answer choices made sense. One said “ no ,” offering no explanation; one had th is to say after answering affirmatively that the answers made sense : “ I don't think they make sense . Because they're both the same,” and another student who answered in the
affirmative offered this caveat : “ Yeah. Only B does though. Because why would you l earn it if you’re not learning much looking at it once or twice? It shouldn't be there. Should be three times or more. ” Student #15 Revised/VH040205

During 4th grade, have you used books, videos, or other materials (including online resources) about American Indian or Alaska Native people outside of school ?

A. Yes, once or twice

B. Yes, three or more times

C. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 49 

## Solutions to Improve Item

Respondent: Differentiate answer choices

— opinion was not shared by the majority of respondents.

Interviewer: N A


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of respondents seemed to understand this item with ease and interpret it as it was

mean t . As with the previous question, the only change might be to consider replacing the phrase “online
resources” with the term s “ I nternet” or “websites” as they may be more recognizable to students.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources” with the phrase "internet resources" which is likely more recognizable to students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 50 

## Item 16

Reading: NA

Understandin g: Some issues that arose with the interpretation of the item came from confusion around the term “media center” and students not interpreting the question solely in terms of school

— also taking community resources into account.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- six students out of 28 respondents seemed to generally understand the question as it related to having a place to go to get information such as a library, and some of these s tudents directly related it to AI/ANs. Not all students seemed to interpret this question as being located in a school per se.

Probe 3: What does the phrase “school library, media center, or resource center” mean to you?

This probe drew a number of different answers from the 28 respondents , with students referencing different pieces of the phrase “school library, media center, or resource center.” Ten students said something about computer labs, nine mentioned libraries or books, five felt all of the different entities were essentially the same thing, five mentioned resources, one said something about it being in school, one said it would be elsewhere, and one said they did not know. The term “ media center ” seemed to
throw some students off, as did “ resource center ” (though to a lesser extent).

Prove 4: What does the phrase “have access” mean to you?

Twenty

- three out of 31 respondents felt that “have access” was related to being able to go somewhere or having permission t o go somewhere

— sometimes this was seen as a physical place, or the ability to go
to certain sites on the I nternet. Three students indicated that they did not understand what “access” meant, one thought the question was related to having the time to go some where

, and another thought it had to do with being physically close to a place. Four gave difficult

- to

- categorize answers such as, “Access more food and more computers” and

, “Really enjoy and really into learning about Native American people.”



## Solutions to Improve Item

Respondent: NA Student #16 Revised/VH040207

Do you have access to a school libra ry, media center, or resource center with books, videos, or other materials (including online resources) about American Indian or Alaska Native people?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 51 Interviewer: Perhaps look at rephrasing the item to use easier

- to

- recognize words and phrases. For instance, perhaps look at alternatives for “have access” and “media center.”



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The item seemed to be generally understandable by the majority of respondents. One

r ecommendation may be to simplify the language with regard to “access” and “media center.” Additionally, if the aim is to specifically ask about these resources at sch ool, that should be highlighted, perhaps by only bolding the word “school” and leaving the other words without bolding, or alternatively using the word “ school ” before every component of the question such as: “school library” “school media
center” or “scho ol resource center.”



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically asking about these resources in school by adding the bolded words " In school " to the beginning of the
item. For consistency with the proposed revision to the grade 4 student items VH040200 and VH040205, ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is likely more recognizable to students. The proposed revised item wording is consistent with other NAEP student questionnaires, such as the 2014 TEL Probe questionnaire.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 52 

## Item 18

Reading: NA

Understanding: The grand majority of students were able to understand the qu estion.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- six out of 31 respondents indicated that putting in effort equated to trying hard in one way or another. Only one student said that they did not know, and one student mentioned teachers. Two students gave hard

- to

- characterize answers : “I really think about what t he question is trying to ask me”
and

, “People effort into the schoolwork by learning about them.” There are p otentially typos from the transcription process.

Probe 3: [Conditional probe: If the participant selects answer choice B or C, the n ask the follow

- up question]: Can you give me examples of how you put a lot of effort into your schoolwork?

Out of 29 respondents , 19 gave examp les of trying hard to show how they put effort into their schoolwork. Four mentioned the importance of listening to teachers, two mentioned getting good grades, and two mentioned studying. The following responses were mentioned once: doing work quickly, do ing homework, getting help when needed, showing one’s work, and taking time to do work.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. Student #18 New/VH040240

I put a lot of e ffort into my schoolwork.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 53 

## Item 21

Reading: One student misread the answer choices.

Understanding: One student seemed to have trouble grasping the overall concept of the question.

Changing An swers: One student changed their answer from C to A after the interviewer prompted them

to re

- read the item’s answer choices (as they had been misread).



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 31 of the respond ents grasped the meaning behind this item with some variation. Many felt that they were being asked if they felt safe at school, or if they enjoyed school, or if they wanted to be at
school.

Probe 3: What does the phrase “belong at school” mean to you?

All but one of the 31 respondents seemed to understand the probe and all gave varying answers about what the phrase “belong at school” meant to them. Six said that it had to do with being at school, seven said it had to do with learning, five indicated that it meant you wanted to stay at s chool, four mentioned college aspirations, two said that it felt good to be at school, one said that it meant being part of something, one felt it meant wanting to stay at your school rather than going to another, and one did not know. One gave a difficult

- to

- categorize answer : “I think it means did your parents put you in [art 00:28:43] school or did they have to put you in a private school because maybe the kids aren’t so good at school to you. They treat you so mean but you treat them really nice. ”

Probe 4: [Conditional probe: If the participant selects answer choice B or C, the n ask the follow

- up question]: Can you give me examples of when you feel you “belong at school”?

Of 27 respondents , eight mentioned friends, eight mentioned learning, six brought u p feeling good at school, two mentioned college, two mentioned simply being at school, and one said they were not sure.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA S tudent #21 New/VH040382

I feel that I belong at school.

A. This is not like me.

B. This is a little like me.

C. This is a lot like me.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 54 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The grand majority of students understood the meaning and wording of this question and probes.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising

or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 55 

## Item 24

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- eight of 30 respondents answered this probe in keeping with the meaning of the item, two said that they did not know ( which may be related to interview fatigue

— particularly with what could be construed as repetitive questions )

.
Probe 3: What does the phrase “put in enough effort” mean to you? Can you give me examples of how you could put in enough effort?

Twenty

- three of 31 responding students referenced trying hard, three mentioned getting good grades, two said something about going to college, two mentioned listening to teachers and directions, one mentioned homework, one said staying in school, and one said they did not know.

Probe 4: What does the phrase “s ucceed in school” mean to you? Can you give me examples of what
“succeed in school” would look like?

Answers among 31 respondents were varied for this question. Six referenced trying hard, six said something about passing grades and movi ng from one grade to the next, six mentioned “graduation,” six mentioned getting good grades, three referenced listening to teachers and directions, three said they did not know and indicated the question was somewhat confusing, two spoke of college degree s, one simply said “no,” and one mentioned homework.



## Solutions to Improve Item

Respondent: NA

Interviewer: The question regarding p u t t ing in enough effort question was a little repetitive given I tem
18. S tudent #24 New/VH040391

If I put in enough effort, I will succeed in school.

A. Strongly disagree

B. Disagree

C. Agree

D. Strongly agree

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 56 

## Initial KAI Recommendation (Keep/Change/Discard)

Ke ep: All students seemed to grasp the mea ning and wording of this item.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 57 

# Surve y Instrument: Student Grade 8/Sequence I

Item 3

Reading: NA

Understanding: Most of the students understood the question without trouble ; two had to be redirected by the interviewer for a more appropriate interpretation.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 23 respondents seemed to grasp the meaning of the item, two said they did not know

, and
one did not answer.

Probe 3: In answer choice B, what does the phrase “Family members” mean to y ou?

Thirteen of 23 respondents named different types of family members that the question brought to mind, four mentioned that family members are people related to them, three mention ed that it would be other tribal members as they are all family to some d egree or another, and two mentioned love as a component of the relationship. One gave a difficult

- to

- characterize response : “ Which one of your family members taught you the most of your people? ”

Probe 4: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”?

Seven students (out of 21 respondents ) mentioned that tribal representatives were people who represented the tribe, five students gave specific names o r relationships of people who were representatives or elders, four mentioned people who had more knowledge, four said it had to do with Student #3 New/VH039931

Who taught you most of what you know about issues today that are important to American Indian or Alaska Native people?

A. No one has taught me about issues today that are important to American Indian or Alaska Native people.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. S omeone else (please specify):

________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 58 being older, three mentioned grandparents, one said that they were people higher up in the tribe, one said they did not know, one said people on the council, and two gave uncharacterizable answers.

Probe 5: [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?

Eleven of 17 respondents indicated yes, three said no, and three said maybe . M ost who indicated that they did have a family member who was also a tribal representative noted that they thought of the person as family first (one respondent was the exception).



## Solutions to Improve Item

Respondent: NA

Interviewer: In the future, it might be helpful to separate out P robe 4 into two different questions, same
for P robe 5.



## Ini tial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that the overwhelming majority of respondents understood the question as it was asked. One suggestion for potential change may be to keep the structure of the answers the same.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 59 

## Item 4

Reading: NA

Understanding: Most of the respondents seemed to understand the item as it was, though some students misconstrued the question and thought it had to do with learning about cultu re rather than language.

Changing Answers: One student changed their answer from B to A as a result of answering P robe 6.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Seventeen of 23 responding students seemed to interp ret the item correctly, five thought it had to do with culture or tradition, and one student was confused.

Probe 3: What does the phrase “American Indian or Alaska Native language” mean to you? Can you give me an example of “American Indian or Alaska Nati ve language” that you can speak or know
about?

Twelve of 23 respondents gave examples of specific languages, three said they did not know, three referred to “the language they speak,” two mentioned taking classes, two were confused by the question, and one would not give an example.

Probe 4: How many American Indian or Alaska Native languages do you speak?

Fourteen out of 23 respondents said they spoke at least one language

— or had some sort of knowledge of one language

. A few mentione d being able to understand but not speak. Five mentioned non

- AI/AN languages they could speak, four said they did not speak any AI/AN languages, and one student said they spoke two AI/AN languages.

Probe 5: How would you answer this question if you speak more than one American Indian or Alaska Native language?

Answers to this probe were varied among the 16 respondents , leading interviewers to posit that hypothetical questions are difficult for many respondents to answer. Three students gave D as their new Student #4 New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska Native language?

A. My school does not offer courses that are taught in an American Indian or Alaska Native language.

B. Never or hardly ever

C. Once or twice a month

D. Once or twice a week

E. Every day or almost every day

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 60 answer, three were confused by the question, three said they would answer E , three stated that they do not speak more than one language, two said their answer would stay the same, one said they would speak more often (but did not specify how much more ofte n)

, and one mentioned a non

- AI/AN language.

Probe 6: What is the difference between answer choices A and B?

Twelve students of the 23 respondents who responded to this probe responded in a seemingly appropriate manner ; however, there seemed to be a fair b it of confusion related to the fact that answer
choice A mentions school in the answer options but answer choice B does not. Five respondents gave responses that indicated that they were confused by the question, two mentioned that family would teach them, one said there were too many tribes at the school to offer language classes, one said there
was no difference bet ween the two answer choices, and one felt they were the same.

Probe 7: If there were no courses offered in your school that are taught in an American Indian or Alaska Native language, which answer choice would you select?

Fourteen of 22 students responding answered A , five answered B , and three answered D

. 

## Solutions to Improve Item

Respondent: NA

Interviewer: Make answer choices consistent to cut back on confusion

. S tudents seem to interpret only
answer choice A as being specifically related to s chool . They seem to interpret the other answer choices as having more to do with use, and not necessarily related to classes or school.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: The item seems to get distorted by the way the answer choices are phrased

. We recommend making wording consistent and potentially highlighting the phrase “in school” if the interest is in whether or not students learn a language at school and not at home, informally, or through a tribal or
cultural center.



## ETS Recom mendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer choice “A”, ETS recommends changing the word “courses” to the word “classes” to be consistent with
the item stem.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 61 

## Item 5

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 23 respondents appropriately rephrased the question

. A ll understood that the item was asking them to determine how well they could speak in an AI/AN language.

Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?

Of the 23 respondents , 22 essentially reiterated the probe and said that they thought the question

, “I can speak a few words or phrases

, ” meant being able to spe ak a few words. One student said they did
not know.

Probe 4: In answer choice C, what does the phrase “I can speak well” mean to you?

All 21 students responding said that “I can speak well” meant speaking a language with fluency

— the amount of fluency ranged from one paragraph to being a fully fluent speaker.

Probe 5: Do you think that addit ional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?

Out of the 23 respondents , 14 students did not feel that there was a need for additional answer choices.
Nine student s indicated that there could be additional answer choices

. O ne of the suggestions that came up three times was the notion of being able to understand but not speak a language, o r something between speaking a few words and phrases and speaking a language well.



## Solutions to Improve Item

Respondent: Add an additional answer choice between B and C

.
Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider addin g an additional answer choice between B and C

. Student #5 New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?

A. I cannot speak an American Indian or Alaska Native language.

B. I can speak a few words or phrases.

C. I can speak well.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 62 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 63 

## Item 6

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents interpreted the item as it was intended and understood that it was asking them to determine how well they could read in an AI/AN language.

Probe 3: [Conditional probe: If the participant selects answer choice B or C, the n ask the follow

- up question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?

Eleven of 12 respondents mentioned specific languages, and one mentioned basic literacy in a language but did not specify which one.

Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow

- up question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?

Eighteen of 20 respondents indicated that they felt that the question mean t that you could read some of a language but not all of it, or that you could make sense of some works and that you could pronounce things but not necessarily understand it. Two students indicated that they did not know.

Probe 5: In answer choice C, what d oes the phrase “I can read well” mean to you?

Fourteen of 18 students responding said they thought it meant that you could read something with ease (such as picking up a book and being able to read it)

. O ne mentioned comfort with reading

, while another poi nted to the idea of ease of reading. Two students answered that they did not know.

Probe 6: Do you think that additional answer choices should be added between B and C? If yes, can you give me examples of what those answer choices would be?

Fifteen of 21 respondents said that the answers were fine as they were. Six students felt that additional answer options could be added

— at least three of these students heralded back to their suggestions from the last item having to do with speaking. Their suggestions were to add something about being Student #6 New/VH040067

How do you rate yourself in reading in an American Indian or Alaska Native language?

A. I cannot read in an American Indian or Alaska Native language.

B. I can read a few words or phrases.

C. I can read well.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 64 able to read but still stumbling over words, something between B and C , fluent reading, and sound ing
out words but not mak ing sense of them.



## Solutions to Improve Item

Respondent: There were a few suggestions for additional response options between B and C

.
Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally seemed comfortable with the question. C onsider making answe r choices uniform in phrasing.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 65 

## Item 7

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 22 students responding essentially restated the item using slightly different words.

Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?

There were v aried answers among the 20 respondents

. I t seemed that many of the students did not really understand what the probe was trying to get at. Twelve indicated that they were confused or did not understand the probe, two said it had to do with knowing the language, two said it had to do with

N ative languages, one said they did not know any A laska Native languages, one restated the probe, one said it had to do with the “old ways,” one said it meant you could write and speak a language, and one thought it had to do with culture.

Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska Native language”?

Again, there was a lot of variation among the 20 respondents to the answers

, and the probe seemed to not make sense to a lot of the respondents. Eight were confused by the question, three mentioned specific words in an AI/AN language, two mentioned specific languages, two said it had to do with the tribe, three thought it might refer to words for animals

, shapes

, prayers , or songs. One student mentioned tongue positioning for pronunciation, one mentioned pro nunciation, and one student said they did not have any examples. S tudent #7 New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?

A. No one has taught me about an American Indian or Alaska Native language.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): _________________

____________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 66 Probe 5 : [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?

Of the eight students answering the question, five said that they did have family members who were tribal members, and all said they thought of them as family first. Two students sai d no, and one said they were not sure.

Probe 6:

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: Probes 3 and 4 were confusing to students as they could be considered somewhat vague questions

. We recommend asking more specific probe questions in the future.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The item seems to be fine as is. R econsider keeping answer choices phrased similarly. Probes
could use some re

- tooling if they are to be u sed in the future.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 67 Student #10



## Item 10

Reading: NA

Understanding: One stude nt reportedly had trouble making sense of answer choices.

Changing Answers: One student changed their answer several times while working through options for
the A portion of the question.



## Probes

Probe 1: Can you tell me, in your own words, what the quest ion is asking?

Thirteen of 21 students responding essentially repeated the question with varying degrees of samenes s, referring to whether or not they had learned about N ative culture through different means, such as field trips or presentations. Four students thought that the question was referring only to things that happened within the school, two thought the question was only referring to what they had learned about AI/AN cultural aspects (no mention of school), one student thought the probe was ref erring to their thoughts on the item structure, on e student interpreted the question as having to do with what they had learned in class

, a nd one student thought the question was referring to what presentations they had given at school.

Probe 3: How do yo u know that other 8th

- grade students attended, or that activities were not offered
to 8th

- grade students?

A v ariety of answers were given among the 17 respondents for this probe, indicating some validity issues with understanding the probe. Six students sa id that other students were there (indicating that that was how they knew that other students attended), five were confused by the question, three said that they were there and that was how they knew, two students said it was difficult to answer because

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 68 th ey were unsure about what was happening at other schools, one said they did not know, one said no one was told, and another mentioned that they had gone in another grade.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask th e follow

- up questions]: Do you know whether other 8th

- grade students had the activities? Again, there was a variety of answers among the 18 respondents suggesting that students may not have fully grasped the probe. Thirteen said “yes,” four said they were there, three indicated that they did not understand the question, one said that none of the students had activities, one simply said “no,” and another simply said that they did know.

Probe 5: For questions a. through d., how would you answer this question if you did not attend these activities and did not know whether other 8th graders attended these activities?

Again, there was wide variety among the 19 responses . Four students indicated that they were confused by the question, two said B , seven said C

, two said that these activities were not offered, one said “yes,” one indicated they would still give the same answer, one student was not sure, and one student quipped

, “how would I know.”



## Solutions to Improve Item

Respondent: One student suggested that the questions would be difficult for a new student not familiar with activities prior to their arrival to answer.

Interviewer: Item and probes should be reconsidered for this question as they seem to confuse respondents and the validity of the data is questionable.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: Perhaps it would make more sense to ask this question in a more simplified format as it was for

4 th graders, or just make answer choices “yes” or “no.” Additionally, it may be helpful to determine the most salient pieces of information to be gathered from asking the question and tailor the phrasing
better to best capture that

— perhaps bold “through your school” and consider bolding “During the 8 th grade.”



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the format of the item should be simplified by collapsing the answer
choices to “Yes” and “No”.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 69 Stud ent #14 Revised/VC996624

Here are some sentences about reading. Fill in one oval on each line to show whether the sentence describes a person like you.

This is not
like me. This is a

little like

me. This is a lot

like me.

a. When my teacher talks about American Indian or Alaska Native history or culture, I try to read more about it. A B C Revised/

VH043704

b. I enjoy reading about American Indian or Alaska Native people. A B C Trend/

VC996627

c. I enjoy reading about people who have different traditions and cultures (ways of life, customs) than I have. A B C Trend/

VC996628



## Item 14

Reading: One student misread “sentence” as “sometimes.”

Understanding: Students had varied interpretations of the item that were all peripherally related to the question at hand but were dynamic in the different pieces that student s picked up on

— outlined further

in P robe 1. This variation in response may be due to the fact t hat there are three different questions to
summarize in I tem 14.

Changing Answers: One student changed their answer after responding to P robe 4 as they realized they
had confused or misread the answer choices for A and C

.



## Probes

Probe 1: Can you tell me, in your own words, what question a. is asking?

The 22 responses to the probe were varied with students picking up on different pieces of the overall item questions. For example, eight students felt the question was asking them about learning more or researching on their own, two interpreted the question as having to d o with enjoying learning, two commented on the structure of the item questions and answers, and two gave difficult

- to

- characterize
answers such as

, “It is basically like a little test to see if you are that person or not” and

, “Sounds like they’re talking about what you would describe your friend as.” Other responses had to do with interest, whether or not someone studied, reading, paying attention, if the teacher covered AI/AN issues/cultures, if students thought of their tribe when teachers were talking a bout AI/ANs, and whether or not students were interested in the topic. Again, the wide variety is likely due to the fact that there are three questions and students were essentially being asked to sum them up into one.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 70 Probe 3: In question a. what does t he phrase “when my teacher talks about” mean to you?

All 23 respon dents felt that the phrase “when my teacher talks about” referred to when their teacher was teaching

— either reading something to them, lecturing, or as part of a class activity.

Prove 4: W hich answer choice would you select if your teacher had not talked about American Indian or Alaska Native history or culture? Why would you select this answer choice?

Ten of 22 respondents said they would give A as an answer, six said they would choose B

, and two said they would choose C . Two students said they would “do more” but didn’t elaborate on whether or not that meant they would change their answer to B or C , and two students said they would keep their answer the same. One student was confused by th e question. Interestingly, and primarily among the
students who answered A , five students indicated that if their teacher was not telling them about AI/AN history or culture

, then how would they know to go look anything up ?



## Solutions to Improve Item

Respo ndent: Possibly put answers in boxes rather than at top of column with corresponding letters.

Interviewer: Consider drop ping P robe 3 as the concept is easily understood.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It is hard to tell if anything should be changed with this item given the variety of responses to

P robe 1. Again, it is very likely that the wide variety is due to students trying to describe all three questions from this item at once rather than the item itself being confusing.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 71 

## Item 16

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 23 respondents mentioned helping others with school work. Four specifically mentioned helping classmates or peers with school work, one specifically mentioned friends, and one spoke about helping younger students.

Probe 3: [Conditional probe: If the particip ant selects answer ch oice B, C, or D, the n ask the follow

- up question]: Can you give me e xamples of how you have helped other s tudents with their schoolwork? Are the other students that you have helped with their schoolwork in 8th grade? If not, what grade are the students that you have helped with their schoolwork in?

Of the 19 respondents , 12 students mentioned helping other 8 th graders, five said they helped students in other grades, five mentioned siblings or family members. With regard to areas where help was provided, five said they just worked through and explained things to other students, four mentioned math specifically, four mentioned language or reading, three mentioned helping on projects, three pointed to helping with homework, and one mention ed science.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. Student #16 New/ VH040389

How often do you help other students from your school with their schoolwork? For example, you might help other students to study for a test, help them with a school project, or go over
their homework with them.

A. Never or hardly ever

B. Once or twice a month

C. Once or twice a we ek

D. Every day or almost every day

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 72 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identifie d that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 73 Student #17 Trend/VC996635

How often do any of the following people help you with your schoolwork? For example, they might help you to study for a test, help you with a school project, or go over your homework with you. Fill in

one oval on each line.

Never

or hardly

ever Once

or
t wice a
month Once

or twice a
week Every

day or
almost

every day

a. A parent or someone else from my family A B C D Trend/

VC996636

b. A teacher or another adult from my
school A B C D Trend/

VC996637

c. Another student A B C D Trend/

VC996639

d. A tribal representative or elder A B C D New/VH018268

e. Someone else who lives in my community or is a friend of my family A B C D Trend/

VC996640



## Item 17

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 23 students responding felt the question was asking them about who helps them with their homework, five thought the question was focused on how often others helped them with their homework, one student made a comment abo ut how the item is structured (rather than focusing on content), and one student was focused on the question being about someone else who lived in the community.

Probe 2: Can you tell me, in your own words, what question d. is asking?

Seventeen students out of 23 respondents answered the question in a manner that was in keeping with the meaning of the item though most focused on tribal representatives over elders, likely because the

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 74 two are posed in a question together. Six students seemed confused by the question and gave answers such as, “I try an elder that has been in, that has known a lot about schoolwork and stuff and knows a lot about the tribal stuff,” or

, “Do you have any... Do you have any

- I don't know,” or

, “Not really. Usually a tribal repres entative or something or elder. I'm not really sure what it means ... like elder, or is that like an older person?”

Probe 3: Can you tell me, in your own words, what question e. is asking?

Seventeen of 23 respondents seemed to understand P robe 3 with ease. Two mentioned elders, two mentioned family members, and one mentioned tribal representatives. One pointed to the fact that they lived far from their community so they were unsure how to answer and one though t the question was focused on how often you were helped by the community member or family friend rather than what the phrase meant. Two seemed to not fully grasp what the question was asking and gave varied responses such as

, “Say you're given a month and you had problems within the month, you see how many ... which one of your parent, teacher, other student, tribal representative, or someone else helps you,” and

, “ D , my ... one of the elders, Nana Grace has helped me doing a project on N ative Americans. She was really helpful. [Nana Grace is family ].”

Probe 5: In question e., what does the phrase “someone else who lives in your community or is a friend of your family” mean to you?

Sixteen out of 22 students responding answered the question in keeping with what the question was asking, and a few mention ed that the question was very similar to P robe 3. Six mentioned neighbors, six mentioned family friends, two spoke of family, two said something about random people in the neighborhood, one noted their tutor, and one mentioned that they did not live in a community.

Probe 6: [Conditional probe: If the participant sel ects answer choice B, C, or D, then ask the follow

- up questions]: Can you give me exa mples of these people who have he lped you with your schoolwork? Can you give me examples of how these people have helped you?

Ten of 20 respondents mentioned family

— family included mothers (primarily), parents, grandparents,

aunts
, and uncles. Seven noted their siblings or cousins, eight referenced teachers, four spoke of friends helping, and two mentione d family friends. Two gave difficult

- to

- classify answers such as, “Like, for a tournament... If I'm in a basketball tournament, I'll go to my friend Braden. His uncle is like.... He's like, a coach for us, when tournaments come, so I'll ask him to help me . Like, "How should I do this?" Or, "How should I play in the game?" And he'll help me do it.” Or

, “If I have certain problems with a subject, they make sure I focus on that one subject so I get it easier.”



## Solutions to Improve Item

Respondent: One student wanted the additional answer choice “whenever they can.”

Interviewer: It would be helpful to ask about elders and tribal representatives separately as students choose to respond about one and not the other.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It appears that the item is easily understood by most students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 75 

## ETS Recommendation (05/02/2014)

Keep this matrix item , but drop sub

- item “e” VC996640

. Rationale: ETS recommends dropping this item ( VC996640 ) to reduce student burden. Having five sub

items about who is helping the student with his/her schoolwork is excessive. Asking about "someone else who lives in your community/friend of your family" seems to involve a nuanced understanding that is not being exhibited by some of the respondents to this question.

The similar grade 4 item VC759291 is also proposed to be dropped.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 76 Student #19 Trend/ VC996644

During 8th grade, how many times have you talked to each of the following people about the classes you should take in high school or about what you want to do after high school? Fill in one oval on
each line.

Never One
time Two or
three
times Four or

more times

a. A family member A B C D Trend/VC996645

b. A teacher A B C D Trend/VC996646

c. A school counselor A B C D Trend/VC996647

d. Another student A B C D Trend/VC996648

e. A tribal representative or elder A B C D New/VH026209

f. Someone else who lives in my community or is a friend of my family A B C D Trend/VH026211



## Item 19

Reading: One student misread the answer scale.

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The majority of the 23 students providing answers seemed to understand the gist of the item but took away different pieces from it

, likely based on the number of answer options and trying to rephrase them into a cohesive restatement of the question. Eight felt the question was focused on talking to someone about future plans, five felt it had to do w ith how often, four interpreted the question as having to do with talking to family members, two mentioned classes, two felt that the focus was who they had spoken with, one talked about friends, and one seemed to think the question was tied to success.

Probe 2: Can you tell me, in your own words, what question e. is asking?

This question had a variety of answers among the 18 respondents with 10 students referencing tribal representatives and seven mentioning elders. One student mentioned wisdom, one said something about speaking with a school president, one mentioned family friends, one thought it had to do with

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 77 people who came to the school to t alk about such topics, one mentioned their sister’s boyfriend, and one student thought the question had to do with how many tribal representatives or elders they had spoken with. The combination of elders and tribal representatives often seems to prevent s tudents from parsing out the differences between them

. However, the overlap is often very great because most representatives will be their elders, but not all elders are representatives.

Probe 4: [Conditional probe: If the partic ipant selects answer choic e A, then ask the follow

- up questions]: Do you have any family members who are also tribal representativ es? If yes, do you first think of them as your family members or as tribal representatives?

Of the two respondents , one student said “ yes ” and that th ey thought of them as family first, and one
student said “ no. ”



## Solutions to Improve Item

Respondent: NA

Interviewer: Potentially consider ways to split out elders and tribal representatives as different categories or just be aware of the issues of overlap and what that may mean for data interpretation.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It appears that the question is easily understood by students.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 78 Student #20 New/VH040157

How much do you agree with each of the following statements? Fill in one oval on each line.

Strongly disagree Disagree Agree Strongly

agree

a. If I put in enough effort I will succeed
in school. A B C D New/VH040158

b. If I put in enough effort in school I will get into college. A B C D New/VH040160

c. If I put in enough effort in school I will get a good job. A B C D New/VH040161



## Item 20

Reading: NA

Understanding: NA

Changing Answers: One student changed their answer at P robe 5 from “ strongly agree ” to “ agree ” due to the idea that some people do not have enough money to go to college.



## Probes

Probe 1: Can you tell me, in your own w ords, what the question is asking?

Fourteen of 20 respondents interpreted the question as having to do with the correlation between effort and success. Six students spoke to the item structure.

Probe 3:

Nineteen students of the 23 responding spoke about trying hard and working hard in school, four mentioned doing homework, two pointed to paying attention, one mentioned passing class, and one mentioned grades.

Probe 4:

Fourteen of 23 respondents referred to getting good grades, 11 referenced graduation, three pointed to good behavior, one mentioned getting an education, and one spoke about their future.

Probe 5:

All 23 respon dents seemed to struggle a bit with this probe because it was so straightforward a question. Twenty

- one said it meant getting into college ; of these 21 , three mentioned scholarships, and two mentioned going to a college of choice. Other answers were that the question had to do with money (one student), further education (three students), being smart (one student), having a career

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 79 (one student)

, and how big an achievement that would be for someone from a small town (one student).

Probe 6:

Nineteen of 23 respondents interpreted this question as having to do with being able to get the job that
you want

— your job of choice. Specific jobs mentioned varied among the 23 responses such as lawyers

, doctors

, and teachers were mentioned as was having good hours and pay and not working at McDonalds. Other professions mentioned were architecture, being a judge, photographer, business owner, and river runner among others.



## Solutions to Improve Item

Respondent: NA

Interviewer: Probe 5 is very straightforward and thus actually a little confusing to answer for some students, as if they suspect that it may be a trick question.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 80 Student #21 New/VH0 40216

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student tardiness A B C D New/VH040217

b. Student health problems A B C D New/VH040219

c. Student misbehavior in class A B C D New/VH040221

d. Physical conflicts among students A B C D New/VH040223

e. Bullying A B C D New/VH040226

f. Low student motivation A B C D New/VH040227

g. Low teacher expectations A B C D New/VH040231



## Item 21

Reading: One student stumbled over a few words but reading did not seem generally to be a problem.

Understanding: Two students seemed to be interpreting the item in terms of how it related to them personally

, as if the focus of the question were on them rather tha n on the school they attended.

Changing Answers: There were n o changing answers, but there was some difficulty choosing answers.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Sixteen students out of the 19 respondents f elt the item and subsequent questions wanted them to reflect on problems at their school, one student seemed to interpret the probe on a personal level, one said they did not know what the question was asking, and one thought it had to do with expectations

.
Probe 3: What does the phrase "student tardiness” mean to you?

Twenty of the 23 students responding felt the question had to do with lateness

— both to school and to class. Five students felt it also had to do with absence from school, and one said, “If they actually want to go to school and just learn new things.”

Probe 4: In question a., what does the p h rase "student health problems" mean to you?

Thirteen students out of 23 respondents felt the question was asking about students getting ill or sick,
fou r mentioned obesity, three brought up asthma, one said cancer, one noted broken limbs, one said autism, one mentioned drugs or alcohol, one talked about scoliosis, and one mentioned disabilities in

genera l

.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 81 Probe 5: In question c., what does the phrase “s tudent misbehavior in class” mean to you?

Out of all 23 respondents , 12 students felt that the question had to do with students’ disruptive behavior or “acting up” in class, eight mentioned talking in class or talking back to the teacher, five said
it inc luded not listening to the teacher, and four said it meant not doing work.

Probe 6 : In question d., what does the phrase “physical conflicts among students” mean to you ?

Nineteen of 23 respondents indicated fighting, four mentioned teasing or bullying, a nd two interpreted the question as also being related to verbal conflicts like teasing.

Probe 7: In question e., what does the phrase “bullying” mean to you?

Fourteen of 23 respondents talked about verbal conflicts such as teasing, picking on others, and talking behind someone’s back. Six felt that bullying could also be physical, and five spoke of other aspects such as the jealousy of bullies, the troubles with being “different,” and students sparring with one another

.
Probe 8: In question f., what does t he phrase “low student motivation” mean to you?

This probe and the following one seemed to cause a little bit of confusion to all 23 respondents as they had a harder time explaining what they thought the concepts meant without generally restating the phras e they were being asked about. For instance, 14 students felt that “ low student motivation ” meant that students had no motivation. Some explained a bit more by saying that some students saw no point in trying hard at school, or were not being encouraged to do better by friends or teachers. Nine mentioned that low motivation meant not trying hard

, and one said it meant sleeping in class.

Probe 9: In question g., what does the phrase “low teacher expectations” mean to you?

As with the previous probe, many of the 23 respondents’ answers fell into the same category as many students tended to rephr ase the phrase they were being asked to interpret. To that end, 18 students felt
that “ low teacher expectations ” meant that teachers did not expect much from their student s, that they did not care and were not motivating students. Four students said that t hey did not know what the question was trying to ask, and one interpreted the question as having to do with whether or not people had high expectations of their teachers.



## Solutions to Improve Item

Respondent: NA

Interviewer: It is h ard to say if students were answering for their 8 th grade class or their school overall.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Perhaps consider what is included when asking about the school in general versus just asking
about the 8 th grade.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students as some schools will likely consist of a mixture of both AI/AN and non

- AI/AN students. The purpose of

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 82 the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

Drop sub

- item “g” VH040231.

Rationale: ETS recommen ds dropping this item (VH040231) to reduce student burden. This sub

- item caused much more confusion for respondents than any of the other sub

- items within this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 83 

# Survey Instrument: Student Grade 8/Sequence II

Item 1

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fourteen of 25 responding stud ents felt the item was asking them to think about who had taught them the most of what they knew about AI/AN history. Four said they did not know, three mentioned history and culture, three felt the question was asking if they knew about AI/AN history, and one student felt the question had to do with tribal elders.

Probe 3: What does the phrase “American Indian or Alask a Native history” mean to you? Can you give me examples of “American Indian or Alaska Native history” that you have learned?

Eight out of 23 responding students simply reiterated that the question was asking about history, four spoke about culture, four mentioned specific tribes, and three felt it had to do with AI/AN lifestyles of the past. Four respondents mentioned drumming, art, dance, o r storytelling. Two mentioned language, two mentioned religion, and one student brought up ancestors.

Probe 4: In answer choice B, what does the phrase “Family members” mean to you?

Twelve students out of 24 respondents mentioned specific family members such as grandparents,
parents, aunts and uncles, siblings, and cousins. Five said that “family members” were people who were
“related

, ” five reiterated the term “family,” three mentioned that they considered the reservation or part of their family. One stu dent mentioned friends, and one talked about love.

Probe 5: In answer choice E, what does the phrase “Tribal represent atives or elders” mean to you? Can you give me examples of “Tribal representatives or elders”? Student #1 New/VH039733

Who taught you most of what you know about American Indian or Alaska Native history?

A. No one has taught me about American Indian or Alaska Native history.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify): _____

______________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 84 Eleven of 25 respondents spoke of family, six talked about tribal councils or chair people, three mentioned being old and/or wise, two students mentioned specific people, six mentioned the tribe. The responses to this question were fairly dispersed due to the fact that the question covers two grou ps of people who may or not may not overlap in terms of their roles as elders or tribal representatives.

Probe 6: [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e als o tribal representatives? If yes, do you first think of them as your family members or as tribal representatives?

Seven out of 12 students responding indicated that they did have a family member who was a tribal representative. Of these seven, four said they thought of this person as a family member first, two did not specify, and one said they thought of the person as a tribal representative first. Five students reported that they did not think that they had a family member who was also a tribal represen tative.

Probe 7: [Conditional probe: If the partic ipant selects answer choice F, the n ask the follow

- up question]: Who is the person?

Only one student chose F as their answer for the item and this student gave a specific name.

Probe 8: What does the phr ase “most of what you know” mean to you?

This question seemed to confuse students somewhat as it is a straightforward question, which at times leads to students either repeating almost verbatim what the question was, or trying to come up with an
answer th at is different. Of the 17 students responding , six felt the question was asking them about the most or the majority of what they knew, five thought it meant what they had learned, three felt it had to do with language, two thought it was asking about ever ything that they knew, and two said they did
not know.

Probe 9: Which answer choice would you select if multiple people had taught you about American

In dian or Alaska Native history? How would you decide which answer choice to select?

As with previous hypothetical probes, this one did not generate the same kind of reasoning that may have been needed to answer the question completely. Of the 17 students responding , seven said that they would choose B , five said they would choose D , two s aid E , one said F , one said that they did not know, and one said they would not choose more than one answer.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of the students seem ed to grasp the meaning of the item overall. C onsider keeping all answer choices phrased in a similar fashion, and consider if there is a need to separate out tribal representatives from elders or to keep the question as is.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 85 

## ETS Recommendation (05/02/2014 )

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 86 

## Item 2

Reading: One student misread the answer for E

. Understanding: One student initially thought this was a repeat of I tem 1.

Changing Answers: One student initially chose answer A but then quickly changed to B

.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fourteen of 23 responding students interpreted this question as having to do with who had taught them
the most about what they knew about AI/AN traditions, two mentioned how much they knew, one thought the question was the same as the last, one simply said “ no. ” Five gave difficult

- to

- classify answers ranging from

, “About American Indian N ative tribes. It is about our tr adition,” to

, “What they did or what they do, the people, American Indians” or

, “What do they do? We have Christmas, Thanksgiving, what do they do? ["They?"] The tribes.”

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions” mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you
have learned?

Eight of 25 responding students mentioned dancing and drumming, six brought up festivals and powwows, three spoke of religion, three t alked about food ways and hunting, two brought up specific tribes, two mentioned language, one discussed leaders, one brought up clothes, and yet another
mentioned ceremonies.

Probe 4: [Conditional probe: If the partic ipant selects answer choice B, then a sk the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?

Four of six students responding to this question said that they did not have family members who were tribal representatives, and two said they did and thought of them as family first. Five had already answered the question previously. Student #2 New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions (way of life, customs)?

A. No one has taught me about American Indian or Alaska Native traditions.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please spec ify): ___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 87 Probe 5: [Conditional probe: If the partic ipant selects answer choice F, then ask the follo w - up question]: Who is the person?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: The majority of students appear to understand the question.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 88 

## Item 8

Reading: NA

Understanding: Twelve students expressed confusion over the term “systems of counting” and did not know what it referred to. This calls into questi on the validity of the responses to this question.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twelve out of 25 responding students indicated that they did not know what the question was trying to ask or what systems of counting were. Though several students came up with answers, it seemed that there was a lot of guessing as to what “systems of counting” meant. Six focused on the part about who taught them about systems of counting, four guessed tha t the question meant counting in an AI/AN language, three thought it might have to do with tribes being able to keep tabs on their population counts, one thought it might have to do with money.

Probe 3: In answer choice A, what does the phrase “American I ndian or Alaska Native sys tems of counting” mean to you? Can you give me examples of “American Indian or Alaska Native systems of counting” that you know about?

Eleven of 22 respondents said they did not know, eight said that they thought it had to do with language, two brought up population counts, and one mentioned money.

Probe 4: [Conditional probe: If the participant s elects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?

Only two students responded to this probe and they thought of these people as family first. One
student had already been asked the question and had already answered it. Student #8 New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of counting?

A. No one has taught me about American Indian or Alaska Native systems of counting.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify):

_ __________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 89 Probe 5: [Conditional probe: If the participan t selects answer choice F, then ask the follow

- up question]:

Who is the person?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: Defi ne what is meant by “systems of counting.”



## Initial KAI Recommendation (Keep/Change/Discard)

Discard: Students clearly did not grasp what the question was trying to ask but would often provide

answers B

- E

, thus calling the validity of such responses into qu estion.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS recommends revising the phrase to "ways of counting".

This recommendation is aligned with the similar grade 4 item (VH040103).

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 90 

## Item 9

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The 25 responses given for this probe varied greatly as students were not sure wh at the question was asking or what “symbols and designs used in mathematics” meant. Five students focused on the part of the question regarding who taught them what they know, three mentioned math, four said they did not know, one said shapes, and one refe rence d algebra. Nine gave answers that were difficult to categorize.
Some of these are shared below:

“ About the symbols they used to tell each other what to do and yeah... ”

“ What does it mean like they’re never direct like Egyptians people really have their [inaudible 00:07:07] so they want to know what it is, probably like that. ”

“ I guess it’s the way they do stuff, like making a teepee. They would have to do all this counting and
stuff. ”

“ In mathematics, I'm pretty sure I know that symbols and designs, but I'm not really sure what it is doing
in mathematics. ”

Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native sym bols and designs” mean to you? Can you give me examples of “American Indian or Alaska Native symbols and designs” that you know about?

Eleven of 24 responding students felt that “symbols and designs” had to do with decorative symbols, usually in art. Six suggested that perhaps they were variations on plus, minus, greater than, division, and Student #9 New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and designs used in mathematics?

A. No one has taught me about American Indian or Alaska Native symbols an d designs
used in mathematics.

B. Family members

C. Friends

D. Teachers

E. Tribal representatives or elders

F. Someone else (please specify):

___________________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 91 multiplication symbols in math. Five said they did not know

, and two said that they did not think that symbols and designs were used in math.

Probe 4: [Conditional probe: If the partic ipant selects answer choice B, then ask the follow

- up questions]: Do you have any family members who ar e also tribal representatives? If yes, do you first think of them as your famil y members or as tribal representatives?

Only one student was asked the question ; they answered “ no. ”

Probe 5: [Conditional probe: If the partic ipant selects answer choi ce F, the n ask the follow

- up question]: Who is the person?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: This item should either be clarified or removed.



## Initial KAI Recommendation (Keep/Change/Discard)

Discard: R emove the question as there is a lack of clear understanding about what the question is asking. The lack of understanding calls into question the validity of the answers given by students.



## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting the phrase "symbols and designs", and the specific reference to math caused additional confusion.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 92 Student #11 Revised/VH039972

During 8th grade, how often have you used any of the following materials in school ? Fill in one oval
on each line.

Never

or hardly

ever Once or

twice a
month Once

or twice a
week Every

day or
almost

every day

a. Books, videos, or other materials (including online resources)
about American Indian or Alaska Native history, traditions, or cultures (way of life, customs) A B C D Revised/VH039974

b. Books, videos, or other materials (including online resources) not
related to A merican Indian or Alaska Native people, traditions,
or cultures A B C D Revised/VH040079

c. Books, videos, or other materials (including online resources)
about current issues that are important to American Indian or Alaska Native people A B C D Revised/VH040083



## Item 11

Reading: One student misread question row B , one misread the question as “today” rather than “8 th grade.”

Understanding: Five respondents expressed some confusion over how to make sense of the item and the different questions posed within it.

Changing Answers: One student changed their answer to row B from A to B after a discussion of the
item ’s meaning. Another student ex pressed some difficulty choosing an answer for r ow A and r ow C

.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

The 25 responses to this question were varied in part perhaps because respondents were being asked to restate three different questions as one. Four students thought it had to do with how often resources were used, four felt it was asking if they had ever used them, two r estated the question, one thought it

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 93 had to do with how often they used their AI/AN culture in their everyday life, and one thought it had to do with learning traditions. Two felt that the question was asking about where they learned about AI/AN
culture an d traditions, two said they did not know, and two restated all three portions of the question in their own words. Six students gave difficult

- to

- categorize answers such as:

“ The bad stuff that's happening, not the bad stuff but the important stuff. [Happe ning to what?] Something that’s important to them. [Who?] To the American Indians and Alaska Natives. ”

“ Like what's going on right now, in that culture or whatever. ”

“ Traditions they used, or how they used to work their life. ”

Probe 3: In question b., what does the phrase “books, videos, or other materials not related to
American I ndian or Alaska Native people, tradit ions, or cultures” mean to you? Can you give me examples of these books, videos, or other materials?

There were a variety of answers to this question from the 25 respondents . Twelve students referenced doing research or project work on topics not having to do with AI/ANs, three talked about history books, four mentioned going online to do research about anything (or looking at Facebook or viral videos), one student thought the question still had to do with AI/ANs, three said they did not know, and two mentioned research in general.

Probe 4: In question a., what does the phrase “online resources” mean to you? Can you give me examples of “online resources”?

Twelve of 25 respondents specifically mentioned going to websites to learn more about their AI/AN culture, nine mentioned Google, four talked about Bing, two mentioned YouTube, two mentioned Facebook, one said T witter, one mentioned the media, and one said anything on the I nternet.

Probe 5: What does the phrase “in school” mean to you?

As this question was very straightforward, it produced some interesting answers among the 22 respondents where students seemed to be unsure of the answers because the answer seemed so
obvious. Sixteen said that “ in school ” meant in school, two said they did not know, one thought it had to do with education broadly, one said it had to do with what they were l earning, and five gave difficult

- to

classify answers :

“ That means like spent in at max. ”

“ Having someone come in and talk to us about it or lessons with video clips about it. ”

“ Using books, reading, talking about it socially. ”

“ Boring. [Interviewer prompts .] Prison, learning, get educated, getting you ready for high school. ”

Probe 6: Have you used any of the books, videos, or materials listed in questions a. through c. outside of school, but not for “in scho ol” class work or assignments? If yes, do you cons ider those activities to be “in school” class work or assignments?

Ten of the 21 students responding said that they had used materials outside of school and five of those
considered that work to be school

- related. Five said they did not consider that work school

- related. Ten students said that they did not use materials outside of school, and one said probably.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 94 

## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It is difficult to make a recommend ation on this particular item given the lack of validity in how students interpreted the item

, which may be due in large part to having to summarize three questions
into one.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 95 Student #12 Revised/VH040104

During 8th grade, how often have you used any of the following materials outside of school ? Fill in

one oval on each l ine.

Never

or hardly

ever Once or

twice a
month Once or

twice a
week Every

day or
almost

every day

a. Books, videos, or other materials (including online resources) about American Indian or Alaska Native history, traditions, or cultures (way of life, customs) A B C D Revised/VH040106

b. Books, videos, or other materials (including online resources) not related to
American Indian or Alaska Native history, traditions, or
cultures A B C D Revised/VH040109

c. Books, videos, or other materials (including online resources) about current issues that are important to American Indian or Alaska Native people A B C D Revised/VH040111



## Item 12

Reading: NA

Understanding: Two students had to be redirected about the question referring to materials “outside of
school.”

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fifteen of 24 respondents said that the question was asking about what they did outside of school, two restated the question, and one said they did not un derstand. Six gave hard

- to

- qualify answers such as:

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 96 “ It's asking you how often you would go and look up stuff about Native Americans or the way of life or what's going on with them in oppose to how much you don't look up about them while you're still on

t he I nternet. ”

“ It's talking about outside and we just came back from outside. I get to play basketball, kickball,
stickball. ”

“ Things you don't do outside of school. ”

“ What you do the most or what you like to do. ”

Probe 3: What does the phrase “o utside of school” mean to you? Do you consider “outside of school”
to include all activities that are for schoolwork and not for schoolwork?”

Out of 25 respondents , 15 said that “ outside of school ” meant outside of school. This question seemed
to make s tudents uncomfortable because of how easy it was. Three said doing things on their own time, two said it had to do with hanging out with friends, two said they did not know, one said being at home,
and two gave hard

- to

- categorize answers:

“ Like at home or at people's houses or powwows or something. [All activities included?] Like kind of for
school. ”

“ Mmhmm. Sometimes, but whenever it gets kind of fun for friends and play. I'll start reading just like

- “

Probe 4: In question b., what does the phrase “books, videos, or other materials not related to American Indian or Alaska Native people, traditions, or cultures” mean to you?

Sixteen of 24 respondents said that the phrase in question B just meant researching different topics that did not have to do with AI/A Ns. Four thought it had to do with their or another AI/AN tribe

, and four said they did not know.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: It seems that most respondents seem to unders tand this question based on responses to the
item.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 97 

## Item 13

Reading: NA

Understanding: One student said they were unsure what “media center” meant.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 22 studen ts responding to the question mentioned having access to different materials ; only six mentioned in school. One student said they did not know and two thought the question had to do with where you would go if you did not have access.

Probe 3: What does th e phrase “school library, media center, or resource center” mean to you?

Nineteen of 24 respondents essentially defined the idea of “school library, media center, or resource center” as a place to go to find materials. However, it became clear that many st udents were unsure of what a resource or media center were (10), though they often hazarded guesses, and it seemed that many students interpreted “resource center” as a place outside of school. This is perhaps because “library” is the place they are most f amiliar with, and also it is the only term with “school” before it.

Probe 4: What does the phrase “have access” mean to you?

Twenty

- three of 24 students responding felt that “have access” meant being allowed, having permission ; one said that “have access” did not mean anything to them.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider changing “school library, media center, or resource center” to terms mor e familiar to students. School library seemed universally known

, whereas the other two options were either unknown to students or they thought they were not school

- based. Student #13 Revised/VH040207

Do you have access to a school library, media center, or resource center with books, videos, or other materials (including online resources) about American Indian or Alaska Native people?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 98 

## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically asking about these resources in school by adding the bolded words " In school " to the beginning of the
item. For consistency with the proposed revision to the gra de 4 student items VH040200 and VH040205, ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is likely more recognizable to students. The proposed revised item wording is consistent with other NAEP student que stionnaires, such as the 2014 TEL Probe questionnaire.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 99 Student #15 New/VH040130

Here are some sentences about your school. Fill in one oval on each line to show whether the sentence describes a p erson like you.

This is
not like

me. This is a

little like

me. This is a

lot like

me.

a. I put a lot of effort into my schoolwork. A B C New/

VH040131

b. I want to be one of the best students in my
class. A B C New/

VH040134

c. I enjoy being challenged in my classes. A B C New/

VH040137

d. I feel that I belong at school. A B C New/

VH040138



## Item 15

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what question a. is asking?

Fourteen of 21 responding students said that the question was getting at whether or not they tried hard at school. Three commented on the structure of the item, three mentioned liking school, and one restated the overall question with all four parts.

Probe 2: [Conditional probe: F or question a., if the participant selects answer choice B or C, then ask
the follow

- up question]: Can you give me examples of how you put a lot of effort into your
schoolwork?

Twenty of 23 respondents said the question was about trying hard in school suc h as paying attention or taking one’s time to do good work. One student mentioned getting help from others to succeed, and two gave difficult

- to

- classify answers:

“ Okay. Monday and Tuesday, I told you, we were in Greensboro. Tuesday night, whenever we came back, we had PTA that night. We got our report cards. Before we got report cards, I went to all of my teachers and I asked them to make sure if I'd missed anything, what it was, or if I could get it now, or if I could get it tomorrow at school or whenever . I don't want to miss out of anything, and I don't want to

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 100 be behind whenever we get back in class. Everybody's like, "I know, I know

, " and I'm sitting there like "I don't really know". If that's the case, I should have just stayed at home. I shouldn't ha ve come to school [inaudible 00:22:30]. ”

“ Well me, people try to split me up from my best friend but then we go right back to each other so like I feel like when I am with her I get my work done and stuff but people don't think that because we goof
off a l ot but then we get our work done. ”

Probe 3: Can you tell me, in your own words, what question b. is asking?

Eleven of 24 respondents mentioned being the best or the top of their class, six mentioned getting good grades, four said it was about being the sma rtest, four indicated that it was about being the teacher’s pet, three said it had to do with being better than others, three mentioned good behavior as a component, and two spoke about being popular.

Probe 4: Can you tell me, in your own words, what ques tion c. is asking?

Twelve out of 24 responding students indicated that enjoying a challenge had to do with harder material, nine said that it had to do with liking a challenge, five mentioned competition with others, and one said they did not know.

Probe 5 : Can you tell me, in your own words, what question d. is asking?

Ten of 24 respondents said that belonging had to do with feeling like you should be there, six said it was about feeling safe, four mentioned it had to do with liking school, one mentioned popularity, one said it was about one’s future, and one indicated that they did not know . One gave a difficult

- to

- categorize

answer : “Going against other people in my classroom.”

Probe 7: In question b., what does the phrase “one of the best students” mean to you?

Twelve out of 24 students responding said that “ one of the best students ” meant being one of the top students, six felt it had to do with grades, six brought up the topic of good behavior, one felt it had to do with being the fastest, and one indicated that they did not know what it mean t

.
Probe 8: In question c., what does the phra se “ being challenged” mean to you? Can you give me examples of you “being challenged in your classes”?

When asked about being challenged, 13 of 24 responding students mentioned having harder material to work with, five said it had to do with putting in mor e effort and pushing oneself, four talked about competing with others, one said it had to do with getting help when needed, and one said they did no t know.

Probe 9: In question d., what does the phrase “belong at school” mean to you?

Eight of 23 respond ents said that they felt they belonged at school because being there felt like they fit in, four said they felt safe, four said you should be at school, four indicated that it was because you were good at school, and three said to learn.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 101 Probe 10: [Cond itional probe: For question d. if the participant selects answer choice B or C, then ask
the follow

- up question]: Can you give me examples of when you feel you “belong at school”?

Of the 23 respon dents to the question, 11 said that they felt good at school, eight replied that it had to do with friends, two said that they felt they belonged at school when they were learning, one said that they had to be at school, and one simply replied “no.”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. All respondents seemed to understand the item as written.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive i nterviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 102 Student #18 Revised/VH040140

During 8th grade, have you participated in any of the following activities organized by your school ?
Fill in one oval on each li ne.

Yes No

a. Sports team A B Revised/VH040143

b. Drama club A B Revised/VH040146

c. Music club, orchestra, or band A B Revised/VH040147

d. Cultural dances or other American Indian or Alaska Native activities (for example, drum groups) A B Revised/VH040150

e. Academic club (for example, math club) A B Revised/VH040152



## Item 18

Reading: One student had a hard time reading “orchestra.”

Understanding: Four students had some difficulty defining some of the item’s different components.

Changing Answers:



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 21 responding students essentially restated the question in general terms regarding whether or not they had taken part in any of the after

- school activities mentioned. Tw o students talked about interest rather than participation.

Probe 3: In question a., what does the phrase “sports team” mean to you?

Twenty

- three respondents gave a multitude of examples of what they considered sports teams. Nine mentioned basketball, seven brought up football, four mentioned track or cross

- country, three talked about baseball, three mentioned soccer, three said volleyball, two said tennis, tw o said softball, one said golf, and one said wrestling.

Probe 4: In question b., what does the phrase “drama club” mean to you?

Eleven of 22 responding students felt that “drama club” entailed theatre or acting in plays, seven said they did not know, thr ee said they did not have a drama club at their school, one said they’d never done it, and one thought it had to do with dance. One interpreted the question as students creating drama and referenced groups of girls who created drama.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 103 Probe 5: In questio n c., what does the phrase “music club, orchestra, or band” mean to you?

Eighteen of 21 responding students interpreted “music club, orchestra, or band” as being related to playing instruments, although nine students also mentioned choir or singing. Thre e brought up dance as an example.

Probe 6: In question d., what does the phrase “cultural dances or other American Indian or Alaska

N ative activities” mean to you? Do you think the example “drum groups” is an example of cultural
dances? If not, can you gi ve me examples of what “cultural dances” would be?

When asked about cultural dances, 14 of 22 respondents talked about cultural dances, and five students mentioned powwows. Something interesting about other responses to this question is that most of the

e xamples were not school

- based but rather from outside of school. Additionally there were several students who, when prompted, said that drum groups were a good example; but there was also a contingency that pointed out that drum groups were not a dance.

P robe 7: In question e., what does the phrase “academic club” mean to you?

Fourteen of 22 respondents used math as an example of an academic club, three felt that it had to do with smart kids, two mentioned spelling bees, two brought up science, one mentioned history, one said they did not know

, and another student said that they did not have academic clu b at their school.

Probe 8: In addition to the activities listed in questions a. through e., do you think any additional
activities should be included? If yes, can you give me examples of additional activities?

Fourteen of 23 responding students felt tha t no additional activities needed to be added to the list. The following categories had one student each suggesting that it be added: AI/AN

- specific organizations, photography, choir, tutoring club, chess, science, art, and gardening.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider putting word “school” in front of each activity to ensure that students remember that the question is contextually based on activities offered by the sch ool

— if indeed that is the key component being measured.



## ETS Recommendation (05/02/2014)

Keep this matrix item, but drop sub

- item “b” VH040146

.
Rationale: ETS recommends dropping this item ( VH040146 ) to reduce student burden. This sub

- item caused much more confusion for respondents than any of the other sub

- items within this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 104

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 105 Student #20 New/VH040157

How much do you agree with each of the following statements? Fill in one oval on eac h line.

Strongly disagree Disagree Agree Strongly

agree

a. If I put in enough effort I will succeed
in school. A B C D New/VH040158

b. If I put in enough effort in school I will get into college. A B C D New/VH040160

c. If I put in enough effort in school I will get a good job. A B C D New/VH040161



## Item 20

Reading: One respondent skipped a word when reading the item.

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 21 respondents felt that the question was asking about trying hard to do well in school. One student restated the question, one thought it was a repeat of an earlier question, and one said they did
not know.

Probe 3: What does the phrase “put in enough effor t” mean to you?

All 23 respondents to this probe said that “put in enough effort” meant trying hard.

Probe 4: In question a., what does the phrase “s ucceed in school” mean to you? Can you give me examples of what “succeed in school” would look like?

Sev enteen of the 23 students responding said that “succeed in school” meant passing or graduating. Eight felt it had to do with getting good grades, three said it was related to being the best, and two mentioned good behavior.

Probe 5: In question b., what does the phrase “get into college” mean to you?

Twenty of 23 respondents felt that “get into college” meant going to college. One mentioned scholarships, two had difficult

- to

- categorize answers : “That you did succeed in school and you did what you are doing and you did it really good” and “It means, I don't know the word for it ... it gets harder, it

- “

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 106 Probe 6: In question c., what does the phrase “get a good job” mean to yo u? Can you give me examples of what “a good job” would be?

Fourteen for 23 respondents mentioned that “get a good job” meant having a job of one’s choice. The options listed at least once by students were: doctor, lawyer, teacher, astronomer, sports player, desk job, restaurant owner, physicist, engineer busines s owner , and others. Five students indicated that a good job meant having enough money, and two said that it was not working at McDonalds.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As i s.


## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 107 Student #21 New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student tardiness A B C D New/VH040217

b. Student health problems A B C D New/VH040219

c. Student misbehavior in class A B C D New/VH040221

d. Physical conflicts among students A B C D New/VH040223

e. Bullying A B C D New/VH040226

f. Low student motivation A B C D New/VH040227

g. Low teacher expectations A B C D New/VH040231



## Item 21

Reading: One student misread “moderate.”

Understanding: A few students had some trouble with some of the vocabulary used in the item, one had trouble understanding “extent

, ” another had a problem with “physical conflict,” and some grappled with adequately understa nding “low student motivation” and “low teacher expectations.”

Changing Answers: One student changed their answer in row A from C to B after the word “extent” was explained to them.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eighteen of 20 students responding felt the question was asking them about problems in their school. One person felt the question was asking if any of the problems were personally experienced ; one respondent’s answer was dropped due to losing audio

.
Probe 3: What does the phrase “student tardiness” mean to you?

All 22 respondents said that “student tardiness” meant being late

— there was a combination of respondents who thought this was just for class or for school overall. Additionally three student s thought that it meant being absent from school entirely.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 108 Probe 4: In question a., what does the phrase “student health problems” mean to you?

Of the 22 respondents

, 10 students felt that “student health problems” meant being sick or ill. Five mentioned asthma, two brought up diabetes, one mentioned obesity, one brought up cancer, one said broken limb, one student mentioned “disease” in general

, and one talked about nutri tion.

Probe 5: In question c., what does the phrase “student misbehavior in class” mean to you?

Eleven students out of 22 respondents felt that the probe about misbehavior could be characterized by not paying attention or listening, seven said it had to do with being disruptive in class, three said it was about being rude, one said talking in class, and one mentioned students acting up.

Probe 6: In question d., what does the phrase ““physical conflicts among students” mean to you?

Of the 22 respondent s, 17 respondents felt that “ physical conflicts ” consisted of fights. Interestingly, three students interpreted the question as having to do with physical activity such as sports and exercise, and two said it meant bothering others.

Probe 7: In question d ., what does the phrase "bullying" mean to you?

Fifteen of 23 students responding said that bullying had to do with picking on others (often those younger or smaller), two mentioned teasing, two said it was being rude or mean, two mentioned talking badly a bout someone, one student felt that it involved physical conflicts, and another student felt it had to do with problem students.

Probe 8: In question f., what does the phrase “low student motivation” mean to you?

Of the 22 respondents , 13 indicated tha t “low student motivation” had to do with not trying in school, three said it was students who did not want to be at school, two felt that they were students who needed help, one indicated that they were kids with problems either at home or at school, one student said they did not know, and two gave hard

- to

- characterize answers in the following:

“Sad, guilty, or it's nothing.”

“ No, I think that's a large extent. Our teachers are always, even if a student doesn't think they can do it, our teacher helps us, "You can do it. You can do it. Don't give up." I'm also in student council, so if there's ever anybody that's feeling like they can't go down, because I'm the president, they will come and talk to me. They'll ask for my advice. I also feel like I'm a guida nce counselor. I guess they feel more comfortable talking to me about it than they would an adult. I guess I would understand more. Low student motivation, no, because we're always trying to help in any way that we can. ”

Probe 9: In question g., what does the phrase “low teacher expectations” mean to you?

Students struggled a bit with “low teacher expectations.” Ten of the 22 respondents said that it was teachers who expect little and do not push their students. Three respondents said it was teachers who were not doing their jobs, two referenced mean teachers, one said it was teachers who did not discipline, and four students said they did not know. There were two difficult

- to

- categorize answers:

“When the teacher would tell you to do something you wouldn' t do it and you would like tell her no or yeah...”

“Like teachers they need to go on the right path, trying to help them out, do everything good to them. Just try to help them out on math, reading, science if a student won't know and a teacher will tell him

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 109 how to do it, give them the rules and everything. Like Rhonda Fry she helped us out on our rules learning from mathematics.”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider rephrasing “physical conflicts among students” to “physical fights among students” and consider easier phrasing for “low student motivation” and “low teacher expectations.”



## ETS Recommendation (05/02/2014)

Revise this matrix item.

Rationale: ETS r ecommends revising this item to clarify that it is specifically asking about AI/AN students as some schools will likely consist of a mixture of both AI/AN and non

- AI/AN students. The purpose of the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools; therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

D rop sub

- item “g” VH040231

.
Rationale: ETS recommends dropping this item ( VH040231 ) to reduce student burden. This sub

- item caused much more confusion for respondents than any of the other sub

- items within this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 110 

# Survey Instrument: Teacher Grade 4/Sequence I

Item 1

Reading: NA

Understanding: Two respondents had a difficult time determining what “culturally specific instructional practices for AI/AN students” meant. There was some question as to whether or not that meant teaching about the culture or teaching in a culturally appropriate manner

. Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Fifteen teachers ( out of 23 respondents ) said that they felt the question was asking them to reflect on whether they use cultural topics when teaching that are applicable to AI/AN students. Eight respondents thought it might have to do with culturally appropriate practices of engaging and meeting the needs of AI/AN students.

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?

Twenty

- one of the 24 respondents felt that the time frame of two years was appropriate. Of the three who did not, reasons had to do with either being very new to teaching, or (for lon g

- tenured teachers) that five years might be a better measure.

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?

Of the 23 respondents , 20 teachers thought of “the last two years” as the last two school years, while three thought of it as the last 24 months. Teacher #1 New/VH040247

During the last two years, to what exte nt have you implemented culturally specific instructional practices for American Indian or Alaska Native students in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 111 Probe 5: What does the phrase “culturally specific instructional practices for American Indian or
Alaska Na tive students” mean to you?

Twenty

- four teachers responded to this item . Of these, 12 felt that “culturally specific instructional practices” had to do with how one might make their teaching more culturally appropriate or responsive with regard to being aw are of and honoring cultural differences among their AI/AN students (i.e., adapting their teaching methods to meet the cultural needs of their students). Eleven mentioned teaching AI/AN topics, three mentioned working to incorporate AI/AN vocabulary into t heir teaching, and two said they made an effort to invite outside speakers to present on different AI/AN

- specific topics.

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can you give me a n example for culturally specific instructional practices for American or Alaska Native students that you have implemented in your classroom?

Seven teachers ( out of 18 respondents ) reported making lesson plans more interactive and hands

- on so as to better engage their AI/AN students, seven said they chose topics of interest for AI/AN students, and four mentioned being mindful of cultural practices (e.g., not touching children, limited eye contact). One mentioned having little leeway in the school

- prescribed curriculum to offer other topics, and one teacher again mentioned bringing in speakers to share different topics with the class.



## Solutions to Improve Item

Respondent: NA

Interviewer: The number and type of probes for this question were reasonable. The qu estion about participants ’ answers should come before asking them to restate the question as that seems to be the
natural flow of the interaction.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems like the question could be clarified to bet ter define for teachers if they are reflecting on topics or instructional practice as the difference between the two. The rather in

- depth analysis many teachers seemed to give the question while trying to determine what it was actually asking suggests that there is some lacking consensus on the actual subject of the question.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 112 

## Item 2

Reading: One participant misread “culturally” as “c ultural.”

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- one of 22 respondents interpreted the question appropriately. One answer was a non

- sequitur

response.

Probe 3: Do you c onsider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?

Fifteen teachers ( of the 19 respondents ) said that they did think that two years was a reasonable time frame, three said no (again, usually calling for either less or more time in the form of one or five years), and one respondent said it depended on what the question was asking.

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?

Of the 17 who responded , 11 teachers said they thought of the last two years in school years, six said they were thinking of 24 consecutive months.

Probe 5: What does the phrase “professional or community

- based development programs” mean to you?

Fourteen teachers out of 21 respondents felt that professional programs would have to do with receiving training by professionals or the district on specific topics or in other areas to help them maintain their teaching certification. Six felt that community programs would entail some kind of training given specifically by the community to address community needs, and six additional respondents said that community progr ams might be programs or workshops help by tribes. Two mentioned that maybe this would have to do with culturally specific training. Teacher #2 Revised/VH040275

During the last two years, how many times have you attended professional or community

based development programs (such as in

- service classes and workshops, including online classes) aimed at developing culturally specific instructional practices for American Indian or
Alaska Native students?

A. Never  Skip Pattern

.

B. 1 or 2 times

C. 3 or 4 times

D. 5 or more times

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 113 Probe 6: What does the phrase “in

- service classes” mean to you?

The answers varied greatly in specifics for this question, but all teachers ( 24 respondents in total ) were
aware of what in

- service classes were and what they meant for them and their certification. Descriptions used to answer this question included that in

- service classes were required, offered by the school, of fered by the district, a workshop, focused on academic topics, for credit, served as knowledge enrichment, done in

- school when students were not in school, were a class with other teachers, were part of their job, a professional development day, and requir ed for licensure.

Probe 7: When you answered this question, did you consider activities that were not in

- service classes and workshops or online classes? If yes, can you give me examples of what these other activities are?

Twelve teachers ( out of 22 respo ndents ) said “yes” and nine said “no.” Examples of other activities mostly related to school and teacher training; six had to do with culturally specific learning related to AI/AN students.

Probe 8a: [Conditional probe: If the participant selects answer ch oice B, C, or D, then ask the follow

up questions]: a. What development programs have you attended?

Of the 14 respondents , six teachers said that they had attended development programs sponsored by the school, three mentioned conferences, three mentioned presentations [Note: it was difficult to determine if these were through the school, community, conference, or other], one mentioned a local historical center and one said s/he still took classes.

Probe 8b: [Conditional probe: If the participant selects a nswer choice B, C, or D, then ask the follow

up questions]: b. Were any of the programs you attended taught as online classes?

Eight teachers ( out of 13 respondents ) said “no” to online classes, three said “yes,” and three indicated that they had done some online activities though these ranged from looking things up on the Internet to attending a webinar.

Probe 8c: [Conditional probe: If the participant selects answ er choice B, C, or D, then ask the follow

- up questions]: c. Have you attended development programs that are not through in

- service classes?

Seven of 13 respondents said “yes,” three said “no,” and three indicated that they possibly had. Examples were a con ference, a class, and learning through peers.



## Solutions to Improve Item

Respondent: NA

Interviewer: Some of the probes have multiple question components imbedded into one probe, making the answers somewhat messy.



## Initial KAI Recommendation (Keep/Change/D iscard)

Change: The question is generally fine as is, but may be helpful to highlight the fact that the question is specifically geared to learning more about AI/AN students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 114 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 115 

## Item 3

Reading: NA

Understanding: There was one question about whether or not the time frame for the question was still

two years.
Changing Answers: One teacher changed their response from C to D when considering their response.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 17 respondents , 15 teachers appropriately restated the I tem 3 question as having to do with whether or not they were using gained knowledge in their classrooms. Two gave difficult

- to

- categorize

answer s with the following:

“Have you taught a lesson that was geared on a topic maybe that your community had a need for i n your classroom?”

“As a part of that class that I took when I first came here, I had to teach a lesson in class and I also had to come back to my specific class and teach a lesson.”

Probe 3: What does the phrase “lessons learned” from these programs mean to you?

Twelve of the 16 respondents felt that “lessons learned” essentially meant that they had gained knowledge that they could put into practice. Three felt the question specifically related to new knowledge gained. One gave a difficult

- to

- qualify answe r :
“Hopefully there’s a positive outcome and that you want your lesson to be... That whoever is attending get something that’s an eye

- opener for them for their community, help them develop whether it’s their
students or the teachers. Make them more aware. I think awareness maybe overall.”

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can you give me examples of how you have implemented lessons learned from these professional or community

- based development programs in your classroom?

Answers to this probe varied in terms of topics noted as implementing lessons learned and new knowledge. Of the 17 respondents , four mentioned that particular teaching practices they had learned Teacher #3 New/VH040281

To what extent have you implemented lessons learned from these professional or community

- based development programs in y our classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 116 and made use of in their classrooms. Other responses had to do with using AI/AN stories ( three ), or cultural practices into their teaching ( two ). The following responses were only mentioned by one teacher for each topic: Singapore math, cultural awareness, common core, history, AI/AN language, social justice, 3D, and using projects.



## Solutions to Improve Item

Respondent: NA

Interviewer: Skip pattern not obvious at first.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 117 

## Item 4

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1 : Can you tell me, in your own words, what the question is asking?

Of the 16 respondents , 11 teachers felt that the question was trying to get at w ho sponsored or paid for the programs in which they had taken part over the last two years. Three felt the question had to do with where they were trained. Two gave difficult

- to

- classify answers:

“ I interpret it what avenues are you pursuing to get professional development. Is it all just coming from your school, is it all just coming from the state, or are you personally going out and looking for other opportunities to get that development. ”

“ Well, I kind of think the point of it is to find out how effective...If you have a program and it was from a certain place, how effective was it. Or what are the resources that are out there that you can take advantage of, like are they coming from the state, are they coming from new district, from tribal, from

In dian education, or are they coming from colleges or universities? So that’s kind of what the question

means to me. ”

Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow

- up question]: What is this funding source?

When asked about the funding source, the eight responses for this probe pointed to the BIE (two), the tribe (two), Teach for America (two), STEP (one), PACER (one), NSF (one), and one “I don’t know.”



## Solutions to Improve Item

Respondent: NA Teacher #4 Revised/VH040284

Who sponsored the professional or community

- based development programs you attended in the last two years? Fill in all ovals that apply.

A. State

B. District

C. Tribal education department

D. Indian education professional associations

E. College or university

F. Other (please specify): ________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 118 Interviewer: NA



## Ini tial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 119 Teacher #5 Trend/VE012668

To what extent do you use the following to assess student progress? Fill in one oval on each line.

Not
at all Small

extent Moderate

extent Large

extent

a. State assessments A B C D Trend/VE012670

b. District assessments A B C D Trend/VE012672

c. Assessments developed by American Indian or Alaska Native organizations A B C D Trend/VE012673

d. Tests supplied by textbook publishers (for example, end of unit or chapter tests) A B C D Trend/VE012674

e. Teacher

- made tests or quizzes A B C D Trend/VE012675

f. Performance

- based assessments A B C D Trend/VE012676

g. Group projects A B C D Trend/VE012678

h. Oral responses of students during class
discussions A B C D Trend/VE012681

i. Assessments to evaluate English language proficiency A B C D New/VH040300



## Item 5

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eight teachers ( out of 21 respondents ) thought the question was asking them what assessments they used, seven thought the question referred to how they assess their students, two thought the question was asking how often they assessed students, and one felt the question had to do with how wel l s/he
assessed students. Four respondents gave difficult

- to

- qualify answers [Note: one is too long to include in this report] :

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 120 “ Well, because one of the things

... Going back to the gentleman who gave me the same, basically
American Indian students learned d ifferently. If you're going to assess their performance, I would argue it's not just American Indian community that I know may sound differently. If you are only using things like state assessments and districts assessment, you're really not getting at the whole picture and I wholeheartedly agree with that especially since a lot of those things I see have definite [inaudible 00:21:40]. ”

“ I only think this because part B surprised me so much because I haven't actually seen any assessments developed by Americ an Indian or Alaska Native organizations, so in my mind it is just to cue my awareness, are you using assessments that are going to be fair and specific and incorporating the culture into the assessment? So often that can be a determining factor in success

. ”

“ I think it's demonstrating more than asking that there's a lack of input from...In assessments from
American Indian and Alaska Native organizations. It could be how important I would state the assessments in my own teaching. Yeah, I mean look at that s tuff. Usually the teacher made it tough, the day to day quizzes and I have districts to place our tests for things like Math. I supposed that helps guide us to see clearly what the students need more help in kind of thing with their stem of understanding. ”

Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency” mean to you?

This question seemed to confuse many respondents: none of the 23 respondents reported using an assessment to evaluate English language profic iency. All 23 teachers who responded had a notion of what English language proficiency testing consisted of, but it seemed from their answers that this was a confusing question. It is hard to say whether or not this is because most seemed to associate lang uage proficiency assessment specifically with students of foreign descent.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask
the follow

- up question]: Can you give me examples of how you have used “assessments to evaluate English language proficiency” to assess student progress?

Answers to this probe varied greatly

, suggesting some validity issues with teacher response choices to
the item . R espondents lacked the ability to explain what they were doi ng in their own classrooms to assess English language proficiency . Of the 15 respondents , four teachers used school

- based tests as their answer ( i.e., tests given to students across the board at the school level), three mentioned that they might better und erstand a student’s English proficiency through the use of math problems, one
used team meetings to go over test results ( bit did not specify a type of test), one said s/he simply
listen ed , one said reports, one said the language proficiency assessment, and one said s/he did not know. There were three answers difficult to categorize:

“ Well, the assessments to evaluate English language, they're only used with students who speak two language s. [ Inaudible 00:18:39] students when English isn't necessarily their first language, yeah. ”

“ I know at first for me, I was like, I don't teach English, I teach Ojibwe. So I have to try and teach the structure and the makeup of the language and how it's se t up different than English. But I don't test English so for me, that's opposite of what I do. Then after I thought, when you told me to focus in on it then I thought oh, maybe it's for first speakers who are in maybe an ELL program. ”

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 121 “ Maybe we’re practici ng for our NMSBA tests that are coming up. From the very beginning, kids will come in and I would ask them a question, they would say no, yes or this and that. Then we have to tell them, you need to answer me in complete sentence, complete thought. We have to practice like this so by the time the tests come around, hopefully, they’ll be able to answer in complete sentences and not just by one or two words. ”

Probe 5: In question c., what specific organizations did you consider when answering this question?

O f the 21 respondents , nine teachers said they did not know of any specific organizations, three said that they did not consider any specific organization, two teachers mentioned the BIA, two mentioned Indian Education departments, and two said they thought of tribal organizations. One reported using the local text, one thought of AI/AN organizations, and one specifically mentioned NIEA.

Probe 6: When answering this question did you consider NIES information about student progress? If yes, in which question specifically did you consider NIES information about student progress?

All 21 respondents said that they had not considered the NIES when answering the question, with many noting that they had never heard of the NIES.



## Solutions to Improve Item

Responden t: NA

Interviewer: Is this question meant to be specific to AI/AN students? If so, that should probably be explicit in the question asked.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: If this item is geared toward AI/AN students, consider high lighting the specification. Also, determine whether or not the item is referring specifically to assessments used by teachers or by the school. And last but not least, is there a time frame to be associated with this question?



## ETS Recommendation (05/02/201 4)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking about AI/AN students.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 122 

## Item 6

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of the 20 respondents seemed to understand the meaning of the question

. O ne response recorded consisted of “No, I didn’t.”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. Teacher #6 New/VH043923

Do you teach reading/language arts to grade 4 students?

A. Yes  Go to Question 7

B. No  Go to Question 8

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 123 Teac her #7 Revised/VH040381

How of ten do you integrate materials about the following topics into your reading/language arts
lessons? Fill in one oval on each line.

Never At least

once a
year At least

once a month At least

once a week Every day

or almost every day

a. American Indian or Alaska Native culture or history A B C D E Revised/VH0

40385

b. Current issues affecting
American Indian or Alaska Native people or
communities A B C D E Revised/VH0

40386



## Item 7

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answer for question A to C from D , and for question

B from D to C

.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 19 respondents interpreted the item as it was intended.

Probe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history” mean to you?

Twenty out of 20 respondents either worked to explain or gave examples of the meaning of “ AI/AN culture or history .” Some respondents felt t he question was straightforward, thus leading to redundant

answers. I t would be better in the future to simply ask for examples to determine understanding of the
concept.

Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska

N ative people or communities” mean to you?

Eighteen teachers ( out of 18 respondents ) gave answers indicating that they did indeed have an understanding of the phrase “current issues affecting AI/ANs.” Examples given in the course of answering included: trea ties, land rights, sovereignty, historical trauma, economies, casinos, lifestyle, health, and poverty.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 124 Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow

- up question]: Can you give me examples of how you have integrated materials about the topics into your reading/language arts lessons?

Five of the 16 respondents said they used state history to integrate topics into their lessons. Five said they used stories, traditional and otherwise. Three mentioned crafts from creating regalia to basket weaving

, one mentioned language, and one mentioned dance. I ssues of how muc h leeway teachers had
with their state or district curriculum was a gain brought up by one respondent

. 

## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 125 

## Item 8

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty teachers (out of 20 respondent s ) seemed to easily understand what the item was asking.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. Teacher #8 New/VH043941

Do you teach mathematics to grade 4 students?

A. Yes  Go to Question 9

B. No  Go to Question 10

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 126 Teacher #9 Revised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in

one oval on each line.

Never At least

once a
year At least

once a month At least

once a week Every day
or almost every day

a. American Indian or Alaska Native culture or history A B C D E Revised/VH

040390

b. Current issues affecting
American Indian or Alaska Native people or communities A B C D E Revised/VH

040392



## Item 9

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 18 respondents , all 18 teachers understood the item.

Probe 3: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow

- up question]: Can you give me examples of how you have integrated materials about the topics into your mathematics lessons?

On ly 11 respondents provided answers for P robe 3 . These ranged from teachers using questions about ancient or current population s, measurements on how to build thi ngs (such as teepees or drums), counting in an AI/AN language, culturally specific word problem s, data from current issues such as water rights, and mapping and scaling using math.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 127 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 128 Teacher #10 New/VH043953

How much do you agree with each of the following statements about the materials available in y our school library, media center, or resource center? Fill in one oval on each line.

Strongly disagree Disagree Agree Strongly

agree

a. The number of books and materials available for 4th grade students is
sufficient. A B C D New/VH043957

b. The quality of the books and materials available for 4th grade students is satisfactory. A B C D New/VH043959

c. The number of books and materials specific to American Indian and
Alaska Native culture available for 4th grade students is sufficient. A B C D New/VH043964

d. The quality of the books and materials specific to American
Indian and Alaska Native culture available for 4th grade students is satisfactory. A B C D New/VH043966



## Item 10

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen teachers (out of 21 respondents ) accurately explained that the question had to do with whether or not teachers thought there were enough satisfactory books for 4 th graders. Only 10 specifically mentioned AI/AN

- specific books.

Probe 3: In questions a. and c., what does the phrase “is sufficient” mean to you?

All 21 respondents mentioned amounts of books in their response to what “sufficient” meant to them.
One felt that the question had to do with quality, one said access, and one felt that it had to do with averages (i.e., sufficiency serving as another word for average, as does satisfactory).

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 129 Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you?

All 24 respondents provided an answer to the probe. Nine teachers felt that the question about what “satisfactory” means talked about the idea of averages. Five felt the question was talking about quality,
four mentioned if the books were co ntemporary, four said if books were appropriate for the reading age, three mentioned the condition of the books, two said it meant if there were enough books, and one
felt it had to do with whether or not books were relevant.

Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean to you?

Fourteen of the 24 respondents said that the quality of books and materials had to do with content (i.e., whether it was appropriate), seven mentioned the condition of the books , seven mentioned whether or not the books were contemporary, and four felt it had to do with whether or not the books were for the appropriate reading age.

Probe 6: In questions c. and d., what does the phrase “specific to American Indian and Alaska Native cultures” mean to you?

Of the 23 respondents , 11 teachers said that they felt that materials specific to AI/AN cultures should be historically accurate or accurate in general. Ten mentioned the word “specific” either in terms of explaining what they thought of the phrase posed to them, or sometimes having to do with the notion that materials should be about specific tribes or groups. Three teachers mentioned that materials should be authored by AI/AN writers.



## Solutions to Improve Item

Respondent: Th ere should be an “I don’t know” category.

Interviewer: The way the probes were phrased generated quite a deal of reflection on the part of teachers, but it is notable that the vast majority of teachers focused only on books as “materials.”



## Initial KAI Rec ommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 130 Teacher #11 New/VH0404 11

To what extent is each of the following a problem in your school? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student absenteeism A B C D New/VH040413

b. Student tardiness A B C D New/VH040414

c. Student health problems A B C D New/VH040416

d. Student misbehavior in class A B C D New/VH040422

e. Physical conflicts among students A B C D New/VH040424

f. Bullying A B C D New/VH040425

g. Low student aspirations A B C D New/VH040427

h. Low teacher expectations A B C D New/VH040428

i. Low family involvement A B C D New/VH040430



## Item 11

Reading: NA

Understanding: One teacher wanted to know if the timeline for this question was still two years. Another seemed to not understand the question but answered all subsequent probes appropriately.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty

- two teachers (out of 22 respondents ) seemed to appropriately understand what the question was trying to invoke.

Probe 3: In question a., what does the phrase “student absenteeism” mean to you?

Nineteen of the 24 respondents felt “absenteeism” meant students not at school; seven felt it had to do with the rate at which students were missing school.

Probe 4: In question b., what does the phrase “student tardiness” mean to you?

Of the 24 respondents , 21 teachers sa id that “tardiness” meant being late; five teachers mentioned the
rate at which students were late.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 131 Probe 5: In question c., what does the phrase “student health problems” mean to you?

Thirteen of the 24 teachers who responded to the probe felt that the question regarding student health problems involved students being sick. Five mentioned asthma, four talked about diabetes, three brought up allergies, two mentioned diet and nutrition, two noted disabilities (including learning disabilities), two mentione d physical problems, two noted mental problems, and one mentioned dental and one vision problems. Other issues mentioned were epilepsy and ADD.

Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you?

Eleven teachers (out of 24 respondents ) mentioned disruptive or distracting behavior as constituting student misbehavior. Nine brought up students not following the rules, and five mentioned students being disrespectful or talking in class.

Probe 7: In question e., what does the phrase “physical conflicts among students” mean to you?

Of the 24 respondents , 18 teachers felt that physical conflicts among students constituted fighting. Six teachers mentioned students putting their hands on each other, five mentioned hitting, th ree brought up kicking, and two said pushing.

Probe 8: In question f., what does the phrase “bullying” mean to you?

Eighteen of the 24 respondents said that bullying constituted repeated behavior such as picking on
others. Ten teachers mentioned verbal bu llying such as name calling, while cyber bullying was also brought up by three teachers. Nine teachers felt that bullying could also be physical.

Probe 9: In question g., what does the phrase “low student aspirations” mean to you?

Fourteen teachers (out of 23 respondents ) mentioned motivation of students as an issue in low student aspirations. Nine brought up students not feeling like they had much to look forward to with regard to their futures, and seven mentioned a lack of self

- esteem.

Probe 10: In qu estion h., what does the phrase “low teacher expectations” mean to you?

All 24 teachers responded to the probe. Of these, 13 teachers felt that low teacher expectations were akin to teachers not pushing their students, and 12 mentioned teachers with low s tandards for their
students’ achievements.

Probe 11: In question i., what does the phrase “low family involvement” mean to you?

All 24 teacher responses were consistent on ideas of what might constitute family involvement, but not
all teachers answered th e question in terms of what they thought low family involvement entailed. Different aspects discussed included whether or not parents volunteered at the school, sent notes to the teacher, were responsive to teacher communication, attended parent

- teacher co nferences, attended school events, etc.



## Solutions to Improve Item

Respondent: One respondent asked about whether or not they should be reflecting on the class level, grade level, or school level.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 132 Interviewer: Is this item meant to elicit reflection on th e entire school (which may be K

- 12 sometimes) or just for the 4 th grade?



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Be more specific about the level on which teachers should be focus when answering the item.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students as some schools will likely consist of a mixture of both AI/AN and non

- AI/AN students. The purpose of the NIES survey is to capture information that is specifi c to AI/AN students, teachers, and schools; therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 133 

# Survey Instrument: Teacher Grade 8/Sequence I

Item 1

Reading: NA

Understanding: Two respondents had trouble with the term “culturally specific instructional practices”
and what that meant.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Validity of this item is questionable: 16 out of 23 respondents felt that “implementing culturally specific instructional practices” had to do with covering topics related to AI/ANs, four felt like it had to do with actual teaching practices, one said that s/he did not know, and two expressed that they were not sure
what the term referred to. As one of the unsure noted:

“ I think the thing that catches me about it is what they might mean by an instructional pr actice because I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s why I would go with a small extent. I actually started a class called Engineering from a Native Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best practices for all students. ”

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame would be?

Of the 23 respondents , 19 expressed that they thought two years was a good time frame. One felt that four to five years was a better time frame, two said that one year was a better time frame, and one felt that six months was a more appropriate time frame. One resp onse was hard to characterize:

“ I have done it throughout my career and I have taught many different grade levels. I think that we always have Native students in classes. We also have Hispanic and Non

- Natives in other schools I have taught in and I think that everybody needs to know how rich the Native culture is. Sometimes we are Teacher #1 New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional practices for American Indian or Alaska Native students in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 134 marginalized to Native American Day. The Hispanics are Cinco de Mayo. That just makes me angry. I think that we need to be integrated and respected for the rich history that we have got. ”

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?

Twenty

- one of 22 respondents felt that “the last two years” referred to school years. One said 24
months.

Probe 5: What does the phrase “culturally specific instructional practices for American Indian or Alaska Native students” mean to you?

There was quite a bit of variance in this question; 15 of 23 respondents felt that “culturally specific instructional practices” referred to incorporating cultural material into lessons, six felt it had to do with culturally sensitive teaching practices, three said language, three said they were u nsure what the phrase meant, and one gave a difficult

- to

- characterize answer:

“ To me, the Lakota culture is an oral culture, and there's a lot more listening and talking in my room, in part because I'm a big talker, but also that's just traditionally how N ative parents would have educated their children. I think it becomes vague because there are so many, I'm sure this is exactly what you want on this recording, I think it becomes vague because there are so many generations of Native parents that have led t o the students that are in my classroom today that were not raised in the traditional household, traditionally by the Native parents. ”

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can y ou give me an example for culturally specific instructional practices for American or Alaska Native students that you have implemented in your classroom?

There were 19 respondents for this question, and the answers varied: six respondents indicated that
th ey implemented culturally specific practices through language, four mentioned science and nature, three said arts and crafts, two mentioned Native American History Month, one reported doing so through a school bulletin, one mentioned ELL practices, one sai d history, one reported not implementing practices, one described being mindful of eye contact, one said honoring shyness, and one described pairing students.



## Solutions to Improve Item

Respondent: One respondent suggested that an example of what is meant by “culturally specific instructional practices” would be helpful.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems that this question is an engaging one; however, as one of the respondents suggested, it would be very help ful to be specific about what is meant by “culturally specific instructional practices,” whether that refers to teaching methods or incorporating culturally appropriate material. And if both are intended, then perhaps this could be reconfigured as a matrix question.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 135 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 136 

## Item 2

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 21 respondents , 20 adequately noted that the question was asking about attending other training opportunities. Only 12 specifically mentioned AI/ANs in their restatement of the answer, one respondent thought the question had to do with implementing what was learned from the trainings, and one thought it had to do with language.

Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me examples of what a reasonable time frame wou ld be?

Fifteen of 19 respondents said that the last two years was a reasonable amount of time. Three said no, that it should be a longer time frame (again, five years was a common suggestion), and one said it
should be less time.

Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the last 24 months? If not, can you give me examples of what “the last two years” would be?

Of 17 respondents , 10 said that they thought of the last two school years as school years , and five said they thought of it as 24 months. One said “year round,” and another said they were thinking of this year.

Probe 5: What does the phrase “professional or community

- based development programs” mean to you?

Responses from 20 respondents were somewhat varied to this probe. Ten felt that the professional development programs were provided by the school or district. Answers about community

- based programs ranged with eight respondents essentially saying that they thought this meant that the

commun ity was involved

— though it varied by respondent as to whether they thought it was hosted by Teacher #2 Revised/VH040275

During the last two years, how many times have you attended professional or community

based development programs (such as in

- service classes and workshops, including online classes) aimed at developing culturally specific instructional practices for American Indian or
Alaska Native students?

A. Never  Skip Pattern

.

B. 1 or 2 times

C. 3 or 4 times

D. 5 or more times

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 137 the community or just involved the community. Five specifically mentioned AI/AN communities, four felt that the programs were offered by an outside source. Other co mments were that it had to do with cultural integration or that it was something you could get credit for, and one person mentioned
conferences.

Probe 6: What does the phrase “in

- service classes” mean to you?

Ten of 20 respondents said that in

- service cla sses were offered by the district, seven said they were held in school, three said they involved professional development, and two respondents seemed to think that it had to do with shadowing someone else (like a student teacher).

Probe 7: When you answer ed this question, did you consider activities that were not in

- service classes and workshops or online classes? If yes, can you give me examples of what these other activities are?

Eight of 22 respondents said “no,” and seven said that other activities the y were thinking of were community events. Two mentioned assemblies, one said they went to see a tribal council meeting, one mentioned conferences, one said they were thinking of a partnership with a local organization, one said they taught online classes a nd learned from that experience, and one said they did not know.

Probe 8a: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

up questions]: a. What development programs have you attended?

Five of 13 teachers resp onding mentioned outside speakers when asked about other development programs they had attended. Five mentioned AI/AN groups, four noted cultural training, three brought up nearby universities as resources, one mentioned community organizations, one said T each for America, one said GLAD (guided language acquisition design), and another said they did not remember.

Probe 8b: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

up questions]: b. Were any of the programs you attended taught as online classes?

Eleven of 13 respondents said that they had not attended any online classes. Two said that they had but that they were not Native

- specific.

Probe 8c: [Conditional probe: If the participant selects answer choice B, C , or D, then ask the follow

- up questions]: c. Have you attended development programs that are not through in

- service classes?

Seven of 14 responding teachers said that they had attended development programs that were not through in

- service classes and six said they had not

. O ne said they did not remember. Of those who said yes, conferences, community events, presentations

, and workshops were mentioned.



## Solutions to Improve Item

Respondent: One respondent suggested a larger time frame than two years for thi s item.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: We recommend highlighting American Indian/Alaska Native students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 138 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 139 

## Item 3

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answer from B to D in response to Probe 3.



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eleven of 13 respondents to this question felt that the question had to do with whether or not they had implemented lessons learned. Two gave hard

- to

- classify answers in the following:

“ I think this is trying to ask that, I’m saying that the professional and community

- based development programs in my classroom has happened according to the language that I teach and the history that I bring. However, somebody has to teach some others like me so that all o f us teachers can do what I do. ”

“ I think the thing that catches me about it is what they might mean by an instructional practice because I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s why I would go with a small extent. I actually started a class called Engineering from a Native Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best practices for all students. ”

Probe 3: What does the phrase “lessons learned” from these programs mean to you?

Eight out of 15 respondents felt the question had to do with what implementation they had done of lessons learned. Seven indicated that it had to do with what they had taken away from trainings. Two

responses were hard to categorize:

“ I’m 65 years old. I think some of these lessons learned, I think I need to know some something about the Southwest, you know, for me to make inference of what they do and make a direct connection to maybe the Ojibwe way of doing that also. Let’s say, they do maple sugar right now here in Minnesota...During this time, what are they doing in New Mexico? What are they doing in the Pacific Northwest? What do they do up in Six Nations Iroquois Confederacy area? What are they doing over
her e about the Blue Corn Festival? What’s going on with the drum? These kind of things. People do not know anything about that and that’s what I think is real. We have a living, truly beautiful, living culture, Teacher #3 New/VH040281

To what extent have you implemented lessons learned from these professional or community

- based development programs in your classroom?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 140 but it’s all in books and these kids are going t o read and make their worksheets and they’re going to be done and get graded, move on. ”

“ Well, I was hoping that'd be the in

- service lessons that we learned. Well, as an educator, it's just something that you learned in the classroom situation or something like that, not necessarily learned on the street (which could be ‘ lessons learned, ’ but I meant it literally ).”

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can you give me examples of how you have implemented lessons learned from these professional or community

- based development programs in your classroom?

Six of the 14 respondents gave examples of including specific topics into their teaching (not all of these answers were specific to AI/ANs), six responded that they had made sure to be more culturally sensitive in their teaching methods, two brought up professional development in general, and one mentioned the GLAD program.



## Solutions to Improve Item

Respondent: One respondent noted that they found the item too Euro

- centric.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Is this question meant to invoke whe ther or not teachers implement lessons learned in general or whether they are implementing lessons learned specifically targeting AI/AN students? If the latter, it should be noted in the question and highlighted.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 141 

## Item 4

Reading: NA

Understanding: One subject indicated that they were a little confused about answer option C regarding the tribal education department and whether that referred specifically to a department on their

res ervation or if it was meant more generally.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twelve of 14 respondents felt that question was asking who had sponsored any community

- based development programs they may have attended. One respondent thought the question was asking if they had gone, and two responses were hard to categorize:

“ I'm a little confused about whet her that means a department specifically within our tribe that we have on our reservation or if that's just more general. I know CCSD puts on trainings. I don't know whether the ones that I attended were sponsored by CCSD. I don't have the impression that they were. I have the impression that it was more like our individual schools reach out to the tribal reservation and set it up that way, but I do know that the district offers trainings. ”

“ Professional development, which we do it every day, or we have it every other week. We do it every other week, because I'm the head of the math department. We have some kind of peer share, we pull videos up from the web site teaching channel and I implement it a large extent. We have a lady coming in also, doing professi onal development, she comes about every three months and she asked us to read a book and to implement it into our teaching. ” Teacher #4 Revised/VH040284

Who sponsored the professional or community

- based development programs you attende d in the last two years? Fill in all ovals that apply.

A. State

B. District

C. Tribal education department

D. Indian education professional associations

E. College or university

F. Other (please specify): ________________

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 142 Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow

- up question]: What is this funding s ource?

Of the five teachers who responded , one said they did not know, one said Teach for America, one said ELO (unsure of what this stands for), one said the school where they taught, and one said the Minnesota Literary Council.



## Solutions to Improve Item

Respondent: One respondent said, “ To me it automatically makes me think of the school district, which out here, like for our school, is kind of meaningless, because we're the only school in our school district. Tribal Education Department, Indian Educatio n Professional Associations, college university, for me, I guess the people that actually sponsored mine were a college or a university, Indian Education Professional Associations, because I feel like the Tribal Education Department is nested within that, and then the Tribal Education Department and the district. ”

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 143 Teacher #5 Trend/VE012668

To what extent do you use the following to assess student progress? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. State assessments A B C D Trend/VE012670

b. District assessments A B C D Trend/VE012672

c. Assessments developed by American Indian or Alaska Native organizations A B C D Trend/VE012673

d. Tests supplied by textbook publishers (for example, end of unit or chapter tests) A B C D Trend/VE012674

e. Teacher

- made tests or quizzes A B C D Trend/VE012675

f. Performance

- based assessments A B C D Trend/VE012676

g. Group projects A B C D Trend/VE012678

h. Oral responses of students during
class discussions A B C D Trend/VE012681

i. Assessments to evaluate English language proficiency A B C D New/VH040300



## Item 5

Reading: NA

Understanding: One respondent noted that they did not think of grades as a measure of student progress and further indicated that they did not know what “assessments developed by AI/AN organizations” meant.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 23 respondents , 22 thought the question had to do with how they assessed their students. One respondent thought it had to do with whether or not AI/AN students could understand questions.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 144 Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency” mean to you?

Out of a total of 22 respondents to this question, 12 referenced either ELL, ESL, or ELP as assessments for English language. Many of these respondents ad ditionally commented that they were unsure if AI/AN students received these assessments, mentioned that the counselor administered these tests, or noted that all of their students were English speakers. Eight teachers responding felt that the question

was simply asking about testing English, and two indicated that they did not know.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask
the follow

- up question]: Can you give me examples of how you have used “assessments to evaluate English language proficiency” to assess student progress?

There were 12 respondents for this question and three said they looked at vocabulary, two mentioned writing assignments, and one said ESL

. O ne brought up that they were a math teacher and another said they were a science teacher

— both indicated that they did not do English assessments. One teacher said they used oral assessments and another said they used the ACCESS test but said this was used when they taught in the Cari bbean. Two gave difficult

- to

- classify answers:

“ That’s the only thing that I’m thinking about when I think of assessments to evaluate English. Specific involves what is in that test or other examples? ”

“ The only thing I can think of is the language program that OLC has and then Red Cloud has their own language program too so they develop their own curriculum, all of that stuff. ”

Probe 5: In question C, what specific organizations did you consider when answering this question?

Eight of 21 respondents said th at they had not considered any specific organizations when answering questions, seven mentioned some specific AI/AN organizations or groups including tribes. One said the state, four said they did not know, and one said the Northwest Regional Lab.

Probe 6: When answering this question did you consider NIES information about student progress? If yes, in which question specifically did you consider NIES information about student progress?

Out of 20 respondents , 17 indicated that they had not considered the NI ES, many noting they had never heard of it before. One said yes and two gave difficult

- to

- classify answers:

“ Yeah. When I read it, I actually froze up because I am anti

- assessments or anti

- standardized assessments. When I read that, that's all I felt righ t away and it makes me defensive because I do not follow this prescription of those types of assessments. That being said though, my students, they have high success rates. They're high achievers. The way I assess them is at a different level. No. It's jus t a feeling. I just have a general stereotypical feeling, I guess. ”

“ I guess when I read it, I was really thinking about how I assess my students in class and where I get that
from. ”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 145 

## Initial KAI Reco mmendation (Keep/Change/Discard)

Keep: Consider clarifying the question by adding “in your classroom” if the question is intended to capture what teachers themselves are doing rather than all assessments used at school. Otherwise, this item seems generally to be understood.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking about AI/AN students.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 146 

## Item 6

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Interestingly the 21 responses given varied more than expected, likely because th e question is straightforward, causing some respondents to overly analyze the question. Seventeen thought the question was asking them if they taught reading or language arts. Seven of these specifically mentioned
the 8 th grade, and one said middle school. One teacher simply said “no,” and three teachers gave
difficult

- to

- categorize answers:

“ Teaching them strategies on how to understand what they're reading and what they're writing, how to
write sentences. ”

“ I think specifically if you have a reading langu age arts license, I would have answered yes. ”

“ I think that’s a departmentalized kind of a thing. We have reading teachers. We have language arts
teachers. As a n elementary teacher, this would’ve referred to me living on a reservation. [I either 00:32:34] do everything. ”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02 /2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. Teacher #6 Trend/VE012771

Do you teach reading/language arts to grade 8 students?

A. Yes  Go to Question 7

B. No  Go to Question 8

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 147

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 148 Teacher #7 Revised/VH040381

How often do you integrate materials about the following topics into your reading/language arts
lessons? Fill in one oval on each line.

Never At least

once a
year At least

once a month At least

once a week Every day

or almost every day

a. American Indian or Alaska Native culture or history A B C D E Revised/

VH0403
85

b. Current issues affecting
American Indian or Alaska Native people or communities A B C D E Revised/

VH0403
86



## Item 7

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 10 respondents , six mentioned AI/ANs specifically when answering the question about integrating materials. Four simply mentioned implementing culturally specific material.

Pr obe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history” mean to you?

Answers from the 10 respondents to this question were all over the map. Three teachers mentioned stories, four mentioned history, one pointed to tradition, one said identity, and one said that they did
not know.

Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska Native people or communities” mean to you?

All nine respondents pointed to current issues (such as the Keystone pipeline); other themes mentioned by the remaining respondents were treaties, mascots, suicide, government, issues with the state, alcohol, land rights, and suffering.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 149 Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow

- up question]: Can you give me examples of how you have integrated materials about the topics into your reading/language arts lessons?

The 10 responses re ceived for this question were very varied. Five respondents pointed to covering issues in their classes, two said they talked about history, one teacher said something about MLK Day (the rest of the answer had to do with having an African American presiden t and needing an AI/AN president), one said they had students read Native authors, and one pointed to Native American History
Month.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. Res pondents seem comfortable with the item. We suggest asking for examples in the probes rather than asking what phrases mean to respondents as the simplicity of the question often leads to unnecessarily complicated answers.



## ETS Recommendation (05/02/2014)

K eep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 150 

## Item 8

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 20 respondents , 17 felt the question was asking them if they were math teachers; of these 17, eight teachers mentioned 8 th graders. Three teachers stated that the question was trying to get at what one’s subject area was.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initi al KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. Teacher #8 Trend/VE012783

Do you teach mathematics to grade 8 students?

A. Yes  Go to Question 9

B. No  G o to Question 10

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 151 Teacher #9 R evised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in

one oval on each line.

Never At least

once a
year At least

once a month At least

once a week Every day
or almost every day

a. American Indian or Alaska Native culture or history A B C D E Revised/

VH0403
90

b. Current issues affecting
American Indian or Alaska Native people or communities A B C D E Revised/

VH0403
92



## Item 9

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All six respondents understood the question to be asking how they implemented AI/AN

- specific culture or current issues into their teaching.

Probe 3: [Conditional probe: If the parti cipant selects answer choice B, C, D, or E to questions a. or b., or to both, then ask the follow

- up question]: Can you give me examples of how you have integrated materials about the topics into your mathematics lessons?

Out of eight respondents , six spok e to using local context

— whether that be local animals or fruits or history. One mentioned using the local language and one made a statement about Native American History Month but did not elaborate on what that looked like in their math class.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 152 

## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 153 Teacher #10 New/VH040395

How much do you agree with each of the following statements about the materials available in your school library, media center, or resource center? Fill in one oval on each line.

Strongly disagree Disagree Agree Strongly

agree

a. The number of books and materials available for 8th grade students is
sufficient. A B C D New/VH040399

b. The quality of the books and materials available for 8th grade students is satisfactory. A B C D New/VH040401

c. The number of books and materials specific to American Indian and
Alaska Native culture available for 8th grade students is sufficient. A B C D New/VH040402

d. The quality of the books and materials specific to American
Indian and Alaska Native culture available for 8th grade students is satisfactory. A B C D New/VH040404



## Item 10

Reading: NA

Understanding: Two respondents assumed that rows A and B also referred to AI/AN materials.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents felt that the question was asking about the types of materials available to their students. Ten teachers specifically mentioned the 8 th grade and eight mentioned AI/ANs specifically.

Probe 3 : In questions a. and c., what does the phrase “is sufficient” mean to you?

Answers from 22 respondents to this question varied greatly with seven saying they thought that “sufficient” meant enough, five said it meant minimum or adequate, four said it was a subjective term, four said it referred to quality, four felt it meant availability, one said variety, and one gave a hard

- to

qualify answer: “ It's looking at resources in the building so that if we do want to work in ... I'm thinking

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 154 especially in terms of history and language arts. I'm doing things in ... Type into it to N ative American culture and kind of geared toward that, do we have the resources in the building. ”

Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you?

A gain, answers from 22 respondents were very varied to this question. Thirteen said that “satisfactory” meant meets minimum quality, two said the term was the same as “sufficient.” One said it had to do with reading level, one said it was needed to be curre nt, one said it was a subjective term, one felt it had to do with the condition of books, one said it meant “enough,” and another said it meant “needs work.” Two gave hard

- to

- qualify answers:

“ Let’s have a Native

- American section here, don’t I? Look in the library yourself. Check it out. Look
around. Where’s the Native

- American side? You’ll find out. Go to [Harlem 00:38:02]. Go over there. You know what, these 2 schools are American Indian studies [sites 00:38:10]. ”

“ Sufficient means enough, enough that we can ... whatever it is that we’re doing in our classroom. Do we have enough books that we can use in order to get across to our students what it is that we’re teaching, you know, that objective? Do we have not only specific things, but do we have a variety of things to where the kids can do a lot of research? Do they h ave a lot of places to go to find materials from different tribes or different languages and other customs? We just don’t have that. ”

Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean to you?

Eight of 22 respondents said that the question had to do with the content of books, for instance if it was

up

- to

- date and accurate. Six said it meant if the quality was good, four said it was the condition of materials, two said it was a subjective question, one said it should be by Native authors, one said it should be at appropriate reading level for grade, one mentioned variety, and one said it should be chosen by Native teachers.

Probe 6: In questions c . and d., what does the phrase “specific to American Indian and Alaska Native cultures” mean to you?

Nineteen out of 21 respondents said it had to do with AI/ANs, and four mentioned that it had to do with being authored by AI/ANs.



## Solutions to Improve Ite m

Respondent: Add an “I don’t know” option.

Interviewer: Consider specifying in rows A and B that the question refers to school books.



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Respondents generally seemed to grasp what the question was asking.



## ETS Recommendation (05/02/2014)

Keep this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 155 This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 156 

## Item 11

Reading: NA

Understanding: It was hard for some teachers who taught multiple grades to specify about their 8 th grade students. Additionally, in responding to Probe 1, many respondents did not mention their AI/AN students specifically, but this does not necessarily have bearing on understanding. We recommend highlighting the term “American Indian or Alaska Native” to ensure that is the focus for respondents.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Nineteen of 21 respondents said they were being asked to predict how many of their students would pass the 8 th grade, and five mentioned AI/AN students specifically.

Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students influence the answer choice you selected?

There were 22 responses to this question

, and given the incredibly wid e variety of answers

, there was a total lack of validity for this probe. The term “proportion” in the question led respondents to give actual numbers, percentages, or numbers of their AI/AN students compared to the rest of the class. And their
answers abou t whether or not the proportions had an influence were often not answered, though they usually fell into three categories: it makes no difference, makes it easier to answer because the proportion is small, or makes it easier to answer because the proportio n is high (usually for schools where AI/AN population is 100%).

Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the
follow

- up question]: Please explain why you selected the answer choice. In other words, ho w did you figure out the answer? Teach er #11 New/VH040406

What proportion of your American Indian or Alaska Native students will complete the 8th grade?

A. None

B. A few

C. Some

D. Most

E. All

F. I don’t know.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 157 There were 21 responses to this question, and again, the answers varied greatly

. Some teachers gave answers that heralded back to the previous probe having to do with whether or not they had a high or low proportion ( thus m aking it easy to calculate )

, while other respondents made blanket statements about expecting most of their students to pass. It is hard to say if the wide variety and vague answers given to this probe are a result of the way the probe is written or, more l ikely, that teachers are sensitive about the subject of the question.

Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

When asked if it was difficult or easy to answer, 15 of the 21 respondents said it was easy, though several of these respondents noted that they thought this was because they were at a smaller school and had smaller classes, whereas teachers at larger schools may have trouble. Three said it was difficult because they didn’t know who t heir AI/AN students were, and another said they did not like to admit to the numbers of where students were. Lastly there were three difficult

- to

- categorize answers, one of which was the following: “ To answer totally honestly, it could have been written be tter and I could have answered it easily. But since it was, "A proportion," is a math

- bound, and there's no proportions listed, in my view, anyhow, that made it difficult. Not the question ... or, the concept itself was not difficult, just the way it was w orded. ”

Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of
other answer choices that would be more clear?

Out of 20 responses to this question, 18 respondents felt that the answers were clear, while two fel t that the question was a little difficult as it was vague. As one stated, “ Well, you need to know who they're comparing with. Who are you comparing with ? You don't have a comparison, so how can you make a proportion if you don't have a comparison. To answ er those, if I was to do it, this is how I'd do it, I'd be thinking okay, whites 60

- 40. We had 54 last year, we have 60 this year. It's more than some, I'd say most, because I think of the kids that left, I'd be thinking of the kids that left school. One e xpelled,
went home at Christmas. Our turnover rate is about 45% here.”



## Solutions to Improve Item

Respondent: One respondent said, “ I think the proportion part should be taken out, because honestly if somebody is taking that survey and they haven't like recently looked at the, I don't go around thinking I've got this many of these students. I don't think people would do that. So, I jus t think the question maybe should ask, how many of your Native American students will complete the 8 th grade, or do you think a big amount would? But the propor tional part, I don't like that.”

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discar d)

Change: Given the wide variety of answers and the push back from some of the respondents, consider alternate wordings for this item. Also consider bold ing “American Indian or Alaska Native students.”



## ETS Recommendation (05/02/2014)

Revise this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 158 Rationale: ETS recommends revising this item to simplify the phrase "What proportion", which caused confusion for the respondents, by changing it to the phrase "About how many".

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 159 

## Item 12

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 20 respondents to this question felt that the question was referring to how many of their students would be prepared for high school. AI/AN students were only referenced by a few respondents, and it
should be kept in mind that some of the schools where interviews took place were 100% AI/AN, though the majority were not.

Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what extent does having a high or low proporti on of American Indian or Alaska Native students influence the answer choice you selected?

There were 17 responses to this question. Six teachers said that they had a small proportion of students; four said they had a large proportion. Seven said that the p roportion had no effect, three said that it did. One teacher said they did not know. Two teachers had difficult

- to

- categorize responses:

“ Most because it has to be a pair with what I answer for 1 1 if I believe that most of my 8th grader were going to pass then most of them will be ready for high school as well. ”

“ Just like the last one. I think that the answer choices can be more specific. What percentages or different words to use. But I think that I had a good sense of what most meant related to some or all. So I think it was okay.” Teacher #12 New/VH040408

What proportion of your American Indian or Alaska Native students will be prepared for high school?

A. None

B. A few

C. Some

D. Most

E. All

F. I don’t know.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 160 Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the
follow

- up question]: Please explain why you selected the answer choice. In other words, how did you figure out the answer?

The 18 respondents for this question gave very varied answers that ranged from counting the number of students to answers about how students pass no matter what, to pointing out that if the question actually used the term “8 th graders

, ” that would change h ow they answered, to saying things about how they know their students pretty well and that is what they based their answers on.

Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

Twelve of the 19 respo ndents said that the question was easy to answer, six said it was hard, and one
said “in the middle.”

Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of
other answer choices that would be more clear?

Fiftee n of the 17 respondents said that the answer choices were clear, and two said they were not

— largely because they were seen as vague.



## Solutions to Improve Item

Respondent: One teacher felt that “proportion” was the wrong term and should be changed to "port ion." Another suggested that the word “prepared” should be clarified as having to do with academic preparedness or social preparedness.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Consider changing the term “proportion” and specifying what is meant by “prepared.” Additionally, specify if teachers should be reflecting on their 8 th grade students.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising t his item to simplify the phrase "What proportion", which caused confusion for the respondents, by changing it to the phrase "About how many".

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 161 Teacher #13 New/VH040411

To what extent is each of the following a problem in your school? Fill in one o val on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student absenteeism A B C D New/VH040413

b. Student tardiness A B C D New/VH040414

c. Student health problems A B C D New/VH040416

d. Teen pregnancies A B C D New/VH040417

e. Drug or alcohol use by students A B C D New/VH040421

f. Student misbehavior in class A B C D New/VH040422

g. Physical conflicts among students A B C D New/VH040424

h. Bullying A B C D New/VH040425

i. Low student aspirations A B C D New/VH040427

j. Low teacher expectations A B C D New/VH040428

k. Low family involvement A B C D New/VH040430



## Item 13

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Twenty of 21 respondents understood what the item was asking. One respondent provided a difficult

to

- quantify answer: “ Behavior, I guess involvement from students, teachers, and the community to
parents. ”

Probe 3: In question a., what does the phrase “student absenteeism” mean to you?

All 21 respondents felt that “student absenteeism” meant students missing school.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 162 Probe 4: In question b., what does the phrase “student tardiness” mean to you?

All 21 respondents said that “student tardiness” meant students being late. Most of the respondents interpreted this as being late to school, others as late to class, and still others as late to both school and
class.

Probe 5: In question c., what does the phrase “student health problems” mean to you?

Answers varied for this question from the 21 teachers providing answers . Nine mentioned being sick, four said it could be emotional, three brought up asthma, two said lice, two said diabetes, two mentioned ADHD, two said it w ould be a physical issue, two said allergies, one mentioned obesity, two brought up hygiene, and one said broken bones.

Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then ask
the follow

- up question]: Can you give me examples of how teen pregnancies are a problem in your
school?

There were only seven respondents to this question about teen pregnancies

. T wo said bullying of the girls was a problem, two said that missing school was an issue

, and four gave dif ficult

- to

- qualify

responses:

“ After hours there's not enough supervision. We had three or four pregnant. That got pregnant the last year that were 8 th graders. Some of them got pregnant at home, at Christmas and came back. So it all comes down to talking t o your kids, interacting with your kids, and basically talking to them. But then that doesn't help. ”

“ I had a pregnant 8th grader last year. I feel that’s a problem, not that I know her. She’s great but I feel when you’re pregnant and you’re in 8th grade, that’s a problem. I think we have four, not 8th graders. We had one 8th grader and then I think three or four high school girls that were pregnant last year. We have 50 students in our high school. We have 60 in our middle school. Proportionality wise, tha t’s a good number. ”

Probe 7: In question e., what does the phrase “drug or alcohol use by students” mean to you?

All 21 respondents felt that “drug or alcohol use among students” meant whether or not students were using drugs. Most respondents expanded to explain that they thought that most of this was occurring outside of school, though some said it could happen in school.

Probe 8: In question f., what does the phrase “student misbehavior in class” mean to you?

There were 20 responses to this question. Seven said that student misbehavior had to do with students not following rules, five mentioned students disrupting class, four said it was about students acting up, one said it was students being rude, another felt it was about students making bad choices , another said insubordination, and yet another said it had to do with students not listening.

Probe 9: In question g., what does the phrase “physical conflicts among students” mean to you?

Sixteen out of 21 respondents used the word “fight” to describe what they felt “physical conflicts among students” meant to them. Three used the word “physical” and two mentioned pushing.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 163 Probe 10: In question h., what does the phrase “bullying” mean to you?

Out of 21 respondents , eight said it was systematic or repeated behavior, seven felt that “bullying” was verbal in nature, six mentioned cyber bullying, four felt it was physical in nature, and four brought up students being picked on, harassed, or intimidated.

Probe 11: In q uestion i., what does the phrase “low student aspirations” mean to you?

When asked about “low student aspirations,” the 21 respondents had a variety of answers. Thirteen felt that it had to do with a lack of initiative or lack of vision for the future, si x said it had to with low self

esteem, one said “yes,” and another said that they did not know.

Probe 12: In question j., what does the phrase “low teacher expectations” mean to you?

Of the 20 responses received for this question, six said that they thou ght “low teacher expectations” had to do with teachers underestimating students, five indicated that it was about teachers who did not set that bar high, three mentioned teachers who do not care, two said it was about teachers who had given up, one said th at low expectations led to bad behavior, one said it had to do with the obstacles teachers face, one mentioned little administrative support, and one simply disagreed that it was a problem.

Probe 13: In question k., what does the phrase “low family involv ement” mean to you?

There were 20 respondents on the issue of “low family involvement.” Seven of these mentioned parents missing parent

- teacher conferences, seven mentioned low investment or engagement with children’s education, six said it was about paren ts not attending school activities (such as opportunities to volunteer, or going to games or concerts), five said it was about not communicating with teachers, one pointed to the fact that they were at a boarding school, and another mentioned that by the 8 th grade, many parents see their children as adults and so are more hands

- off.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: All respondents seem to understand the item and what it is asking, although one point may be that the question does not specify a grade level or a time frame. The time frame issue did not come up during participants’ answers, but it became clear that some teachers were answering for the entire school (as stated in the question) while others were responding only for 8 th grade.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically as king about AI/AN students as some schools will likely consist of a mixture of both AI/AN and non

- AI/AN students. The purpose of

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 164 the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 165 

# Survey Instrument: Administrator Grade 4/Sequence I

Item 1

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 11 respondents , six felt the question was asking about who funded or who had authority over their school, and five saw the question asking what kind of school it was.

Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain why.

Eight of the 12 respondents said that the y did not think they needed to supply more than one answer, and four said that they could choose more than one answer due to the way the funding of their school
was set up.

Probe 4: In answer choice E, what does the phrase “other nonpublic school” mean to you? Can you give me examples of “other nonpublic schools?”

Six of the 11 respondents mentioned parochial schools as an example, five brought up private schools, three mentioned homeschooling, one said tribal schools, and one said alternative schools.



## So lutions to Improve Item

Respondent: NA

Interviewer: NA School #1 Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.

A. Regular public school

B. Charter public school

C. Bureau of Indian Education contracted or grant school

D. Bureau of Indian Education operated school

E. Other nonpublic school

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 166 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 167 

## Item 2

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 11 respondents appropriately responded to the item and said that a boarding school was one where
students lived at the school.

Probe 3: What does the phrase “boarding school” mean to you ?

Of the 11 teachers responding , 10 indicated that it meant living on campus and two mentioned dorms.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow

- up questions]: Are there any students at your school that do not board? If yes, what proportion of your students do not board? To what extent does having a high or low proportion of students who do not board influence the answer choice you selected?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initia l KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item. School #2 New/VH040433

Is your school a boarding school?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 168 

## Item

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 respondents , eight essentially said that the question was asking if their school was on a reservation, one said “tribal land,” and two gave difficult

- to

- categorize answers:

“ This one is a double

- sided question because to the [tribe] , we are on tribal land, all of California but, no, we are not on the reservation. We know. Whether the site that that school is on happens to coincide
with the reservation that's been established b y the Federal Government and it doesn't. But for the tribal land, because it's supposed to all tribal land, they still hold on to view that gap. We are on tribal land whenever or anywhere in this region. ”

“ That the property that is around us is tribal prop erty, and that obviously we have a lot of tribal members living in this area, it doesn’t mean to me that the school is sitting on tribal land, but everywhere around us is and I don’t know if it is tribal land right here, I’m not sure. It’s all around us, I
know that. ”

Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?

When asked what “located on a reservation/tribal land” meant, nine of the 12 respondents said that it meant being on a federal reservation, three felt that re servations and tribal land were the same thing,
and one said that it meant self

- governance.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is. School #3 New/VH040437
Is your school located on a reservation/on tribal land?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 169 

## ETS Recommendation (05/02/2014)

Keep this i tem.
This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 170 

## Item

Reading: NA

Understanding: One respondent was confused by what “immersion school” meant.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of the 11 respondents for this question, nine said it meant teaching in another language, one said they did not know what the phrase meant, and one said, “ I think it might be getting at trying to see whether to track whether languages that migh t become extinct are still being taught. It might be looking
also at curriculum. ”

Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school” mean to you?

Of the 12 respondents for this question, one restated that they did not know what an immersion school was, nine said it meant teaching in an AI/AN language all day, one mentioned teaching in another language, and one talked about how their Native language was gone.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow

- up question]: What proportion of students in your school participates in your schools’ language immersion program?

NA 

## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Disc ard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item. School #4 New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 171 This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 172 School #5 Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the following ways? Fill in one oval on each line.

Yes No

a. Academic club (for example, math club) A B Revised/VH040576

b. Arts club A B Revised/VH040577

c. Cultural dances or activities (for example, drum groups) A B Revised/VH040578

d. Drama club A B Revised/VH040580

e. Making school curriculum decisions A B Revised/VH040581

f. Open houses or back

- to

- school nights A B Revised/VH040584

g. Parent

- teacher conferences A B Revised/VH040588

h. Parent

- teacher organizations A B Revised/VH040589

i. School sports A B Revised/VH040592

j. Volunteer programs A B Revised/VH040596

k. Other (please specify): _____________________________ A B Revised/VH040597



## Item

Reading: NA

Understanding: NA

Changing Answers: One respondent changed their answers once realizing that the answers were supposed to relate to families. Another respondent changed their answers once they reached the probes specific to each question section.



## Probes

Probe 1: Can yo u tell me, in your own words, what the question is asking?

Of the 12 respondents to this question, 10 felt it was asking about parent involvement in the school, one thought it had to do with the tribe, and one felt it had to do with what the school was inv olved in.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 173 Probe 3: Are your answer choices for questions a. through k. specific to the families of your grade 4 American Indian or Alaska Native students, or do they apply to the families of all American Indian or Alaska Native students in your school?

Eight of the 11 respondents said they considered the question as relating to all of the school’s families, two said just the AI/AN families, one said all families K

- 4, one said all 3

- 5 grades, and another said all K

5. In several of these schools, it was n oted that the population of AI/AN students and families was very high.

Probe 4: In question a., what does the phrase “academic club” mean to you?

Out of 12 respondents , nine brought up math, six said science, five said reading/language, two said
social st udies, one said history, one said spelling, one said to enhance subject material, and one said it
was after school.

Probe 5: In question b., what does the phrase “arts club” mean to you?

Ten out of the 11 respondents said that “arts club” meant that it wa s fine art, three just said their school did not have such an activity, and one mentioned music.

Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you?

Seven of the 12 responding administrators said that they felt that “cultural dances or activities” were AI/AN

- related. One mentioned drums, three said powwows, and one brought up ORF (did not explain what this stands for).

Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the
follow

- up question]: Can you give me examples of cultural dances or activities? Do you consider “drum groups” a good example of “cultural dances or activities”?

Four out of 10 responding administrators said tha t drum groups were a good example, two mentioned powwows, three mentioned different kinds of crafts, one said pit house activities, one mentioned sweat lodges, one brought up hand games, another said that for their tribe it was rattles and not drum groups, one mentioned songs, and one had the following to say:

“ To me it means either assemblies or students doing these things. They’re either taking lessons or showing off with what they’ve learned where they is some theme to what they’re engaged in. [Drum

gro ups good example?] AC/DC plays drum too so that’s a little weird to have it be the one thing that somehow the drums are ethnic or somehow. I don't know. I think because this questionnaire is in its infancy. Someone is just writing down some random stuff. T hat’s probably an example that needs to be refined with a couple of more activities. That would be clear if you decided to lift this as extracurricular or parent involvement. It would depend on what category they’re putting that in. ”

Probe 8: In question d ., what does the phrase “drama club” mean to you?

Nine of the 11 responses were that plays or theater constituted a drama club, four mentioned the performing arts in general, poetry was mentioned twice, and music, dance, and art were each
mentioned once.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 174 Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?

When asked about making school curriculum decisions, seven of the 11 responding administrators said that it meant having curriculum input from parents, and fi ve mentioned parent advisory committees.

Probe 10: In question f., what does the phrase “open houses or back

- to

- school nights” mean to you?

All 11 respondents appropriately characterized back

- to

- school nights as open houses or opportunities to
involve fam ilies in familiarizing themselves with the school.

Probe 11: In question g., what does the phrase “Parent

- teacher conferences” mean to you?

Again, all 11 respondents were in agreement that “parent

- teacher conferences” meant parents and teachers meeting to discuss children’s progress.

Probe 12: In question h., what does the phrase “Parent

- teacher organizations” mean to you?

All 11 respondents saw parent

- teacher organizations as an opportunity for parents and teachers to work together for the improvement of the school.

Probe 13: In question i., what does the phrase “school sports” mean to you?

All 11 respondents spoke about sports in terms of identifying what sports constituted a school sport (and to that end basketball, baseball, soccer, volleyball, track, and softball were mentioned). And some administrators pointed to the fact that their elementary schools did not have school

- sponsored sports but rather that was something managed by the wider community or to be found in middle or high
school.

Probe 14: I n question j., what does the phrase “Volunteer programs” mean to you?

Volunteer programs were seen by all 10 respondents to correspond with non

- paid work on the part of community or family members.

Probe 15: What proportion of your students are American I ndian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students in your school influence the answer choice you selected?

The 10 answers to this question on the proportion ranged from 2% to 33% to 100 %. Seven said that the proportion amount had no effect on their answers, one said they did not know, one said it made answering easier, and one said this was very influential in how they chose to answer.

Probe 16: [Conditional probe: If the participant se lects answer choice A for question k, then ask the
follow

- up question]: What are the other ways in which families of your American Indian or Alaska Native students are involved with your school?

Among the nine answers given to this question, the following themes arose: Bible study, festivals, budget decisions, leadership groups, grad school programs, reading to kids (mentioned twice), fund raising, and career day.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 175 

## Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider bold ing “families of AI/AN students.”



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 176 S chool #6 Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions and culture are offered at your school each year? Fill in one oval on each line.

None 1 2 3 4 or

more

a. Year

- long course A B C D E Trend/VC962954

b. Semester

- long course A B C D E Trend/VC962956

c. Workshop or unit A B C D E Trend/VC962957

d. Clubs A B C D E Trend/VC962958

e. Other programs (for example, study groups before or after regular class periods) A B C D E Revised/VH018332



## Item

Reading: NA

Understanding: Three respondents were confused about some of the terms used in this item such as “year

- long” vs. “semester long” or “other courses,” and another felt that the question was better geared toward middle or high schools.

Chang ing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

All 11 respondents felt that the question was asking whether or not their schools were teaching about AI/AN culture.

Probe 3: What does the phrase “study groups before or after regular class periods” mean to you?

All 11 respondents said that “study groups before or after regular class periods” meant before or after

school programs either as activities or additional help for students.

Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E to question e., then
ask the follow

- up question]: Can you give me examples of other programs offered at your school?

Answers from the seven respondents were varied and included youth development, AI/AN

- specific activities such as field trips, summer camp, tutoring, study groups, and, “ The counselor could possibly have say maybe one. The counselor could pull a group of kids and work with them specifically . There could be one there. Again, there's just ... ”

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 177 Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of
other answer choices that would be more clear?

All 12 administrators responded to this question and fi ve felt that the answer choices (or at least part of the answer choices) were clear, but nine of the responses indicated that there was some confusion over the answer choices. One mentioned that the chart was unnecessary, one pointed to the possible need
f or a “daily basis” category, one felt that E was not clear, three felt that the answer choices were in keeping with high school or middle school rather than elementary, another felt that clubs was repetitive of the other questions. Points made about the st ructure and content of questions are below:

“ Like I said, instead of how many of the following courses or programs are offered each year, I would maybe word it like is American Indian or Alaska Native traditions and culture taught in your school in any
of the following methods maybe. Not methods, but ... How is the American Indian or Alaska Native traditions and culture taught in the curriculum, or offered in the curriculum at your school? Again, we don't have a year

- long course, and we don't have a semest er

- long course, but the teachers do teach
units that would address that.”

“ It’s just kind of silly to have CBB but whatever. That’s a problem with this. You got to come up of some other way to be like answer to 6B is B

. You know what I mean? There’s a diff erent way to do that somehow. I don't know. [ Number is not clear?] Right. If the answer to 6B is B then you haven’t clarified.
I would do it like 6.1, 6.2, 6.3, 6.4 , the answer is B so that you get a little bit clearer there. A, B, C, D, E

, there’s no 6 as sociated with it. You should associate this as 6A, 6B, 6C, 6D, 6E not just to the .1. ”

“ Yeah, this is ambiguous...Yes, because it is a different groups, we have a couple different workshops and then we have one that goes every week, two times a week, so it 's a same answer, two for if you meet two times or if it is two times a week. See what I'm saying? ”



## Solutions to Improve Item

Respondent: Some respondents felt that the chart was unnecessary. Course structure is different in the

4 th grade and so the struct ure of answers is inappropriate or inapplicable. Row D is repetitive of Item 5

. Change language from “courses” to “class.” Find a different numbering system for question rows or

answers

— don’t use letters, just use numbers. “Courses” does not allow for summ er programs or after

school activities.

Interviewer: Consider including “language” along with traditions and culture in the question.



## Initial KAI Recommendation (Keep/Change/Discard)

Change: Given the respondent feedback, particularly to Probe 5, this question should be changed to better reflect the realities of elementary school. Additionally, may consider highlighting if this question is only to refer to the 4 th grade, as it currently does not.


## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to a more simplified format by collapsing the answer choices to “Yes” and “No”. The answer choices for this existing item caused confusion among

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 178 responden ts. Also, response frequencies indicate that the strong majority of the responses fall under the following two out of the five categories: “None” or “1”. As a result of changing this item to a “Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs about American Indian or Alaska Native traditions and culture offered at your school each year?”

. This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 179 School #7 Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your
school? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student absenteeism A B C D Trend/VC963030

b. Student tardiness A B C D Trend/VC963031

c. Student health problems A B C D Trend/VC963032

d. Student misbehavior in class A B C D Trend/VC963034

e. Physical conflicts among students A B C D Trend/VC963035

f. Bullying A B C D Trend/VC963037

g. Low student aspirations A B C D Trend/VC963038

h. Low teacher expectations A B C D Trend/VC963039

i. Low family involvement A B C D Trend/VC963040

j. Inadequate transportation for
students A B C D New/VH040475



## Item

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Out of 12 respondents , 10 felt that the question was referring to problems at the school. One person specifically mentioned problems for AI/AN students, one person felt it had to do with problems for students, staff, and family, and another mentioned that they were thinking of all students.

Probe 3: In question a., what does the phrase “student absenteeism” mean to you?

All 12 respondents felt that this question was asking about students who were missing from school or
class or both.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 180 Probe 4: In question b., what does the phrase “student tardiness” mean to you?

All 12 respondents thought this question was about asking about students who were late to school or
class or both.

Probe 5: In question c., what does the phrase “student health problem s” mean to you?

Of the 12 respondents providing an answer to this question, six mentioned children being sick or ill, four brought up diabetes, three said asthma, two mentioned mental or emotional problems, one said dental, one said sleep deprivation, one mentioned physical issues, one brought up allergies, one noted obesity,
and one mentioned lice.

Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you?

There were 12 respondents for this question and four mentioned disr uptive behavior as student misbehavior in class, three said it was students not listening, two said that it had to do with poor classroom management, two noted rule breaking and two brought up interrupting and disrespect.

Probe 7: In question e., what doe s the phrase “physical conflicts among students” mean to you?

When asked about physical conflicts, out of 12 responses , four mentioned pushing, three said shoving, three used the word “fight,” three said it had to do with students putting their hands on o ne another, two said hitting, one said bullying, and one said it had to do with classroom management.

Probe 8: In question f., what does the phrase “bullying” mean to you?

When discussing bullying, seven of the 12 respondents said that the behavior had t o be consistent and repetitive, five said it was generally verbal, three mentioned physical aspects, one brought up cyber bullying, one said that it had an effect on children’s self

- esteem, and one said it was related to poor classroom management/poorly or ganized adults. Interestingly it seemed that many of the administrators did not think that many of the complaints about bullying at their schools were actual bullying, but rather just kids being kids.

Probe 9: In question g., what does the phrase “low stu dent aspirations” mean to you?

Out of the 12 respondents , six intimated that low student aspirations were due to the low bar that teachers set, four mentioned motivation on the part of students, one said it had to do with students not trying hard, one bro ught up disabilities, and one said that some children did not like coming to school because they were bullied.

Probe 10: In question h., what does the phrase “low teacher expectations” mean to you?

With regard to low teacher expectations, five of the 12 respondents said that it had to do with teachers setting a low bar, two said it was because teachers were overwhelmed or had given up, one noted poor organization among teachers, one said it was teachers who did not bother to push their students, and
three gave hard

- to

- qualify answers:

“ A lot of them coming up with that problem. Is there a problem? But I don’t see. Teachers aren’t coming to us and saying that there’s a problem. ”

“ That says to me how much the administration expects of them as far as putting effort into educating
our children. ”

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 181 “ I would say ... Overall, here the teachers actually have really high expectations. I still answered that small extent, because I think sometimes ... One of the areas we struggle with as a school is we have a high level of proficiency. About 80% of our kids are proficient, but there's still that other 20%. We've got to focus on that. Even the students that are already proficient, we don't show a lot of growth. It's how are we challenging the kids that are already profici ent, because proficient isn't really good enough. ”

Probe 11: In question i., what does the phrase “low family involvement” mean to you?

Of the 12 respondents to this question, nine felt that low family involvement was related to families not physically com ing to events or activities, and four mentioned parents being non

- responsive to teacher
communication.

Probe 12: In question j., what does the phrase “inadequate transportation” mean to you?

There were 12 responses to this question and of those, eight fel t that “inadequate transportation” meant not having a way to get to and from somewhere, one interpreted it as students having dangerous transportation options, and three gave hard

- to

- categorize answers:

“ Somebody is trying to figure out. That’s a kind of d ifferent category. That’s like a jacked up district. ”

“ That means they're asking if the ability to transport our students hurts their education. ”

“ There’s none. I don’t see any inadequate transportation. ”



## Solutions to Improve Item

Respondent: Consider rephrasing "considering entire student body" or elimi nating it completely; also consider increasing the scale to five or adding proportion/ percentage. One respondent did not care for
the term "extent." A nother suggested answer categories: r arely, sometimes, usually, always.

Interviewer: Is this question meant to apply to the entire school (which may be K

- 8 or higher) or just to
the 4 th grade?



## Initial KAI Recommendation (Keep/Change/Discard)

Change: This item seems generally understandable. Conside r making it specific to 4 th grade. Additionally, should administrators be reflecting on all students or just AI/AN students?



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specifically a sking about AI/AN students as some schools will likely consist of a mixture of both AI/AN and non

- AI/AN students. The purpose of the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 182 School #8 Trend/VC963041

Which of the following types of student and family services are offered at your school on a regular
basis? Fill in one oval on each line.

Offered Not offered

a. Health services for students such as a school nurse A B Trend/VC963042

b. Health services for families such as a community clinic A B Trend/VC963043

c. Social or counseling services for students A B Trend/VC963045

d. Social or counseling services for families A B Trend/VC963047

e. Adult education programs for all ages A B Trend/VC963048

f. Tutoring services for students A B New/VH017944

g. Career counseling services for students A B New/VH017945



## Item

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 responses given, nine concentrated on providing or offering services to families, and two focused on providing services to students.

Probe 3: What does the phrase “a regular basis” mean to you?

Answers from the 12 respondents were varied in how they determined what “a regular basis” meant. Five said it meant regularly occurring, other answers were that it depended, would be most of the week, weekly, daily, three times a week, not once a year, all the time, or two times a month.

Probe 4: In question f., what does th e phrase “tutoring service for students” mean to you?

There were 12 responses to this question. Ten were that “tutoring services” meant extra help for students, one person said it was to help students elevate their scores, and another simply said “yes.”

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 183 Probe 5: [Conditional probe: If the participant selects answer choice A to question f., then ask the
follow

- up question]: Can you give me examples of the tutoring services offered at your school?

Eleven out of 12 respondents said that their school offers tutoring before, after, and during classes. Only one respondent said that their school did not have such offerings.

Probe 6: In question g., what does the phrase “career counseling services for students” mean to you?

With regard to career counseling serv ices, it is difficult to determine the group that the 12 respondents were thinking about when giving their answers. At least four mentioned that they thought the question was more appropriate for secondary

- level students, two mentioned job fairs generally, and seven said that their schools offered career counseling

— something it is hard to imagine that 4 th graders would be involved with given their age.

Probe 7: [Conditional probe: If the participant selects answer choice A to question g., then ask the
fol low

- up question]: Can you give me examples of the career counseling services offered at your
school?

There were only seven responses to this question. Three mentioned job fairs, three brought up outside speakers, and two mentioned teachers as people provid ing career information to students. Again, it is difficult to say if the focus here was purely on 4 th grade students.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep/Change: Consider bold ing if the question is focused on the whole school or only 4 th graders.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 184 

## Item

Reading: NA

Understanding: One respondent found the item difficult to answer but could not clearly convey why, and also noted that they were from Guatemala originally and so had a different definition for what a pueblo was.

Changing Answers: NA



## Pro bes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 12 responses given to this question, seven mentioned preparedness, and six thought that the question meant incorporating the culture.

Probe 3: What does the phrase “prepar ing American Indian or Alaska Native students for life in their community, tribe, group, pueblo, or village” mean to you?

Out of 12 responses , five mentioned the tribe with regard to preparing AI/AN students, one said they did not know, and three mentioned the community. Several respondents gave hard

- to

- qualify answers:

“ That’s really a touchy question there, because I am not Native American and some of this is allowing them to have excuses why they are late for school and my opinion is that I think we oug ht to treat everybody the same. And they want to be treated different and they think the teachers need to learn the culture of the Native American and accept that they’re late all the time or they’re not at school and
it’s hard for me to deal with this one . (Chuckle.) ”

“ It it's specific to American Ind ian or Alaska Native in living like ... the dynamics on a reservation or something like that, I don't know the answer to that. ”

“ That means that how much emphasis is being put in and to get the kids ready to utilize their way of living and resources to be better citizens in the future. However, when we say, “To what extent do your school curriculum emphasize,” if I just put the 4th grade as a block of learning time, then I’m talking
about the curriculum that we, all the schools have, the buy curriculum. But if our school has learning every day, that our school is an active school with activities related to their life, I think we are doing a better, a very good job in that. ” School #9 New/VH040496

To what extent do your school’s 4th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo, or village?

A. Not at all

B. Small extent

C. Moderate extent

D. Large extent

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 185 Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow

- up question]: Can you give me examples of how your school’s 4th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, puebl o or village?

There were eight responses to this question. Three administrators stated that they did not think that
their 4 th grade curricula did emphasize preparing AI/AN students for life in the different communities listed. Three said that their schools offered culturally relevant programming, and two brought up encouraging common core educational components.

Probe 5: What d oes the word “pueblo” mean to you?

Of the 11 responses given, eight felt that “pueblos” were related to American Indian people in the Southwest, one person (from Guatemala) said that it meant city, one said “house drum,” and another
said that it did not re ally have any meaning to them.

Probe 6: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

When asked if the item was easy or difficult to answer, four of the 12 respondents said it was easy, and eight felt th at it was difficult. Of those indicating that it was difficult, the reasons had to do with the fact that several respondents did not see “community” and “tribe” (among the other phrases) as being the same thing. For instance, many felt that the community r eferred to being among the wider community, whereas being with the tribe was a separate kind of community. Additionally, many respondents mentioned that the item’s emphasis on the 4 th grade curricula was problematic because they often do not emphasize cult ural learning that may happen at the school level. This is important as many curricula are approved at the district or state level.



## Solutions to Improve Item

Respondent: Expand examples on what is meant by “preparing youth for their community.”

Interviewer : NA


## Initial KAI Recommendation (Keep/Change/Discard)

Change: It seems that there was some confusion over the multiple terms to describe what were seen as different kinds of communities. The language around “curriculum” caused many respondents confusion

ov er how they should answer the question as it is currently worded. If the wording could be changed to be more inclusive around the curriculum piece then it should be kept. Consider removing the word “community” unless rephrasing it to “tribal community.”



## ET S Recommendation (05/02/2014)

Drop this item.

Rationale: ETS recommends dropping this item as respondents indicated that life in their community is not necessarily the same as life in their tribes. Respondents indicated that the school curricula may

succe ed in preparing students for life in their community, for instance, but not necessarily life in their tribe. In addition, not all students may have a “tribal life” and may live in a community without other

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 186 AI/AN families. Furthermore, sub

- items "b" and "c" may be more applicable to higher grades, as expressed by some of the respondents, therefore they may not be appropriate for grade 4.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 187 

# Survey Instrument: Administrator Grade 8/Sequence I

Item

Reading: NA

Understanding: NA

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 10 respondents , six administrators felt the question item asked them to ide ntify school type, three felt it asked about the school’s funding source, and one felt the question was trying to determine
whether or not the school was effective.

Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain why.

All 11 administrators who responded report ed needing only one answer choice.

Probe 4: In question e., what does the phrase “other nonpublic school” mean to you? Can you give me examples of “other nonpublic schools?”

Of the 11 responses to this probe, six pointed to religious or parochial schools, five focused on private
schools or institutions

, and two highlighted charter schools as types of “other nonpublic schools.” One
administrator cited alliance or alternative schools.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA School #1 Revised/VH040432

Which of the following describe s your school best? Please fill in one oval only.

A. Regular public school

B. Charter public school

C. Bureau of Indian Education contracted or grant school

D. Bureau of Indian Education operated school

E. Other nonpublic school

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 188 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising

or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 189 

## Item

Reading: NA

Understanding: None of the respondents demonstrated any difficulty understanding the question; however, one administrator pointed out that some lay/regular survey participants may not be familiar with the term “boarding school.”

Changing Answers: NA



## Probes

Probe 1: Ca n you tell me, in your own words, what the question is asking?

Of the 11 respondents , 10 administrators felt that the question asked whether the school boarded students/students lived at the school, while one felt that the question was looking at the type of school operated, specifically.

Probe 3: What does the phrase “boarding school” mean to you?

Seven administrators (out of 12 respondents ) defined “boarding school” in terms of providing students with housing or students living on campus or in dorms. Two referred to students living on campus or at the school “24/7,” while one felt that students in boarding schools lived independently while on campus. Three respondents made it a point to reference the historical context of boarding schools in AI/AN communi ties.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow

- up questions]: Are there any students at your school that do not board? If yes, what proportion of your students do not board? To what extent does having a high or low proportion of students who do not board influence the answer choice you selected?

Two administrators responded to this probe. The first reported a 60% boarding population, while the
other noted that 33 of the 460 enrolled students boarded. Neit her respondent felt that this proportion
influenced their answer choice selection.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA School #2 New/VH040433

Is your school a boarding school?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 190 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 191 

## Item

Reading: NA

Understanding: One respondent admitted to experiencing some difficulty determining the meaning of the phrase “on a reservation/on tribal land.”

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in y our own words, what the question is asking?

Of the nine respondents , six felt that the question was asking if the school was located on land owned by the tribe or within the boundaries of the reservation, while one felt that it was ask ing about which

group or entity had jurisdiction over the provision of educational services. Two administrators referenced special or complex circumstances in their responses:

“Yes, it was a reservation land originally here in Oklahoma, but because of the Treaty it still is id entified with us, but it’s not reservation land. It’s not set aside specifically anymore, just for the tribe its set aside for individuals. But you have to know that kind of in your head, because some people will say, ‘Yeah, I’m
on a reservation land.’ But in Oklahoma we are not.

“Up here, this could be a little confusing but I'm interpreting it whether it's on reservation or tribal land, which the answer would be no. The uniqueness of Wellpinit is we're on a reservation but we're on publics land within the reservation.”

Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?

Nine administrators (out of 12 respondents ) felt that “ reservation or tribal lands ” referred to reservations, N ative

- owned land

, or federal land set aside for tribes. One felt that it referred to land where taxes were not paid on the land, while one other administrator felt that the term was nested within a historical definition. One admitted to being unsure of the definition of the phrase:

“That’s where I’m struggling. I know what a reservation is but I thought it was the same thing. Me being

a non

- N ative maybe that’s why I don’t know.”



## Solutions to Improve Item

Respondent: NA

Interviewer: NA School #3 New/VH040437
Is your school located on a reservation/on tribal land?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 192 

## Initial KAI Recommendation (Keep/Change/Discard)

Change: While none of the respondents demonstrated difficulty responding to the question, a few noted situations where the presence of trust lands could make it more challenging to answer. Consider adding “trust land” to the list of tribal land options within the text of the question.

Another respondent noted that nontribal members or individuals unfamiliar with the concept or definition of tribal lands/reservations may experience confusion or difficulty when answering the question.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 193 

## Item

Reading: NA

Understanding: One respondent t hought the question was asking about American Indian and Alaska Native languages separately. A second administrator demonstrated some difficulty interpreting the phrase “immersion program.”

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 11 respondents , eight felt that the question was trying to ask if an AI/AN language was used all day or included in the general school curriculum. Two administrators felt that the question only asked if the school ran an immersion program, while one felt that the questi on wanted to determine if staff and students spoke a N ative language instead of simply teaching it.

Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school” mean to you?

Seven administrators (out of 12 respondents ) fe lt that an immersion program incorporated an AI/AN language throughout the day and throughout course curriculum. One felt that the term meant that the school had certified teachers to provide appropriate language immersion instruction. Five of the
administ rators discussed current or past plans to implement an immersion program in their responses.

Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow

- up question]: What proportion of students in your school participates in your schools’ language immersion program?

NA



## Solutions to Improve Item

Respondent: NA

Interviewer: NA School #4 New/VH040477

Is your school an American Indian or Alaska Native language immersion school?

A. Yes

B. No

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 194 

## Initial KAI Recommendation (Keep/Change/Discard)

Keep: As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 195 School #5 Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the following ways? Fill in one oval o n each line.

Yes No

a. Academic club (for example, math club) A B Revised/VH040576

b. Arts club A B Revised/VH040577

c. Cultural dances or activities (for example, drum groups) A B Revised/VH040578

d. Drama club A B Revised/VH040580

e. Making school curriculum decisions A B Revised/VH040581

f. Open houses or back

- to

- school nights A B Revised/VH040584

g. Parent

- teacher conferences A B Revised/VH040588

h. Parent

- teacher organizations A B Revised/VH040589

i. School sports A B Revised/VH040592

j. Volunteer programs A B Revised/VH040596

k. Other (please specify): _____________________________ A B Revised/VH040597



## Item

Reading: NA

Understanding: One respondent appeared to equate references to clubs and activities with actual
school course activities.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the 10 respondents , eight felt that the question was asking about types and levels of family involvement, while two felt that the question asked about the opportunities for involvement offered to
families.

Probe 3: Are your answer choices for questions a. through k. specif ic to the families of your grade 8 American Indian or Alaska Native students, or do they apply to the families of all American Indian or Alaska Native students in your school?

Six administrators (out of nine respondents ) referenced all families when answer ing. Two referenced only the families of 8 th grade students

, and one limited their answer to AI/AN families.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 196 Probe 4: In question a., what does the phrase “academic club” mean to you?

When asked about academic clubs, eight out of 10 respondents felt that the term referred to academic

or course

- specific groups; one felt that it referred to a group for gifted or talented students; and one felt
that it referred to a type of peer

- based mentoring.

Probe 5: In question b., what does the phrase “arts club” mean to you?

Six of the 10 administrators who responded felt that the item referred to any visual or performance art. Three administrators, rather than defining the term, described past or present examples of arts club s at the school, while one simply noted tha t the school did not offer such a n activity.

Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you?

Ten administrators responded to the probe. Of these, two made general references to powwows and drum groups, while four cited specific examples of cultural activities offered at or by the school. Three
administrators discussed the traditional and cultural value of such activities in their responses. One respondent defined the probe item as visual or performance art.

Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the
follow

- up question]: Can you give me examples of cultural dances or activities? Do you consider “drum groups” a good example of “cultural dances or activities”?

Of the nine respondents , five described powwows and drum groups offered by or at the school. Two respondents described specific types of dance taught to students, while one made a more general reference to teaching students traditional types of dance. One administrator failed to provide an example, simply noting that drum groups are a good example of traditional activity. Three other administrators similarly noted t hat drum groups are, in fact, a good example.

Probe 8: In question d., what does the phrase “drama club” mean to you?

All but one administrator (out of 10 respondents ) described “ drama club ” in terms of acting, plays,
stage
, or theater. The 10 th administr ator’s response described the school’s plans to develop a student drama group.

Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?

Of the 11 respondents , six described the probe item as parental involvement or i nput into curriculum or course decisions. Three administrators felt it referred to a specific council or committee that included parents. One administrator’s response defined it simply as parental feedback on school curriculum,
while one other administrato r noted that only the s tate drove curriculum

- based decisions for the school.

Probe 10: In question f., what does the phrase “open houses or back

- to

- school nights” mean to you?

Ten of the 11 respondents to this probe defined this probe item in terms of a s pecific school event where parents met or interacted with teachers and school staff. One administrator referred to it as a type of public relations activity.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 197 Probe 11: In question g., what does the phrase “Parent

- teacher conferences” mean to you?

All 10 responses described “ parent

- teacher conferences ” as a meeting between parents and teachers (both formal and informal in nature).

Probe 12: In question h., what does the phrase “Parent

- teacher organizations” mean to you?

Three administrators (out of 10 respondents ) specifically referred to PTAs or PTOs in their responses, while five made more general references to some type of parent

- teacher committee. One respondent described a past example of this type of organization, while two others noted that no su ch organization currently existed.

Probe 13: In question i., what does the phrase “school sports” mean to you?

Of the 10 responses describing “ school sports, ” six referred to a specific sport (e.g., football, basketball, and track) and four referred to sp orts as a collective, general category.

Probe 14: In question j., what does the phrase “Volunteer programs” mean to you?

Six administrators (out of 10 respondents ) described “ volunteer programs ” as occasions when parents or family members provided assista nce to the school. Two administrators described specific programs or examples in their responses, while two others noted that no such programs currently exist.

Probe 15: What proportion of your students are American Indian or Alaska Native? To what extent does having a high or low proportion of American Indian or Alaska Native students in your school influence the answer choice you selected?

Of the nine total responses given, one administrator reported a proportion of 2%, one a proportion of 20%, and one a proportion of 33%. Five administrators reported a proportion of 95% or greater. Two respondents noted that these proportions had no influence over their answer selection. One respondent found this probe very confusing and needed to re read it before respond ing, while three other respondents found the probe too confusing to answer.

Probe 16: [Conditional probe: If the participant selects answer choice A for question k, then ask the
follow

- up question]: What are the other ways in which families of your America n Indian or Alaska Native students are involved with your school ?

Of the seven responses provided, two described alumni activities, one described families serving as chaperones or assisting with supervision during field trips, and four described more miscellaneous activities (e. g. , cheerleading, band, and archery contests).



## Solutions to Impr ove Item

Respondent: One respondent noted that not all question row items were available or offered at the
school.

Interviewer: NA

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 198 

## Initial KAI Recommendation (Keep/Change/Discard)

Change: Several respondents noted that some of the question rows listed did not apply as they were not
available at the school. Consider adjusting the answer choices to include an option for “N/A” or “Do not offer/Not available.” Also specify and emphasize if the question focuses on all AI/AN families or just the
families of 8 th grade students.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 199 School #6 Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions and culture are offered at your school each year? Fill in one oval on each line.

None 1 2 3 4 or

more

a. Year

- long course A B C D E Trend/VC962954

b. Semester

- long course A B C D E Trend/VC962956

c. Workshop or unit A B C D E Trend/VC962957

d. Clubs A B C D E Trend/VC962958

e. Other programs (for example, study groups before or after regular class periods) A B C D E Revised/VH018332



## Item

Reading: NA

Understanding: Two respondents expressed some confusion when reading and answering the question item, one because the answer choices are not specifically offered at the respondent’s school

, while the other could not understand the question’s meaning.

Chan ging Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One respondent (out of 12 respondents ) was not sure what the question was trying to ask. Seven administrators felt the question was asking about AI/AN culture

- specific classes and courses. Two administrators felt that the question explored the types of cultural programming offered at the scho ol, outside of a traditional curriculum, and one thought the question wanted to know how culture was taught at the school. One respondent thought the question was asking about t he i r own level of knowledge of N ative culture.

Probe 3: What does the phrase “ study groups before or after regular class periods” mean to you?

One of the 11 respondents was unsure of the definition of the probe item. Four defined the phrase in terms of activities taking place outside of the classroom or regular school day. Three res pondents described it as a formal group or tutoring program. Three administrators described the phrase as topic

related discussion groups ( one ), independent study ( one ), or follow

- up study or instruction ( one ).

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 200 Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E for question e., then
ask the follow

- up question]: Can you give me examples of other programs offered at your school?

The four responses to the probe referred to the Century 21 program ( one ), the MEFE ( one ), a program involving NASA ( one ), and a professional performance group ( one ).

Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of
other answer choices that would be more clear?

All 11 administrators who responded noted that the answer choices seemed clear.



## Solutions to Improve Item

Respondent: One administrator suggested rephrasing some of the question rows to read more clearly:

"That jus t meant a year

- long course ... or and programs so it's really, is there a specific course, is there a specific program that they're talking about?"

Another respondent suggested separating out questions, as the current format can be confusing and overly com plicated to answer

.

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Change: Consider provid ing examples of or definitions for the question row items. This should help alleviate some of the respondent confusion. Also specify if these courses are school

- wide or 8 th grade

specific.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS rec ommends revising this item to a more simplified format by collapsing the answer choices to “Yes” and “No”. The answer choices for this existing item caused confusion among respondents. Also, response frequencies indicate that the strong majority of the res ponses fall under the following two out of the f ive categories: “None” or “1”. As a result of changing this item to a “Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs
about American Indian or Alaska Native t raditions and culture offered at your school each year?”.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 201 School #7 Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your
scho ol? Fill in one oval on each line.

Not at
all Small

extent Moderate

extent Large

extent

a. Student absenteeism A B C D Trend/VC963030

b. Student tardiness A B C D Trend/VC963031

c. Student health problems A B C D Trend/VC963032

d. Teen pregnancies A B C D New/VH040470

e. Drug or alcohol use by students A B C D Trend/VC963033

f. Student misbehavior in class A B C D Trend/VC963034

g. Physical conflicts among students A B C D Trend/VC963035

h. Bullying A B C D Trend/VC963037

i. Low student aspirations A B C D Trend/VC963038

j. Low teacher expectations A B C D Trend/VC963039

k. Low family involvement A B C D Trend/VC963040

l. Inadequate transportation for
students A B C D New/VH040475



## Item

Reading: NA

Understanding: One respondent was unsure of the time frame covered by the question. This same individual later interpreted the question as specific to AI/ANs rather than to the entire school.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One administra tor (out of eight respondents ) answered in reference to specific answer rows rather than
the entire question. Five administrators felt the question was asking about the types of problems or issues experienced at the school. Two administrators thought the question focused on issues that specifically inhibited students’ ability to learn or succeed i n school. One respondent felt that the question was trying to explore the school’s management abilities.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 202 Probe 3: In question a., what does the phrase “student absenteeism” mean to you?

Of the 12 respondents , 11 defined the probe phrase in terms of th e student being away from or out of school without an excuse. One respondent described it as a lack of parental support.

Probe 4: In question b., what does the phrase “student tardiness” mean to you?

Ten administrators (out of 11 respondents ) described “ t ardiness ” as being late for school or class. One administrator described a circumstance where the student’s physical distance from the school could impact his or her ability to come to school on time.

Probe 5: In question c., what does the phrase “student health problems” mean to you?

All 11 respondents referred to the phrase in terms of a health or medical condition. Among these responses, two specifically referred to diabetes, one referred to lice, one to men tal health issues, one to
the flu

, and one to any condition requiring medication.

Probe 6: In question e., what does the phrase “drug or alcohol use by students” mean to you?

Twelve administrators responded to this probe. Of these, seven defined the phra se as the use or abuse of drugs, alcohol

, or other illegal substances. Four administrators described the prevalence of the issue in their school setting; one described a specific instance of addressing the issue with students. One administrator’s response described the school’s response to the issue in general.

Probe 7: In question f., what does the phrase “student misbehavior in class” mean to you?

Of the 11 respondents

, 10 described the phrase as some kind of problem behavior or misconduct. One
administ rator defined it more specifically as behavior resulting in some kind of formal discipline.

Probe 8: In question g., what does the phrase “physical conflicts among students” mean to you?

Ten of the 12 respondents described the phrase as physical fighting or harm done to other students, with one other respondent detailing a specific instance of physical conflict that took place in the school. One administrator’s response was that the item was not an issue.

Probe 9: In question h., what does the phrase “bul lying” mean to you?

The 12 respondents to this probe had varying definitions of bullying. Five described it as harassing another student or causing another student stress or harm. Four described specific examples of bullying, with one other noting that bu llying is something that happens more than once (i.e., the behavior is repetitive). Three defined it in terms of one student overpowering another, while two respondents
noted that the term itself is difficult to define:

“I hear this all the time, large ex tent. I think it may be due to a definition problem. They just don't understand the definition of it. So any type of verbal words looking at.”

“This one is so hard. Everyone got their own definition for it. I would say people lump a lot of things into
bull ying that may not be...”

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 203 Probe 10: In question i., what does the phrase “low student aspirations” mean to you?

Eleven of the 12 respondents defined the probe item as students lacking goals, direction

, or motivation. One respondent ’ s answer described a specific example of low student aspirations.

Probe 11: In question j., what does the phrase “low teacher expectations” mean to you?

Seven administrator responses (out of 12 respondents ) consisted of general descriptions of teacher expectations. Four admin istrator responses described the high levels of teacher expectations present at their schools. One response was difficult to code:

“Well, these kids just come from the reservation and they haven’t been in schools, so they’ll probably not going to help my scores in any ... I mean instead of ... I’m a firm believer that kids will do what you expect them to do and I always think of the s ame few teachers that all year, they always came to mind and just ... I don’t know, I’m just kind of almost borderline, just don’t care.”

Probe 12: In question k., what does the phrase “low family involvement” mean to you?

Of the 12 responses to the probe , nine discussed low family involvement in general terms

, and two described how the school currently struggled with low involvement. One administrator had difficulty responding to the probe:

“For some reason when they get out of elementary we lose them at the middle school level usually,
because in 8 th grade they are involved in the sports and the other activities. So I don’t get a lot of family involvement, but again you need to say, ‘Is that involvement in the school or is it involvement in our program?’ So that might be... You might want to say something about you know are they involved in the school or... Because they may be involved in our program, but not in the school. You see what I mea n? They might come to my meetings, but they won’t go to the parent

- teacher meetings at the school. So I don’t know. That question was just a little... You might want to be a little more specific on that one.”

Probe 13: In question l., what does the phrase “inadequate transportation” mean to you?

Six administrators (out of 10 respondents ) described the probe items as a lack of transportation to or from the school. Two administrators noted that inadequate transportation was not an issue for their school, whil e two others described it as an issue involving students’ parents.

Probe 14: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then
ask the follow

- up question]: Can you give me examples of how teen pregnancies are a p roblem in your school?

Of the five administrators who responded , two noted that it is not a current problem but went on to describe past experiences with teen pregnancy. Three other administrators described how the issue impacted students’ ability to succe ed in school.



## Solutions to Improve Item

Respondent: One respondent requested more clarification on the time frame of the question (e.g., t his school year? m ultiple school years?)

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 204 Interviewer: Consider rephrasing the question about teen pregnancies to focus on how they impact schools rather than how they are a "problem.”



## Initial KAI Recommendation (Keep/Change/Discard)

Keep: Question seems generally understandable; however, potential changes co uld include specifying the time frame to which respondents should refer when answering, and rephrasing “teen pregnancy” to seem less stigmatic. Additionally, consider whether or not answers are wanted for the whole school, which could include up to 12 th gr ade, or just for 8 th grade students.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS recommends revising this item to clarify that it is specificall y asking about AI/AN
students.

This recommendation is the same at grades 4 and 8.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 205 School #8 Trend/VC963041

Which types of student and family services are offered at your school on a regular basis? Fill in one oval on
each line.

Offered Not
Offered

Health services for students such as a school nurse A B Trend/VC963042

Health services for families such as a community clinic A B Trend/VC963043

Social or counseling services for students A B Trend/VC963045

Social or counseling services for families A B Trend/VC963047

Adult education programs for all ages A B Trend/VC963048

Tutoring services for students A B New/VH017944

Career counseling services for students A B New/VH017945



## Item

Reading: NA

Understanding: One respondent found the language in question row B confusing.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Eight of the 10 respondents felt that the question tried to identify the types of services or programs available to students and families. Two administrators limited their responses to r ow A specifically.

P robe 3: What does the phrase “a regular basis” mean to you?

Of the 10 respondents , nine administrators referred to a recurring schedule or regular time. One administrator simply referred to things taking place “on average.”

Probe 4: In question f., what d oes the phrase “tutoring service for students” mean to you?

Of the 11 administrators who responded , five defined tutoring services as the provision of academic support, five specifically described such services as taking place outside of regular school ho urs and three mentioned specific examples of types of tutoring services currently provided.

Probe 5: [Conditional probe: If the participant selects answer choice A for question f., then ask the
follow

- up question]: Can you give me examples of the tutoring services offered at your school?

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 206 Seven of the 11 respondents referred to generic tutoring services that provide students with additional academic support , and three specifically mentioned the Century 21/21 st Century tutoring program

.

Probe 6 : In question g., what does the phrase “career counseling services for students” mean to you?

Six administrators (out of 11 respondents ) referred to guidance or decision

- making assistance about
students’ futures

, and one spoke of how it informs students of options outside the reservation. Four administrators described specific programs or services provided by the school.

Probe 7 : [Conditional probe: If the participant selects answer choice A for question g., then ask the
follow

- up question]: Can you give me examples of the career counseling services offered at your
school?

Of the five responses provided , career counseling was described as career days ( four ), staff counselors ( four ), college visits or partnerships ( three ), and bringing in outside speakers or programs ( four ).



## Solutions to Improve Item

Respondent: One respondent suggested adjusting the question language to accommodate alternative terms, such as "school nurse."

Interviewer: NA


## Initial KAI Recommendation (Keep/Change/Discard)

Keep : As is.



## ETS Recommendation (05/02/2014)

Keep this item.

This item performed well in cognitive interviews. No issues were identified that would warrant revising or dropping this item.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 207 

## Item

Reading: NA

Understanding: One administrator expressed some confusion over the meaning of the phrase “dropping out” ( i.e., Does it refer to a student leaving school or leaving the entire school system?)

. Another administrator questioned the time frame referenced by “last school yea r” ( i.e., Does it refer to a calendar year or the student’s last year in school?)

. Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

One administrator (out of nine respondents ) noted that t he y lacked su fficient data to accurately answer the question. Five administrators felt that the question was asking about dropout rates for the school. Two thought the question was asking them to define the problem, while one administrator felt that the question was as king for reasons why students drop out.

Probe 3: Please explain why you selected the answer choice. In other words, how did you calculate
the answer?

While four of the 11 respondents simply stated that student dropouts were not an issue in their schools ( thus, the answer selection would be A ), four other respondents conducted a general count of students still in school. Three administrators, though, noted that they lacked the information to come up with an
accurate calculation.

Probe 4: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?

Of the 11 responses to the probe, five respondents found the question easy to answer, five found it difficult, and one found it in between easy and hard to answer. When describ ing why, three respondents School #9 Revised/VH040486

What percentage of your 8th grade students dr opped out of school during the last school year?

A. 0

– 2 %

B. 3

– 5 %

C. 6

– 10 %

D. 11

– 20 %

E. 21

– 30 %

F. 31

– 40 %

G. 41

– 50 %

H. More than 50 %

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 208 cited the phrasing of the question, while two noted they lacked the information to calculate an answer. One respondent explained that the question was easy to answer because student dropouts were not an
issue at the school.



## Solut ions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Keep (with reservations): Question generally seems fine as is. However, given the reservations expressed by multiple respondents regarding the lack of info rmation (affecting their ability to accurately calculate a figure for their response), there could be concern regarding the reliability of respondents’ answers and the validity of the question item. If the question is meant to assess how many AI/AN
student s dropped out, consider adding that in and highlighting it.



## ETS Recommendation (05/02/2014)

Revise this item.

Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking about AI/AN students.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 209 School #10 New/VH040507

To what extent do your school’s 8th grade curricula emphasize preparing American Indian or Alaska Native students for the following? Fill in one oval on each line.

Not
at all Small

extent Moderate

extent Large

extent

a. Prepare students for life in their community, tribe, group, pueblo, or village A B C D New/VH0

40511

b. Prepare students for the workforce A B C D New/VH0

40518

c. Prepare students for high school and college A B C D New/VH0

40522



## Item

Reading: NA

Understanding: One respondent needed to reread the question item a few times in orde r to understand the phrasing.

Changing Answers: NA



## Probes

Probe 1: Can you tell me, in your own words, what the question is asking?

Of the nine respondents , six felt the question was asking about how the school generally prepared students. One respondent felt that the question was asking about teacher capabilities, while another respondent thought it was asking if the school had high expectations for its stu dents. One respondent was confused by the question and thought it could be interpreted in multiple ways:

“I don’t know what they mean by “I prepare students for life.” I mean, if we’re preparing students for a community that has 80% unemployment and no jo b opportunities, then we’re doing a good job because a lot of our students don’t have opportunities when they graduate high school. But are we preparing them to be active participants in their community’s culture? That’s different then ... I guess without an y qualifying awards or statements, every single kid is being prepared to be part of their community, but it’s like, what kind of preparation are they coming with? Are they prepared ... Does that make sense? I would interpret it ...”

Probe 3: In question a., wh at does the phrase “prepare students for life in their community, tribe, group, pueblo, or village” mean to you?

Three of the 11 respondents felt that the students were prepared for community life, but not tribal life. Ten administrators felt that the prob e item referred to general preparation for the work and behaving appropriately in the community.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 21 0 Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question a., then ask
the follow

- up question]: Can you give me examples of how your school’s 8th grade curricula emphasize preparing American Indian or Alaska Native students for life in their community, tribe, group, pueblo or village?

Four administrators (out of 11 respondents ) noted that their school curricula did not prepa re students for life in their tribe. Six administrators felt that general school curricula and teacher interactions prepared students, while two administrators discussed how students take part in community service activities or otherwise engaged with the c ommunity. One administrator’s response proved difficult to
code:

“We're going to Reno for about two weeks in March to a summit that I've gone to for about the last five years. When I read this though, I think in like history or in language arts, are they getting stories that belong to them or to the American Indian culture? I still think that has a long way to go. That's my goal, is that why wouldn't we take time to talk about this specific tribe in this area and their history when that
would make them hav e ownership, be a part of ... validate them.”

Probe 5: What does the word “pueblo” mean to you?

Six administrators (out of nine respondents ) defined pueblos in terms of geographic location, particularly the southwest. Four administrators noted that the ter m mean s “village” or “community,” and three highlighted that the term was specifically N ative in origin and reference.

Probe 6: In question b., what does the phrase “prepare students for the workforce” mean to you?

Of the 12 respondents , four defined the p robe item as general preparation for the work place, two provided specific examples of how the school prepared students, and five discussed how student preparation included development of math, reading, speaking, reasoning, and social skills. One responden t’s response was difficult to code:

“Well, not you're getting into personal beliefs of mine where I think, I'll just go back to begin with the end in mind ... I'm getting philosophical in here, but a lot of times we're not thinking about the end game and what it is that we want them to know and be able to do and I really dislike how sometimes we don't make anything apply to different situations. I don't know if you saw the visuals out there, did you see any of those out there?”

Probe 7: [Conditional probe: If the participant selects answer choice B, C, or D for question b., then
ask the follow

- up question]: Can you give me examples of how your school’s 8th grade curricula emphasize preparing American Indian or Alaska Native students for workforce?

Six of th e 10 respondents reported that school courses and curricula prepared students for the workforce. Three respondents described the use of team work or a general discussion of student preparation when answering. One administrator’s response was difficult to c ode:

“I work with Fredina Drye

- Romero and she is the American Indian Consultant there in Reno. Her mom actually lives on the reservation. I went to a thing in Vegas the other day and I've gone on five different American Indian college tours in different s tates from Clark County with kids checking out colleges. But as far as this school, this is my first year here and already, like I said in there, it's probably more so than

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 211 your average principle, we've had the student being in boarding school presentation and he came here taking the 7th grade US Nevada history teacher.”

Probe 8: In question c., what does the phrase “prepare students for high school and college” mean to you?

All 11 administrators who responded said that the probe item referred to the academic preparation of students for graduation.

Probe 9: [Conditional probe: If the participant selects answer choice B, C, or D for question c., then ask
the follow

- up question]: Can you give me examples of how your school’s 8th grade curricula emphasizes preparing American Indian or Alaska Native students for high school and college?

Ten administrators responded to the probe item. Of these, six cited college preparation coursework or curricula, three cited the u se of outside speakers or college prep programs, and one described general discussions that staff had with students about preparing for high school and college.

Probe 10: Which answer choice would you select for questions a. through c. if the curricula is focused on high school only?

Two of the 10 respondents found the question confusing. Three administrators reported that their answer would not change, while four stated that their answer would. One administrator’s response proved difficult to code:

“I mea n, you really have to let them know that you do have high expectations and they've got to do better than what they're doing in high school.”

Probe 11: Do you think the question is easy or difficult to answer? Why is it easy or difficult to
answer?

Ten admi nistrators responded to the probe item. Of these, five found the question easy to answer, while four found it difficult. One administrator responded that the question was in between (i.e., both easy and difficult to answer.) Four respondents explained that the question’s phrasing is specific to high school rather than middle or elementary school, while one commented on the complex and interrelated nature of the various elements within the question.



## Solutions to Improve Item

Respondent: NA

Interviewer: NA



## Initial KAI Recommendation (Keep/Change/Discard)

Change: Several respondents noted that school curricula succeeds in preparing students for life in the community but fails in preparing them for life in their tribes. Consider separating this into two different questions, as it examines two different (though admittedly intertwined) aspects of a student life.

## ETS CogLab Report: NIES | 2015

ETS Post

- Cognitive Interview Item Recommendations Page 212 

## ETS Recommendation (05/02/2014)

Drop this item.

Rationale: ETS recommends dropping this item as respondents indicated that life in their community is not necessarily the same as life in their tribes. Respondents indicated that the school curricula may succeed in preparing students for life in their community, for instance, but not necessarily life in their tribe. In addition, not all students may ha ve a “tribal life” and may live in a community without other AI/AN families. Furthermore, sub

- items B and C may be more applicable to higher grades, as expressed by some of the respondents, therefore they may not be appropriate for grade 8.