### Appendix 4

Cognitive Interview Results

This appendix contains the following reports and memos that provide results from the NHES:2016 cognitive interviews:

Cognitive Interview Findings for the Pare nt and Family Involvement and Early Childhood Program Participati on Surveys 2016: Final Report

Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2016: Draft Report

Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews: Memo

Preliminary Findings from the 2015 Spanish Cognitive Interviews: Memo

# Cognitive Interview Findings for the Parent and Family Involvement and Early Childhood Program Participation Surveys 2016: Final Report

Prepared for the National Center for Education Statistics (NCES) by

Susan Cole, Raquel Gonzalez, Kaitlin Fronberg, and Suzette Chavez

May 2015

## Contents

1. Introduc tion .......................................................................................................... 4

1.1 Cognitive Intervie w Methodology .......................................................................... 5
1.2 Sample .................................................................................................................... ... 5

1.3 Data Collection and Analytic Approach ................................................................ 8

Data Collection

.............................................................................................................................. 8

Data Analysis Methods

................................................................................................................. 9



### 2. Findi ngs ................................................................................................................. 9

2.1 Screener It ems .......................................................................................................... 9

2.2 PFI Items ................................................................................................................. 10

2.2.1 Child’s Schooling ....................................................................................................... .. 10

School Assignment ............................................................................................................. ............ 10

Advanced Placement ............................................................................................................ ........... 11

Internet Courses .............................................................................................................. ................ 12

Homeschooling ................................................................................................................. .............. 14

Other Items in the Child’s Schooling Section ................................................................................. 17

2.2.2 Child’s Health .......................................................................................................... ..... 18

Services ...................................................................................................................... ..................... 18

Other Items in the Child ’s Health Section ..................................................................................... . 20
2.2.3 Child’s Background ...................................................................................................... 20

Joint Custody ................................................................................................................. ................. 20

Language at Home

........................................................................................................................... 21

Other Items in the Child’s Background Section .............................................................................. 22

2.2.4 Your Household .......................................................................................................... . 22

Other Items in the Your Household Section ................................................................................... 2 4
2.2.5 Child’s Family .......................................................................................................... .... 25

### 2.3 ECPP It ems ............................................................................................................. 25

2.3.1 Finding and Choosing Care for Your Child ................................................................. 25

Difficulty Finding Child Care ................................................................................................. ........ 25

Importance of Reasons for Choosing Child Care ........................................................................... 26

Ratings from a Website

................................................................................................................... 26

Early Childhood Program Curriculum ............................................................................................ 28

Other ECPP Items .............................................................................................................. ............. 29 

### Appendix A

......................................................... .................................................... 

### 30

### 1. Introduction In 2016, the National Household Education Survey s Program (NHES) will field the Parent and Family Involvement in Education Survey (PF I) and the Early Childhood Program Participation Survey (ECPP). This report summarizes the findi ngs from cognitive interviews conducted to test and revise a subset of items from the PFI survey based on potential issues with a few items that
were observed in the 2012 collection and th e 2014 Feasibility Study. This report also summarizes findings from a series of new and re vised ECPP items that cover topics suggested by the Office of Nonpublic Education (ONPE) and th e Policy and Program Studies Service (PPSS) in consultation with the Office of Early Lear ning. The report also includes findings from cognitive testing of th e household screener.

In order to reduce the responde nt’s cognitive and time burden, th e interviews included only the sections of the NHES questionnaires that contai ned items requiring testing. The objective of the cognitive interviews in 2015 was to identif y and correct problems of ambiguity or misunderstanding in item wording that were ev ident from NHES:2012 data. The research was iterative, in that item wording and format desi gn changed during the testing period in response to early findings. The cognitive interviews should result in a set of questionnai res that are easier to
understand and therefore less burde nsome for respondents, while also yielding more accurate
information.

A total of 67 items underwent cognitive testing, including 60 items from the PFI survey and 7 items from the ECPP survey. The items from the PFI survey that were tested included items about the following topics:

 School assignment

 Advanced Placement courses

 Internet courses

 Homeschooling

 Joint child-custody arrangements

 Language spoken by the child at home

 Your household

 Child’s family

The items from the ECPP questionnaire that were tested in cognitive interviews included items about the following topics:

 Difficulty finding child care

 Importance of reasons for choosing child care

 Early childhood pr ogram curriculumThe selected survey items were tested with me mbers of the appropriate target respondent group
to determine their understa ndability, the need for revision, and fo r retention in or exclusion from the surveys. The target respondent groups are described in section 1.2.

Interviews were conducted betw een January 20 and March 2, 2015.



### 1.1 Cognitive Interview Methodology

Each cognitive interview lasted approximatel y one hour and was conducted using primarily a “think-aloud” approach with conc urrent probing as needed, base d on a structured protocol. The methodology was developed by a senior researcher at AIR in consultation with NCES and drawing on best practices and methods from cogn itive science (see Appendix A for the protocol). These interviews were designed to identify probl ems of ambiguity or misunderstanding in item wording. The interviews were conducted by interv iewers who received tr aining in the cognitive interview technique and in the in terview protocol by a senior researcher at AIR. Researchers
from AIR and NCES listened to a subset of interviews and provided guidance and feedback throughout the study.

During each interview, the study participant was welcomed by the interviewer and led to an interview room. The participant was then told th at the purpose of the interview was to help answer questions about how peopl e respond to items on schooling. A ll participants were assured that their participation was voluntary and that their answers would be used only for research purposes and not disclosed, or us ed, in identifiable form for a ny other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the think-aloud process (see below) and modeled a think-aloud practice with a sample question. Participants were then given a paper copy of the household screener and questionnaire to fill out.

The interviews were conducted using “concurrent probing,” a technique in which interviewers asked respondents to “think aloud” as they answ er the survey items. The cognitive interviews also included a verbal probing component, conduc ted after completion of the think-aloud portion for a given questionnaire. The verbal probes included a combination of pre-planned, itemspecific questions identified before the session as important and ad hoc questions that the interviewer identified as important from observations during the interview, su ch as clarifications or expansions on points rais ed by the participant.



### 1.2 Sample

Cognitive interview respondents were recruited through local recruitment firms in San Mateo, CA, the Washington, DC metro area, and Waltham, MA and through additional interviewer outreach to homeschooling parents. Respondents were given a $40 American Express gift card for their participation. The re cruitment criteria included:

 15 parents or guardians of a child enrolled in public school and 7 in private school; 12 parents or guardians of a child enrolled in grades K-5 and 7 enrolled in grades 6-12;

 12 parents or guardians of a child w ho is in a joint custody arrangement;

 12 parents or guardians for whom English is their second language but who can read English fluently;

 6 parents or guardians of a child who is homeschooled;

 9 parents or guardians who also have a child or children ages 0 to 5 and not yet in kindergarten and also have a care arrangement for the child;

 6 parents or guardians of a child who receiv es instruction from sc hool over the internet; and

 10 parents or guardians with a high sc hool completion or less education.

Note: The sum of the categories listed is greater than the total because several respondents are expected to fall into multiple categories.

A total of 51cognitive interviews were conducted across the three locations . We sought to recruit a diverse set of respondents; however the respon dent characteristics repo rted during recruitment did not always match the respondent’s reported ch aracteristics during the cognitive interviews. Table 1 provides more detailed informati on on the recruitment and interview reported characteristics of the respondents.Table 1. Number of participants in PFI/ECPP c ognitive interviews, by selected recruitment and interview characteristics

Reported Characteristics from Recruitment Reported Characteristics from Cognitive Interviews

Total Respondents 1 51 51

Child’s	Grade

K	‐	5	 14 14

6	‐	12	 32 36 Type	of	School 2
Public	 36 38

Private	 12 11

Homeschool 3,4 5	 6 English	as	a	Second	Language 1

Yes	 8 7

No	 43 43 Highest	Degree High	School	Diploma	or	Less	 5 6 More	than	just	High	School	 46 44 Joint	Custody	Arrangement 5

Yes	 11 10

No	 40 43
Internet	Instruction	for	Credit 6

Yes	 10 8

No	 41 42

Second	Child	Under	5

Yes	 9 9

No	 42 42

Location

DC	 16 16

CA	 25 25

MA	 10 10

1 One	respondent	only	filled	out the	ECPP	items	on	the	questionna ire	as	he/she	did	not	have	a	second	child	K‐ 12;	n=50	for	the	PFI	items

2 Four	respondents	also	indicated that	their	child	attended	a	pu blic	or	private	school.

3 Two	respondents	were	former	home schoolers	and	filled	out	the	sc reener	and	questionnaire	as	if	they	were still	homeschooling	their	child.

4 One	respondent	believed	that	“t utoring”	was	a	form	of	homescho oling.

5 Some	respondents	were	confused	a bout	the	meaning of	joint	custo dy.

6 For	Internet	courses,	there	was initially	a	discrepancy	between the	language	on	the	recruitment	screener and	the	survey.Table 2 provides information on the total numbe r of respondents who were scheduled for an interview, the number of interviews that were completed, and the cancellations and no shows by demographic characteristics.

Table 2. Number of interviews scheduled, comp leted, cancellations and no shows, by demographics

Demographics Interviews Scheduled Interviews Completed Cancellations No Shows

Child’s	Grade
K	‐	5 29 14 8 7
6	‐	12 59 36 10 13 Type	of	School
Public 69 38 12 19
Private 16 11 3 2 Homeschool 11 5 6 0 English	as	a	Second	Language 10 7 1 2 High	School	Diploma	or
Less 6 1 7

Joint	Custody	Arrangemen t 21 10 3 8 Internet	Instruction	for	Credi t 16 8 4 4 Second	Child	Under	5 16 9 5 2
Location
DC 49 16 18 15
CA 34 25 3 6
MA 12 10 1 1
TOTAL 95 51 22 22

Note: The	sum	of	the	categories	listed	is	greater	than	the	total	bec ause	several	respondents	fall	into	multiple categories.



### 1.3 Data Collection and Analytic Approach

Data Collection Respondents were asked to complete a paper vers ion of the screener a nd questionnaire; while doing this, they were asked to “think aloud” when responding to the items. After the respondent completed an item of interest, the interviewer asked the respondent to stop and asked probing questions about the item as need ed. If the respondent had a probl em or issue with items other than the items of interest, the interviewer aske d probing questions about those items. While the respondent filled out the screener and questionnaire, the interviewe r took handwritten notes in an interview protocol that contains the items and probes for the cogni tive interview. The screener and questionnaire were kept by th e interviewer as a r ecord of the respondent ’s responses to the items. Once the interview was completed, the interviewer entered notes in an electronic note template. This template is a standardized format for qualitative data entries, which allowed the analysis team to process and analyze the data in NVivo, a qualitative data processing softwaresystem. The interviewer then saved the interview notes and an audio recording of the interview in a secure server for analysis.

Data Analysis Methods
Once the interview notes were ente red, a senior researcher checked the data to ensure that the
most accurate information was included about th e errors, issues, and concerns that respondents had about the items in the questionnaire and screen er. After the interview notes were checked for quality, all items of interest as well as any it ems that respondents had difficulty answering were coded into the following categories:

a. No problems for questions of interest
b. Respondent was hesitant in answering the question

c. Respondent self-corrected his or her answer
d. Respondent made an item or survey suggestion

e. Skip problem (questionnaire navigation)
f. How the respondent interpreted an item term

g. Special situation (when a res pondent responds to an item diffe rently than expected based on a special situation )
h. Response problem or issue
i. Respondent had difficulty understa nding the survey formatting
ii. Respondent was unable to find a response that matched his or her experience
iii. Respondent had difficulty understanding the question
iv. Respondent did not include a response

v. Respondent misinterpreted wh at the question was asking
vi. Respondent had trouble understandi ng what a specific term meant
vii. Other problems or issues

These categories allowed the analysis team to de termine the types of problems that each item of
interest has. Once we were able to determine the item-by-item issues, we were able to do a higher level analysis of topic-level problems.



### 2. Findings

2.1 Screener Items At the beginning of the cognitive interviews, all respondents completed the National Household Education Survey household screener. Below we re port on two items of interest (month and year of birth and school enrollment) and discuss a common skip error.

For the item on the month and year of birth, the ma jority (n=45) of respondents did not have any difficulty reporting the month and year of birth for household members. Only a few respondents struggled to remember the year of birth for memb ers of the household. In most of these cases, the respondent had some difficulty recalling the year of birth for a child, but subtracted his/herchild’s age from the current year to identify th e year of birth. Only two respondents were unable to recall the year of birth for adult member s of the household (i.e., a mother-in-law’s and respondent’s parents’ year of birth).

For the item that asks if household members ar e currently homeschooled or enrolled in school (including college, university or vocational school), most partic ipants were able to answer without any problems. In a couple of instances respondents were uns ure what option to select for their young children (i.e., a one year old and 10 month old) who are in child care. One respondent selected “public or private school, or preschool” and the other respondent left the
item blank.

The most common problem in the household screener was a skip error. If the respondent reports that a household member is “not in school”, the su rvey instructs the respo ndent to go to the next
household member. If a household member is in school, respondents should proceed to the next item that asks for the person’s current grade or eq uivalent. Half of the respondents failed to skip out of the current grade or e quivalent item for household member s not currently enrolled in school. These respondents selected either “none of these” for current grade or equivalent or the household member’s highest level of schooling.

None of the items on the screener were ch anged during cognitive interview testing.



### 2.2 PFI Items

2.2.1 Child’s Schooling

School Assignment The item on a child’s school assignment was to be completed only by respondents whose children attend public schools. Table 3 lists the three versions of th is item that were tested in cognitive interviews.

Table 3. Wording changes for item on school assignment Original Wording Wording Change Final Wording Is	it	his/her	regularly‐ assigned	school?	(No/Yes) Is	it	his/her	distric t ‐assigned school?	(No/Yes) Is	it	his/her	distric t ‐assigned school?
A district

‐ assigned school is the school that your local public school district told you that this child can attend, based on the location of your residence. (No/Yes)

The original item that asks about “his/her regularly-assigned sc hool” was tested by 12 respondents. After respondents answ ered the item, the in terviewer presented th e respondent with an alternative item (“his/her district assigned school”). During testi ng, the two items wereswitched and 13 respondents respon ded to the item on “his/her di strict-assigned” school and were presented with the item on “his/her regula rly-assigned school” as th e alternative. For the remainder of the cognitive interviews (n=26), th e item was changed to include a definition of “district-assigned” and the item with the original wording was no longer tested.

Although many respondents understood the initial wording (“regularly-assigned school”), some respondents misinterpreted “regularly” to mean daily – a school the st udent goes to everyday during the week. Most respondents di d not have a problem with the term “district-assigned,” and seemed to prefer this over “regularly-assigned” because they were more familiar with the term. They described it as the closest school to the home or the school assigned to a student based on address/geographic zone.

For at least one respondent, inte rpretation of “district-assigned ” and “regularly-assigned” was informed by local policy. One respondent, base d in California, responded “Yes,” to “districtassigned school,” and reported th at California education code pe rmits parents to indicate a preference for which school his/her child will atte nd within the district regardless of where the family resides within the dist rict. The respondent applied to a preferred high school and was allowed to enroll the child in that high school , and therefore, reporte d that the school was district-assigned. However, when asked if the re spondent’s child attended the “regularly assigned school,” the respondent reported ”N o” – this school was not the child’s regularly assigned school, but rather the preferred sc hool. Adding a definition seemed to be helpful for respondents. Five respondents completed this item even though they should have skippe d out of it since their child attends a private school.

Advanced Placement One item on Advanced Placement (AP) enrollment was tested in cognitive interviews. This item was tested by all 50 respondents who completed the PFI items. The response option “Does not apply” was removed during testing (See Table 4 below).Table 4. Wording changes for Ad vanced Placement (AP) item

2012 Wording Original Wording on 2016 Survey Wording Change Final Wording

Is	he/she	currently enrolled	in	advanced placement	classes? (No/Yes/Does	not apply) Is	he/she	currently enrolled	in	any	high school	Advanced Placement	(AP)	classes? Advanced Placement is a program that offers college

‐ level courses to high school students, with the option for students to take AP

exams to earn college credit. (No/Yes/Does not	apply) Is	he/she	currently enrolled	in	any	high school	Advanced Placement	(AP) classes? Advanced Placement is

a program that offers college

‐ level courses to high school students, with the option for students to take AP

exams to earn college credit. (No/Yes) Is	he/she	currently enrolled	in	any	high school	Advanced Placement	(AP) classes? Advanced Placement is

a program that offers college

‐ level courses to high school students, with the option for students to take AP

exams to earn college credit. (No/Yes)

Of the 12 respondents who responded to the original wording of the item, seven did not have any difficulty answering the item. Some responde nts whose children at tend high school were confused as to whether they should respond “No” or “Does not apply.” Examples include one respondent who responded “No” stated that her child took an AP cour se the year before, but that
the child’s school did not offer AP courses in 11 th grade. Another respondent who marked “Does not apply,” misinterpreted the item and believed that elementary school students could take AP classes. Another respondent could not tell the difference between AP and IB courses, and, lastly, one respondent did not understand the meaning of Advanced Placement.

“Does not apply,” was removed as a response opti on as it was determined that sufficient data would be captured for respondents who res ponded “No” for children not in high school. Additionally, respondents with high school students at schools that do not offer AP might select “Does not apply.”

Of the total 38 respondents who responded to the fina l wording of this item, the majority did not have any difficulty answering it, although a fe w were confused as to how to respond. One respondent who incorrectly marked “Yes” for this item ended up revising the answer to “No”
when she reread the item and realized there was a difference between advanced classes and Advanced Placement. Another respondent who responded “No” said that the term was “foreign” to him or her, although the respondent attr ibuted that to the child being in 9 th grade.

Internet Courses
Three items of interest on courses taken over th e internet were initially tested by respondents. The first item in the table below was tested by all 50 respondents, and th e other two items were tested by 11 respondents who mark ed “Yes” in the first item. The following wording changes
were made to two of the items and one item was deleted during testing.Table 5. Wording changes/deletions for it ems on courses taken over the internet

Wording in 2012 Original Wording on 2016 Survey Wording Change Final Wording

Some	students	take school‐related	courses over	the	Internet.	Is this	child	receiving any	instruction	this way? Some	students	take school‐related	courses for	credit	over	the Internet.	Is	this	child receiving	any instruction	this	way? (No/Yes) Is	this	child	taking	any school‐related	courses online	instead	of	in‐ person	with	the teacher?
Do not include courses that use the Internet only for selected assignments? (No/Yes) Is	this	child	taking	any school‐related	courses online	instead	of	in‐ person	with	the teacher?
Do not include courses that use the Internet only for selected assignments? (No/Yes)
Is	that	instruction provided	by	any	of	the following	places?

Mark all that apply. Your	local	public school

A	charter	school

Another	public school

A	private	school

A	college, community	college,	or university

Someplace	else— Specify : Is	that	instruction provided	by	any	of	the following	places?

Mark all that apply. Your	local	public school

Your	state

A	charter	school

Another	public school

A	private	school

A	college, community	college,	or university

A	website

Someplace	else— Specify: Is	that	instruction provided	by	any	of	the following	places?

Mark all that apply. Your	local	public school

Your	state

A	charter	school

Another	public school

A	private	school

A	college,	community college,	or	university

Someplace	else— Specify : Is	that	instruction provided	by	any	of	the following	places?

Mark all that apply. Your	local	public school

Your	state

A	charter	school

Another	public school

A	private	school

A	college,	community college,	or	university

Someplace	else— Specify :

What	is	the	main reason	this	child	took school‐related	courses over	the	internet? Deleted Deleted

Although most respondents understo od the initial wording “school-re lated courses for credit over the Internet” in the first item, seven respondent s misinterpreted the it em, including four who incorrectly responded “Yes” on the questionnaire. Th ese responses included materials, tests, and
homework at the child’s school that are accessible on the Internet, extra credit for courses done at home, extra coursework done at school for additional grades, or the completion of a math course on-line over the summer. One respondent who responded “No,” in terpreted “Internetcourses” to mean reading, spelli ng, or alphabet games, and said: “it could be anything, it could be listening to some songs over the Internet.”

The wording was changed on this item because of the misinterpretations and also because the intent of the item. After the wording was cha nged and the definition wa s added to the initial item, many respondents correctly responded, although three respondent s misinterpreted the item, even when they correctly responded “No.” One thought the item was asking if the student is advanced enough to be taking online college course s, and another referred to courses outside of school, for example, tutoring for a calculus class. A third respondent showed some confusion in his or her reasoning for not including an informal online typing class as an online course: “there is no human being on the other end, th e entire class is completely el ectronic, and he [the student]
does not receive a certificate or credit for it.”

Respondents suggested the following changes to this item to make it more comprehensible: 1) Restructuring the sentence to read, “all-school related courses in pe rson with the teacher” or, “are any [courses] online,” and 2) replacing the wordin g “instead of” to “in addition to,” which might lessen the chance of glos sing over the question.”

On the second item of interest that asks who pr ovided the internet courses, of the 11 respondents who responded to the item, four respondents, he sitated to respond or mi sinterpreted the item because they did not understand what the item was asking, or because they were not sure whether to respond about the instituti on that pays for the course or the website used to facilitate the instruction. Based on these cognitive interviews , it was decided to delete “the website” as a response option for the remainder of the respondents.

After “the website” was deleted on this item, a couple of respondents were not sure how to respond because courses were through a free on- line system or an academy affiliated with homeschooling, and they were not su re of the funding source.

On the third item that asks what the main reason is that the child took sc hool-related courses over the internet, three respondents mi sinterpreted the item based on thei r misinterpretation of the first item, for example, two respondents selected “imp rove basic reading, writing, math, or science skills,” although the courses were either not for credit or for extra credit, and one respondent selected “Other,” which corresponds to the co mpletion of a math course on-line over the
summer. This item was eliminated because of conflic t with the first item on internet courses.

Homeschooling Initially, six items of interest on homeschooling we re tested in cognitive interviews. On the 2012 item, there was an overreporting of homeschooli ng and it was suspected that some respondents did not understand that the item was asking abou t formal homeschooling programs, and instead reported on casual instruction at home. In the 2014 feasibility study, the number of hours the child attended public or private school was adde d to the item, which allowed NCES to evaluate the number of respondents who indicated that the child was homeschooled but who likelymisinterpreted the item. A majority of 2014 re spondents who indicated that their child was homeschooled also reported that the child was attending a public or private school full-time. Based on these issues, it was decided to test a series of homeschool items on the 2016 PFIEnrolled survey.

The items tested in this section on homeschooling were:

 Some parents decide to educat e their child at home rather th an send them to a public or private school. Is this child be ing schooled at home instead of at school for at least some classes or subjects? (No, Yes)

 Which of the following statements best de scribes your homeschooling arrangement for
this child?

o This child is homeschooled for all classes or subject areas

o This child is homescooled for some cla sses and subject areas and also attends a public or private school

o This child is not homeschooled. This child attends a public or private school for all classes or subject areas

 How many hours each week does this child usua lly go to a public or private school for instruction? Do not include time spent in extracurricular activities. [__] Hours

 There are many different reasons that parent s choose to homeschool their children. Did your family choose to homeschool this child because: (Mark X ONE box for each item below.) (Responses a-j)

 Of the reasons your family chose to homesc hool this child, which one would you say is the most important to you? Write the letter from question [above] for the most important reason you chose to homeschool your child. [__]

 Thinking about all years this child has been homeschooled, which of the following subject areas has this child been taught during his or her home instruction Mark X all that apply .(list of subject areas)

 [New item added] Which of the following subj ect areas are being taught to this child
now? Mark X all that apply. (list of subject areas)

On the first item that asks if the child is be ing schooled at home, one respondent initially responded “No,” and then changed the response to “Yes” after the probin g questions, and said that he or she is providing tutoring for the child and considers it to be homeschooling. This respondent incorrectly completed all of the re maining items on homeschooling. On the item on description of the homeschooling arrangement th is respondent chose the response option: “this child is homeschooled for some classes and subj ect areas and also attends a public or private
school.” On the item on the number of hours eac h week the child usually goes to school for instruction, this respondent misread “hours each week” and wrote 9 hours and explained he or she drops off the child at school at 7 a.m. and pi cks the child up at 3:30 p.m. On the item about the subject areas the child has been taught durin g home instruction, this respondent selected“Calculus” and “Chemistry or physics” because those are the subject areas for which the child was receiving tutoring.

The majority of the other 49 res pondents did not have any problems or issues with the first item that asks if the child is being schooled at home, although a couple of respondents had difficulty with key vocabulary in the item. One of thes e respondents defined homeschooling as educating the child after school, although correctly res ponded “No,” and anothe r respondent defined “homeschool” as “a supplement to what [the ch ild] is learning in school,” although also correctly responded “No.” One homeschooling respondent made a skip error and did not continue to the next item on homeschooling. At the end of the cognitive testing a decision was made to add a downward arrow, similar to other arrows on th e questionnaire, to dir ect the respondent to
continue to the rest of the items on homesc hooling if they responde d “Yes” to the first homeschooling item.

The remaining homeschooling items were complete d by five respondents: three current and two former homeschooling parents, a nd a respondent who tutors the ch ild and considers it to be homeschooling. A couple of respondents had proble ms with the item on the description of the homeschooling arrangement. The item on the numbe r of hours each week the child usually goes to school for instruction presented problem s for two respondents. Two respondents were confused on how to respond to this item, one because the online homeschooling program is considered a private school and the responde nt thought the item on th e questionnaire was referring to a “brick and mortar” school, and another respondent was not sure how to respond about a homeschooling academy that was not traditionally public or private.

There were no problems on the item on the differe nt reasons that parents choose to homeschool their children. There were also no problems on the item on the most important reason the respondent chose to homeschool the child.

There were no problems on the item about the su bject areas the child has been taught during home instruction. Four responde nts recommended adding “physical education” and “health” to the list of subject areas. Based on these recommendations, at the end of cognitive testing these were added to the list of subj ect areas, although these items were not tested during this phase of cognitive interview testing (see Table 6 below).

An additional item on subject ar eas being taught “now” was added during the cognitive interview testing because there is not enough space to obtain every subject the child was taught in every year that they were homeschooled. Overall, this item did not pr esent any problems for respondents. One respondent who had only starte d to homeschool the child this year inquired whether the item is asking about today, this year, this semester, or this month, although responded with the same answ er as the previous item.Table 6. Wording changes and new item on s ubject areas taught during home instruction

Original Wording Wording Change Final Wording Thinking	about	all	years	the child	has	been	homeschooled, which	of	the	following	subject areas	has	this	child	been	taught during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas] Thinking	about	all	years	the child	has	been	homeschooled, which	of	the	following	subject areas	has	this	child	been	taught during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas]

Physical	education	or	gym

Health Thinking	about	all	years	the child	has	been	homeschooled, which	of	the	following	subject areas	has	this	child	been	taught during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas]

Physical	education	or	gym

Health
New Item Wording Change Final Wording Which	of	the	following	subject areas	are	being	taught	to	this child	now?

Mark	X	all	that	apply.

[List	of	subject	areas] Which	of	the	following	subject areas	are	being	taught	to	this child	now?

Mark	X	all	that	apply.

[List	of	subject	areas]

Physical	education	or	gym

Health Which	of	the	following	subject areas	are	being	taught	to	this child	now?

Mark	X	all	that	apply.

[List	of	subject	areas]

Physical	education	or	gym

Health

Other Items in the Child’s Schooling Section In this section, there were 19-21 other items tested by respondents. The majority of these items were tested by all 50 respondents except for a few skip patterns. Of these items, respondents had varying degrees of difficulty responding to the item s as well as the instructions. Only 13 of the 50 respondents had no difficulty when completing these items.

Five respondents were confused with the forma tting on the item asking the child’s current grade or year of school, which has the response boxe s for grade 1 through 12 under the kindergarten response options. All five said that they expected the boxes de signating their response to be placed directly under the item text and were ini tially confused when it was the last response option.

On the item asking respondents what type of sc hool the child attends, three respondents who all responded “private school,” continued to fill ou t the next two items relating to public school, when they should have skipped to a later item.Three respondents had difficulty responding to th e item asking if the child attends a charter school, and noted that they did not understand the meaning of “charter school.” Additionally some respondents were confused about the di fference between a charter school and a magnet school when reading the definition in the item asking whether or not a respondent’s school
district let them choose what public school they want the chil d to attend. On this same item several respondents were uncertain as to how to answer and noted that they have the option to
transfer schools in th eir district, however the district did not guarantee the transfer.

Three respondents did not follow the skip patter n on the item asking whether or not they had
considered other schools for the child. One of these respondents ignored the skip pattern and completed the following item, one incorrectly foll owed the skip pattern, and the third respondent turned the page and skipped three items inst ead of the item they were directed to.

On the item asking respondents about how many times teachers or school staff had contacted their household, a couple of respondents referred to contact they had made with teachers at an open house, and some could not understand w hy “very good behavior” and “very good school work,” were included in the questionnaire when teachers only contact them about problems with
the child’s behavior or school work.

Several respondents had difficulty responding to the item asking about their child’s overall grades across subjects. Several respondents sele cted responses other th an “This child’s school does not give these grades,” and reported that their ch ild’s school had a 1-4 point system and did not use traditional grading system s. Similarly, several respondents had difficulty determining an average grade. One respondent wrote on the questi onnaire, “all over the map” and did not select one of the response options.

On the item asking about the child’s suspension and expulsion history, a few respondents did not know how to interpret the meaning of “in-schoo l suspension.” Two respondents inquired about the meaning and suggested that it means that th e student was to attend school but not attend classes at all during school hours, and another respondent reported that it means that the child
could not attend certain activities such as Chorus.

Several respondents who had difficulty determin ing an average grade also had difficulty responding to the item asking how the respondent would describe their child’s work at school because their child excelled in some subjects and did poorly in others, and it was difficult for them to mark only one response option, e .g., “above average,” acr oss all subjects. .

2.2.2 Child’s Health

Services
Two items of interest on child’s health were te sted in cognitive interviews. Only respondents
whose children received services for health c onditions responded to these items. The followingwording change was made to one of the s ub-items on “sources for where services were provided” to be more parallel with other sub-items (see Table 7).

Table 7. Wording changes for item on sources for services provided

Original Wording Wording Change Final Wording Are	these	services	provided	by any	of	the	following	sources? Mark X ONE box for each item below.

No/Yes

a.		 Your	local	school district............... b.		 A	state	or	local	health	or social	service

agency........................... c.		 A	doctor,	clinic,	or	other health	care provider................................. d.		 A	teacher	or	health	provide r from	a	private school........................... Are	these	services	provided	by any	of	the	following	sources? Mark X ONE box for each item below.

No/Yes

a.	 Your	local	school district............... b.	 A	state	or	local	health	or social	service

agency........................... c.	 A	doctor,	clinic,	or	other health	care provider..............................

... d.	 This	child’s	private school........................... Are	these	services	provided	by any	of	the	following	sources? Mark X ONE box for each item below.

No/Yes

a.	 Your	local	school district............... b.	 A	state	or	local	health	or social	service

agency........................... c.	 A	doctor,	clinic,	or	other health	care provider..............................

... d.	 This	child’s	private school...........................

Of the 12 respondents who responded to the item on the sources where services were provided, two respondents were unsure whether to mark “Yes” for the sub-item “your local school district,” reasoning that either th e school or the district provided the service, or they were unsure altogether whether the local sc hool district provided the serv ice their child received. Two respondents indicated that the “Y es” and “No” response options te nd to be reversed on other surveys (with “Yes” before “No”). One of thes e respondent accidentally marked “No” on all of
the sub-items when he or she intended to ma rk “Yes.” One respondent incorrectly marked “doctor or clinic” for the person who was providi ng services for a condition that was not listed on the previous item.

On the second item about whether the services are provided through an IEP or services plan, respondents provided varying responses for the m eaning of IEP and services plan, or did not understand the meaning of the terms. Two res pondents defined “IEP” or “services plan” as
instances when children receive extra time to co mplete coursework or their assignments. One respondent indicated uncertainly ab out the meaning of an IEP, but said that if he or she was answering the survey at home, would look up th e meaning online to confirm. One respondent responded “No” to this item, but his or her ex planation for the “504 plan” his or her child received largely aligned with the definition of a services plan.Other Items in the Child’s Health Section Two other items on child’s health presented some problems, although they were not prevalent. Of the 50 respondents who responded to the item on “conditions told by health and education professionals,” two respondents noted issues with the item’s wording, citing that professionals do not “tell” you that your child has one of the conditions; inst ead the “health or education professional” recommends that you seek testing fo r your child to see if they have one of the conditions listed on the item. Two respondents ha d difficulty responding to this item because they were uncertain about the timeframe. One res pondent initially marked sub-item “a specific learning disability” then went dow n the list and identified sub-it em “attention deficit disorder, ADD or ADHD” as more fitting for the child’s co ndition and removed selection of “a specific learning disability.” It was deci ded to move the sub-item “a specific learning disability” from the top of the response options to just after ADD/ADHD.

There were also comprehension issues on th is item. Two respondents confused sub-item “orthopedic” with orthodontic matters. One respondent did not know the definition of “orthopedic” altogether. Additionally, four resp ondents could not define or did not recognize “Pervasive Developm ent Disorder (PDD).

For the item “if the child is receiving services for his/her condition,” one respondent indicated confusion about how to respond since the child no longer received services for the condition marked on the previous item. Two respondents had problems with the skip patterns. One respondent marked “Yes” and missed the skip patte rn, citing the arrow as less conspicuous than the box indicating a skip to the next set of ques tions. One respondent mark ed “No,” but read the following items and marked “Yes” for the “IEP and services plan” item.

2.2.3 Child’s Background

Joint Custody Two items of interest on joint custody were tested in cognitive interviews. No changes were
made to these items. The first item that asks if th e child “lives at this a ddress and another address (for example, because of a joint custody arra ngement)” was tested by all 50 respondents. The second item of interest on where the child spends his/her time (“at this address,” “at another address,” or “equal time at both addresses”) was asked only of 11 respondents who marked
“Yes” to the first item.

Although many respondents did not have any co mprehension problems with the first item on joint custody, some respondents had to re-read th e item a few times before responding, and some changed their response either from “No” to “Yes” or vice versa. Four respondents were not clear on the meaning of “this” address, although assu med it meant their home address – this response confusion could be a function of the cognitive interview setting outside of the respondent’s home. Some were not clear on the meaning of “j oint custody,” or whet her they were asking
about one or two addresses. For example, one respondent who responded “Yes” said that she and her husband are married and are the two people who ar e the parents of that child that have joint-custody or joint decision making. Another respo ndent who responded “Yes” said that her child goes to the father’s house on the weekends, alth ough when asked what the item was asking, the respondent said, “it doesn’t necessarily mean divor ce; children could go over to grandparents. It is asking if children spend time in another home.”

Four respondents described speci al situations where they had physical custody of a child, but they also had legal custody a rrangements. For example, where the child is supposed to spend time with one parent but prefers not to go to that parent’s house, so instead spends all of the time with the respondent, and the res pondents responded “No” to the item. In a couple of instances, respondents stated that their res ponse would be different for the fi rst part of the question, “Does
this child live at this address and another address” than the second part, “for example, because of a joint custody arrangement.”

Two suggestions were made by respondents for th e first item on joint custody: 1) to move the item to the screener; and 2) to simplify the ite m to read: “Does your child have more than one residency?” This respondent said that since a respondent might answer the item quickly, he or she might “skip the residency component altogether.””

Of the nine respondents who responded to the sec ond item of interest on joint custody that asks where the child spends his/her time, a few res ponded incorrectly because they did not understand the first item described above, and one responded “No” to the first item and should have skipped
the second item.

Language at Home One item on what language the child speaks most at home was tested in cognitive interviews. The majority of the 50 respondents who responded to this item found it relativ ely easy to answer. Four respondents were confused by the formatting of the first two response options: “Child is not able to speak,” and “English” and thought it was one option “Child is not ab le to speak English”
because there is a bracket around th ese two options to indicate that the respondent should skip to another question, and the response boxes were fa int. Respondents were looking for an English only response and most located it during the think aloud, although one respondent marked “English and another language equa lly” because he or she could not find the correct response.

The response boxes on this item were filled with white to make it clearer that these were two separate response options. Four respondents were still confused with the new formatting and thought the first two response op tions were grouped together. One respondent suggested adding an arrow after each response option rather than the bracket around the two options.Other Items in the Child’s Background Section A few other items in the Child’s Background se ction presented problems for respondents. A couple of respondents suggested ch anging “sex” to “gender” for the item on the child’s sex. A few respondents were confused and frustrated wh y the item on Hispanic, Latino, or Spanish origin is separate from the item on the child’s ra ce, and suggested that th ere should be a Hispanic option in the race item. Some responded “White” or left the item blank when responding “Yes” to Hispanic, Latino, or Spanish origin.

Some respondents had problems with skip patter ns in two other items. Five respondents who responded that the child was born in the U.S. did not skip out of the next item on how old the
child was when he/she first moved to the U. S. In two instances, respondents incorrectly responded to the item on being enrolled in ESL, bilingual education, or an English immersion program when they should have skipped out af ter the previous item on the language the child speaks most at home.

2.2.4 Your Household

Two items of interest on how many people live in the respondent’s househ old were tested in cognitive interviews. The following wording change s and order were made to the two items (see Table 8 below) during cognitive testing.Table 8. Wording changes for the items on how many people live in household

2012 Wording Original Wording (and Order) on 2016 Survey Wording Change [and Order Reversed) Final Wording (and Order) Including	yourself,	how many	total	people	live in	this	household?

[____]	people How	many	of	the	following people	live	in	this household	with	this	child?

Do not include this child in

your answer.

Example: Brother(s) 2
Write ‘0’ if none.

Brother(s) Sister(s) Parent(s) Aunt(s) Uncle(s) Grandmother(s) Grandfather(s) Cousin(s) Parent’s	girlfriend/ boyfriend/partner Other	relative(s) Other	non‐relative(s) How	many	people	live	in this	household?

Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home.

[____] How	many	people	live	in this	household?

Include adults and children who are temporarily away from home (for example, living
in college housing) if they have no other permanent home.

[____]

Other	than	the	parents or	guardians	already reported,	how	many	of the	following	people live	in	the	household with	this	child?

Example: Brother(s) 2

Write ‘0’ if none.

This	child’s...

Brother(s) Sister(s) Aunt(s) Uncle(s) Grandmother(s) Grandfather(s) Cousin(s) Parent’s girlfriend/boyfriend/pa

rtner Other	relative(s) Other	non‐relative(s) Enter	the	total	number	of people	living	in	this household	with	this	child (This	number	should	be equal	to	the	sum	of	a through	k	above).

Do not include this child in

your answer. [____]		people How	many	of	the	following people	live	in	this household	with	this	child?

Do not include this child in

your answer.

Example: Brother(s) 2

Write ‘0’ if none.

Brother(s) Sister(s) Mother	(birth,	adoptive, step,	or	foster) Father	(birth,	adoptive, step,	or	foster) Aunt(s) Uncle(s) Grandmother(s) Grandfather(s) Cousin(s) Parent’s girlfriend/boyfriend/part

ner Other	relative(s) Other	non‐relative(s) How	many	of	the following	people	live	in this	household	with	this child?

Do not include this child
in your answer.

Example: Brother(s) 2

Write ‘0’ if none.

Brother(s) Sister(s) Mother	(birth,	adoptive, step,	or	foster) Father	(birth,	adoptive, step,	or	foster) Aunt(s) Uncle(s) Grandmother(s) Grandfather(s) Cousin(s) Parent’s girlfriend/boyfriend/pa

rtner Other	relative(s) Other	non‐relative(s)Of the 50 respondents who responded to the item, three respondents had problems with the original item. For who is in the household, problems include d searching for an option for “mother” and not including themselves in the co unt; and being uncertain how to respond about a boyfriend who is at the house the majority of the time but who does not legally live in the household. The latter respondent deci ded not to include the boyfriend.

Five respondents had difficulty responding to the count. Problems included respondents’ difficulty with the “a through k” language in th e parentheses and reading “a” as an article, and including the child in th e count. A respondent claimed that the item had “a lot of words,” and “a lot to read to ask a pretty si mple question,” and suggested that the item should ask how many people live in this household excluding the child, and then to break it down further below.

In the next round of testing, the item order was reversed and the exact wording from the household screener was implemented, based on fee dback from the initial cognitive interviews. All the remaining respondents comprehended th e revised first item “H ow many people live in
this household?”

The revised item “How many of the following peopl e live in this household with this child?” that only had a change from “Par ent(s)” to “Mother (birth, adoptive, step, or foster)” and “Father (birth, adoptive, step, or foster)” to match an other item in the “Your H ousehold” section, still presented problems similar to what was reported on the original item. Four respondents included the child in the counts, three respondents were uncertain whether to include some household members, i.e., two children away at college, a friend of the child’s who was living with the family for a period of time, and a brother-in-law who is a long-term houseguest. All three decided not to include these pe ople. One of these respondents s uggested that more information
should be included in this item similar to th e other household item about whether or not to include long-term houseguests. On e respondent made a suggestion th at “it might be worthwhile to indicate on the questionnaire that if you want the total number or pe ople in this item to be the
other household item minus 1.”

Other Items in the Your Household Section The majority of the respondents did not have an y problems with the othe r items in the Your
Household section. The item on the total income of all persons in the household was confusing for a few respondents because they did not know w ho in the household to include in the response and what income to include, for example, child support or disability. The item on which language(s) are spoken at home by the adults cause d some confusion for a couple of respondents because they thought it was referring to the lang uage spoken with the child, or they incorrectly did not include English when they were flue nt in English. Responde nts who lived in an apartment or a converted garage found the item on whether “this house” is owned, rented or occupied by another arrangement problematic, and one respondent suggested that the item should say “house or apartment.”2.2.5 Child’s Family

Two items of interest on the respondent’s educa tion level were tested in cognitive interviews. The majority of the respondents did not have any problems when responding to these items on the highest grade or level of sc hool that the respondent comple ted and whether he or she is currently attending or enrolled in school. Th e most common problem for a few respondents was that they were not sure if they should be re sponding about themselves or their child on both
items.

Additionally, on the firs t item a couple of respondents did not understand what “vocational diploma after high school” means, and one of these respondents actually had a “vocational certificate” although responded “high school diploma or equivalent (GED)” on the questionnaire. On the second item, a couple respondents did not initially see “job traini ng” at the end of the item, and one respondent was confused because the item is asking tw o questions (whether respondent is in school or receiving vocational ed ucation or job training) and suggested that it
would be more clear to ask for the “student status.”

No changes were made to these items during cognitive testing.



### 2.3 ECPP Items

2.3.1 Finding and Choosing Care for Your Child

Respondents with children age five or younger completed additional items related to finding and choosing care for a child. Three of these are items of interest and are discussed below.

Difficulty Finding Child Care One item of interest on the primary reason for difficulty finding child care was tested in cognitive interviews. This item was tested by th e five respondents who had indicated in the previous item that they faced some difficulty finding the type of ch ild care or early childhood program they wanted for their child. The following wording changes were made to the item during testing (see Table 9 below).Table 9. Wording changes for the item on prim ary reason for difficulty finding child care

Original Wording Wording Change Final Wording What	was	the	primary	reason for	the	difficulty	finding	care?

Cost

Location
Quality

Lack	of	open	slots	for	new children

Needed	a	program	for children	with	special	needs

Other	reason:	Specify

______________________________ What	was	the	primary	reason	for the	difficulty	finding	care?

Mark X ONE only.

Cost

Location
Quality

Lack	of	open	slots	for	new children

Needed	a	program	for	children with	special	needs

Other	reason:	Specify

______________________________ What	was	the	primary	reason	for the	difficulty	finding	child	care or	early	childhood	programs?

Mark X ONE only.

Cost

Location
Quality

Lack	of	open	slots	for	new children

Needed	a	program	for	children with	special	needs

Other	reason:	Specify

______________________________

Of the two respondents who responded to the in itial item, one did not have any difficulty responding. The other respondent ha d difficulty understanding whethe r or not it was possible to mark more than one option and marked several options for this item including “ Cost,” “Quality,” and, “Lack of open slots for new children.”

After the instructions “Mark X ONE only” was added to the item, the remaining respondents correctly responded to the item w ithout any confusion. In the final version, the wording was changed to mirror the language of another ite m on choices for child care “or early childhood programs” where the respondent lives.

Importance of Reasons for Choosing Child Care One item of interest asks how important it was in choosing the childcare arrangement or program where their child spends the most time. Five new sub-items were tested in cognitive interviews and they are described below.

Ratings from a Website Of the nine respondents who answered this sub- item, all but one had no difficulty understanding the item or finding a response that matched their experience. One respondent did not fill out this item and was uncertain whether or not the questi on was referring to the child care provider’s
website or to an external website.Recommendations from friends and family

Of the nine respondents who answered the sub- item on how important recommendations from friends and family were in choosing a child care arrangement or program, none of the respondents had difficulty understanding th e question or selecting a response.

Accreditation of care provider

Two versions of this sub-item were tested on how important accreditation of the care provider was in choosing a childcare arrangeme nt or program (see Table 10 below).

Table 10. Wording changes for sub-item on im portance of accreditation of care provider

Original Wording Wording Change/Final Wording Accreditation	of	the	care provider	(e.g.	from the	National	Association	for	the	Education of	Young	Children	or	the	National	Early Childhood	Program	Accreditation)?

Not	at	all	important

A	little	important

Somewhat	important

Very	important Accreditation	of	th e	care	provider? “Accreditation" is another way to judge the quality of a child care program. Family child care homes and child

care centers can choose to get accredited by a child

care accrediting organization, such as the National
Association for the Education of Young Children or the National Early Childhood Program Accreditation.

I	do	not	know	abou t	the	accreditation	of	my	child’s care	provider.

Not	at	all	important

A	little	important

Somewhat	important

Very	important

The original item was tested by four respondents; three of these respondents were unsure what accreditation was. Based on these cognitive intervie ws, the item was changed to add a definition of accreditation and the following response option “I do not know about the accreditation of my child’s care provider.” The revi sed version of the item was test ed by five respondents. Adding the definition seemed to be helpful for res pondents to better understand what the item was asking; several were able to better articulate the meaning of accreditation.

Religious orientation of the program

Several of the nine respondents who answered this sub-item on how important the religious orientation of the program was in choosing a chil dcare arrangement interpre ted this item to mean that the child would attend a program run by a church or religious group. Respondents did not appear to have difficulty understand ing the item or finding a response.QRIS

Three versions of this sub-item on how importa nt a local Quality Rati ng Improvement System (QRIS) was in choosing a childcare arrangement or program were te sted in cognitive interviews (see Table 11below).

Table 11. Wording changes for sub-item on importance of QRIS

Original Wording Wording Change Final Wording Did	not	use	a	QRIS

Not	at	all	important

A	little	important

Somewhat	important

Very	important I	have	not	heard	of	a	QRIS

Did	not	use	a	QRIS

Not	at	all	important

A	little	important

Somewhat	important

Very	important I	have	not	heard	of	a	QRIS

Not	at	all	important

A	little	important

Somewhat	important

Very	important

The original response options were tested by four respondent s. All four respondents were unaware what a QRIS was and chose “did not us e a QRIS.” Based on these cognitive interviews and to make the item more parallel to the acc reditation sub-item, a res ponse option, “I have not heard of a QRIS” was added. Two respondents test ed the item with this additional response option, which both respondents selected. To stre amline the response options , the option of “did not use a QRIS” was removed and the revised item was tested. Responde nts did not have any problems with selecting from th e final response options.

Early Childhood Program Curriculum
Three versions of this item of interest that asks about the type of curricu lum offered at the early childhood program were tested in cognitive interviews (see Table 12 below).Table 12. Wording changes for item on early childhood program’s curriculum

Original Wording Wording Change Final Wording Which	of	the	following	best describes	this	program’s curriculum?

Montessori

Special	program	emphasis‐ such	as	science	or	math emphasis,	performing	arts, talented	or	gifted	preschool, foreign	language	immersion,

etc.

Special	education‐	primarily serves	students	with disabilities

The	Creative	Curriculum®

None	of	the	above Which	of	the	following	best describes	this	program’s curriculum?

Montessori

Special	program	emphasis‐ such	as	science	or	math emphasis,	performing	arts, talented	or	gifted	preschool, foreign	language	immersion,

etc.

Special	education‐	primarily serves	students	with disabilities

Play‐based	curriculum,	such	as The	Creative	Curriculum®	or Reggio®	curriculum

None	of	the	above Which	of	the	following	best describes	this	program’s curriculum?

Montessori

Another	play‐based	curriculum, such	as	The	Creative Curriculum®,	High	Scope	or Reggio	Approach

Special	program	emphasis‐ such	as	science	or	math emphasis,	performing	arts, talented	or	gifted	preschool, foreign	language	immersion,

etc.

Special	education‐	primarily serves	students	with	disabilities

None	of	the	above

This item of interest was tested by nine responde nts. They were generally not familiar with The Creative Curriculum on the original response option, although one res pondent said that the child attends a program with a play-based curriculum similar to The Creative Curriculum. Only a couple of respondents were familiar with Montes sori programs although not familiar with the specifics of the curriculum. Respondents were also not familiar with the Reggio curriculum. Based on these cognitive interviews the term “play-based” and a nother example of a play-based curriculum were added to this response option.

The final response options were further revised sin ce Montessori is also a play-based curriculum, another example of a play-based curriculum wa s added, and the order of the response options was changed to make the item clearer. Respondent s were typically unaware of the curriculum
used in their child’s early childhood program.

Other ECPP Items Respondents did not have any problem s or issues with the other four ECPP items that were tested in cognitive interviews.

### Appendix A

– Interview Protocol312016ParentandFamilyInvolvementinEducationSurvey(PFI)andEarlyChildhoodProgramParticipation(ECPP)SurveyCognitiveInterviewProtocolInterview ID:Date ofInterview:Parent FirstName:InterviewerAge of Child:STEP 1: WELCOME ANDINTRODUCTION SCRIPTInterviewer Instructions:The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer shouldbe thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.Hello, my name is<your name> and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.[Create small talk to build rapport with the parent]:You are here today to help us out with a new survey that weare working on for the U.S. Department of Education. This survey asks about yourchild’s schooling [and finding and choosing child care for your child for parents with children 5 and younger]. I will ask youto take the survey as ifyou were taking a real survey. There are noright or wrong answers, and it’s ok if you donot know some of the answers. As youare answering thequestions I will ask you about your answers to the questions. This will help us make the survey better and clearer for other parents just like you.Because the information you provide is so important, I am going to be taking notes while you are answering the survey and whilewe are talking.[IF OBSERVER PRESENT]One of my colleagues will be observing our discussion through a live video stream. It is not being recorded by video.Is that OK with you?[IF NOT, TURN OFF WEBEX]32Also, this interview will be audio-recorded so researchers can review the recording later. Any information that refers to you or your familypersonally, like your name or name of your child’s school, willnot be included in our report, and your responses will be usedfor research purposesonly.Remember, you are not being tested; we want you to tell us what you think about the survey.If at any time you want to stop, just let me know.[If, for any reason, the participant is no longer interested inparticipating, thank the participant for his/her time and end the interview.]Before we continue, do you have any questions about what I just said?[Answer any questions the parent may ask.]IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER WITH A COPY OF THE FORM. ASK THERESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANYQUESTIONS, AND HAVE THE RESPONDENT SIGNTHE FORM. LEAVE A SEPARATE COPY OFTHE FORM WITHTHE RESPONDENT.IF THE RESPONDENT HAS CONSENTED TORECORDING, START THE RECORDER.STEP 2: THINK-ALOUD PRACTICEWhile you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you figure out your answersto the questions will help me understand how to make the questionsbetter. I am more interested inhow you arrive at your answers than in theanswers you give me. To help you get started on thinking aloud, we are going to do a practice activity.First, I will show you an example of how to think out loud. Then I will give you a chance to practice it.QUESTION: How many windows are there inthe house or apartment where you live?[Model the think aloud process for the parent, including reading the question out loud. Elaborate on counting all of the windows in your house orapartment and as needed you can make up pieces of information to emphasize the think aloud process e.g., “there is a door at the back of the houseand it has windows on it; I think I’ll count those as windows. In the living room there are windows with panes in them and I think I’ll count each painas a separate window...” When you are modeling, write down the number of windows in your house/apartment as you are thinking aloud.][Once finished the model practice think aloud]I’m going to give you a piece of paper with the question on it. I’m going to ask you to read out loudthe question and think aloud as you decide on your answer.33Give respondent handout with think aloud practice.How many windows are there in the house or apartment where you live?Please tell me what you are thinking while you are figuring out your answer.Probe as needed for detail:How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tellme more about what you are including in your count?When complete:Thank you; that’s the kind of detail I’m looking for when youtalk to me about your answers tothe questions in the survey.HANDRESPONDENTACOPYOFTHEQUESTIONNAIREANDAPENORPENCIL.STEP 3: COMPLETIONOF THE QUESTIONNAIRENow I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything youwould have read to yourself if I were not here.Remember that I am very interested in hearing you talk about what you are thinking as you figure out your answers to the questions.REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOBTHINKING ALOUD, AND THEIR ANSWERS TOYOUR QUESTIONS ARE REALLY HELPFUL.34Useful ProbesCan you tell me in your own words what that question was asking?What does the word [term] in this question mean to you?How easy or difficult is it to remember [topic]?How easy or hard was it to choose an answer?Tell me what you are thinking?How did you come up with that answer?You said [answer]. Can you tell me more about that?[In response to R actions or comments]: Tell mewhat happened? Tell me what you were doing?[If you pick up on a visual cue like a furrowed brow ora puzzled look]: “Tell me what you are thinking.”[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering thisquestion]: “Tell me what justhappened. Tell me about what you were just doing.”35ScreenerASK THE RESPONDENT TO FILL OUTTHE ENTIRE SCREENER. REMIND THEMTO THINK ALOUD IF THEY ARE NOTDOING SO.Q3.What is the person’s month and year ofbirth?[If the respondent is having difficulty with themonth and/or year]How did you come up withyour answer?Q5.Is this person currently in...?How did you come up with your answer?[If respondent chooses “homeschool”] Whatdoes “homeschool” mean to you?DO NOT PROBE SPECIFICALLY ABOUTHOMESCHOOLING HERE. FOR THESCREENER WE ONLY WANT TOUNDERSTAND HOW THEY AREINTERPRETING “HOMESCHOOL.”PFI/ECPP SurveyChild’s SchoolingASK THE RESPONDENT TO FILL OUTTHE SURVEY. ASK QUESTIONS AND USEPROBES ONLY AS NECESSARY; REMINDRESPONDENT TO THINK ALOUDInstructions:It is not necessary to probe onthe instructions. Note if the respondent has anyproblems if they read them.36Q1-Q2: It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q3. Is it his/her district-assigned school? Adistrict-assigned school is the school that yourlocal public school district told you that thischild can attend, based on the location of yourresidence.[Ask only if responded to this question]Howdid you decide on your answer?What does “district assigned” mean to you?How did you know what school your child wasgoing to attend? How do other parents in yourchild’s district know which schools their childwill attend?Q4-Q13: It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q14.Is he/she currently enrolled in any highschool Advanced Placement (AP) classes?What were you thinking of when you answeredthis question?What do “Advanced Placement (AP) classes”mean to you?What grade-level are the students who usuallytake AP classes in your child’s school district?37Q15-Q21:It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q22.Is this child taking any school-relatedcourses online instead of in-person with theteacher?Do not include courses that use theInternet only for selected assignments.How did you decide on your answer?Q23.Is that instruction [internet] provided byany of the following places?How did you decide on your answer?[If chooses a response option or “Someplaceelse:”]Tell me more about that. How doesyour child access the classes? Where is he/shewhen accessing the classes? Who provides theclass?Q24:It is not necessary to probe on thisquestion. Let the respondent continue the thinkaloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.38Q25.Some parents decide to educate theirchildren at home rather than send them to apublic or private school. Is this child beingschooled at home instead of at school for atleast some classes or subjects?In your own words, tell me what this questionis asking about.How did you determine whether to answer yesor no?[If respond “yes,”] Do not probe on themeaning of homeschool until Q30.[If respond “no” and if not evident from thinkaloud to previous questions]What does“homeschool” mean to you?Make a note if the respondent did not skipproperly, but do not probe on skip pattern.]Q26.Which of the following statements bestdescribes your homeschooling arrangementfor this child?How did you come up with this answer?[Do not probe on the meaning of homeschooluntil Q30. Check to see if respondent skippedproperly, but do not probe on skip pattern.]Q27.How many hours each week does thischild usually go to a public or private schoolfor instruction? Do not include time spent inextracurricular activities.How did you come up with your answer?39[Check to see if respondent filled in thisquestion although they indicated that the childis homeschooled.]Q28.There are many different reasons thatparents choose to homeschool their children.Did your family choose to homeschool thischild because:How did you determine how to answer yes orno to each of these questions[a-j]?Tell me more what you were thinking whenanswering these questions.[Do not probe on the meaning of homeschooluntil Q30.]Q29.Of the reasons your family chose tohomeschool this child, which one would yousay is the most important to you?How did you come up with your answer?[If not evident from think aloud to previousquestions]What does “homeschool” mean toyou?Q30.Thinking about all years this child hasbeen homeschooled, which of the followingsubject areas has this child been taughtduring his or her home instruction?How did you decide what to mark?How do you cover these subjects during homeinstruction?Is there anything missing?40Q31.Which of the following subject areas arebeing taught to this child now?How did you come up with your answer? Canyou tell me more about what the child is beingtaught now?Child’s HealthREMIND RESPONDENT TO THINKALOUDQ32-Q35:It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q36.Are these services provided by any of thefollowing sources?How did you decide on your answer?Q37.Are any of these services providedthrough an Individualized Education Plan(IEP) or services plan?How did you come up with your answer?What does a “services plan” mean to you?41Child’s BackgroundQ38-Q43:It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q44.Does this child live at this address andanother address (for example, because of ajoint custody arrangement)?How did you come up with your answer?In your own words, what is this questionasking?[If answered No]What does joint custodyarrangement mean to you?Q45.If yes, does this child spend most of thetime at this address, at another address orequal time at both addresses?How did you come up with your answer?[If needed] Can you describe the arrangementyour child has?Q46.What language does this child speakmost at home?It is not necessary to probe on this question.Note the respondent’s response, especially forthose who are parents of ELL students. Recordin your notes whether the respondent reads theresponse options below the top two responses.42Q47. It is not necessary to probe on thisquestion. Let the respondent continue the thinkaloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Your HouseholdQ48.How many people live in thishousehold?How did you come up with your answer?Who are you including in the counts?Q49.How many of the following people livein this household with this child?Tell me more about what you were thinkingwhen you answered this question?Who are you including in the counts?In your own words, what is this questionasking?Q50-Q58:It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondentsChild’s FamilyREMIND RESPONDENT TO THINKALOUDQ59.What is the highest grade or level ofschool that this parent or guardiancompleted?43[Only if respondent chooses “high schooldiploma or equivalent”, “vocational diplomaafter high school”, or “some college, but nodegree”:]How did you come up with your answer?What does “vocational diploma after highschool” mean to you?Q60.Is he or she currently attending orenrolled in a school, college, university, oradult learning center, or receiving vocationaleducation or job training?In your own words, what is this questionasking?If parent wants to discuss another child in K-12 (not the first listed on the screener), there can be an informal follow-up, but it should beparent-directed- they can provide additional insight regarding howthey might have answered some questions for another child if there istime.THE REMAINING QUESTIONS ARE FOR PARENTS WHO ALSO HAVE A CHILD UNDER AGE-5, NOT YET IN KINDERGARTEN,AND WHO IS IN A CARE ARRANGEMENT.Tell the respondent the following:The remaining questions are about your other child who is underage 5, not yet in kindergarten and in a care arrangement. How old is this child?What type of care arrangement does this child have?[Record this in your notes.]Please continue and answer the remaining questions. Please remember to think aloud.44Finding and Choosing Care for your ChildREMIND RESPONDENT TO THINKALOUDQ61-64:It is not necessary to probe on thesequestions. Let the respondent continue thethink aloud and only probe if respondent sayssomething that indicates a potential problem, orif there is confusion or an inconsistentinterpretation compared to other respondents.Q65.What was the primary reason for thedifficulty finding child care or early childhoodprograms?Please tell me more about how you chose youranswer.Can you tell me more about that?Q66.How important was each of thesereasons when you chose the child carearrangement or program where this childspends the most time?66h. Ratings on a website?How did you decide what to mark?In your own words, what is 66h asking?Could you provide some examples?66i.Recommendations from friends andfamily?How did you decide what to mark?45In your own words, what is 66i asking?66j.Accreditation of the care provider?"Accreditation" is another way to judge thequality of a child care program. Family childcare homes and child care centers can chooseto get accredited by a child care accreditingorganization, such as the National Associationfor the Education of Young Children or theNational Early Childhood ProgramAccreditation.How did you decide what to mark?In your own words, what is 66j asking?[If definition seems confusing:]In your ownwords, what is the text[point to definition]telling you?66k.The religious orientation of theprogram?How did you decide what to mark?In your own words, what is 66k asking?66l.Your city or state’s Quality Rating andImprovement System (QRIS)?How did you decide what to mark?In your own words, what is 66l asking?What does QRIS mean to you?46[If applicable:]How did you learn about theQRIS? How did you use it? Do you rememberwhat it is called for your state or area—that is,which QRIS did you use?Q67.Which of the following best describesthis program’s curriculum?How did you come up with your answer?[If respondent chose Montessori:]Tell mewhat Montessori means to you.[If respondent chose Play-based curriculum:]Can you tell me more about your response?What does “play-based” mean to you?Are you familiar with The CreativeCurriculum?How about HighScope?How about Reggio Approach?Are there other play-based curriculum that youare aware of?[If respondent chose None of the above:]How would you describe the type ofcurriculum used in your child’s program?Do you know the name of the curriculum usedin your child’s program?47FINAL QUESTIONDo you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?STEP 4. THANK YOUFOR PARTICIPATINGWe are finished. Thank you for helping us. The information you’veprovided will be very helpful for making this survey better.Interviewer Instructions:Collect materials and escort parent out of the interviewing room and to the lobby. Please give the respondent incentiveenvelope [$40 gift card], and thank them for their time and effort.

# Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2016: Draft Report

Prepared for the National Center for Education Statistics (NCES) by

Stephanie Cronen & Kirsten Manville

June 20151000 Thomas Jefferson Street NW Washington, DC 20007-3835 202.403.5000 | TTY 877.334.3499

www.air.org 

# Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2016: Draft Report

June 2015

Stephanie Cronen & Kirsten Manville

## Contents

Page Introduction .................................................................................................................. ....................3
Methodology ................................................................................................................... .................3
Sample......................................................................................................................... .....................4
Key Findings .................................................................................................................. ..................4

Appendix A. Cognitive Interview Protocol .....................................................................................9American Institutes for Research Co gnitive Interview Report for ATES 2016—3 

## Introduction

Beginning in January 2016, the National Household Education Survey (NHES) is expected to field the first full-scale administration of the Adult Training and Education Survey (ATES). The American Institutes for Research (AIR) is wo rking with the National Center for Education Statistics (NCES) and staff from numerous other federal statistical agencies to develop a strategy for measuring the prevalence of subbaccalaureate credentials—in particul ar, industry-recognized certifications, licenses, and edu cational certificates—and work-rela ted training and education in the United States. This multiphase research effo rt began in 2009 and includes the development and testing of questions that will be included in the ATES. In Phase I, AIR contracted with Shugoll Research to conduct focus groups and cognitive interviews with respondents holding less than a bachelor’s de gree who work in fields that may offer subbaccalaureate opportunities. In Phase II, the Center for Survey Methods at AIR designed and conducte d cognitive interviews of potential survey items that may be used to meas ure certifications and licen ses. In Phase III, we extended this work in two stages—for the or iginally planned 2015 NHES administration and for the 2016 rescheduled administration—to include meas ures of participation in and characteristics of work-related training and education. In addi tion, we conducted furthe r testing of revised measures of credential attainment.

This draft report documents the key findings fr om the English language cognitive interviews conducted in Phase III for the 2016 administrati on. The revised report will include key findings from the Spanish language cognitive interv iews that are currently being conducted.


## Methodology

Each cognitive interview lasted approximatel y one hour and was conducted using primarily a “think-aloud” approach with conc urrent probing as needed, base d on a structured protocol. The methodology was developed by a senior researcher at AIR in consultation with NCES and drawing on best practices and methods from cogn itive science (see Appendix A for the protocol). These interviews were designed to identify probl ems of ambiguity or misunderstanding in item wording. The interviews were conducted by interv iewers who received tr aining in the cognitive interview technique and in the in terview protocol by a senior researcher at AIR. Researchers from AIR and NCES listened to a subset of interviews and provided guidance and feedback throughout the study.

During each interview, the study participant was welcomed by the interviewer and led to an interview room. The participant was then told th at the purpose of the interview was to help answer questions about how peopl e respond to items on schooling. A ll participants were assured that their participation was voluntary and that their answers would be used only for research purposes and not disclosed, or us ed, in identifiable form for a ny other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the think-aloud process (see below) and modeled a think-aloud practice with a sample question. Participants were then given a paper copy of the questionnaire to fill out.American Institutes for Research Co gnitive Interview Report for ATES 2016—4 The interviews were conducted using “concurrent probing,” a technique in which interviewers asked respondents to “think aloud” as they answ er the survey items. The cognitive interviews also included a verbal probing component, conduc ted after completion of the think-aloud portion for a given section of the questionnaire. The verbal probes included a combination of preplanned, item-specific questions id entified before the session as important and ad hoc questions that the interviewer identified as important fr om observations during th e interview, such as clarifications or expansions on points raised by the participant. 

## Sample

Cognitive interview participants were recruite d through the recruitment firm Elliott Benson in Sacramento and San Mateo, CA and by AIR staff in Washington, DC and Grand Rapids, MI. All interviews were conducted in person at these sa me locations. Respondent received $40 for their participation.

The majority of respondents were required to either have a professi onal certification, an educational certificate, or other work-related tr aining. A small number of additional cases were included to represent the genera l population. Based on screening da ta, the number of respondents within each target group was: 1

 Certification or license: 28

 Certificates: 18

 Work-related educati on or training: 22

 General population: 2

AIR conducted a total of 48 cognitive interv iews between March 3, 2015, and April 17, 2015. The interviews were conducted in four rounds to facilitate revi sion and re-testing. 

## Key Findings

Findings are presented by survey section, and ch anges made to the survey to address response issues are described below.



### Items on Education

The follow-up item that asks for the field of stud y for highest level of education was difficult for some respondents to follow, either because they di d not see their field listed, or because they did not notice the instructions to mark one field only. To address these issues, we simplified the label for the two most problematic fields of study (hea lthcare and social studies, political science, economics, or history) and underlined the word “one” in the stem:

1 These target group numbers sum greater to the total number of interviews completed because respondents could have combinations of target criteria.American Institutes for Research Co gnitive Interview Report for ATES 2016—5 “Which one of the following best describes the field of study for the highest level of school you have completed? (Mark ONE only.)

The item intended to measure participation in adult basic education was eliciting false positive responses, due to respondents’ generalization of the term “literacy.” Classes such as computer literacy and college composition courses were being reported under this item, and people who taught rather than took classes pertaining to lite racy or reading were reporting participation. Therefore, we underlined the word “your” in the stem and instru cted respondents not to include college-level classes:

Since leaving high school, have you taken any literacy classes to help improve your reading? Do not include college-level classes.

The item designed to measure college enrollment was confusing to some respondents for several reasons. There were two different time referents in the stem (“currently ” and “after high school”) and the item appeared after the question about literacy classes, wh ich seemed to be creating an order effect. Therefore, some respondents inte rpreted the question as being about remedial classes. To address these issues , we moved the item to appear after the first two educational attainment items, and de leted “after high school”:

Are you currently enrolled at a college, university, technical or trade school, or other school?



### Items on Certifications and Licenses

In prior administrations, follo w-up questions referred to the most recent certification or license; however, we found that most recent credentials were not always being reported due to low salience; i.e., not all of these credentials were considered curr ently relevant or as the most important credential to all respondents, particular ly in fields where resp ondents are required to obtain multiple credentials. For 2016, we ther efore tried a new approach by asking about respondents’ most important certification or license. Based on interview findings, we determined that this approach is valid, and th at most respondents can easily identify what they consider to be their most important certification or license.

A familiar issue in this set of follow-up questions persisted, however; some respondents conflated multiple certificati ons or licenses when responding. Each item already reminds respondents to report on their most important certification or license; therefore, no further revisions were made.

The item that serves as the primary indicator of the credential being a license rather than a certification was difficult to an swer for several respondents who were not sure if a government entity issued their credential. For some, it was th e parenthetical examples that were causing their confusion; therefore, we deleted thos e examples and simplified the item to:

Is your most important certification or license required by a federal, state, or local government agency in orde r to do that kind of work?

We also added a second question from the 2010-11 ATES to help identify false positive reports:American Institutes for Research Co gnitive Interview Report for ATES 2016—6 Can your most important certification or license be revoked or suspended for any reason?



### Items on Certificates

The main certificate item continue s to be a challenge, due to the overuse of the term among education and training providers. The result of this overuse is confusion be tween certificates and certifications and between educational certificat es (based on a postsecondary program of study) and certificates of completion (b ased on a single course or trai ning). In the current round of interviews, we found these same issues, but to a le sser extent. It appears to be helping to (a) ask about certifications and licenses be fore certificates; and (b) ask about other t ypes of certificates (e.g., from an employer for completing a training) before asking about edu cational certificates. Therefore, only minor modifications were made to the certificates items for readability purposes. Specifically, we deleted some unne cessary text in the stem and response options to make the item less cognitively burdensome.



### Items on Work Experience Programs

One of the primary purposes of th e 2016 interviews was to furthe r test the relatively new items on participation in and characteristics of work experience programs. Overall, we found that these items performed well, with the exception of a moderate amount of over-reporting by respondents who misinterpreted the items to pertain to more general on-the-job experience. We addressed this to the extent possible by revising th e stem of the main item to refe r explicitly to types of work experience programs:

Have you ever completed an internship, co-op, practicum, clerkship, externship, residency, clinical experience, apprenticeship, or similar program?

Some respondents also had difficu lty finding the field of study fo r their program. Therefore, we made the healthcare field easier to find and added nursing assi stant to the list.

Another relatively common area of overreporting was for respondent s to report that their work experience program was part of various types of schools’ education or training program. To get respondents to better attend to th e response options, we revised the stem to be more general and the response options to be more specific:
Which one of the following best describes your last work experience program? (Mark ONE only.)  It was not part of a formal education program  It was part of a high school program  It was part of a school program after hi gh school and below an Associate’s degree

program  It was part of an Associate’s degree program  It was part of a Bachelor’s degree program  It was part of an advanced degree program or other program beyond a Bachelor’s degreeAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—7 

### General Issues

We identified two areas of respons e error across multiple survey s ections of the English version of ATES. The first type of error was respondent s’ tendency to mark a response to only one item among a list of items that calls for a response to each item in the list. We tried several variations on item formatting and instructions, and ultimately chose to include the instruction “(Mark one box for EACH ITEM below.)"

The second type of error we encountered ac ross sections was unemployed respondents who missed the “Not applicable, not currently workin g” response option that applied to them. To increase the salience of this option, we moved it to appear as the first opti on rather than the last. 

## Summary

AIR conducted four rounds of cogn itive interviews to test questions for their potential to measure the prevalence of subbaccal aureate credentials and participation in and ch aracteristics of workrelated training among adults in the United Stat es. During the cognitive testing, the AIR team found that some respondents had problems (1) unde rstanding the purpose of several items, and (2) locating the correct response option. To addr ess these problems, the questions and response options were simplified and re-ordered in some cases, and instructions we re added or modified. In addition, one new item was added to the certifi cations and licenses sec tion to help identify false positive reports.American Institutes for Research Co gnitive Interview Report for ATES 2016—8American Institutes for Research Co gnitive Interview Report for ATES 2016—9 

## Appendix A. Cognitive Interview Protocol

Interview ID: Date of Interview: First Name: Interviewer



### STEP 1: WELCOME AND INTRODUCTION SCRIPT
Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer. Hello, my name is < your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today. [Create small talk to build rapport with the respondent] You are here today to help us out with a survey that we are working on for the U.S. Department of Education. This survey asks about adults’ educati on and training experiences. I will ask you to complete the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about your answers. This will help us make the survey better and clearer for other adults like you. To make sure we learn everything we can from this session, I am going to be taking notes while you are answering the survey and while we are talking [ INSERT IF THERE IS AN OBSERVER : and my colleague will also be observing our session today behind the glass or by video ]. Also, this interview will be audio recorded so re searchers can review the recording later. Any information that refers to you personally, like your name or employer, will not be included in our report, and your respon ses will be used for research purposes only. Remember, you are not being tested; all we want is to learn how well the survey works. If at any time you want to stop, just let me know. [If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.] Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on res pondent preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent. If the respondent has consented to recording, start the recorder.]



### STEP 2: THINK-ALOUD PRACTICE While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you decide on your answers will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give. To help you get started on thinking aloud, we are going to do a practice activity. [Show an example by asking yourself the question and answering it using ‘think-aloud’] I’m going to give you a piece of paper with a question in it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

[Give respondent handout with think aloud practice .]American Institutes for Research Co gnitive Interview Report for ATES 2016—10 How many glasses of water did you drink yesterday?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

[ Hand respondent a copy of the questionnaire and a pen or pencil.] 

### STEP 3: COMPLETION OF THE QUESTIONNAIRE Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would read to yourself if I were not here. Remember that I am interested in hearing you talk about what you are thinking as you figure out your answers to the questions. [REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

Useful Probes—Use only as needed when res ponse is unclear or R appears confused or

unsure Can you tell me in your own words what that question was asking? What does the word [term] in this question mean to you? How easy or difficult is it to remember [topic]? How easy or hard was it to choose an answer? Tell me what you are thinking. How did you come up with that answer? You said [answer]. Can you te ll me more about that? [In response to R actions or comments]: Tell me what happened? Tell me what you were doing? [If you pick up on a visual cue like a furrowed br ow or a puzzled look]: Tell me what you are thinking. [If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: Tell me what just happened. Tell me about what you were just doing.



### Education

STOP RESPONDENT AFTER Q5 – Probe on Q1-5 NOTES

ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.

Q1-Q4 : It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something thatAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—11 indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q5.

Since leaving high school, have you taken any literacy classes to help improve your reading? Do not include college-level classes.

( Ask everyone ) What types of classes would you include as literacy classes? “Your” underlined. Probe change – to ask what classes are literacy classes.



### Certifications and Licenses

STOP RESPONDENT AFTER Q15 – Probe on Q6-15

Q6-7: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q8: [ Only probe if response to Q7 > 1. ]

The next few questions ask about the certification or license that you consider to be your most important. What is the name of your most important certification or license?

What do the words “most important” mean to you in this question?

How did you decide which certification or license was your most important?

How easy or hard was it to choose your most important certification or license?

Q9: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confus ion or an inconsistent interpretation compared to other respondents.

Q10: Is your most important certification or license required by a federal, state, or local government agency in order to do that kind of work?

Who issued your certification or license?

(If R seems confused) How easy or hard was it to answer this question?

Q11-14: It is not necessary to probe on this question. Let the respondentAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—12 continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q15. How useful has your most important certification or license been for each of the following? (a-d)

[If respond to Q7 is > 1]: How easy or hard was it to only answer about your most important certification or license for each question?

Q8 through 15 Tell me a little bit more about your most important certification or license— in your own words, what was the process you went through in order to get it?

Note to interviewers: What we’d like to find out is whether R really has a certification or license, and whether they have one but also have a certificate from a course or a program (a series of courses). Information about courses or training they took to prepare, where they took it, whether they had to pass a certification or license exam, where they took that exam, and who issued the certification or license all provide good clues.

Keep in mind that someone might go to a school (or other organization) to take course(s) to prepare for this ty pe of credentialing exa m, then take the exam and have both (1) a certificate of completion (30a) or post-secondary certificate (30d) from going through this preparation process as well as (2) a certification and/or license issued after passing the credentialing exam. Therefore, some Rs report someth ing under both certifications and certificates and it’s legitimate for them to do so. Other Rs just confuse the two and report certificates under certifications and vice versa.

To help you understand the difference, a certification or license is recognized in the R’s field as proof of q ualification to do a particular type of job, and usually requires an “official” exam that isn’t specific to a particular school or other training provider—everyone in the state or country would take the same exam, regardless of whether they took any preparation courses or not. A state usually issues licenses, and professional associations (like the American Association of Medical Assistants) usually issue certifications for their particular field.

In contrast, a certificate simply indicat es that the R has participated in some type of education or training to le arn content in an area. It does not “authorize” them to practice in that area. A certificate of completion (30a) is usually based on a single course and offered by a variety of providers, and a post-secondary certificate (30d) is awar ded by a school after someone has completed a specific series of courses (pr ogram of study), just like a college degree. It is considered a college creden tial, but generally takes less time to get than an associate’s degree.

Note also that many colleges or other organizations partner withAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—13 credentialing bodies to host their exams , so a R might take courses or go through a certificate program at a co mmunity college and then take the certification or licensing exam there. The school doesn’t “issue” the certification, however; they are just ser ving as proctors for the state or association who does issue it. This makes it a challenge for us to differentiate between certifications and certificates, so we try to learn about the educational process Rs went throu gh as well as the type of exam they took, if any.

IF RUNNING BEHIND OR R IS REPORTING SOMETHING OTHER THAN LICENSES AND CERTIFICATIONS, MOVE THEM ON TO CERTIFICATES SECTION.

OTHERWISE, STOP RESPONDENT AFTER Q24 – Probe on Q16-24

Q16-24 (if answer “Yes” to Q16 – probe) : Tell me a little bit more about your second-most important certification or license—in your own words, what was the process you went through in order to get it?

Q16-24 . Do you have another currently active certification or license? (if answer “Yes” to Q16 – probe)

How did you decide which certification or license was your second most important?

How easy or hard was it to choose your second most important certification or license?

Q17 through 24 For questions 17 through 24, was it easy or hard to answer for just your second most important certification or license?

Did you think about other certifications or licenses when you answered these questions?

Q19. Is your second most important certification or license required by a federal, state, or local government agency in order to do that kind of work?

Who issued your certification or license?

Q20-23: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q24. How useful has your second-most important certification or license been for each of the following?(a-d) How easy or hard was it to only answer about your second most importantAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—14 certification or license for each question?

STOP RESPONDENT AFTER Q26 – Probe on Q25-29

Q25-29: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.



### Certificates

STOP RESPONDENT AFTER Q38 – Probe on Q30-38

Q30a. People sometimes earn certificates from an education or training program. These are different from certificatio ns or licenses. Do not include certifications or licenses here. Have you ever earned any of the following types of certificates? a. A certificate for completing a tr aining program from an employer, employment agency, union, software or equipment manufacturer, or other training provider.

(If R choose “yes”) Tell me a little more about what you’re reporting for
30a. In your own words, what was th e process you went through in order to get it?

(If R appears to be reporting same credential as in certifications/licenses) How is this related to the certification or license you reported earlier?

Note to interviewers: Similar to what we’re trying to learn in the certification and license section, we need to know more about how R got the certificate, from what type of organiza tion (get the name if you can), and if it is the same thing or related to the certification or license reported earlier (if applicable).

Q30d. People sometimes earn certificates from an education or training program. These are different from certifications or licenses. Do not include certifications or licenses here . Have you ever earned any of the following types of certificates? d. A certificate—not a degree—for completing a program at a community or technical college, or other school after high school. Do not include teaching certificates or college degrees.

(If R choose “yes”) Tell me a little more about what you’re reporting for 30d. In your own words, what was the process you went through in order toAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—15 get it? (If R appears to be reporting same credential as in certifications/licenses) How is this related to the certification or license you reported earlier?

Q32. Who gave you your last post-secondary certificate?

Who gave you your last post-secondary certificate (get name) ?

How easy or difficult was it to answer this question?

(

If R chooses “another college or university” or “Someplace else” and it’s not clear based on the name ): Can you tell me more about where you got your last post-secondary certificate?

Q33. About how many hours of instructio n did you complete in order to earn your last post-secondary certificate?

How did you determine your answer?

How easy or difficult was it to answer this question?

Q34: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q35. To earn your last post-secondary certificate did you have to complete...

Were there any other instructional re quirements needed to complete this program other than what is listed?

Q36: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q37 Is your current job related to yo ur last post-secondary certificate?

How easy or difficult was it to answer this question? Question order changed.

Q38: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.American Institutes for Research Co gnitive Interview Report for ATES 2016—16 

### Work-Based Learning Program

STOP RESPONDENT AFTER Q49 – Probe on Q39-49 Section name change – from Work Experience Program

Q39: Have you ever completed an internship, co-op, practicum, clerkship, externship, residency, clinical e xperience, apprenticeship, or similar program?

(IF YES) Tell me about your program:

 What job was it for?

 Could you describe a typical day in the program?

 What type of organizations were involved in it and what were their roles?

 (If applicable) Was it related to the [certification/license/certificate] that you reported earlier? (If yes) How?

Note to interviewers: Similar to th e previous sections, we’d like to find out if Rs are reporting something new here or something that they have reported previously Revised question wording. Probe on this item higher priority now.

Q40. ( If applicable ) If yes, What type of work was your last work-based learning program for? (Mark ONE only.)

Can you tell me how you chose this category? How easy or hard was it to find your answer on that list?

Were there any categories that are missing?

[Pay attention to R and if they want to select the category headers that don’t have check boxes; probe if they are uncertain of where to mark their response]

Q41-42: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q43 a-c. ( If applicable ) As a part of your last work-based learning program did you...

(If yes to any of a-c) : Can you tell me a bit more about the connection between your training program and the classes, training, or studying you did? Question wording changedAmerican Institutes for Research Co gnitive Interview Report for ATES 2016—17 Q44: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q45. (If applicable) Which one of the follow best describes your last workbased learning program?

Was it easy or hard to answer this question?

Please tell me about how you chose your answer for this one? New response option (3 rd & 6 th option); Changed response order

Q46. ( If applicable ) Did (or will) your last work-based learning program help you earn a professional certification or license?

Can you tell me in your own words what that question is asking?

Q47-Q49: It is not necessary to probe on these questions. Let the respondent continue the think alou d and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation co mpared to other respondents.



### Employment [If short on time these sections are not priority]

STOP RESPONDENT AFTER Q70 – Probe on Q50-70

Q50-Q67: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q68. ( If applicable) Did you have a license that was required by a federal, state, or local government agency to do this job?

In your own words, what is this question asking?

[Check for consistency, if they answered inconsistently between either Q10/19/28 and Q68 then probe] You reported in (Q10/Q19/Q28) “yes/no”American Institutes for Research Co gnitive Interview Report for ATES 2016—18 but you reported “yes/no” in Q68. Could you tell me the difference between these two questions?

Q69-Q70: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. 

### Background [If short of time have them skip to Q82-84 after the Work-Based Learning section]

Q71-81: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Q82. Do you have Internet access on a cell phone?

Please tell me about how you chose [no, yes] for this one. Revised question wording (changed smartphone to cell phone)

Q83. Do you have Internet access at home on a computer or tablet?

Please tell me about how you chose [no, yes] for this one.

[If not obvious from response], Where are you thinking about? [Find out where “at home” was interpreted by R to be.] Q84. How often do you use the Internet?

Please talk to me about how you got to your answer here.

[Look for indications that the response options are not inclusive enough/ adequate.]



### FINAL QUESTION Is there anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? (IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS 

### STEP 4. THANK YOU FOR PARTICIPATING We are finished. Thank you for helping us. The info rmation you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort them out of the interviewing room and to the lobby. Please give the respondent incentive envelope [ $40], and thank them for their time and effort.LOCATIONS

Domestic

Washington, D.C.

Atlanta, GA

Baltimore, MD

Chapel Hill, NC

Chicago, IL

Columbus, OH

Frederick, MD

Honolulu, HI

Indianapolis, IN

Naperville, IL

New York, NY

Rockville, MD

Sacramento, CA

San Mateo, CA

Waltham, MA

International

Egypt

Honduras

Ivory Coast

Kyrgyzstan

Liberia

Tajikistan

Zambia1000 Thomas Jefferson Street NW Washington, DC 20007-3835 202.403.5000 | TTY 877.334.3499

www.air.org

# Memo

1 Date: May 20, 2015

To: Sarah Grady, NCES

From Danielle Battle and Meghan McQuiggan, AIR

Re: Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews



## Introduction

In 2016, the National Household Education Survey (NHES) will field the Parent and Family Involvement in Education Survey (PFI), which contains a homeschool component for households with homeschooled children. This report summ arizes the findings from the supplementary homeschool cognitive interviews conducted to te st and revise a subset of PFI items based on potential issues initially observed in the 2012 collection and the 2014 Feasibility Study. This report provides supplemental information to the “2016 Parent and Family Involvement and Early Childhood Program Participation Surveys Cognitive Laboratory Final Report” (Cole, Gonzalez, Fronberg, & Chavez, 2015).

In 2014, the National Center for Education Stat istics (NCES) conducted a feasibility study using the NHES. Sampled households were first sent a screener survey, which asked individuals to enumerate the members of their household, as well as answer a few questions about each person. The responses to the screener items then determined if someone in the household was eligible for one or both topical surveys: the Afterschool Programs and Activities (ASPA) survey and the Adult Training and Education (ATES) survey.

One screener item asked respondents to report th e enrollment status of each household member. The response options included “public or private school, or preschool”; “homeschool instead of school for some or all classes”; “college, univers ity, or vocational school” ; and “not in school”. Depending on the version of the screener the house hold received, the homeschool option would have been presented either first or second.

Cognitive interviewing and analysis of the feasibility study data revealed that the concept of “homeschooling” may be unclear to some responde nts. All households who received the ASPA survey had indicated on the screen er that the sampled child was enrolled in public or private school or preschool. However, in the ASPA topi cal questionnaire, about 2 percent of sampled children were reported to be homeschooled. If respondents selected “homeschool” on the topical survey, they were also asked how many hours pe r week the child attended a public or private school. Of those who reported that the child wa s homeschooled, 34 percent indicated the child spent 25 hours or less per week in public or priv ate school, suggesting that these children are2 likely “true homeschoolers.” The other 66 percent, who reported 25 hours or more per week in a traditional school setting, likely misinterpreted the meaning of “homeschooling” (Jackson, McQuiggan, & Megra, forthcoming).

Although these findings only apply to a small pe rcentage of NHES respondents, they suggest that there is ambiguity over the term “homesc hooling,” causing some parents of homeschooled children to receive a topical survey intended for parents of enrolled children. As a result, NCES decided to conduct a supplemental round of cognitiv e interviews specifica lly with parents of homeschooled children. In these in terviews, parents were asked to complete the PFI survey intended for parents of children enrolled in public or private school (PFI-Enrolled) but which contained a section about homeschooling. The go al of the interviews was to examine how parents of homeschooled children would respond to questions about traditional “brick and mortar” school environments, since homeschooli ng households that misreported on the screener might receive this instrument in future NHES administrations.

Cognitive interview respondents were recruite d through personal contacts and Craigslist in Washington, DC (4) and an interviewer contact in rural Michigan (1). The homeschooled children reported in the topical instruments ranged in age from seven to 18. When asked the children’s grade equivalenc ies, two parents reported second grade, one reported sixth grade, one reported ninth grade, and anothe r reported twelfth grade. Three respondents reported that their child attended a traditional school before hom eschooling and two respondents reported their child had always been homeschooled. All resp ondents indicated that their children were homeschooled full-time. Respondents were given a $40 American Express gift card for their participation.

The purpose of this memo is to provide key results from the five supplemental homeschool cognitive interviews conducted using the PFI-E nrolled questionnaire. This memo summarizes the main issues found, as well as provides reco mmendations for making these items clearer for both parents of homeschooled and enrolled children. All questionnaire changes based on interview findings are bolded in the item discussions below.

This memo highlights six key findings:

 Screener: In total, respondents were able to navi gate the screener questionnaire without any problems. When probed on the meaning of “homeschool,” all resp ondents articulated a clear understanding of this concept. Some parents noted that the idea of gradeequivalency may be more difficult for homesch ooled children than enrolled children.

 Traditional school items: Homeschool parents often verba lized that these questions did not apply to them. At times, pa rents left items blank, and at other times, they satisficed by answering based on their homeschooling situation.

 Online instruction items: Multiple homeschool respondents did indicate that their child participated in online courses, but they were often unable to find an instruction provider among the response options provided that matche d their child’s educa tional experience.

 Homeschool items: Parents were able to easily answ er questions about their child’s homeschooling. The items about subjects ta ught during all homeschooling and taught3 currently were the same for most respondents, depending on th e flexibility of the child’s curriculum. It may be difficult for parents to report on subjects taught “now” based on how the child’s curriculum is delivered acro ss a school year and how the term “now” is interpreted.

 Health, Child’s Background, Househol d Members, Child’s Family sections: Respondents were able to navigate these sections without many problems.

 Your Household section: This new section worked well, and respondents were able to easily answer the new Internet items. All respondents indicated frequent Internet use, including for homeschooling purposes, and repo rted access on smartphones, tablets, and computers.


## Key Findings

Screener Instrument

Overall, respondents encountered few problems with the screener instrument. On question 5, which asks about the enrollment status of house hold members, two respo ndents initially marked “homeschool” when reporting in formation about themselves. Afte r they realized their mistake and corrected their responses to “not in school ”, both parents verbalized that they were mistakenly thinking about their children when answer ing. It is likely that th is pattern was due to a priming effect in which respondents knew they were recruited as homeschool parents, which impacted their responses and should not be reflected in the larger data collection.

After respondents selected “homeschool” for thei r children’s enrollment status, interviewers probed on the meaning of homeschooling, and it was evident that these homeschooling parents had a clear understanding of th is concept. One respondent sa id his child is homeschooled because “we elect not to send her to a public scho ol or private institution and to educate her ourselves”. Another said that his child is homes chooled because he is “enrolled in an accredited program that we administer at home...most pe ople consider it homeschool because we are the principal educators.” One respondent added that homeschooling is a “global term” encompassing a spectrum of arrangements and philosophies. He de scribed how his children are enrolled in an accredited and structured homeschooling curri culum, while others, sometimes called “unschoolers,” practice more expe riential learning. This respondent did note that he is unsure if “unschoolers” would categorize their childre n as “homeschooled,” and suggested adding language to clarify that “homeschooling” includ es all curriculum-base d, distance education, and experiential learning methods.

For all household members enroll ed in a school program, respondents were supposed to answer question 6, which asked for the person’s current gr ade equivalent. One res pondent noted that this question was difficult for him to answer since his child does not progress through traditional grades, but said he would write 2 nd grade since she was seven years old. This respondent suggested using “age appropriate grade” to make this language applicable to homeschooled children. Another respondent commented that this could be challenging since some homeschool instruction is self-paced, and a child may be on- grade for some subjects, while ahead or behind4 grade-level for others. Overall, these suggestions should result in furthe r testing of this item wording and alternative item wording before used in data collection.



### Topical Instrument

Child’s Schooling

The first half of this section was challenging for homeschool parents. Multiple respondents expressed that these items seemed to apply only to children in public a nd private schools. For example, question 2 asked the type of school the child attends. Three respondents left this item blank, saying that no response option applied to their situation. One pare nt marked “private, nonreligious,” but added that most homeschool pa rents wouldn’t categorize homeschooling as such. Respondents noted that several items in this se ction did not apply to them, and one respondent recommended rewording the items to say “sc hool/ homeschool.” Another parent suggested general instructions telling ho meschool parents to think of their “home learning environment” when reading the questions about “school.” Another respondent me ntioned that they would have thrown away the survey if they received it in the mail after trying to answer the first few items that did not apply to their homeschool child.

Question 3 on the topical questionnaire asked respondents if the child attended his or her “district-assigned” school. Three respondents marked “no” and one le ft it blank, noting that this item did not apply to homeschoolers. One responde nt marked “yes,” and when probed, said it was because his child is homeschooled in the ward where they live.

Question 7 asked if parents considered other schools for the child. One parent interpreted “schools” as “curriculum,” explai ning that they did investigat e other homeschooling curriculums for the child.

Question 8 asked respondents if they considered th e performance of schools, such as test scores and dropout rates, when selec ting a school for their child. On e respondent explained that monetary reasons might be another factor, as he prefers a non-traditional approach to education, but found that non-traditional private schools were too expensive. Another respondent added that learning outcomes were an important factor in selecting a program, as he wanted a homeschooling curriculum that emph asized reading and writing.

Question 14 asked parents if the child was en rolled in any high-school level Advanced Placement courses. All respondents reported “no” for this item, but one respondent noted that parents unfamiliar with public schools may not know what AP courses are. Another respondent added that homeschool parents may confuse this with community college courses, which many upper-level homeschool children take.

Question 22 asked if the child pa rticipates in any online course s or instruction. Two respondents reported “no,” saying that their children were too young. Another respond ent reported “no,” because they did not use the Internet for speci fic coursework but as a supplement to their teaching. One respondent said “yes,” explaining th at his family pays tuition for access to a distance education school in which the children have access to online courses, transcripts, and an5 advisor. Another marked “yes” and said that he r child uses K12, which also links the child to other online programs to practice the concepts.

If respondents indicate that the child is taking online courses, he or she is asked in question 23 to report who provides the online instruction. The two respondent s both wrote in the “other— specify” box, and reported “other online hom eschooling program” and “our homeschool.” NHES: 2012 data showed that homeschoolers often wrote other online schools or virtual academies into this box. This suggests that for this item, the response options may not be suitable for homeschooled students, re quiring additional testing fo r future administrations.

Question 25 asked if the sampled ch ild is “schooled at home instead of at school for at least some classes or subjects.” All respondents marked “yes” for this ite m, and two added that they could have just skipped to this question as the prev ious items about traditional schooling did not apply to them. It was discussed to move this item to the first page (where the item was located in NHES:2012) but NCES decided th at it could introduce additi onal error as those who do homeschool would be skipped out of items that could provide a dditional information about some types of homeschool households.

Question 26 asked parents to mark the statement that best described their child’s homeschooling arrangement. All respondents marked “homeschool ed for all classes or subject areas,” but one parent mentioned that he initially wasn’t sure ho w to answer since his chil dren participate in coops, which are outside of the home.

Question 28 lists a variety of reasons parent s might homeschool their children, and asked respondents to mark if each was a factor in the parent’s decision to homeschool the child. Among the respondents, “concerned about the school environment, such as safety, drugs, or negative peer pressure,” “dissa tisfied with the academic inst ruction at other schools,” and “interested in a nontraditional approach to ch ildren’s education” were the most common responses. One respondent’s main reason was his child’s deafness, which he categorized as a “physical or mental health pr oblem,” and another parent expl ained that her child was being bullied, which she considered “other special ne eds.” Another respondent reported in the “OtherSpecify” box that one reason not on the list was supervision. The responden t felt that if their child was homeschooled, they would be unde r the respondent’s direct supervision.

Although no parents selected “yes” for question 28i, “you prefer the flexibility homeschooling provides to this child during freque nt moves,” two respondents initiall y just read the first half of the item, interpreting it simply as the flexibil ity homeschooling provides. After one respondent read the remainder of the option, he wrote “fle xibility” in the “other—specify” box, explaining that homeschooling allowed his child to expl ore other educational and extracurricular opportunities. Further analysis wa s conducted using NHES: 2012 data and it was determined that “frequent moves” was not a common write-in response. Therefore, it was recommended that item 28i be deleted from the PFI-Enrolled a nd PFI-Homeschool instruments. The item was removed from the NHES:2016 instruments.

For this set of items, one respondent noted that it was difficult to mark “yes” or “no” since some were considerations, but not main reasons fo r homeschooling. He suggested a scale so that6 parents could indicate how much each factor weighed in their decision. Two respondents verbalized that some of the f actors were benefits they discov ered since starting homeschooling, but not initial reason s, and they did not mark those options.

When asked to indicate the main reason for homeschooling in questi on 29, one respondent marked “h—nontraditional approach to education” and explained that they wanted to educate their child through a love of l earning. Others marked “g—speci al needs” and “e—physical or mental health problem,” explaining their child ren’s experiences with bullying and deafness, respectively. Two other respondents marked “a— concerned about the school environment.” One respondent described that after researching schools, he was wo rried about school security, and the other said that the child wa s enrolled in a public school that the parent felt was unsafe and they wanted to ensure the safety of th eir child and be able to protect them

Question 30 asked respondents to mark all of th e subjects their child had been taught since starting homeschooling. Most respon dents selected several subjects, but in general, parents of younger children did not select upper level mathem atics and science courses. When probed on how those courses were covered, respondents had a variety of responses. One respondent explained that his child attends a curriculum-based co-op, as well as a co-op where parents teach courses such as yoga, museum studies, performing arts, etc. Another parent explained that her child’s academic subjects are taught online thro ugh K12, and the child meets up with other homeschoolers weekly for subjects like art, mu sic, and physical education. The third parent described a similar situation, where his childr en complete academic courses through an online curriculum, but go to an art studio for art, lear n piano from local colle ge students, and go to physical education clas ses with other homesc hoolers. Another responde nt explained that a certified instructor comes to his home daily to deliver the academic curriculum, and the child does not participate in activi ties outside of the home.

Question 31 asked parents to report on the cour ses the child is taking “now”. Three respondents marked all of the same subjects as in question 30, noting that these two items are the same for homeschoolers. However, two respondents selected different subjects for this item based on the courses covered this year in the child’s curricul um. One parent questioned if “now” meant in the current season or in the academic school year, e xplaining that the child takes physical education in the spring, but not in the winter. Another pa rent had difficulty with the term “now” and was not certain if it meant today or the last few mont hs, and chose to select every subject she taught her child because they do not use a strict curri culum. The parent also recommended including an item about curriculum instead of this item.

Respondents were also probed on if any subj ects were missing from these lists. Some suggestions that parents provided included: performing arts, dance, handwriting, logic, religion/spirituality, gardening, a nd current affairs. One responde nt mentioned including a writein box since there are so many t opics covered through their homesc hooling that do not fall in the traditional school curriculum.7 

### Child’s Health

Respondents encountered no problems with the “Ch ild’s Health” items.



### Child’s Background

Overall, respondents had no problems with the “Child’s Background” items. Question 44 asked if the child lives “at this addre ss and another address.” One parent initially marked “yes,” reading only the first half of the item. After seeing “and another address,” she changed her response, but verbalized that parents may answer this item in correctly if they are ru shing through the survey. Therefore, it is recommended that this option be revised to “does this child live at this address and another address (for example, beca use of a joint custody arrangement)?” Thisrecommended revision was implemented for the NHES:2016 PFI-Enrolled, PFIHomeschool, and ECPP instruments.



### Household Members

Respondents had no trouble accurately reporti ng household totals and roster counts.



### Child’s Family

Overall, respondents easily navi gated the questions a bout the child’s pare nts. One respondent noted that he would prefer if the Parent 1 ques tions said “are you...” instead of “is this...”. For question 60, which asks about em ployment status, one respondent mentioned that the wording “employed for pay or income” was odd, and anothe r questioned why “for pay or income” was not included with the “self-employed” option. One respondent had difficulty answering this item for his wife, as she is both a stay-at-ho me parent and employed part-time.



### Your Household

Due to inaccuracies in repor ting household members in NHES: 2012, it was decided that the “Your Household” section would be moved ahead of the “Child’s Family” section. However, this led to concerns about the income item bei ng too early in the instrument. Additionally, one homeschool respondent noted that th e internet access items in this section seemed out of place since the previous questi ons were about education. Therefore, it was recommended that the “Your Household” section be renamed “Household Members,” and a new “Your Household” section follow “Child’s Family,” co ntaining the items about benefits, income, home ownership, and Internet access. Th is recommended revision was implemented for the NHES:2016 PFI-Enrolled, PFI-Homeschool, and ECPP instruments. This revised format was tested in these supplemental intervie ws, and respondents had no trouble accurately responding to these items.

For Question 99, all respondents in terviewed indicated that th ey had Internet access on a smartphone, either through a mobile service provider or Wi-Fi. Four respondents indicated in question 100 that they had Internet on a comput er or tablet, and severa l mentioned that their8 children use the Internet on Kindles and iPads. One respondent did not have Internet access through a computer or tablet because they live in a rural setting and it is difficult to get Internet except through their phone s. In question 101, all respondents re ported using the Internet “every day,” noting that they check their own email dail y and use it for their children’s education. One respondent added that homeschool pa rents in particular are usually well-connected, as they use the Internet to participate in on line homeschooling groups and forums.



## Summary

Respondents in these interviews provided a subs tantial amount of inform ation regarding their interpretations of “homeschoo ling,” their children’s homeschoo ling arrangements, methods of instruction, and academic coursework. However, respondents did struggle with some schooling items, revealing questions that may benefit from additional testing as they are particularly challenging for parents of home schooled children. Despite di fficulty with select items, respondents were able to answer the majority of questions incl uded on the instrument, resulting in few changes to this and othe r applicable NHES surveys.

1. Since “frequent moves” was not a comm on write-in in NHES: 2012, and respondents may confuse this option as simply the flexib ility of homeschooling, item 28i should be deleted from the instrument. This cha nge applies to the PFI-Enrolled and PFIHomeschool instruments.

2. “And” should be underlined in question 44 to ensure that this ite m captures children who live at multiple addresses as opposed to just on e. This change applies to the PFI-Enrolled, PFI-Homeschool, and ECPP instruments.

3. The “Your Household” section should be renamed “Household Members,” and the benefits, income, home owners hip, and Internet access item s should be moved to a new “Your Household” section afte r “Child’s Family”. This change applies to the PFIEnrolled, PFI-Homeschool, and ECPP instruments.9 

## References

Cole, S., Gonzalez, R., Fronberg, K., & Chavez, S. (2015). 2016 Parent Family Involvement in Education and Early Childhood Program Partici pation Surveys Cognitive Laboratory Final Report . Washington, DC: U.S. Department of Ed ucation, National Center for Education Statistics.

Jackson, M., McQuiggan, M., and Megra, M. (forthcoming). NHES:2014 Feasibility Study Report . Washington, DC: U.S. Department of Ed ucation, National Center for Education Statistics.10 

## Appendix: Questionnaire and Interview Protocol111213141516171819202122232425262728 

## 2016 Parent and Family Involve ment in Education Survey (PFI) Cognitive Interview Protocol (for parents of homeschooled children)

Interview ID: Date of Interview: Parent First Name: Interviewer

Age of Child:



### STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is < your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the parent]:

You are here today to help us out with a new survey that we are working on for the U.S. Department of Education. This survey asks about your child’s schooling. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questi ons I will ask you about your answers to the questions. This will help us make the survey better and clearer for other parents just like you. Because the information you provide is so important, I am going to be taking notes while you are answering the survey and while we are talking.29 Also, this interview will be audio-recorded so re searchers can review the recording later. Any information that refers to you or your family personally, like your name or the name of your child, will not be included in our report, and your res ponses will be used for research purposes only. Remember, you are not being tested; we want you to tell us what you think about the survey. If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the parent may ask.]

IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER WITH A COPY OF THE FORM. ASK THE RE SPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WI TH THE RESPONDENT. IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.



### STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you figure out your answers to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give me. To help you get started on thinking aloud, we are going to do a practice activity.

First, I will show you an example of how to think out loud. Then I will give you a chance to practice
it.

QUESTION: How many glasses of water did you drink yesterday?

[Model the think aloud process for the parent, including reading the question out loud. Elaborate on counting all of the glasses of water and as needed you can make up pieces of information to emphasize the think aloud process.

[Once finished the model practice think aloud]

I’m going to give you a piece of paper with the question on it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

Give respondent handout with think aloud practice

.

How many glasses of water did you drink yesterday?

Please tell me what you are thinking while you are figuring out your answer.30 Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL



### .

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here. Remember that I am very interested in hearing y ou talk about what you are thinking as you figure out your answers to the questions.

REMEMBER TO TELL THE PARENT TH ROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.31 Useful Probes Can you tell me in your own words what that question was asking? What does the word [term] in this question mean to you? How easy or difficult is it to remember [topic]? How easy or hard was it to choose an answer? Tell me what you are thinking? How did you come up with that answer? You said [answer]. Can you te ll me more about that? [In response to R actions or comments]: Tell me what happened? Tell me what you were doing? [If you pick up on a visual cue like a furrowed br ow or a puzzled look]: “Tell me what you are thinking.” [If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Screener
ASK THE RESPONDENT TO FILL OUT THE ENTIRE SCREENER. REMIND THEM TO THINK ALOUD IF THEY ARE NOT DOING SO. Q3. What is the person’s month and year of birth?

[If the respondent is having difficulty with the month and/or year] How did you come up with your answer?

Q5. Is this person currently in...?

How did you come up with your answer?

[If respondent chooses “homeschool”] What does “homeschool” mean to you?

PFI Survey CHILD SELECTION: SELECT THE FIRST CHILD AGES 5-17 ON THE SCREENER THE R REPORTED TO BE CURRENTLY HOMESCHOOLED.

Child’s Schooling

ASK THE RESPONDENT TO FILL OUT THE SURVEY ABOUT THE CHILD SELECTED. ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY; REMIND RESPONDENT TO THINK ALOUD32 Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.

Q1-24: P ay attention to how the R reacts to these items and take notes and probe accordingly. Since the child is homeschooled these items may not apply. If R is having difficulties remind the R to fill out the survey as they would if they received it at home.

[ Probe if necessary ] How would you react if you were to receive this survey in the mail at home? Q1-Q2 : It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Q3 . Is it his/her district-assigned school? A district-assigned school is the school that your local public school district told you that this child can attend, based on the location of your residence.

[Ask only if responded to this question] How did you decide on your answer?

What does “district assigned” mean to you?

How did you know what school your child was going to attend? How do other parents in your child’s district know which schools their child will attend? Q4-Q13 : It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q14. Is he/she currently enrolled in any high school Advanced Placement (AP) classes?

What were you thinking of when you answered this question?

What do “Advanced Placement (AP) classes”33 mean to you?

What grade-level are the students who usually take AP classes in your child’s school district?

Q15-Q21: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q22. Is this child taking any school-related courses online instead of in-person with the teacher? Do not include courses that use the Internet only for selected assignments.

How did you decide on your answer?

Q23. Is that instruction [internet] provided by any of the following places?

How did you decide on your answer?

[If chooses a response option or “Someplace else:”] Tell me more about that. How does your child access the classes? Where is he/she when accessing the classes? Who provides the class? Q24: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q25. Some parents decide to educate their children at home rather than send them to a public or private school. Is this child being schooled at home instead of at school for at least some classes or subjects?

In your own words, tell me what this question is asking about.

How did you determine whether to answer yes or no?

[If respond “yes,”] Do not probe on the meaning of homeschool until Q29.34 [If respond “no” and if not evident from think aloud to previous questions] What does “homeschool” mean to you?

Make a note if the respondent did not skip properly, but do not probe on skip pattern.] Q26. Which of the following statements best describes your homeschooling arrangement for this child?

How did you come up with this answer?

[Do not probe on the meaning of homeschool until Q29. Check to see if respondent skipped properly, but do not probe on skip pattern.]

Q27. How many hours each week does this child usually go to a public or private school for instruction? Do not include time spent in extracurricular activities.

How did you come up with your answer?

[Check to see if respondent filled in this question although they indicated that the child is homeschooled.]

Q28. There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool this child because:

How did you determine how to answer yes or no to each of these questions [a-j] ?

Tell me more what you were thinking when answering these questions.

[Do not probe on the meaning of homeschool until Q29.] Q29. Of the reasons your family chose to homeschool this child, which one would you say is the most important to you?

How did you come up with your answer?

[If not evident from think aloud to previous questions] What does “homeschool” mean to you?35 Q30. Thinking about all years this child has been homeschooled, which of the following subject areas has this child been taught during his or her home instruction?

How did you decide what to mark?

How do you cover these subjects during home instruction?

Is there anything missing?

Q31. Which of the following subject areas are being taught to this child now?

How did you come up with your answer? Can you tell me more about what the child is being taught now?

Child’s Health

REMIND RESPONDENT TO THINK ALOUD Q32-Q35: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q36. Are these services provided by any of the following sources?

How did you decide on your answer?

Q37. Are any of these services provided through an Individualized Education Plan (IEP) or services plan?

How did you come up with your answer?

What does a “services plan” mean to you?36 Child’s Background Q38-Q43: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Q44. Does this child live at this address and another address (for example, because of a joint custody arrangement)?

How did you come up with your answer?

In your own words, what is this question asking?

[If answered No] What does joint custody arrangement mean to you?

Q45. If yes, does this child spend most of the time at this address, at another address or equal time at both addresses? How did you come up with your answer?

[If needed] Can you describe the arrangement your child has?

Q46. What language does this child speak most at home?

It is not necessary to probe on this question. Note the respondent’s response, especially for those who are parents of ELL students. Record in your notes whether the respondent reads the response options below the top two responses.

Q47 . It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Your Household Q48. How many people live in this household?

How did you come up with your answer?37 Who are you including in the counts?

Q49. How many of the following people live in this household with this child?

Tell me more about what you were thinking when you answered this question?

Who are you including in the counts?

In your own words, what is this question asking? Q50-Q57: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Pay particular attention Q53, noting if the respondent seems uncomfortable answering the income item. Q58a. Do you have Internet access on a smartphone?

Please tell me about how you chose [no, yes] for this one.

Q58b. Do you have Internet access at home on a computer or tablet?

Please tell me about how you chose [no, yes] for this one.

[If not obvious from response], Where are you thinking about? [Find out where “at home” was interpreted by R to be.] Q59. How often do you use the Internet?

Please talk to me about how you got to your answer here.

[Look for indications that the response options are not inclusive enough/ adequate.]

Child’s Family

REMIND RESPONDENT TO THINK ALOUD Q60-Q71: It is not necessary to probe on these questions. Let the respondent continue the38 think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Q72

. What is the highest grade or level of school that this parent or guardian completed?

[Only if respondent chooses “high school diploma or equivalent”, “vocational diploma after high school”, or “some college, but no degree”:] How did you come up with your answer?

What does “vocational diploma after high school” mean to you?

Q73. Is he or she currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

In your own words, what is this question asking?

Q74-Q100: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.39 FINAL QUESTION

Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? 

### STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The info rmation you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to the lobby. Please give the respondent incentive envel ope [$40 gift card], have the respondent sign the receipt, and thank them for their time and effort.

# Memo

1 Date: June 3, 2015

To: Sharon Boivin and Sarah Grady, NCES

From Anna Sandoval Giron and Meghan McQuiggan, AIR

Re: Preliminary Findings from the 2015 Spanish Cognitive Interviews



## Introduction

In 2016, the National Center for Education Statistics (NCES) will conduct the second full-scale, mail-based administration of the National H ousehold Education Survey (NHES). The NHES uses a two-stage design in which sampled households are first sent a screener survey and asked to list the members of their household, as well as provide some key information on each individual, including their date of birth, sex, and school enroll ment status. The responses from the screener survey determine if any household me mbers are eligible to receive one of the NHES topical instruments, which ask additional questio ns about the sampled i ndividual’s education.

In 2016, the NHES will contain thr ee topical surveys: the Early Childhood Program Participation (ECPP) survey, the Parent and Family Involveme nt in Education (PFI) survey (Enrolled and Homeschool), and the Adult Training and Educa tion Survey (ATES). All contact materials, screener instruments, and topical instruments are administered in both English and Spanish. Additionally, the NHES:2016 will contain a web experiment in which some respondents will be asked to complete the screener and topical surveys online.

In order to reduce measurement error and ensure high-quality translations, all Spanish materials are being tested using cognitive interview methods. Spanish interviews are being conducted in three phases. Phase 1 tested the Spanish screener and topical letters, postc ard reminder, screener questionnaire, and new ATES items that had not pr eviously been translated. Phase 2 tested the topical questionnaires, includi ng the ECPP, PFI- Enrolled, an d ATES. These instruments contained translations from the 2012 administration, as well as new translations due to item revisions after the English cognitive interviews. Phase 3 will test the contact materials for the web survey, however those findings are not yet available.

Phase 2 and 3 were ongoing but mostly complete, at the time this memo was written The memo documents all findings to date from the Spanis h interviews, and no further substantive changes are anticipated.2 

## Sample

Phase 1 interviews took place in Washington, DC, San Mateo, CA, and Houston, TX with 26 respondents who are Spanish dominant speakers (t hey either speak only Spanish or are most comfortable speaking Spanish). Phase 1 interviews were conduc ted between March and April 2015.

Phase 2 interviews took place in Washington, DC, San Mateo, CA, Houston, TX, and Tucson, AZ with 82 respondents (to date ) who are Spanish dominant; – 20 on ECPP, 38 on PFI-Enrolled, and 24 on ATES. Phase 2 interviews were conducted between April and May 2015.


## Key Findings: Phase 1

Screener and Topical Letters

General Issues

In the first round, respondents had difficulty with the names of government agencies, as translating these organizations of ten led to sentences that were quite lengthy. For example, the opening sentence from the screener letter, incl uding the Spanish version, is shown below:

I am pleased to inform you that the U.S. Census Bureau has selected your household to participate in the 2015 National Household Ed ucation Survey, which we are conducting on behalf of the U.S. Department of Education

.
Me place informarle que la Oficina del Censo de los Estados Unidos ha escogido su hogar para que participe en la 2015 Encuesta Nacional en Hogares sobre Educación, la cual estamos realizando para el Departamento de Educación de los Estados Unidos

.
Respondents often skipped these sentences or did not understand the information. When asked who sponsored the study, six of the 20 respondent s understood that it was the U.S. Department of Education. Four respondents sa id “Census,” and others repor ted “Department of Commerce,” “the government,” “teachers,” a nd “the schools.” Three respondent s said they did not know who was sponsoring the study.

Based on these results, it was recommended that the opening paragraph of all letters be simplified so that the language is easier for r eaders to understand. For ro und 2, this wording was changed to the following:

I am pleased to inform you that your househol d has been selected to participate in the 2015 National Household Education Survey. This is a U.S. Department of Education survey that is being conducted by the U.S. Census Bureau

.3 Me complace informarle que su hogar ha sido es cogido para participar en la Encuesta Nacional en Hogares sobre Educación del 2015. Esta es una encuesta del departamento de Educación realizada por la Oficina del Censo de los Estados Unidos.

After the wording was changed, six of the seven respondents in Round 2 interviews reported that the study’s sponsor was the “Depar tment of Education,” “Census Bu reau,” or “the government.” Therefore, it is believed that simplifying this language made it clearer to respondents that the study was sponsored by a government entity.

At the request of NCES, the term “conducted” wa s changed to “administered” in the English version, but the Spanish tran slation remains the same.

Translation Issues

Both the screener and topical letters include the phrase “toke n of appreciation,” which is translated as “dinero en efec tivo.” In round 1, 19 of 20 responde nts believed that this meant “cash.” For round 2, the phrase was re-translated as “muestra de aprecio.” No respondents in round 2 had any difficulty with this language, and it is believed that th is translation better captures the concept of “token of appreciation.”

The letters also contained the phrase propósitos estadísticos, “statistical purposes.” Several respondents did not understand this wording, sugg esting the need for an improved translation. For round 2, the wording was changed to fines estadísticos in an attempt to make the text shorter and more readable.

The ECPP topical letter sp ecifies that the study pr ovides information about educación de niños a temprana edad , “children’s early learning,” however, in the questionnaires, this was translated

as educación a niños pequeños , “education for small children.” In order to be consistent, it was recommended after round 2 that all references to children’s early learning match the language used in the questionnaires.

The ATES topical lett er specifies that th e survey is about adiestramientos y credenciales que los adultos reciben para trabajar, “training and credentials that adults get for work.” Several respondents reported that the word adiestramientos, “training,” seem s to refer to animal training, and instead suggested using the word entrenamiento . Once this revision was made for round 2, respondents had no issues with this language.

At the request of NCES, some references to “w ork” were dropped from both the English and Spanish letters to place less emphasis on employment because respondents are not required to be working to be eligible for the survey.4 

### Screener Questionnaire

General Issues

Question 5 on the screener asks respondents fo r the individual’s school enrollment status. For people not in school, the response option pr ovided is a word-for-word translation, no está en la escuela , “not in school.” In round 1, respondents had trouble with this it em, often looking for a response option that said ninguna , “none,” as is used in many other Spanish surveys. For round 2, ninguna was used in place of no está en la escuela, and respondents who were not in any of the school categories had less trouble finding an applicable response.

Respondents also had trouble with the homeschooling response op tion in question 5. This option, in English and Spanish, is below:

Homeschool instead of attending a public or private school for so me or all classes.

Escuela en el hogar (homeschool) en lugar de asistir a una escuela pública o privada para algunas o todas las clases.

The concept of homeschooling confused multiple respondents. Generally, they either thought this was referring to a parent supporting a ch ild’s education at home , or an adult using educational materials for their own self-imp rovement purposes. Due to the perceived low prevalence of homeschooling among Spanish-spea king families, this is believed to be a conceptual issue because Spanish respondent s simply are unfamiliar with homeschooling. Therefore, it was recommended that any translat ion revisions should only be made to address substantial changes in the E nglish homeschooling questions, wh ich were tested separately.

Respondents also had difficulty understanding qu estion 6, which asks for each individual’s current grade equivalent, or “gra do actual de esta persona, o su equivalente.” Some respondents in cognitive testing interpreted th is as meaning the highest level of education the person had completed, particularly for indivi duals not enrolled in school. Ho wever, no changes were made.



## Key Findings: Phase 2

ECPP, PFI, and ATES

General Issues

Overall, respondents across the th ree topical surveys had difficulty following skip patterns. In round 1, eight of 10 respondents did not skip to ot her sections or questi ons when they should have. When further probed, two respondents comme nted that in other “official” questionnaires they had been told to fill in all possible informa tion, so they chose to complete as many questions as possible, disregardi ng skip instructions.5 Across all surveys, respondents al so found questions related to race challenging. In the first round, four of 10 respondents commented that they could not find an adequate response option (Hispanic or Latino) among those provided. Some eventually chose White, commenting that that is what they are “supposed” to choose.

No changes were made to skip instructions in the questionnaire since respondents were making the more desirable reporting error of commissi on instead of error of omission, however, NCES intends to implement post-survey edits for skip patterns which will take into account this cognitive finding.



### PFI and ECPP

General Issues

These two surveys contain a question asking if “thi s child is currently enrolled in English as a second language, bilingual education or an English immersion program?” Three parents expressed that they thought this was asking if the teacher could communi cate with the child in Spanish. It is believed that this is both a translat ion issue, and a conceptual one in that parents are unfamiliar with “immersion programs.” In Round 2 we discovered that the issue was the translation of immersion programs. The or iginal translation of the question used programa especial de inglés to refer to immersion programs. In a ddition the original translation used the

term lengua , “tongue” to refer to language. We determ ined these two terms/ phrases were causing confusion to respondents and suggested a new tr anslation of the questi on that reflects the language commonly used by schools and parents. NCES implemented this recommendation.

New suggested translation: Actualmente, ¿asiste este niño(a) a clases de inglés como segundo idioma, educación bilingüe o a un programa de inmersión al inglés?

Translation Issues

These topical instruments also as k parents to report any of the sampled child’s health conditions. Although two respondents were confused by this s ection and chose to leave it blank, the others had no problems. However, two terms in this section proved challenging for respondents. The phrase impedimiento del habla , “language impediment,” was marked with a yes by some parents. During probing these parents explained that they chose yes for this option because their children are learning to talk in two languages and have not mastered one of th e languages. NCES has decided not to make any changes to the survey based on this finding.

Another respondent had difficulty with the term, Trastorno generalizado del desarrollo (PPD)

, “Pervasive Developmental Di sorder (PDD).” AIR research ers have followed-up on this translation and reviewed the wording in the U.S. Department of Health and Human Services. The phrase Trastorno generalizado del desarrollo (PPD) is the commonly used translation to refer to these disorders. Currently we believe that pare nts that have a child diagnosed with Pervasive Developmental Disorder will be able to identif y themselves and correctly answer the question. No change is recommended.6 

### ECPP

General Issues

In round 2, parents with children in home-based daycare had difficu lty categorizing this type of care. When asked if the child receives home-ba sed care from a non-relative, parents of children in home-based daycares often responded “yes.” Ho wever, when respondents got to the series of questions about daycares, presc hool, and prekindergarten programs not in private homes, they wanted to report the programs in this section as well, and it seemed th at respondents were not reading the “not in a private home” text. Overall, four of seven respondents in this round had response error in this set of questions. No change was recommended.



### ATES

General Issues

Similar to the English ATES interviews, res pondents had difficulty distinguishing between certifications, certificates, and licenses.

In the second round, it was discovered that re spondents with educat ion and licensing from foreign organizations and governments often had difficulty categorizing their credentials. One respondent explained that in her country, her high school educati on is considered two separate certificates- she can earn both a high school degree and technical career degree. It is believed that this issue may not be limited to Spanis h speakers, but to many immigrant groups.

Translation Issues

The ATES questionnaire asks res pondents if their most important certification or license can be revoked or suspended for any reason. “Revoke” is currently translated as “revocada,” but the

term revocada produced confusion for some respondents. Due to the confusion around this term, it was recommended that this language be revised to quitar , which, is a synonym of revocada meaning means “remove” or “take away.” .The cu rrent and recommended tran slations are below.

English version: Can your most important certification or license be revoked or suspended for any reason?

Current translation: Su certificación o li cencia más importante, ¿podría ser revocada o suspendida por alguna razón?

Suggested translation: ¿Le podrían quitar o su spender su certificación o licencia más importante por alguna razón?7 

## Summary

Overall, these two phases of cognitive interviews provided a lot of insight into Spanish speakers’ interpretation and understanding of NHES contact materials and questionnaires. Respondents were able to highlight areas of confusion a nd problematic language, which led to improved wording and translations. For example, in the sc reener and topical lett ers, introductory text containing government agency names was simplifie d to make it more readable. In the screener questionnaire, the response option for individuals not in school wa s revised to reflect a response option that Spanish speakers are more familiar wi th. Testing of the screener questionnaire also reinforced that homeschooling continues to be a challenging concept for Spanish speakers.

Cognitive testing also revealed concepts and lan guage that needed to be revised on the topical questionnaires. While some translation change s were made during testing, others require additional testing and investigati on before revisions can be made. Across all of the instruments, some of the items that were difficult incl uded Spanish immersion programs, child language impediments, school detentions and suspensi ons, home-based daycare programs, and the revocation of adult credentials. The final analysis of the interview data, as well as future cognitive interviews, will continue to inform the development and refinement of Spanish NHES

surveys.8 

## Appendix A: Phase 1 and Phase 2 ProtocolsPhase 1 Protocol

9 Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total interview time with the respondent lasts 1 hour or less.

Before starting the interview, use this table to confirm the participant info that we received from the recruitment firm. If the respondent indicates that some of the listed information is incorrect, please cross it out and enter the correct information in the relevant cell.*

Background Information Confirmed 

Date & Time

Name of Interviewer

Parent/Respondent First Name

Respondent’s Gender  Male  Female

Respondent’s Age

If parent fill following questions:

Child Enrolled in School  Yes  No

Name of School (only if R is willing to share)

Type of School  Pu blic  Charter  Private  Homeschool

Student’s Grade

Age of Student

Assigned ID



## *This cover sheet should be kept separate from the rest of the data and be turned in with the rest of the paperwork. This cover sheet should not be copied, shared, or left in an unlocked office.10 

### Documento Adjunto 3
Protocolo en Español para las Entrevistas Cognitivas

MATERIALS NEEDED FOR INTERVIEW



### □ INTERVIEWER PROTOCOL



### □ RESPONDENT LETTERS AND QUESTIONNAIRE



### □ DO YOU HAVE THE CORRECT VERSI ON OF THE LETTERS, PROTOCOL, AND QUESTIONNAIRE?



### □ CONSENT FORM (TWO COPIES)



### □ INCENTIVE



### □ PAYMENT RECEIPT



### □ DIGITAL RECORDER AN D EXTRA BATTERIES



### □ PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be r ead verbatim. Text in italics is suggested content with which th e interviewer should be thoroughly familiar in advance. Text in brackets [ ] are instructions for the interviewer.

Hola, mi nombre es < su nombre > y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. Gracias por venir a ayudarnos el día de hoy.

[Platique de algún tema casual para establecer una buena relación c on el participante.]:

Usted está aquí hoy para ayudarnos con un nue vo estudio de investigación que estamos realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta acerca de [la educación de sus hijos, la búsque da y selección de cuidado infantil para los padres con hijos de cinco años de edad o menores, o acerca de sus certificados de estudios].

Este estudio es sobre [la educaci ón de su hijo(a), cómo buscaron y escogieron el para su hijo(a) para aquellos padres con niños de cinco años o menores, y sobre sus certificados de educación. Como parte de este estudio, le voy a pedir que lea al gunas cartas y que me diga lo que piensa que estas cartas le están pidiendo que haga. Adem ás, voy a pedirle que tome una breve encuesta como que si estuviera tomando una encuesta real. No existen respuestas correctas o incorrectas y está bien si usted no sabe algunas de las re spuestas. A medida que usted vaya leyendo las cartas y respondiendo a las preguntas, yo le vo y a hacer algunas preguntas acerca de sus respuestas. Esto nos ayudará a que las cartas y la encuesta sean mejores y más claras.

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas mientras estamos hablando [ INSERTE SI HAY UN OBSERVADOR : y mi colega también va a observar nuestra sesión de hoy].11 Además, esta entrevista será grabada en audi o para que los investigadores del estudio puedan revisar la grabación más tarde. Cualquier inform ación que se refiera a usted o su familia personalmente, tal como su nombre o la escuela de su hijo(a), no será incluida en nuestro informe y sus respuestas serán utilizadas para fines de investigación únicamente.

Recuerde que usted no está tomando un examen. Queremos que nos diga lo que usted piensa acerca de las cartas y la encuesta.
Si en algún momento usted desea dete nerse, solamente déjemelo saber.

[Si por algún motivo el participante ya no tiene interés en participar, ag radézcale por su tiempo y termine la entrevista.]

Antes de que continuemos, ¿tiene alg una pregunta sobre lo que acabo de decir? [Responda a cualquier pregunta que pueda hacer el participante.

PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT

SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.



### STEP 2: SCREENER LETTER

Ahora me gustaría que complete esta actividad como que si estuviera sucediendo en su casa. Voy a darle este sobre y quiero que usted haga lo que usted haría normalmente si usted recibiera y abriera estos materiales en su casa.
Voy a salir del cuarto por aproximadamente cinco minutos para dejar que usted lea los materiales y luego regresaré para que podamos hablar al respecto. [ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL CUARTO/AREA.]
[Cuando el entrevistado termine:]

Gracias por tomarse el tiempo para leer estos materiales.

Screener letter Ahora tengo unas cuantas preguntas par a usted, pero me gustaría que las responda sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias.

Me gustaría preguntar acerca de su reacción a la carta. Por favor, recuerde que no hay respuestas correctas o inco rrectas, sólo estamos tratando de saber esta carta puede ser interpretada.

PREGUNTE:

 En general, ¿Quién manda esta carta?

 ¿Qué se espera que haga usted después de leer la carta?12  ¿Para qué se va a usar la información recolectada por este estudio?

Gracias. Ahora quiero preguntarle acerca de los detalles de la carta. Voy a devolverle la carta y podemos hablar más acerca de ella.

[Anote cualquier lugar en el que el entrevis tado dude al leer, secciones que se salte el entrevistado, palabras que el en trevistado pronuncie incorrectamente o cualquier cosa acerca de la que haga comentarios el entrevistado.]

Quién

Por favor, lea en voz alta el primer párrafo de la carta.

[Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Después de leer este párrafo, puede decirme quien le está enviando esta carta?

 ¿Puede decirme quién cree usted que es el patrocinador de esta encuesta?

 ¿Acerca de que cree usted que se tr ata esta encuesta? En términos generales, usted no tiene que darme detalles específicos, solo lo que usted piensa acerca de lo que leyó en este párrafo. Qué

Continuemos. Por favor, ¿puede leer el segundo párrafo?

[Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 Hablemos acerca de la primera oración acerca de que su hogar fue elegido al azar. En sus propias pal abras, ¿qué quiere decir esto?

 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles? Cómo Continuemos. Por favor, ¿puede leer la siguiente oración y los puntos?

[Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Qué dice esto que se le envi ará en la siguiente semana?

 Usamos el término “muestra de aprecio.” ¿Qué significa este término para usted?
Privacidad

Continuemos y hablemos acerca de este texto en el cuadro. Por favor, ¿Puede leerlo en voz alta?13 [Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Hay algunas palabras aquí que no le sean familiares?

 ¿Qué cree que está tratando de decir este párrafo?



### STEP 3: THINK-ALOUD PRACTICE

Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así entender cómo mejorar estos materiales. Me intere sa más saber cómo llega a sus respuestas que en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer un ejercicio de práctica.

Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una oportunidad para que practique.

[Model the think aloud process for the responde nt, including reading the question out loud. Elaborate on counting all of the glasses of water as needed you can make up pieces of information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee I drank in the morning, I drink three cups of coffee, I am going to count those since that is the only liquids I have in the mo rning...” When you are modeling, wr ite down the number of glasses of water as you are thinking aloud.]

[Once finished the m odel practice think aloud]

Voy a darle un papel con una pregunta. Le voy a pe dir que lea en voz alta la pregunta y que piense en voz alta mientr as decide su respuesta.

[Give respondent handout with think aloud practice.]

¿Cuántos vasos de agua tomó usted ayer?

Por favor dígame lo que está pensan do mientras me da su respuesta.

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta? ¿Qué estaba visualizando mientras estaba decidiendo que responder? ¿Puede decirme más acerca de lo que usted está incluyendo en su conteo?

Cuando haya completado el ejercicio: Gracias. Ese es el tipo de detalle que busco cuando usted me hable acerca de sus respuestas a las preguntas de la encuesta.

Preguntas de seguimiento (probes) En sus propias palabras, ¿puede decirme lo que dice esa pregunta? ¿Qué significa la palabra [término] en esta pregunta para usted? ¿Qué tan fácil o difícil es recordar [tema]?14 ¿Qué tan fácil o difícil es escoger una respuesta? Dígame lo que está pensando. ¿Cómo llegó a esa respuesta? Usted dijo [respuesta]. ¿Puede decirme más acerca de esto? [En respuesta a las acciones o comentarios del entrevistado]: Dígame que pasó. Dígame que estaba haciendo. [Si nota alguna señal visual, como cejas fruncid as o una expresión de confusión]: “Dígame que está pensando.” [Si nota alguna señal visual que indique confus ión como voltear las pá ginas para regresar a secciones anteriores o volver a leer preguntas anteriores antes de responder a esta pregunta: Dígame qué es lo que acaba de pasar. Dí game acerca de lo que acaba de hacer.”



### STEP 4: SCREENER QUESTIONNAIRE

Ahora voy a darle una breve encuesta. Me gustaría que trate de completar las preguntas como si lo estuviera haciendo en casa, excepto que quiero que piense en voz alta mientras decide su respuesta, así como lo hicimos durante el ejercicio. Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma, como si yo no estuviera aquí.

Recuerde que estoy muy interesada/o en escuc harle hablar acerca de lo que está pensando
mientras decide las res puestas para las preguntas.

RECUERDE DECIRLE AL ENTREVISTA DURANTE LA ENTREVISTA QUE ESTA
HACIENDO UN BUEN TRABAJO AL HABLAR EN VOZ ALTA, Y QUE SUS
RESPUESTAS A LAS PREG UNTAS SON MU UTILES.

Screener
POR FAVOR LLENE TODA LA ENCUES TA Y PIENSE EN VOZ ALTA COMO HICIMOS EN EL EJERCICIO. Q3. ¿Cuál es el mes y año de nacimiento de la persona ?

[Si el participante tiene difi cultad con el mes yo año] ¿Cómo llegó a esta respuesta? Q5. ¿Asiste esta persona actualmente a. . .?

¿Cómo llegó a esta respuesta?

[Si no es evidente por lo que dijo al pe nsar en voz alta al responder preguntas previas] ¿Qué quiere decir para usted “Esc uela en el hogar (homeschool) en lugar de asistir a una escuela pública o pr ivada para algunas o todas las clases?15 

### STEP 5: TOPICAL LETTERS

[PARA PADRES DE FAMILIA O TUTORE S: LLENE EL NOMBRE DEL PRIMER NIÑO/A
LISTADO EN EL CUESTIONARIO.

PARA ADULTOS: LLENE EL NOMBRE DEL PRIMER ADULTO LISTADO EN LA ENCUESTA.]
Ahora tengo otra carta para usted. Por favor tome unos minutos para leerla. [ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL CUARTO/AREA.]
[Cuando el entrevistado termine:]

Gracias por tomarse el tiempo para leer estos materiales.

Topical Letters Al igual que antes, tengo unas cuantas preguntas para usted, pero me gustaría que las responda sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias.

Ahora me gustaría preguntarle acerca de su rea cción a la carta. Por favor, recuerde que no hay respuestas correctas o incorrectas. Sólo estamo s tratando de entender cómo usted interpreta la

carta.

PREGUNTAR:

 En general, ¿quién manda esta carta?

 ¿Qué se espera que haga usted después de leer la carta?

 ¿Para qué se va a usar la información recolectada por este estudio?

 ¿Acerca de qué se trata este estudio?

Gracias. Ahora quiero preguntarle acerca de los detalles de la carta. Voy a devolverle la carta y podemos hablar más acerca de ella.

[Anote cualquier lugar en el que el entrevis tado dude al leer, secciones que se salte el entrevistado, palabras que el en trevistado pronuncie incorrectamente o cualquier cosa acerca de la que haga comentarios el entrevistado.] Razón

Por favor, lea en voz alta el primer párrafo de la carta.

[Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Después de leer este párrafo, puede deci rme quien le está enviando esta carta?

 ¿Puede decirme quién cree usted que es el patrocinador de esta encuesta? Qué

Continuemos. Por favor, ¿puede leer el segundo y tercer párrafo?16 [Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Acerca de quién debe responder la encuesta?

 ¿Acerca de qué cree usted que se trata esta encuesta? En términos generales, usted no tiene que darme detalles específicos, sólo lo que usted piensa acerca de lo que leyó.

 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles? Cómo Continuemos. Por favor, ¿puede le er los siguientes dos párrafos?

[Espere a que el entrevistado termine de leer el párrafo.]

PREGUNTE:

 ¿Dónde puede encontrar más in formación acerca del estudio?



### STEP 6: SCREENER POSTCARD

El patrocinador de la encuesta también envía esta postal. Por favor, tómese un minuto para
leerla.

SCREENER POSTCARD

PREGUNTE:

 ¿Cuál es el propósito de la postal?

 ¿Quién manda la postal?

 ¿Hay algunas palabras aquí que no le sean familiares?

otros comentarios

Gracias.



### Continúe con la entrevista17 

### STEP 7: ATES ITEMS

La última actividad que le voy a pedir que hagamos hoy es responder a algunas preguntas acerca de su adiestramiento laboral. Estas son preguntas nueva s que se están considerando para la encuesta y sus comentarios acerca de ellas serán útiles. Al igual que hicimos antes, por favor, responda a las preguntas y " piense en voz alta " mientras contesta las preguntas.

ATES ITEMS [haga preguntas de seguimiento (probe s) si es necesario y aplican al encuestado]

Q5. Ahora veamos lo que hizo para preparase [Q5. ¿Hizo usted alguna de las siguientes cosas al prepararse pa ra obtener su certificación o licencia más importante? (Marque UNA casilla en cada una de las siguientes líneas.)]

 Por favor lea la letra D y dígame, en sus palabras, ¿que significa para usted esto? [opción d: Obtuve un título universitario].

 ¿Ahora hablemos de la opción E? [opción e: Participé en formación o capacitación laboral, pasa ntía o práctica profesional, o un programa de aprendiz.]

Q6. Veamos esta pregunta: ¿Qué tan útil ha sido su certificació n o licencia más importante para cada una de las siguientes situaciones?

 Quiero que lea la opción C otra vez y me diga, en sus propias palabras, ¿qué significa esto para usted? [opción c. Mantener el interés de los empleadores o clientes en usted].

 Ahora veamos la opción D ¿Qué si gnifica esto para usted? [opción d: Mejorar sus habilidades de trabajo].

Q7. Ahora veamos las preguntas sobre su experiencia de trabajo. Por favor lea la pregunta 7 otra vez. ¿Me puede decir que significa el termino “educación o formación/capacitación formal ” para usted?

 Hablemos de “pasantía o práctica pr ofesional”, ¿que significa para usted esto?

 Y finalmente programa “de aprend iz”, ¿que significa para usted esto?18 FINAL QUESTION

¿Tiene algo más que le gustaría hablar sobr e estos materiales que no hemos tenido la oportunidad de hablar?



### THANK YOU FOR PARTICIPATING

Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para mejorar las cartas y encuesta.

Interviewer Instructions: Collect materials and escort respo ndent out of the interviewing room and to the lobby. Please give th e respondent incentive envelope [$40 cash] and thank them for
their time and effort.Phase 2 Protocols

1000 Thomas Jefferson Street NW, Washington, DC 20007-3835 | 202.403.5000 | TTY 877.334.3499 | www.air.org Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total in terview time with the re spondent lasts 1 hour or less.

Before starting the interview, use this table to confirm the participant information we received from the recruitment firm. If the respondent indi cates that some of th e listed information is incorrect, please cross it out and enter the correct information in the relevant cell.*

Background Information Confirmed 

Date & Time

Location

Name of Interviewer

Parent/Respondent First Name

Respondent’s Gender  Male  Female

Respondent’s Age

If parent fill following questions:

Child Enrolled in School  Yes  No

Name of School (only if R is willing to share)

Type of School  Public  Charter  Private  Homeschool

Student’s Grade

Age of Student

Assigned ID


### *This cover sheet should be kept separate from the rest of the data and be turned in with the rest of the paperwork. This cover sheet should not be copied, shared, or left in an unlocked office.20 

### Esta página se ha dejado en blanco intencionalmente21 

### Documento Adjunto 

## 3 NHES 2016 PFI, ECPP, ATES

Protocolo en Español para las Entrevistas Cognitivas

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA

Instrucciones para el entrevistador: El siguie nte protocolo NO debe ser leído textualmente.
El texto en letra cursiva solo es contenido con el cual el entrevistador debe familiarizarse antes de la entrevista. El texto entre pa réntesis [] contiene instrucciones para el
entrevistador.

Hola, mi nombre es < su nombre > y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. Gracias por venir a ayudarnos el día de hoy.

[Platique de algún tema casual para establecer una buena relación c on el participante.]:

Usted está aquí hoy para ayudarnos con un nue vo estudio de investigación que estamos realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta acerca de [la educación de sus hijos, la búsque da y selección de cuidado infantil para los padres con hijos de cinco años de edad o menores, o acerca de sus certificados o licencias].

Como parte de este estudio, le voy a pedir que complete una encuesta como que si estuviera completando una encuesta real en su casa. No exi sten respuestas correctas o incorrectas y está bien si usted no sabe algunas de las respue stas. A medida que usted vaya respondiendo a las preguntas, yo le voy a hacer algunas preguntas a cerca de sus respuestas. Esto nos ayudará a que las preguntas en la encues ta sean mejores y más claras.

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas mientras estamos hablando [ INSERTE SI HAY UN OBSERVADOR : y mi colega también va a observar nuestra sesión de hoy].

Además, esta entrevista será grabada en audi o para que los investigadores del estudio puedan revisar la grabación más tarde. Cualquier inform ación que se refiera a usted o a su familia personalmente, tal como su nombre [o la escuel a de su hijo(a)], no será incluida en nuestro informe y sus respuestas serán utilizadas únicamente para fines de investigación.

Recuerde que esto no es un examen. Queremos que nos diga lo que usted piensa acerca de las preguntas de la encuesta.
Si en algún momento usted desea parar, solo déjemelo saber.

[Si por algún motivo el participante ya no tiene interés en participar, ag radézcale por su tiempo y termine la entrevista.]

Antes de que continuemos, ¿tiene alg una pregunta sobre lo que acabo de decir? [Responda a cualquier pregunta que pueda hacer el participante.]22 DELE AL ENCUESTADO UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO. PEDIRLE QUE LEA LA FORMA (O LEASELA, DEPENDINEDO DE LA PREFERENCIA DEL ENCUESTADO) RESPONDA CUALQUIER PREGUNTA Y PEDIRLE QUE FIRME LA FORMA.
DELE UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO AL ENCUENTADO

SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACION



### PASO 2: PRACTICA DE PENSAR VOZ ALTA

Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así entender cómo mejorar estos materiales. Me intere sa más saber cómo llega a sus respuestas que en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer un ejercicio de práctica.

Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una oportunidad para que practique.

[Model the think aloud process for the responde nt, including reading the question out loud. Elaborate on counting all of the glasses of water as needed you can make up pieces of information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee I drank in the morning, I drink three cups of coffee, I am going to count those since that is the only liquids I have in the mo rning...” When you are modeling, wr ite down the number of glasses of water as you are thinking aloud.]

[Once finished the m odel practice think aloud]

Voy a darle un papel con una pregunta. Le voy a pe dir que lea en voz alta la pregunta y que piense en voz alta mientr as decide su respuesta.

[Give respondent handout with think aloud practice.]

¿Cuántos vasos de agua tomó usted ayer?

Por favor dígame lo que está pensan do mientras me da su respuesta.

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta? ¿Qué estaba visualizando mientras estaba decidiendo qué responder? ¿Puede decirme más acerca de lo que usted está incluyendo en su cuenta?

Cuando haya completado el ejercicio: Gracias. Ése es el tipo de detalle que busco cuando usted me hable acerca de sus respuestas a las preguntas de la encuesta.

ENTREGUE AL ENCUESTADO UNA COPI A DEL CUESTIONARIO Y UNA PLUMA O UN LAPIZ23 PASO 3: RESPUESTAS A LA ENCUESTA

Ahora voy a darle la encuesta. Me gustaría que pi ense en voz alta mientr as decide su respuesta. Por favor lea en voz alta cualquier cosa que us ted hubiera leído para sí misma(o), como si yo no estuviera aquí.

Recuerde que estoy muy interesada(o) en escu charle hablar acerca de lo que está pensando mientras decide sus respuestas a las preguntas.

RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL
CUANDO PIENSA EN VOZ ALTA Y QUE SU S RESPUESTAS A LAS PREGUNTAS SON DE
GRAN UTILIDAD.

Preguntas Útiles En sus propias palabras, ¿qué dice esta pregunta?

¿Qué significa para usted la palabra [término] en esta pregunta?

¿Qué tan fácil o difícil es recordar [tema]?

¿Qué tan fácil o difícil fu e escoger una respuesta?

Dígame lo que está pensando.

¿Cómo llegó a esa respuesta?

Usted dijo [respuesta]. ¿Puede decirme un poco más acerca de cómo llego a esa respuesta?

[En respuesta a las acciones o comentarios del entrevistado]: Por favor, dígame que sucedió. Dígame que estaba usted haciendo.

[Si nota alguna señal visual, como cejas frun cidas o una expresión de confusión]: Dígame lo que está pensando24 PFI: Encuesta sobre la experiencia de los estudiantes y las familias con sus escuelas (NHES41-BC)

Educación del niño PÍDALE AL ENCUESTADO QUE POR FAVOR LLENE EL CUESTIONARIO. RECUERDELE QUE TIENE QUE PENSAR EN VOZ ALTA , SI NO LO ESTÁ HACIENDO

HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO Instrucciones: No es necesario preguntar sobre las in strucciones, nada más anote si el encuestado tiene problemas con ellas.

Q1-Q2 No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q3 ¿Es esta la escuela asignada del distrito?

[ Preguntar solamente si el encu estado respondió esta pregunta] ¿Cómo llegó a esta respuesta?

¿Qué significa para usted “escu ela asignada del distrito”?

Q4-Q10 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q11 ¿En qué mes de este año escolar comenzó es te niño(a) a asistir a su escuela actual?

¿Cómo decidió la respuesta a esta pregunta? ¿Le resulta fácil o di fícil contestar esta pregunta? ¿Por qué? Q14. ¿Está este niño inscrito en algún cu rso avanzado llamado Advanced Placement AP en la escuela?

Si respondió si, ¿me puede explicar sobre estos cu rsos en los que su niño( a) está inscrito? ¿En qué pensaba cuando respondió a esta pregunta?

¿En qué grado académico se encuentran los estudiantes que generalmente están inscritos en un curso avanzado en el distrito escolar de su hijo(a)?

Q15-21 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q22. ¿Está este niño(a) tomando algún curso relaciona do con la escuela por el internet en lugar25 de hacerlo de manera presencial con un maestro(a)?

Si responde si - por favor dígame un poco más acerca de esto. ¿Me puede decir porque usted eligió esta respuesta?

¿Me puede describir a que se refiere?

Q23. ¿Cuál de los siguientes lu gares proporciona esas clases?

¿Cómo decidió su respuesta?

[Si seleccionó una respuesta o la opción de “A lgún otro lugar”] Por favor dígame un poco más acerca de esto. ¿Cómo tiene acceso su hijo(a) a estas clases? ¿En qué lugar se encuentra cuando está tomando dicha clase? ¿Quién provee esa clase?

Q24. No es necesario indagar sobre esta pregunta. De je que el encuestado continúe su proceso de pensar en voz alta y solamente haga pregunt as si el encuestado dice algo que indique un problema potencial, o si hay conf usión o alguna inconsistencia en la interpretación comparado con otros encuestados

Q25. Algunos padres deciden educar a sus hijo s en el hogar en vez de mandarlos a una escuela pública o privada. ¿Recibe este niño sus cl ases en el hogar en lugar de en la escuela para algunas clases o asignaturas?

¿Por favor, me puede decir con su s propias palabras qué es lo que se le está preguntando aquí?

¿Cómo determinó su respuesta?

[Si respondió que “si”] ¿Me puede decir porque respondió si?

[Si respondió que “no”] ¿Me puede decir porque respondió no?

[Si se aplica] Q26. ¿Cuál de las siguientes fras es describe mejor el arreglo de educación en el hogar para este niño?

[Si el encuestado elige la respuesta a o b]

¿Porque eligió esta respuesta? ¿Me puede explicar cómo funci ona este arreglo?

[Si el encuestado elige una de las siguientes re spuestas: “Este niño no reci be la educación en el hogar” o “Este niño asiste a la escuela pública o privada para todas las clases o asignaturas]

¿Porque eligió esta respuesta?26 [Si no es evidente indague el signi ficado de la escuela del hogar]

[Si se aplica] Q27. Pensando en todos los años en los que este niño(a) ha recibido educación en el hogar, ¿en cuál de las áreas a continuaci ón ha recibido este niño(a) enseñanza durante su instrucción en el hogar?

[Se han añadido 16 opciones de respuesta a esta pr egunta. Por favor anote si el entrevistado tiene problemas con el número y el tipo de asignaturas enumeradas]

[Si no es evidente indague el signi ficado de la escuela del hogar]

¿Cómo determinó cómo responder en cada una de las preguntas?

¿Cómo enseña las asignatu ras durante el periodo de instrucción en el hogar?

¿Cree que falta algo?

[Si se aplica] Q30. Hay diferentes razones por la s que los padres eligen la educación en el hogar para sus hijos. ¿Eligió su familia la educación en el hogar para su hijo porque:

¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]?

Platíqueme un poco más sobre lo que estaba pensando cuando respondió las preguntas.

[Si no es evidente indague el significado de la escuela del hogar]

[Si se aplica] Q31. Entre las razones por las que su familia decidió educar a este niño(a) en el hogar, ¿cuál diría usted que es la más impo rtante para usted? Escriba la letra de la pregunta 30 para la razón más importante por la que decidió educar a su niño(a) en el hogar.

¿Qué tuvo en cuenta al elegir esta razón?

Platíqueme un poco más sobre lo que estaba pensando cuando respondió a esta pregunta.

Familias y Escuela Q32-36. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.27 Tareas Q37-43. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Al final de la sección pregunte:

Quiero hacerle preguntas sobre algunas palabras que usamos en esta sección de tareas.

¿Me puede decir cuando en esta parte di ce tarea o tareas a que se refiere?

Actividades en familia Q44-46. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Salud del niño(a) Q47 No es necesario indagar sobre esta pregunta. De je que el encuestado continúe su proceso de pensar en voz alta y solamente haga pregunt as si el encuestado dice algo que indique un problema potencial, o si hay conf usión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q48. ¿Le ha dicho un profesional de la salud o de la educación que este niño(a) tiene alguno de los siguientes problemas de salud?

[Note como responde el encuestado cada una de las preguntas de la a-m.]

¿Hay alguna palabra o frase que no le es familiar?

¿Hay algún problema de salud que su niño(a) tiene que no está listado?

Q49- 50. No es necesario indagar sobre estas p reguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente ha ga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q51. ¿Ofrecen estos servicios algunas de las siguientes fuentes?

[note como responde cada una de las pr eguntas de la a-d el encuestado]28 ¿Cómo determinó su respuesta?

¿Hay alguna palabra o frase que no le es familiar?

Q52. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de servicios para la familia (IFSP por sus siglas en inglés), un progr ama individualizado de educación (IEP por sus siglas en inglés) o plan de servicios?

[en esta pregunta estamos tratando de determinar que vocabulario utilizan los padres de familia para referirse a el IFSP y al IEP]

¿Cómo determinó su respuesta?

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus siglas en inglés)?

¿Qué significa para usted un “programa individualizado de educación” o IEP?

Q53-56. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Historia del niño(a) Q57-62. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q63. ¿El niño(a) vive en este domicilio y en ot ro domicilio (por ejemplo, como resultado del acuerdo de custodia)?

¿Cómo determinó su respuesta?

En sus propias palabras, ¿qué dice esta pregunta?

[Si la respuesta fue NO] ¿Qué signifi ca “arreglos de custodia compartida”?

Q64. Si contestó afirmativamente, este niño(a ) ¿pasa la mayor parte del tiempo en este domicilio, otros domicilios, o el mismo tiempo en ambos?

¿Cómo determinó su respuesta?29 Q66. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación bilingüe o a un programa especial de inglés?

¿Cómo determinó su respuesta?

¿Hay alguna palabra o frase que no le es familiar?

Los miembros de su familia

Q68. Además de los padres o tutores lega les que ya se incluyeron, ¿cuántas de las siguientes personas viven en el hogar con este niño(a)?

¿Cómo determinó su respuesta?

¿A quién está incluyendo en su cuenta?

La familia del niño(a)

Q71-115. No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Su hogar

Q116-Q124. No es necesario indaga r sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.30 PFI: Encuesta sobre la experiencia de los estudian tes y las familias con sus escuelas: Escuela del Hogar

ESTAS PREGUNTAS SON PA RA LOS PAPAS QUE SUS HIJOS ESTAN EN LA
ESCUELA DEL HOGAR

Dígale al encuestado lo siguiente:

Estas preguntas son sobre su hijo que está en la escuela del hogar. ¿Qué edad tiene el
niño?

Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta

PIDALE AL ENCUESTADO QUE PORFAV OR RELLENE EL CUESTIONARIO. RECUERDELE QUE TIENE QUE PENSAR EN VO Z ALTA, SI NO LO ESTA HACIENDO

HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO Instrucciones: No es necesario preguntar sobre las in strucciones, nada más anote si el encuestado tiene problemas con ellas.

Escuela del Hogar Q1-Q5 No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q6. ¿Cuántas horas a la semana va normalmente este niño(a) a una escuela para recibir instrucción? No incluya el tiempo que dedica a actividades extracurriculares.

¿Cómo determinó su respuesta?

[Verifique si la persona respondió a esta pregunta a pesar de haber indicado que el niño(a) está en escuela domiciliaria (o escolarización en el hogar). Q10. ¿Cuál de las siguientes afirmaciones descri be mejor el estilo de enseñanza que se usa para la educación en el hogar de este niño(a)?

¿Cómo determinó su respuesta?

Q17

. Hay diferentes razones por las que los padres eligen la educación en el hogar para sus hijos. ¿Eligió su familia la educación en el hogar para su hijo porque:31 ¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]?

Dígame un poco más sobre lo que estaba pensando cuando respondió estas preguntas.

Q18. Entre las razones por las que su familia decidió educar a este niño(a) en el hogar, ¿cuál diría usted que es la má s importante para usted?

¿Cómo determinó qué opción escoger [a-j]?

Platíqueme un poco más sobre lo que estaba pensando cuando escogió esta opción.

[Si no es evidente, haga preguntas sobre el significado de la escuela del hogar]

Q20. Pensando en todos los años que este niño(a) ha recibido la educación en el hogar, ¿en cuál de las áreas a continuación ha recibido este niño(a) enseñanza durante su instrucción en el hogar?

¿Cómo determinó qué responder s en cada una de las preguntas?

¿Cómo cubre usted las asignaturas durant e el periodo de inst rucción en el hogar?

¿Hace falta algo?

La salud del niño(a) Q27-30. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o s i hay confusión o algu na inconsistencia en la interpretación comparado con otros encuestados. Q31. ¿Ofrecen estos servicios algunas de las siguientes fuentes?

¿Cómo determinó su respuesta?

[Solo si el encuestado tiene a su niño(a) en un a escuela privada]¿Qué significa para usted que estos servicios se ofrezcan en la escuela privada del niño(a)?

Q33. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de servicios para la familia (IFSP por sus siglas en inglés), un progr ama individualizado de educación (IEP en inglés) o plan de servicios?

¿Cómo determinó su respuesta?

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus siglas en inglés)?32 ¿Qué significa para usted un “programa individualizado de educación” o IEP?

Historia del niño(a) Q38-42. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q43. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado del acuerdo de custodia)?

¿Cómo determinó su respuesta?

En sus propias palabras, ¿qué dice esta pregunta?

[Si la respuesta fue NO] ¿Qué significa para usted arreglos de custodia compartida?

Q44. Si contestó afirmativamente, este niño(a ) pasa la mayor parte del tiempo en este domicilio, otros domicilios, o el mismo tiempo en ambos?

¿Cómo determinó su respuesta? Q45. ¿Qué idioma habla este niño (a) principalmente en el hogar?

¿Cómo decidió su respuesta?

Los miembros de su familia

Q47. Además de los padres o tutores lega les que ya se incluyeron, ¿cuántas de las siguientes personas viven en el hogar con este niño(a)? ¿Cómo determinó su respuesta?

¿A quién está incluyendo en su cuenta de las personas que viven en su hogar?

La familia del niño(a) Q62. ¿Cuál es el grado o nivel escola r más alto que esta persona completó?

[Solamente si el respondiente escoge “diplo ma de estudios secundarios o equivalente”, “diploma vocacional obtenido después de la escuela secundaria”, o “algo de33 colegio/universidad, pero sin título”:]

¿Cómo determinó su respuesta?

¿Qué quiere decir para usted “diploma vocacional obtenido después de la escuela secundaria”?

Q63. Actualmente, ¿asiste esta persona a una escuela, universidad o centro de educación para adultos o recibe educación vocacional o capacitación laboral?

En sus propias palabras, ¿qué dice esta pregunta?

ECPP: El Futuro de nuestros niños: Una Encuesta sobre el cuidado y la educación de niños pequeños. (NHES-21BC(SP))

Estas preguntas son para los padres que tienen algún niño(a) menor de cinco años., que todavía no está inscrito en preescolar (kindergarten).

Dígale al encuestado lo siguiente:

Estas preguntas son sobre su hijo que es menor de cinco años de edad, que todavía no está inscrito en preescolar (kindergar ten). ¿Qué edad tiene este niño(a)? ¿Qué tipo de arreglos u opciones ha establecido para el cuidado de su hijo(s)? [Anote la respuesta]

Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta

Cuidado Infantil y Programas para la Niñez Q1-11 No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q12. ¿Le ayuda alguna de las siguientes pers onas, programas u organizaciones a pagarle a este pariente por el cuidado del niño(a)?

¿Me podría decir que información cree que esta pregunta le está pidiendo?

Si la persona contesto la pregunta Q12b. b. Asistencia temporal a familias necesitadas (TANF)

Si el encuestado respondió sí . ¿Me puede describir este programa?

Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta?34 Q13-Q16 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q17 Actualmente cuida a este niño(a), con regularidad, una persona que no es su pariente ya sea en su casa o en otra casa?

Si el encuestado respondió sí. ¿M e puede describir el lugar?

Si el encuestado respondió no. ¿Me puede d ecir porque respondió no a esta pregunta?

¿Qué significa para usted una persona que no es su pariente?

Q18-Q37 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q38. (Si se aplica) ¿Es este programa de Head Start o de Early Head Start?

¿Me podría decir un poco más sobre qué le están preguntando aquí?

Si el encuestado respondió sí . ¿Me puede describir este programa?

Si el encuestado respondió no. ¿Me puede decir por qué respondió que no a esta pregunta?

Q39-48 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

[pregunte solo si no ha hecho esta s preguntas en la pregunta Q12]

Q49 ¿Le ayuda alguna de las siguientes personas, programas u organizaciones a pagarle a este pariente por el cuidado del niño(a)?

¿Me podría decir que información cree que esta pregunta le está pidiendo?

Si la persona contesto la pregunta Q49b. b. Asistencia temporal a familias necesitadas (TANF)

Si el encuestado respondió sí . ¿Me puede describir este programa?35 Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta?

Q49-53 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Buscar y Seleccionar Cuidado Infantil para su Niño(a) Q54. (Solo si no se le ha preguntado ya sobre el programa de Head St art) ¿Ha asistido este niño(a) alguna vez a un programa de Head Start o de Early Head Start?

¿Me podría decir que información cree que esta pregunta le está pidiendo?

Si el encuestado respondió sí . ¿Me puede describir este programa?

Si el encuestado respondió no . ¿Me puede decir por qué respondió no a esta pregunta?

Q55-Q56 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q57. ¿Cuánta dificultad tuvo para encontrar el tipo de cuidado infantil o programa de educación temprana que usted quería para este niño(a)?

¿Cómo determinó su respuesta?

¿Me podría decir un poco má s sobre su respuesta?

¿Qué significa para usted el término “educación temprana”?

Q58. ¿Qué tan importante fue cada una de estas razones cuando usted seleccionó el cuidado infantil o el programa en el que este niño(a) pasa la mayor parte del tiempo?

¿Hay alguna otra razón que es importante para usted y que no ha encontrado en la lista de opciones?

Q59h. ¿El Sistema de Calificación y Mejoram iento de la Calidad (Quality Rating and Improvement System, QRIS) de su ciudad o estado?

¿Me podría decir un poco más de su respuesta?36 Q59k. ¿Está acreditado el proveedor del servicio de cuidado infantil?

¿Me podría decir un poco más de su respuesta?

Q60 No es necesario indagar sobre esta pregunta. De je que el encuestado continúe su proceso de pensar en voz alta y solamente haga pregunt as si el encuestado dice algo que indique un problema potencial, o si hay conf usión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Actividades en familia Q61-67. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Cosas Que su Niño(a) Podría estar Aprendiendo Q68-75. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Salud del niño(a) Q76-80. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q81. ¿Ofrecen estos servicios algunas de las siguientes fuentes?

¿Cómo determinó su respuesta?

¿Qué significa para usted que estos servicios se ofrezcan en la escuela privada del niño(a)?

Q82. ¿Se ofrecen algunos de estos servicios por medio de un Plan individualizado de servicios para la familia (IFSP, por sus sigl as en inglés) o un Progr ama individualizado de educación (IEP, por sus siglas en inglés)?

¿Cómo determinó su respuesta?

Si el encuestado responde si:

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus siglas en inglés)?37 ¿Qué significa para usted un “pla n individualizado de educación”?

Q83-86 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Historia del niño(a) Q87 – 91. No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q92. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado del acuerdo de custodia)? No in cluya propiedades vacacionales.

¿Cómo determinó su respuesta?

En sus propias palabras, ¿qué dice esta pregunta?

[Si la respuesta fue NO] ¿Qué significa “acuerdo de custodia compartida”?

Q94. ¿Qué idioma habla este niño (a) principalmente en el hogar?

¿Cómo determinó su respuesta?

¿Hay alguna palabra o frase que no le es familiar?

Q95. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación bilingüe o a un programa especial de inglés?

¿Qué cree le está preguntando la pregunta?

¿Hay alguna frase o palabra que no le sea familiar en esta pregunta?

Los miembros de su familia

Q97. Además de los padres o tutores lega les que ya se incluyeron, ¿cuántas de las siguientes personas viven en el hogar con este niño(a)?38 ¿Cómo determinó su respuesta?

¿A quién está incluyendo en su cuenta de las personas que viven en el hogar?

La familia del niño(a)

Q100-138. No es necesario indagar sobre estas pregunt as. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Su hogar Q139-Q145. No es necesario indagar sobre estas pre guntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encues tado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Encuesta ATES

PASO 3: LLENADO de la encuesta

Ahora voy a darle una encuesta. Me gustaría que piense en voz alta mientras decide su respuesta. Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma, como si yo no estuviera aquí.

Recuerde que estoy muy interesada/o en escuc harle hablar acerca de lo que está pensando
mientras decide las res puestas para las preguntas.

RECUERDA DE DECIRLE AL ENCUESTADO QUE ESTA HACIENDO UN MUY BUEN
TRABAJO PENSANDO EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON

DE GRAN UTILIDAD

Escolaridad HAGA PREGUNTAS SEGÚN SEA NECESARIO

Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el encuestado tiene problemas con ellas. Q1-Q2 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q3. ¿Está actualmente inscrito en una universid ad, escuela técnica o comercial, u otra escuela?39 En sus propias palabras, ¿qué dice esta pregunta?

[Si respondió Si] ¿Puede decirme algo más sobre las clases que toma? ¿Cuál es el campo de estudio?

¿Dónde toma las clases?

¿Qué significa para usted medio tiempo y tiempo completo?

Q5. Desde que salió de la escuela secundaria, ¿ha tomado usted clases de lectura para mejorar cómo lee? No incluya clases a nivel universitario.

¿A qué se refiere esta pregunta?

Certificación y Licencias Q6. ¿Tiene usted una certificación profesional o una licenci a estatal o industrial que este actualmente activa? No incluya licencias comerciales, tales como de lic or o de máquinas expendedoras (vending machines).

[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.]

¿Me puede decir que cree que esta pregunta le esta preguntando?

Q7. (si APLICA) ¿Si contestó que sí, cuantas certifica ciones y licencias tiene usted que estén actualmente activas?

¿Cómo determinó su respuesta?

¿Qué significa para usted el término “actualmente activas” en esta pregunta?

En sus propias palabras, ¿qué dice esta pregunta?

Q8. ¿Cuál es el nombre de su certif icación o licencia más importante?

¿Cómo determinó su respuesta?

¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la más importante?

Q9. ¿Para qué tipo de trabajo es su certificación o licencia más importante?

¿Qué tan fácil o difícil fue para usted recordar el tipo de trabajo para el qu e la certificación o licencia le sirve?

Q10. ¿Es su certificación o licencia más importante requeri da por un gobierno federa l, estatal o local para realizar este trabajo?

En sus propias palabras, ¿qué dice esta pregunta?

¿A qué se refiere esta pregunta cuando dice “requerida”?

Q11. ¿Le podrían quitar o suspender su certificación o licencia más importante por alguna razón?

En sus propias palabras, ¿qué dice esta pregunta?

¿A qué se refiere esta pregunta cuando dice quitar o suspender?40 Q12. ¿En qué año obtuvo su certif icación o licencia más importante?

¿Cómo determinó su respuesta?

¿Qué tan fácil o difícil fue para usted recordar en qué año obtuvo la certificación o licencia más importante?

Q13a. ¿Se preparó usted para obtener su certificación o licencia más importante realizando alguna de las siguientes actividades? Tomando clases en una universidad , escuel a técnica o escuela vocacional (escuela para aprender oficios)

¿Cómo determinó que contestar en esta pregunta?

Q14. ¿Es su certificación o licencia más importante para su empleo actual?

¿Cómo determinó qué contestar en esta pregunta?

En sus propias palabras, ¿qué dice esta pregunta?

[SI contesto afirmativamente] ¿Cómo puede usted determinar si la cer tificación o licencia es requerida para su trabajo actual?

Q15. ¿Qué tan útil ha sido su certificación o licenci a más importante para cada una de las siguientes situaciones? (a-d)

¿Qué tan fácil o difícil fue para usted contestar solamente sobre la certificación o licencia más importante para cada una de las opciones?

Q16. ¿Tiene otra certificación o licencia actualmente activa?

¿Cómo determinó qué certifi cación o licencia poner aquí?

Q17-Q22 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados. Q22a (Si aplica). ¿Se preparó uste d para obtener su segunda certific ación o licencia más importante realizando alguna de las siguientes actividades?

22a ¿Tomando clases en una univ ersidad, escuela técnica o escuela vocacional (escuela para aprender oficios)?

¿Cómo determinó qué contestar en esta pregunta?

Q23 (Si se aplica). ¿Es su segunda certificación o licencia más importante para su empleo actual

¿Cómo determinó qué contestar en esta pregunta?

¿Con sus propias palabras, describa qué es lo que le están preguntando?

Después de la pregunta 23

¿Qué tan fácil o difícil fue para usted contestar solamente pensando en la certificación o licencia que usted decidió fue la segunda más importante?

¿Mientras contestaba, pensó en sus otras certificaciones o licencias?41 Q24. ¿Qué tan útil ha sido su segunda certificación o li cencia más importante para cada uno de las siguientes situaciones? (a-d)

¿Qué tan fácil o difícil fue para usted contestar solamente sobre su segunda certificación o licencia más importante para cada una de las preguntas?

¿Me podría decir cuál piensa usted que es la diferencia en tre las respuestas de estas categorías? ¿Fue fácil o difícil seleccionar la respuesta?

Q26. (Si aplica). ¿Cuál es el nombre de su tercera certificación o li cencia más importante?

¿Cómo determinó qué contestar en esta pregunta?

¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la tercera más importante?

Q27. (Si se aplica). ¿Para qué tipo de trabajo su tercera certificación o licencia más importante?

¿Cómo determinó qué contestar en esta pregunta?

Q29. ¿Es su tercera certificación o li cencia más importante requerida por un gobierno federal, estatal o local?

¿Con sus propias palabras, describa qué es lo que le están preguntando?

Q25-29 (Si aplica)

¿Qué tan fácil o difícil fue para usted contestar solamente pensando en su tercera certificación o licencia más importante?

¿Mientras contestaba, pensó en sus otras certificaciones o licencias?

Certificados Q30. Las personas a veces reciben certificados por completar un programa educativ o o de capacitación. Éstos
son diferentes de una certificación o licencia. No incluya a quí certificaciones/ licen cias. ¿Alguna vez ha obtenido usted algunos de los siguientes tipos de certificados?

[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.]

¿Me puede decir que cree que esta pregunta le está preguntando?

Q30d. Un certificado—no una titulaci ón—por completar un programa en una universidad técnica o de la comunidad, o alguna otra escuela después de la escu ela secundaria. No incluya certificados de maestro o títulos universitarios?

¿Porque eligió esta respuesta (si/no)?

Q31. (Si contestó que sí.) Nos referiremos a los certificados en la pregunta 30d como certificados ‘postsecundaria’. ¿Cuál fue el campo de estudio de su úl timo certificado de post-secundaria? (Marque UNA casilla solamente.)

¿Pudo encontrar alguna opción que encaje con su situación o cree que falta alguna opción?

¿Me podría hablar un poco más sobre su respuesta?42 SI contesto “otro” ¿podría decirme más sobre su respuesta?

Q34. ¿Cuál de las siguientes opciones fue un requisito para matricularse en el último programa de certificado post-secundaria?

¿Con sus propias palabras, describa qué es lo que le están preguntando?

¿Me podría hablar un poco más sobre su respuesta?

Q35. Para obtener su último certificado de post-secundaria, ¿tuvo que completar...? [a-b]

¿Cómo determinó las respuestas a las preguntas (a-b)?

Q37. ¿Está su empleo actual relacionado con su último certificado de post-secundaria?

¿Cómo determinó qué contestar a esta pregunta?

Q38 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente indague si el encuestado dice algo que seña le un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Programas de Experiencia laboral Q39 ¿Ha completado alguna vez una pasantía o práctica profesional, programa de cooperación educativa, practicum, asistente jurídico (clerk ship), residencia médica, experienci a clínica, programa de aprendiz o similar?

[Si aplica] Me podría decir sobre su programa:

¿Para qué trabajo fue?

¿Puede describir un día típico en el programa?

¿Qué tipo de organizaciones estaban involucradas y cuáles eran sus roles?

Q40. ¿Si su respuesta es sí, para qué tipo de trab ajo fue su último programa de experiencia laboral?

¿Me podría decir un poco más acerca de cómo seleccionó su respuesta?

¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difí cil fue seleccionar su respuesta?

Q41. ¿Cuánto tiempo duró su programa de experiencia laboral más reciente?

¿Cómo llegó a su respuesta?

Q42. (Si aplica) ¿Qué sueldo ganaba como part e de su último programa de experiencia laboral?

[Si no contesto rápidamente] ¿Pudo encontrar una respuesta que estuviera acorde a su experiencia? De no serlo así, ¿cuál hubiese sido una mejor opción para usted?

[Si ninguna opción fue seleccionada] ¿Me podría hablar un poco más sobre el salario que recibió?

Q43 a-c. Como parte de su último programa de experi encia laboral, ¿realizó usted alguna de las siguientes actividades?43 ¿Cómo llegó a su respuesta?

(Si contestó Sí a alguna de las preguntas) ¿Me podría hablar un poco más sobre la relación entre su programa de experiencia laboral las clases, su entrenamiento, o el estudio que llevó a cabo?

Q44 a-d. ¿Describe alguna de las siguientes actividades su programa de experiencia laboral más reciente?

¿Cómo llegó a su respuesta?

Q44a (Si contestó Sí) ¿Me podría hablar sobre las evaluaciones?

Q44c (Preguntar solo si contestó Sí) ¿Qué significa pa ra usted tener “clasificación de obrero especializado”?

Q45. ¿Cuál de las siguientes opciones describe mej or su último programa de experiencia laboral?

¿Me podría decir cómo un poco más de cómo seleccionó su respuesta?

¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difícil fue seleccionar su respuesta?

Q46. (Si aplica) ¿Considera que su último programa de experiencia laboral le ayudó (o le ayudará) a lograr una certificación o licencia profesional?

¿Me podría decir con sus propias palabras, que es lo que le están preguntando?

Q47a-c. (Si aplica) ¿Qué tan útil fue su programa de experiencia laboral más recien te para cada uno de las siguientes opciones?

¿Cómo seleccionó su respuesta pa ra cada una de las preguntas?

¿Me podría decir cuál cree usted que es la diferencia entre las categorías de las diferentes respuestas?

¿Fue fácil o difícil seleccionar su respuesta?

Q48. (Si aplica) En su trabajo actual, ¿qué tan a me nudo usa las habilidades o el conocimiento que aprendió durante su último programa de experiencia laboral?

¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando?

¿Me podría hablar más sobre lo que usted estaba pensando cuando respondió esta pregunta?

Q49. (Si aplica) ¿Está su trabajo actual relacionado con su último programa de experiencia laboral?

¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando?

Empleo Q50-Q67 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz alta y solamente indague si el participante dice algo que señale un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q68. (Si aplica) ¿Tenía usted una licencia requerida por una agencia del gobierno federal, estatal o local para realizar este trabajo?

¿Con sus propias palabras, describa qué es lo que le están preguntando?44 [Verificar si es congruente con las respuestas anteriores]

Q69-70 No es necesario indagar sobre estas preguntas. Deje qu e el encuestado continúe su proceso de pensar en voz alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Historial Q71-84 No es necesario indagar sobre estas preguntas. Deje qu e el encuestado continúe su proceso de pensar en voz alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

PREGUNTA FINAL

¿Hay algo más sobre lo que le gustaría hablar acerca de estos materiales? ¿Hay algo acerca de lo que no ha tenido oportunidad de hablar?



### PASO 4. GRACIAS POR PARTICIPAR

Hemos terminado. Gracias por ayudarnos. La informac ión que nos ha dado será muy útil para mejorar las cartas y la encuesta.

Instrucciones para el entrevistador: Recoja los materiales y acompañe a los encuestados al
vestíbulo. Por favor dele a los encuestados el sobre con el incentivo, y agradézcales por su tiempo y su ayuda.