Appendix 4 

Cognitive Interview Results 

This appendix contains the following reports and memos that provide results from the 

NHES:2016 cognitive interviews: 

Cognitive Interview Findings for the Parent and Family Involvement and Early 

Childhood Program Participation Surveys 2016: Final Report 

Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2016: 

Draft Report 

Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews: Memo 

Preliminary Findings from the 2015 Spanish Cognitive Interviews: Memo 
Cognitive Interview Findings 

for the Parent and Family 

Involvement and Early 

Childhood Program 

Participation Surveys 2016: 

Final Report 

Prepared for the National Center for 

Education Statistics (NCES) by 

Susan Cole, Raquel Gonzalez, Kaitlin Fronberg, and 

Suzette Chavez 

 

May 2015 
Contents

1. Introduction .......................................................................................................... 4 

1.1 Cognitive Interview Methodology .......................................................................... 5 

1.2 Sample ....................................................................................................................... 5 

1.3 Data Collection and Analytic Approach ................................................................ 8 

Data Collection.............................................................................................................................. 8

Data Analysis Methods................................................................................................................. 9

2. Findings ................................................................................................................. 9

2.1 Screener Items .......................................................................................................... 9

2.2 PFI Items ................................................................................................................. 10 

2.2.1 Child’s Schooling ......................................................................................................... 10 

School Assignment ......................................................................................................................... 10 

Advanced Placement ....................................................................................................................... 11 

Internet Courses .............................................................................................................................. 12 

Homeschooling ............................................................................................................................... 14 

Other Items in the Child’s Schooling Section ................................................................................. 17 

2.2.2 Child’s Health ............................................................................................................... 18 

Services ........................................................................................................................................... 18 

Other Items in the Child’s Health Section ...................................................................................... 20 

2.2.3 Child’s Background ...................................................................................................... 20 

Joint Custody .................................................................................................................................. 20 

Language at Home........................................................................................................................... 21

Other Items in the Child’s Background Section .............................................................................. 22

2.2.4 Your Household ........................................................................................................... 22 

Other Items in the Your Household Section ................................................................................... 24 

2.2.5 Child’s Family .............................................................................................................. 25 
2.3 ECPP Items ............................................................................................................. 25

2.3.1 Finding and Choosing Care for Your Child ................................................................. 25 

Difficulty Finding Child Care ......................................................................................................... 25 

Importance of Reasons for Choosing Child Care ........................................................................... 26 

Ratings from a Website ................................................................................................................... 26

Early Childhood Program Curriculum ............................................................................................ 28 

Other ECPP Items ........................................................................................................................... 29 

Appendix A............................................................................................................. 30 

 
1. Introduction

In 2016, the National Household Education Surveys Program (NHES) will field the Parent and 

Family Involvement in Education Survey (PFI) and the Early Childhood Program Participation 

Survey (ECPP). This report summarizes the findings from cognitive interviews conducted to test 

and revise a subset of items from the PFI survey based on potential issues with a few items that 

were observed in the 2012 collection and the 2014 Feasibility Study. This report also 

summarizes findings from a series of new and revised ECPP items that cover topics suggested by 

the Office of Nonpublic Education (ONPE) and the Policy and Program Studies Service (PPSS) 

in consultation with the Office of Early Learning. The report also includes findings from 

cognitive testing of the household screener. 

In order to reduce the respondent’s cognitive and time burden, the interviews included only the 

sections of the NHES questionnaires that contained items requiring testing. The objective of the 

cognitive interviews in 2015 was to identify and correct problems of ambiguity or 

misunderstanding in item wording that were evident from NHES:2012 data. The research was 

iterative, in that item wording and format design changed during the testing period in response to 

early findings. The cognitive interviews should result in a set of questionnaires that are easier to 

understand and therefore less burdensome for respondents, while also yielding more accurate 

information. 

A total of 67 items underwent cognitive testing, including 60 items from the PFI survey and 7 

items from the ECPP survey. The items from the PFI survey that were tested included items 

about the following topics: 

 School assignment 

 Advanced Placement courses 

 Internet courses 

 Homeschooling 

 Joint child-custody arrangements 

 Language spoken by the child at home 

 Your household 

 Child’s family 

The items from the ECPP questionnaire that were tested in cognitive interviews included items 

about the following topics: 

 Difficulty finding child care 

 Importance of reasons for choosing child care 

 Early childhood program curriculum 
The selected survey items were tested with members of the appropriate target respondent group 

to determine their understandability, the need for revision, and for retention in or exclusion from 

the surveys. The target respondent groups are described in section 1.2. 

Interviews were conducted between January 20 and March 2, 2015. 

1.1 Cognitive Interview Methodology

Each cognitive interview lasted approximately one hour and was conducted using primarily a 

“think-aloud” approach with concurrent probing as needed, based on a structured protocol. The 

methodology was developed by a senior researcher at AIR in consultation with NCES and 

drawing on best practices and methods from cognitive science (see Appendix A for the protocol). 

These interviews were designed to identify problems of ambiguity or misunderstanding in item 

wording. The interviews were conducted by interviewers who received training in the cognitive 

interview technique and in the interview protocol by a senior researcher at AIR. Researchers 

from AIR and NCES listened to a subset of interviews and provided guidance and feedback 

throughout the study. 

During each interview, the study participant was welcomed by the interviewer and led to an 

interview room. The participant was then told that the purpose of the interview was to help 

answer questions about how people respond to items on schooling. All participants were assured 

that their participation was voluntary and that their answers would be used only for research 

purposes and not disclosed, or used, in identifiable form for any other purpose except as required 

by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the 

think-aloud process (see below) and modeled a think-aloud practice with a sample question. 

Participants were then given a paper copy of the household screener and questionnaire to fill out. 

The interviews were conducted using “concurrent probing,” a technique in which interviewers 

asked respondents to “think aloud” as they answer the survey items. The cognitive interviews 

also included a verbal probing component, conducted after completion of the think-aloud portion 

for a given questionnaire. The verbal probes included a combination of pre-planned, itemspecific questions identified before the session as important and ad hoc questions that the 

interviewer identified as important from observations during the interview, such as clarifications 

or expansions on points raised by the participant. 

1.2 Sample

Cognitive interview respondents were recruited through local recruitment firms in San Mateo, 

CA, the Washington, DC metro area, and Waltham, MA and through additional interviewer 

outreach to homeschooling parents. Respondents were given a $40 American Express gift card 

for their participation. The recruitment criteria included: 

 15 parents or guardians of a child enrolled in public school and 7 in private school; 
 12 parents or guardians of a child enrolled in grades K-5 and 7 enrolled in grades 6-12; 

 12 parents or guardians of a child who is in a joint custody arrangement; 

 12 parents or guardians for whom English is their second language but who can read 

English fluently; 

 6 parents or guardians of a child who is homeschooled; 

 9 parents or guardians who also have a child or children ages 0 to 5 and not yet in 

kindergarten and also have a care arrangement for the child; 

 6 parents or guardians of a child who receives instruction from school over the internet; 

and 

 10 parents or guardians with a high school completion or less education. 

Note: The sum of the categories listed is greater than the total because several respondents are expected to 

fall into multiple categories. 

A total of 51cognitive interviews were conducted across the three locations. We sought to recruit 

a diverse set of respondents; however the respondent characteristics reported during recruitment 

did not always match the respondent’s reported characteristics during the cognitive interviews. 

Table 1 provides more detailed information on the recruitment and interview reported 

characteristics of the respondents. 

 
Table 1. Number of participants in PFI/ECPP cognitive interviews, by selected recruitment 

and interview characteristics 

		

Reported Characteristics

from Recruitment

Reported Characteristics

from Cognitive Interviews

								Total Respondents1 51 51

Child’s	Grade 	 	

K	‐	5	 14 14

6	‐	12	 32 36

Type	of	School2	

Public	 36 38

Private	 12 11

Homeschool3,4	 5	 6	

English	as	a	Second	Language1

Yes	 8 7

No	 43 43

Highest	Degree	

High	School	Diploma	or	Less	 5 6

More	than	just	High	School	 46 44

Joint	Custody	Arrangement5	

Yes	 11 10

No	 40 43

Internet	Instruction	for	Credit6

Yes	 10 8

No	 41 42

Second	Child	Under	5	

Yes	 9 9

No	 42 42

Location	 	 	

DC	 16 16

CA	 25 25

MA	 10 10 1One	respondent	only	filled	out	the	ECPP	items	on	the	questionnaire	as	he/she	did	not	have	a	second	child	K‐

12;	n=50	for	the	PFI	items		

2	Four	respondents	also	indicated	that	their	child	attended	a	public	or	private	school.		 3	Two	respondents	were	former	homeschoolers	and	filled	out	the	screener	and	questionnaire	as	if	they	were	

still	homeschooling	their	child.	

4	One	respondent	believed	that	“tutoring”	was	a	form	of	homeschooling.	

5	Some	respondents	were	confused	about	the	meaning	of	joint	custody.	

6	For	Internet	courses,	there	was	initially	a	discrepancy	between	the	language	on	the	recruitment	screener	

and	the	survey.	
Table 2 provides information on the total number of respondents who were scheduled for an 

interview, the number of interviews that were completed, and the cancellations and no shows by 

demographic characteristics. 

Table 2. Number of interviews scheduled, completed, cancellations and no shows, by 

demographics 

								Demographics

Interviews 

Scheduled 

Interviews 

Completed 

Cancellations No Shows 

Child’s	Grade 

K	‐	5	 29 14 8 7

6	‐	12	 59 36 10 13

Type	of	School

Public	 69 38 12 19

Private	 16 11 3 2

Homeschool 11 5 6 0

English	as	a	Second	Language	 10 7 1 2

High	School	Diploma	or	

Less	

6 1 7

Joint	Custody	Arrangement 21 10 3 8

Internet	Instruction	for	Credit 16 8 4 4

Second	Child	Under	5	 16 9 5 2

Location	

DC	 49 16 18 15

CA	 34 25 3 6

MA	 12 10 1 1

TOTAL		 95 51 22 22

Note:	The	sum	of	the	categories	listed	is	greater	than	the	total	because	several	respondents	fall	into	multiple	

categories.	

1.3 Data Collection and Analytic Approach

Data Collection

Respondents were asked to complete a paper version of the screener and questionnaire; while 

doing this, they were asked to “think aloud” when responding to the items. After the respondent 

completed an item of interest, the interviewer asked the respondent to stop and asked probing 

questions about the item as needed. If the respondent had a problem or issue with items other 

than the items of interest, the interviewer asked probing questions about those items. While the 

respondent filled out the screener and questionnaire, the interviewer took handwritten notes in an 

interview protocol that contains the items and probes for the cognitive interview. The screener 

and questionnaire were kept by the interviewer as a record of the respondent’s responses to the 

items. Once the interview was completed, the interviewer entered notes in an electronic note 

template. This template is a standardized format for qualitative data entries, which allowed the 

analysis team to process and analyze the data in NVivo, a qualitative data processing software 
system. The interviewer then saved the interview notes and an audio recording of the interview 

in a secure server for analysis. 

Data Analysis Methods

Once the interview notes were entered, a senior researcher checked the data to ensure that the 

most accurate information was included about the errors, issues, and concerns that respondents 

had about the items in the questionnaire and screener. After the interview notes were checked for 

quality, all items of interest as well as any items that respondents had difficulty answering were 

coded into the following categories: 

a. No problems for questions of interest 

b. Respondent was hesitant in answering the question 

c. Respondent self-corrected his or her answer 

d. Respondent made an item or survey suggestion 

e. Skip problem (questionnaire navigation) 

f. How the respondent interpreted an item term 

g. Special situation (when a respondent responds to an item differently than expected based 

on a special situation ) 

h. Response problem or issue 

i. Respondent had difficulty understanding the survey formatting 

ii. Respondent was unable to find a response that matched his or her experience 

iii. Respondent had difficulty understanding the question 

iv. Respondent did not include a response 

v. Respondent misinterpreted what the question was asking 

vi. Respondent had trouble understanding what a specific term meant 

vii. Other problems or issues 

These categories allowed the analysis team to determine the types of problems that each item of 

interest has. Once we were able to determine the item-by-item issues, we were able to do a 

higher level analysis of topic-level problems. 

2. Findings

2.1 Screener Items

At the beginning of the cognitive interviews, all respondents completed the National Household 

Education Survey household screener. Below we report on two items of interest (month and year 

of birth and school enrollment) and discuss a common skip error. 

For the item on the month and year of birth, the majority (n=45) of respondents did not have any 

difficulty reporting the month and year of birth for household members. Only a few respondents 

struggled to remember the year of birth for members of the household. In most of these cases, the 

respondent had some difficulty recalling the year of birth for a child, but subtracted his/her 
child’s age from the current year to identify the year of birth. Only two respondents were unable 

to recall the year of birth for adult members of the household (i.e., a mother-in-law’s and 

respondent’s parents’ year of birth). 

For the item that asks if household members are currently homeschooled or enrolled in school 

(including college, university or vocational school), most participants were able to answer 

without any problems. In a couple of instances respondents were unsure what option to select for 

their young children (i.e., a one year old and 10 month old) who are in child care. One 

respondent selected “public or private school, or preschool” and the other respondent left the 

item blank. 

The most common problem in the household screener was a skip error. If the respondent reports 

that a household member is “not in school”, the survey instructs the respondent to go to the next 

household member. If a household member is in school, respondents should proceed to the next 

item that asks for the person’s current grade or equivalent. Half of the respondents failed to skip 

out of the current grade or equivalent item for household members not currently enrolled in 

school. These respondents selected either “none of these” for current grade or equivalent or the 

household member’s highest level of schooling. 

None of the items on the screener were changed during cognitive interview testing. 

2.2 PFI Items

2.2.1 Child’s Schooling

School Assignment

The item on a child’s school assignment was to be completed only by respondents whose 

children attend public schools. Table 3 lists the three versions of this item that were tested in 

cognitive interviews. 

Table 3. Wording changes for item on school assignment 

Original Wording Wording Change Final Wording

Is	it	his/her	regularly‐

assigned	school?	(No/Yes)	

Is	it	his/her	district‐assigned	

school?	(No/Yes)

Is	it	his/her	district‐assigned	

school?	

A district‐assigned school is the

school that your local public

school district told you that this

child can attend, based on the

location of your residence.

(No/Yes)	

The original item that asks about “his/her regularly-assigned school” was tested by 12 

respondents. After respondents answered the item, the interviewer presented the respondent with 

an alternative item (“his/her district assigned school”). During testing, the two items were 
switched and 13 respondents responded to the item on “his/her district-assigned” school and 

were presented with the item on “his/her regularly-assigned school” as the alternative. For the 

remainder of the cognitive interviews (n=26), the item was changed to include a definition of 

“district-assigned” and the item with the original wording was no longer tested. 

Although many respondents understood the initial wording (“regularly-assigned school”), some 

respondents misinterpreted “regularly” to mean daily – a school the student goes to everyday 

during the week. Most respondents did not have a problem with the term “district-assigned,” and 

seemed to prefer this over “regularly-assigned” because they were more familiar with the term. 

They described it as the closest school to the home or the school assigned to a student based on 

address/geographic zone. 

For at least one respondent, interpretation of “district-assigned” and “regularly-assigned” was 

informed by local policy. One respondent, based in California, responded “Yes,” to “districtassigned school,” and reported that California education code permits parents to indicate a 

preference for which school his/her child will attend within the district regardless of where the 

family resides within the district. The respondent applied to a preferred high school and was 

allowed to enroll the child in that high school, and therefore, reported that the school was 

district-assigned. However, when asked if the respondent’s child attended the “regularly assigned 

school,” the respondent reported ”No” – this school was not the child’s regularly assigned 

school, but rather the preferred school. Adding a definition seemed to be helpful for respondents. 

Five respondents completed this item even though they should have skipped out of it since their 

child attends a private school. 

Advanced Placement

One item on Advanced Placement (AP) enrollment was tested in cognitive interviews. This item 

was tested by all 50 respondents who completed the PFI items. The response option “Does not 

apply” was removed during testing (See Table 4 below). 

 
Table 4. Wording changes for Advanced Placement (AP) item 

2012 Wording Original Wording on

2016 Survey

Wording Change Final Wording

Is	he/she	currently	

enrolled	in	advanced	

placement	classes?	

(No/Yes/Does	not	

apply)	

Is	he/she	currently	

enrolled	in	any	high	

school	Advanced	

Placement	(AP)	classes?	

Advanced Placement is a

program that offers

college‐level courses to

high school students,

with the option for

students to take AP

exams to earn college

credit.	(No/Yes/Does	

not	apply)

Is	he/she	currently	

enrolled	in	any	high	

school	Advanced	

Placement	(AP)	

classes?	

Advanced Placement is

a program that offers

college‐level courses to

high school students,

with the option for

students to take AP

exams to earn college

credit.	(No/Yes)	

Is	he/she	currently	

enrolled	in	any	high	

school	Advanced	

Placement	(AP)	

classes?	

Advanced Placement is

a program that offers

college‐level courses to

high school students,

with the option for

students to take AP

exams to earn college

credit.	(No/Yes)

Of the 12 respondents who responded to the original wording of the item, seven did not have any 

difficulty answering the item. Some respondents whose children attend high school were 

confused as to whether they should respond “No” or “Does not apply.” Examples include one 

respondent who responded “No” stated that her child took an AP course the year before, but that 

the child’s school did not offer AP courses in 11th grade. Another respondent who marked “Does 

not apply,” misinterpreted the item and believed that elementary school students could take AP 

classes. Another respondent could not tell the difference between AP and IB courses, and, lastly, 

one respondent did not understand the meaning of Advanced Placement. 

“Does not apply,” was removed as a response option as it was determined that sufficient data 

would be captured for respondents who responded “No” for children not in high school. 

Additionally, respondents with high school students at schools that do not offer AP might select 

“Does not apply.” 

Of the total 38 respondents who responded to the final wording of this item, the majority did not 

have any difficulty answering it, although a few were confused as to how to respond. One 

respondent who incorrectly marked “Yes” for this item ended up revising the answer to “No” 

when she reread the item and realized there was a difference between advanced classes and 

Advanced Placement. Another respondent who responded “No” said that the term was “foreign” 

to him or her, although the respondent attributed that to the child being in 9th grade. 

Internet Courses

Three items of interest on courses taken over the internet were initially tested by respondents. 

The first item in the table below was tested by all 50 respondents, and the other two items were 

tested by 11 respondents who marked “Yes” in the first item. The following wording changes 

were made to two of the items and one item was deleted during testing. 
Table 5. Wording changes/deletions for items on courses taken over the internet 

Wording in 2012 Original Wording on

2016 Survey

Wording Change Final Wording

Some	students	take	

school‐related	courses	

over	the	Internet.	Is	

this	child	receiving	

any	instruction	this	

way?	

Some	students	take	

school‐related	courses	

for	credit	over	the	

Internet.	Is	this	child	

receiving	any	

instruction	this	way?	

(No/Yes)	

Is	this	child	taking	any	

school‐related	courses	

online	instead	of	in‐

person	with	the	

teacher?		

Do not include courses

that use the Internet

only for selected

assignments? (No/Yes)

Is	this	child	taking	any	

school‐related	courses	

online	instead	of	in‐

person	with	the	

teacher?		

Do not include courses

that use the Internet

only for selected

assignments? (No/Yes)	

Is	that	instruction	

provided	by	any	of	the	

following	places?	

Mark all that apply.

	Your	local	public	

school		

	A	charter	school		

	Another	public	

school		

	A	private	school		

	A	college,	

community	college,	or	

university		

	Someplace	else—

Specify: 

Is	that	instruction	

provided	by	any	of	the	

following	places?		

Mark all that apply.

	Your	local	public	

school		

	Your	state		

	A	charter	school		

	Another	public	

school		

	A	private	school		

	A	college,	

community	college,	or	

university		

	A	website		

	Someplace	else—

Specify:		

Is	that	instruction	

provided	by	any	of	the	

following	places?		

Mark all that apply.

	Your	local	public	

school		

	Your	state		

	A	charter	school		

	Another	public	

school		

	A	private	school		

	A	college,	community	

college,	or	university		

	Someplace	else—

Specify: 

Is	that	instruction	

provided	by	any	of	the	

following	places?		

Mark all that apply.

	Your	local	public	

school		

	Your	state		

	A	charter	school		

	Another	public	

school		

	A	private	school		

	A	college,	community	

college,	or	university		

	Someplace	else—

Specify: 

	 What	is	the	main	

reason	this	child	took	

school‐related	courses	

over	the	internet?	

Deleted Deleted	

Although most respondents understood the initial wording “school-related courses for credit over 

the Internet” in the first item, seven respondents misinterpreted the item, including four who 

incorrectly responded “Yes” on the questionnaire. These responses included materials, tests, and 

homework at the child’s school that are accessible on the Internet, extra credit for courses done 

at home, extra coursework done at school for additional grades, or the completion of a math 

course on-line over the summer. One respondent who responded “No,” interpreted “Internet 
courses” to mean reading, spelling, or alphabet games, and said: “it could be anything, it could 

be listening to some songs over the Internet.” 

The wording was changed on this item because of the misinterpretations and also because the 

intent of the item. After the wording was changed and the definition was added to the initial 

item, many respondents correctly responded, although three respondents misinterpreted the item, 

even when they correctly responded “No.” One thought the item was asking if the student is 

advanced enough to be taking online college courses, and another referred to courses outside of 

school, for example, tutoring for a calculus class. A third respondent showed some confusion in 

his or her reasoning for not including an informal online typing class as an online course: “there 

is no human being on the other end, the entire class is completely electronic, and he [the student] 

does not receive a certificate or credit for it.” 

Respondents suggested the following changes to this item to make it more comprehensible: 1) 

Restructuring the sentence to read, “all-school related courses in person with the teacher” or, “are 

any [courses] online,” and 2) replacing the wording “instead of” to “in addition to,” which might 

lessen the chance of glossing over the question.”   

On the second item of interest that asks who provided the internet courses, of the 11 respondents 

who responded to the item, four respondents, hesitated to respond or misinterpreted the item 

because they did not understand what the item was asking, or because they were not sure 

whether to respond about the institution that pays for the course or the website used to facilitate 

the instruction. Based on these cognitive interviews, it was decided to delete “the website” as a 

response option for the remainder of the respondents. 

After “the website” was deleted on this item, a couple of respondents were not sure how to 

respond because courses were through a free on-line system or an academy affiliated with 

homeschooling, and they were not sure of the funding source. 

On the third item that asks what the main reason is that the child took school-related courses over 

the internet, three respondents misinterpreted the item based on their misinterpretation of the first 

item, for example, two respondents selected “improve basic reading, writing, math, or science 

skills,” although the courses were either not for credit or for extra credit, and one respondent 

selected “Other,” which corresponds to the completion of a math course on-line over the 

summer. This item was eliminated because of conflict with the first item on internet courses. 

Homeschooling

Initially, six items of interest on homeschooling were tested in cognitive interviews. On the 2012 

item, there was an overreporting of homeschooling and it was suspected that some respondents 

did not understand that the item was asking about formal homeschooling programs, and instead 

reported on casual instruction at home. In the 2014 feasibility study, the number of hours the 

child attended public or private school was added to the item, which allowed NCES to evaluate 

the number of respondents who indicated that the child was homeschooled but who likely 
misinterpreted the item. A majority of 2014 respondents who indicated that their child was 

homeschooled also reported that the child was attending a public or private school full-time. 

Based on these issues, it was decided to test a series of homeschool items on the 2016 PFIEnrolled survey. 

The items tested in this section on homeschooling were: 

 Some parents decide to educate their child at home rather than send them to a public or 

private school. Is this child being schooled at home instead of at school for at least some 

classes or subjects? (No, Yes) 

 Which of the following statements best describes your homeschooling arrangement for 

this child? 

o This child is homeschooled for all classes or subject areas 

o This child is homescooled for some classes and subject areas and also attends a 

public or private school 

o This child is not homeschooled. This child attends a public or private school for 

all classes or subject areas 

 How many hours each week does this child usually go to a public or private school for 

instruction? Do not include time spent in extracurricular activities. [__] Hours 

 There are many different reasons that parents choose to homeschool their children. Did 

your family choose to homeschool this child because: (Mark X ONE box for each item 

below.) (Responses a-j) 

 Of the reasons your family chose to homeschool this child, which one would you say is 

the most important to you? Write the letter from question [above] for the most important 

reason you chose to homeschool your child. [__] 

 Thinking about all years this child has been homeschooled, which of the following 

subject areas has this child been taught during his or her home instruction Mark X all 

that apply .(list of subject areas)

 [New item added] Which of the following subject areas are being taught to this child 

now? Mark X all that apply. (list of subject areas)

On the first item that asks if the child is being schooled at home, one respondent initially 

responded “No,” and then changed the response to “Yes” after the probing questions, and said 

that he or she is providing tutoring for the child and considers it to be homeschooling. This 

respondent incorrectly completed all of the remaining items on homeschooling. On the item on 

description of the homeschooling arrangement this respondent chose the response option: “this 

child is homeschooled for some classes and subject areas and also attends a public or private 

school.” On the item on the number of hours each week the child usually goes to school for 

instruction, this respondent misread “hours each week” and wrote 9 hours and explained he or 

she drops off the child at school at 7 a.m. and picks the child up at 3:30 p.m. On the item about 

the subject areas the child has been taught during home instruction, this respondent selected 
“Calculus” and “Chemistry or physics” because those are the subject areas for which the child 

was receiving tutoring. 

The majority of the other 49 respondents did not have any problems or issues with the first item 

that asks if the child is being schooled at home, although a couple of respondents had difficulty 

with key vocabulary in the item. One of these respondents defined homeschooling as educating 

the child after school, although correctly responded “No,” and another respondent defined 

“homeschool” as “a supplement to what [the child] is learning in school,” although also correctly 

responded “No.” One homeschooling respondent made a skip error and did not continue to the 

next item on homeschooling. At the end of the cognitive testing a decision was made to add a 

downward arrow, similar to other arrows on the questionnaire, to direct the respondent to 

continue to the rest of the items on homeschooling if they responded “Yes” to the first 

homeschooling item. 

The remaining homeschooling items were completed by five respondents: three current and two 

former homeschooling parents, and a respondent who tutors the child and considers it to be 

homeschooling. A couple of respondents had problems with the item on the description of the 

homeschooling arrangement. The item on the number of hours each week the child usually goes 

to school for instruction presented problems for two respondents. Two respondents were 

confused on how to respond to this item, one because the online homeschooling program is 

considered a private school and the respondent thought the item on the questionnaire was 

referring to a “brick and mortar” school, and another respondent was not sure how to respond 

about a homeschooling academy that was not traditionally public or private. 

There were no problems on the item on the different reasons that parents choose to homeschool 

their children. There were also no problems on the item on the most important reason the 

respondent chose to homeschool the child. 

There were no problems on the item about the subject areas the child has been taught during 

home instruction. Four respondents recommended adding “physical education” and “health” to 

the list of subject areas. Based on these recommendations, at the end of cognitive testing these 

were added to the list of subject areas, although these items were not tested during this phase of 

cognitive interview testing (see Table 6 below). 

An additional item on subject areas being taught “now” was added during the cognitive interview 

testing because there is not enough space to obtain every subject the child was taught in every 

year that they were homeschooled. Overall, this item did not present any problems for 

respondents. One respondent who had only started to homeschool the child this year inquired 

whether the item is asking about today, this year, this semester, or this month, although 

responded with the same answer as the previous item. 

 
Table 6. Wording changes and new item on subject areas taught during home instruction

Original Wording Wording Change Final Wording

Thinking	about	all	years	the	

child	has	been	homeschooled,	

which	of	the	following	subject	

areas	has	this	child	been	taught	

during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas]	

Thinking	about	all	years	the	

child	has	been	homeschooled,	

which	of	the	following	subject	

areas	has	this	child	been	taught	

during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas]	

Physical	education	or	gym	

Health

Thinking	about	all	years	the	

child	has	been	homeschooled,	

which	of	the	following	subject	

areas	has	this	child	been	taught	

during	her	home	instruction?

Mark X all that apply.

[List	of	subject	areas]	

Physical	education	or	gym	

Health	

New Item Wording Change Final Wording

Which	of	the	following	subject	

areas	are	being	taught	to	this	

child	now?	

Mark	X	all	that	apply.	

[List	of	subject	areas]	

Which	of	the	following	subject	

areas	are	being	taught	to	this	

child	now?	

Mark	X	all	that	apply.	

[List	of	subject	areas]	

Physical	education	or	gym	

Health	

Which	of	the	following	subject	

areas	are	being	taught	to	this	

child	now?	

Mark	X	all	that	apply.	

[List	of	subject	areas]	

Physical	education	or	gym	

Health	

Other Items in the Child’s Schooling Section

In this section, there were 19-21 other items tested by respondents. The majority of these items 

were tested by all 50 respondents except for a few skip patterns. Of these items, respondents had 

varying degrees of difficulty responding to the items as well as the instructions. Only 13 of the 

50 respondents had no difficulty when completing these items. 

Five respondents were confused with the formatting on the item asking the child’s current grade 

or year of school, which has the response boxes for grade 1 through 12 under the kindergarten 

response options. All five said that they expected the boxes designating their response to be 

placed directly under the item text and were initially confused when it was the last response 

option. 

On the item asking respondents what type of school the child attends, three respondents who all 

responded “private school,” continued to fill out the next two items relating to public school, 

when they should have skipped to a later item. 
Three respondents had difficulty responding to the item asking if the child attends a charter 

school, and noted that they did not understand the meaning of “charter school.” Additionally 

some respondents were confused about the difference between a charter school and a magnet 

school when reading the definition in the item asking whether or not a respondent’s school 

district let them choose what public school they want the child to attend. On this same item 

several respondents were uncertain as to how to answer and noted that they have the option to 

transfer schools in their district, however the district did not guarantee the transfer. 

Three respondents did not follow the skip pattern on the item asking whether or not they had 

considered other schools for the child. One of these respondents ignored the skip pattern and 

completed the following item, one incorrectly followed the skip pattern, and the third respondent 

turned the page and skipped three items instead of the item they were directed to. 

On the item asking respondents about how many times teachers or school staff had contacted 

their household, a couple of respondents referred to contact they had made with teachers at an 

open house, and some could not understand why “very good behavior” and “very good school 

work,” were included in the questionnaire when teachers only contact them about problems with 

the child’s behavior or school work.

Several respondents had difficulty responding to the item asking about their child’s overall 

grades across subjects. Several respondents selected responses other than “This child’s school 

does not give these grades,” and reported that their child’s school had a 1-4 point system and did 

not use traditional grading systems. Similarly, several respondents had difficulty determining an 

average grade. One respondent wrote on the questionnaire, “all over the map” and did not select 

one of the response options. 

On the item asking about the child’s suspension and expulsion history, a few respondents did not 

know how to interpret the meaning of “in-school suspension.” Two respondents inquired about 

the meaning and suggested that it means that the student was to attend school but not attend 

classes at all during school hours, and another respondent reported that it means that the child 

could not attend certain activities such as Chorus. 

Several respondents who had difficulty determining an average grade also had difficulty 

responding to the item asking how the respondent would describe their child’s work at school 

because their child excelled in some subjects and did poorly in others, and it was difficult for 

them to mark only one response option, e.g., “above average,” across all subjects. . 

2.2.2 Child’s Health

Services

Two items of interest on child’s health were tested in cognitive interviews. Only respondents 

whose children received services for health conditions responded to these items. The following 
wording change was made to one of the sub-items on “sources for where services were 

provided” to be more parallel with other sub-items (see Table 7). 

Table 7. Wording changes for item on sources for services provided 

Original Wording Wording Change Final Wording

Are	these	services	provided	by	

any	of	the	following	sources?		

Mark X ONE box for each item

below.

	 No/Yes	

a.		 Your	local	school	

district...............		

b.		 A	state	or	local	health	or	

social	service	

agency...........................		

c.		 A	doctor,	clinic,	or	other	

health	care	

provider.................................

d.		 A	teacher	or	health	provider

from	a	private	

school...........................		

Are	these	services	provided	by	

any	of	the	following	sources?		

Mark X ONE box for each item

below.

	 No/Yes

a.	 Your	local	school	

district...............		

b.	 A	state	or	local	health	or	

social	service	

agency...........................		

c.	 A	doctor,	clinic,	or	other	

health	care	

provider.................................

d.	 This	child’s	private	

school...........................		

Are	these	services	provided	by	

any	of	the	following	sources?		

Mark X ONE box for each item

below.

No/Yes	

a.	 Your	local	school	

district...............		

b.	 A	state	or	local	health	or	

social	service	

agency...........................		

c.	 A	doctor,	clinic,	or	other	

health	care	

provider.................................

d.	 This	child’s	private	

school...........................		

Of the 12 respondents who responded to the item on the sources where services were provided, 

two respondents were unsure whether to mark “Yes” for the sub-item “your local school 

district,” reasoning that either the school or the district provided the service, or they were unsure 

altogether whether the local school district provided the service their child received. Two 

respondents indicated that the “Yes” and “No” response options tend to be reversed on other 

surveys (with “Yes” before “No”). One of these respondent accidentally marked “No” on all of 

the sub-items when he or she intended to mark “Yes.” One respondent incorrectly marked 

“doctor or clinic” for the person who was providing services for a condition that was not listed 

on the previous item. 

On the second item about whether the services are provided through an IEP or services plan, 

respondents provided varying responses for the meaning of IEP and services plan, or did not 

understand the meaning of the terms. Two respondents defined “IEP” or “services plan” as 

instances when children receive extra time to complete coursework or their assignments. One 

respondent indicated uncertainly about the meaning of an IEP, but said that if he or she was 

answering the survey at home, would look up the meaning online to confirm. One respondent 

responded “No” to this item, but his or her explanation for the “504 plan” his or her child 

received largely aligned with the definition of a services plan. 
Other Items in the Child’s Health Section

Two other items on child’s health presented some problems, although they were not prevalent. 

Of the 50 respondents who responded to the item on “conditions told by health and education 

professionals,” two respondents noted issues with the item’s wording, citing that professionals do 

not “tell” you that your child has one of the conditions; instead the “health or education 

professional” recommends that you seek testing for your child to see if they have one of the 

conditions listed on the item. Two respondents had difficulty responding to this item because 

they were uncertain about the timeframe. One respondent initially marked sub-item “a specific 

learning disability” then went down the list and identified sub-item “attention deficit disorder, 

ADD or ADHD” as more fitting for the child’s condition and removed selection of “a specific 

learning disability.” It was decided to move the sub-item “a specific learning disability” from the 

top of the response options to just after ADD/ADHD. 

There were also comprehension issues on this item. Two respondents confused sub-item 

“orthopedic” with orthodontic matters. One respondent did not know the definition of 

“orthopedic” altogether. Additionally, four respondents could not define or did not recognize 

“Pervasive Development Disorder (PDD). 

For the item “if the child is receiving services for his/her condition,” one respondent indicated 

confusion about how to respond since the child no longer received services for the condition 

marked on the previous item. Two respondents had problems with the skip patterns. One 

respondent marked “Yes” and missed the skip pattern, citing the arrow as less conspicuous than 

the box indicating a skip to the next set of questions. One respondent marked “No,” but read the 

following items and marked “Yes” for the “IEP and services plan” item. 

2.2.3 Child’s Background

Joint Custody

Two items of interest on joint custody were tested in cognitive interviews. No changes were 

made to these items. The first item that asks if the child “lives at this address and another address 

(for example, because of a joint custody arrangement)” was tested by all 50 respondents. The 

second item of interest on where the child spends his/her time (“at this address,” “at another 

address,” or “equal time at both addresses”) was asked only of 11 respondents who marked 

“Yes” to the first item. 

Although many respondents did not have any comprehension problems with the first item on 

joint custody, some respondents had to re-read the item a few times before responding, and some 

changed their response either from “No” to “Yes” or vice versa. Four respondents were not clear 

on the meaning of “this” address, although assumed it meant their home address – this response 

confusion could be a function of the cognitive interview setting outside of the respondent’s 

home. Some were not clear on the meaning of “joint custody,” or whether they were asking 

about one or two addresses. For example, one respondent who responded “Yes” said that she and 

her husband are married and are the two people who are the parents of that child that have joint-
custody or joint decision making. Another respondent who responded “Yes” said that her child 

goes to the father’s house on the weekends, although when asked what the item was asking, the 

respondent said, “it doesn’t necessarily mean divorce; children could go over to grandparents. It 

is asking if children spend time in another home.” 

Four respondents described special situations where they had physical custody of a child, but 

they also had legal custody arrangements. For example, where the child is supposed to spend 

time with one parent but prefers not to go to that parent’s house, so instead spends all of the time 

with the respondent, and the respondents responded “No” to the item. In a couple of instances, 

respondents stated that their response would be different for the first part of the question, “Does 

this child live at this address and another address” than the second part, “for example, because of 

a joint custody arrangement.” 

Two suggestions were made by respondents for the first item on joint custody: 1) to move the 

item to the screener; and 2) to simplify the item to read: “Does your child have more than one 

residency?” This respondent said that since a respondent might answer the item quickly, he or 

she might “skip the residency component altogether.”” 

Of the nine respondents who responded to the second item of interest on joint custody that asks 

where the child spends his/her time, a few responded incorrectly because they did not understand 

the first item described above, and one responded “No” to the first item and should have skipped 

the second item. 

Language at Home 

One item on what language the child speaks most at home was tested in cognitive interviews. 

The majority of the 50 respondents who responded to this item found it relatively easy to answer. 

Four respondents were confused by the formatting of the first two response options: “Child is not 

able to speak,” and “English” and thought it was one option “Child is not able to speak English” 

because there is a bracket around these two options to indicate that the respondent should skip to 

another question, and the response boxes were faint. Respondents were looking for an English 

only response and most located it during the think aloud, although one respondent marked 

“English and another language equally” because he or she could not find the correct response. 

The response boxes on this item were filled with white to make it clearer that these were two 

separate response options. Four respondents were still confused with the new formatting and 

thought the first two response options were grouped together. One respondent suggested adding 

an arrow after each response option rather than the bracket around the two options. 
Other Items in the Child’s Background Section 

A few other items in the Child’s Background section presented problems for respondents. A 

couple of respondents suggested changing “sex” to “gender” for the item on the child’s sex. A 

few respondents were confused and frustrated why the item on Hispanic, Latino, or Spanish 

origin is separate from the item on the child’s race, and suggested that there should be a Hispanic 

option in the race item. Some responded “White” or left the item blank when responding “Yes” 

to Hispanic, Latino, or Spanish origin. 

Some respondents had problems with skip patterns in two other items. Five respondents who 

responded that the child was born in the U.S. did not skip out of the next item on how old the 

child was when he/she first moved to the U.S. In two instances, respondents incorrectly 

responded to the item on being enrolled in ESL, bilingual education, or an English immersion 

program when they should have skipped out after the previous item on the language the child 

speaks most at home. 

2.2.4 Your Household

Two items of interest on how many people live in the respondent’s household were tested in 

cognitive interviews. The following wording changes and order were made to the two items (see 

Table 8 below) during cognitive testing. 

 
Table 8. Wording changes for the items on how many people live in household 

2012 Wording Original Wording (and

Order) on 2016 Survey

Wording Change [and

Order Reversed)

Final Wording (and

Order)

Including	yourself,	how	

many	total	people	live	

in	this	household?	

[____]	people	

How	many	of	the	following	

people	live	in	this	

household	with	this	child?		

Do not include this child in

your answer.		

Example: Brother(s)		2

Write ‘0’ if none.

Brother(s)	

Sister(s)	

Parent(s)	

Aunt(s)	

Uncle(s)	

Grandmother(s)	

Grandfather(s)	

Cousin(s)	

Parent’s	girlfriend/	

boyfriend/partner	

Other	relative(s)	

Other	non‐relative(s)		

How	many	people	live	in	

this	household?		

Include adults and children

who are temporarily away

from home (for example,

living in college housing) if

they have no other

permanent home.

[____]

How	many	people	live	in	

this	household?		

Include adults and

children who are

temporarily away from

home (for example, living

in college housing) if they

have no other permanent

home.

[____]	

Other	than	the	parents	

or	guardians	already	

reported,	how	many	of	

the	following	people	

live	in	the	household	

with	this	child?	

Example: Brother(s)		2

Write ‘0’ if none.

This	child’s...	

Brother(s)	

Sister(s)	

Aunt(s)	

Uncle(s)	

Grandmother(s)	

Grandfather(s)	

Cousin(s)	

Parent’s	

girlfriend/boyfriend/pa

rtner	

Other	relative(s)	

Other	non‐relative(s)	

Enter	the	total	number	of	

people	living	in	this	

household	with	this	child	

(This	number	should	be	

equal	to	the	sum	of	a	

through	k	above).		

Do not include this child in

your answer.

[____]		people	

How	many	of	the	following	

people	live	in	this	

household	with	this	child?		

Do not include this child in

your answer.		

Example: Brother(s)		2

Write ‘0’ if none.

Brother(s)	

Sister(s)	

Mother	(birth,	adoptive,	

step,	or	foster)	

Father	(birth,	adoptive,	

step,	or	foster)	

Aunt(s)	

Uncle(s)	

Grandmother(s)	

Grandfather(s)	

Cousin(s)	

Parent’s	

girlfriend/boyfriend/part

ner	

Other	relative(s)	

Other	non‐relative(s)	

How	many	of	the	

following	people	live	in	

this	household	with	this	

child?		

Do not include this child

in your answer.		

Example: Brother(s)		2

Write ‘0’ if none.

Brother(s)	

Sister(s)	

Mother	(birth,	adoptive,	

step,	or	foster)	

Father	(birth,	adoptive,	

step,	or	foster)	

Aunt(s)	

Uncle(s)	

Grandmother(s)	

Grandfather(s)	

Cousin(s)	

Parent’s	

girlfriend/boyfriend/pa

rtner	

Other	relative(s)	

Other	non‐relative(s)	
Of the 50 respondents who responded to the item, three respondents had problems with the 

original item. For who is in the household, problems included searching for an option for 

“mother” and not including themselves in the count; and being uncertain how to respond about a 

boyfriend who is at the house the majority of the time but who does not legally live in the 

household. The latter respondent decided not to include the boyfriend. 

Five respondents had difficulty responding to the count. Problems included respondents’ 

difficulty with the “a through k” language in the parentheses and reading “a” as an article, and 

including the child in the count. A respondent claimed that the item had “a lot of words,” and “a 

lot to read to ask a pretty simple question,” and suggested that the item should ask how many 

people live in this household excluding the child, and then to break it down further below. 

In the next round of testing, the item order was reversed and the exact wording from the 

household screener was implemented, based on feedback from the initial cognitive interviews. 

All the remaining respondents comprehended the revised first item “How many people live in 

this household?” 

The revised item “How many of the following people live in this household with this child?” 

that only had a change from “Parent(s)” to “Mother (birth, adoptive, step, or foster)” and “Father 

(birth, adoptive, step, or foster)” to match another item in the “Your Household” section, still 

presented problems similar to what was reported on the original item. Four respondents included 

the child in the counts, three respondents were uncertain whether to include some household 

members, i.e., two children away at college, a friend of the child’s who was living with the 

family for a period of time, and a brother-in-law who is a long-term houseguest. All three 

decided not to include these people. One of these respondents suggested that more information 

should be included in this item similar to the other household item about whether or not to 

include long-term houseguests. One respondent made a suggestion that “it might be worthwhile 

to indicate on the questionnaire that if you want the total number or people in this item to be the 

other household item minus 1.” 

Other Items in the Your Household Section

The majority of the respondents did not have any problems with the other items in the Your 

Household section. The item on the total income of all persons in the household was confusing 

for a few respondents because they did not know who in the household to include in the response 

and what income to include, for example, child support or disability. The item on which 

language(s) are spoken at home by the adults caused some confusion for a couple of respondents 

because they thought it was referring to the language spoken with the child, or they incorrectly 

did not include English when they were fluent in English. Respondents who lived in an 

apartment or a converted garage found the item on whether “this house” is owned, rented or 

occupied by another arrangement problematic, and one respondent suggested that the item 

should say “house or apartment.” 
2.2.5 Child’s Family

Two items of interest on the respondent’s education level were tested in cognitive interviews. 

The majority of the respondents did not have any problems when responding to these items on 

the highest grade or level of school that the respondent completed and whether he or she is 

currently attending or enrolled in school. The most common problem for a few respondents was 

that they were not sure if they should be responding about themselves or their child on both 

items. 

Additionally, on the first item a couple of respondents did not understand what “vocational 

diploma after high school” means, and one of these respondents actually had a “vocational 

certificate” although responded “high school diploma or equivalent (GED)” on the questionnaire. 

On the second item, a couple respondents did not initially see “job training” at the end of the 

item, and one respondent was confused because the item is asking two questions (whether 

respondent is in school or receiving vocational education or job training) and suggested that it 

would be more clear to ask for the “student status.” 

No changes were made to these items during cognitive testing. 

2.3 ECPP Items

2.3.1	Finding and Choosing Care for Your Child

Respondents with children age five or younger completed additional items related to finding and 

choosing care for a child. Three of these are items of interest and are discussed below. 

Difficulty Finding Child Care

One item of interest on the primary reason for difficulty finding child care was tested in 

cognitive interviews. This item was tested by the five respondents who had indicated in the 

previous item that they faced some difficulty finding the type of child care or early childhood 

program they wanted for their child. The following wording changes were made to the item 

during testing (see Table 9 below). 

 
Table 9. Wording changes for the item on primary reason for difficulty finding child care 

Original Wording Wording Change Final Wording

What	was	the	primary	reason	

for	the	difficulty	finding	care?	

	Cost		

	Location		

	Quality	

	Lack	of	open	slots	for	new	

children		

	Needed	a	program	for	

children	with	special	needs	

	Other	reason:	Specify	

______________________________		

What	was	the	primary	reason	for	

the	difficulty	finding	care?	

Mark X ONE only.		

	Cost		

	Location		

	Quality	

	Lack	of	open	slots	for	new	

children		

	Needed	a	program	for	children	

with	special	needs	

	Other	reason:	Specify	

______________________________		

What	was	the	primary	reason	for	

the	difficulty	finding	child	care	

or	early	childhood	programs?	

Mark X ONE only.		

	Cost		

	Location		

	Quality	

	Lack	of	open	slots	for	new	

children		

	Needed	a	program	for	children	

with	special	needs	

	Other	reason:	Specify	

______________________________		

Of the two respondents who responded to the initial item, one did not have any difficulty 

responding. The other respondent had difficulty understanding whether or not it was possible to 

mark more than one option and marked several options for this item including “Cost,” “Quality,” 

and, “Lack of open slots for new children.” 

After the instructions “Mark X ONE only” was added to the item, the remaining respondents 

correctly responded to the item without any confusion. In the final version, the wording was 

changed to mirror the language of another item on choices for child care “or early childhood 

programs” where the respondent lives. 

Importance of Reasons for Choosing Child Care		

One item of interest asks how important it was in choosing the childcare arrangement or program 

where their child spends the most time. Five new sub-items were tested in cognitive interviews 

and they are described below. 

Ratings from a Website 

Of the nine respondents who answered this sub-item, all but one had no difficulty understanding 

the item or finding a response that matched their experience. One respondent did not fill out this 

item and was uncertain whether or not the question was referring to the child care provider’s 

website or to an external website. 
Recommendations from friends and family 

Of the nine respondents who answered the sub-item on how important recommendations from 

friends and family were in choosing a childcare arrangement or program, none of the 

respondents had difficulty understanding the question or selecting a response. 

Accreditation of care provider 

Two versions of this sub-item were tested on how important accreditation of the care provider 

was in choosing a childcare arrangement or program (see Table 10 below). 

Table 10. Wording changes for sub-item on importance of accreditation of care provider 

Original Wording Wording Change/Final Wording		

Accreditation	of	the	care	provider	(e.g.	from	

the	National	Association	for	the	Education	

of	Young	Children	or	the	National	Early	

Childhood	Program	Accreditation)?		

Not	at	all	important	

A	little	important		

Somewhat	important		

Very	important	

Accreditation	of	the	care	provider?	

“Accreditation" is another way to judge the quality of a

child care program. Family child care homes and child

care centers can choose to get accredited by a child

care accrediting organization, such as the National

Association for the Education of Young Children or the

National Early Childhood Program Accreditation.		

I	do	not	know	about	the	accreditation	of	my	child’s	

care	provider.	

Not	at	all	important	

A	little	important		

Somewhat	important		

Very	important

The original item was tested by four respondents; three of these respondents were unsure what 

accreditation was. Based on these cognitive interviews, the item was changed to add a definition 

of accreditation and the following response option “I do not know about the accreditation of my 

child’s care provider.” The revised version of the item was tested by five respondents. Adding 

the definition seemed to be helpful for respondents to better understand what the item was 

asking; several were able to better articulate the meaning of accreditation. 

Religious orientation of the program 

Several of the nine respondents who answered this sub-item on how important the religious 

orientation of the program was in choosing a childcare arrangement interpreted this item to mean 

that the child would attend a program run by a church or religious group. Respondents did not 

appear to have difficulty understanding the item or finding a response.
QRIS 

Three versions of this sub-item on how important a local Quality Rating Improvement System 

(QRIS) was in choosing a childcare arrangement or program were tested in cognitive interviews 

(see Table 11below). 

Table 11. Wording changes for sub-item on importance of QRIS 

Original Wording Wording Change Final Wording

Did	not	use	a	QRIS	

Not	at	all	important	

A	little	important	

Somewhat	important	

Very	important	

I	have	not	heard	of	a	QRIS

Did	not	use	a	QRIS	

Not	at	all	important	

A	little	important	

Somewhat	important	

Very	important	

I	have	not	heard	of	a	QRIS

Not	at	all	important	

A	little	important	

Somewhat	important	

Very	important	

The original response options were tested by four respondents. All four respondents were 

unaware what a QRIS was and chose “did not use a QRIS.” Based on these cognitive interviews 

and to make the item more parallel to the accreditation sub-item, a response option, “I have not 

heard of a QRIS” was added. Two respondents tested the item with this additional response 

option, which both respondents selected. To streamline the response options, the option of “did 

not use a QRIS” was removed and the revised item was tested. Respondents did not have any 

problems with selecting from the final response options. 

Early Childhood Program Curriculum

Three versions of this item of interest that asks about the type of curriculum offered at the early 

childhood program were tested in cognitive interviews (see Table 12 below). 

 
Table 12. Wording changes for item on early childhood program’s curriculum 

Original Wording Wording Change Final Wording

Which	of	the	following	best	

describes	this	program’s	

curriculum?	

Montessori	

Special	program	emphasis‐	

such	as	science	or	math	

emphasis,	performing	arts,	

talented	or	gifted	preschool,	

foreign	language	immersion,	

etc.	

Special	education‐	primarily	

serves	students	with	

disabilities	

The	Creative	Curriculum®	

None	of	the	above

Which	of	the	following	best	

describes	this	program’s	

curriculum?	

Montessori	

Special	program	emphasis‐	

such	as	science	or	math	

emphasis,	performing	arts,	

talented	or	gifted	preschool,	

foreign	language	immersion,	

etc.	

Special	education‐	primarily	

serves	students	with	

disabilities	

Play‐based	curriculum,	such	as	

The	Creative	Curriculum®	or	

Reggio®	curriculum	

None	of	the	above		

Which	of	the	following	best	

describes	this	program’s	

curriculum?	

Montessori	

Another	play‐based	curriculum,	

such	as	The	Creative	

Curriculum®,	High	Scope	or	

Reggio	Approach	

Special	program	emphasis‐	

such	as	science	or	math	

emphasis,	performing	arts,	

talented	or	gifted	preschool,	

foreign	language	immersion,	

etc.	

Special	education‐	primarily	

serves	students	with	disabilities

None	of	the	above		

This item of interest was tested by nine respondents. They were generally not familiar with The 

Creative Curriculum on the original response option, although one respondent said that the child 

attends a program with a play-based curriculum similar to The Creative Curriculum. Only a 

couple of respondents were familiar with Montessori programs although not familiar with the 

specifics of the curriculum. Respondents were also not familiar with the Reggio curriculum. 

Based on these cognitive interviews the term “play-based” and another example of a play-based 

curriculum were added to this response option. 

The final response options were further revised since Montessori is also a play-based curriculum, 

another example of a play-based curriculum was added, and the order of the response options 

was changed to make the item clearer. Respondents were typically unaware of the curriculum 

used in their child’s early childhood program. 

Other ECPP Items

Respondents did not have any problems or issues with the other four ECPP items that were tested 

in cognitive interviews. 

	 	
Appendix A – Interview Protocol
31

2016 Parent and Family Involvement in Education Survey (PFI) and Early Childhood Pro

(ECPP) Survey Cognitive Interview Protocol

Interview ID: Date of 

Interview: 

Parent First Name: 

Interviewer 

Age of Child: STEP 1: WELCOME AND INTRODUCTION SCRIPTInterviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with wbe thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer. Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming [Create small talk to build rapport with the parent]: You are here today to help us out with a new survey that we are working on for the U.S. Department of Education. This suchild’s schooling [and finding and choosing child care for your child for parents with children 5 and younger]. I will ask yyou were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. Aquestions I will ask you about your answers to the questions. This will help us make the survey better and clearer for other Because the information you provide is so important, I am going to be taking notes while you are answering the survey an[IF OBSERVER PRESENT] One of my colleagues will be observing our discussion through a live video stream. It is noIs that OK with you? [IF NOT, TURN OFF WEBEX]
32

Also, this interview will be audio-recorded so researchers can review the recording later. Any information that refers to yopersonally, like your name or name of your child’s school, will not be included in our report, and your responses will be uonly. Remember, you are not being tested; we want you to tell us what you think about the survey. If at any time you want to stop, just let me know. [If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the Before we continue, do you have any questions about what I just said? [Answer any questions the parent may ask.] IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER WITH A COPY OF THRESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCQUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM W

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER. 

STEP 2: THINK-ALOUD PRACTICE While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how yo

to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers you give me. To help you get started on thinking aloud, we are going to do a practice activity. First, I will show you an example of how to think out loud. Then I will give you a chance to practice it. QUESTION: How many windows are there in the house or apartment where you live? [Model the think aloud process for the parent, including reading the question out loud. Elaborate on counting all of the winapartment and as needed you can make up pieces of information to emphasize the think aloud process e.g., “there is a door and it has windows on it; I think I’ll count those as windows. In the living room there are windows with panes in them and as a separate window...” When you are modeling, write down the number of windows in your house/apartment as you are 

[Once finished the model practice think aloud] I’m going to give you a piece of paper with the question on it. I’m going to the question and think aloud as you decide on your answer. 
33

Give respondent handout with think aloud practice. How many windows are there in the house or apartment where you live? Please tell me what you are thinking while you are figuring out your answer. Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on thme more about what you are including in your count? When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the queHAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL.

STEP 3: COMPLETION OF THE QUESTIONNAIRE 

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please rewould have read to yourself if I were not here. Remember that I am very interested in hearing you talk about what you are thinking as you figure out your answers to the 

REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL. 
34

Useful Probes Can you tell me in your own words what that question was asking? What does the word [term] in this question mean to you? How easy or difficult is it to remember [topic]? How easy or hard was it to choose an answer? Tell me what you are thinking? How did you come up with that answer? You said [answer]. Can you tell me more about that? [In response to R actions or comments]: Tell me what happened? Tell me what you were doing? [If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.” [If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this quhappened. Tell me about what you were just doing.” 
35

Screener ASK THE RESPONDENT TO FILL OUT THE ENTIRE SCREENER. REMIND THEM TO THINK ALOUD IF THEY ARE NOT DOING SO. Q3. What is the person’s month and year of birth?[If the respondent is having difficulty with the month and/or year] How did you come up with your answer? Q5. Is this person currently in...?How did you come up with your answer? [If respondent chooses “homeschool”] What does “homeschool” mean to you? DO NOT PROBE SPECIFICALLY ABOUT HOMESCHOOLING HERE. FOR THE SCREENER WE ONLY WANT TO UNDERSTAND HOW THEY ARE INTERPRETING “HOMESCHOOL.”PFI/ECPP Survey 

Child’s Schooling

ASK THE RESPONDENT TO FILL OUT THE SURVEY. ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY; REMIND RESPONDENT TO THINK ALOUD Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them. 
36

Q1-Q2: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. 

Q3. Is it his/her district-assigned school? A district-assigned school is the school that your local public school district told you that this child can attend, based on the location of your residence.[Ask only if responded to this question] How did you decide on your answer? What does “district assigned” mean to you? How did you know what school your child was going to attend? How do other parents in your child’s district know which schools their child will attend?Q4-Q13: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Q14. Is he/she currently enrolled in any high school Advanced Placement (AP) classes?What were you thinking of when you answered this question? What do “Advanced Placement (AP) classes” mean to you? What grade-level are the students who usually take AP classes in your child’s school district? 
37

Q15-Q21: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. 

Q22. Is this child taking any school-related courses online instead of in-person with the teacher? Do not include courses that use the 

Internet only for selected assignments.How did you decide on your answer? 

Q23. Is that instruction [internet] provided by any of the following places? How did you decide on your answer? [If chooses a response option or “Someplace else:”] Tell me more about that. How does your child access the classes? Where is he/she when accessing the classes? Who provides the class?Q24: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. 
38

Q25. Some parents decide to educate their children at home rather than send them to a public or private school. Is this child being schooled at home instead of at school for at 

least some classes or subjects?In your own words, tell me what this question is asking about. How did you determine whether to answer yes or no? [If respond “yes,”] Do not probe on the meaning of homeschool until Q30. [If respond “no” and if not evident from think aloud to previous questions] What does “homeschool” mean to you? Make a note if the respondent did not skip properly, but do not probe on skip pattern.]Q26. Which of the following statements best describes your homeschooling arrangement for this child?How did you come up with this answer? [Do not probe on the meaning of homeschool until Q30. Check to see if respondent skipped properly, but do not probe on skip pattern.]Q27. How many hours each week does this child usually go to a public or private school for instruction? Do not include time spent in extracurricular activities.How did you come up with your answer? 
39

[Check to see if respondent filled in this question although they indicated that the child is homeschooled.] 

Q28. There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool this child because:How did you determine how to answer yes or no to each of these questions [a-j]? Tell me more what you were thinking when answering these questions. [Do not probe on the meaning of homeschool until Q30.] Q29. Of the reasons your family chose to homeschool this child, which one would you say is the most important to you?How did you come up with your answer? [If not evident from think aloud to previous questions] What does “homeschool” mean to you? Q30. Thinking about all years this child has been homeschooled, which of the following subject areas has this child been taught during his or her home instruction?How did you decide what to mark? How do you cover these subjects during home instruction? Is there anything missing? 
40

Q31. Which of the following subject areas are being taught to this child now? How did you come up with your answer? Can you tell me more about what the child is being taught now? Child’s Health REMIND RESPONDENT TO THINK ALOUD Q32-Q35: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents. Q36. Are these services provided by any of the following sources? How did you decide on your answer? Q37. Are any of these services provided through an Individualized Education Plan (IEP) or services plan? How did you come up with your answer? What does a “services plan” mean to you? 
41

Child’s Background Q38-Q43: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.Q44. Does this child live at this address and another address (for example, because of a joint custody arrangement)? How did you come up with your answer? In your own words, what is this question asking? [If answered No] What does joint custody arrangement mean to you? Q45. If yes, does this child spend most of the time at this address, at another address or equal time at both addresses? How did you come up with your answer? [If needed] Can you describe the arrangement your child has? Q46. What language does this child speak most at home? It is not necessary to probe on this question.Note the respondent’s response, especially for those who are parents of ELL students. Record in your notes whether the respondent reads the response options below the top two responses. 
42

Q47. It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.

Your Household Q48. How many people live in this household?

 How did you come up with your answer? Who are you including in the counts? 

Q49. How many of the following people live in this household with this child?Tell me more about what you were thinking when you answered this question? Who are you including in the counts? In your own words, what is this question asking? Q50-Q58: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondentsChild’s Family REMIND RESPONDENT TO THINK ALOUD Q59. What is the highest grade or level of 

school that this parent or guardian completed?
43

[Only if respondent chooses “high school diploma or equivalent”, “vocational diploma after high school”, or “some college, but no degree”:] How did you come up with your answer? What does “vocational diploma after high school” mean to you? 

Q60. Is he or she currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?In your own words, what is this question asking? If parent wants to discuss another child in K-12 (not the first listed on the screener), there can be an informal follow

parent-directed- they can provide additional insight regarding how they might have answered some questions for a

time. THE REMAINING QUESTIONS ARE FOR PARENTS WHO ALSO HAVE A CHILD UNDER AGE-5, NOT YEAND WHO IS IN A CARE ARRANGEMENT.Tell the respondent the following: The remaining questions are about your other child who is under age 5, not yet in kindergarten and in a care arrangemenWhat type of care arrangement does this child have? [Record this in your notes.]Please continue and answer the remaining questions. Please remember to think aloud. 
44

Finding and Choosing Care for your Child REMIND RESPONDENT TO THINK ALOUD Q61-64: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.Q65. What was the primary reason for the difficulty finding child care or early childhood programs? Please tell me more about how you chose your answer. Can you tell me more about that? Q66. How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time? 66h. Ratings on a website? How did you decide what to mark? In your own words, what is 66h asking? Could you provide some examples? 66i. Recommendations from friends and family? How did you decide what to mark? 
45

In your own words, what is 66i asking? 66j. Accreditation of the care provider?"Accreditation" is another way to judge the quality of a child care program. Family child care homes and child care centers can choose to get accredited by a child care accrediting organization, such as the National Association for the Education of Young Children or the National Early Childhood Program Accreditation. How did you decide what to mark? In your own words, what is 66j asking? [If definition seems confusing:] In your own words, what is the text [point to definition]telling you? 66k. The religious orientation of the program?How did you decide what to mark? In your own words, what is 66k asking? 66l. Your city or state’s Quality Rating and Improvement System (QRIS)? How did you decide what to mark? In your own words, what is 66l asking? What does QRIS mean to you? 
46

[If applicable:] How did you learn about the QRIS? How did you use it? Do you remember what it is called for your state or area—that is, which QRIS did you use? 

Q67. Which of the following best describes this program’s curriculum? How did you come up with your answer? [If respondent chose Montessori:] Tell me what Montessori means to you. [If respondent chose Play-based curriculum:] Can you tell me more about your response? What does “play-based” mean to you? Are you familiar with The Creative Curriculum? How about HighScope? How about Reggio Approach? Are there other play-based curriculum that you are aware of? [If respondent chose None of the above:] How would you describe the type of curriculum used in your child’s program? Do you know the name of the curriculum used in your child’s program? 
47

FINAL QUESTION Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? STEP 4. THANK YOU FOR PARTICIPATING We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey bet

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to the lobby. Please give t

envelope [$40 gift card], and thank them for their time and effort.
Cognitive Interview Findings 

for the Adult Training and 

Education Survey (ATES) 2016: 

Draft Report 

Prepared for the National Center for 

Education Statistics (NCES) by 

Stephanie Cronen & Kirsten Manville 

 

June 2015 

1000 Thomas Jefferson Street NW 

Washington, DC 20007-3835 

202.403.5000 | TTY 877.334.3499 

www.air.org 

Cognitive Interview Findings for the 

Adult Training and Education Survey 

(ATES) 2016: Draft Report 

June 2015 

Stephanie Cronen & Kirsten Manville 

 

Contents 

Page 

Introduction ......................................................................................................................................3

Methodology ....................................................................................................................................3

Sample..............................................................................................................................................4

Key Findings ....................................................................................................................................4

Appendix A. Cognitive Interview Protocol .....................................................................................9

 
American Institutes for Research Cognitive Interview Report for ATES 2016—3 

Introduction 

Beginning in January 2016, the National Household Education Survey (NHES) is expected to 

field the first full-scale administration of the Adult Training and Education Survey (ATES). The 

American Institutes for Research (AIR) is working with the National Center for Education 

Statistics (NCES) and staff from numerous other federal statistical agencies to develop a strategy 

for measuring the prevalence of subbaccalaureate credentials—in particular, industry-recognized 

certifications, licenses, and educational certificates—and work-related training and education in 

the United States. This multiphase research effort began in 2009 and includes the development 

and testing of questions that will be included in the ATES. In Phase I, AIR contracted with 

Shugoll Research to conduct focus groups and cognitive interviews with respondents holding 

less than a bachelor’s degree who work in fields that may offer subbaccalaureate opportunities. 

In Phase II, the Center for Survey Methods at AIR designed and conducted cognitive interviews 

of potential survey items that may be used to measure certifications and licenses. In Phase III, we 

extended this work in two stages—for the originally planned 2015 NHES administration and for 

the 2016 rescheduled administration—to include measures of participation in and characteristics 

of work-related training and education. In addition, we conducted further testing of revised 

measures of credential attainment. 

This draft report documents the key findings from the English language cognitive interviews 

conducted in Phase III for the 2016 administration. The revised report will include key findings 

from the Spanish language cognitive interviews that are currently being conducted. 

Methodology 

Each cognitive interview lasted approximately one hour and was conducted using primarily a 

“think-aloud” approach with concurrent probing as needed, based on a structured protocol. The 

methodology was developed by a senior researcher at AIR in consultation with NCES and 

drawing on best practices and methods from cognitive science (see Appendix A for the protocol). 

These interviews were designed to identify problems of ambiguity or misunderstanding in item 

wording. The interviews were conducted by interviewers who received training in the cognitive 

interview technique and in the interview protocol by a senior researcher at AIR. Researchers 

from AIR and NCES listened to a subset of interviews and provided guidance and feedback 

throughout the study. 

During each interview, the study participant was welcomed by the interviewer and led to an 

interview room. The participant was then told that the purpose of the interview was to help 

answer questions about how people respond to items on schooling. All participants were assured 

that their participation was voluntary and that their answers would be used only for research 

purposes and not disclosed, or used, in identifiable form for any other purpose except as required 

by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the 

think-aloud process (see below) and modeled a think-aloud practice with a sample question. 

Participants were then given a paper copy of the questionnaire to fill out. 
American Institutes for Research Cognitive Interview Report for ATES 2016—4 

The interviews were conducted using “concurrent probing,” a technique in which interviewers 

asked respondents to “think aloud” as they answer the survey items. The cognitive interviews 

also included a verbal probing component, conducted after completion of the think-aloud portion 

for a given section of the questionnaire. The verbal probes included a combination of preplanned, item-specific questions identified before the session as important and ad hoc questions 

that the interviewer identified as important from observations during the interview, such as 

clarifications or expansions on points raised by the participant. 

Sample 

Cognitive interview participants were recruited through the recruitment firm Elliott Benson in 

Sacramento and San Mateo, CA and by AIR staff in Washington, DC and Grand Rapids, MI. All 

interviews were conducted in person at these same locations. Respondent received $40 for their 

participation. 

The majority of respondents were required to either have a professional certification, an 

educational certificate, or other work-related training. A small number of additional cases were 

included to represent the general population. Based on screening data, the number of respondents 

within each target group was:1

 Certification or license: 28 

 Certificates: 18 

 Work-related education or training: 22 

 General population: 2 

AIR conducted a total of 48 cognitive interviews between March 3, 2015, and April 17, 2015. 

The interviews were conducted in four rounds to facilitate revision and re-testing. 

Key Findings 

Findings are presented by survey section, and changes made to the survey to address response 

issues are described below. 

Items on Education 

The follow-up item that asks for the field of study for highest level of education was difficult for 

some respondents to follow, either because they did not see their field listed, or because they did 

not notice the instructions to mark one field only. To address these issues, we simplified the label 

for the two most problematic fields of study (healthcare and social studies, political science, 

economics, or history) and underlined the word “one” in the stem: 

 

1

 These target group numbers sum greater to the total number of interviews completed because respondents could 

have combinations of target criteria. 
American Institutes for Research Cognitive Interview Report for ATES 2016—5 

“Which one of the following best describes the field of study for the highest level 

of school you have completed? (Mark ONE only.) 

The item intended to measure participation in adult basic education was eliciting false positive 

responses, due to respondents’ generalization of the term “literacy.” Classes such as computer 

literacy and college composition courses were being reported under this item, and people who 

taught rather than took classes pertaining to literacy or reading were reporting participation. 

Therefore, we underlined the word “your” in the stem and instructed respondents not to include 

college-level classes: 

Since leaving high school, have you taken any literacy classes to help improve 

your reading? Do not include college-level classes. 

The item designed to measure college enrollment was confusing to some respondents for several 

reasons. There were two different time referents in the stem (“currently” and “after high school”) 

and the item appeared after the question about literacy classes, which seemed to be creating an 

order effect. Therefore, some respondents interpreted the question as being about remedial 

classes. To address these issues, we moved the item to appear after the first two educational 

attainment items, and deleted “after high school”: 

Are you currently enrolled at a college, university, technical or trade school, or 

other school? 

Items on Certifications and Licenses 

In prior administrations, follow-up questions referred to the most recent certification or license; 

however, we found that most recent credentials were not always being reported due to low 

salience; i.e., not all of these credentials were considered currently relevant or as the most 

important credential to all respondents, particularly in fields where respondents are required to 

obtain multiple credentials. For 2016, we therefore tried a new approach by asking about 

respondents’ most important certification or license. Based on interview findings, we determined 

that this approach is valid, and that most respondents can easily identify what they consider to be 

their most important certification or license. 

A familiar issue in this set of follow-up questions persisted, however; some respondents 

conflated multiple certifications or licenses when responding. Each item already reminds 

respondents to report on their most important certification or license; therefore, no further 

revisions were made. 

The item that serves as the primary indicator of the credential being a license rather than a 

certification was difficult to answer for several respondents who were not sure if a government 

entity issued their credential. For some, it was the parenthetical examples that were causing their 

confusion; therefore, we deleted those examples and simplified the item to: 

Is your most important certification or license required by a federal, state, or 

local government agency in order to do that kind of work? 

We also added a second question from the 2010-11 ATES to help identify false positive reports: 
American Institutes for Research Cognitive Interview Report for ATES 2016—6 

 Can your most important certification or license be revoked or suspended for any 

reason?

Items on Certificates 

The main certificate item continues to be a challenge, due to the overuse of the term among 

education and training providers. The result of this overuse is confusion between certificates and 

certifications and between educational certificates (based on a postsecondary program of study) 

and certificates of completion (based on a single course or training). In the current round of 

interviews, we found these same issues, but to a lesser extent. It appears to be helping to (a) ask 

about certifications and licenses before certificates; and (b) ask about other types of certificates 

(e.g., from an employer for completing a training) before asking about educational certificates. 

Therefore, only minor modifications were made to the certificates items for readability purposes. 

Specifically, we deleted some unnecessary text in the stem and response options to make the 

item less cognitively burdensome. 

Items on Work Experience Programs 

One of the primary purposes of the 2016 interviews was to further test the relatively new items 

on participation in and characteristics of work experience programs. Overall, we found that these 

items performed well, with the exception of a moderate amount of over-reporting by respondents 

who misinterpreted the items to pertain to more general on-the-job experience. We addressed this 

to the extent possible by revising the stem of the main item to refer explicitly to types of work 

experience programs: 

Have you ever completed an internship, co-op, practicum, clerkship, externship, 

residency, clinical experience, apprenticeship, or similar program?

Some respondents also had difficulty finding the field of study for their program. Therefore, we 

made the healthcare field easier to find and added nursing assistant to the list. 

Another relatively common area of overreporting was for respondents to report that their work 

experience program was part of various types of schools’ education or training program. To get 

respondents to better attend to the response options, we revised the stem to be more general and 

the response options to be more specific: 

Which one of the following best describes your last work experience program? (Mark ONE 

only.) 

 It was not part of a formal education program 

 It was part of a high school program 

 It was part of a school program after high school and below an Associate’s degree 

program 

 It was part of an Associate’s degree program 

 It was part of a Bachelor’s degree program 

 It was part of an advanced degree program or other program beyond a Bachelor’s degree
American Institutes for Research Cognitive Interview Report for ATES 2016—7 

General Issues 

We identified two areas of response error across multiple survey sections of the English version 

of ATES. The first type of error was respondents’ tendency to mark a response to only one item 

among a list of items that calls for a response to each item in the list. We tried several variations 

on item formatting and instructions, and ultimately chose to include the instruction “(Mark one 

box for EACH ITEM below.)" 

The second type of error we encountered across sections was unemployed respondents who 

missed the “Not applicable, not currently working” response option that applied to them. To 

increase the salience of this option, we moved it to appear as the first option rather than the last. 

Summary 

AIR conducted four rounds of cognitive interviews to test questions for their potential to measure 

the prevalence of subbaccalaureate credentials and participation in and characteristics of workrelated training among adults in the United States. During the cognitive testing, the AIR team 

found that some respondents had problems (1) understanding the purpose of several items, and 

(2) locating the correct response option. To address these problems, the questions and response 

options were simplified and re-ordered in some cases, and instructions were added or modified. 

In addition, one new item was added to the certifications and licenses section to help identify 

false positive reports. 

 
American Institutes for Research Cognitive Interview Report for ATES 2016—8 
American Institutes for Research Cognitive Interview Report for ATES 2016—9 

Appendix A. Cognitive Interview Protocol 

Interview ID: Date of 

Interview: 

First Name: Interviewer 

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is 

suggested content with which the interviewer should be thoroughly familiar in advance. Text in 

brackets [ ] presents instructions for the interviewer. 

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet 

you, thanks for coming to help us out today. 

[Create small talk to build rapport with the respondent] 

You are here today to help us out with a survey that we are working on for the U.S. Department of 

Education. This survey asks about adults’ education and training experiences. I will ask you to 

complete the survey as if you were taking a real survey. There are no right or wrong answers, and 

it’s ok if you do not know some of the answers. As you are answering the questions I will ask you 

about your answers. This will help us make the survey better and clearer for other adults like you. 

To make sure we learn everything we can from this session, I am going to be taking notes while you 

are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and 

my colleague will also be observing our session today behind the glass or by video]. 

Also, this interview will be audio recorded so researchers can review the recording later. Any 

information that refers to you personally, like your name or employer, will not be included in our 

report, and your responses will be used for research purposes only. 

Remember, you are not being tested; all we want is to learn how well the survey works. 

If at any time you want to stop, just let me know. 

[If, for any reason, the participant is no longer interested in participating, thank the participant for 

his/her time and end the interview.] 

Before we continue, do you have any questions? [Answer any questions they may ask.] 

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the 

form (or read it to him/her, depending on respondent preference), answer any questions, and 

have the respondent sign the form. Leave a separate copy of the form with the respondent. 

If the respondent has consented to recording, start the recorder.] 

STEP 2: THINK-ALOUD PRACTICE 

While you fill out the survey I want you to talk about how you are answering the questions. Hearing 

you talk about how you decide on your answers will help me understand how to make the questions 

better. I am more interested in how you arrive at your answers than in the answers you give. To help 

you get started on thinking aloud, we are going to do a practice activity. [Show an example by 

asking yourself the question and answering it using ‘think-aloud’]

I’m going to give you a piece of paper with a question in it. I’m going to ask you to read out loud the 

question and think aloud as you decide on your answer. 

[Give respondent handout with think aloud practice.] 
American Institutes for Research Cognitive Interview Report for ATES 2016—10 

How many glasses of water did you drink yesterday? 

Please tell me what you are thinking while you are figuring out your answer. 

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when 

you were deciding on that answer? Can you tell me more about what you are including in your 

count? 

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about 

your answers to the questions in the survey. 

[Hand respondent a copy of the questionnaire and a pen or pencil.]

STEP 3: COMPLETION OF THE QUESTIONNAIRE 

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on 

your answer. Please read aloud anything you would read to yourself if I were not here. 

Remember that I am interested in hearing you talk about what you are thinking as you figure out 

your answers to the questions. 

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY 

ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR 

QUESTIONS ARE REALLY HELPFUL.] 

Useful Probes—Use only as needed when response is unclear or R appears confused or 

unsure 

Can you tell me in your own words what that question was asking? 

What does the word [term] in this question mean to you? 

How easy or difficult is it to remember [topic]? 

How easy or hard was it to choose an answer? 

Tell me what you are thinking. 

How did you come up with that answer? 

You said [answer]. Can you tell me more about that? 

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing? 

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Tell me what you are 

thinking. 

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old 

question before answering this question]: Tell me what just happened. Tell me about what you 

were just doing. 

Education 

 

STOP RESPONDENT AFTER Q5 – Probe on Q1-5 NOTES

ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY 

Instructions: It is not necessary to probe on the instructions. Note if the 

respondent has any problems if they read them.

Q1-Q4: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 
American Institutes for Research Cognitive Interview Report for ATES 2016—11 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q5.

Since leaving high school, have you taken any literacy classes to help 

improve your reading? Do not include college-level classes. 

(Ask everyone) What types of classes would you include as literacy classes? 

“Your” underlined. 

Probe change – to ask 

what classes are literacy 

classes. 

Certifications and Licenses 

STOP RESPONDENT AFTER Q15 – Probe on Q6-15 

Q6-7: It is not necessary to probe on this question. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q8: [Only probe if response to Q7 > 1.] 

The next few questions ask about the certification or license that you 

consider to be your most important. What is the name of your most 

important certification or license? 

 

What do the words “most important” mean to you in this question? 

How did you decide which certification or license was your most important? 

How easy or hard was it to choose your most important certification or 

license? 

Q9: It is not necessary to probe on this question. Let the respondent continue 

the think aloud and only probe if respondent says something that indicates a 

potential problem, or if there is confusion or an inconsistent interpretation 

compared to other respondents. 

Q10: 

Is your most important certification or license required by a federal, state, 

or local government agency in order to do that kind of work? 

Who issued your certification or license? 

(If R seems confused) How easy or hard was it to answer this question? 

Q11-14: It is not necessary to probe on this question. Let the respondent 
American Institutes for Research Cognitive Interview Report for ATES 2016—12 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q15. How useful has your most important certification or license been for 

each of the following? (a-d)

[If respond to Q7 is > 1]: How easy or hard was it to only answer about 

your most important certification or license for each question? 

Q8 through 15

Tell me a little bit more about your most important certification or license—

in your own words, what was the process you went through in order to get 

it? 

Note to interviewers: What we’d like to find out is whether R really has a 

certification or license, and whether they have one but also have a certificate 

from a course or a program (a series of courses). Information about courses 

or training they took to prepare, where they took it, whether they had to 

pass a certification or license exam, where they took that exam, and who 

issued the certification or license all provide good clues. 

Keep in mind that someone might go to a school (or other organization) to 

take course(s) to prepare for this type of credentialing exam, then take the 

exam and have both (1) a certificate of completion (30a) or post-secondary 

certificate (30d) from going through this preparation process as well as (2) 

a certification and/or license issued after passing the credentialing exam. 

Therefore, some Rs report something under both certifications and 

certificates and it’s legitimate for them to do so. Other Rs just confuse the 

two and report certificates under certifications and vice versa. 

To help you understand the difference, a certification or license is 

recognized in the R’s field as proof of qualification to do a particular type of 

job, and usually requires an “official” exam that isn’t specific to a 

particular school or other training provider—everyone in the state or 

country would take the same exam, regardless of whether they took any 

preparation courses or not. A state usually issues licenses, and professional 

associations (like the American Association of Medical Assistants) usually 

issue certifications for their particular field. 

In contrast, a certificate simply indicates that the R has participated in some 

type of education or training to learn content in an area. It does not 

“authorize” them to practice in that area. A certificate of completion (30a) 

is usually based on a single course and offered by a variety of providers, and 

a post-secondary certificate (30d) is awarded by a school after someone has 

completed a specific series of courses (program of study), just like a college 

degree. It is considered a college credential, but generally takes less time to 

get than an associate’s degree. 

Note also that many colleges or other organizations partner with 
American Institutes for Research Cognitive Interview Report for ATES 2016—13 

credentialing bodies to host their exams, so a R might take courses or go 

through a certificate program at a community college and then take the 

certification or licensing exam there. The school doesn’t “issue” the 

certification, however; they are just serving as proctors for the state or 

association who does issue it. This makes it a challenge for us to 

differentiate between certifications and certificates, so we try to learn about 

the educational process Rs went through as well as the type of exam they 

took, if any. 

IF RUNNING BEHIND OR R IS REPORTING SOMETHING OTHER 

THAN LICENSES AND CERTIFICATIONS, MOVE THEM ON TO 

CERTIFICATES SECTION. 

OTHERWISE, STOP RESPONDENT AFTER Q24 – Probe on Q16-24 

Q16-24 (if answer “Yes” to Q16 – probe): Tell me a little bit more about 

your second-most important certification or license—in your own words, 

what was the process you went through in order to get it? 

Q16-24. Do you have another currently active certification or license? (if 

answer “Yes” to Q16 – probe) 

 

How did you decide which certification or license was your second most 

important? 

How easy or hard was it to choose your second most important certification 

or license? 

Q17 through 24 

For questions 17 through 24, was it easy or hard to answer for just your 

second most important certification or license? 

Did you think about other certifications or licenses when you answered these 

questions? 

Q19. Is your second most important certification or license required by a 

federal, state, or local government agency in order to do that kind of work? 

Who issued your certification or license? 

Q20-23: It is not necessary to probe on this question. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q24. How useful has your second-most important certification or license 

been for each of the following?(a-d) 

How easy or hard was it to only answer about your second most important 
American Institutes for Research Cognitive Interview Report for ATES 2016—14 

certification or license for each question? 

STOP RESPONDENT AFTER Q26 – Probe on Q25-29 

 

Q25-29: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Certificates

STOP RESPONDENT AFTER Q38 – Probe on Q30-38 

Q30a. 

People sometimes earn certificates from an education or training program. 

These are different from certifications or licenses. Do not include 

certifications or licenses here. Have you ever earned any of the following 

types of certificates? 

a. A certificate for completing a training program from an employer, 

employment agency, union, software or equipment manufacturer, or other 

training provider. 

(If R choose “yes”) Tell me a little more about what you’re reporting for 

30a. In your own words, what was the process you went through in 

order to get it?

(If R appears to be reporting same credential as in certifications/licenses) 

How is this related to the certification or license you reported earlier? 

Note to interviewers: Similar to what we’re trying to learn in the 

certification and license section, we need to know more about how R got the 

certificate, from what type of organization (get the name if you can), and if it 

is the same thing or related to the certification or license reported earlier (if 

applicable).

Q30d. People sometimes earn certificates from an education or training 

program. These are different from certifications or licenses. Do not 

include certifications or licenses here. Have you ever earned any of the 

following types of certificates? d. A certificate—not a degree—for 

completing a program at a community or technical college, or other 

school after high school. Do not include teaching certificates or 

college degrees.

(If R choose “yes”) Tell me a little more about what you’re reporting for 

30d. In your own words, what was the process you went through in order to 
American Institutes for Research Cognitive Interview Report for ATES 2016—15 

get it?(If R appears to be reporting same credential as in 

certifications/licenses) How is this related to the certification or license you 

reported earlier? 

Q32. Who gave you your last post-secondary certificate? 

Who gave you your last post-secondary certificate (get name)? 

How easy or difficult was it to answer this question? 

 (If R chooses “another college or university” or “Someplace else” and it’s 

not clear based on the name): Can you tell me more about where you got 

your last post-secondary certificate? 

Q33. About how many hours of instruction did you complete in order to 

earn your last post-secondary certificate?

How did you determine your answer? 

How easy or difficult was it to answer this question? 

Q34: It is not necessary to probe on this question. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q35. To earn your last post-secondary certificate did you have to 

complete... 

Were there any other instructional requirements needed to complete this 

program other than what is listed? 

Q36: It is not necessary to probe on this question. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q37 Is your current job related to your last post-secondary certificate? 

 

 How easy or difficult was it to answer this question? 

Question order changed. 

Q38: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 
American Institutes for Research Cognitive Interview Report for ATES 2016—16 

Work-Based Learning Program 

STOP RESPONDENT AFTER Q49 – Probe on Q39-49 Section name change – 

from Work Experience 

Program 

Q39: Have you ever completed an internship, co-op, practicum, clerkship, 

externship, residency, clinical experience, apprenticeship, or similar 

program? 

(IF YES) Tell me about your program: 

 What job was it for? 

 Could you describe a typical day in the program? 

 What type of organizations were involved in it and what were their 

roles? 

 (If applicable) Was it related to the [certification/license/certificate] 

that you reported earlier? (If yes) How? 

Note to interviewers: Similar to the previous sections, we’d like to find 

out if Rs are reporting something new here or something that they have 

reported previously 

Revised question 

wording. 

Probe on this item 

higher priority now.

Q40. (If applicable) If yes, What type of work was your last work-based 

learning program for? (Mark ONE only.) 

Can you tell me how you chose this category? How easy or hard was 

it to find your answer on that list? 

Were there any categories that are missing? 

[Pay attention to R and if they want to select the category headers that 

don’t have check boxes; probe if they are uncertain of where to mark their 

response]

Q41-42: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q43 a-c. (If applicable) As a part of your last work-based learning 

program did you... 

(If yes to any of a-c): Can you tell me a bit more about the connection 

between your training program and the classes, training, or studying 

you did? 

Question wording 

changed 
American Institutes for Research Cognitive Interview Report for ATES 2016—17 

Q44: It is not necessary to probe on this question. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q45. (If applicable) Which one of the follow best describes your last workbased learning program? 

 

Was it easy or hard to answer this question? 

Please tell me about how you chose your answer for this one? 

New response option 

(3rd & 6th option); 

Changed response order 

Q46. (If applicable) Did (or will) your last work-based learning program 

help you earn a professional certification or license? 

Can you tell me in your own words what that question is asking? 

Q47-Q49: It is not necessary to probe on these questions. Let the 

respondent continue the think aloud and only probe if respondent says 

something that indicates a potential problem, or if there is confusion or an 

inconsistent interpretation compared to other respondents. 

Employment [If short on time these sections are not priority]

STOP RESPONDENT AFTER Q70 – Probe on Q50-70 

Q50-Q67: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents.

Q68. (If applicable) Did you have a license that was required by a 

federal, state, or local government agency to do this job?

 

In your own words, what is this question asking? 

[Check for consistency, if they answered inconsistently between either 

Q10/19/28 and Q68 then probe] You reported in (Q10/Q19/Q28) “yes/no” 
American Institutes for Research Cognitive Interview Report for ATES 2016—18 

but you reported “yes/no” in Q68. Could you tell me the difference between 

these two questions? 

Q69-Q70: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents.

Background [If short of time have them skip to Q82-84 after the Work-Based 

Learning section]

Q71-81: It is not necessary to probe on these questions. Let the respondent 

continue the think aloud and only probe if respondent says something that 

indicates a potential problem, or if there is confusion or an inconsistent 

interpretation compared to other respondents.

Q82. Do you have Internet access on a cell phone? 

Please tell me about how you chose [no, yes] for this one. 

Revised question 

wording (changed 

smartphone to cell 

phone) 

Q83. Do you have Internet access at home on a computer or tablet? 

Please tell me about how you chose [no, yes] for this one. 

[If not obvious from response], Where are you thinking about? [Find out 

where “at home” was interpreted by R to be.]

Q84. How often do you use the Internet? 

Please talk to me about how you got to your answer here. 

[Look for indications that the response options are not inclusive enough/ 

adequate.] 

FINAL QUESTION 

Is there anything else you would like to tell me about this questionnaire that you haven’t had a 

chance to mention? 

(IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS

STEP 4. THANK YOU FOR PARTICIPATING 

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for 

making this survey better. 

Interviewer Instructions: Collect materials and escort them out of the interviewing room and to the 

lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.
 

LOCATIONS

Domestic 

Washington, D.C. 

Atlanta, GA 

Baltimore, MD 

Chapel Hill, NC 

Chicago, IL 

Columbus, OH 

Frederick, MD 

Honolulu, HI 

Indianapolis, IN 

Naperville, IL 

New York, NY 

Rockville, MD 

Sacramento, CA 

San Mateo, CA 

Waltham, MA 

International 

Egypt 

Honduras 

Ivory Coast 

Kyrgyzstan 

Liberia 

Tajikistan 

Zambia 
1000 Thomas Jefferson Street NW

Washington, DC 20007-3835 

202.403.5000 | TTY 877.334.3499 

www.air.org 
Memo

1 

Date: May 20, 2015 

To: Sarah Grady, NCES 

From Danielle Battle and Meghan McQuiggan, AIR 

Re: Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews 

Introduction 

In 2016, the National Household Education Survey (NHES) will field the Parent and Family 

Involvement in Education Survey (PFI), which contains a homeschool component for households 

with homeschooled children. This report summarizes the findings from the supplementary 

homeschool cognitive interviews conducted to test and revise a subset of PFI items based on 

potential issues initially observed in the 2012 collection and the 2014 Feasibility Study. This 

report provides supplemental information to the “2016 Parent and Family Involvement and Early 

Childhood Program Participation Surveys Cognitive Laboratory Final Report” (Cole, Gonzalez, 

Fronberg, & Chavez, 2015). 

In 2014, the National Center for Education Statistics (NCES) conducted a feasibility study using 

the NHES. Sampled households were first sent a screener survey, which asked individuals to 

enumerate the members of their household, as well as answer a few questions about each person. 

The responses to the screener items then determined if someone in the household was eligible for 

one or both topical surveys: the Afterschool Programs and Activities (ASPA) survey and the 

Adult Training and Education (ATES) survey. 

One screener item asked respondents to report the enrollment status of each household member. 

The response options included “public or private school, or preschool”; “homeschool instead of 

school for some or all classes”; “college, university, or vocational school”; and “not in school”. 

Depending on the version of the screener the household received, the homeschool option would 

have been presented either first or second. 

Cognitive interviewing and analysis of the feasibility study data revealed that the concept of 

“homeschooling” may be unclear to some respondents. All households who received the ASPA 

survey had indicated on the screener that the sampled child was enrolled in public or private 

school or preschool. However, in the ASPA topical questionnaire, about 2 percent of sampled 

children were reported to be homeschooled. If respondents selected “homeschool” on the topical 

survey, they were also asked how many hours per week the child attended a public or private 

school. Of those who reported that the child was homeschooled, 34 percent indicated the child 

spent 25 hours or less per week in public or private school, suggesting that these children are 
2 

likely “true homeschoolers.” The other 66 percent, who reported 25 hours or more per week in a 

traditional school setting, likely misinterpreted the meaning of “homeschooling” (Jackson, 

McQuiggan, & Megra, forthcoming). 

Although these findings only apply to a small percentage of NHES respondents, they suggest 

that there is ambiguity over the term “homeschooling,” causing some parents of homeschooled 

children to receive a topical survey intended for parents of enrolled children. As a result, NCES 

decided to conduct a supplemental round of cognitive interviews specifically with parents of 

homeschooled children. In these interviews, parents were asked to complete the PFI survey 

intended for parents of children enrolled in public or private school (PFI-Enrolled) but which 

contained a section about homeschooling. The goal of the interviews was to examine how 

parents of homeschooled children would respond to questions about traditional “brick and 

mortar” school environments, since homeschooling households that misreported on the screener 

might receive this instrument in future NHES administrations. 

Cognitive interview respondents were recruited through personal contacts and Craigslist in 

Washington, DC (4) and an interviewer contact in rural Michigan (1). The homeschooled 

children reported in the topical instruments ranged in age from seven to 18. When asked the 

children’s grade equivalencies, two parents reported second grade, one reported sixth grade, one 

reported ninth grade, and another reported twelfth grade. Three respondents reported that their 

child attended a traditional school before homeschooling and two respondents reported their 

child had always been homeschooled. All respondents indicated that their children were 

homeschooled full-time. Respondents were given a $40 American Express gift card for their 

participation. 

The purpose of this memo is to provide key results from the five supplemental homeschool 

cognitive interviews conducted using the PFI-Enrolled questionnaire. This memo summarizes 

the main issues found, as well as provides recommendations for making these items clearer for 

both parents of homeschooled and enrolled children. All questionnaire changes based on 

interview findings are bolded in the item discussions below. 

This memo highlights six key findings: 

 Screener: In total, respondents were able to navigate the screener questionnaire without 

any problems. When probed on the meaning of “homeschool,” all respondents articulated 

a clear understanding of this concept. Some parents noted that the idea of gradeequivalency may be more difficult for homeschooled children than enrolled children. 

 Traditional school items: Homeschool parents often verbalized that these questions did 

not apply to them. At times, parents left items blank, and at other times, they satisficed by 

answering based on their homeschooling situation. 

 Online instruction items: Multiple homeschool respondents did indicate that their child 

participated in online courses, but they were often unable to find an instruction provider 

among the response options provided that matched their child’s educational experience. 

 Homeschool items: Parents were able to easily answer questions about their child’s 

homeschooling. The items about subjects taught during all homeschooling and taught 
3 

currently were the same for most respondents, depending on the flexibility of the child’s 

curriculum. It may be difficult for parents to report on subjects taught “now” based on 

how the child’s curriculum is delivered across a school year and how the term “now” is 

interpreted. 

 Health, Child’s Background, Household Members, Child’s Family sections: 

Respondents were able to navigate these sections without many problems. 

 Your Household section: This new section worked well, and respondents were able to 

easily answer the new Internet items. All respondents indicated frequent Internet use, 

including for homeschooling purposes, and reported access on smartphones, tablets, and 

computers.

Key Findings 

Screener Instrument 

Overall, respondents encountered few problems with the screener instrument. On question 5, 

which asks about the enrollment status of household members, two respondents initially marked 

“homeschool” when reporting information about themselves. After they realized their mistake 

and corrected their responses to “not in school”, both parents verbalized that they were 

mistakenly thinking about their children when answering. It is likely that this pattern was due to 

a priming effect in which respondents knew they were recruited as homeschool parents, which 

impacted their responses and should not be reflected in the larger data collection. 

After respondents selected “homeschool” for their children’s enrollment status, interviewers 

probed on the meaning of homeschooling, and it was evident that these homeschooling parents 

had a clear understanding of this concept. One respondent said his child is homeschooled 

because “we elect not to send her to a public school or private institution and to educate her 

ourselves”. Another said that his child is homeschooled because he is “enrolled in an accredited 

program that we administer at home...most people consider it homeschool because we are the 

principal educators.” One respondent added that homeschooling is a “global term” encompassing 

a spectrum of arrangements and philosophies. He described how his children are enrolled in an 

accredited and structured homeschooling curriculum, while others, sometimes called 

“unschoolers,” practice more experiential learning. This respondent did note that he is unsure if 

“unschoolers” would categorize their children as “homeschooled,” and suggested adding 

language to clarify that “homeschooling” includes all curriculum-based, distance education, and 

experiential learning methods. 

For all household members enrolled in a school program, respondents were supposed to answer 

question 6, which asked for the person’s current grade equivalent. One respondent noted that this 

question was difficult for him to answer since his child does not progress through traditional 

grades, but said he would write 2nd grade since she was seven years old. This respondent 

suggested using “age appropriate grade” to make this language applicable to homeschooled 

children. Another respondent commented that this could be challenging since some homeschool 

instruction is self-paced, and a child may be on-grade for some subjects, while ahead or behind 
4 

grade-level for others. Overall, these suggestions should result in further testing of this item 

wording and alternative item wording before used in data collection. 

Topical Instrument 

Child’s Schooling 

The first half of this section was challenging for homeschool parents. Multiple respondents 

expressed that these items seemed to apply only to children in public and private schools. For 

example, question 2 asked the type of school the child attends. Three respondents left this item 

blank, saying that no response option applied to their situation. One parent marked “private, nonreligious,” but added that most homeschool parents wouldn’t categorize homeschooling as such. 

Respondents noted that several items in this section did not apply to them, and one respondent 

recommended rewording the items to say “school/ homeschool.” Another parent suggested 

general instructions telling homeschool parents to think of their “home learning environment” 

when reading the questions about “school.” Another respondent mentioned that they would have 

thrown away the survey if they received it in the mail after trying to answer the first few items 

that did not apply to their homeschool child. 

Question 3 on the topical questionnaire asked respondents if the child attended his or her 

“district-assigned” school. Three respondents marked “no” and one left it blank, noting that this 

item did not apply to homeschoolers. One respondent marked “yes,” and when probed, said it 

was because his child is homeschooled in the ward where they live. 

Question 7 asked if parents considered other schools for the child. One parent interpreted 

“schools” as “curriculum,” explaining that they did investigate other homeschooling curriculums 

for the child. 

Question 8 asked respondents if they considered the performance of schools, such as test scores 

and dropout rates, when selecting a school for their child. One respondent explained that 

monetary reasons might be another factor, as he prefers a non-traditional approach to education, 

but found that non-traditional private schools were too expensive. Another respondent added that 

learning outcomes were an important factor in selecting a program, as he wanted a 

homeschooling curriculum that emphasized reading and writing. 

Question 14 asked parents if the child was enrolled in any high-school level Advanced 

Placement courses. All respondents reported “no” for this item, but one respondent noted that 

parents unfamiliar with public schools may not know what AP courses are. Another respondent 

added that homeschool parents may confuse this with community college courses, which many 

upper-level homeschool children take. 

Question 22 asked if the child participates in any online courses or instruction. Two respondents 

reported “no,” saying that their children were too young. Another respondent reported “no,” 

because they did not use the Internet for specific coursework but as a supplement to their 

teaching. One respondent said “yes,” explaining that his family pays tuition for access to a 

distance education school in which the children have access to online courses, transcripts, and an 
5 

advisor. Another marked “yes” and said that her child uses K12, which also links the child to 

other online programs to practice the concepts. 

If respondents indicate that the child is taking online courses, he or she is asked in question 23 to 

report who provides the online instruction. The two respondents both wrote in the “other—

specify” box, and reported “other online homeschooling program” and “our homeschool.” 

NHES: 2012 data showed that homeschoolers often wrote other online schools or virtual 

academies into this box. This suggests that for this item, the response options may not be suitable 

for homeschooled students, requiring additional testing for future administrations. 

Question 25 asked if the sampled child is “schooled at home instead of at school for at least some 

classes or subjects.” All respondents marked “yes” for this item, and two added that they could 

have just skipped to this question as the previous items about traditional schooling did not apply 

to them. It was discussed to move this item to the first page (where the item was located in 

NHES:2012) but NCES decided that it could introduce additional error as those who do 

homeschool would be skipped out of items that could provide additional information about some 

types of homeschool households. 

Question 26 asked parents to mark the statement that best described their child’s homeschooling 

arrangement. All respondents marked “homeschooled for all classes or subject areas,” but one 

parent mentioned that he initially wasn’t sure how to answer since his children participate in coops, which are outside of the home. 

Question 28 lists a variety of reasons parents might homeschool their children, and asked 

respondents to mark if each was a factor in the parent’s decision to homeschool the child. 

Among the respondents, “concerned about the school environment, such as safety, drugs, or 

negative peer pressure,” “dissatisfied with the academic instruction at other schools,” and 

“interested in a nontraditional approach to children’s education” were the most common 

responses. One respondent’s main reason was his child’s deafness, which he categorized as a 

“physical or mental health problem,” and another parent explained that her child was being 

bullied, which she considered “other special needs.” Another respondent reported in the “OtherSpecify” box that one reason not on the list was supervision. The respondent felt that if their 

child was homeschooled, they would be under the respondent’s direct supervision. 

Although no parents selected “yes” for question 28i, “you prefer the flexibility homeschooling 

provides to this child during frequent moves,” two respondents initially just read the first half of 

the item, interpreting it simply as the flexibility homeschooling provides. After one respondent 

read the remainder of the option, he wrote “flexibility” in the “other—specify” box, explaining 

that homeschooling allowed his child to explore other educational and extracurricular 

opportunities. Further analysis was conducted using NHES: 2012 data and it was determined that 

“frequent moves” was not a common write-in response. Therefore, it was recommended that 

item 28i be deleted from the PFI-Enrolled and PFI-Homeschool instruments. The item was 

removed from the NHES:2016 instruments.

For this set of items, one respondent noted that it was difficult to mark “yes” or “no” since some 

were considerations, but not main reasons for homeschooling. He suggested a scale so that 
6 

parents could indicate how much each factor weighed in their decision. Two respondents 

verbalized that some of the factors were benefits they discovered since starting homeschooling, 

but not initial reasons, and they did not mark those options. 

When asked to indicate the main reason for homeschooling in question 29, one respondent 

marked “h—nontraditional approach to education” and explained that they wanted to educate 

their child through a love of learning. Others marked “g—special needs” and “e—physical or 

mental health problem,” explaining their children’s experiences with bullying and deafness, 

respectively. Two other respondents marked “a—concerned about the school environment.” One 

respondent described that after researching schools, he was worried about school security, and 

the other said that the child was enrolled in a public school that the parent felt was unsafe and 

they wanted to ensure the safety of their child and be able to protect them 

Question 30 asked respondents to mark all of the subjects their child had been taught since 

starting homeschooling. Most respondents selected several subjects, but in general, parents of 

younger children did not select upper level mathematics and science courses. When probed on 

how those courses were covered, respondents had a variety of responses. One respondent 

explained that his child attends a curriculum-based co-op, as well as a co-op where parents teach 

courses such as yoga, museum studies, performing arts, etc. Another parent explained that her 

child’s academic subjects are taught online through K12, and the child meets up with other 

homeschoolers weekly for subjects like art, music, and physical education. The third parent 

described a similar situation, where his children complete academic courses through an online 

curriculum, but go to an art studio for art, learn piano from local college students, and go to 

physical education classes with other homeschoolers. Another respondent explained that a 

certified instructor comes to his home daily to deliver the academic curriculum, and the child 

does not participate in activities outside of the home. 

Question 31 asked parents to report on the courses the child is taking “now”. Three respondents 

marked all of the same subjects as in question 30, noting that these two items are the same for 

homeschoolers. However, two respondents selected different subjects for this item based on the 

courses covered this year in the child’s curriculum. One parent questioned if “now” meant in the 

current season or in the academic school year, explaining that the child takes physical education 

in the spring, but not in the winter. Another parent had difficulty with the term “now” and was 

not certain if it meant today or the last few months, and chose to select every subject she taught 

her child because they do not use a strict curriculum. The parent also recommended including an 

item about curriculum instead of this item. 

Respondents were also probed on if any subjects were missing from these lists. Some 

suggestions that parents provided included: performing arts, dance, handwriting, logic, 

religion/spirituality, gardening, and current affairs. One respondent mentioned including a writein box since there are so many topics covered through their homeschooling that do not fall in the 

traditional school curriculum. 
7 

Child’s Health 

Respondents encountered no problems with the “Child’s Health” items. 

Child’s Background 

Overall, respondents had no problems with the “Child’s Background” items. Question 44 asked 

if the child lives “at this address and another address.” One parent initially marked “yes,” reading 

only the first half of the item. After seeing “and another address,” she changed her response, but 

verbalized that parents may answer this item incorrectly if they are rushing through the survey. 

Therefore, it is recommended that this option be revised to “does this child live at this 

address and another address (for example, because of a joint custody arrangement)?” This 

recommended revision was implemented for the NHES:2016 PFI-Enrolled, PFIHomeschool, and ECPP instruments. 

Household Members 

Respondents had no trouble accurately reporting household totals and roster counts. 

Child’s Family 

Overall, respondents easily navigated the questions about the child’s parents. One respondent 

noted that he would prefer if the Parent 1 questions said “are you...” instead of “is this...”. For 

question 60, which asks about employment status, one respondent mentioned that the wording 

“employed for pay or income” was odd, and another questioned why “for pay or income” was 

not included with the “self-employed” option. One respondent had difficulty answering this item 

for his wife, as she is both a stay-at-home parent and employed part-time. 

Your Household 

Due to inaccuracies in reporting household members in NHES: 2012, it was decided that the 

“Your Household” section would be moved ahead of the “Child’s Family” section. However, 

this led to concerns about the income item being too early in the instrument. Additionally, one 

homeschool respondent noted that the internet access items in this section seemed out of place 

since the previous questions were about education. Therefore, it was recommended that the 

“Your Household” section be renamed “Household Members,” and a new “Your 

Household” section follow “Child’s Family,” containing the items about benefits, income, 

home ownership, and Internet access. This recommended revision was implemented for 

the NHES:2016 PFI-Enrolled, PFI-Homeschool, and ECPP instruments. This revised format 

was tested in these supplemental interviews, and respondents had no trouble accurately 

responding to these items.

For Question 99, all respondents interviewed indicated that they had Internet access on a 

smartphone, either through a mobile service provider or Wi-Fi. Four respondents indicated in 

question 100 that they had Internet on a computer or tablet, and several mentioned that their 
8 

children use the Internet on Kindles and iPads. One respondent did not have Internet access 

through a computer or tablet because they live in a rural setting and it is difficult to get Internet 

except through their phones. In question 101, all respondents reported using the Internet “every 

day,” noting that they check their own email daily and use it for their children’s education. One 

respondent added that homeschool parents in particular are usually well-connected, as they use 

the Internet to participate in online homeschooling groups and forums. 

Summary

Respondents in these interviews provided a substantial amount of information regarding their 

interpretations of “homeschooling,” their children’s homeschooling arrangements, methods of 

instruction, and academic coursework. However, respondents did struggle with some schooling 

items, revealing questions that may benefit from additional testing as they are particularly 

challenging for parents of homeschooled children. Despite difficulty with select items, 

respondents were able to answer the majority of questions included on the instrument, resulting 

in few changes to this and other applicable NHES surveys. 

1. Since “frequent moves” was not a common write-in in NHES: 2012, and respondents 

may confuse this option as simply the flexibility of homeschooling, item 28i should be 

deleted from the instrument. This change applies to the PFI-Enrolled and PFIHomeschool instruments. 

2. “And” should be underlined in question 44 to ensure that this item captures children who 

live at multiple addresses as opposed to just one. This change applies to the PFI-Enrolled, 

PFI-Homeschool, and ECPP instruments. 

3. The “Your Household” section should be renamed “Household Members,” and the 

benefits, income, home ownership, and Internet access items should be moved to a new 

“Your Household” section after “Child’s Family”. This change applies to the PFIEnrolled, PFI-Homeschool, and ECPP instruments. 
9 

References 

Cole, S., Gonzalez, R., Fronberg, K., & Chavez, S. (2015). 2016 Parent Family Involvement in 

Education and Early Childhood Program Participation Surveys Cognitive Laboratory Final 

Report. Washington, DC: U.S. Department of Education, National Center for Education 

Statistics.

Jackson, M., McQuiggan, M., and Megra, M. (forthcoming). NHES:2014 Feasibility Study 

Report. Washington, DC: U.S. Department of Education, National Center for Education 

Statistics. 
10 

Appendix: Questionnaire and Interview Protocol 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 

2016 Parent and Family Involvement in Education Survey (PFI) 

Cognitive Interview Protocol (for parents of homeschooled children) 

Interview ID: Date of 

Interview: 

Parent First 

Name: 

 Interviewer 

Age of Child: 

STEP 1: WELCOME AND INTRODUCTION SCRIPT 

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is 

suggested content with which the interviewer should be thoroughly familiar in advance. Text in 

brackets [ ] presents instructions for the interviewer. 

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet 

you, thanks for coming to help us out today. 

[Create small talk to build rapport with the parent]: 

You are here today to help us out with a new survey that we are working on for the U.S. Department 

of Education. This survey asks about your child’s schooling. I will ask you to take the survey as if you 

were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some 

of the answers. As you are answering the questions I will ask you about your answers to the 

questions. This will help us make the survey better and clearer for other parents just like you. 

Because the information you provide is so important, I am going to be taking notes while you are 

answering the survey and while we are talking. 

 
29 

Also, this interview will be audio-recorded so researchers can review the recording later. Any 

information that refers to you or your family personally, like your name or the name of your child, 

will not be included in our report, and your responses will be used for research purposes only. 

Remember, you are not being tested; we want you to tell us what you think about the survey. 

If at any time you want to stop, just let me know. 

[If, for any reason, the participant is no longer interested in participating, thank the participant for 

his/her time and end the interview.] 

Before we continue, do you have any questions about what I just said? [Answer any questions the 

parent may ask.] 

IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER 

WITH A COPY OF THE FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ 

IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY 

QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE 

COPY OF THE FORM WITH THE RESPONDENT. 

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER. 

STEP 2: THINK-ALOUD PRACTICE 

While you fill out the survey I want you to talk about how you are answering the questions. Hearing 

you talk about how you figure out your answers to the questions will help me understand how to 

make the questions better. I am more interested in how you arrive at your answers than in the 

answers you give me. To help you get started on thinking aloud, we are going to do a practice 

activity. 

First, I will show you an example of how to think out loud. Then I will give you a chance to practice 

it. 

QUESTION: How many glasses of water did you drink yesterday? 

[Model the think aloud process for the parent, including reading the question out loud. Elaborate on 

counting all of the glasses of water and as needed you can make up pieces of information to 

emphasize the think aloud process. 

[Once finished the model practice think aloud] 

I’m going to give you a piece of paper with the question on it. I’m going to ask you to read out loud 

the question and think aloud as you decide on your answer. 

Give respondent handout with think aloud practice. 

How many glasses of water did you drink yesterday? 

Please tell me what you are thinking while you are figuring out your answer. 
30 

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when 

you were deciding on that answer? Can you tell me more about what you are including in your 

count? 

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about 

your answers to the questions in the survey. 

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL. 

STEP 3: COMPLETION OF THE QUESTIONNAIRE 

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on 

your answer. Please read aloud anything you would have read to yourself if I were not here. 

Remember that I am very interested in hearing you talk about what you are thinking as you figure out 

your answers to the questions. 

REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT 

THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO 

YOUR QUESTIONS ARE REALLY HELPFUL. 

 
31 

Useful Probes 

Can you tell me in your own words what that question was asking? 

What does the word [term] in this question mean to you? 

How easy or difficult is it to remember [topic]? 

How easy or hard was it to choose an answer? 

Tell me what you are thinking? 

How did you come up with that answer? 

You said [answer]. Can you tell me more about that? 

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing? 

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are 

thinking.” 

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old 

question before answering this question]: “Tell me what just happened. Tell me about what you 

were just doing.” 

Screener 

ASK THE RESPONDENT TO FILL OUT 

THE ENTIRE SCREENER. REMIND THEM 

TO THINK ALOUD IF THEY ARE NOT 

DOING SO. 

Q3. What is the person’s month and year of 

birth?

[If the respondent is having difficulty with the 

month and/or year] How did you come up with 

your answer? 

Q5. Is this person currently in...?

How did you come up with your answer? 

[If respondent chooses “homeschool”] What 

does “homeschool” mean to you? 

PFI Survey 

CHILD SELECTION: SELECT THE FIRST CHILD AGES 5-17 ON THE SCREENER 

THE R REPORTED TO BE CURRENTLY HOMESCHOOLED. 

Child’s Schooling

 

ASK THE RESPONDENT TO FILL OUT 

THE SURVEY ABOUT THE CHILD 

SELECTED. ASK QUESTIONS AND USE 

PROBES ONLY AS NECESSARY; REMIND 

RESPONDENT TO THINK ALOUD 
32 

Instructions: It is not necessary to probe on 

the instructions. Note if the respondent has any 

problems if they read them. 

Q1-24: Pay attention to how the R reacts to 

these items and take notes and probe 

accordingly. Since the child is homeschooled 

these items may not apply. If R is having 

difficulties remind the R to fill out the 

survey as they would if they received it at 

home. 

[Probe if necessary] How would you react if 

you were to receive this survey in the mail at 

home?

Q1-Q2: It is not necessary to probe on these 

questions. Let the respondent continue the 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q3. Is it his/her district-assigned school? A 

district-assigned school is the school that your 

local public school district told you that this 

child can attend, based on the location of your 

residence.

[Ask only if responded to this question] How 

did you decide on your answer? 

What does “district assigned” mean to you? 

How did you know what school your child was 

going to attend? How do other parents in your 

child’s district know which schools their child 

will attend? 

Q4-Q13: It is not necessary to probe on these 

questions. Let the respondent continue the 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q14. Is he/she currently enrolled in any high 

school Advanced Placement (AP) classes?

What were you thinking of when you answered 

this question? 

What do “Advanced Placement (AP) classes” 
33 

mean to you? 

What grade-level are the students who usually 

take AP classes in your child’s school district? 

Q15-Q21: It is not necessary to probe on these 

questions. Let the respondent continue the 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q22. Is this child taking any school-related 

courses online instead of in-person with the 

teacher? Do not include courses that use the 

Internet only for selected assignments.

How did you decide on your answer? 

Q23. Is that instruction [internet] provided by 

any of the following places? 

 

How did you decide on your answer? 

[If chooses a response option or “Someplace 

else:”] Tell me more about that. How does 

your child access the classes? Where is he/she 

when accessing the classes? Who provides the 

class? 

Q24: It is not necessary to probe on this 

question. Let the respondent continue the think 

aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q25. Some parents decide to educate their 

children at home rather than send them to a 

public or private school. Is this child being 

schooled at home instead of at school for at 

least some classes or subjects?

In your own words, tell me what this question 

is asking about. 

How did you determine whether to answer yes 

or no? 

[If respond “yes,”] Do not probe on the 

meaning of homeschool until Q29. 
34 

[If respond “no” and if not evident from think 

aloud to previous questions] What does 

“homeschool” mean to you? 

Make a note if the respondent did not skip 

properly, but do not probe on skip pattern.] 

Q26. Which of the following statements best 

describes your homeschooling arrangement 

for this child?

 

How did you come up with this answer? 

[Do not probe on the meaning of homeschool 

until Q29. Check to see if respondent skipped 

properly, but do not probe on skip pattern.]

Q27. How many hours each week does this 

child usually go to a public or private school 

for instruction? Do not include time spent in 

extracurricular activities.

How did you come up with your answer? 

[Check to see if respondent filled in this 

question although they indicated that the child 

is homeschooled.] 

Q28. There are many different reasons that 

parents choose to homeschool their children. 

Did your family choose to homeschool this 

child because:

 

How did you determine how to answer yes or 

no to each of these questions [a-j]? 

Tell me more what you were thinking when 

answering these questions. 

[Do not probe on the meaning of homeschool 

until Q29.] 

Q29. Of the reasons your family chose to 

homeschool this child, which one would you 

say is the most important to you?

How did you come up with your answer? 

[If not evident from think aloud to previous 

questions] What does “homeschool” mean to 

you? 
35 

Q30. Thinking about all years this child has 

been homeschooled, which of the following 

subject areas has this child been taught 

during his or her home instruction?

How did you decide what to mark? 

How do you cover these subjects during home 

instruction? 

Is there anything missing? 

Q31. Which of the following subject areas are 

being taught to this child now? 

How did you come up with your answer? Can 

you tell me more about what the child is being 

taught now? 

Child’s Health 

 

REMIND RESPONDENT TO THINK 

ALOUD 

Q32-Q35: It is not necessary to probe on 

these questions. Let the respondent continue 

the think aloud and only probe if respondent 

says something that indicates a potential 

problem, or if there is confusion or an 

inconsistent interpretation compared to other 

respondents. 

Q36. Are these services provided by any of the 

following sources? 

How did you decide on your answer? 

Q37. Are any of these services provided 

through an Individualized Education Plan 

(IEP) or services plan? 

How did you come up with your answer? 

What does a “services plan” mean to you? 
36 

Child’s Background 

Q38-Q43: It is not necessary to probe on these 

questions. Let the respondent continue the 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Q44. Does this child live at this address and 

another address (for example, because of a 

joint custody arrangement)? 

How did you come up with your answer? 

In your own words, what is this question 

asking? 

[If answered No] What does joint custody 

arrangement mean to you?

Q45. If yes, does this child spend most of the 

time at this address, at another address or 

equal time at both addresses? 

How did you come up with your answer? 

[If needed] Can you describe the arrangement 

your child has? 

Q46. What language does this child speak 

most at home? 

It is not necessary to probe on this question.

Note the respondent’s response, especially for 

those who are parents of ELL students. Record 

in your notes whether the respondent reads the 

response options below the top two responses. 

Q47. It is not necessary to probe on this 

question. Let the respondent continue the think 

aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Your Household 

Q48. How many people live in this 

household?

How did you come up with your answer? 
37 

Who are you including in the counts?

Q49. How many of the following people live 

in this household with this child?

 

Tell me more about what you were thinking 

when you answered this question? 

Who are you including in the counts? 

In your own words, what is this question 

asking? 

Q50-Q57: It is not necessary to probe on these 

questions. Let the respondent continue the 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents. 

Pay particular attention Q53, noting if the 

respondent seems uncomfortable answering the 

income item. 

Q58a. Do you have Internet access on a 

smartphone? 

Please tell me about how you chose [no, yes] 

for this one. 

Q58b. Do you have Internet access at home 

on a computer or tablet? 

 

Please tell me about how you chose [no, yes] 

for this one. 

[If not obvious from response], Where are you 

thinking about? [Find out where “at home” was 

interpreted by R to be.] 

Q59. How often do you use the Internet? 

Please talk to me about how you got to your 

answer here. 

[Look for indications that the response options 

are not inclusive enough/ adequate.] 

Child’s Family 

 

REMIND RESPONDENT TO THINK 

ALOUD 

Q60-Q71: It is not necessary to probe on these 

questions. Let the respondent continue the 
38 

think aloud and only probe if respondent says 

something that indicates a potential problem, or 

if there is confusion or an inconsistent 

interpretation compared to other respondents.

Q72. What is the highest grade or level of 

school that this parent or guardian 

completed?

[Only if respondent chooses “high school 

diploma or equivalent”, “vocational diploma 

after high school”, or “some college, but no 

degree”:] 

How did you come up with your answer? 

What does “vocational diploma after high 

school” mean to you? 

Q73. Is he or she currently attending or 

enrolled in a school, college, university, or 

adult learning center, or receiving vocational 

education or job training?

In your own words, what is this question 

asking? 

Q74-Q100: It is not necessary to probe on 

these questions. Let the respondent continue 

the think aloud and only probe if respondent 

says something that indicates a potential 

problem, or if there is confusion or an 

inconsistent interpretation compared to other 

respondents.
39 

FINAL QUESTION 

Do you have anything else you would like to tell me about this questionnaire that you haven’t had a 

chance to mention? 

STEP 4. THANK YOU FOR PARTICIPATING 

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for 

making this survey better. 

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to 

the lobby. Please give the respondent incentive envelope [$40 gift card], have the respondent sign the 

receipt, and thank them for their time and effort.
Memo

1 

Date: June 3, 2015 

To: Sharon Boivin and Sarah Grady, NCES 

From Anna Sandoval Giron and Meghan McQuiggan, AIR 

Re: Preliminary Findings from the 2015 Spanish Cognitive Interviews 

Introduction 

In 2016, the National Center for Education Statistics (NCES) will conduct the second full-scale, 

mail-based administration of the National Household Education Survey (NHES). The NHES 

uses a two-stage design in which sampled households are first sent a screener survey and asked 

to list the members of their household, as well as provide some key information on each 

individual, including their date of birth, sex, and school enrollment status. The responses from 

the screener survey determine if any household members are eligible to receive one of the NHES 

topical instruments, which ask additional questions about the sampled individual’s education. 

In 2016, the NHES will contain three topical surveys: the Early Childhood Program Participation 

(ECPP) survey, the Parent and Family Involvement in Education (PFI) survey (Enrolled and 

Homeschool), and the Adult Training and Education Survey (ATES). All contact materials, 

screener instruments, and topical instruments are administered in both English and Spanish. 

Additionally, the NHES:2016 will contain a web experiment in which some respondents will be 

asked to complete the screener and topical surveys online. 

In order to reduce measurement error and ensure high-quality translations, all Spanish materials 

are being tested using cognitive interview methods. Spanish interviews are being conducted in 

three phases. Phase 1 tested the Spanish screener and topical letters, postcard reminder, screener 

questionnaire, and new ATES items that had not previously been translated. Phase 2 tested the 

topical questionnaires, including the ECPP, PFI- Enrolled, and ATES. These instruments 

contained translations from the 2012 administration, as well as new translations due to item 

revisions after the English cognitive interviews. Phase 3 will test the contact materials for the 

web survey, however those findings are not yet available. 

Phase 2 and 3 were ongoing but mostly complete, at the time this memo was written The memo 

documents all findings to date from the Spanish interviews, and no further substantive changes 

are anticipated. 
2 

Sample 

Phase 1 interviews took place in Washington, DC, San Mateo, CA, and Houston, TX with 26 

respondents who are Spanish dominant speakers (they either speak only Spanish or are most 

comfortable speaking Spanish). Phase 1 interviews were conducted between March and April 

2015. 

Phase 2 interviews took place in Washington, DC, San Mateo, CA, Houston, TX, and Tucson, 

AZ with 82 respondents (to date) who are Spanish dominant; – 20 on ECPP, 38 on PFI-Enrolled, 

and 24 on ATES. Phase 2 interviews were conducted between April and May 2015. 

Key Findings: Phase 1 

Screener and Topical Letters 

General Issues 

In the first round, respondents had difficulty with the names of government agencies, as 

translating these organizations often led to sentences that were quite lengthy. For example, the 

opening sentence from the screener letter, including the Spanish version, is shown below: 

I am pleased to inform you that the U.S. Census Bureau has selected your household to 

participate in the 2015 National Household Education Survey, which we are conducting on 

behalf of the U.S. Department of Education. 

 

Me place informarle que la Oficina del Censo de los Estados Unidos ha escogido su hogar para 

que participe en la 2015 Encuesta Nacional en Hogares sobre Educación, la cual estamos 

realizando para el Departamento de Educación de los Estados Unidos. 

Respondents often skipped these sentences or did not understand the information. When asked 

who sponsored the study, six of the 20 respondents understood that it was the U.S. Department 

of Education. Four respondents said “Census,” and others reported “Department of Commerce,” 

“the government,” “teachers,” and “the schools.” Three respondents said they did not know who 

was sponsoring the study. 

Based on these results, it was recommended that the opening paragraph of all letters be 

simplified so that the language is easier for readers to understand. For round 2, this wording was 

changed to the following: 

I am pleased to inform you that your household has been selected to participate in the 2015 

National Household Education Survey. This is a U.S. Department of Education survey that is 

being conducted by the U.S. Census Bureau. 
3 

Me complace informarle que su hogar ha sido escogido para participar en la Encuesta Nacional 

en Hogares sobre Educación del 2015. Esta es una encuesta del departamento de Educación

realizada por la Oficina del Censo de los Estados Unidos.

After the wording was changed, six of the seven respondents in Round 2 interviews reported that 

the study’s sponsor was the “Department of Education,” “Census Bureau,” or “the government.” 

Therefore, it is believed that simplifying this language made it clearer to respondents that the 

study was sponsored by a government entity. 

At the request of NCES, the term “conducted” was changed to “administered” in the English 

version, but the Spanish translation remains the same. 

Translation Issues 

Both the screener and topical letters include the phrase “token of appreciation,” which is 

translated as “dinero en efectivo.” In round 1, 19 of 20 respondents believed that this meant 

“cash.” For round 2, the phrase was re-translated as “muestra de aprecio.” No respondents in 

round 2 had any difficulty with this language, and it is believed that this translation better 

captures the concept of “token of appreciation.” 

The letters also contained the phrase propósitos estadísticos, “statistical purposes.” Several 

respondents did not understand this wording, suggesting the need for an improved translation. 

For round 2, the wording was changed to fines estadísticos in an attempt to make the text shorter 

and more readable. 

The ECPP topical letter specifies that the study provides information about educación de niños a 

temprana edad, “children’s early learning,” however, in the questionnaires, this was translated 

as educación a niños pequeños, “education for small children.” In order to be consistent, it was 

recommended after round 2 that all references to children’s early learning match the language 

used in the questionnaires. 

The ATES topical letter specifies that the survey is about adiestramientos y credenciales que los 

adultos reciben para trabajar, “training and credentials that adults get for work.” Several 

respondents reported that the word adiestramientos, “training,” seems to refer to animal training, 

and instead suggested using the word entrenamiento. Once this revision was made for round 2, 

respondents had no issues with this language. 

At the request of NCES, some references to “work” were dropped from both the English and 

Spanish letters to place less emphasis on employment because respondents are not required to be 

working to be eligible for the survey. 

 
4 

Screener Questionnaire 

General Issues 

Question 5 on the screener asks respondents for the individual’s school enrollment status. For 

people not in school, the response option provided is a word-for-word translation, no está en la 

escuela, “not in school.” In round 1, respondents had trouble with this item, often looking for a 

response option that said ninguna, “none,” as is used in many other Spanish surveys. For round 

2, ninguna was used in place of no está en la escuela, and respondents who were not in any of 

the school categories had less trouble finding an applicable response. 

Respondents also had trouble with the homeschooling response option in question 5. This option, 

in English and Spanish, is below: 

Homeschool instead of attending a public or private school for some or all classes. 

Escuela en el hogar (homeschool) en lugar de asistir a una escuela pública o privada para 

algunas o todas las clases. 

The concept of homeschooling confused multiple respondents. Generally, they either thought 

this was referring to a parent supporting a child’s education at home, or an adult using 

educational materials for their own self-improvement purposes. Due to the perceived low 

prevalence of homeschooling among Spanish-speaking families, this is believed to be a 

conceptual issue because Spanish respondents simply are unfamiliar with homeschooling. 

Therefore, it was recommended that any translation revisions should only be made to address 

substantial changes in the English homeschooling questions, which were tested separately. 

Respondents also had difficulty understanding question 6, which asks for each individual’s 

current grade equivalent, or “grado actual de esta persona, o su equivalente.” Some respondents 

in cognitive testing interpreted this as meaning the highest level of education the person had 

completed, particularly for individuals not enrolled in school. However, no changes were made. 

Key Findings: Phase 2 

ECPP, PFI, and ATES 

General Issues 

Overall, respondents across the three topical surveys had difficulty following skip patterns. In 

round 1, eight of 10 respondents did not skip to other sections or questions when they should 

have. When further probed, two respondents commented that in other “official” questionnaires 

they had been told to fill in all possible information, so they chose to complete as many questions 

as possible, disregarding skip instructions. 
5 

Across all surveys, respondents also found questions related to race challenging. In the first 

round, four of 10 respondents commented that they could not find an adequate response option 

(Hispanic or Latino) among those provided. Some eventually chose White, commenting that that 

is what they are “supposed” to choose. 

No changes were made to skip instructions in the questionnaire since respondents were making 

the more desirable reporting error of commission instead of error of omission, however, NCES 

intends to implement post-survey edits for skip patterns which will take into account this 

cognitive finding. 

PFI and ECPP 

General Issues 

These two surveys contain a question asking if “this child is currently enrolled in English as a 

second language, bilingual education or an English immersion program?” Three parents 

expressed that they thought this was asking if the teacher could communicate with the child in 

Spanish. It is believed that this is both a translation issue, and a conceptual one in that parents are 

unfamiliar with “immersion programs.” In Round 2 we discovered that the issue was the 

translation of immersion programs. The original translation of the question used programa 

especial de inglés to refer to immersion programs. In addition the original translation used the 

term lengua, “tongue” to refer to language. We determined these two terms/phrases were causing 

confusion to respondents and suggested a new translation of the question that reflects the 

language commonly used by schools and parents. NCES implemented this recommendation. 

New suggested translation: Actualmente, ¿asiste este niño(a) a clases de inglés como segundo 

idioma, educación bilingüe o a un programa de inmersión al inglés? 

Translation Issues 

These topical instruments also ask parents to report any of the sampled child’s health conditions. 

Although two respondents were confused by this section and chose to leave it blank, the others 

had no problems. However, two terms in this section proved challenging for respondents. The 

phrase impedimiento del habla, “language impediment,” was marked with a yes by some parents. 

During probing these parents explained that they chose yes for this option because their children 

are learning to talk in two languages and have not mastered one of the languages. NCES has 

decided not to make any changes to the survey based on this finding. 

Another respondent had difficulty with the term, Trastorno generalizado del desarrollo (PPD), 

“Pervasive Developmental Disorder (PDD).” AIR researchers have followed-up on this 

translation and reviewed the wording in the U.S. Department of Health and Human Services. The 

phrase Trastorno generalizado del desarrollo (PPD) is the commonly used translation to refer to 

these disorders. Currently we believe that parents that have a child diagnosed with Pervasive 

Developmental Disorder will be able to identify themselves and correctly answer the question. 

No change is recommended. 
6 

ECPP 

General Issues 

In round 2, parents with children in home-based daycare had difficulty categorizing this type of 

care. When asked if the child receives home-based care from a non-relative, parents of children 

in home-based daycares often responded “yes.” However, when respondents got to the series of 

questions about daycares, preschool, and prekindergarten programs not in private homes, they 

wanted to report the programs in this section as well, and it seemed that respondents were not 

reading the “not in a private home” text. Overall, four of seven respondents in this round had 

response error in this set of questions. No change was recommended. 

ATES 

General Issues 

Similar to the English ATES interviews, respondents had difficulty distinguishing between 

certifications, certificates, and licenses. 

In the second round, it was discovered that respondents with education and licensing from 

foreign organizations and governments often had difficulty categorizing their credentials. One 

respondent explained that in her country, her high school education is considered two separate 

certificates- she can earn both a high school degree and technical career degree. It is believed 

that this issue may not be limited to Spanish speakers, but to many immigrant groups. 

Translation Issues 

The ATES questionnaire asks respondents if their most important certification or license can be 

revoked or suspended for any reason. “Revoke” is currently translated as “revocada,” but the 

term revocada produced confusion for some respondents. Due to the confusion around this term, 

it was recommended that this language be revised to quitar, which, is a synonym of revocada

meaning means “remove” or “take away.” .The current and recommended translations are below. 

English version: Can your most important certification or license be revoked or suspended for 

any reason? 

Current translation: Su certificación o licencia más importante, ¿podría ser revocada o 

suspendida por alguna razón? 

Suggested translation: ¿Le podrían quitar o suspender su certificación o licencia más 

importante por alguna razón? 
7 

Summary 

Overall, these two phases of cognitive interviews provided a lot of insight into Spanish speakers’ 

interpretation and understanding of NHES contact materials and questionnaires. Respondents 

were able to highlight areas of confusion and problematic language, which led to improved 

wording and translations. For example, in the screener and topical letters, introductory text 

containing government agency names was simplified to make it more readable. In the screener 

questionnaire, the response option for individuals not in school was revised to reflect a response 

option that Spanish speakers are more familiar with. Testing of the screener questionnaire also 

reinforced that homeschooling continues to be a challenging concept for Spanish speakers. 

Cognitive testing also revealed concepts and language that needed to be revised on the topical 

questionnaires. While some translation changes were made during testing, others require 

additional testing and investigation before revisions can be made. Across all of the instruments, 

some of the items that were difficult included Spanish immersion programs, child language 

impediments, school detentions and suspensions, home-based daycare programs, and the 

revocation of adult credentials. The final analysis of the interview data, as well as future 

cognitive interviews, will continue to inform the development and refinement of Spanish NHES 

surveys. 
8 

Appendix A: Phase 1 and Phase 2 Protocols 
Phase 1 Protocol 

9 

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. 

Also, keep track of the time so that your total interview time with the respondent lasts 1 hour or 

less. 

Before starting the interview, use this table to confirm the participant info that we received from 

the recruitment firm. If the respondent indicates that some of the listed information is incorrect, 

please cross it out and enter the correct information in the relevant cell.* 

Background Information Confirmed



Date & Time 

Name of Interviewer 

Parent/Respondent First Name 

Respondent’s Gender  Male  Female 

Respondent’s Age 

If parent fill following questions: 

Child Enrolled in School  Yes  No 

Name of School (only if R is willing to share) 

Type of School  Public  Charter  Private  Homeschool 

Student’s Grade 

Age of Student 

Assigned ID 

*This cover sheet should be kept separate from the rest of the data and 

be turned in with the rest of the paperwork. This cover sheet should not 

be copied, shared, or left in an unlocked office. 

  

   
10 

Documento Adjunto 3

Protocolo en Español para las Entrevistas Cognitivas  

MATERIALS NEEDED FOR INTERVIEW 

□ INTERVIEWER PROTOCOL 

□ RESPONDENT LETTERS AND QUESTIONNAIRE 

□ DO YOU HAVE THE CORRECT VERSION OF THE LETTERS, PROTOCOL, 

AND QUESTIONNAIRE? 

□ CONSENT FORM (TWO COPIES) 

□ INCENTIVE 

□ PAYMENT RECEIPT 

□ DIGITAL RECORDER AND EXTRA BATTERIES 

□ PENS AND PENCILS 

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is 

suggested content with which the interviewer should be thoroughly familiar in advance. Text in 

brackets [ ] are instructions for the interviewer. 

Hola, mi nombre es <su nombre> y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. 

Gracias por venir a ayudarnos el día de hoy. 

[Platique de algún tema casual para establecer una buena relación con el participante.]: 

Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos 

realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta 

acerca de [la educación de sus hijos, la búsqueda y selección de cuidado infantil para los padres 

con hijos de cinco años de edad o menores, o acerca de sus certificados de estudios]. 

Este estudio es sobre [la educación de su hijo(a), cómo buscaron y escogieron el para su hijo(a) 

para aquellos padres con niños de cinco años o menores, y sobre sus certificados de educación. 

Como parte de este estudio, le voy a pedir que lea algunas cartas y que me diga lo que piensa 

que estas cartas le están pidiendo que haga. Además, voy a pedirle que tome una breve encuesta 

como que si estuviera tomando una encuesta real. No existen respuestas correctas o incorrectas 

y está bien si usted no sabe algunas de las respuestas. A medida que usted vaya leyendo las 

cartas y respondiendo a las preguntas, yo le voy a hacer algunas preguntas acerca de sus 

respuestas. Esto nos ayudará a que las cartas y la encuesta sean mejores y más claras. 

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas 

mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va 

a observar nuestra sesión de hoy]. 
11 

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan 

revisar la grabación más tarde. Cualquier información que se refiera a usted o su familia 

personalmente, tal como su nombre o la escuela de su hijo(a), no será incluida en nuestro 

informe y sus respuestas serán utilizadas para fines de investigación únicamente. 

Recuerde que usted no está tomando un examen. Queremos que nos diga lo que usted piensa 

acerca de las cartas y la encuesta. 

Si en algún momento usted desea detenerse, solamente déjemelo saber. 

[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y 

termine la entrevista.] 

Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a 

cualquier pregunta que pueda hacer el participante. 

PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE 

RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON 

RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT 

SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT. 

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER. 

STEP 2: SCREENER LETTER

Ahora me gustaría que complete esta actividad como que si estuviera sucediendo en su casa. 

Voy a darle este sobre y quiero que usted haga lo que usted haría normalmente si usted 

recibiera y abriera estos materiales en su casa. 

Voy a salir del cuarto por aproximadamente cinco minutos para dejar que usted lea los 

materiales y luego regresaré para que podamos hablar al respecto. 

[ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL 

CUARTO/AREA.] 

[Cuando el entrevistado termine:] 

Gracias por tomarse el tiempo para leer estos materiales. 

Screener letter 

Ahora tengo unas cuantas preguntas para usted, pero me gustaría que las 

responda sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias. 

Me gustaría preguntar acerca de su reacción a la carta. Por favor, recuerde 

que no hay respuestas correctas o incorrectas, sólo estamos tratando de saber 

esta carta puede ser interpretada. 

PREGUNTE: 

 En general, ¿Quién manda esta carta? 

 ¿Qué se espera que haga usted después de leer la carta? 
12 

 ¿Para qué se va a usar la información recolectada por este estudio? 

Gracias. 

Ahora quiero preguntarle acerca de los detalles de la carta. Voy a devolverle la carta y podemos 

hablar más acerca de ella. 

[Anote cualquier lugar en el que el entrevistado dude al leer, secciones que se salte el 

entrevistado, palabras que el entrevistado pronuncie incorrectamente o cualquier cosa acerca de 

la que haga comentarios el entrevistado.] 

Quién 

Por favor, lea en voz alta el primer párrafo de la carta. 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Después de leer este párrafo, puede decirme quien le está enviando 

esta carta? 

 ¿Puede decirme quién cree usted que es el patrocinador de esta 

encuesta? 

 ¿Acerca de que cree usted que se trata esta encuesta? En términos 

generales, usted no tiene que darme detalles específicos, solo lo que 

usted piensa acerca de lo que leyó en este párrafo. 

Qué 

Continuemos. Por favor, ¿puede leer el segundo párrafo? 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 Hablemos acerca de la primera oración acerca de que su hogar fue 

elegido al azar. En sus propias palabras, ¿qué quiere decir esto? 

 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles? 

Cómo 

Continuemos. Por favor, ¿puede leer la siguiente oración y los puntos? 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Qué dice esto que se le enviará en la siguiente semana? 

 Usamos el término “muestra de aprecio.” ¿Qué significa este término 

para usted? 

Privacidad 

Continuemos y hablemos acerca de este texto en el cuadro. Por favor, ¿Puede 

leerlo en voz alta? 
13 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Hay algunas palabras aquí que no le sean familiares? 

 ¿Qué cree que está tratando de decir este párrafo? 

STEP 3: THINK-ALOUD PRACTICE 

Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las 

preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así 

entender cómo mejorar estos materiales. Me interesa más saber cómo llega a sus respuestas que 

en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer 

un ejercicio de práctica. 

Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una 

oportunidad para que practique. 

[Model the think aloud process for the respondent, including reading the question out loud. 

Elaborate on counting all of the glasses of water as needed you can make up pieces of 

information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee 

I drank in the morning, I drink three cups of coffee, I am going to count those since that is the 

only liquids I have in the morning...” When you are modeling, write down the number of glasses 

of water as you are thinking aloud.] 

[Once finished the model practice think aloud]

Voy a darle un papel con una pregunta. Le voy a pedir que lea en voz alta la pregunta y que 

piense en voz alta mientras decide su respuesta. 

[Give respondent handout with think aloud practice.] 

¿Cuántos vasos de agua tomó usted ayer? 

Por favor dígame lo que está pensando mientras me da su respuesta. 

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta? 

¿Qué estaba visualizando mientras estaba decidiendo que responder? ¿Puede decirme más 

acerca de lo que usted está incluyendo en su conteo? 

Cuando haya completado el ejercicio: Gracias. Ese es el tipo de detalle que busco cuando usted 

me hable acerca de sus respuestas a las preguntas de la encuesta. 

Preguntas de seguimiento (probes) En sus propias palabras, ¿puede decirme lo que dice esa 

pregunta? 

¿Qué significa la palabra [término] en esta pregunta para usted? 

¿Qué tan fácil o difícil es recordar [tema]? 
14 

¿Qué tan fácil o difícil es escoger una respuesta? 

Dígame lo que está pensando. 

¿Cómo llegó a esa respuesta? 

Usted dijo [respuesta]. ¿Puede decirme más acerca de esto? 

[En respuesta a las acciones o comentarios del entrevistado]: Dígame que pasó. Dígame que 

estaba haciendo. 

[Si nota alguna señal visual, como cejas fruncidas o una expresión de confusión]: “Dígame que 

está pensando.” 

[Si nota alguna señal visual que indique confusión como voltear las páginas para regresar a 

secciones anteriores o volver a leer preguntas anteriores antes de responder a esta pregunta: 

Dígame qué es lo que acaba de pasar. Dígame acerca de lo que acaba de hacer.” 

STEP 4: SCREENER QUESTIONNAIRE

Ahora voy a darle una breve encuesta. Me gustaría que trate de completar las preguntas como si 

lo estuviera haciendo en casa, excepto que quiero que piense en voz alta mientras decide su 

respuesta, así como lo hicimos durante el ejercicio. Por favor lea en voz alta cualquier cosa que 

usted hubiera leído para sí misma, como si yo no estuviera aquí. 

Recuerde que estoy muy interesada/o en escucharle hablar acerca de lo que está pensando 

mientras decide las respuestas para las preguntas. 

RECUERDE DECIRLE AL ENTREVISTA DURANTE LA ENTREVISTA QUE ESTA 

HACIENDO UN BUEN TRABAJO AL HABLAR EN VOZ ALTA, Y QUE SUS 

RESPUESTAS A LAS PREGUNTAS SON MU UTILES. 

Screener 

POR FAVOR LLENE TODA LA ENCUESTA Y PIENSE EN VOZ ALTA 

COMO HICIMOS EN EL EJERCICIO. 

Q3. ¿Cuál es el mes y año de nacimiento de la persona?

[Si el participante tiene dificultad con el mes yo año] ¿Cómo llegó a esta 

respuesta? 

Q5. ¿Asiste esta persona actualmente a. . .? 

¿Cómo llegó a esta respuesta? 

[Si no es evidente por lo que dijo al pensar en voz alta al responder preguntas 

previas] ¿Qué quiere decir para usted “Escuela en el hogar (homeschool) en 

lugar de asistir a una escuela pública o privada para algunas o todas las clases? 

 
15 

STEP 5: TOPICAL LETTERS

[PARA PADRES DE FAMILIA O TUTORES: LLENE EL NOMBRE DEL PRIMER NIÑO/A 

LISTADO EN EL CUESTIONARIO. 

PARA ADULTOS: LLENE EL NOMBRE DEL PRIMER ADULTO LISTADO EN LA 

ENCUESTA.] 

Ahora tengo otra carta para usted. Por favor tome unos minutos para leerla.

[ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL 

CUARTO/AREA.] 

[Cuando el entrevistado termine:] 

Gracias por tomarse el tiempo para leer estos materiales. 

Topical Letters 

Al igual que antes, tengo unas cuantas preguntas para usted, pero me gustaría que las responda 

sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias. 

Ahora me gustaría preguntarle acerca de su reacción a la carta. Por favor, recuerde que no hay 

respuestas correctas o incorrectas. Sólo estamos tratando de entender cómo usted interpreta la 

carta. 

PREGUNTAR: 

 En general, ¿quién manda esta carta? 

 ¿Qué se espera que haga usted después de leer la carta? 

 ¿Para qué se va a usar la información recolectada por este estudio? 

 ¿Acerca de qué se trata este estudio? 

Gracias. 

Ahora quiero preguntarle acerca de los detalles de la carta. Voy a devolverle la carta y podemos 

hablar más acerca de ella. 

[Anote cualquier lugar en el que el entrevistado dude al leer, secciones que se salte el 

entrevistado, palabras que el entrevistado pronuncie incorrectamente o cualquier cosa acerca de 

la que haga comentarios el entrevistado.] 

Razón 

Por favor, lea en voz alta el primer párrafo de la carta. 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Después de leer este párrafo, puede decirme quien le está enviando esta carta? 

 ¿Puede decirme quién cree usted que es el patrocinador de esta encuesta?

Qué 

Continuemos. Por favor, ¿puede leer el segundo y tercer párrafo? 
16 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Acerca de quién debe responder la encuesta? 

 ¿Acerca de qué cree usted que se trata esta encuesta? En términos generales, usted no 

tiene que darme detalles específicos, sólo lo que usted piensa acerca de lo que leyó. 

 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles? 

Cómo 

Continuemos. Por favor, ¿puede leer los siguientes dos párrafos? 

[Espere a que el entrevistado termine de leer el párrafo.] 

PREGUNTE: 

 ¿Dónde puede encontrar más información acerca del estudio? 

STEP 6: SCREENER POSTCARD

El patrocinador de la encuesta también envía esta postal. Por favor, tómese un minuto para 

leerla. 

SCREENER POSTCARD 

PREGUNTE: 

 ¿Cuál es el propósito de la postal? 

 ¿Quién manda la postal? 

 ¿Hay algunas palabras aquí que no le sean familiares? 

otros comentarios 

Gracias. 

Continúe con la entrevista 

   
17 

STEP 7: ATES ITEMS   

La última actividad que le voy a pedir que hagamos hoy es responder a algunas preguntas acerca de su 

adiestramiento laboral. Estas son preguntas nuevas que se están considerando para la encuesta y sus 

comentarios acerca de ellas serán útiles. Al igual que hicimos antes, por favor, responda a las preguntas 

y " piense en voz alta " mientras contesta las preguntas.

ATES ITEMS 

[haga preguntas de seguimiento (probes) si es necesario y aplican al 

encuestado] 

Q5. Ahora veamos lo que hizo para preparase [Q5. ¿Hizo usted alguna de 

las siguientes cosas al prepararse para obtener su certificación o licencia 

más importante? (Marque UNA casilla en cada una de las siguientes 

líneas.)] 

 Por favor lea la letra D y dígame, en sus palabras, ¿que significa 

para usted esto? [opción d: Obtuve un título universitario]. 

 ¿Ahora hablemos de la opción E? [opción e: Participé en 

formación o capacitación laboral, pasantía o práctica profesional, o 

un programa de aprendiz.] 

Q6. Veamos esta pregunta: 

¿Qué tan útil ha sido su certificación o licencia más importante para cada 

una de las siguientes situaciones? 

 Quiero que lea la opción C otra vez y me diga, en sus propias 

palabras, ¿qué significa esto para usted? [opción c. Mantener el 

interés de los empleadores o clientes en usted]. 

 Ahora veamos la opción D ¿Qué significa esto para usted? [opción 

d: Mejorar sus habilidades de trabajo]. 

Q7. Ahora veamos las preguntas sobre su experiencia de trabajo. Por favor 

lea la pregunta 7 otra vez. ¿Me puede decir que significa el termino 

“educación o formación/capacitación formal” para usted? 

 Hablemos de “pasantía o práctica profesional”, ¿que significa para 

usted esto? 

 Y finalmente programa “de aprendiz”, ¿que significa para usted 

esto?
18 

FINAL QUESTION 

¿Tiene algo más que le gustaría hablar sobre estos materiales que no hemos tenido la 

oportunidad de hablar?

THANK YOU FOR PARTICIPATING 

Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para 

mejorar las cartas y encuesta.

Interviewer Instructions: Collect materials and escort respondent out of the interviewing room 

and to the lobby. Please give the respondent incentive envelope [$40 cash] and thank them for 

their time and effort. 
Phase 2 Protocols 

1000 Thomas Jefferson Street NW, Washington, DC 20007-3835 | 202.403.5000 | TTY 877.334.3499 | www.air.org

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. 

Also, keep track of the time so that your total interview time with the respondent lasts 1 hour or 

less. 

Before starting the interview, use this table to confirm the participant information we received 

from the recruitment firm. If the respondent indicates that some of the listed information is 

incorrect, please cross it out and enter the correct information in the relevant cell.* 

Background Information Confirmed 



Date & Time 

Location 

Name of Interviewer 

Parent/Respondent First Name 

Respondent’s Gender  Male  Female 

Respondent’s Age 

If parent fill following questions: 

Child Enrolled in School  Yes  No 

Name of School (only if R is willing to share) 

Type of School  Public  Charter  Private  Homeschool 

Student’s Grade 

Age of Student 

Assigned ID 

*This cover sheet should be kept separate from the rest of the data and 

be turned in with the rest of the paperwork. This cover sheet should not 

be copied, shared, or left in an unlocked office. 

   
20 

Esta página se ha dejado en blanco intencionalmente		
21 

Documento Adjunto 3 NHES 2016 PFI, ECPP, ATES  

Protocolo en Español para las Entrevistas Cognitivas  

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA

Instrucciones para el entrevistador: El siguiente protocolo NO debe ser leído textualmente. 

El texto en letra cursiva solo es contenido con el cual el entrevistador debe familiarizarse 

antes de la entrevista. El texto entre paréntesis [] contiene instrucciones para el 

entrevistador. 

Hola, mi nombre es <su nombre> y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. 

Gracias por venir a ayudarnos el día de hoy. 

[Platique de algún tema casual para establecer una buena relación con el participante.]: 

Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos 

realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta 

acerca de [la educación de sus hijos, la búsqueda y selección de cuidado infantil para los padres 

con hijos de cinco años de edad o menores, o acerca de sus certificados o licencias]. 

Como parte de este estudio, le voy a pedir que complete una encuesta como que si estuviera 

completando una encuesta real en su casa. No existen respuestas correctas o incorrectas y está 

bien si usted no sabe algunas de las respuestas. A medida que usted vaya respondiendo a las 

preguntas, yo le voy a hacer algunas preguntas acerca de sus respuestas. Esto nos ayudará a 

que las preguntas en la encuesta sean mejores y más claras. 

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas 

mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va 

a observar nuestra sesión de hoy]. 

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan 

revisar la grabación más tarde. Cualquier información que se refiera a usted o a su familia 

personalmente, tal como su nombre [o la escuela de su hijo(a)], no será incluida en nuestro 

informe y sus respuestas serán utilizadas únicamente para fines de investigación. 

Recuerde que esto no es un examen. Queremos que nos diga lo que usted piensa acerca de las 

preguntas de la encuesta. 

Si en algún momento usted desea parar, solo déjemelo saber. 

[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y 

termine la entrevista.] 

Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a 

cualquier pregunta que pueda hacer el participante.] 
22 

DELE AL ENCUESTADO UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO. PEDIRLE 

QUE LEA LA FORMA (O LEASELA, DEPENDINEDO DE LA PREFERENCIA DEL 

ENCUESTADO) RESPONDA CUALQUIER PREGUNTA Y PEDIRLE QUE FIRME LA FORMA. 

DELE UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO AL ENCUENTADO 

SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACION 

PASO 2: PRACTICA DE PENSAR VOZ ALTA 

Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las 

preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así 

entender cómo mejorar estos materiales. Me interesa más saber cómo llega a sus respuestas que 

en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer 

un ejercicio de práctica. 

Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una 

oportunidad para que practique. 

[Model the think aloud process for the respondent, including reading the question out loud. 

Elaborate on counting all of the glasses of water as needed you can make up pieces of 

information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee 

I drank in the morning, I drink three cups of coffee, I am going to count those since that is the 

only liquids I have in the morning...” When you are modeling, write down the number of glasses 

of water as you are thinking aloud.] 

[Once finished the model practice think aloud]

Voy a darle un papel con una pregunta. Le voy a pedir que lea en voz alta la pregunta y que 

piense en voz alta mientras decide su respuesta. 

[Give respondent handout with think aloud practice.] 

¿Cuántos vasos de agua tomó usted ayer? 

Por favor dígame lo que está pensando mientras me da su respuesta. 

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta? 

¿Qué estaba visualizando mientras estaba decidiendo qué responder? ¿Puede decirme más 

acerca de lo que usted está incluyendo en su cuenta? 

Cuando haya completado el ejercicio: Gracias. Ése es el tipo de detalle que busco cuando usted 

me hable acerca de sus respuestas a las preguntas de la encuesta. 

ENTREGUE AL ENCUESTADO UNA COPIA DEL CUESTIONARIO Y UNA PLUMA 

O UN LAPIZ 
23 

PASO 3: RESPUESTAS A LA ENCUESTA 

Ahora voy a darle la encuesta. Me gustaría que piense en voz alta mientras decide su respuesta. 

Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma(o), como si yo no 

estuviera aquí. 

Recuerde que estoy muy interesada(o) en escucharle hablar acerca de lo que está pensando 

mientras decide sus respuestas a las preguntas. 

RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL 

CUANDO PIENSA EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON DE 

GRAN UTILIDAD. 

Preguntas Útiles 

En sus propias palabras, ¿qué dice esta pregunta? 

¿Qué significa para usted la palabra [término] en esta pregunta? 

¿Qué tan fácil o difícil es recordar [tema]? 

¿Qué tan fácil o difícil fue escoger una respuesta? 

Dígame lo que está pensando. 

¿Cómo llegó a esa respuesta? 

Usted dijo [respuesta]. ¿Puede decirme un poco más acerca de cómo llego a esa respuesta? 

[En respuesta a las acciones o comentarios del entrevistado]: Por favor, dígame que sucedió. 

Dígame que estaba usted haciendo. 

[Si nota alguna señal visual, como cejas fruncidas o una expresión de confusión]: Dígame lo 

que está pensando 
24 

PFI: Encuesta sobre la experiencia de los estudiantes y las familias con sus escuelas

(NHES41-BC) 

Educación del niño 

PÍDALE AL ENCUESTADO QUE POR FAVOR LLENE EL CUESTIONARIO. RECUERDELE 

QUE TIENE QUE PENSAR EN VOZ ALTA, SI NO LO ESTÁ HACIENDO 

HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO

Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el 

encuestado tiene problemas con ellas.

 

Q1-Q2 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Q3 ¿Es esta la escuela asignada del distrito?

[Preguntar solamente si el encuestado respondió esta pregunta] ¿Cómo llegó a esta respuesta? 

¿Qué significa para usted “escuela asignada del distrito”? 

Q4-Q10 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q11 ¿En qué mes de este año escolar comenzó este niño(a) a asistir a su escuela actual? 

¿Cómo decidió la respuesta a esta pregunta? ¿Le resulta fácil o difícil contestar esta pregunta? 

¿Por qué? 

Q14. ¿Está este niño inscrito en algún curso avanzado llamado Advanced Placement AP en 

la escuela?

Si respondió si, ¿me puede explicar sobre estos cursos en los que su niño(a) está inscrito? ¿En 

qué pensaba cuando respondió a esta pregunta? 

¿En qué grado académico se encuentran los estudiantes que generalmente están inscritos en un 

curso avanzado en el distrito escolar de su hijo(a)? 

Q15-21 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q22. ¿Está este niño(a) tomando algún curso relacionado con la escuela por el internet en lugar 
25 

de hacerlo de manera presencial con un maestro(a)? 

Si responde si - por favor dígame un poco más acerca de esto. ¿Me puede decir porque usted 

eligió esta respuesta? 

¿Me puede describir a que se refiere? 

Q23. ¿Cuál de los siguientes lugares proporciona esas clases? 

¿Cómo decidió su respuesta? 

[Si seleccionó una respuesta o la opción de “Algún otro lugar”] Por favor dígame un poco más 

acerca de esto. ¿Cómo tiene acceso su hijo(a) a estas clases? ¿En qué lugar se encuentra cuando 

está tomando dicha clase? ¿Quién provee esa clase? 

 

Q24. No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de 

pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un 

problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados 

Q25. Algunos padres deciden educar a sus hijos en el hogar en vez de mandarlos a una 

escuela pública o privada. ¿Recibe este niño sus clases en el hogar en lugar de en la escuela 

para algunas clases o asignaturas? 

¿Por favor, me puede decir con sus propias palabras qué es lo que se le está preguntando aquí? 

¿Cómo determinó su respuesta? 

[Si respondió que “si”] ¿Me puede decir porque respondió si? 

[Si respondió que “no”] ¿Me puede decir porque respondió no? 

[Si se aplica] Q26. ¿Cuál de las siguientes frases describe mejor el arreglo de educación en 

el hogar para este niño? 

[Si el encuestado elige la respuesta a o b] 

¿Porque eligió esta respuesta? ¿Me puede explicar cómo funciona este arreglo? 

[Si el encuestado elige una de las siguientes respuestas: “Este niño no recibe la educación en el 

hogar” o “Este niño asiste a la escuela pública o privada para todas las clases o asignaturas] 

¿Porque eligió esta respuesta? 
26 

[Si no es evidente indague el significado de la escuela del hogar] 

[Si se aplica] Q27. Pensando en todos los años en los que este niño(a) ha recibido educación 

en el hogar, ¿en cuál de las áreas a continuación ha recibido este niño(a) enseñanza durante 

su instrucción en el hogar? 

[Se han añadido 16 opciones de respuesta a esta pregunta. Por favor anote si el entrevistado tiene 

problemas con el número y el tipo de asignaturas enumeradas] 

[Si no es evidente indague el significado de la escuela del hogar] 

¿Cómo determinó cómo responder en cada una de las preguntas? 

¿Cómo enseña las asignaturas durante el periodo de instrucción en el hogar? 

¿Cree que falta algo? 

[Si se aplica] Q30. Hay diferentes razones por las que los padres eligen la educación en el 

hogar para sus hijos. ¿Eligió su familia la educación en el hogar para su hijo porque: 

¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]? 

 

Platíqueme un poco más sobre lo que estaba pensando cuando respondió las preguntas. 

[Si no es evidente indague el significado de la escuela del hogar]

[Si se aplica] Q31. Entre las razones por las que su familia decidió educar a este niño(a) en 

el hogar, ¿cuál diría usted que es la más importante para usted? Escriba la letra de la 

pregunta 30 para la razón más importante por la que decidió educar a su niño(a) en el 

hogar. 

¿Qué tuvo en cuenta al elegir esta razón? 

Platíqueme un poco más sobre lo que estaba pensando cuando respondió a esta pregunta. 

Familias y Escuela 

Q32-36. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 
27 

Tareas 

Q37-43. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Al final de la sección pregunte: 

Quiero hacerle preguntas sobre algunas palabras que usamos en esta sección de tareas. 

¿Me puede decir cuando en esta parte dice tarea o tareas a que se refiere? 

Actividades en familia 

Q44-46. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Salud del niño(a) 

Q47 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de 

pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un 

problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q48. ¿Le ha dicho un profesional de la salud o de la educación que este niño(a) tiene alguno 

de los siguientes problemas de salud? 

[Note como responde el encuestado cada una de las preguntas de la a-m.] 

¿Hay alguna palabra o frase que no le es familiar? 

¿Hay algún problema de salud que su niño(a) tiene que no está listado? 

Q49- 50. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que 

indique un problema potencial, o si hay confusión o alguna inconsistencia en la 

interpretación comparado con otros encuestados. 

Q51. ¿Ofrecen estos servicios algunas de las siguientes fuentes? 

[note como responde cada una de las preguntas de la a-d el encuestado] 
28 

¿Cómo determinó su respuesta? 

¿Hay alguna palabra o frase que no le es familiar? 

Q52. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de 

servicios para la familia (IFSP por sus siglas en inglés), un programa individualizado de 

educación (IEP por sus siglas en inglés) o plan de servicios? 

[en esta pregunta estamos tratando de determinar que vocabulario utilizan los padres de familia 

para referirse a el IFSP y al IEP] 

¿Cómo determinó su respuesta? 

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus 

siglas en inglés)? 

¿Qué significa para usted un “programa individualizado de educación” o IEP? 

Q53-56. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Historia del niño(a) 

Q57-62. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Q63. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado del 

acuerdo de custodia)? 

¿Cómo determinó su respuesta? 

En sus propias palabras, ¿qué dice esta pregunta? 

 

[Si la respuesta fue NO] ¿Qué significa “arreglos de custodia compartida”? 

Q64. Si contestó afirmativamente, este niño(a) ¿pasa la mayor parte del tiempo en este 

domicilio, otros domicilios, o el mismo tiempo en ambos? 

¿Cómo determinó su respuesta? 
29 

Q66. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación 

bilingüe o a un programa especial de inglés? 

¿Cómo determinó su respuesta? 

¿Hay alguna palabra o frase que no le es familiar? 

Los miembros de su familia 

Q68. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las 

siguientes personas viven en el hogar con este niño(a)? 

¿Cómo determinó su respuesta? 

¿A quién está incluyendo en su cuenta?

La familia del niño(a) 

Q71-115. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Su hogar 

Q116-Q124. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

 
30 

PFI: Encuesta sobre la experiencia de los estudiantes y las familias con sus escuelas: Escuela del 

Hogar 

ESTAS PREGUNTAS SON PARA LOS PAPAS QUE SUS HIJOS ESTAN EN LA 

ESCUELA DEL HOGAR 

Dígale al encuestado lo siguiente: 

Estas preguntas son sobre su hijo que está en la escuela del hogar. ¿Qué edad tiene el 

niño? 

Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta 

PIDALE AL ENCUESTADO QUE PORFAVOR RELLENE EL CUESTIONARIO. 

RECUERDELE QUE TIENE QUE PENSAR EN VOZ ALTA, SI NO LO ESTA HACIENDO 

HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO

Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el 

encuestado tiene problemas con ellas.

 

Escuela del Hogar 

Q1-Q5 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q6. ¿Cuántas horas a la semana va normalmente este niño(a) a una escuela para recibir 

instrucción? No incluya el tiempo que dedica a actividades extracurriculares. 

¿Cómo determinó su respuesta? 

[Verifique si la persona respondió a esta pregunta a pesar de haber indicado que el niño(a) está 

en escuela domiciliaria (o escolarización en el hogar). 

Q10. ¿Cuál de las siguientes afirmaciones describe mejor el estilo de enseñanza que se usa 

para la educación en el hogar de este niño(a)? 

¿Cómo determinó su respuesta? 

Q17. Hay diferentes razones por las que los padres eligen la educación en el hogar para sus 

hijos. ¿Eligió su familia la educación en el hogar para su hijo porque: 
31 

¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]? 

Dígame un poco más sobre lo que estaba pensando cuando respondió estas preguntas. 

Q18. Entre las razones por las que su familia decidió educar a este niño(a) en el hogar, 

¿cuál diría usted que es la más importante para usted? 

¿Cómo determinó qué opción escoger [a-j]? 

Platíqueme un poco más sobre lo que estaba pensando cuando escogió esta opción. 

[Si no es evidente, haga preguntas sobre el significado de la escuela del hogar]

Q20. Pensando en todos los años que este niño(a) ha recibido la educación en el hogar, ¿en 

cuál de las áreas a continuación ha recibido este niño(a) enseñanza durante su instrucción 

en el hogar? 

¿Cómo determinó qué responder s en cada una de las preguntas? 

¿Cómo cubre usted las asignaturas durante el periodo de instrucción en el hogar? 

¿Hace falta algo? 

La salud del niño(a) 

Q27-30. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o s i hay confusión o alguna inconsistencia en la interpretación 

comparado con otros encuestados.

Q31. ¿Ofrecen estos servicios algunas de las siguientes fuentes? 

¿Cómo determinó su respuesta? 

[Solo si el encuestado tiene a su niño(a) en una escuela privada]¿Qué significa para usted que 

estos servicios se ofrezcan en la escuela privada del niño(a)? 

Q33. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de 

servicios para la familia (IFSP por sus siglas en inglés), un programa individualizado de 

educación (IEP en inglés) o plan de servicios? 

¿Cómo determinó su respuesta? 

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus 

siglas en inglés)? 
32 

¿Qué significa para usted un “programa individualizado de educación” o IEP? 

Historia del niño(a) 

Q38-42. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Q43. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado 

del acuerdo de custodia)? 

¿Cómo determinó su respuesta? 

En sus propias palabras, ¿qué dice esta pregunta?

[Si la respuesta fue NO] ¿Qué significa para usted arreglos de custodia compartida? 

Q44. Si contestó afirmativamente, este niño(a) pasa la mayor parte del tiempo en este 

domicilio, otros domicilios, o el mismo tiempo en ambos? 

¿Cómo determinó su respuesta?

Q45. ¿Qué idioma habla este niño(a) principalmente en el hogar? 

¿Cómo decidió su respuesta? 

Los miembros de su familia 

Q47. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las 

siguientes personas viven en el hogar con este niño(a)? 

¿Cómo determinó su respuesta? 

¿A quién está incluyendo en su cuenta de las personas que viven en su hogar?

La familia del niño(a) 

Q62. ¿Cuál es el grado o nivel escolar más alto que esta persona completó? 

[Solamente si el respondiente escoge “diploma de estudios secundarios o equivalente”, 

“diploma vocacional obtenido después de la escuela secundaria”, o “algo de 
33 

colegio/universidad, pero sin título”:] 

¿Cómo determinó su respuesta? 

¿Qué quiere decir para usted “diploma vocacional obtenido después de la escuela secundaria”? 

Q63. Actualmente, ¿asiste esta persona a una escuela, universidad o centro de educación 

para adultos o recibe educación vocacional o capacitación laboral? 

En sus propias palabras, ¿qué dice esta pregunta?

ECPP: El Futuro de nuestros niños: Una Encuesta sobre el cuidado y la educación de niños 

pequeños. (NHES-21BC(SP)) 

Estas preguntas son para los padres que tienen algún niño(a) menor de cinco años., que todavía 

no está inscrito en preescolar (kindergarten). 

Dígale al encuestado lo siguiente: 

Estas preguntas son sobre su hijo que es menor de cinco años de edad, que todavía no está 

inscrito en preescolar (kindergarten). ¿Qué edad tiene este niño(a)? ¿Qué tipo de arreglos u 

opciones ha establecido para el cuidado de su hijo(s)? [Anote la respuesta] 

Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta 

Cuidado Infantil y Programas para la Niñez 

Q1-11 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q12. ¿Le ayuda alguna de las siguientes personas, programas u organizaciones a pagarle a 

este pariente por el cuidado del niño(a)? 

¿Me podría decir que información cree que esta pregunta le está pidiendo? 

Si la persona contesto la pregunta Q12b. b. Asistencia temporal a familias necesitadas 

(TANF)

Si el encuestado respondió sí. ¿Me puede describir este programa? 

Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta? 
34 

Q13-Q16 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q17 Actualmente cuida a este niño(a), con regularidad, una persona que no es su pariente 

ya sea en su casa o en otra casa? 

Si el encuestado respondió sí. ¿Me puede describir el lugar? 

Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta? 

¿Qué significa para usted una persona que no es su pariente? 

Q18-Q37 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Q38. (Si se aplica) ¿Es este programa de Head Start o de Early Head Start? 

¿Me podría decir un poco más sobre qué le están preguntando aquí? 

Si el encuestado respondió sí. ¿Me puede describir este programa? 

Si el encuestado respondió no. ¿Me puede decir por qué respondió que no a esta pregunta? 

Q39-48 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

[pregunte solo si no ha hecho estas preguntas en la pregunta Q12] 

Q49 ¿Le ayuda alguna de las siguientes personas, programas u organizaciones a pagarle a 

este pariente por el cuidado del niño(a)? 

¿Me podría decir que información cree que esta pregunta le está pidiendo? 

Si la persona contesto la pregunta Q49b. b. Asistencia temporal a familias necesitadas 

(TANF)

Si el encuestado respondió sí. ¿Me puede describir este programa? 
35 

Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta? 

Q49-53 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Buscar y Seleccionar Cuidado Infantil para su Niño(a) 

Q54. (Solo si no se le ha preguntado ya sobre el programa de Head Start) ¿Ha asistido este 

niño(a) alguna vez a un programa de Head Start o de Early Head Start? 

¿Me podría decir que información cree que esta pregunta le está pidiendo? 

Si el encuestado respondió sí. ¿Me puede describir este programa? 

Si el encuestado respondió no. ¿Me puede decir por qué respondió no a esta pregunta? 

Q55-Q56 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q57. ¿Cuánta dificultad tuvo para encontrar el tipo de cuidado infantil o programa de 

educación temprana que usted quería para este niño(a)? 

¿Cómo determinó su respuesta? 

¿Me podría decir un poco más sobre su respuesta? 

¿Qué significa para usted el término “educación temprana”? 

Q58. ¿Qué tan importante fue cada una de estas razones cuando usted seleccionó el 

cuidado infantil o el programa en el que este niño(a) pasa la mayor parte del tiempo? 

¿Hay alguna otra razón que es importante para usted y que no ha encontrado en la lista de 

opciones? 

Q59h. ¿El Sistema de Calificación y Mejoramiento de la Calidad (Quality Rating and 

Improvement System, QRIS) de su ciudad o estado?

¿Me podría decir un poco más de su respuesta? 
36 

Q59k. ¿Está acreditado el proveedor del servicio de cuidado infantil? 

¿Me podría decir un poco más de su respuesta? 

Q60 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de 

pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un 

problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Actividades en familia 

Q61-67. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Cosas Que su Niño(a) Podría estar Aprendiendo 

Q68-75. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Salud del niño(a) 

Q76-80. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados.

Q81. ¿Ofrecen estos servicios algunas de las siguientes fuentes? 

¿Cómo determinó su respuesta? 

¿Qué significa para usted que estos servicios se ofrezcan en la escuela privada del niño(a)? 

Q82. ¿Se ofrecen algunos de estos servicios por medio de un Plan individualizado de 

servicios para la familia (IFSP, por sus siglas en inglés) o un Programa individualizado de 

educación (IEP, por sus siglas en inglés)? 

¿Cómo determinó su respuesta? 

Si el encuestado responde si: 

¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus 

siglas en inglés)? 
37 

¿Qué significa para usted un “plan individualizado de educación”? 

Q83-86 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Historia del niño(a) 

Q87 – 91. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Q92. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado 

del acuerdo de custodia)? No incluya propiedades vacacionales. 

¿Cómo determinó su respuesta? 

En sus propias palabras, ¿qué dice esta pregunta? 

[Si la respuesta fue NO] ¿Qué significa “acuerdo de custodia compartida”? 

Q94. ¿Qué idioma habla este niño(a) principalmente en el hogar? 

¿Cómo determinó su respuesta? 

¿Hay alguna palabra o frase que no le es familiar? 

Q95. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación 

bilingüe o a un programa especial de inglés? 

¿Qué cree le está preguntando la pregunta? 

¿Hay alguna frase o palabra que no le sea familiar en esta pregunta? 

Los miembros de su familia 

Q97. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las 

siguientes personas viven en el hogar con este niño(a)? 
38 

¿Cómo determinó su respuesta? 

¿A quién está incluyendo en su cuenta de las personas que viven en el hogar?

La familia del niño(a) 

Q100-138. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Su hogar 

Q139-Q145. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su 

proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique 

un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado 

con otros encuestados. 

Encuesta ATES 

PASO 3: LLENADO de la encuesta 

Ahora voy a darle una encuesta. Me gustaría que piense en voz alta mientras decide su 

respuesta. Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma, como 

si yo no estuviera aquí. 

Recuerde que estoy muy interesada/o en escucharle hablar acerca de lo que está pensando 

mientras decide las respuestas para las preguntas. 

RECUERDA DE DECIRLE AL ENCUESTADO QUE ESTA HACIENDO UN MUY BUEN 

TRABAJO PENSANDO EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON 

DE GRAN UTILIDAD 

Escolaridad 

HAGA PREGUNTAS SEGÚN SEA NECESARIO 

Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el encuestado tiene problemas 

con ellas.

Q1-Q2 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz 

alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o 

alguna inconsistencia en la interpretación comparado con otros encuestados. 

Q3. ¿Está actualmente inscrito en una universidad, escuela técnica o comercial, u otra escuela? 
39 

En sus propias palabras, ¿qué dice esta pregunta? 

[Si respondió Si] ¿Puede decirme algo más sobre las clases que toma? ¿Cuál es el campo de estudio? 

¿Dónde toma las clases? 

¿Qué significa para usted medio tiempo y tiempo completo? 

Q5. Desde que salió de la escuela secundaria, ¿ha tomado usted clases de lectura para mejorar cómo lee? No 

incluya clases a nivel universitario. 

¿A qué se refiere esta pregunta? 

Certificación y Licencias 

Q6. ¿Tiene usted una certificación profesional o una licencia estatal o industrial que este actualmente activa? 

No incluya licencias comerciales, tales como de licor o de máquinas expendedoras (vending machines).

[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.] 

¿Me puede decir que cree que esta pregunta le esta preguntando? 

Q7. (si APLICA) ¿Si contestó que sí, cuantas certificaciones y licencias tiene usted que estén actualmente 

activas? 

¿Cómo determinó su respuesta? 

¿Qué significa para usted el término “actualmente activas” en esta pregunta? 

En sus propias palabras, ¿qué dice esta pregunta? 

Q8. ¿Cuál es el nombre de su certificación o licencia más importante? 

¿Cómo determinó su respuesta? 

¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la más importante? 

Q9. ¿Para qué tipo de trabajo es su certificación o licencia más importante? 

¿Qué tan fácil o difícil fue para usted recordar el tipo de trabajo para el que la certificación o licencia le sirve? 

Q10. ¿Es su certificación o licencia más importante requerida por un gobierno federal, estatal o local para 

realizar este trabajo? 

En sus propias palabras, ¿qué dice esta pregunta? 

¿A qué se refiere esta pregunta cuando dice “requerida”? 

Q11. ¿Le podrían quitar o suspender su certificación o licencia más importante por alguna razón?

En sus propias palabras, ¿qué dice esta pregunta? 

¿A qué se refiere esta pregunta cuando dice quitar o suspender? 
40 

Q12. ¿En qué año obtuvo su certificación o licencia más importante? 

¿Cómo determinó su respuesta? 

¿Qué tan fácil o difícil fue para usted recordar en qué año obtuvo la certificación o licencia más importante? 

Q13a. ¿Se preparó usted para obtener su certificación o licencia más importante realizando alguna de las 

siguientes actividades? Tomando clases en una universidad , escuela técnica o escuela vocacional (escuela para 

aprender oficios) 

¿Cómo determinó que contestar en esta pregunta? 

Q14. ¿Es su certificación o licencia más importante para su empleo actual? 

¿Cómo determinó qué contestar en esta pregunta? 

En sus propias palabras, ¿qué dice esta pregunta? 

[SI contesto afirmativamente] ¿Cómo puede usted determinar si la certificación o licencia es requerida para su 

trabajo actual? 

Q15. ¿Qué tan útil ha sido su certificación o licencia más importante para cada una de las siguientes 

situaciones? (a-d)

¿Qué tan fácil o difícil fue para usted contestar solamente sobre la certificación o licencia más importante para cada 

una de las opciones? 

Q16. ¿Tiene otra certificación o licencia actualmente activa? 

¿Cómo determinó qué certificación o licencia poner aquí? 

Q17-Q22 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en 

voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay 

confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.

Q22a (Si aplica). ¿Se preparó usted para obtener su segunda certificación o licencia más importante 

realizando alguna de las siguientes actividades? 

22a ¿Tomando clases en una universidad, escuela técnica o escuela vocacional (escuela para aprender 

oficios)? 

¿Cómo determinó qué contestar en esta pregunta? 

Q23 (Si se aplica). ¿Es su segunda certificación o licencia más importante para su empleo actual

¿Cómo determinó qué contestar en esta pregunta? 

¿Con sus propias palabras, describa qué es lo que le están preguntando? 

 Después de la pregunta 23

¿Qué tan fácil o difícil fue para usted contestar solamente pensando en la certificación o licencia que usted decidió 

fue la segunda más importante? 

¿Mientras contestaba, pensó en sus otras certificaciones o licencias? 
41 

Q24. ¿Qué tan útil ha sido su segunda certificación o licencia más importante para cada uno de las siguientes 

situaciones? (a-d)

¿Qué tan fácil o difícil fue para usted contestar solamente sobre su segunda certificación o licencia más importante 

para cada una de las preguntas? 

¿Me podría decir cuál piensa usted que es la diferencia entre las respuestas de estas categorías? ¿Fue fácil o difícil 

seleccionar la respuesta? 

Q26. (Si aplica). ¿Cuál es el nombre de su tercera certificación o licencia más importante? 

¿Cómo determinó qué contestar en esta pregunta? 

¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la tercera más importante? 

Q27. (Si se aplica). ¿Para qué tipo de trabajo su tercera certificación o licencia más importante?

¿Cómo determinó qué contestar en esta pregunta? 

Q29. ¿Es su tercera certificación o licencia más importante requerida por un gobierno federal, estatal o local? 

¿Con sus propias palabras, describa qué es lo que le están preguntando? 

Q25-29 (Si aplica)

¿Qué tan fácil o difícil fue para usted contestar solamente pensando en su tercera certificación o licencia más 

importante? 

¿Mientras contestaba, pensó en sus otras certificaciones o licencias? 

Certificados 

Q30. Las personas a veces reciben certificados por completar un programa educativo o de capacitación. Éstos 

son diferentes de una certificación o licencia. No incluya aquí certificaciones/ licencias. ¿Alguna vez ha 

obtenido usted algunos de los siguientes tipos de certificados?

[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.] 

¿Me puede decir que cree que esta pregunta le está preguntando? 

Q30d. Un certificado—no una titulación—por completar un programa en una universidad técnica o de la 

comunidad, o alguna otra escuela después de la escuela secundaria. No incluya certificados de maestro o 

títulos universitarios?

¿Porque eligió esta respuesta (si/no)? 

Q31. (Si contestó que sí.) Nos referiremos a los certificados en la pregunta 30d como certificados ‘postsecundaria’. ¿Cuál fue el campo de estudio de su último certificado de post-secundaria? (Marque UNA casilla 

solamente.)

¿Pudo encontrar alguna opción que encaje con su situación o cree que falta alguna opción? 

¿Me podría hablar un poco más sobre su respuesta? 
42 

SI contesto “otro” ¿podría decirme más sobre su respuesta? 

Q34. ¿Cuál de las siguientes opciones fue un requisito para matricularse en el último programa de certificado 

post-secundaria? 

¿Con sus propias palabras, describa qué es lo que le están preguntando? 

¿Me podría hablar un poco más sobre su respuesta? 

Q35. Para obtener su último certificado de post-secundaria, ¿tuvo que completar...? [a-b] 

¿Cómo determinó las respuestas a las preguntas (a-b)? 

Q37. ¿Está su empleo actual relacionado con su último certificado de post-secundaria? 

¿Cómo determinó qué contestar a esta pregunta? 

Q38 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta 

y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna 

inconsistencia en la interpretación comparado con otros encuestados.

Programas de Experiencia laboral 

Q39 ¿Ha completado alguna vez una pasantía o práctica profesional, programa de cooperación educativa, 

practicum, asistente jurídico (clerkship), residencia médica, experiencia clínica, programa de aprendiz o 

similar? 

[Si aplica] Me podría decir sobre su programa: 

¿Para qué trabajo fue? 

¿Puede describir un día típico en el programa? 

¿Qué tipo de organizaciones estaban involucradas y cuáles eran sus roles? 

Q40. ¿Si su respuesta es sí, para qué tipo de trabajo fue su último programa de experiencia laboral?

¿Me podría decir un poco más acerca de cómo seleccionó su respuesta? 

¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difícil fue seleccionar su respuesta? 

Q41. ¿Cuánto tiempo duró su programa de experiencia laboral más reciente? 

¿Cómo llegó a su respuesta? 

Q42. (Si aplica) ¿Qué sueldo ganaba como parte de su último programa de experiencia laboral?

[Si no contesto rápidamente] ¿Pudo encontrar una respuesta que estuviera acorde a su experiencia? De no serlo así, 

¿cuál hubiese sido una mejor opción para usted? 

[Si ninguna opción fue seleccionada] ¿Me podría hablar un poco más sobre el salario que recibió? 

Q43 a-c. Como parte de su último programa de experiencia laboral, ¿realizó usted alguna de las siguientes 

actividades? 
43 

¿Cómo llegó a su respuesta? 

(Si contestó Sí a alguna de las preguntas) ¿Me podría hablar un poco más sobre la relación entre su programa de 

experiencia laboral las clases, su entrenamiento, o el estudio que llevó a cabo? 

Q44 a-d. ¿Describe alguna de las siguientes actividades su programa de experiencia laboral más reciente? 

¿Cómo llegó a su respuesta? 

Q44a (Si contestó Sí) ¿Me podría hablar sobre las evaluaciones? 

Q44c (Preguntar solo si contestó Sí) ¿Qué significa para usted tener “clasificación de obrero especializado”? 

Q45. ¿Cuál de las siguientes opciones describe mejor su último programa de experiencia laboral?

¿Me podría decir cómo un poco más de cómo seleccionó su respuesta? 

¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difícil fue seleccionar su respuesta? 

Q46. (Si aplica) ¿Considera que su último programa de experiencia laboral le ayudó (o le ayudará) a lograr 

una certificación o licencia profesional? 

¿Me podría decir con sus propias palabras, que es lo que le están preguntando? 

Q47a-c. (Si aplica) ¿Qué tan útil fue su programa de experiencia laboral más reciente para cada uno de las 

siguientes opciones? 

¿Cómo seleccionó su respuesta para cada una de las preguntas? 

¿Me podría decir cuál cree usted que es la diferencia entre las categorías de las diferentes respuestas? 

¿Fue fácil o difícil seleccionar su respuesta? 

Q48. (Si aplica) En su trabajo actual, ¿qué tan a menudo usa las habilidades o el conocimiento que aprendió 

durante su último programa de experiencia laboral? 

¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando? 

¿Me podría hablar más sobre lo que usted estaba pensando cuando respondió esta pregunta? 

Q49. (Si aplica) ¿Está su trabajo actual relacionado con su último programa de experiencia laboral? 

¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando? 

Empleo 

Q50-Q67 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en 

voz alta y solamente indague si el participante dice algo que señale un problema potencial, o si hay confusión o 

alguna inconsistencia en la interpretación comparado con otros encuestados.

Q68. (Si aplica) ¿Tenía usted una licencia requerida por una agencia del gobierno federal, estatal o local para 

realizar este trabajo?

¿Con sus propias palabras, describa qué es lo que le están preguntando? 
44 

[Verificar si es congruente con las respuestas anteriores] 

Q69-70 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz 

alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna 

inconsistencia en la interpretación comparado con otros encuestados.

Historial 

Q71-84 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz 

alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna 

inconsistencia en la interpretación comparado con otros encuestados.

PREGUNTA FINAL 

¿Hay algo más sobre lo que le gustaría hablar acerca de estos materiales? ¿Hay algo acerca de lo que 

no ha tenido oportunidad de hablar?

PASO 4. GRACIAS POR PARTICIPAR 

Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para mejorar 

las cartas y la encuesta.

Instrucciones para el entrevistador: Recoja los materiales y acompañe a los encuestados al 

vestíbulo. Por favor dele a los encuestados el sobre con el incentivo, y agradézcales por su 

tiempo y su ayuda.