Adapting Cognitive Interview Techniques for Use in Pretesting Spanish 

Language Survey Instruments 

Author: Patricia L. Goerman 

U.S. Census Bureau 

4700 Silver Hill Road Stop 9100, Washington DC 20233-9100, Patricia.L.Goerman@census.gov 

Abstract 

Pretesting methods that are currently used in survey research to develop and improve survey questions have been evolving 

for the past 20 years. While there has been increased focus on issues related to multicultural and multilingual survey design 

in recent years (e.g. Harkness, Van de Vijver, and Mohler [eds.], 2003), there has been relatively little research on the 

appropriateness of specific pretesting methods with respondents from different cultural and/or linguistic groups. 

Previous research has shown that some respondents have a great deal of difficulty with common cognitive interview probes 

and techniques. English-speaking respondents with low educational levels have been shown to have difficulty with both 

paraphrasing and think aloud probes (Willis, 2005; Bickert & Felcher, 1996; Wellens, 1994). Both Spanish and Chinesespeaking respondents have been shown to experience even greater difficulty with translated cognitive interview probes and 

techniques, including the use of think aloud, paraphrasing, process oriented probes, and meaning oriented probes (Pan, 2004; 

Carrasco, 2003; Potaka and Cochrane, 2002 & 2004; Coronado & Earle, 2002; Blumberg & Goerman, 2000; Kissam, et al., 

1993). In some cases these difficulties have been observed regardless of the educational level of respondents. 

This paper presents an approach for continuing the process of tailoring and refining cognitive interview techniques to 

enhance their effectiveness across cultural and linguistic groups, with a focus on Spanish-speaking respondents. The paper 

discusses a plan of research for: 

• Evaluating common cognitive interview techniques and probes translated into Spanish 

• Developing and testing alternative cognitive interview techniques for use with Spanish-speaking respondents. 

Finally, the paper presents preliminary results and findings based on an implementation of this approach. 

1. Introduction 

In recent years, both federal and private survey research institutions throughout the world have recognized an increasing need 

for multilingual survey instruments to allow for the collection of representative data on culturally and linguistically diverse

populations. This is true for both research that crosses national boundaries and research within single countries such as the 

United States. 

In 2000, the U.S. Census Bureau identified 380 categories of languages or language groups within the United States alone 

(Shin & Bruno, 2003). Many of the people who speak these languages have low levels of English language proficiency. In 

2000, the Census Bureau classified 4.4 million households and 11.9 million people as “linguistically isolated.”1

 Spanish was 

the most common non-English language spoken in the home in 2000, with 28.1 million speakers. Almost half of these people 

reported that they spoke English less than “very well.” Chinese was the second most common language spoken in the home 

with 2 million speakers. 

A great deal of research has been done on best practices for the translation of survey instruments (Harkness, et al., 2003; U.S. 

Census Bureau, 2004). In addition, researchers are increasingly recognizing the need to pretest multiple language versions of 

survey instruments to ensure that data collected from different language groups will be comparable. At the same time, little 

 

1

 The Census Bureau defines a linguistically isolated household as “...one in which no person aged 14 or over speaks English 

at least ‘very well.’” A linguistically isolated person is defined as “...any person living in a linguistically isolated household.” 

It should be noted that this definition can include household members under 14 years of age who do in fact speak English 

fluently or very well (Shin and Bruno, 2003). 
specific methodological research has been done to evaluate pretesting methods in non-English languages (see Willis, et al., 

forthcoming, for an exception to this). 

This paper first presents a review of the literature on pretesting multilingual surveys through cognitive interviews. It goes on 

to discuss the methodology and preliminary findings of ongoing Census Bureau research on methodological issues related to 

conducting cognitive interviews in Spanish. Finally, the paper presents recommendations for future research. 

2. Review of Literature on Pretesting Non-English Language Survey Instruments 

2.1 The Pretesting of Multilingual Surveys 

In both public and private survey research institutions throughout the world, it has not been common practice to pretest all 

language versions of surveys before they are fielded (Harkness, 2004; Willis, 2004a; Potaka & Cochrane, 2004). There are a 

number of reasons for this lack of extensive pretesting, but first and foremost is the cost and added time that testing multiple 

versions of a single survey requires. There has historically been a belief that the most literal translation is the best translation 

and for longitudinal surveys, there is often a great deal of pressure to maintain “tried and true” question wording. Many 

research institutions also have a lack of access to experienced staff who are both bilingual and who have expertise in survey 

methodology. Finally, it can be difficult to gain access to respondents who speak the languages in question. 

Many research institutions, including the Census Bureau, are becoming increasingly concerned with addressing these issues. 

In 2000, the Census Bureau made the Decennial Census questionnaire available in five non-English languages: Spanish, 

Chinese, Korean, Vietnamese, and Tagalog. In addition, respondents could request language assistance guides in 49 

languages, which could be used to assist them in filling out the English language questionnaire. 

In addition to translating survey materials into multiple languages, the Census Bureau has a long term goal of expanding the 

pretesting of these materials to include all language versions. In 2003, the Census Bureau released a Pretesting Standard 

which includes requirements for non-English language survey materials. The standard lays out a “minimal standard,” which 

requires that all new questions or changes to questions on Census Bureau surveys be shown to “work.” In other words, that 

they can both be administered properly by interviewers and understood and answered by respondents. The inclusion of nonEnglish language materials in this standard is an important first step in an effort to improve the quality of all of the Census

Bureau’s survey instruments and processes. 

2.2 The Cognitive Interview as a Method for Pretesting Multilingual Surveys 

Many researchers point to the cognitive interview as a potentially powerful tool for pretesting multilingual survey 

instruments (Harkness, et al., 2003; Potaka & Cochrane, 2004; Willis, et al., forthcoming). The cognitive interview is 

basically an in-depth interview in which researchers “...study the manner in which targeted audiences understand, mentally 

process, and respond to the materials we present–with a special emphasis on breakdowns in this process” (Willis, 2005, p. 3). 

While there is no official, standard method for conducting cognitive interviews, there are a variety of widely accepted and 

practiced techniques (Willis, 2005; Gerber, 2004; Pan, 2004; U.S. Census Bureau, 2003). Many of these correspond to the 

cognitive processes emphasized by Tourangeau (1984): “question comprehension, information retrieval, judgment and 

estimation, and response" (Willis, 2004b, p. 24). These methods include think aloud protocols and a variety of probing 

techniques, which can be administered either concurrently while the respondent goes through a questionnaire or 

retrospectively as part of a debriefing discussion. Probes can also be scripted in advance or “emergent,” occurring to the 

interviewer as the interview progresses (Willis, 2005). Some examples of common cognitive interview probe types are: 

meaning oriented probes, paraphrasing probes, process oriented probes and recall probes. 

A think aloud protocol involves a researcher asking a respondent to discuss his or her thought processes while completing a 

survey interview. This technique can be employed with both self-administered and interviewer-administered surveys. 

Cognitive interview probes are used in a number of ways. An example of a meaning oriented probe would be “What does the 

term foster child mean to you in this question?” Another way for a researcher to examine a respondent’s interpretation of a 

question is to ask him or her to paraphrase the question. A typical paraphrasing probe might be, “Can you tell me in your own 

words what that question is asking?” Process oriented probes help the researcher to examine a respondent’s decisions or 

calculations required to provide an answer to a question. A typical process oriented probe would be, “How did you arrive at 
or choose that answer?” Finally, a typical recall probe would be “How do you remember that you moved into your home in 

1976?” 

5.3 Problems with Cognitive Interview Techniques: Monolingual, English Surveys

A number of researchers have found that cognitive interview respondents can experience difficulty with common cognitive 

interview techniques. For example, Bickart and Felcher (1996) and Wellens (1994) found that respondents of low educational 

levels sometimes have difficulty producing think aloud protocols. In Wellens’ case, respondents had varying levels of 

English proficiency but were interviewed in English. Based on a review of the literature on cognitive testing, Willis (2005) 

points out respondents with low educational levels often have difficulty paraphrasing. Finally, Miller (2003) found that low 

income and low educational level respondents often lack experience with survey interviews and can have generalized 

difficulty with the cognitive interview process. 

5.4 Problems with Cognitive Interview Techniques: Multilingual Surveys 

Some researchers have found that non-English speaking respondents can have difficulty with cognitive interview techniques 

regardless of their educational level. Through cognitive interviews with Chinese-speaking respondents, Pan (2004) found that 

many of her respondents had difficulty with think aloud protocols, process oriented probes and meaning oriented probes. Pan 

found that even respondents with advanced degrees experienced difficulties and she attributed this to the fact that many of 

them were completely unfamiliar with the social context of a survey interview. Pan also attributed some of the difficulties to 

translation issues. Many common English-language cognitive interview probes contain idiomatic expressions and there are 

times when literal translation might not be linguistically or culturally appropriate. 

Many researchers have also encountered problems using common cognitive interview techniques in Spanish (Blumberg & 

Goerman, 2000; Carrasco, 2003; Coronado & Earle, 2002; Kissam, et al., 1993). It should be noted that these studies were 

not designed specifically to examine cognitive interview methodology or techniques; nevertheless, the researchers include 

discussions of difficulties that they encountered in their reports. 

Spanish-speaking cognitive interview respondents have exhibited difficulty with meaning oriented probes and paraphrasing 

and what is even more of a concern, a number of respondents have expressed negative emotions, such as anger, disgust and 

humiliation during the course of the cognitive interview (Coronado & Earle, 2002). A number of researchers discuss the 

respondents’ frequent perception of the cognitive interview as some sort of a “test” (Coronado & Earle, 2002; Kissam, et al., 

1993). Kissam, et al. (1993) reported that some of their respondents asked for the tape recorder to be turned off so as not to

record either their requests for clarification or answers that they perceived to be incorrect. Many of their respondents also 

wondered whether they were the appropriate person to be participating in the research. Additionally, some cognitive

interview respondents have expressed irritation with what they perceived to be repetitiveness in the cognitive interview. 

Finally, some of the researchers discuss the idea that they have had difficulty keeping respondents “on track” and some 

respondents have digressed a great deal. Clearly these interactions could be going more smoothly. The present research aims 

to examine exactly what is happening when the cognitive interview interaction breaks down. 

While the numbers of expert cognitive interviewers who are fluent in non-English languages (particularly Spanish) is 

increasing, many survey research organizations do not have full-time staff members with both survey methodology expertise 

and language skills. It is often still the case that these agencies face the necessity to train bilingual people to conduct 

cognitive interviews in non-English languages (Willis, forthcoming). This situation often leads to a cognitive interview 

protocol being created in English and translated into other languages. Some inexperienced interviewers have been known to 

read the interview protocol exactly as worded with little flexibility or adaptation. This would not be considered a “best 

practice” for cognitive interviewing regardless of language and this type of factor may explain some of the difficulties 

encountered by researchers. 

6. Research Questions 

The present research is designed to evaluate the use of common cognitive interview probes and techniques in Spanish and to 

develop and test alternative techniques. It examines cognitive issues, status issues and sociolinguistic issues, and includes a

number of research questions. For example, 

1. What kind of introduction works best to explain the purpose of the cognitive interview to different respondents? 
2. How do perceived status differences between the interviewer and respondent affect rapport between the two parties? 

3. What are the most effective ways to minimize perceived status differences between interviewer and respondent? 

4. Does the interaction go more smoothly when the interviewer engages in small talk before starting the interview? 

5. What types of respondents have more difficulty with common probe wording? 

6. Are there alternative probes that work best for specific types of respondents? 

The findings from this study should add to our “toolkit” of cognitive interview probes and techniques and may also offer 

suggestions as to the type of training that would be useful for new cognitive interviewers who will conduct interviews in nonEnglish languages. 

7. Research Methodology 

In order to examine the effectiveness of common cognitive interview techniques, the present study includes both “structured” 

and more “experimental” cognitive interviews with monolingual, native Spanish-speakers of both high and low educational 

levels (see Table 1). The high educational level group consists of respondents who have completed 12 or more years of 

schooling (high school in the United States or la preparatoria or preparatory school in Mexico). The lower educational level 

group consists of respondents who have completed less than 12 years of schooling. To reduce variability due to national 

origin and/or ethnicity, all respondents in this study are of Mexican origin. Respondents also vary by age, gender and length 

of time in the United States. 

Table 1: Respondents’ Educational Level by Type of Interview 

Educational Level Structured interviews Experimental interviews 

Low level (less than la preparatoria or 12 yrs.) 10 13 

High level (La preparatoria or 12 yrs. or more) 10 13 

Total Respondents 20 26 

For the structured interviews, probes were translated as directly as possible from English to Spanish. We examined standard 

cognitive interview protocols used by the Census Bureau and included the same introduction, explanations and types of 

probes. In order to document what might be going wrong in some cognitive interview studies, we are using scripted probes 

for the structured interviews and the interviewers are deviating as little as possible from the scripted protocol. In addition we 

are collecting extensive background information about each respondent, including their familiarity and experience with 

surveys. 

The experimental interviews are being conducted using less scripted probes and more emergent probes. For these interviews, 

the interviewers spend more time with respondents prior to the interview engaging in “small talk” to increase rapport, 

discussing the work of the Census Bureau and explaining the purpose of the cognitive interview. The researchers also provide 

time for respondents to ask questions or bring up concerns that they might have before the interview begins. The 

experimental interviews also contain a sample think aloud protocol in which the researcher provides the respondent with an 

example of how to think aloud. In addition, the researcher goes through a sample of a meaning oriented probe and explains 

why such questions might be asked during the interview. 

The interviews are being conducted by two Spanish speakers. One is an experienced cognitive interviewer who is a native 

English-speaker but who speaks Spanish fluently. The second interviewer is a novice interviewer who is a native Spanishspeaker. All interviews are being conducted using a segment of the Spanish version of the Computer Assisted Personal 

Interview (CAPI) version of the Census Bureau’s American Community Survey (ACS).2

 The interview protocols for both 

the structured and experimental interviews were created based on the same group of survey questions. 

All interviews are being coded for factors such as whether or not a probe was understood the first time it was heard, whether 

respondents experienced discomfort during specific parts of the interview, and whether probes elicited useful information for 

the evaluation of the survey questions. 

 

2

 The ACS is a new nationwide survey designed to provide more frequent data than the Decennial census. It will replace the 

long form in future censuses. 
8. Preliminary Findings 

To date we have conducted one structured pilot interview and 20 experimental interviews. The pilot interview was conducted 

in the Washington DC area and the experimental interviews were conducted in Texas. All remaining interviews, both 

structured and experimental, will be conducted in Texas. 

8.1 The Structured Interview 

During the planning phase of the research, we conducted one structured pilot interview with an immigrant from Bolivia in the 

Washington DC area. A number of problems arose during the course of the interview and a description of the interaction is 

illustrative of the types of issues that can arise. 

The interviewer greeted the respondent by exchanging pleasantries, asking the respondent how she was doing and 

commenting on the weather. The interviewer did not talk in depth about herself, the Census Bureau, or the cognitive 

interview process. The respondent seemed nervous at the start of the interview. She had assumed that the person who had 

recruited her to do the interview would also be present during the interview and seemed disconcerted to find out that she 

would be alone with the interviewer. 

The respondent was reluctant to sign an informed consent form and asked if it would be okay to sign it after the interview. 

She expressed concern about the types of questions that the interviewer would be asking and wondered how she could give 

consent without knowing what the interview questions were in advance. The interviewer explained that the form was 

basically needed to gain the respondent’s permission to tape record the interview and needed to be signed prior to the 

interview. The interviewer spent time discussing the confidentiality of the interview and assured the respondent that she 

could decline to answer any question that made her uncomfortable. In addition, the interviewer told the respondent that if she 

felt uncomfortable after the interview the tape could be erased and she could be excused from the study. 

The interviewer began to talk about the interview process and to give standard instructions. The respondent seemed 

uninterested and asked the interviewer to continue, saying, “Siga nomás” or “Just go on.” 

There were a number of problems with this cognitive interview interaction. First of all, when the interviewer read survey 

questions that were related to things that the respondent had already alluded to, the respondent became extremely irritated. 

For example, in her answer to one of the survey questions, the respondent said that she lived in an efficiency apartment. A 

subsequent question asked whether anyone else lived or stayed with the respondent. To that question she replied, “No. From 

where? I am [in] an efficiency, I’m in the building, there’s nobody there, just me, and different tenants and different rooms. 

Am I clear?” 

A number of problems with the standard cognitive interview probes arose as well. Probably as a result of the earlier 

conversation regarding the signing of the consent form, this respondent was eager to “pass” on some of the interview 

questions. Interestingly, she passed on the actual cognitive interview probes as opposed to any survey questions that asked for

personal information. For example, when asked to describe her race, the respondent chose the term “blanca” or “white.” 

When asked a process oriented probe, “How did you choose that answer?” She replied, “That’s just what it is.” The 

interviewer then asked a meaning oriented probe, saying “What does the word ‘white’ mean to you in that question?” The 

respondent replied, “Pass.” 

An additional issue that arose with this respondent was related to the characteristics of the cognitive interviewer. This 

respondent was interviewed by the non-native Spanish speaking interviewer. The respondent apparently interpreted some of 

the meaning oriented probes to be an indication that the interviewer was having trouble with the Spanish language terms and 

therefore sometimes provided an English translation for the term in question. For example, when asked “¿Qué significa la 

palabra ‘se queda’ para usted en esta pregunta?” or “What does the word ‘stays’ mean to you in this question?” She replied, 

“Stays” in English. This interview clearly contained some breakdowns in communication between the interviewer and the 

respondent. 

8.2 Experimental Interviews 

At present we have conducted a total of 20 experimental interviews with Mexican origin respondents in Texas. All of these 

interviews were conducted by the non-native Spanish speaking interviewer. Sixteen of the respondents were women and four 
were men. They ranged in age from 19 to 68 with the average age being 42.8 years old. The respondents ranged from having 

had 4-14 years of schooling, with the average being 10.4 years. Eleven respondents had lower than a high school level

education and nine had higher than a high school level education. The interviews were conducted in an office on a university 

campus. For these interviews, we tried a variety of techniques both to see if we could make the interactions go more 

smoothly and to see if we could improve the collection of useful information for the evaluation of the survey questions. 

Pre-Interview Interaction 

First of all, the interviewer met respondents in the hallway as they arrived for the interview. The interviewer generally offered 

to shake hands with respondents but a few respondents leaned forward to kiss the interviewer on the cheek. When this 

happened, the interviewer went along with this more typical Mexican greeting. It was most often older women respondents 

who greeted the interviewer in this manner. 

When they entered the interview room, the interviewer offered respondents a glass of water. Before beginning the interview, 

the interviewer spent time engaging in “small talk” with the respondents in order to establish good rapport. The interviewer 

often initiated conversation about the interview location, discussing the difficulty in finding parking in the area. She also told 

respondents that she was visiting from out of town and complimented them on their area. In many cases a respondent shared 

information about himself or herself and in some cases the interviewer did so as well. The interviewer discussed her family 

background, how she had learned Spanish and her work at the Census Bureau. The interviewer also asked respondents if they 

had heard about the Census Bureau and talked about the types of surveys that the agency conducts. She also discussed uses of

survey data and explained that she was not collecting data, but rather testing out a questionnaire. 

Prior to the interview, some respondents expressed nervousness or concern that they were not the “right” person to be 

participating in the study. The interviewer explained that she hoped to talk to as many different kinds of people as possible 

and reassured each respondent that he or she was exactly the person with whom she wanted to speak. 

The interviewer noticed that occasionally, when the “small talk” touched on personal issues, topics that would be included in 

the survey questions came up. She deemed that this was not ideal, so in later interviews personal topics were avoided a bit 

more. 

Interview Instructions and Procedures 

Once the experimental interviews started, the interviewer went through a sample think aloud protocol with the respondents, 

as recommended by Gerber (2004). The interviewer gave respondents a sample question, “How many windows are there in 

your home?” The interviewer then gave a sample response, such as “Let’s see, there are two in the bedroom and two in the 

living room, so that’s four. There’s also a small window on the top of my front door that looks out into the hallway of my 

apartment building but I don’t know if I should count that. I guess I will count it, so my answer is five.” 

The interviewer went on to give a sample of a meaning oriented probe along with an explanation of why such a question 

might be asked. She told respondents that during the course of the interview she might ask questions that would sound a little 

bit strange, such as, “What does the word ‘window’ mean to you in this question?” She then explained that the reason she 

might ask something like that is that some people might include a small window at the top of their front door while others 

might not. She also explained that there was no correct answer for this type of question and that she was really trying to learn 

about individual people’s interpretations of different terms or ideas to help in designing the survey. 

Measuring the Effectiveness of Cognitive Interview Techniques and Probes 

The coding and data analysis of the completed interviews are still underway, but this section offers some preliminary 

observations. Overall, respondents in the experimental interviews seemed to experience very little difficulty with the 

cognitive interview process. 

With think aloud protocols, these 20 respondents had very little difficulty. A few lower educational level respondents seemed 

to have difficulty understanding the concept of thinking aloud, but in general most people were able to do so. A few lower 

educational level respondents read aloud when looking at a flashcard, rather than discussing what they were thinking, but this 

did not cause a great deal of confusion. 
Regarding meaning oriented probes, the presence of a sample probe up front and the acknowledgement by the interviewer 

that the questions might sound strange seemed to be extremely helpful. Almost all of the respondents were able to answer the 

meaning oriented probes in a meaningful way, and while a few of these respondents knew some English, none of them gave 

English translations for the Spanish terms in question. In addition, none of the respondents expressed the feeling that they 

were being “tested” during the cognitive interview process. 

Paraphrasing probes did cause respondents some difficulty. The paraphrasing probes were originally translated as: ¿Podría 

decirme con sus propias palabras qué información pide esa pregunta?” or “Could you tell me in your own words what 

information this question is asking for?” Many respondents stared at the interviewer blankly when asked that question or 

seemed deep in thought. When the interviewer rephrased the question as, “Qué le parece que están preguntando allí?” or 

“What do you think they’re asking there?” or “¿Por qué le parece que preguntan eso?” or “Why do you think they’re asking 

that?” respondents were able to answer with much greater ease. For future interviews, I would recommend the testing of a 

sample paraphrasing explanation or probe to practice with respondents prior to an interview. 

Caveat on my Respondents 

While most of the 20 respondents in Texas were monolingual Spanish speakers, a number of them had lived in the U.S. for 

up to 20 or 30 years. Some of them knew more English than they had reported in the recruitment screening phase of the 

study, perhaps due to cultural norms regarding modesty. Many of them were also quite familiar with U.S. forms and surveys, 

so these respondents might not be typical of the larger immigrant population in the U.S. 

9. Next Steps for the Current Study 

Next steps in this research project include interviewing 20 additional Mexican origin respondents of high and low educational 

levels in the same region in Texas using a more structured protocol. The findings from the second round of interviews will be 

used to examine interviewer/respondent interaction and a host of other factors when done through a structured approach. A 

more structured approach is not highly recommended in the field, but it appears to be common in the testing of non-English 

versions of surveys in the U.S. due to the difficulty of finding experienced, bilingual survey methodologists. 

10. Recommendations for Future Research 

It would be useful to conduct similar research with English speakers of low educational levels and with respondents from 

additional language/cultural groups in order to see if similar experimental techniques will be more or less successful with 

different kinds of respondents. 

This type of approach could also be applied to additional pretesting methods such as focus groups, behavior coding and 

respondent or interviewer debriefing in order to tailor these additional techniques for use in testing multilingual survey 

instruments. With increasing diversity within countries due to globalization and immigration and with the increased need to

conduct international surveys, the comparability between versions of multilingual surveys will only increase in importance in 

the coming years. 
References 

Bickart, B., & Felcher, E.M. (1996). Expanding and enhancing the use of verbal protocols in survey research. In N. Schwartz 

& S. Sudman (Eds.) Answering Questions: Methodology for Determining Cognitive and Communicative Processes 

in Survey Research. (Pp. 115-142). San Francisco: Jossey Bass. 

Blumberg, R.L. and Goerman, P. L. (2000). Family complexity among Latino immigrants in Virginia: An ethnographic study 

of their households aimed at improving Census categories.” Report submitted to the United States Census Bureau. 

Briggs, C. (2001). Assessment Report: Language Program. Unpublished Census Bureau Report (November 20, 2001). 

Carrasco, L. (2003). The American Community Survey (ACS) en español: Using cognitive interviews to test the functional 

equivalency of questionnaire translations. Study Series: Survey Methodology #2003-17. Methodology and Standards 

Directorate, U.S. Census Bureau. 

Coronado, I., and Earle, D. (2002). Effectiveness of the American Community Survey of the U.S. Census in a borderlands 

colonia setting.” Draft report submitted to the U.S. Census Bureau, March 19, 2002. 

Gerber, E. (2004). U.S. Census Bureau Official Cognitive Interview Training Course. (Unpublished materials). 

Gerber, E. and Pan, Y. (2004). Developing cognitive pretesting for survey translations. Paper presented at the Second 

International Workshop on Comparative Survey Design and Implementation (CSDI). April 1-3, 2004. Paris, France. 

Harkness, J.A. (2004). Problems in establishing conceptually equivalent health definitions across multiple cultural groups. 

Paper presented at the Eighth Conference on Health Survey Research Methods. Cohen, S.B., Lepkowski, J.M. 

 (eds.) Hyattsville, MD: National Center for Health Statistics, 2004. 

Harkness, J.A., Van de Vijver, F.J.R., Mohler, P. Ph. (eds.) (2003). Cross-Cultural Survey Methods. Hoboken, New Jersey: 

John Wiley & Sons 

Hunter, J. and Landreth, A. (2005). Behavior coding analysis report: Evaluating bilingual versions of the Non-Response 

Follow-Up (NRFU) for the 2004 Census Test.” U.S. Census Bureau report, submitted February, 18, 2005. 

Jabine, T.B., Straf, M.L., Tanur, J.M. & Tourangeau, R. (1984). Cognitive Aspects of Survey Methodology: Building a bridge 

between disciplines. Report of the Advanced Seminar on Cognitive Aspects of Survey Methodology. Washington, 

DC: National Academy Press. 

Kissam, E., Herrera, E., Nakamoto, J.M. (1993). Hispanic response to Census enumeration forms and procedures.” Contract 

Report submitted to the U.S. Census Bureau, March, 1993. 

McGovern, P.D. (2004). A quality assessment of data collected in the American Community Survey (ACS) from households 

with low English proficiency. Study Series: Survey Methodology #2004-01. Methodology and Standards 

Directorate, U.S. Census Bureau. 

McNally, J.W. (2000). Overweight and underexposed: The measurement of non-English speaking populations in the U.S. 

representative surveys.” Paper prepared for the National Institute of Child Health and Human Development Institute

on Aging Conference on Inclusion of Language-Minority Populations in National Studies: Challenges, 

Opportunities and Best Practices. Bethesda, MD, July 27-28, 2000. 

Miller, K. (2003). Conducting cognitive interviews to understand question-response limitations among poorer and less 

educated respondents. American Journal of Health Behavior Vol. 27, (3) pp. 264-272. 

Pan, Y. (2004). Cognitive interviews in languages other than English: Methodological and research issues. Paper presented at 

the American Association for Public Opinion Research conference, May 13-16, 2004, Phoenix, AZ. 2004. 

Proceedings of the American Statistical Association (to appear). 
Pan, Y. (2003). The role of sociolinguistics in the development and conduct of federal surveys.” Proceedings of the Federal 

Committee on Statistical Methodology Research conference. November 17-19, 2003, Arlington, VA. 

Potaka, L. and Cochrane, S. (2004). Developing bilingual questionnaires: Experiences from New Zealand in the development 

of the 2001 Maori language survey. Journal of Official Statistics, Vol. 20, No. 2, pp. 289-300. 

Potaka, L., Cochrane, S. (2002). Developing bilingual questionnaires: Experiences from New Zealand in the development of 

the 2001 Maori language survey.” Paper presented at the International Conference on Questionnaire Development, 

Evaluation, and Testing Methods (QDET), Charleston, SC, November 14-17, 2002. 

Presser, S., Rothgeb, J., Couper, M.P., Lessler, J.T., Martin, E., Martin, J. & Singer, E. (eds). (2004). Methods for testing and 

evaluating survey questionnaires. Hoboken, NJ: John Wiley & Sons, Inc. 

Rogler, L.H. (1999). Methodological sources of cultural insensitivity in mental health research.” American Psychologist, Vol. 

54, No. 6, pp. 424-433. 

Shin, H.B. & Bruno, R. (October 2003). Language use and English-speaking ability: 2000. Census 2000 Brief. U.S. 

Department of Commerce, Economics and Statistics Administration, U.S. Census Bureau. Retrieved from 

http://www.census.gov/prod/2003pubs/c2kbr-29.pdf on 2/15/05. 

Sirken, M.G., Herrmann, D.J., Schechter, S., Schwartz, N., Tanur, J.M., and Tourangeau, R. (eds.) (1999). Cognition and 

survey research. New York: John Wiley & Sons, Inc. 

Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective, in Jabine, T. Straf, M. Tanur, J. and 

Tourangeau, R. (eds.) Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, 

DC: National Academy Press. 

United States Census Bureau (2004). Census Bureau guideline: Language translation of data collection instruments and 

supporting materials. U.S. Census Bureau, U.S. Department of Commerce, Washington, D.C. 

United States Census Bureau (2003). Census Bureau standard: Pretesting questionnaires and related materials for surveys and 

censuses. Methodology and Standards Directorate, U.S. Census Bureau, U.S. Department of Commerce, 

Washington D.C., July 25, 2003. 

Wellens, T. (1994). The cognitive evaluation of the nativity questions for the Current Population Survey. Proceedings of the 

Section on Survey Research Methods, American Statistical Association. Pp. 204-1209. 

Willis, G., Lawrence, D., Thompson, F., Kudela, M., Levin, K., & Miller, K. (forthcoming). The use of cognitive 

interviewing to evaluate translated survey questions: Lessons learned. Proceedings of the Federal Committee on 

Statistical Methodology Research conference. November 14-16, 2005, Arlington, VA. 

Willis, G.B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage 

Publications. 

Willis, G. (2004a). Overview of methods for developing equivalent measures across multiple cultural groups.” Paper 

presented at the Eighth Conference on Health Survey Research Methods. Cohen, S.B., Lepkowski, J.M. (eds.) 

Hyattsville, MD: National Center for Health Statistics, 2004. 

Willis, G. (2004b). Cognitive Interviewing Revisited: A Useful Technique, in Theory? In Presser, S., Rothgeb, J., Couper, 

M.P., Lessler, J.T., Martin, E., Martin, J. & Singer, E. (eds). Methods for testing and evaluating survey 

questionnaires. Hoboken, NJ: John Wiley & Sons, Inc.