**Spring, 2007 Advisory Committee Meetings**

**Research on Questionnaire Development and Language**

Manuel de la Puente, ADC for Survey Methodology

Statistical Research Division

[\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}

[Introduction]{.underline}

Since the early 1990s the Census Bureau has been researching survey
respondents who speak a language other than English and how they
interpret and answer survey questions.

This document focuses on the qualitative language research conducted or
sponsored by the Census Bureau as part of its questionnaire pretesting
research. The paper has two major sections. The first covers research
conducted using the decennial census Spanish language questionnaire and
the second highlight cognitive research conducted to pretest American
Community Survey (ACS) informational materials in five languages.

The aim of this document is to provide a brief overview of the type of
the qualitative language research conducted by the Census Bureau in the
recent past. This document also seeks the advice of Census Advisory
Committee members by posing key questions on future research needed to
inform the Census Bureau's ongoing language research program.

1.  [Decennial Census Research]{.underline}

*1.1* *In depth interviews conducted by Aguirre International*

> In 2003, the Census Bureau hired Aguirre International to conduct 60
> in-depth interviews with Spanish speaking respondents of different
> national origins in order to better understand how Spanish speakers
> would interpret and respond to census questions. In addition to
> interviews, this study also relied on nine focus groups to examine how
> Spanish speaking respondents view the census process. The in-depth
> interviews explored specific semantic, syntactic, and conceptual
> issues affecting how Spanish speakers respond to the Spanish language
> 1990 census long form. Results showed that Spanish speakers with less
> than 6 years of formal education found it very difficult to complete
> the 1990 Spanish language census form without assistance. Most Spanish
> speaking respondents (especially those with low levels of formal
> education) encountered syntactic (or phrasing) and conceptual
> difficulties with key questions on the long form. These included the
> year of entry, home language and schooling questions. Lexical (or
> vocabulary) difficulties were encountered in some of the housing
> questions. Lastly, although the research showed that low-literate
> respondents had the most difficulty with the Spanish language forms,
> intermediate or level literate respondents also encountered numerous
> syntactic and lexical problems. This study helped the Census Bureau
> identify specific language and conceptual issues that Spanish speaking
> respondents encountered in the Spanish language questionnaire.

2.  *Sociolinguistic analysis of Hispanics of Mexican origin*

> In 1990 the Census Bureau commissioned a sociolinguistic study of
> Mexican origin Hispanics in Chicago, Ill. A total of 39 in-depth
> ethnographic interviews were conducted with both English and Spanish
> speaking Hispanic respondents of Mexican origin. While the major focus
> of this study was to elicit the views and attitudes of respondents
> toward the decennial census, the study used the 1990 short and long
> census forms (in both English and Spanish) to determine how study
> participants responded to questions on the form. Most respondents in
> this study were bilingual and could technically have filled out the
> English language forms. Twenty four of the 39 respondents chose to use
> the Spanish language census form. Virtually all had low functional
> literacy skills in both English and Spanish. Numerous problems were
> identified with vocabulary and terms used in the Spanish language
> census form. This research provided us with empirical evidence that
> improvements need to be made to the Spanish language census form.

3.  *Ethnographic exploratory research*

> During the 1990 Census the Census Bureau commissioned 29 ethnographic
> studies across 29 different communities. Twenty eight study sites were
> located in the continental U.S and one site was situated in Puerto
> Rico. The principal goal of this research was to identify and explain
> the reasons for omissions and other erroneous enumerations in the
> census. A number of the study sites were situated in communities with
> high concentrations of non-English language speakers. This research
> identified specific factors that help explain census errors. One of
> these factors was little or no knowledge of English. This research
> provided us with research-based evidence that little or no knowledge
> of English can be a barrier to census enumeration.
>
> During the year 2000 ethnographic studies in six research areas were
> conducted in conjunction with Census 2000. Of these six research areas
> the ethnographies of two of the areas -- Spanish speaking respondents
> living in colonias and Spanish speaking respondents living in complex
> households in Central Virginia\-- provided the Census Bureau with
> additional insight into how Spanish speakers with little or no
> knowledge of English understand and respond to questions on the census
> form. Colonias are generally unincorporated low-income residential
> subdivisions, lacking basic infrastructure and services along the
> border between the U.S. and Mexico. These ethnographies demonstrated
> that little or no knowledge of English and low levels of literacy (in
> both English and Spanish) presented many challenges to respondents and
> to Census Bureau enumerators. In Census 2000 interviews in colonias
> were conducted in-person because a regular address is often lacking.
> Ethnographers on the site observed that some enumerators had
> difficulty using Spanish language translation provided and some
> respondents had difficulty understanding key words and concepts.
>
> The second relevant research area \-- complex households - focused on
> recent Hispanic immigrants. The researchers conducted in-depth
> interviews aimed at getting information on complex household
> structure. During the in-depth interviews respondents were asked to
> fill out the Spanish language Census 2000 short form. Of interest to
> the researchers was the relationship question on the census form and
> the extent to which this question was able to capture the complexity
> of many of the households in the study. In the process of conducting
> their research the researchers identified words and phrases in the
> Spanish language form that were not readily understood by some
> respondents. This research served to further document the difficulties
> that some Spanish speaking respondents have with the Spanish language
> census form.

4.  *Cognitive research and focus groups on bilingual (English/Spanish)
    census questionnaire format*

> In early 2002 the Census Bureau began to explore having a bilingual
> (Spanish/English) language Census form Census 2010. In 2003, we hired
> RTI International to conduct cognitive interviews and focus groups to
> pretest four bilingual census form options. These options were: A
> "swim lane" form (with English in the left hand column and Spanish in
> the right hand column on the same page) an "embedded booklet" (with
> English language text directly followed by Spanish language text on
> the same page) a separate forms option with one form in English and
> one in Spanish included in the same mailing package and a back-to-back
> form (a form with English on one side of the page and Spanish on the
> other). The main goal of this research was to determine which
> bilingual form respondents found easier to navigate and use. To
> address this issue RTI conducted a total of eight focus groups and 96
> cognitive interviews in Chicago, Dallas, Los Angeles, and Miami. Using
> these methods RTI researchers identified both the strengths and
> weaknesses of all four bilingual census form options. Results showed
> that the swim lane form was rated most favorably by respondents and
> appeared to cause fewer problems for most participants. This research
> provided the Census Bureau with empirical guidance on developing a
> bilingual census form.
>
> *1.5 Cognitive interviews on the bilingual swim lane form*
>
> Additional research on the swim lane form was conducted in 2006 and
> 2007 when Census Bureau researchers collaborated with researchers from
> RTI to conduct two rounds of cognitive interviews. The aim of the
> first round of cognitive interviews was to cognitively test the
> Spanish-language translation contained in the questionnaire. In this
> round a total of 44 cognitive interviews were conducted in Chicago,
> Miami, Los Angeles and Raleigh/Durham. Twenty of these interviews were
> conducted with monolingual Spanish speakers and 24 were conducted with
> Spanish speaking respondents with minimal knowledge of English. Round
> one results provided information on the extent to which Spanish
> speakers understand the Spanish translation used in the swim lane
> form. Probing was done on selected terms and concepts and research
> results led to recommendations for alternative wording of a number of
> key Spanish language words and phrases. A number of Spanish language
> terms were changed as a result of the testing.
>
> After round 1 recommendations were evaluated and implemented the same
> Census Bureau and RTI research team conducted a second round of
> cognitive testing. The focus of the second round of cognitive testing
> was broader. That is, in addition to testing the new Spanish language
> wording and respondents' understanding of census questions, cognitive
> testing in round 2 examined how respondents opened and handled the
> mailing package which includes cover letter, the questionnaire and a
> return envelope. Round 2 was also designed to provide insight
> regarding why selected items on the bilingual form used in the 2005
> Census Test had a higher rate of missing data than the English only
> census form. In round 2 a total of 66 Spanish and English language
> cognitive interviews were conducted across the same sites as in
> round 1. Round 2 research provided a better understanding of wording
> and formatting issues of the bilingual questionnaire as well as
> information on how to improve the mailing package envelope and
> introductory letter. A number of Spanish language terms are being
> changed based on the results of Round 2 of the cognitive testing and
> two versions of the bilingual form will be field test in 2007. Results
> will help inform the 2010 bilingual census form.

6.  *Cognitive interviews and behavior coding of the English and Spanish
    versions of the NRFU instrument for the 2004 and 2006 Census Test*.

> In preparation for the 2004 and 2006 Census Test, Census Bureau
> researchers and contractors conducted cognitive testing of the Spanish
> and English language text used in the Hand Held Computing (HHC) device
> used to collect data in the Census Bureau's Non Response Follow Up
> (NRFU) operation.
>
> In addition the Census Bureau conducted behavior coding of the 2004
> and 2006 NRFU instruments in the field during the Census tests in both
> Spanish and English. In 2004 a total of 119 English language, 72
> Spanish language, and 29 English/Spanish NRFU interviews were audio
> taped in the NRFU operations conducted in Queens, New York. The
> interviews were behavior coded to keep track of how often they made
> change to question wording, how often respondents were unable to
> answer questions without further assistance, etc. Findings specific to
> the Spanish language text indicate that Spanish language questions had
> consistently lower rates of ideal interview behavior than the English
> language counterparts. That is, interviewers using the Spanish
> language NRFU automated instrument tended to skip questions or not
> read questions as worded more often than interviewers who used the
> English language NRFU automated instrument.
>
> The NRFU instrument was revised based on results from the 2004
> behavior coding study. The revised instrument was used in the 2006
> Census Test. In 2006, Census Bureau researchers conducted behavior
> coding research based on 72 audio taped interviews (54 interviews were
> in English and 18 were in Spanish) from the Austin, TX test site. The
> findings indicated that despite changes that were made in the
> automated NRFU instrument between 2004 and 2006, there was still a
> higher rate of non-ideal interviewer and respondent behavior in the
> Spanish language interviews.
>
> *Question: Given the qualitative pretesting research described above,
> what does the committee recommend as the Census Bureau's next steps
> for qualitative research in this area?*

[2.0 American Community Survey Research]{.underline}

*2.1 Cognitive testing of American Community Survey (ACS) informational*

*materials in five languages*

> In 2006 researchers from the Census Bureau and RTI collaborated on the
> cognitive testing of four ACS documents that are used by interviewers
> during personal visit follow-up. The test included two letters; one
> that introduces the survey and includes important informed consent
> messages and a thank you letter. We also tested two brochures; a short
> informational brochure and a detailed brochure that included questions
> and answers about the survey.
>
> These materials were translated from English into Spanish, Chinese,
> Russian, and Korean, and Vietnamese. There were two main study
> objectives. The first objective was to evaluate whether respondents
> who use translated documents have a similar understanding of the
> intended communication as English-speaking respondents. The second
> objective was to identify the informed-consent messages that are
> conceptually difficult to translate and to develop strategies to
> overcome these difficulties.
>
> Both the English language materials and the non-English language
> materials were cognitively tested in the target languages. A total of
> 112 cognitive interviews were conducted across the five languages.
> These interviews were conducted in the greater Washington DC area,
> Chicago, IL, and North Carolina. 2004 ACS data were used to identify
> the characteristics of the persons who would most likely be exposed to
> these materials. These characteristics were used to develop recruiting
> profiles. Probing was done on key informed-consent messages contained
> in these materials.
>
> This research showed that some of the wording problems identified in
> the non-English target languages stemmed from problems in the wording
> of the English version. In many cases respondents' comprehension was
> compromised because of problems associated with the translation of the
> materials. While language specific problems were identified, most of
> these problems could be grouped under three categories: inaccurately
> translated terms, overly complex sentence structures and vocabulary
> choices, and inappropriate rhetorical styles. Numerous changes were
> made to the translated materials based on the results of the testing
> and a second round of interviews tested these changes before they were
> implemented Research results led to recommendations for changes in the
> English original materials as well as in the translations. Findings of
> this study will help improve Census Bureau translations and the
> translation pretesting process.
>
> *Question: While we are very happy with the improvements in the
> translations and the lessons learned about translating informed
> consent messages, this testing required a great amount of time and
> effort. Timing and budget constraints often limit our ability to
> conduct such a thorough testing cycle. What alternatives to this
> methodology would you suggest when time and budget impose such
> limitations?*