Two Sides of a Single Coin: Dimensions of Change

Suggested in Different Settings

Luann Moy, U.S. General Accounting Office

Linda L. Stinson, Bureau of Labor Statistics

Luann Moy

R C E Division

U.S. General Accounting Office,

Washington, DC 20548

telephone: 202-512-6852 FAX: 202-512-6171

MoyL.RCED@GAO.gov

Paper prepared for the Joint Statistical Meetings in Baltimore, Maryland.

Any opinions expressed in this paper are those of the authors and do not

constitute policy of the U.S. General Accounting Office or the Bureau of Labor

Statistics.

August, 1999
Abstract

In March, 1998, the Office of Management and Budget (OMB) granted approval to test the wording and

format of materials designed to collect aggregate ‘race’ and ‘Hispanic origin’ data according to new

standards. The first step in the testing process was to develop three different forms, each approaching the

problem of aggregate reporting in a slightly different way. The second step was to test the three forms to

determine their strengths and weaknesses and, ultimately, to select a single form to present to OMB.

The first waves of testing were conducted in cognitive labs at the Bureau of Labor Statistics (BLS) and

the National Center for Health Statistics (NCHS) by professional staff from 3 different agencies: BLS,

NCHS, and the U.S. General Accounting Office (GAO). After assessing the information collected in the

lab about the three forms, the next step was to take the forms into the field for cognitive testing in the

workplace. This paper presents the results of three waves of testing. It also considers the differences and

similarities in the results obtained from cognitive testing in different settings and reviews the possible

sources of those differences and similarities in findings. In addition, we will present a brief summary of

the positive and negative aspects of doing cognitive research using an interagency team and multiple labs.
1

Two Sides of a Single Coin? Dimensions of Change

Suggested in Different Settings

Luann Moy, U.S. General Accounting Office, Washington, DC 20548

Linda Stinson, Bureau of Labor Statistics, Washington, DC 20212

Key words: cognitive testing, race/ethnicity reporting, forms development

I. Introduction

The survey literature is filled with examples of both directed and non-directed probing questions,

generally referred to as “cognitive interview” techniques, used successfully to study respondents’

understanding of and reaction to the content of questions (e.g., De Maio and Rothgeb, 1996, Hess and Singer,

1995; Bates, 1992; and Esposito et al., 1991). Likewise, there has been much discussion about the reliability

of these methods, including reviews of how and when they are used, debates over the use of “cognitive”

techniques versus other pretest methods, and descriptions of specific protocols used in laboratory and on-site

establishment settings (Willis, et. al., 1999, Presser and Blair, 1994, and de Maio and Rothgeb, 1996, Tucker,

1997). However, much of the discussion on the use of cognitive methods is focused on data collection at the

household level and only a few authors have focused on the usefulness of this technique for establishment

surveys (Goldberg, et. al., 1993).

Turning specifically to the literature of establishment surveys, the emphasis is often placed on the

various protocols used to identify and gather information from knowledgeable officials. For example, Edwards

and Cantor (1991) developed and described a response model for establishment surveys that treats the

establishment’s response as an ‘information system’ in which the respondent is asked to respond on behalf of

the establishment. 

The actual interaction between the respondent representing the establishment and the

form/questionnaire requesting information from that establishment may be problematic. In many instances, the

information required by the form/questionnaire is neither collected nor presented in a similar manner. This

represents what Edwards and Cantor describe as the record formation phase in establishment surveys--

comparable to the cognitive encoding process for individual respondents. As they pointed out, "...the
2

respondent must have encoded information about how to get the answer to a question. If the response is

obtained from an information system, some set of steps must have occurred for the information to be in the

system."(p. 218). The authors go on to comment that respondent selection in establishment surveys is also a

critical element, since establishment information tends to be highly compartmentalized and it is essential that

the most knowledgeable respondent be identified and contacted.

Since establishment forms/questionnaires often request information that is retrieved from records,

pretesting of the instruments is often conducted on-site at the establishment, rather than in a cognitive lab (de

Maio and Rothgeb, 1996). The decision to go on-site to establishments is grounded in the belief that the

information retrieval processes used by establishments can and will affect the quality of the responses. That is

to say, dynamic processes such as the (a) records retrieval systems, (b) availability of colleagues for assistance,

and (c) general working conditions can have as much or more influence on the quality of establishment

responses than the individual who actually fills out the form/questionnaire. As pointed out by Edwards and

Cantor (1991), the impact of such environmental forces (which also may be at work in regard to household

surveys) are magnified for establishments surveys: "Thus, an establishment's environment may have even

more effect on the survey response process than does the household respondent's environment on his or her

survey responses." (p. 216)

The belief that cognitive interviews designed to pretest establishment forms/ questionnaires are best done on-site

at the establishment rather than in a laboratory setting has several practical implications. Since it saves travel time, it is

generally less expensive and quicker to conduct pretests in the laboratory. However, if on-site interviews are more

productive in identifying the information retrieval processes linking the individual respondent with records and other

informed individuals within their system, then testing in the laboratory would not really be more efficient. In this paper,

we examine one study where we are able to compare what we learned in the laboratory with our finding on-site at

establishments, in order to reflect upon those conditions under which laboratory testing may be most useful. Using the

proposed Office of Management and Budget (OMB) revised guidelines for administrative reports of aggregated race and

Hispanic origin data, we examine and contrast the information gained from interviews with administrative
3

personnel, in (1) government cognitive laboratories and (2) on-site at establishments. In this way, we hope to identify the

contributions of the two sources of information.

II. Race and Ethnicity Study: Background and Study Design.

Before 1997, the OMB Statistical Directive 15 required that agencies report the number of individuals identified as:

(a) American Indian or Alaskan Native,

(b) Asian or Pacific Islander,

(c) Black,

(d) White, and

(e) Hispanic origin or not of Hispanic origin. 

The combined version included the categories:

(a) American Indian or Alaskan Native,

(b) Asian or Pacific Islander,

(c) Black, not of Hispanic origin,

(d) Hispanic, and

(e) White, not of Hispanic origin. 

In October 1997, these standards were revised, so that federal agencies should report the number of

individuals identified as:

 (a) White,

(b) Black/African American,

(c) Asian,

(d) American Indian or Alaska Native, and

(e) Native Hawaiian or other Pacific Islander,

(f) Those indicating membership in more than one racial category, and

(g) of Hispanic origin. 

The task of developing reporting forms was a collaborative effort among committee members from the

National Center for Health Statistics, the Bureau of the Census, the Bureau of Labor Statistics, the U.S.

General Accounting Office, the National Center for Education Statistics, the Office of Minority Health, and the

Department of Veteran Affairs

We initially began with a panel of experts who reviewed and made suggestions on an initial draft of a

form design. From this, we constructed three aggregate reporting forms that we tested. The study design

consisted of nine interviews conducted in government cognitive laboratories with administrative personnel
4

similar to those who would be using the reporting forms in the future. An additional series of nine interviews

were conducted with similar administrative personnel located on-site at their establishments. During the

course of the interviews, we asked a battery of standardized questions and conducted a debriefing session after

each interview. The comments from test participants were assessed and the forms were revised accordingly.

III. Discussion of Findings

Three different forms were developed for testing (see Exhibits 1, 2, and 3). The process of developing

these initial forms was a collaborative effort among the committee members, experts in questionnaire design

and survey research, as well as policy and statistical analysts from the Federal Government.

Form RH-1 : This form was designed to collect every possible race combination generated from the

five minimum race categories. These combinations are further classified under the column headings:

“Yes, of Hispanic or Latino origin;” “No, not of Hispanic or Latino origin;” “No Hispanic or Latino

origin information provided.” This form represents a template for the information that could be

programmed by an automated data reporting system. The categories can be easily altered depending

on the specific race combinations listed.

Form RH-2: This form has two sections. The first section asks for aggregated totals for those who

reported each single race, the totals reporting more than one race, and the total number of individuals

for whom there is race information. The second section focuses upon the totals reporting multiple race

combinations and asks the number of times each race was included in combination. As with form RH1, these totals are further specified as “Hispanic or Latino,” “non-Hispanic or Latino,” or “no Hispanic

or Latino information available.”

Form RH-3: This form is similar to RH-2 in that it also has two sections, but it provides the

opportunity to report aggregated race data crosstabulated with other variables (e.g., gender). The first

sections asks for the total number of individuals reporting each single race or a multiple race

combination and this information was crosstabulated by Hispanic or Latino origin and gender. The

second section records the number of times each race was mention in a multi-race combination and this

information is likewise crosstabulated with Hispanic or Latino origin and gender. 

It was these three forms that were tested through a series of interviews with respondents in the lab and in the

field.
5

A. Lab Interviews

In total, 9 interviews were conducted in government cognitive laboratories. Participants included both

Federal and private sector employees whose jobs typically involved the use of administrative forms. To test the

usability of our draft reporting forms, test participants were given stacks of fictitious applications containing

single and multiple race responses, as well as information about Hispanic or Latino origin. These ‘dummy

records’ were used to see how participants would complete the forms using different kinds of source data. In

one set of dummy records (referred to as the “Combined Form”), the Hispanic origin information was

embedded along with the race information, as is typical practice among many institutions currently collecting

race and ethnicity information from their constituents. Another set of dummy records (referred to as the

“Separate Form”) used one question to record whether the fictitious applicant was of Hispanic origin and

another question to record race. Examples of these questions from the dummy records are given in Exhibit 4. 

After test participants completed their task of aggregating the information from the dummy records and

recording it on either test form RH-1, 2, or 3, interviewers then probed for additional information on how

participants understood and interpreted the various sections of the forms. Interviewers also asked participants

to explain the meanings of the terms used and to identify any parts of the forms that were confusing, offensive

or problematic in any other way.

In brief, the key findings from the laboratory interviews may be summarized as follows:

• Some respondents had difficulty grasping the concept of ‘multi-race’ reporting.

• Some participants treated “Hispanic” as a race and included it in the multi-race counts.

• Some participants were unsure how to report missing Hispanic/Latino origin information.

• Some participants were unsure how to report missing race information.

• An accompanying tally sheet could be useful for manual reporting.

• The forms needed detailed instructions with examples.

• The acronyms used in some versions were misleading and needed to be changed.

• The 3 "Hispanic information" columns on form RH-1 confused some participants.

• Section C of form Rh-2 was difficult because it requires “double-counting” of individuals.

• The layout of form RH-2 permitted users to identify miscalculations quickly.

While respondents, in many cases, understood what they needed to do, they still had problems with the words

and categories on the forms. We began to see and understand that the participants who came into the lab were

having cognitive difficulties with the forms because we were using categories and concepts (e.g., multi-racial)
6

which did not match their own experiences. It was our first glimpse of the complex conceptual issues that lay

at the heart of our own difficulties designing an aggregate reporting form.

B. On-Site Interviews

Our on-site interviews included public agencies (e.g., government agencies and jails), private not-forprofit organizations (e.g., private school), and private sector businesses. In total, we conducted 9 interviews on

location at these establishments and, whenever possible, used their actual data-collection forms or computergenerate summaries of those forms to replace the dummy records used in the laboratory interviews.

For consistency, the on-site interviews were structured the same as our laboratory interviews and

included the same questions. In addition, however, we were able to view the establishments’ data

“processing” systems and learn details of their data collection processes. It was by going into the field and

using actual establishment records that we quickly discovered the chasm between our draft forms and the

source data with which these participants worked. In many cases, our test sites used 1977 OMB reporting

guidelines (and included “Hispanic” as part of the listing of races. Apart from this common element, however,

we discovered a great deal of variability in the classification of race and ethnicity (see Exhibit 5 for examples). 

We also learned that race and ethnicity information is not always collected through self-reports. Visiting jails,

we learned that the arresting officers made race and ethnicity assignments for prisoners in these particular

systems, most typically on the basis of appearance. In the past, this had produced situations such as having a

prisoner who spoke only Spanish and who had a Spanish surname and yet was classified simply as “Black.” In

another case, we learned of an administrative secretary who changed the self-reports of her clients if the

information did not correspond with information she knew about the family and its history (e.g., she changed

self-reports of “white” to “Hispanic” when she knew the individuals were born in Mexico, spoke Spanish and

had a Spanish surname). While this information did
7

not directly impinge upon our design and revision of the reporting forms, it did provide new insights into the

quality of race and ethnicity data.

The findings from the field interviews may be summarized as follows:

• Some participants treated “Hispanic” as a race and included it in the multi-race counts.

• Some participants were unsure how to report missing Hispanic/Latino origin information.

• Some participants were unsure how to report missing race information.

• An accompanying tally sheet could be useful for manual reporting.

• The forms needed detailed instructions with examples.

• The acronyms used in some versions were misleading and needed to be changed.

• Some participants were confused by the 3 “Hispanic information” columns on form RH-1

• Section C of form Rh-2 was difficult because it requires “double-counting” of individuals.

• Some sites classify multi-race combinations as “other” and these cannot be disaggregated.

• The forms did not match very well with the participants’ ‘intake’ forms.

Clearly the findings mirror those obtained during the lab interviews. In other words, conducting them

on site at the establishments did not, in these particular cases, produce an advantage over what was obtained in

the lab. By looking at the similarity in the findings between the lab and the field, we concluded that the

problems with the form were conceptual rather than procedural. That is to say, the difficulties resulted from

the interaction of the respondent with the form and were not location specific or driven by establishment

procedures.

Having said this, however, we did find one additional difference in the field which was a very

important finding-- the forms we designed didn't match with the participant's intake forms. We could have

learned this in the lab, though, if we had asked respondents to bring their forms with them. Nevertheless,

going outside the lab did give us insights into the environment in which the forms would be completed, helped

us estimate the length of time needed to complete the form, and the quality of the data the aggregate forms

would be collecting.

IV. Comparisons of Lab and Field Findings

As mentioned, the findings from both the lab and the field were remarkably similar. While this triangulation of

information was reassuring, it is necessary to reflect on the conditions that would lead to these similarities,

since this may not always be the case.
8

• First of all, the task was conceptually simple. The forms involved only two variables, namely, race and

ethnicity.

• Second, the problems were with the form design and how the individual interacted with it.

Unfortunately, the forms appeared complex, with many lines and blanks for entering data, so that the

respondents had to interpret the forms before they could complete them. Yet, the process of

understanding how the individual interacted with the form was not dependent upon the location where

the interview occurred.

• Third, since the central issues were largely cognitive and conceptual, they were not the result of

information system retrieval problems. This is especially true for large establishments with many

employees and state-of-the-art computer systems. For the larger establishments, it was mainly an issue

of being able to convey to them how to program their existing variables; they simply needed "computer

specs" rather than a form.

Thus the main difficulties were how to (1) convey to an individual the multitude of race and ethnicity categories

that were being requested and (2) provide a conceptual bridge between their own racial categories and those

requested by the form. Consequently, the location for testing the form was as portable as the individual

respondent with knowledge of the organization's racial categories. The work could be done in the lab or in the

field. Doing the work in the field simply made it easier to collect copies of the various organizations' intake

forms and racial categories (which they could have brought to the lab, if asked) and identify the appropriate

person who typically would be filling out such a form.

V. Conclusions

While we were successful in uncovering many of the difficulties that respondents were having with our three

draft reporting forms, we were not equally successful in discovering solutions to those problems. No simple

answers or easy fixes appeared; the magic bullet never materialized. On the one hand, this may reflect the

complexity of the task at hand. Designing a form that can be used during a transitional period between one set
9

of definitions and a different set of definitions, may be an impossible task that has no solution. On the other

hand, the fact that we uncovered problems without simultaneously discovering solutions may reflect the

qualitative nature of the study. Since the work was largely exploratory, the study did not have a strictly defined

experimental design that might have allowed us to clearly recognize causes and effects. Likewise, we lacked

the systematic controls that would have allowed us to test various manipulations and generalize the results. 

VI. What does this all mean?

The good news is that whatever we found in terms of respondents’ difficulties with the forms appears 'real.' 

Where we conducted our interviews seemed to have few differences in our findings. The bad news is that

despite the different settings where we tested, even with real data at hand, there were problems with the

categories and the words on the form that we have not yet been able to overcome. We tried to improve the

forms with better instructions and item labeling, but respondents still had difficulty with the separation of the

'Hispanic' category and the concept of multi-racial reporting. 

In addition to these findings, however, we also learned more about the quality of the race data that will be

reported back on our aggregate reporting forms. While we learned about the cognitive difficulties respondents

had with the form in both locations, by going outside the lab setting, we obtained more helpful insights about

the way people are collecting the data.
10

Exhibit 1 - Form RH-1 (original)

FORM RH-1 For Use in Reporting Populations by Every Combination of Multiple Race and Hispanic Origin Responses

Individuals

who marked

Yes, Hispanic

Origin

Individuals

who marked

NO, Hispanic

Origin

Individuals

who did NOT

provide

Hispanic

Origin

Information

1 White

2 Black/African American

3 Asian

4 American Indian/Alaska Native

Individuals

who marked

ONLY

ONE

Race

5 Native Hawaiian/Other Pacific Islander

6 White + Black/African Am.

7 White + Asian

8 White + Am Indian/Alaska Nat

9 White + Nat Hawaiian/OPI

10 Black/African Am + Asian

11 Black/African Am + Am Indian/Alaska Nat

12 Black/African Am + Nat Hawaiian/OPI

13 Asian + Am Indian/Alaska Nat

14 Asian + Nat Hawaiian/OPI

Individuals

who 

marked TWO

races

15 Am Indian + Nat Hawaiian/OPI

16 White + Black/African Am + Asian

17 White + Black/African Am + Am Indian/Alaska Nat

18 White + Black/African Am + Nat Hawaiian /OPI

19 White + Asian + Am Indian/Alaska Nat

20 White + Asian + Nat Hawaiian/OPI

21 White + Am Indian/Alaska Nat + Nat Hawaiian/OPI

22 Black/African Am + Asian + Am Indian/Alaska Nat

23 Black/African Am + Asian + Nat Hawaiian/OPI

24 Black/African Am + Am Indian/Alaska Nat + Nat Hawaiian/OPI

Individuals

who marked

THREE races

25 Asian + Am Indian/Alaska Nat + Nat Hawaiian/OPI

26 White + Black/African Am + Asian + Am Indian/Alaska Nat

27 White + Black/African Am + Asian + Nat Hawaiian/OPI

28 White + Black/African Am + Am Indian/Alaska Nat + Nat

29 White + Asian + Am Indian/Alaska Nat + Nat Hawaiian/OPI

Individuals

who marked

FOUR races

30 Black/African Am + Asian + Am Indian/Alaska Nat + Nat

Individuals

who marked

FIVE

races

31 White + Black/African Am + Asian + Am Indian/Alaska Nat + 

Nat Hawaiian/OPI

Race Missing 32 Individuals who DID NOT provide race information

Total 33 Count of total population (Sum of rows 1 through 32)
11

Exhibit 1a - Form RH-1 (revised)

Instructions for Completing Form on Race and Ethnicity

Race Categories and Definitions

• White: A person having origins in any of the original peoples of Europe, the

Middle East, or North Africa.

• Black or African American (Black): A person having origins in any of the

black racial groups of Africa.

• Asian: A person having origins in any of the original peoples of the Far East,

Southeast Asian, or the Indian subcontinent including Cambodia, China, India,

Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and

Vietnam.

• American Indian or Alaska Native (AIAN): A person having origins in any

of the original peoples of North and South America (including Central

America), and who maintains tribal affiliations or community attachment.

• Native Hawaiian or Other Pacific Islander (NHOPI): A person having

origins in any of the original peoples of Hawaii, Guam, Samoa, or other

Pacific Islands.

Ethnicity Category and Definition

• Hispanic or Latino: A person of Cuban, Mexican, Puerto Rican, South or

Central American, or other Spanish culture or origin, regardless of race. 

Instructions Prior to Completing This Form

Step 1 Classify individuals by the number of races reported.

• Individuals can report one race or more than one race. This form allows

up to 5 race combinations to be reported for both Hispanics or Latinos

and for non-Hispanic or non-Latinos.

• Items 3 through 7 include individuals who reported one race, followed

by individuals who reported two races, and so forth. 

• Individuals who did not report race or for whom race information is

unavailable should be reported in Item 2.

• Individuals who reported race but did not report Hispanic or Latino

origin (whether blank, missing, or unavailable) should be reported under

the Not Hispanic/Not Latino columns for the appropriate race group.

Step 2 Classify individuals as Hispanic or Latino or Not Hispanic or Latino

and report the number under the appropriate column. 

• For purposes of this form, Hispanic or Latino origin is NOT a race

category.

• Example 1: If an individual reported Hispanic, White, and Black, report

this individual in Item 4, Line 4a under the Hispanic or Latino column. 

• Example 2: If an individual reported Hispanic and no race information is

available, report this individual in Item 2, Line 2a.

• Example 3: If an individual reported Asian, report this individual in

Item 3, Line 3c, under the Not Hispanic/Not Latino column.

Step 3 Count subgroup entries into 1 of the 5 race groups as defined above. 

• For example, if an individual reported Chinese and Japanese, count both

as Asian. Report this individual in Item 3, Line 3C under the Not

Hispanic or Latino column.

1. Total population . ____

Should equal sum of Items 2-7 below).

Please enter the number of employees that represent the following race/ethnicity 

categories.

2. Race information not available/not reported

Total Hispanics for whom no race

information is available ____

F Do not report these individuals below

b. Total individuals for whom no race or

Hispanic information is available ____

F Do not report these individuals below

3. Individuals who reported ONE race

Race Hispanic/ Not Hispanic/

Latino Not Latino

a. White .................. ____ ____

b. Black .................. ____ ____

c. Asian ..........____ ____

d. AIAN .................. ____ ____

e. NHOPI .................. ____ ____

4. Individuals who reported TWO races

Race Hispanic/ Not Hispanic/

Latino Not Latino

a. White+Black................ ____ ____

b. White+Asian ................ ____ ____

c. White+AIAN................ ____ ____

d. White+NHOPI.............. ____ ____

e. Black+Asian................. ____ ____

f. Black+AIAN................ ____ ____

g. Black+NHOPI.............. ____ ____

h. Asian+AIAN ................ ____ ____

i. Asian+NHOPI.............. ____ ____

j. AIAN+NHOPI.............. ____ ____

5. Individuals who reported THREE races

Race Hispanic/ Not Hispanic/

Latino Not Latin

a. White+Black+Asian..... ____ ____

b. White+Black+AIAN .... ____ ____

c. White+Black+NHOPI____ ____

d. White+Asian+AIAN..... ____ ____

e. White+Asian+NHOPI____ ____

f. White+AIAN+NHOPI____ ____

g. Black+Asian+AIAN..... ____ ____

h. Black+Asian+NHOPI... ____ ____

i. Black+AIAN+NHOPI____ ____

j. Asian+AIAN+NHOPI____ ____

6. Individuals who reported FOUR races

Race Hispanic/ Not Hispanic/

Latino Not Latino

a. White+Black+Asian+AIAN ____ ____

b. White+Black+Asian+NHOPI ____ ____

c. White+Black+AIAN+NHOPI ____ ____

d. White+Asian+AIAN+NHOPI ____ ____

e. Black+Asian+AIAN+NHOPI ____ ____

7. Individuals who reported FIVE races

Race Hispanic/ Not Hispanic/

Latino Not Latino

a. White+Black+Asian+AIAN

+NHOPI .................. ____ ____
12

Exhibit 2: Form RH-2

Form RH2-a For Use in Reporting Single and Multiple Race Responses Separately from Hispanic Origin 

SECTION A AGGREGATE REPORTING OF POPULATION BY HISPANIC ORIGIN

If your records list Hispanic as a race and individuals did not select the Hispanic category, then they would be counted as “Not Hispanic” on line 2.

If your records include a separate question asking for “Hispanic Origin” and individuals omitted that question completely, they would be counted as “No Hispanic Origin

information” on line 3.

1 Number of individuals who are Hispanic, Latino, or of Spanish Origin 1

2 Number of individuals who are NOT Hispanic, Latino, or of Spanish Origin 2

Hispanic or NOT

Hispanic

Identification

3 Number of individuals with NO Hispanic Origin information 3

Total Population 4 ADD boxes 1 + 2 + 3 and enter sum in box 4 → → 4

SECTION B AGGREGATE REPORTING OF POPULATION BY RACE 

If your records list “Hispanic” as one of the races, please do not count it as a race for the purposes of this summary report. Tally the races only as “White, Black/African

American, Asian, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander.” Please do NOT include Hispanic as a race.

Example: Those who report that they are “American Indian and Hispanic” would be counted on line 8 with those individuals who marked “American Indian/Alaska

 Native” only. Their “Hispanic Origin” information would be recorded in Section A.

5 Number of individuals who marked White only 5

6 Number of individuals who marked Black/African American only 6

7 Number of individuals who marked Asian only 7

8 Number of individuals who marked American Indian/Alaska Native only 8

9 Number of individuals who marked Native Hawaiian/Other Pacific Islander only 9

Single Race

Count

10 ADD boxes 5 + 6 + 7 + 8 + 9 and enter sum in box 10 → → 10

Multiple Race

Count

11 Number of individuals who marked MORE THAN ONE race 11

Missing Race

Count

12 Number of individuals who DID NOT report race 12

Total Population 13 ADD boxes 10 + 11 + 12 and enter sum in box 13 → → 13

SECTION C AMONG INDIVIDUALS REPORTING MULTIPLE RACES. AGGREGATE

 REPORTING OF TIMES EACH RACE IS MARKED.

This section is for providing information about those who reported more than one race. The sum of lines 14-18 should be greater than the total population counts given in lines 4

and 13.

Example: Those who report that they are “White, Black, and Hispanic” would be included in this section of “those who marked more than one race.” They would be

 included in line 14 because they reported “White” and on line 15 because they also reported “Black/African American.”

Example: Those who report that they are “Asian and Hispanic” would NOT be included in this section because they only reported 1 race, “Asian.” Reminder: for the

 purposes of this form, “Hispanic” is not included as a race.

14 Number of times White was marked 14

15 Number of times Black/African American was marked 15

16 Number of times Asian was marked 16

17 Number of times American Indian/Alaska Native was marked 17

Count of TIMES

each race was

marked for

individuals who

marked MORE

THAN ONE race

18 Number of times Native Hawaiian/Other Pacific Islander was marked 18
13

Exhibit 3a: Form RH-3 (original)

FORM RH-3A - For Use in Tabulating Aggregated Race Data

AGGREGATE REPORTING OF POULATION BY RACE, HISPANIC ORIGIN AND GENDER

Hispanic and Gender Individuals Who Marked Only ONE Race

Characteristics White Black/

African

American

Asian American

Indian/

Alaska Native

Native Hawaiian/

OtherPacific 

Islander

Individuals

Who Marked

MORE

THAN ONE

Race

Total Population

Individuals

Who Did NOT

Report Race

Total

Population

Hispanic Male

Female

Total

Not Hispanic Male

Female

Total

No Hispanic

Information

Male

Female

Total

 ↑

 |

 |

_____________________________________________________________| 

 |

 ↓

NOTE: For individuals who makred MORE THAN ONE RACE, also complete form RH-3B on next page
14

EXHIBIT 3b : Form RH-3 (continued)

FORM-3B - For Use in Tabulating Multiple Race Responses with Other Variables

AMONG INDIVUDALS REPORTING MULTIPLE RACES, AGGREGATE REPORTING OF TIMES EACH RACE IS MARKED BY

HISPANIC ORIGIN AND GENDER

Hispanic and Gender

Characteristics

Number of times

White was

marked

Number of times

Black/African Am

was marked

Number of times

Asian

was marked

Number of

times

Am Indian/Alaska Nat

was marked

Number of

times

Nat Hawaiian/OPI

was marked

Total

Male

Hispanic

Female

Total

Male

Not Hispanic

Female

Total

Male

No Hispani

Information

c

Female
15

Exhibit 4: Examples of Race and Ethnicity Questions from the Dummy Records

Example 1: Combined Form

Race: Mark one or more

01 _ White 

02 _ Black or African American

03 _ Hispanic or Latino

04 _ American Indian or Alaska Native

05 _ Asian

06 _ Native Hawaiian or other Pacific Islander

Example 2: Separate Form

9. Are you Spanish/Hispanic/Latino?

 01 _ Yes 

 02 _ No

10. Race: Mark one or more

01 _ White 

02 _ Black or African American

03 _ American Indian or Alaska Native

04 _ Asian

05 _ Native Hawaiian or other Pacific Islander
16

Exhibit 5: Examples of Race and Ethnicity Classifications Used by Establishments in our 

Field Tests

Example 1:

Select one:

Black

Hispanic

Asian / Pacific Islander

American Indian

Multi Racial

Example 2:

Select one:

Black, not Hispanic

American Indian

Oriental / Asian

Hispanic

White /Caucasian

Other (select this for multi-racial combinations)

Example 3:

Select one:

Back

White

Oriental

Indian

Black Hispanic

White Hispanic

Oriental Hispanic

Indian Hispanic

Unknown
17

REFERENCES

DeMaio, Theresa J., and Jennifer M. Rothgeb, "Cognitive Interviewing Techniques: In the Lab and in the Field" in 

Answering Questions, Seymour Sudman and Norbert Schwarz, ed.,

1996.

Esposito, James, Pamela C. Campanelli , Jennifer Rothgeb, and Anee Polivka, "Determining Which Questions are

Best: Methodologies for Evaluating Survey Qeustions", in Proceedings of the Section on Survey Methods Research,

American Statistical Association, 1991, pp. 45-55

Goldenberg, Karen L., Shall Butani, and Polly A. Phipps, 'Response Analysis Surveys for Assessing Response Errors

in Establishment Surveys', paper presented at the International Conference on Establishment Surveys, June 1993.

W. Sherman Edwards and David Cantor, "Toward a Response Model in Establishment Surveys", in Measurement

Errors in Surveys, Paul Biemer, et. al., eds, John Wiley & Sons, 1991.

Presser, Stanley and Johnny Blair, "Survey Pretesting: Do Different Methods Produce Different Results?" in

Sociological Methodology, 1994, pp. 73-104.

Tucker, Clyde, "Methodological Issues Surrounding the Application of Cognitive Psychology in Survey Research" in 

Bulletin de Methodologie Sociologique, June 1977.

Role of Respondent Debriefing Questions in Questionnaire Development

Wallman, Katherine, "Draft Provisional Guidance on the Implementation of the 1997 Standards for the Collection of

Federal Data on Race and Ethnicity", OMB, Feb. 1999.

Willis, Gordon B., and Theresa J. DeMaio and Brian Harris-Kojetin, "Is the Bandwagon Headed to the Methodological

Promised Land? Evaluating the Validitiy of Cognitive Interviewing Techniques" in Cognition and Survey Research. 

Monroe G. Sirken, Douglas J. Hermann, et. al., eds, John Wiley & Sons, Inc.. 1999