Improving Industry Descriptions for the Annual Refiling Survey 

Karen L. Goldenberg and Monica Dashen1 

U.S. Bureau of Labor Statistics

Karen L. Goldenberg, BLS, 2 Massachusetts Avenue, N.E., Room 3985, Washington, DC Goldenberg.Karen@bls.gov 

Monica Dashen, BLS, 2 Massachusetts Avenue, N.E., Room 1950, Washington, DC Dashen.Monica@bls.gov 

I. Introduction 

Industry is a fundamental variable in government economic statistics, and the accurate classification of industry is important 

to ensuring high quality industry-based data. U.S. statistical agencies base industry on the North American Industry 

Classification System (NAICS). The sampling frame at the Bureau of Labor Statistics (BLS) includes a NAICS designation 

for each of approximately 8.4 million business establishments. BLS ensures that the NAICS designations are current and 

accurate through the Annual Refiling Survey (ARS), an annual survey conducted as part of the Quarterly Census of 

Employment and Wages (QCEW). QCEW is a Federal-State Cooperative Program in which state agencies act as contractors 

to collect data for BLS from or about establishments in their states. 

During the ARS, respondents see a written description of the industry that corresponds to the NAICS code assigned to their 

establishment. The written descriptions consist of a general statement characterizing an economic activity, followed by lists 

of included examples and excluded items. Respondents are asked to indicate whether the description corresponds to the 

establishment's main business activity. Anecdotal reports from state agencies indicate that some respondents are confused by 

these descriptions, especially the “Does Not Include” component, and report incorrectly as a result. 

The purpose of this study is to explore the presentation of industry descriptions on ARS forms in a controlled laboratory 

setting. We use vignettes describing fictitious business establishments to compare respondent behavior with currently-used 

(original) industry descriptions and alternative (test) industry descriptions from which we have omitted the “Does not 

include” statements. Our goal is to determine whether the presence of the “Does not include” statement causes respondents 

to make more errors, as suggested by anecdotal reports from state agency staff. 

II. Background

NAICS Classifications 

The NAICS classification system was introduced in 1997 and replaced the Standard Industrial Classification (SIC) system, 

which had been in place since the 1930s. Under NAICS, industries are classified according to their production processes, so 

that businesses using similar raw materials, equipment, and labor to perform similar activities are grouped together (Walker 

and Murphy, 2001). The initial NAICS classification was updated in 2002 and will be further refined in 2007. 

History. There were several motivations driving the development of NAICS. First, the SIC system did not allow for an 

adequate representation of an economy that was increasingly service-oriented, rather than goods-producing, nor did it readily 

allow for the identification of new and emerging industries. In addition, the North American Free Trade Agreement 

(NAFTA) required a common classification system that would permit comparison of economic statistics across the three 

partner nations, Canada, Mexico, and the U.S. Still another concern was that some SICs were based on production processes 

while other SICs were shaped around demand characteristics--i.e., activities or products that appeared similar to the users or 

customers.2

 As a result, the system did not have a single underlying conceptual basis (Murphy, 1998). 

1

 The opinions expressed in this paper are those of the authors and do not reflect official policy of the Bureau of Labor 

Statistics. The authors gratefully acknowledge the contributions of Stella Godbolt in recruiting test subjects, Kathy Herring 

for data entry and research assistance, Martha Duff for help in reviewing the responses to the debriefing interviews, and 

William Mockovak for his helpful comments on an earlier draft of the paper. 2

 An example of a demand-based classification is “Beauty and cosmetology schools.” This classification was included under 

the SICs for “Beauty Shops” and “Barber Shops” because of the services provided to the public. These schools were moved 
 

 

 

 

 

 

 

 

 

 

 

Because of the importance of industry classification in structuring national economic indicators, development of NAICS took 

place at high levels in the three participating governments. In the U.S., the responsibility fell to the Office of Management 

and Budget (OMB), which is part of the Executive Office of the President. OMB established the Economic Classification 

Policy Committee (ECPC) and charged it with overseeing the NAICS structure and its content. The U.S. work was 

performed by subject matter subcommittees, with contributions from numerous government agencies, as well as input from 

trade groups, data users, and others (Murphy, 1998). The ECPC represents the U.S. in discussions with Canada and Mexico, 

and continues to direct updates and changes to the industry classification structure. 

Structure. NAICS is a hierarchical structure consisting of 20 major sectors, each corresponding to a different category of 

economic activity (for example, manufacturing, construction, finance and insurance, health care). Within the hierarchy, the 

most detailed individual industries are represented by unique 6-digit codes. The first 2 digits identify the sector, the third is 

the subsector, the fourth represents the industry group, and the fifth digit is the international (three nation) industry level. 

The sixth digit is used by the individual countries if they need more detailed classifications than those agreed upon by the 

NAFTA consortium. 

The 6-digit level usually specifies a single economic activity, but the NAICS structure characterizes a few industries as 

combinations of two or more specific products or activities, both of which must be present (OMB, 2002). Gasoline stations 

with convenience stores (NAICS 447110) are an example of a combined activity. In addition, within many industry groups, 

the structure provides for a residual or “Other” category that encompasses products, services, or activities not specifically 

included anywhere else. There are 1,179 different 6-digit industries in the NAICS 2002 hierarchy. 

NAICS Manual. The NAICS manual (OMB, 1997; 2002) is the primary reference for assigning industry codes. The 

manual defines each industry sector and subsector, provides a narrative description of each industry group and internationallevel industry, and if there are U.S.-specific industries, includes a complete specification of each one. At the most detailed 

level, the manual may include “illustrative examples” of activities or products (called index items) that serve as criteria for 

classification. In addition, the definitions may specify exceptions, especially in the “Other” categories. For example, the 

definition of 442299, Other Home Furnishings Stores, is: 

This U.S. industry comprises establishments primarily engaged in retailing new home furnishings (except floor 

coverings, furniture, and window treatments) (OMB, 2002: 560) 

The illustrative examples for the preceding description (442299) include bath shops, chinaware stores, lamp stores, and some 

other types of stores. Each industry also lists cross references to similar activities and shows the NAICS classifications for 

those activities. In the example shown here, the cross references are to Floor Coverings Retailers, Furniture Stores, Window 

Treatment Stores, and several other industries, not all of which are addressed in the definition. In effect, the combination of 

exceptions and cross references define an industry by what it is not as well as by what it is. This process of definition by 

exception is meaningful and easy to follow in the NAICS manual, where a reader can review the cross references. It is less 

meaningful when it is removed from its supporting context, as used on the ARS forms. 

The Annual Refiling Survey 

The Annual Refiling Survey is the BLS mechanism for maintaining current information in its sampling frame. Every year, 

the state QCEW offices mail ARS forms to approximately one-third of the Unemployment Insurance account holders in their 

states.3

 These BLS-designed forms ask respondents to verify the business name, mailing address, physical location address, 

county or township, and economic activity for the location(s) in that state. The economic activity (industry) item consists of 

a short description and a question about whether the description accurately reflects the main business activity of the 

establishment in the state during the previous 12 months. If it does not, respondents are asked to list their main activities, and 

to indicate the approximate percentage each contributes to sales or revenues. 

to “Educational Services” because their production process was defined as "establishments with students who learn and 

teachers who teach" (Jordan, 2005). 3

 For ARS purposes, the basic unit of analysis is an Unemployment Insurance account. Accounts are assigned by each state 

and can apply to either single-unit businesses or multi-establishment firms. The materials created for this study are singleestablishment versions, although the same industry descriptions appear on single-unit and multi-unit ARS forms. 
 

 

 

 

 

 

 

 

 

 

 

 

Figure 1 illustrates the basic structure of the industry verification item on the form. The top part of the illustration shows the 

generic template, while the bottom half shows what the questions look like as presented to a respondent in industry 238221 

from the fictitious state of Utana. Note that the question refers to the “main” business activity, leaving the definition of 

“main” up to the respondent. This ambiguity is intentional. According to the NAICS manual, if an establishment performs 

more than one activity, the classification should be based on the “principal product or group of products produced or distributed, or services rendered.” While the ideal determination of this product or service is the establishment's “relative share of 

current production costs and capital investments,” revenue, shipments, or employment are often used as proxies (OMB, 2002: 

22). During the questionnaire design process, the design team considered the tradeoff of making the question more precise, 

at the price of making it more complex and more challenging to understand. The team decided that for verification purposes 

we would use the more respondent-friendly wording now in place. The questionnaire requests more detailed information 

when the respondent does not agree with the description, and states use the additional detail for coding purposes. 

Figure 1. Industry Verification Item from ARS Form 

9

10 

9

10 

Our records show that the main activity of the business using U.I. number

9999999999 in UTANA is:

 999999 

While you may not do everything listed above, does the information in Item 9 accurately describe the main business in Utana 

during the past 12 months? (If the business has been closed, sold, or moved out of this state, please answer in terms of its former

activity.) 

F YES...Please SKIP to Item 12

F NO....Continue with Item 11

Our records show that the main activity of the business using U.I. number

9999999999 in UTANA is: 

Plumbing, heating, or air-conditioning contractors primarily working on RESIDENTIAL buildings. 

Examples of their work may include, but are not limited to: 

* Duct work (cooling, dust collection, * Natural gas fireplace installation 

exhaust, heating, ventilation) * Lawn sprinkler system installation 

* Fire sprinkler system installation * Sewer hook-ups and connections 

* Furnace installation * Sump pump installation 

RESIDENTIAL contractors may provide both parts and labor when performing work. The work 

performed may include new work, additions, alterations, maintenance, and repairs. 

DOES NOT INCLUDE primarily installing electrical controls for HVAC systems; or duct cleaning. 

238221 

While you may not do everything listed above, does the information in Item 9 accurately describe the main business in Utana 

during the past 12 months? (If the business has been closed, sold, or moved out of this state, please answer in terms of its former

activity.) 

F YES...Please SKIP to Item 12

F NO....Continue with Item 11
 

 

 

 

BLS economists rewrite the industry descriptions from the NAICS manual for the ARS forms. During the rewriting process, 

they simplify the language, remove the cross references, and fit the description into the space available. As illustrated above, 

the manual is written using economic jargon, and tends to present long lists of included and excluded items in paragraph 

form. The economists worked from a set of guidelines, including: 

● Describe what the industry is well ahead of what it is not. 

● Use bullets instead of paragraphs to the extent possible, especially for lists 

● Where appropriate, emphasize that the examples are illustrative and that the list is not exhaustive 

● Use “Does not include” to refer to excluded items 

● Avoid presenting excluded activities in parentheses. 

Some industries are particularly challenging to describe, however. The “all other” industries generally take the form of: 

This industry comprises establishments primarily engaged in [activity] (except ...) 

It is not uncommon for the list of exclusions to be as long or longer than the list of inclusions. Since respondents do not see 

the related industry descriptions, the result is lengthy “Does not include” statements on their ARS forms. For example, the 

“Does not include” statement for All Other Home Furnishings Stores, cited above, is: 

DOES NOT INCLUDE retail sales of floor coverings, furniture, window treatments, or

new mirrored glass lighting fixtures.

DOES NOT INCLUDE electronic home shopping, mail-order, or other non-store retail

sales of home furnishings. 

State agencies tell BLS that some respondents reject descriptions because their business involves some amount of an 

excluded item, even though the “includes” list covers their main activity. Other respondents reject correctly coded industry 

descriptions because their businesses engage in an activity that is absent from the list of examples.4 

III. Methodology 

Research Objectives 

Stated broadly, the objective of this research is to look for principles that can be adapted to writing industry descriptions that 

will make these descriptions clearer to ARS respondents. The specific goal of the present study is to explore the effect of 

using exclude statements in the ARS industry descriptions. Because of the complexity of some industry definitions, it is 

important that the examination of excludes be tailored to the industry. 

Approach 

Our goal for the current investigation is to compare industry descriptions with and without the “Does not include” statements. 

We wanted to see if the presence or absence of these statements has an effect on the likelihood that a respondent will agree 

with a correct description or disagree with an incorrect description. We conducted a laboratory study with members of the 

general public as study participants. We created vignettes to represent businesses that could be compared to BLS industry 

descriptions. Each vignette is a short descriptive paragraph describing a fictional business. 

In the laboratory, we presented study participants with mock ARS forms containing industry descriptions, some with and 

some without the “Does not include” statements. We asked participants to consult the industry description on the forms and 

to determine whether or not the descriptions accurately reflected the business described in the scenario.5

 We also asked study 

participants to rate the difficulty of the verification task, and we debriefed them about the task. 

4

 BLS has not conducted a systematic cognitive evaluation of the industry descriptions for the ARS forms. The large number 

of descriptions makes it impractical and costly. However, state QCEW staffs interact with respondents, often telephoning for 

clarification of a business' activities, and these personnel are a useful source of information for QCEW. We systematically 

ask state QCEW offices about problems with industry descriptions, especially those that seem to be misunderstood. 5

 In this paper, we use the terms “vignette” and “scenario” interchangeably. 
 

 

 

 

 

 

 

 

 

Our research approach differs from other establishment studies where researchers generally visit the respondents at their 

places of business and make inquiries about their business activities (e.g., Phipps et al., 1993; Eldridge et al., 2000). The 

more familiar methodology would not meet the needs of our study because respondents are expected to provide an answer for 

more than one industry. It would be unreasonable to expect individual ARS respondents to answer any questions about a 

business or industry other than their own. 

Research Design 

We used scenarios and mock ARS forms to compare the effect of having “Does not include” statements as part of the 

industry verification process. 

Selection of Industries. The selection of industries to test was a key component of the research. We began with 

Employment and Wages: Annual Averages (BLS, 2003), the primary publication of the QCEW program, and compiled a list 

of all 6-digit NAICS industries that had 20,000 or more establishments in 2002. This process yielded a list of 87 industries6 

and ensured that we would be working within the realm of widely visible and familiar industries. We then analyzed the 

industry descriptions printed on the ARS forms for those 87 industries. We coded: 

● The total number of words in the description (as counted by Microsoft Word) 

● The number of examples of items or activities in the industry description “includes.” Some of these “includes” are part 

of the narrative descriptions, while others appear on ARS forms as “Examples include, but are not limited to” 

● The number of items shown under “Does not include” 

The industry descriptions ranged in length from 20 to 144 words and the number of examples and included items ranged from 

0 to 21. We selected industries in the middle range (60-100 words, 2 to 16 “includes”), in order to ensure that the 

descriptions were relatively homogeneous and to increase the likelihood that our results could be attributed to the 

experimental condition and not to an uncontrolled attribute of the industry description.7 We chose not to use the number of 

“excludes” as a criterion, because the number varied widely and nonlinearly relative to the “includes.” Within these limits, 

we divided the remaining 29 industries into four approximately equal groups and chose 3 industries from each group, for a 

total of 12 industries. Our subjective choices were based on our perception of how common the industries were, the clarity of 

the BLS descriptions, our wish to represent as many different industry sectors as possible, and the ease of creating vignettes 

for them. All of the selected industries have “Does not include” statements in their current descriptions. Appendix 1 shows 

the 12 selected industries and their descriptions. 

Vignettes. Vignettes create a “reality” for respondents. They are especially useful in assessing potential sources of error 

because researchers know the correct answer for a situation. In recent years vignettes have moved from the realm of 

household surveys into establishment surveys (Morrison et al., 2004), where they have been presented as mock records to test 

questionnaire layouts (Stettler et al, 2000), the questionnaire completion process (Goldenberg et al., 2002), and selfadministered questionnaires (Goldenberg, 1998).8 

By using vignettes in a laboratory setting, we can ask about a variety of industries. We control the situations being described, 

the extent to which they do or do not match industry definitions, and the amount of ambiguity in those situations. On the 

other hand, because laboratory subjects are members of the general public, and we are testing industry-specific descriptions, 

we have to ensure that the scenarios deal with familiar businesses and everyday situations. 

We created two hypothetical scenarios for each industry. One is straightforward, by which we mean the scenario reflects 

information from the “include” side of the NAICS industry definition (description or examples). The other scenario is 

complex, in that it incorporates information from the “Does not include” statement. We used the complex scenario because in 

the real world many businesses encompass a variety of products, services, or activities, some of which could overlap with 

other NAICS classifications. For example, using the Other Home Furnishings stores described earlier, a straightforward 

scenario that is correctly coded in this industry might describe a store that sells sheets, towels, and other bedroom and 

bathroom accessories. A straightforward scenario that is not correctly coded for this industry could describe a business that 

6

 The list included a few known "problem" industries with fewer than 20,000 establishments, and some that represented 

"Other" categories. However, none of the smaller industries made the final cut. 7

 This range is approximately the mean number of words (79) ± one standard deviation (27). 

8

 See Morrison et al., 2004 for a review. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

sells refrigerators, stoves, washing machines, dryers, and other household appliances—that is, the business does something 

different from what is described, but is not specifically addressed by the exclusions. A complex scenario that is correctly 

coded might be a business that sells lamps (30 percent of total products sold), other home accessories (45 percent), and rugs 

(25 percent), while a complex scenario that does not fit the description could be a furniture store or a store that sells rugs (60 

percent) and other home accessories (40 percent). What makes these situations complex is that they refer to one of the 

exclusions for the NAICS industry. In the case of the store with 60 percent of its sales in rugs, the main product or activity is 

an excluded item. 

Appendix 2 shows the vignettes for each of the 12 selected industries. Based on the scenarios, study participants should 

reject some of the industry descriptions. The Y or N at the end of each scenario indicates whether the description should be 

accepted (Y) or rejected (N), given the scenario.9 

Original and test conditions. Once we had selected the industries, we turned again to the industry descriptions currently 

printed on the ARS forms. We defined the current descriptions as the original condition. We removed the “Does not 

include” statement from the description to create the test condition. We determined that study participants would see both 

original and test conditions, but for different industries. That is, they would always see either the original or the test 

condition for a specific industry. Since nothing else in the industry description changes, this allows us to evaluate the effect 

of the “Does not include” statement. By holding the version of the industry description constant across both types of 

scenarios, we increase the likelihood that any differences that emerge are in fact due to the presence or absence of the “Does 

not include” statement. We assigned original and test conditions so that study participants would see all 12 industries and an 

equal number of original and test descriptions. 

Materials. We prepared the following materials for the study: 

● Mock ARS forms corresponding to each vignette scenario. The forms show a fictitious business at a specific physical 

location in the fictitious state of Utana. The industry questions (Figure 1) contain the industry description 

corresponding to the scenario, in either the original or the test condition. While participants received the entire form, 

the questions following the industry description were grayed out so as not to distract the study participants. We 

prepared two versions of each industry description, one with the “Does not include” statement (original) and one 

without the “Does not include” statement (test), for a total of 48 different ARS forms. Appendix 3 shows sample mock 

ARS forms for the original and test conditions. 

● Worksheets for each scenario. The worksheets contain a paragraph with the vignette, as well as a self-administered 

difficulty rating. We asked participants to rate each industry verification task from 1 to 5, where 1 is very easy and 5 is 

very hard, before going on to the next scenario. 

● Participant packets. We compiled 24 sets of scenario worksheets and their corresponding ARS forms into packets for 

each study participant. We used a randomization procedure to allocate industries and scenarios so that study 

participants saw either the original or the test version of each description. We also developed four different sequences 

in which we presented the scenarios and industries. This procedure ensured that participants did not see both scenarios 

for an industry together, scattered correct and incorrect descriptions, and provided a good mix of complex and 

straightforward scenarios. 

● An interviewer-administered debriefing questionnaire. The questionnaire asked about the industry descriptions on the 

ARS form, focusing on the examples to include, the “Does not include” statements, whether the includes were 

helpful/confusing, whether the excludes were helpful/confusing, and how the study participant understood the phrase 

“Examples include but are not limited to." We also showed a study participant a complex scenario and asked if having 

the “Does not include” information would have been helpful while working with the scenario. Finally, we asked the 

participants to paraphrase the industry questions and asked how they understood different phrases in those questions. 

Administering the task. We briefed study participants on the nature of the ARS survey and the present task. We asked them 

to read and sign an Informed Consent form. Then we gave participants the packet described above, reviewed the instructions, 

and let them go to work. When they finished all 24 scenarios, an interviewer conducted the debriefing interview and 

recorded the answers. 

9

 In creating the vignettes, we wanted to reflect the reality that some businesses are not assigned the correct industry code. 

However, we did not incorporate this decision into the research design, and as a result the number of correct and incorrect 

vignettes is not equal. 
 

 

 

 

 

V. Results 

 

 

 

 

 

Participants. Thirty-nine individuals responded to an advertisement in a local newspaper and received $35.00 each to 

compensate for the expenses incurred by participating in the study. We subsequently dropped 3 of the participants’ data from 

the analysis,10 leaving us with a base of 36 usable cases. The participants ranged in age from 22 to 75 years, with a mean age 

of 45. All were high school graduates, with 12 to 19 years of education (mean 15.5 years); 25 percent had bachelor's degrees 

and an additional 36 percent had graduate training. Fourteen of the study participants were male and 22 were female. 

IV. Predicted Outcomes 

Our basic measure is the percentage of correct responses for each vignette, tabulated separately for the original (with “Does 

not include”) and test (without “Does not include”) versions of the industry descriptions. In some cases, the correct 

responses are “No"—the description on the mock ARS form does not fit the scenario. 

Recall that state QCEW staff have attributed errors to the presence of “Does not include” statements. If these staff reports are 

correct, the test version of an industry description should result in a higher number of correct answers than the original 

version. Similarly, we should see more correct responses for the straightforward vignettes than for the complex ones, 

because the straightforward scenarios present easier tasks. However, we noted that the complex scenarios contain references 

to information from the “Does not include” statement. Therefore, we anticipate an interaction effect, where the test version 

of the industry description produces fewer correct answers for the circumstances described in the complex vignettes. 

In addition to looking at whether the study participant correctly assessed each ARS form, we have two additional tools with 

which to examine the results and to consider the effect of the experimental treatment. The first is the perceived ease or 

difficulty attributed to the response for each scenario, recorded on the vignette worksheet. Our expectation is that the easier 

the task, the greater the likelihood of a correct response. Since the “Does not include” statements may make the task more 

difficult, it is likely that the ratings for the original industry descriptions will reflect greater difficulty than those for the 

descriptions in the test condition. However, this might be reversed for the complex scenarios, since they address information 

in the “Does not include” statements, and that information that could make the task easier. 

The second tool is the debriefing interview. The short interview focused on the industry description item and included 

general questions about the use of examples and the presence of the “Does not include” statements. 

Experimental treatment and response accuracy

Table 1 summarizes the results of the experiment. This table shows each of the industries we tested, tabulates the number 

(first line) and percentage of correct responses (second line) for the original and test versions of the industry description, and 

presents the differences between the percentages correct (original minus test). The data appear separately for the 

straightforward and complex scenarios. Each original and test condition has a maximum of 18 cases. The percentage correct 

at the bottom of the table summarizes our findings: across all 12 industries, there was no difference in correct response for the 

straightforward scenarios. On the complex side, the test condition showed a slightly higher percentage correct than the 

original condition, a difference which is not statistically significant, and which is counter to our prediction that the test 

condition would not perform as well for complex cases. 

Looking at the individual industries, correct responses for straightforward scenarios under the original condition ranged from 

33.3 to 88.9 percent, and from 44.4 to 83.3 percent under the test situation. Similarly, for the complex vignettes, correct 

responses ranged from 27.8 to 83.3 percent under the original condition and from 50.0 to 94.4 percent for the test condition. 

For each industry, we can compare the percentage correct between the original version and the test condition (the “Diff O- T” 

column in Table 1). Here again we see essentially no difference. Among the 12 industries, in 5 instances the original version 

of the industry description had a higher percentage correct than the test version (O >T), in 5 instances the test version 

outperformed the original version (O < T), and in the remaining two industries the results were identical (O = T). Among the 

complex scenarios, the original outperformed the test version in 5 cases, the test version was more successful in 6 cases, and 

the results were identical once. 

10 One participant left without completing the activity, and two others apparently did not understand the task. Their data were 

not usable. 
Table 1. Scenarios and Industry Descriptions: Results 

Number and Percentage Correct for Original and Test Conditions, by Type of Scenario 

 Straightforward Scenario Complex Scenario 

NAICS Industry Correct Industry description Correct Industry description 

response Original Test Diff O-T response Original Test Diff O-T 

238221 N Y 

Residential plumbing and Nr 12 11 15 14 

HVAC Contractors Pct 66.7 61.1 5.6 83.3 77.8 5.5 

441110 N Y 

New car dealers Nr 15 14 13 14 

Pct 83.3 77.8 5.5 72.2 77.8 -5.6 

444190 Y N 

Other building material Nr 11 13 13 11 

dealers Pct 61.1 72.2 -11.1 72.2 61.1 11.1 

447190 Y Y 

Other Gasoline Nr 14 15 15 15 

Stations Pct 77.8 83.3 -5.5 83.3 83.3 0 

453220 N Y 

Gift, novelty, and souvenir Nr 16 14 5 15 

stores Pct 88.9 77.8 11.1 27.8 83.3 -55.5 

453998 Y N 

Store retailers not Nr 6 8 16 13 

specified elsewhere Pct 33.3 44.4 -11.1 72.2 88.9 -16.7 

541940 Y N 

Veterinary services Nr 15 11 14 10 

 Pct 83.3 61.1 22.2 77.8 55.6 22.2 

624410 Y N 

Child daycare services Nr 14 14 15 14 

 Pct 77.8 77.8 0 83.3 77.8 5.5 

713940 Y N 

Fitness and recreational Nr 14 15 10 9 

sports centers Pct 77.8 83.3 -5.5 55.6 50 5.6 

811111 Y N 

General automotive Nr 14 14 14 15 

repairs Pct 77.8 77.8 0 82.4 83.3 -0.9 

812112 N Y 

Beauty salons Nr 12 10 11 17 

 Pct 66.7 55.6 11.1 61.1 94.4 -33.3 

813410 Y Y 

Civic and social Nr 8 12 10 11 

organizations Pct 44.4 66.7 -22.3 55.6 61.1 -5.5 

Total correct 151 151 151 158 

Percent correct 70.2 70.2 70.2 73.5 

We believe that the study participants generally understood the task and used the scenarios as we had intended. Most of the 

scenarios⎯19 of 24 (79 percent) in the original condition, and 20 of 24 (83 percent) of those in the test condition⎯were 

answered correctly at least 60 percent of the time. Where the scenarios did not perform well, the problem could have been in 

the way we worded the vignette.11 

 

11 This seems likely in two cases (NAICS 453998, Store retailers not specified elsewhere, and NAICS 713940, Fitness and 

recreational sports centers), where both the original and the test condition fell below the (admittedly arbitrary) 60 percent 

level for one of the vignette pairs. "Store retailers not specified elsewhere" is one of the residual categories described earlier, 

and the problematic vignette was for the straightforward condition. Both of the less-successful vignettes for NAICS 713940 

described complex situations. In another case (NAICS 813410, Civic and social organizations), both the original conditions 

but not the test condition failed to achieve 60 percent correct. 
 

 

 

Looking within specific industries, and comparing the straightforward and complex situations, we find some evidence of an 

interaction effect. Moving from the straightforward to the complex, the difference between the original and test conditions 

changes direction in five industries, and goes to/from zero in three additional industries. The change in direction is 

inconsistent, raising the possibility that something other than the experimental treatment might be responsible for the 

observed results. Given the lack of differences overall, however, the hypothesized interaction between straightforward and 

complex scenarios did not appear. 

Effect of the underlying decision task. As noted, some of the industry descriptions on the ARS forms were correct for the 

vignette and some were not. Since these two activities represent different cognitive tasks, we looked at the performance of 

the industry descriptions under the original and test conditions. Table 2 summarizes the difference in the proportion of correct 

responses according to whether the correct response is to verify a description or to reject it as incorrect. For the 12 industries 

used in this study, the post hoc analysis shows different patterns of responses based on the underlying task. When the 

industry on the ARS form fits the scenario (“Yes” response), the test version outperforms the original version. That is, 

omitting the “Does not include” statement appears to help with the recognition task, perhaps by eliminating irrelevant 

information. When the industry does not match the scenario, however, a larger number of study participants seeing the 

original industry description selected the correct answer that those who saw the test version. Although the number of 

industries to be verified and rejected is quite small, the pattern of data suggests that the “Does not include” statements were a 

factor in our study participants’ decisions to reject the ARS descriptions. 

Table 2. Decision Task (Verify Correct versus Reject Incorrect): Data Pattern 

Performance of Original and Test conditions across Industries (Number of Industries) 

Decision Task Straightforward Complex Total 

Verify correct 

Orig > Test 1 1 2 

Orig < Test 5 4 9 

Orig = Test 

Reject incorrect

2 1 3 

Orig > Test 4 4 8 

Orig < Test 0 2 2 

Orig = Test 0 0 0 

Taking the analysis a step further, Table 3 shows the mean percentage correct under each set of conditions in Table 2. We 

used analysis of variance (ANOVA) to evaluate the differences. In the ANOVA model, scenario type (straightforward/ 

complex) and the underlying task (verify correct or reject incorrect) are within-subject factors and industry description 

(original/test) is a between-subject factor. The main effects and 3-factor interaction were not statistically significant. Only 

one of the three possible 2-factor interactions, underlying decision task and industry description, reached statistical 

significance [F(1,70)= 4.32, P<.05], suggesting that study participants answered differently depending upon the original or 

test condition of the industry description. Indeed, Table 3 shows a significantly higher proportion of correct verifications for 

complex scenarios using the test version of the industry description (79.6 percent) than for the original version (63.9 percent) 

[F (1,70) = 4.24, p<.05]. The table points to an interaction—for the complex scenarios, the pattern is reversed. Study participants presented with an original version of the industry description had a higher percentage of correct rejections (75.9 

percent) than did those who saw the test version (66.7 percent), but this difference was not statistically significant. The 

interaction holds for the straightforward scenarios, but again the differences are not statistically significant. 

Table 3. Verify Correct versus Reject Incorrect: 

Mean Percentage of Correct Responses Across all Scenarios 

Straightforward Complex Total Decision Task Original Test Original Test Original Test 

Verify correct 66.7 70.8 63.9* 79.6* 65.5 74.6 

Reject incorrect 76.4 68.1 75.9 66.7 76.1 66.9 

Total 70.2 70.2 70.2 73.5 70.2 71.9 

 *Difference between means p < .05 
 

 

 

 

 

Taken together, these findings suggest that “Does not include” statements are helpful when the individual should reject the 

industry description as incorrect for the scenario. However, if the description is correct and should be verified, removing the 

“Does not include” statements seems to help with the recognition task and thus to increase the likelihood of a correct answer. 

Experimental Treatment and Ease/Difficulty of Response

Study participants assigned a rating to each vignette to reflect the ease or difficulty of completing the mock ARS form. 

Ratings ranged from 1 (very easy) to 5 (very difficult). We recoded the participant-assigned ratings so that they ranged from 

1 (very difficult) to 5 (very easy) for each ARS form/vignette combination—that is, the higher the mean, the easier the task. 

We compared the mean ease/difficulty ratings for each vignette between original and test conditions for the straightforward 

and complex scenarios. Given the input from state QCEW personnel, we would expect participants to find the original 

descriptions more difficult to work with than the test conditions, even though information in the “Does not include” 

statements might directly address the situation described in the vignettes. To some extent, the perceived ease or difficulty 

parallels that of the overall accuracy ratings. Table 4 shows the ratings for each of the 12 industries and the overall rating 

across all industries. Generally speaking the differences were negligible. For the straightforward scenarios, the mean for the 

original industry descriptions was 3.97, while for the test descriptions it was 3.95. For the complex scenarios, the means 

were a little lower: 3.87 for the original descriptions and 3.83 for the test versions. These differences are not statistically 

significant. Looking at individual industries and the straightforward vignettes, we see means ranging from 3.67 to 4.28 under 

the original condition and from 3.61 to 4.39 under the test condition. The differences are very small, and statistically 

significant for only 2 of the 12 vignettes. In addition, the direction of the statistically significant differences is not the same; 

these results could have happened by chance. The situation for the complex scenarios is similar, but the ratings are a little 

lower, suggesting the (expected) greater difficulty in those scenarios. Means for the original condition ranged from 3.33 to 

4.22, and from 3.44 to 4.22 for the test condition. None of the individual differences between conditions are statistically 

significant for the complex scenarios. In short, removing the “Does not include” statements from the industry descriptions 

does not appear to affect the perceived difficulty of the respondent's task. 

We repeated the analysis, examining whether the study participant was verifying correct information or rejecting incorrect 

information. Table 5 shows the response pattern. This table compiles the results of the “Diff O – T” column in Table 4, 

showing the results of each comparison according to the respondent's decision task. It appears from this table that having the 

original or the test version of the industry description had little effect for study participants who were verifying an industry 

description. However, when the appropriate action was to reject an incorrect industry, the original version noticeably 

outperformed the test version. That is, the original definition—with the “Does not include” statement—appears to facilitate 

the task of recognizing and rejecting an incorrect description. 

This pattern is borne out by the mean ease ratings shown in Table 6, although none of the mean differences or interaction 

terms is statistically significant. The numbers are small, but the interaction observed earlier holds for the present situation. 

None of the differences is statistically significant, but if the underlying task is to verify a correct description, our study 

participants found the test version of the description (without the “Does not include” statement) easier to work with than the 

original. On the other hand, the original version rated “easier” when the underlying task was to reject an incorrect 

description. This result mirrors the mean correct responses shown in Table 3. 

Ease/difficulty and correct response 

We created a dummy variable for the correct response for each vignette, using 1 for a correct answer and 0 for an incorrect 

answer. We then correlated the ease/difficulty rating with the likelihood that a study participant answered the vignette 

correctly. Our expectation was that the easier the task (i.e., the higher the recoded ease rating), the more likely the study 

participant would be to provide the correct answer. Larger positive r values support the hypothesis; smaller values and 

negative correlations tend to refute it. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 4. Mean Ease/Difficulty Ratings for Each Scenario, Original and Test Conditiona 

NAICS Industry 

Straightforward Scenarios Complex Scenarios 

Industry description Industry description 

Original Test Diff O-T Original Test Diff O-T 

238221 

Residential plumbing and HVAC Contractors 4.17 4.06 0.11 4.11 3.72 0.39 

441110 

New car dealers 4.11 4.06 0.05 3.94 4.06 -0.11 

444190 

Other building material dealers 4.06 4.06 0.0 3.56 3.83 -0.28 

447190 

Other Gasoline Stations 4.00 4.06 0.06 3.44 3.94 -0.50 

453220 

Gift, novelty, and souvenir stores 4.06 3.61 0.45 3.83 3.55 0.28 

453998 

Store retailers not specified elsewhere 3.67 4.00 -0.33 4.00 3.44 0.56 

541940 

Veterinary services 3.72 4.39 -0.67* 4.00 3.78 0.22 

624410 

Child daycare services 4.28 3.67 0.61* 4.22 4.12 0.10 

713940 

Fitness and recreational sports centers 4.28 4.11 0.16 3.78 3.65 0.13 

811111 

General automotive repairs 3.76 3.83 -0.07 4.06 3.83 0.22 

812112 

Beauty salons 3.78 3.89 -0.11 4.17 4.22 -0.06 

813410 

Civic and social organizations 3.72 3.61 0.11 3.33 3.79 -0.44 

Overall total 3.97 3.95 3.87 3.83 

a Ease/difficulty rating: 1 = Very difficult, 5 = very ease * p < .05 ** p < .01 

Table 5. Decision Task (Verify Correct versus Reject Incorrect): Data Pattern 

Comparison of Mean Ease/Difficulty Ratings Across Industries (Number of Industries) 

Decision Task Straightforward Complex Total 

Verify correct 

Orig > Test 

Orig < Test 

Orig = Test 

3 

4 

1 

3 

3 

0 

6 

7 

1 

Reject incorrect

Orig > Test 

Orig < Test 

Orig = Test 

3 

1 

0 

5 

1 

0 

8 

2 

0 

Table 6. Decision Task (Verify Correct versus Reject Incorrect): Mean Ease/Difficulty Ratings 

Decision Task Straightforward Complex Total 

Original Test Original Test Original Test 

Verify correct 3.94 3.97 3.80 3.88 3.88 3.92 

Reject incorrect 4.03 3.90 3.94 3.78 3.97 3.83 

Total 3.97 3.95 3.87 3.83 3.96 3.85 
 

 

 

 

 

 

 

 

Table 7 shows the correlations between correct response and reported ease or difficulty of obtaining that response. Looking 

first at the straightforward vignettes, under the original industry condition, all but one of the correlations was in the expected 

positive direction. (The exception had an r value of .00, indicating no relationship between ease of response and correct 

response.) Correlations ranged from 0 to .533. Under the test condition, the range was from -.127 to .665. Eight of the 

correlations were positive and four were negative. The negative correlations all represented a change of direction between 

the original and test conditions, which provides additional evidence of an interaction. That is, the absence of the “Does not 

include” statement in the test condition seems to have altered the relationship between ease of the industry verification task 

and the likelihood of reaching the correct answer. For example, in one scenario (Gift, novelty, and souvenir stores), the 

difference in the percentage correct between the original and test conditions was 11.1 percent. The correlations between ease 

and correct response for this scenario were in the same direction, but differed by a factor of 10 (0.533 for the original 

condition, 0.053 for the test). Although study participants in the test condition found the task more difficult, they still 

generally got the correct answer.12 

Turning to the complex scenarios, correlations ranged from -.410 to .604 for the original industry descriptions and from -.261 

to .360 for the test condition. Ten of the 12 correlations were positive under the original condition, while 8 of the 12 were 

positive for the test condition. 

What do these correlations tell us? Under both sets of scenarios, using the original version of the industry descriptions, there 

is a positive relationship between the likelihood that a study participant arrived at the correct answer and the reported ease or 

difficulty of the scenario evaluation task. The situation for the test condition is mixed. Four of the correlations were negative 

under both the straightforward and the complex scenarios. A negative correlation indicates that the harder the task, the more 

likely the study participant was to provide the correct answer. We can speculate that, in some cases, the study participants 

found the test versions of the industry descriptions more challenging and therefore worked harder to get the correct answer. 

Overall, there is no consistent pattern of relationships across industry description conditions within industries. Looking at the 

straightforward scenarios, half have higher correlations between ease or difficulty and correct response for the test conditions 

than for the original conditions. The other half have either lower positive correlations or correlations that change direction. 

Under the complex environment, the pattern is even more mixed, which we interpret to mean that the relationship between 

ease or difficulty and correct response is not affected by the presence or absence of the “Does not include” statements. These 

results again call into question the state QCEW staff expectations (i.e., that removing the “Does not include” statements will 

make the respondent task easier and result in more accurate responses). 

We also looked separately at the correlations between ease/difficulty and correct response for the verify and reject tasks, but 

found nothing to suggest a pattern or an effect. 

Insights from Debriefing Interviews 

We now turn to the debriefing interview, which we administered after the study participants had finished working with a set 

of 24 vignettes. The interviews had two goals: 

• To further explore perceptions about the industry descriptions, especially the use of examples and the “Does not 

include” statements 

• To see how study participants understood the industry description questions on the ARS form.13 

12 This straightforward scenario did not perform well overall. The vignette presents a business that specializes in children’s 

books and reading-oriented toys, while the industry on the ARS form describes “in-store retail sales of new gifts, novelty 

merchandise, souvenirs, greeting cards, seasonal and holiday decorations, and curios.” The description did not fit the 

category, and comments from several study participants recognized that fact. A few interpreted the business in the vignette to 

be a novelty store rather than a bookstore. 

13 This section requires a caveat: We did not audiotape the interviews, so respondent feedback was affected by the speed and 

clarity of the interviewer’s notes. However, we believe we have captured the essence of the comments. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7. Correlations between Ease/Difficulty Ratings and Correct Response, 

by Scenario and Industry Description Condition 

NAICS Industry 

Straightforward 

Scenarios 

Complex 

Scenarios 

Industry description Industry description 

Original Test Original Test 

238221 

Residential plumbing and HVAC Contractors 0.131 0.562* 0.270 0.444 

441110 

New car dealers 0.246 0.665** 0.115 -0.261 

444190 

Other building material dealers 0.178 -0.115 0.086 0.360 

447190 

Other Gasoline Stations 0.000 0.315 0.508* -0.029 

453220 

Gift, novelty, and souvenir stores 0.533* 0.053 0.108 -0.178 

453998 

Store retailers not specified elsewhere 0.112 -0.127 0.375 -0.197 

541940 

Veterinary services 0.217 0.411 0.142 0.027 

624410 

Child daycare services 0.203 0.094 0.271 0.275 

713940 

Fitness and recreational sports centers 0.386 0.531* -0.410 0.023 

811111 

General automotive repairs 0.132 -0.099 0.604* 0.083 

812112 

Beauty salons 0.076 0.438 0.174 0.347 

813410 

Civic and social organizations 0.431 -0.033 -0.320 0.180 

a Ease/difficulty rating: 1 = Very difficult, 5 = very ease * p < .05 ** p < .01 

We addressed the first goal with general questions about Items 9 and 10 on the ARS form (see Figure 1). Our intent was to 

look at whether the includes and the “Does not include” statements affected the study participants’ ability to understand the 

industry descriptions printed on the forms. Observations from the debriefing interviews are qualitative, and conclusions from 

them are suggestive. Nevertheless, they are useful for helping to round out the picture from the experimental results. 

The second goal was to see how our study participants understood the basic industry questions, which have been used in the 

field since 1999. The ARS forms were designed with the expectation that the survey respondent would be part of the business 

establishment and would have some knowledge of an establishment’s products or services. Obviously this was not true in the 

laboratory setting, where “establishment knowledge” was imposed through the vignettes. At the same time, the QCEW 

program is aware that many ARS respondents are not part of the business establishments named on the forms. These 

respondents may be accountants, professional employer organizations, or other third parties, including company headquarters 

personnel.14 That is, real respondents (as well as laboratory respondents) might not be familiar with an establishment’s 

industry, so it is important to ensure that these questions can be readily understood outside the business establishment. 

14 The questionnaire instructs third party respondents to consult with their clients about the industry description. We have no 

data on how often these third-party respondents actually make the contact. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Use of examples. Almost all of the study participants found the examples or statements of what to include helpful; and about 

three-fifths also said the examples were not confusing. We asked how they used examples, and several mentioned comparing 

the vignette’s business activities with the examples. Those who found the examples confusing made references to industry 

descriptions that did not fit the scenario (i.e., the rejection task). 

“Does not include” statements. Virtually all of the study participants noticed the “Does not include” statements, and almost 

all found them helpful. At the same time, roughly a fourth of them also indicated that these statements could be confusing. 

Study participants said that they used the information in the “Does not include” statements by comparing it to the business 

described in the vignette. Some used the “Does not include” information as a decision-making tool, especially if they were 

able to see immediately that the scenario didn’t fit the industry in Item 9. 

During the interview, we showed study participants an example of an original industry description for one of the scenarios 

where they had worked with the test condition. We asked if having the “Does not include” statement would have helped 

them with the vignette in question. Two-thirds of the study participants said that it would have, and many indicated that they 

would have used the “excluded” information to make their decision. 

Industry description question. On the ARS form, the Item 9 and Item 10 questions are: 

Our records show that the main activity of the business using U.I number 9999999999 

in Utana is: 

followed by the formatted description associated with the NAICS code on record for the business. Item 10 says: 

While you may not do everything listed above, does the information in Item 9 accurately describe the

main business in Utana during the past 12 months? (If the business has been closed, sold, or moved

out of this state, please answer in terms of its former activity.) 

We assessed comprehension of Item 10 by asking study participants to paraphrase the question, and then probed further about 

specific key phrases. Some of the participants struggled with the idea of putting the question into their own words. Those 

who tried generally grasped the intent of Item 10, which focuses on the firm's primary economic activity. However, a few 

seemed to miss the point. “Whatever will keep the business afloat” is not a correct representation of these items. “Based on 

description” is an inadequate answer, which might or might not reflect an understanding of the question. And some 

responses were similarly ambiguous, e.g. “Even though the scenario does not include what is listed you can use what is 

available to give an accurate answer.” 

Most of the study participants also grasped the idea of the “main business” and “the past 12 months,” although a few had 

different ideas. One study participant thought the main business referred to “one of the main businesses in that particular 

area” or “the major business in the state, the number one business in the area.” Most understood “the past 12 months.” 

Finally, we asked whether or not the study participant thought the business had to do everything in the write-up. About half 

said “No,” which is the correct answer. Seven of the remaining study participants answered “Yes,” and added a qualification 

to the effect that otherwise the business would be guilty of false advertising. These study participants apparently did not 

understand the debriefing question, a problem we inadvertently created by using the word “write-up” instead of referring 

precisely to the industry description. 

VI. Discussion 

The objective of this research was to see whether the presence or absence of “Does not include” statements in industry 

descriptions affects the likelihood that a business respondent will agree with (verify) a correct description or disagree with 

(reject) an incorrect description. We selected 12 common industries and created two vignettes describing fictitious 

businesses in each of those industries. The straightforward scenario for each industry is drawn from the general information 

and examples of the industry description currently used on the ARS form. The complex vignette addresses information that 

only appears in the “Does not include” portion of that industry description. Vignettes were written so that slightly more than 

half were accurately described on the ARS forms, while the remainder did not correctly fit the fictitious businesses. 

In a laboratory setting, we presented study participants with mock ARS forms containing industry descriptions, some with the 

“Does not include” statements and some without them. The laboratory study allowed us to investigate several industries at 
 

 

 

 

 

 

 

one time, and to know in advance whether the answer given was correct or not. We asked study participants to consult the 

industry description on the form and to determine whether or not that description accurately reflected the business described 

in the scenario. We also asked them to rate the ease or difficulty of the verification task on a five-point scale. 

Across all scenarios, there was essentially no difference in the percentage of correct responses between the original and test 

versions of the industry descriptions on the ARS forms. Industry descriptions with and without the “Does not include” 

statements worked equally well for both the straightforward and the complex vignettes. We noted that across industries, the 

“Original minus Test” difference changed direction in inconsistent ways. This further supports our contention that, overall, 

the “Does not include” statements have little influence on whether or not the industry item is answered correctly. 

Bearing in mind that not all of the scenarios were created to match the ARS forms descriptions, we separated the scenarios 

according to the underlying decision task, i.e., whether the correct response for the ARS form was “Yes” or “No.” We found 

that when a study participant should verify the existing information (i.e., the correct response is “Yes”), there was a tendency 

for the test version of the industry description to yield a greater number of correct responses, a finding that was statistically 

significant for complex scenarios. That is, the absence of the “Does not include” seemed to produce better data, a result in 

keeping with our expectations. On the other hand, when the study participant should reject the ARS description, the original

version of the industry description worked better. Although we cannot confirm this conclusion without further research, we 

speculate that the presence of the “Does not include” statement gives study participants more information with which to 

work, and particularly with which to judge that there is a mismatch between the business in the vignette and the industry 

shown on the ARS form. We saw some additional evidence of this when we looked at the reported ease/difficulty ratings in 

conjunction with the need to verify or reject the ARS industry description (Table 6). While there was no difference between 

the original and test versions for industries that were correct, when the right answer for a scenario was “No,” the original 

descriptions had higher ease ratings than the test versions. Since the only difference between the original and the test 

conditions was the “Does not include” statements, we infer that study participants are using the information in these 

statements to confirm that the industry does not fit their situation. In fact, we learned during the debriefing interviews that 

the participants used the “Does not include” statements to help rule out the description on the ARS form. 

Even though these findings are based on a very limited set of data, there are implications for the ARS. On one hand, the vast 

majority of ARS respondents verify the industry descriptions on their survey forms. State coders are well trained and know 

how to assign correct NAICS codes, and businesses are contacted every three years, offering ample opportunities for the 

states to “get it right.” On the other hand, the trend among state Unemployment Insurance offices is toward automated 

respondent self-coding, and toward allowing new business registrants only a few words to describe their activities. The new 

business registration process may be handled by a third party rather than an employee of the establishment. This trend 

increases the probability that the initially-assigned code will not be the correct one. The ARS is used to maintain the BLS 

business establishment sampling frame, so miscoded establishments have serious consequences for future samples, 

weighting, and published economic statistics. Since the “Does not include” statements help to rule out incorrect industries, 

the ARS might benefit from keeping the “Does not include” statements as part of their industry descriptions for the survey 

forms. These statements seem to be especially helpful in recognizing and correcting wrong NAICS codes. 

We cannot ignore the fact that the differences—and in many cases the lack of those differences—identified in this research 

could derive from the vignettes themselves, rather than from the presence or absence of the “Does not include” statement in 

the industry descriptions. We attempted to create realistic business settings that do not fit neatly into the NAICS schema. 

The laboratory subjects may have tried to read more into the vignettes than we intended. 

Another area that this study did not address is the wording of the industry descriptions. This wording, rather than the 

presence or absence of “Does not include,” may be what we see reflected in the ease ratings. During the debriefing, a 

comment that we heard quite often had to do with the wordiness of the descriptions, and the inclusion of seemingly irrelevant 

material. The descriptions list the NAICS index items as examples, and quite a few of the cross references are written into 

the “Does not include” statements. For example, many of the retail descriptions have a “Does not include” statement that 

refers to “nonstore sales of...” x, y, or z. Since the descriptions begin with “In-store sales of...,” perhaps this part of the 

description could be deleted without harm. It might also be helpful if a non-economist (perhaps from the BLS Publications 

Staff) could review the language in the descriptions, and work with the QCEW economists to ensure that industries are 

accurately described but in lay terminology. The final product would be a better industry classification tool, resulting in 

more accurate economic statistics. 
 

 

 

 

 

 

VII. References 

Eldridge, Jack, Jean Martin, and Amanda White. 2000. “The Use of Cognitive Methods to Improve Establishment Surveys 

in Britain.” ICES II: The Second International Conference on Establishment Surveys. Invited papers. Alexandria, VA: 

American Statistical Association, pp. 307-316. 

Goldenberg, Karen. 1998. “Results of NCA Form Test.” Unpublished report, U.S. Bureau of Labor Statistics. 

Goldenberg, Karen L., Diane K. Willimack, Sylvia K. Fisher, and Amy E. Anderson. 2002. “Measuring Key Economic 

Indicators in U.S. Government Establishment Surveys.” Paper presented at the International Conference on Improving 

Surveys, Copenhagen, Denmark. 

Jordan, Robert. 2005. Personal communication. 

Morrison, Rebecca L., Kristin Stettler, and Amy E. Anderson. “Using Vignettes in Cognitive Research on Establishment 

Surveys.” Journal of Official Statistics 20 (2): 319-340. 

Murphy, John B. 1998. “Introducing the North American Industry Classification System.” Monthly Labor Review 121 (7): 

pp. 43-47. 

Phipps, Polly A., Shail Butani, and Young I. Chun. 1993. “Designing Establishment Survey Questionnaires.” BLS Statistical

Note Number 35. Washington, D.C.: U.S. Bureau of Labor Statistics. 

Stettler, Kristen, Rebecca L. Morrison, and Amy E. Anderson. 2000. “Results of Cognitive Interviews Studying Alternative 

Formats for Economic Census Forms.” Paper presented at the Second International Conference on Establishment Surveys, 

Buffalo, NY. 

Walker, James A. and John B. Murphy. 2001. “Implementing the North American Industry Classification System at BLS.” 

Monthly Labor Review 124 (12):15-21. 

U.S. Office of Management and Budget (OMB). 1997. North American Industry Classification System. United States, 

1997. Washington, DC: Executive office of the President. 

U.S. Office of Management and Budget (OMB). 2002. North American Industry Classification System. United States, 

2002. Washington, DC: Executive office of the President. 

U.S. Bureau of Labor Statistics. 2003. Employment and Wages. Annual Averages, 2002. Washington, DC. 
Appendix 1. Twelve Industry Descriptions as Currently Used on Annual Refiling Survey Form

NAICS Title Description 

238221 Residential plumbing 

and HVAC contractors 

Plumbing, heating, or air-conditioning contractors primarily working on REExamples of their work may include, but are not limited to:

* Natural gas fireplace inst

* Duct work (cooling, dust collection, * Lawn sprinkler system instexhaust, heating, ventilation) * Sewer hook-ups and connect* Fire sprinkler system installation * Sump pump installation * Furnace installation RESIDENTIAL contractors may provide both parts and labor when performing wperformed may include new work, additions, alterations, maintenance, and rDOES NOT INCLUDE primarily installing electrical controls for HVAC systems

441110 New car dealers Retailing NEW automobiles, light trucks, or both. Retailing these new vehi

combination with vehicle repair services, retailing used cars, or selling and accessories for these vehicles. Examples include retail sales of:

* Light trucks * Pickup trucks * Automobiles * Passenger vans * Sport utility vehicles * Cargo vans 

DOES NOT INCLUDE selling ONLY used vehicles.DOES NOT INCLUDE repairing vehicles without also selling new vehicles.

444190 Other building 

material dealers 

In-store retail sales of specialized lines of new building materials. Exalines include, but are not limited to, retail sales of:* Ceramic tiles * Fencing * Lumber * 

* Doors and windows * Glass * Plumbing supplies * * Electrical supplies * Lighting fixtures * Prefabricated buildings * DOES NOT INCLUDE hardware stores, paint and wallpaper stores, and home cenDOES NOT INCLUDE electronic home shopping, mail-order, or other non-store 

447190 Other gasoline 

stations 

specialized lines of new building materials. (1) Retailing gasoline, diesel fuel, gasohol, or other automotive fuels at without convenience store*s or food marts. OR (2) Retailing automotive fuels in combination with one or more other activrepairing vehicles, selling automotive oils, selling replacement parts or or serving food at restaurants. Examples include, but are not limited to:

* Truck stops withou

* Gasoline stations without convenience stores * Marine service stations DOES NOT INCLUDE repairing motor vehicles without retailing automotive fue

453220 Gift, novelty, and 

souvenir stores 

automotive fuels in combination with a convenience store or food mart. In-store retail sales of new gifts, novelty merchandise, souvenirs, greeti

and holiday decorations, and curios. Examples include, but are not limite

* Gift shops * Souvenir shops * Balloon shops * Greeting card shops * Christmas stores * Novelty shops * Curio shops 

DOES NOT INCLUDE retailing stationery.DOES NOT INCLUDE electronic home shopping, mail-order, or other non-store 

and gifts and novelties. 
Appendix 1. Twelve Industry Descriptions as Currently Used on Annual Refiling Survey Forms (con

NAICS Title Description 453998 Store retailers not 

specified elsewhere 

Retail sales of specialized lines of merchandise in stores. Also includes line of new and used merchandise on an auction basis. Examples include, b* Art supplies * Closet organizers * Hot tu

* Candles * Fireworks, permanent location * Police 

* Calendars * Flags and banners * Religi

* Cemetery memorials * Flowers, artificial or dried * Swimmi

* Collectors' items (autographs, * General merchandise auctions * Trophi

* Home security equipment 

coins, cards, stamps) DOES NOT INCLUDE retailing specialized lines of merchandise via electronic 

541940 Veterinary services order, or direct sale. Veterinary services. Licensed veterinary practitioners practicing of vete

dentistry, or surgery for animals; and providing testing services for licepractitioners.Examples include, but are not limited to:

* Veterinary clinics * Animal hospitals * Veterinary testing laboratories * Veterinarian offices 

DOES NOT INCLUDE providing veterinary research and development services, pveterinary pet care services, such as boarding or grooming pets, providing services or boarding horses, and transporting pets. 

624410 Child daycare 

services 

Child day care services of infants or children. Generally care is for prebut may include caring for older children when they are not in school and prekindergarten educational programs. Examples include, but are not limit

* Nursery schools * Child day care babysitting services * Preschool centers 

* Child or infant day care centers DOES NOT INCLUDE kindergarten educational programs provided in elementary 

713940 Fitness and 

recreational sports centers 

Operating fitness and recreational sports facilities featuring exercise anphysical fitness conditioning or recreational sports activities, such as sracquet sports. Examples include, but are not limited to:

* Physical fitness health spas wi

* Aerobic dance or exercise centers * Ice or roller skating rinks * Gymnasiums * Swimming pools * Handball, racquetball, or tennis club facilities DOES NOT INCLUDE diet and weight reducing centers; operating health resortrecreational facilities are combined with lodging; and recreational sports not operating sports facilities. 
Appendix 1. Twelve Industry Descriptions as Currently Used on Annual Refiling Survey Forms 

NAICS Title Description 

811111 General automotive 

repairs 

Providing a wide range of mechanical and electrical repair and maintenance automotive vehicles (such as passenger cars, trucks and vans) and all trailor engine repair and replacement.Examples include, but are not limited to:

* General automotive repa

* Automotive engine repair shops * Garages (except gasoline service stations) DOES NOT INCLUDE gasoline stations, automotive parts and accessories storesautomobile dealers also providing automotive vehicle repair service, automobile repair shops sparticular repair service, such as vehicle exhaust systems or transmissions

812112 Beauty salons repair and maintenance services, and automotive oil change and lubrication 

Beauty salons cutting, trimming, shampooing, weaving, coloring, waving, or providing facials and body waxing and wraps; or applying make-up (except peInclude any combination of these activities. Examples include, but are not 

* Facial salons or shops * Beauty parlors or shops * Hairdressing salons or shops * Combined beauty and barber shops * Unisex or women's hair stylist sh

* Cosmetology salons or shops DOES NOT INCLUDE barber shops or beauty schools.

813410 Civic and social 

organizations 

Promoting the civic and social interests of their members. These organizatand restaurants for their members. Examples include, but are not limited t

* Fraternal lodges * 

* Alumni associations * Granges * 

* Automobile clubs (except travel) * Non-federally recognized * 

* Booster clubs ` Indian tribal councils * * Ethnic associations DOES NOT INCLUDE insurance offices operated by fraternal benefit organizatiresidential fraternity and sorority houses; and providing travel arrangemenservices, such as automobile travel clubs or motor travel clubs. organizations 
Appendix 2: Straightforward and Complex Scenarios for each industry description 

NAICS Industry Scenario Type: Straightforward Scenario Type: Complex 

238221 Residential 

plumbing and HVAC contractors 

Call Jonathon, Inc. has 20 to 25 licensed plumbers on staff at any one time. Most of the time these plumbers work at construction sites for 

new office buildings. They may install ductwork for heating systems, or they may install piping, bathroom fixtures, sinks, and specialpurpose equipment such as showers in executive bathrooms. Once in a while a builder hires Call Jonathon to install kitchens, baths, and 

laundry rooms in new houses. However, the office construction business has been so good that owner Jonathan Underwood generally turns away other work. N

The owners of Frankfred Heating and C

business since 1978, maintaining and insconditioners, heat pumps, and similar eqhomeowners and apartment managers alof them have annual service agreementsrequests, Frankfred has added duct cleanan additional fee), but so far only a handadvantage of it. 

441110 New car 

dealers 

Would you buy a car from Cal? Cal Worthington Cars is owned and 

operated by a famous comedian, and his specialty is “good used cars with no mystery about them.” Since Cal hails from Texas, the sales 

staff dresses in western-style clothes, including cowboy hats and boots. Each customer receives a cowboy hat along with the car's ownership 

and maintenance history. N 

Chevy Chase Cars is owned and operatefun to shop here because the selling flooThe owner's notoriety and humor, along 

deals around,' result in a high volume of 

are a big part of those good deals, Chevy 

selection of quality pre-owned cars for s

444190 Other building 

material dealers 

Tile Inc. specializes in marble and ceramic European tiles for high-end and custom homes. The Beverly Hills Home and Garden Design Show 

recently showcased the store, featuring Tile Inc.'s Italian terra-cotta clay roof tiles and marble fireplace tiles. Y 

The Hoyt family has owned and operated 

generations. The store sells the usual co

electrical, and mechanical items that servcontractors. They can't compete with the 

don't sell lumber, plumbing fixtures, cabInstead, they offer personal attention and that Grandpa Hoyt set as the family stand

447190 Other gasoline 

stations 

In and Out Gas sells gas and other fuels. The station is open 24 hours a 

day, 7 days a week, with an attendant on hand. The station offers basic auto repair services on weekdays. Y

Truck Time is located near highway 53. 

it as a familiar and comfortable place to of basic maintenance and repair services

of the vehicles, next door there is a motea restaurant and a convenience store. 

453220 Gift, novelty, 

and souvenir stores 

Where's Miss Kittycat? specializes in children's books and readingoriented toys. The store features a reading room sized for kids ten and 

younger, and an adult-child area furnished with comfortable chairs and sofas where parents can read to their children. Kitty Sperling and her staff hold morning story hours each week for preschool children, and after-school book sessions for older kids. N 

Gail's Gallerie specializes in unusual glaitems for the home. Handblown glass cr

and dinnerware, vases—many with silk few examples of the merchandise Gail seto offer her merchandise online. Her onabout 15% of her business. 

453998 Store retailers 

not specified elsewhere 

Amy's Art caters to professional artists and to art enthusiasts at all 

levels. She stocks her store with drawing supplies, paints, brushes, 

canvases, easels, a complete line of artist papers, frames, and nearly 

everything else an artist might need. To encourage beginners, Amy's Art offers a handful of specialty classes in watercolor, acrylic, and oil painting and basic drawing. She is currently deciding whether the classes generate enough revenue to justify continuing to offer them. Y 

Autograph Here specializes in celebrity and online over the web. Customers wh

autographs for movie stars, television pepolitical leaders, where the most popular Tom Hanks and Julia Roberts. Howeverdeveloped an international following ove

for nearly three-fourths of its sales. The 

stocking rare autographs. Former presidLennon are among two recent online sale
Appendix 2: Straightforward and Complex Scenarios for each industry description (contin

NAICS Industry Scenario Type: Straightforward Scenario Type: Complex 

541940 Veterinary 

services 

The Middletowne Veterinary Clinic is a private animal hospital run by 

Dr. Mark Fisher and two associate veterinarians. Dr. Fisher and his 

associates see patients during regular office hours, and perform scheduled and emergency surgeries as needed. The Clinic has facilities 

for keeping its patients while they are being treated or recovering from surgery. The doctors also make “barn calls” when their services are needed to treat cows, horses, sheep, and other large farm animals. Y

Paws Here, a pet care facility with a staf

narian Julie Jenkins. Paws Here service7-day-a week, around-the-clock pet boaralso provide general animal health care (of minor illnesses) at their clinic 2 days 

624410 Child daycare 

services 

Tot Heaven is a preschool day care center. The center accepts children from ages 2 through 5. The owners believe that children benefit from educational programs even at those young ages, so they offer nursery school and other educational activities geared to preschoolers. Y 

Anderson Academy is a private school wIn response to many requests from parenDirectors recently approved adding a new 

Board authorized money for the school tclassrooms for the kindergarteners. 

713940 Fitness and 

recreational sports centers 

LadyFit is a fitness facility geared for women. The club has a completely equipped gym, and offers classes in aerobics and yoga. The facility also offers tennis and racquet ball classes. Y

Dorsey Lodge is a resort that promises soIn the winter, there is skiing, snowboard

and swimming pools are popular summe

course and tennis courts. Hiking is availDorsey Lodge spa offers massages and vtreatments. The Lodge also has a fully-erestaurants, and luxurious hotel rooms. 

811111 General 

automotive repirs 

Euromart Automotive Repair is a garage that specializes in maintaining 

and repairing Volvos, Saabs, BMWs, and other European-designed vehicles. Clients bring their cars in for services such as engine tune-ups and oil changes, as well as diagnosis of various car problems and repairs. Euromart Automotive repairs and replaces brakes, exhaust 

systems, electrical systems, and other automotive components. Y

Ken Wilmot owns and operates a Lube Ta specialty automotive service that only lubrications, but does them quickly. The

changes within an hour on weekdays. B

national franchise, if you have taken youanywhere in the country, the staff at Kenthe records and see when your car last ha

812112 Beauty salons The Hardwick Shop has a staff of three barbers, including owner Tom 

Hardwick, whose primary business is cutting hair for men and boys. The staff also offers shaves and trims beards. The sign in the window 

says “Ladies Haircuts,” but the overwhelming majority of the customers are men—many bringing in their young boys. N

The Cuts and Curls salon has a staff of 1manicurist, and one person who shamposervices such as permanents and hair colblow-dries. While most of the customer

children's hair. Cuts and Curls serves a s

customers in a separate room designated 

813410 Civic and 

social organizations 

Members of the local Historic Car Association (HCA) like old cars. About half of the members own vehicles manufactured before 1950 and proudly display them in parades each year. The most valuable vehicles are owned directly by the HCA. The Association has a paid staff that maintains the membership roster, produces a monthly newsletter, and cares for the HCA's cars in their “clubhouse,” an old but functional garage. Y

Hudson University's football and basketbpride and of funds. The Hudson U Club University's athletic program with financcourse the members attend games and chalso raises money to fund athletic scholaraiser is the Club-sponsored away-game pay a fixed price that includes game tickselected away games. Popular though thscholarship money still comes from genedirectly to the Hudson U Club.