Ensemble Modeling Techniques for NAICS 

Classification in the Economic Census

Federal Committee on Statistical Methodology

Machine Learning Applications

October 24, 2023

Daniel Whitehead

Brian Dumbacher

Economic Statistical Methods Division

U.S. Census Bureau

1
Disclaimer

Any opinions and conclusions expressed herein are those of the authors 

and do not reflect the views of the U.S. Census Bureau. The Census 

Bureau has reviewed this data product to ensure appropriate access, 

use, and disclosure avoidance protection of the confidential source data 

used to produce this product [Data Management System (DMS) 

number: P-7504847, subproject P-7514952; Disclosure Review Board 

(DRB) approval number: CBDRB-FY23-ESMD002-034].

2
Outline

• Background: Slides 4 - 10

• BEACON Methodology: Slides 11 - 13

• Model Stacking: Slides 14 - 18

• Results: Slides 19 - 25

• Conclusions/Contacts: Slides 26 - 27

3
Background: North American Industry 

Classification System (NAICS)

• U.S. Census Bureau classifies business establishments by NAICS code based on primary 

business activity

• NAICS is utilized throughout the survey life cycle

• Sample selection

• Data collection

• Analytical review

• Publication

• Hierarchical 6-digit coding structure

• First two digits of NAICS code represent economic sector (22 – Utilities)

• Additional non-zero digits add industry detail (221210 – Natural Gas Distribution)

4
Background: Primary Business or Activity 

Question from the Economic Census

• Question asks respondents 

to describe their business

• There are prelisted 

descriptions, but the 

respondent also has the 

option of writing in a 

business description

• Manual coding of write-in

text is resource-intensive

5

Source: 2022 Economic Census
Background: What is BEACON?

• Business Establishment Automated Classification of NAICS

• A machine learning tool developed by the Economic Statistical 

Methods Division (U.S. Census Bureau) to classify NAICS for 

establishments based on a write-in business description

6

Respondent 

provides write-in 

description

Text is outputted 

to BEACON API

API returns most 

relevant NAICS 

codes to 

respondent
Background: Goals of BEACON

• Assist respondents in selfdesignating their NAICS 

codes

• Improve accuracy of selfdesignated NAICS codes

• Reduce manual coding of 

write-ins

7
Background: Training Data

• Historic write-in responses to the Economic Census (EC)

• Frequent write-in text that was autocoded during 2017 EC

• Business descriptions from IRS SS-4 forms*

• Classification Analytical Processing System (CAPS) items

• Harmonized System commodity descriptions

• Variables

• Business description text

• Corresponding NAICS code

8

Business Description Text NAICS

This is a car dealership. 441110

R&D lab – medical/health 541715

we mainly repair furniture, 

some sales

811420

*IRS data used for internal statistical purposes only, in accordance with Title 26.
Background: Training Data

9

Data

Source

Number of

Observations

Advantages Disadvantages

EC ~ 1,611,000

(single-unit)

• Represents target population

• Reflects natural language

• Descriptions not perfectly classified

• Descriptions contain misspellings

EC Autocoded ~ 98,000 * • Improves consistency with 

autocoding during 2017 EC

• Relatively small data source

IRS SS-4 ~ 865,000

(single-unit)

• Provides timely data

• Reflects natural language

• Descriptions not perfectly classified

• Descriptions contain misspellings

CAPS ~ 1,508,000 * • Provides a rich vocabulary

• Descriptions are classified 

correctly

• Does not always reflect natural language

Harmonized 

System

~21,000 • Provides examples of industryspecific 

abbreviations/terminology

• Relatively small data source

• Does not always reflect natural language

*Includes duplicates and variations of original observations.
10

Data Sources: Economic Census (2002–2022), 2021 Industry Classification Report, Internal Revenue Service SS4 (2002–2016), Classification Analytical Processing System, Harmonized System

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

11 21 22 23 31 42 44 48 51 52 53 54 55 56 61 62 71 72 81 92

Frequency

BEACON Training Data Breakdown by Sector and Source

EC EC Autocoded IRS SS-4 Harmonized Sys CAPS
Methodology: Text Cleaning

• Removal of extraneous words/symbols

• Remove extra white space and common “stop” words (“the”, “and”, “or”, etc.) 

• Account for numbers and punctuation

• Correct common misspellings

• Map stems of misspelled words to stems of correctly spelled words

• For example, “manifactur”  “manufactur”

• Stem

• Apply prefix/suffix stripping rules to reduce number of word variations

• For example, “manufacturing”  “manufactur”, “cars”  “car”

• Lemmatize

• Map synonyms and abbreviations to a common concept

• For example, “mfg”  “manufactur”, “auto”  “car”

11
Methodology: Dictionary

• Words and word combinations that BEACON recognizes 

• All model features are based on the data dictionary

• Associations between words and NAICS codes in the training data 

influence predictions

• “tutor” is highly associated with NAICS 611691 – Exam Preparation and Tutoring

• “store” occurs in many NAICS codes and is therefore less predictive

12
Methodology: Model Ensemble

• Model ensemble

• Information retrieval models look at how words, combinations, and entire 

descriptions are distributed across NAICS codes

• Individual predictions are averaged, yielding relevance scores

• Relevance scores

• Range in value between 0 and 100

• Reflect how confident BEACON is that the NAICS code is correct

13
Model Stacking: Overview

• Separate prediction problem into two parts

• Create multiple models to generate initial predictions

• Use these predictions as inputs to meta-model

• Proposal:

• Generate predictions from component models within BEACON

• Refine initial predictions with predictions from meta-models

• Sources: 

• Todorovski, L. and Džeroski, S. (2003). Combining Classifiers with Meta Decision Trees. Machine Learning, 50, 223– 249. Netherlands: Kluwer Academic Publishers.

• Merz, C. (1999). Using Correspondence Analysis to Combine Classifiers. Machine Learning, 36, 33–58.

14
Model Stacking: Within BEACON

Standard

Umbrella

Exact

•Considers all words and combinations of words 

(up to 2)

•Ex. “car repair”

•Includes: “car”, “repair”, “car repair” (all 

qualifying write-ins)

•Excludes words/combs that are subsets of 

other combs

•Ex. “car repair”

•Includes: “car repair” (all qualifying write-ins)

•Excludes: “car”, “repair”

•Only considers complete write-in description

•Ex. “car repair”

•Includes: “car repair” (includes only write-ins 

consisting only of terms “car” and “repair”)

•Excludes: “car”, “repair”

15
Model Stacking: Within BEACON

• “Standard” and “Umbrella” models

• The NAICS distributions of the words/stems and word/stem combinations are 

averaged using “purity weights” that give more weight to the NAICS 

distributions of words that are more predictive.

• The purity weight is a function of the maximum proportion.

• Final scores

• The scores from the “Standard”, “Umbrella”, and “Exact” models are averaged

• Three model weight parameters wwssssssssssssssss , wwuuuuuu, and wweeee ee ( = 1 - ww aaaa -

wwuuuuuu )

16
Model Stacking: Proposal

• Generate predictions from component models within BEACON

• “Standard”

• “Umbrella”

• “Exact”

• Refine initial predictions with predictions from meta-models

• Logistic Regression

• Decision Tree

• Random Forest

17
Model Stacking: Evaluation Metrics

• Top-k

• Measures success rate where success is % of times that true NAICS code is 

found in top-k predictions within sector

• Evaluated for k = 1, 2, 3, 4, 5

• F1 score

• Harmonic mean of precision and recall

18
Results

Exact model is least accurate 

of individual component 

models.

BEACON model tends to 

produce more accurate 

predictions than any of its 

component models.

19

Data Sources: Economic Census (2002–2022), 

2021 Industry Classification Report, Internal 

Revenue Service SS-4 (2002–2016), 

Classification Analytical Processing System, 

Harmonized System
Results

Meta-models offer similar 

potential for incremental 

improvement.

Random forest and decision 

trees methods appear most 

promising.

Potential improvement in 

manufacturing sector (31) is 

encouraging. 

20

Data Sources: Economic Census (2002–2022), 

2021 Industry Classification Report, Internal 

Revenue Service SS-4 (2002–2016), 

Classification Analytical Processing System, 

Harmonized System
Results

Similar results were found at 

other NAICS levels.

21

Data Sources: Economic Census (2002–2022), 

2021 Industry Classification Report, Internal 

Revenue Service SS-4 (2002–2016), 

Classification Analytical Processing System, 

Harmonized System
Results

Meta-models improved 

performance in sectors where 

BEACON was already doing 

well.

F1 score of BEACON in NAICS 

codes where meta-model 

outperformed BEACON is 

higher than that of NAICS 

codes where BEACON 

outperformed meta-model. 

22

Data Sources: Economic Census (2002–2022), 2021 

Industry Classification Report, Internal Revenue 

Service SS-4 (2002–2016), Classification Analytical 

Processing System, Harmonized System

RF Tree LR

NAICS codes where F1 score 

of meta-model > F1 score of 

BEACON 0.93 0.93 0.95

NAICS codes where F1 score 

of BEACON > F1 score of 

meta-model 0.85 0.85 0.89

Median BEACON F1 score: 

3-digit NAICS codes
Results

Meta-models improved 

performance in sectors where 

BEACON was already doing 

well.

F1 score of BEACON in NAICS 

codes where meta-model 

outperformed BEACON is 

higher than that of NAICS 

codes where BEACON 

outperformed meta-model. 

23

Data Sources: Economic Census (2002–2022), 2021 

Industry Classification Report, Internal Revenue 

Service SS-4 (2002–2016), Classification Analytical 

Processing System, Harmonized System

RF Tree LR

NAICS codes where F1 score 

of meta-model > F1 score of 

BEACON 0.89 0.89 0.89

NAICS codes where F1 score 

of BEACON > F1 score of 

meta-model 0.85 0.86 0.85

Median BEACON F1 score: 

6-digit NAICS codes
Results Meta-models may offer 

more potential for 

providing a single 

predicted NAICS than for 

providing multiple NAICS 

codes to a respondent. 

24

Data Sources: Economic Census (2002–2022), 2021 

Industry Classification Report, Internal Revenue 

Service SS-4 (2002–2016), Classification Analytical 

Processing System, Harmonized System
Results

BEACON performed as well 

or better than meta-models 

as top-k goal was relaxed.

BEACON is well-equipped at 

providing multiple NAICS 

codes to respondent. 

25

Data Sources: Economic Census (2002–

2022), 2021 Industry Classification Report, 

Internal Revenue Service SS-4 (2002–2016), 

Classification Analytical Processing System, 

Harmonized System
Conclusions

• Meta-models performed best, compared to BEACON, at predicting 

single best NAICS code.

• BEACON performed as well or better than meta-models when goal 

was to include best NAICS code as one of several potential NAICS 

codes.

• Meta-models may be helpful in manufacturing sector, which has more 

NAICS codes than any other sector.

26
Contacts

27

• Email: Daniel.Whitehead@Census.gov

• Email: Brian.Dumbacher@Census.gov