An Overview of Business Establishment Automated Classification of NAICS (BEACON) for the Economic Census

Federal Committee on Statistics Methodology

Machine Learning Applications

November 4, 2021

Daniel Whitehead

Brian Dumbacher

Economic Statistical Methods Division

U.S. Census Bureau

1

1

Disclaimer	

Any views expressed are those of the author(s) and not those of the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance applied. (Approval ID: CBDRB-FY22-ESMD001-001)

2

Outline

Background: Slides 4-12

Methodology: Slides 13-19

Example: Slides 20-25

Summary: Slides 26-28

Wrap-up: Slides 29-30

3

Background: North American Industry Classification System (NAICS)

U.S. Census Bureau classifies business establishments by NAICS code based on primary business activity

NAICS is utilized throughout the survey life cycle

Sample selection

Data collection

Analytical review

Publication

Hierarchical 6-digit coding structure

First two digits of NAICS code represent economic sector (22 – Utilities)

Additional non-zero digits add industry detail (221210 – Natural Gas Distribution)

4

Background: Economic Census

Conducted every five years for years ending in “2” or “7”

Represents approximately eight million establishments, covering most industries and all geographic areas of the U.S.

Key statistics include

Total number of establishments

Total number of employees

Value of sales, shipments, receipts, and revenue

Total annual payroll

Data products are presented by NAICS and geography

5

Background: What is BEACON?

Business Establishment Automated Classification of NAICS

A machine learning tool developed by the Economic Statistical Methods Division (U.S. Census Bureau) to classify NAICS for establishments based on a write-in business description

After the respondent provides the write-in, the text is outputted to the BEACON Application Programming Interface (API), which then applies the machine learning model

The BEACON API returns the most relevant NAICS codes for the respondent to choose from based on the provided write-in

6

6

Background: Principal Business or Activity Question from the 2017 Economic Census

Question asks respondents to describe their business

There are prelisted descriptions, but the respondent also has the option of writing in a business description

Manual coding of write-in text is resource-intensive

7

Source: 2017 Economic Census

Background: BEACON Overview

Goals

Assist respondents in self-designating their NAICS codes

Improve accuracy of self-designated NAICS codes

Reduce manual coding of write-ins

General idea

The respondent inputs a business description

BEACON returns a ranked list of NAICS codes at the 6-digit level

Methodology is based on machine learning, text analysis, and information retrieval (e.g., internet search)

8

8

Background: Current Status of BEACON

Plan is to use BEACON in the 2022 Economic Census (EC)

Three rounds of instrument usability testing conducted

Economic Census 2021 Industry Classification Report (currently in the field) will allow further testing with respondents

We continually refine methodology and incorporate new data

Refinement of methodology is ongoing, both now and after 2022 EC

9

Background: Training Data

Historic write-in responses to the Economic Census (EC)

Frequent write-in text that was autocoded during 2017 EC

Business descriptions from IRS SS-4 forms

Classification Analytical Processing System (CAPS) items

Harmonized System commodity descriptions

Variables

Business description text

Corresponding NAICS code

10
| Business Description Text              | NAICS  |
|----------------------------------------|--------|
| This is a car dealership.              | 441110 |
| R&D lab – medical/health           | 541715 |
| we mainly repair furniture, some sales | 811420 |

Background: Training Data

11
| DataSource        | Number ofObservations    | Advantages                                                       | Disadvantages                                                          |
|-------------------|--------------------------|------------------------------------------------------------------|------------------------------------------------------------------------|
| EC                | ~ 1,200,000(single-unit) | Represents target populationReflects natural language            | Descriptions not perfectly classifiedDescriptions contain misspellings |
| EC Autocoded      | ~ 98,000 *               | Improves consistency with autocoding during 2017 EC              | Relatively small data source                                           |
| IRS SS-4          | ~ 860,000(single-unit)   | Provides timely dataReflects natural language                    | Descriptions not perfectly classifiedDescriptions contain misspellings |
| CAPS              | ~ 1,490,000 *            | Provides a rich vocabularyDescriptions are classified correctly  | Does not always reflect natural language                               |
| Harmonized System | ~21,000                  | Provides examples of industry-specific abbreviations/terminology | Relatively small data sourceDoes not always reflect natural language   |

* Includes duplicates and variations of original observations

12

Sources: Economic Census (2002-2017), 2017 Economic Census Autocoder, IRS SS-4, Harmonized System, Classification Analytical Processing System

12

BEACON Training Data Breakdown by Sector and Source

Frequency

Methodology: Overview

Text cleaning

Remove common words and phrases (e.g., “the”, “has”, “for instance”)

Correct common misspellings

Dictionary

Words and word combinations that BEACON recognizes 

Words are cleaned, stemmed, and meet minimum frequency requirements

Associations between words and NAICS codes in the training data

“tobacconist” is highly associated with NAICS 453991 – Tobacco Stores

“retail” occurs in many NAICS codes and is therefore less predictive

13

Methodology: Overview

Model ensemble

Information retrieval models look at how words, combinations, and entire descriptions are distributed across NAICS codes

NAICS distributions are averaged, yielding relevance scores

Relevance scores

Range in value between 0 and 100

Reflect how confident BEACON is that the NAICS code is correct

14

Methodology: Text Cleaning

Convert to lowercase

Account for numbers and punctuation

Remove extra white space and common “stop” words (“the”, “and”, “or”, etc.) 

Stem

Apply prefix/suffix stripping rules to reduce number of word variations

For example, “manufacturing”  “manufactur”, “cars”  “car”

Correct common misspellings

Map stems of misspelled words to stems of correctly spelled words

For example, “manifactur”  “manufactur”

Lemmatize

Map synonyms and abbreviations to a common concept

For example, “mfg”  “manufactur”, “auto”  “car”

15

15

Methodology: Text Cleaning

Fictional examples

16
| Input Text                       | Clean Text           |
|----------------------------------|----------------------|
| This is a convenence store.      | conveni store        |
| automobile mfg                   | car manufactur       |
| We repiar watches & jewelry. | repair watch jewelri |

16

Methodology: Dictionary

Underlying BEACON is a dictionary of words/combinations that occur frequently in the cleaned training data

Current dictionary size: 399,590 words/combinations

All words/combinations in dictionary must occur at least 10 times in training data

All model features are based on this dictionary

17

Methodology: Model Ensemble

Three information retrieval models

“All” 

Consider all words/combinations (combs)

For each word/comb, look at how observations are distributed across NAICS

“Umbrella” 

Exclude words/combs that are subsets of other combinations

For each remaining word/comb, look at how observations are distributed across NAICS

“Exact” 	

Consider observations that use the exact same words/combs

Look at how these observations are distributed across NAICS

18

Methodology: Model Ensemble

For the “All” and “Umbrella” models

The NAICS distributions of the various words/combs are averaged using “purity weights” that give more weight to the NAICS distributions of words/combs that are more pure/predictive

The purity weight is a function of the maximum proportion.

Final scores

The scores from the “All”, “Umbrella”, and “Exact” models are averaged

Three model weight parameters  , , and  ( = 1 -  -  )

19

19

Example: Model Ensemble

20
| All                                                                                                                                                               | Umbrella                                                                                                                                         | Exact                                                                                                                                                       |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Look at how {retail} is distributedLook at how {bakeri} is distributedLook at how {retail, bakeri} is distributedAverage NAICS distributions using purity weights | The words {retail} and {bakeri} are subsets of {retail, bakeri}, so they are excluded from this modelLook at how {retail, bakeri} is distributed | Focus attention on observations in the training data that consist entirely of the words {retail} and {bakeri}Look at how these observations are distributed |

Input text: “This is a retail bakery.”      Clean text: “retail bakeri”

The words {retail}, {bakeri}, and the two-word combination {retail, bakeri} are in BEACON’s dictionary

Example: Sector Distribution of “retail”

21

Proportion

Example: Sector Distribution of “bakeri”

22

Proportion

Example: Sector Distribution of [“retail”,“bakeri”]

23

Proportion

Example: Sector Dist. of exact[“retail”,“bakeri”]

24

Proportion

Example: Abbreviated Final Results
| Rank | NAICS  | Score | Sector Description              | NAICS Description                                                  |
|------|--------|-------|---------------------------------|--------------------------------------------------------------------|
| 1    | 311811 | 52.24 | Manufacturing                   | Retail bakeries, baking bread, cakes, and other bakery products...   |
| 2    | 722515 | 16.29 | Accommodation and Food Services | Snack and nonalcoholic beverage bars,...                             |
| 3    | 445291 | 12.91 | Retail Trade                    | Baked goods stores                                                 |
| 4    | 445110 | 3.14  | Retail Trade                    | Supermarkets and other grocery stores (except convenience stores)  |
| 5    | 454390 | 2.44  | Retail Trade                    | Other direct selling establishments,...                              |

25

Summary: Performance

26

Under the latest testing, BEACON’s average response time was less than .2 seconds for up to 10 concurrent requests and less than 2 seconds for up to 250 concurrent requests.

BEACON achieved 87% accuracy on the most recent training/test split performed.

Summary: Facts About BEACON

There are currently 10,642 words, 141,515 two-word combinations and 247,433 three-word combinations in the BEACON data dictionary

BEACON has been tested on independent test data from the 2017 EC

BEACON returns 3-10 possible NAICS codes

Over 15 analysts have reviewed the BEACON output

27

I don't know if you want to specify how sub-second BEACON is (e.g., 0.2 seconds).

27

Summary: What can BEACON do for other Surveys?

Team continually refines methodology and incorporates new data

BEACON is currently being used in the Economic Census 2021 Industry Classification Report

BEACON can assist respondents in self-classifying their business activities for other programs

Economic Census will use respondent’s selection to direct respondent to appropriate questions

BEACON can reduce manual work required by analysts

28

28

Wrap-up: Special Thanks

Kyle Jeong

Justin Nguyen

Justin Z. Smith

Thagendra Timsina

29

Wrap-up: Questions? Email: Daniel.Whitehead@Census.gov

30

# image2.jpg

# image3.png

# Microsoft_Excel_Worksheet.xlsx

# Microsoft_Excel_Worksheet1.xlsx

# Microsoft_Excel_Worksheet2.xlsx

# Microsoft_Excel_Worksheet3.xlsx

# image3.jpg

# image1.png