Aiding a Holistic View of Businesses Through Intensive Respondent Research

Diane K. Willimack

Methodology Director for

Measurement & Response Improvement

Economic Statistical Methods Division

1

Federal Committee on Statistical Methodology Research & Policy Conference

Virtual

November 2-4, 2021

Any views expressed are those of the author and not necessarily those of the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (CBDRB-FY22-ESMD006-004).

1

In-Scope Economic Surveys

2

2

Manufacturing

Manufacturers’ Unfilled Orders Survey (M3UFO)

Trade

Annual Wholesale Trade Survey (AWTS)

Services

Multisector

Annual Survey of Manufactures (ASM)

Annual Retail Trade Survey (ARTS)

Services Annual Survey (SAS)

Annual Capital Expenditures Survey (ACES)

3

HYPOTHETICAL COMPANY

CURRENT Collection Strategy: INDUSTRY-BASED

3

ASM & M3UFO

For each Manufacturing plant:

Revenue

AWTS

For the Wholesale portion of the enterprise:

Revenue

ACES

For each Industry the company had operations:

Capital Expenditures

ARTS

Expenses

Inventory

For the Retail portion of the enterprise:

Revenue

Expenses

Inventory

Expenses

Inventory

Why is this a problem?

Measurement error

Definitions 

Lack of harmonized questions/content

The “unit” problem

Reporting unit vs. unit of analysis

Availability of data at different levels

Burden

Mismatch between requested data and recorded data

Respondent’s access to data

4

4

Research Questions:

Definitions:  how do businesses define themselves relative to the Census Bureau definitions?

Accessibility:  how accessible are key data points at varying business units?

Burden:  how resource intensive is gathering data at these varying business units?

5

5

Record-Keeping Studies (RKS) – Two Phases

Phase I

Focus on accounting practices and purposes

Carried out during autumn 2019

29 in-person interviews

Phase II

Focus on “units” and data accessibility

Carried out during winter 2020-2021

30 remote/virtual interviews

6

6

In-Scope Businesses and Respondents

Eligibility Criteria:

Sampled in at least two in-scope surveys

In at least two industrial sectors

More than one establishment

7
|                                      | Phase 1 | Phase 2 |
|--------------------------------------|---------|---------|
| Total Number of Completed Interviews | 29      | 30      |
|                                      |         |         |
| Number of Industries                 |         |         |
| Three or fewer                       | 23      | 25      |
| Four or more                         | 5       | 5       |
|                                      |         |         |
| Number of establishments             |         |         |
| 30 or fewer                          | 10      | 19      |
| 31 or more                           | 18      | 11      |

7

RKS Phase I: Structure of Business & Records

Chart of Accounts

Assets

Liabilities

Income

Expenses

What does your business do?

Line of business

Industry classification = NAICS

The Mainstay -- Consolidated financial reports

Consolidated figures reliable and reportable

Serve as anchors for breaking out details

NAICS is artificial

Distinction among classifications

Different questionnaires = different industries (4- or 6-digit NAICS)

Mismatch: Recorded data ≠ Requested data

Rs dislike deciding how to map their data to the survey questions. 

8

Interview Protocol

Key Findings

8

RKS Phase II: Data Accessibility

Definitions & Equivalencies

Company

Establishment

Line of business

NAICS

“General” = 4-digit

“Detailed” = 6-digit

Color-coding

9

Interview Protocol

Accessibility  X  Unit  X  Topic

9

10

Results: Color-coded Accessibility  X  Unit  X  Topic

10

Revenue: Accessibility  X  Unit

11
| REVENUE           |       |        |        |     |   |
|-------------------|-------|--------|--------|-----|---|
|                   | GREEN | YELLOW | ORANGE | RED | X |
| COMPANY           | 29    | 0      | 0      | 0   | 1 |
| ESTABLISHMENT     | 15    | 9      | 3      | 1   | 2 |
| LINE OF BUSINESS  | 13    | 10     | 2      | 0   | 5 |
| STATE             | 13    | 8      | 7      | 0   | 2 |
| SPECIFIC INDUSTRY | 6     | 7      | 9      | 2   | 6 |
| GENERAL INDUSTRY  | 9     | 9      | 4      | 1   | 7 |

11

12

12

"EASILY ACCESSIBLE“ (GREEN) DATA BY UNIT BY TOPIC

NEW Collection Strategy: TOPIC/ACCOUNT-BASED

13

HYPOTHETICAL COMPANY

13

Total Revenue

Enterprise Level for:

Total Expenses

Enterprise Level for:

End-of Year Inventory

Enterprise Level for

Retail

Wholesale

Wholesale

Manufacturing

For each plant

Manufacturing

Retail

Manufacturing

Wholesale

Retail

For each plant

For each plant

Next Steps for Respondent Research: Cognitive Testing

Primary research questions:

Understand respondents’ preferences regarding organization of the modules and their content

Evaluate alignment of the module topics and with respondent’s record keeping practices

Learn about respondents’ strategies for compiling the data for the questionnaire, with an eye towards identifying difficulties and mismatches between recorded and requested data, along with how those mismatches are resolved

Assess the capabilities of respondents answering via spreadsheet upload for establishment-level collection

Assess respondent burden

14

14

Next Steps for Respondent Research: Pilot Testing

15

15

Phase I:  Intensive Pretesting

N = 100 production cases

Personal recruitment, online pilot survey instrument, Response Analysis Survey, record check study and debriefing interviews.

Phase II:  Wider Roll-Out 

N = 500 non-production cases and nonresponding production cases.

Phone recruitment (production cases), email and mail invitation (all), pilot survey, Response Analysis Survey, debriefing interviews and record check study.

October 2021 – April 2022

March – September 2022

Phase III:  Systems Checks and Final Testing

N = Up to 2,000 non-production cases + responding Phase I cases

Full contact strategy, pilot survey, Response Analysis Survey, debriefing interviews, statistical analyses.

November 2022 – December 2023

Acknowledging the Respondent Research Team

Melissa Cidade

Kristin Stettler

Demetria Hanna

Rebecca Keegan

Aryn Hernandez

And We Thank Alumni:

Erica Marquette

Struther Van Horn

For questions or 

additional information, 

contact:

Diane K. Willimack

Team Lead diane.k.willimack@census.gov

16

16

# image2.png

# image3.png

# image4.png

# image1.png