1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Shiny Demonstration Utilizing 

Bootstrap Approach to the 

Application of First Digits 

Analysis

Matthew Corrigan

Statistical Methods Staff (SMS)

Current Employment Statistics (CES)
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Views expressed are those of the 

presenter and or author(s) and do 

not necessarily reflect the views or 

policies of the Bureau of Labor 

Statistics.

Disclaimer
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• 240 data collectors collect employment data from 

sampled establishments using the CATI system.

• A data collector using CATI may collect data from 50 to 

450 establishments each month. 

• Data collectors are randomly assigned establishments.

• Randomly selecting an establishment within collector

yields a slim possibility of discovering collection errors.

Problem Space
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Supeヴ┗isoヴs doミ’t use ケuaミtitati┗e マethods 

to prioritize which collectors to evaluate. 

• Currently selecting which collector to 

evaluate is done at random.

• There is difficulty in determining if a collector 

is manufacturing data, also known as 

curbstoning. 

Quality Control
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

How can we look at all collectors 

simultaneously and use the collected 

employment data to increase the 

probability of discovering systemic 

collection errors or curbstoning?

Question

7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Beミfoヴd’s La┘
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Beミfoヴd’s La┘ Iould He ideal iミ ouヴ 

case.

• A spike for a collector would lead 

supervisors to recheck the CATI calls 

for all collected establishments with a 

first digit the same as starting with 

the digit of the detected spike. 

Beミfoヴd’s La┘
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Not all Collector Data follow 

Beミfoヴd’s fiヴst digit pヴopoヴtioミs. Ho┘ 

do we use the concepts here within 

the context of our Problem Space?

Question
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Modified Beミfoヴd’s uses the saマe 

IoミIepts as Beミfoヴd’s La┘.

• Uses the proportion of first digits for 

all data IolleItoヴs as ouヴ Beミfoヴd’s 

first digit proportions.

• Compares the proportion of first 

digits of a single data collector to the 

Modified Beミfoヴd’s fiヴst digit 

proportions.

Modified Beミfoヴd’s
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Rank or prioritize data collectors by 

the likelihood of detecting a systemic 

problem.

• Be fair to the data collector. We do 

not want a method that suggests 

improper collection without 

conducting further investigation.

Goals
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

When looking at the proportion of 

first digits for all data collectors 

combined, what can we do when the 

combined proportions follow the 

Beミfoヴd’s pヴopoヴtioミs alマost exactly?

Question
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Siマplif┞ the data used iミ Modified Beミfoヴd’s 

proportions to only the categories that a 

collector has collected in. 

• Which industries did the collector report data 

from?

• What were the reported employment sizes of 

the establishments collected from? Were 

they Small, Medium, or Large?

Stratifying all Collected Data
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

First Digit Proportions of 

Stratified Collected Data
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Stratified Sample

All Collectors NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small nS1=243 nS2=6273 nS3=1783

Medium nM1=364 nM2=4969 nM3=392

Large nL1=110 nL2=2247 nL3=212

Individual 

Collector

NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Stratified Sample

All Collectors NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small nS2=6273 nS3=1783

Medium nM1=364 nM2=4969

Large nL3=212

Individual 

Collector

NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

What if I drew a stratified sample 

from all collectors similar to the 

Individual Collector I am comparing? 

Question
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Sampled Strata are in the same 

stratum the collector obtained 

employment data for. 

• Sampled size within the stratum are 

identical to that of the collector.

• Representative of the workload an 

individual interviewer has been 

assigned

Profile Sample
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• To compare the Individual Collector 

to the Profile Sample.

• Bootstrap a large number of 

subsamples.

• Collect the resulting first digit 

proportions of the subsamples.

Distribution of Profile Samples
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Bootstrap Subsamples

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=i NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample b=1 NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small uS2=49 uS3=23

Medium uM1=2 uM2=46

Large uL3=14

Subsample

b=10,000

NAICS Super 

Sector 1

NAICS Super 

Sector 2

NAICS Super 

Sector 3

Small u*S2,10000=49 u*S3,10000=23

Medium u*M1,10000=2 u*M2,10000=46

Large u*L3,10000=14
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

First Digit Proportions from 

Bootstrap

B {pヴopふdヱぶ, pヴopふdヲぶ, ..., pヴopふdΒぶ, pヴopふdΓぶ}

1 {0.32, 0.16, 0.18, 0.05, 0.07, 0.05, 0.05, 0.06, 0.04 }

2 {0.33, 0.24, 0.12, 0.11, 0.04, 0.06, 0.03, 0.03, 0.04 }

3 {0.36, 0.23, 0.08, 0.10, 0.06, 0.06, 0.05, 0.06, 0.02 }

4 {0.43, 0.15, 0.10, 0.09, 0.05, 0.06, 0.07, 0.02, 0.03 }

... {...,...,...,...,...,...,...,...,...}

9997 {0.28, 0.21, 0.14, 0.07, 0.08, 0.07, 0.06, 0.05, 0.03}

9998 {0.27, 0.21, 0.10, 0.11, 0.09, 0.07, 0.06, 0.05, 0.03}

9999 {0.26, 0.21, 0.12, 0.09, 0.10, 0.07, 0.05, 0.05, 0.05}

10000 {0.31, 0.20, 0.11, 0.09, 0.07, 0.08, 0.06, 0.04, 0.04}
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

The Violin plot is a doubled kernel density plot 

that is used to show multiple densities at once.

Violin Plot Display of First Digit 

Proportions
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• To determine how similar the IolleItoヴ’s 

distribution is to the Bootstrap of profile 

samples, I utilized the distance from the 

median of the Bootstrapped Samples for 

each first digit value.

• This is analogous to the Bootstrapped pvalue, determined by the percentages of 

Bootstrapped proportions higher or 

lower than that of the Collector being 

evaluated. 

Method of Comparison
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• H0: the collector’s first digit proportions come 

from the distribution estimated by the profile 

sample Bootstrap method. 

• H1: the collector’s values are not from the 

estimated distribution. 

• Test StatistiI: the IolleItoヴ’s first digit proportions. 

• The test measures the percentages of 

Bootstrapped samples that have higher and lower 

first digit proportions than the IolleItoヴ’s.

Hypothesis Testing
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

If the interviewer has a proportion of first digit 

ones that is in the most extreme 5% of the 

Bootstrap values, the p-value would be less than 

or equal to 0.05 for that digit. Since we have 

10,000 profile samples, the interviewer's 

proportion of ones would have to be higher than 

the top 250 or lower than the bottom 250 profile 

sample's proportions of ones to fail.

P-values

27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Compared Collector to profile samples with 

similar workloads.

Bootstrap with more than the standard 

2,000 samples.

Used Bonferonni correction for the Pmin test 

instead of less conservative tests.

Did not account for negative correlation 

between digits.

Fairness in Ranking Collectors
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Application for Supervisors
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• Sort Collectors by their Pmin scores. This 

increases the probability of discovering 

systemic collection errors or curbstoning by a 

specific Collector.

• Review the CATI recordings or perform a call 

back of establishments that provided a 

suspect first digit value for a particular 

Collector.

• Review the month-to-month violin plots of a 

suspect collector. 

Application for Supervisors
30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

• In Beミfoヴd’s DistヴiHutioミ aミal┞sis, theヴe is a tヴaditioミal 

methodology to rely on the Chi-square test to determine 

if the observed first digit frequencies match the 

e┝peIted Beミfoヴd’s DistヴiHutioミ fヴeケueミIies. 

• The issue with this test is that you will generally not 

have a significant result with small sample sizes. We had 

many collectors represented who do not have large 

enough samples to be detected by a Chi-square test.

• The Chi-Square test was not sensitive enough to be a 

meaningful discriminator for what we were trying to

accomplish.

Why Not Chi Squared Tests
31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Beミfoヴd’s La┘ Iマage :

http://www.isaca.org/JOURNAL/ARCHIVES/2011/VOLUME3/Pages/Understanding-and-Applying-Benfords-Law.aspx

Associated Paper: 

https://www.bls.gov/osmr/pdf/st170050.pdf

Sources
Contact Information

32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Matthew Corrigan

SMS CES

202-691-6937

Corrigan.Matthew@bls.gov