First Test of a QCEW Business Supplement

 

QBS Team

Emily Thomas, OEUS DASLT, Team Lead

Tiffany Boulanger, OFO

Rick Clayton, OEUS OEIS

Michael Dalton, OEUS ERPDS

Jennifer Edgar, OSMR

Stacey Edwards, OTSP DEWS

Joseph Guntakatti, OTSP DEWS

Vinod Kapani, OEUS SMS

Nora Kincaid, OA DMS

Amrit Kohli, OTSP DEWS

Carol Rowan, OA DMS

Sharon Stang OEUS DASLT

Hugette Sun, OEUS ERPDS

October 5, 2018
i

First Test of a QCEW Business Supplement

October 5, 2018

Executive Summary 

Overview 

The team developed, tested, and implemented a short survey appended to the Annual Refiling Survey 

(ARS) Web System. The ARS is collected 100% electronically via ARS Web (single establishments) and 

NVM Web (multi-unit firms). This project utilized ARS Web only. Once a respondent clicked to submit 

the ARS, they transitioned to a page that asked them to participate in a new survey, the Business 

Research Survey (QBS). This survey asked the respondent their relationship to the company being 

surveyed (employed by the company or employed by a company filing on behalf of that company) and 

what department they work for in their company. It also asked a series of YES/NO questions about 

whether the respondent had access to and could provide answers to certain questions (ex. How many 

1099s does this establishment file?). The aim of this test was to determine the feasibility of appending a 

survey to the ARS Web system, and to get insight into what type of information ARS respondents could 

provide.

QBS Sampling 

The team used two different sampling methods for this test. 

1. Quota Sample: Opened the QBS system to all ARS respondents until 7,000 responses were 

collected. This collection included only units that are in-scope for the ARS and reported on the 

ARS Web system (single establishments with more than 3 employees). After the desired number 

of completed surveys were collected, the QBS system was closed. The quota sample approach 

did not stratify by industry or size class, and collected all respondents within a three-day time 

period. These respondents received no advance materials related to the QBS.

2. Random Sample: A traditional random sample of 10,000 units, stratified by 2-digit NAICS and 

size class, was drawn from the QCEW frame. This sample included units that were selected for 

the ARS (3,748 singles and 1,957 units selected from known multi-unit firms) and units that are 

excluded from the ARS because they have an annual average employment <3, or are in a lowchange NAICS (4,295 QBS-only units). The random sample of establishments were contacted in 

one of three ways:

a. Single units in the ARS and for which we had email addresses from prior ARS Web 

collection were e-mailed an ARS solicitation. (ARS-Prompted)

b. Single units in the ARS and for which we did not have an email address were mailed a

standard ARS solicitation web letter. (ARS-Prompted)

c. Small units, and multi-units in the QCEW were mailed a QBS solicitation letter. (QBSPrompted) 

d. A group of single units in the ARS were mailed a QBS solicitation letter to test the 

response to the QBS letter vs. the standard ARS web letter. (QBS-Prompted)
QBS Results 

Overall, the test was successful. We demonstrated that it is feasible to program and conduct a survey 

attached to the ARS Web system, and that a majority of ARS respondents continued on to complete the 

QBS. The response rates varied by sample group however, and further research is needed to understand 

whether this is primarily related to the fact that these units were out-of-scope for the ARS (e.g., small 

units not active and individual worksites of multi-units not used to responding to surveys) or to the QBS 

letter. 

Quota Sample: The QBS system was open for 3.5 days, two weeks following an advance letter, 

and collected 6,941 completed surveys, 502 partially complete surveys. 

Of the 74 percent of ARS respondents that clicked through to the QBS, 92.9 

percent completed the QBS. 

In total, there was a 69 percent QBS completion rate for ARS respondents in this 

time window who had the opportunity to participate in the QBS.

Random Sample: The ARS-Prompted (n=2,782) sample was open for 4 months and obtained a 

43 percent response rate. Of those respondents who completed the ARS, 71 

percent went on to complete the QBS.

The QBS-Prompted (n=7,038) sample was open for 4 months and obtained an

18 percent response rate. 

Response rates in the QBS-Prompted sample varied by whether the sample was 

in or out-of-scope for the ARS. Out-of-scope units had a 17 percent completion

rate, while in-scope units had a 24 percent completion rate. 

Within the random sample, there were only a few statistically significant differences in response rate by 

industry (retail trade had a higher response rate than average) or size class (small units had the lowest 

response rate, and units with 10 to 99 employees had the highest). As expected, the quota sample had a 

different distribution of establishments by industry and size class than the random sample; 

establishments were more likely to have 4 to 99 employees. During the 2018 ARS cycle, unclassified 

accounts were solicited and asked to report via ARS Web, thus a portion of the quota sample collected 

for this test included unclassified accounts, while the random sample did not.

A majority (74 percent of the quota sample respondents and 71 percent of the random sample 

respondents) clicked to begin the QBS after completing the ARS. Of the respondents who started the 

QBS, a majority (85 percent) answered all eight questions. Most respondents worked directly for the 

company selected for the survey, and were either in finance/accounting/payroll, general management 

or human resources. Depending on the department they belong to, there were differences in the type of 

information they indicated they could provide. 
Conclusions 

Based on the first pilot test, the team concludes that the QBS approach is feasible. There are 

outstanding questions that need to be addressed before developing an implementation plan:

1. Sampling methodology

2. Improve response rates for units that are out-of-scope for the ARS - include sampling strategies 

that focus on active small businesses and determine the most effective address for multi-units

3. Estimate cost and staff time required to run a QBS

Next Steps 

The team proposes one additional test to evaluate the outstanding questions noted above prior to 

moving the QBS into a production environment. Additionally, given DEWS, ARS and OMB time 

constraints, we propose one QBS per year: 

1. Test 2 July 2019

Test has four goals:

a. Refine sampling methodology for small units 

b. Refine quota sample methodology

i. IDCF development sprint in January 2019

c. Improve response rates

d. Develop a detailed cost model

- Questions will be the same as the first test with the potential to add some specific answers 

rather than Yes/No to some questions to evaluate the validity of the responses 

- New OMB Clearance required – submit package no later than Dec 1, 2018

- On DEWS development schedule for January 2019

2. Production 1 July 2020

Conduct the first production QBS using a topic of interest internally at BLS with the goal of 

producing publishable estimates. 

- Funding source will be internal at BLS, but will need to cover all aspects of QBS collection.

- New OMB Clearance required – submit package no later than Dec 1, 2019

3. Production # July 2021

Conduct a second production QBS using a topic of interest of an external customer, possibly a 

DOL-sponsored survey on employee drug testing 

- New OMB Clearance required – submit package no later than Dec 1, 2020
1

First Test of a QCEW Business Supplement

September 25, 2018

Michael Dalton Jennifer Edgar 

Study Design 

To test the feasibility of a ‘piggy-back’ survey attached to the Annual Refiling Survey (ARS) online 

collection system, two samples were drawn, a quota sample and a random sample. Each was aimed at 

evaluating the QBS in terms of unit and item response rates, as well as coverage by industry and size 

class. 

Sampling 

Quota sample

The QBS was live from 11:00 on 2/14/18 until 14:45 on 2/16/18. QBS was reopened at 07:09 on 2/20/18 

and closed at 10:18 on 2/20/18 in order to collect the remainder of the quota sample. No QBS specific 

letters or prompting was done; respondents received the standard, ARS production email or letter 

invitation. 

While the QBS was live, 9,972 ARS respondents were shown the QBS transition page.1 Of those, 74 

percent of respondents (n=7,025) clicked on the QBS link and answered at least one question. 69 

percent of ARS respondents (n=6,549) completed the QBS. An additional 502 respondents completed 

some, but not all, of the QBS. Of the respondents who clicked through to the QBS, 93 percent completed 

it. 

Results 

Random Sample

A random sample of 10,000 establishments was selected from the universe of the Quarterly Census of 

Employment and Wages (QCEW). An analytical sample of 9,820 remained after removing establishments 

that were determined to be out-of-scope, out of business, had an invalid survey (completed the QBS 

without completing the ARS), or were part of an administrative UI reclassification involving the state of 

Rhode Island. The sample respondents all had the opportunity to use the exact same survey instrument; 

however, the method of contact differed for subsamples of the random sample. 

a. Single units that were included in the ARS and for which we had email addresses from 

prior ARS Web collection were e-mailed an ARS solicitation, and single units that were 

included in the ARS and for which we did not have an email address were mailed an ARS 

 

1

Some from this sample were removed from the response rate calculations: 434 did not have a completed ARS in 

the window that the QBS was collected, and 44 completed the QBS outside the hours the QBS was intended to be 

open.
solicitation web letter. This group of units (2,782) is referred to as ARS-Prompted in the 

table below.

b. Small units2, and multi-units3that were included in the ARS, received a QBS solicitation 

letter. Also, a group of single units in the ARS was mailed a QBS letter to test the 

response to the QBS letter vs. the ARS web letter. This group of units (7,038) is referred 

to as QBS-Prompted in the table below.

Random Samples

Click Through Rate

(Completed at least 1 QBS 

question)

Complete Response Rate

(Completed all 8 QBS 

questions)

ARS-Prompted (n=2,782) 46% 43%

ARS Letter (n=2,032) 45% 42%

ARS Email (n=750) 48% 44%

QBS-Prompted (n=7,038) 19% 18%

ARS-Prompted Random Sample

2,782 units were selected from the ARS sample to be in the QBS. Of these, we had email addresses for 

750, which were used for initial contact, and the remaining 2,032 respondents received a letter.

Standard ARS emails were sent on February 21st with a follow up on March 12th. Respondents for 

whom no email address was available were sent standard ARS letters on March 5th and April 30th. The 

QBS was available for all these units until June 28th. 

Of these 2,782 ARS-Prompted units in this random sample, 46 percent (n=1,279) clicked the QBS link and 

answered at least one question. 43 percent (n= 1,188) of units completed the whole QBS. Conditional on 

completing the ARS, 77 percent clicked through to the QBS and 71 percent completed the QBS. 

Respondents who received their initial survey invitation by email had a 44 percent completion rate, 

compared to 42 percent of respondents who received a letter. 

 

2 We believe that units with employment of 0, 1 and perhaps 2 significantly attributed to the low response rate in 

the QBS-prompted sample. Appendix A explores this theory. 

3 Units were identified as being part of a multi using the MEEI indicator. Typically all sub-units of a multi-unit firm 

are collected in MWRWeb and not ARSWeb, but for this test sub-units of a known multi selected in the QBS 

random sample were collected as singles in ARSWeb.
QBS-Prompted Random Sample 

Included in the random sample are 6,091 establishments that were not part of the normal ARS sample 

(out-of-scope) but were chosen to test the QBS. These QBS-Prompted establishments still completed the 

ARS first, but they were contacted outside of the normal ARS procedures. All contacts for this sample 

were made via mail. The first mail was sent on March 23rd, with a follow-up sent on May 7th. There 

were 500 establishments where the mail was returned as undeliverable. These establishments remain in 

the sample for the purpose of calculating response rates. 

Of the QBS-Prompted respondents, 22 percent completed the ARS, 19 percent (n=1,330) clicked through 

to the QBS and 18 percent (n=1,257) completed the QBS. Conditional on completing the ARS, 87 percent 

clicked through to the QBS and 82 percent completed the QBS.

Additionally, there were 947 respondents selected from the ARS sample (e.g., in-scope for the ARS) who 

received the QBS solicitation letters. The goal of this subset was to evaluate the QBS letter, to see if 

asking respondents to answer a QBS had an effect on response rates rather than asking them to answer 

the ARS. The response rate for this group was 24 percent, significantly lower than those in-scope ARS 

respondents who received an ARS letter (43 percent). Looking just at the ARS response, we also see 

evidence that the QBS letter was not effective; fifty-seven percent of respondents who received the ARS 

letter completed the ARS, while only 20 percent of respondents who received the QBS letter completed 

the ARS. 

Response Timing 

One goal of the QBS is to collect data quickly. To evaluate that, the team looked at the response rates 

within the first 7 days the survey was open. During the first 7 days of collection, the ARS response rate 

was 11 percent. We saw a dramatic difference between the solicitation letters. Those respondents 

receiving an ARS letter had a 19 percent click through rate and an 18 percent completion rate. Those 

respondents receiving a QBS letter (both in- and out-of-scope for the ARS) had a 6 percent click through

and completion rate.

Random Samples:

First 7 days

Click Through Rate

(Completed at least 1 QBS 

question)

Complete Response Rate

(Completed all 8 QBS 

questions)

ARS-Prompted (n=2,782) 19.1% 17.8%

QBS-Prompted (n=7,038) 5.9% 5.6%

Graph 1 shows how the response rates for the two random samples evolved over the first 120 days. In 

both cases, the most completions occurred in the first 7 days. There is a fairly steady flow of 

respondents, with small jumps that correspond with additional contacts described above. As noted, the 

QBS-Prompted sample ended with a significantly lower response rate than the ARS-Prompted sample. 
Comparing completion rates by size and industry for the two random samples (ARS-Prompted and QBSPrompted), establishments of smallest size (0 to 3 employees) have the lowest completion rate (17

percent) and establishments of size 10-99 have the highest (35 percent). Retail Trade has a 21% 

completion rate, which turns out to be a statistically significant difference compared to all other 

industries. 

QBS Response Conditional on ARS Completion

Graph 2 shows the proportion of respondents who completed the QBS, conditional on completing the 

ARS. There are no statistically significant differences in completion rates from those who completed the 

ARS in the first 30 days to after the first 30 days. 
QBS Response Conditional on ARS Completion

The ARS is known for high response rates, and one question this test sought to answer was whether ARS 

respondents would be willing to complete a second survey. The QBS response rates conditional on ARS 

completion are remarkably high. In both samples, a majority of ARS respondents completed the QBS. 

Click Through Rate

(Completed at least 1 QBS 

question)

Complete Response Rate

(Completed all 8 QBS 

questions)

ARS-Prompted (n=1,662) 76.7% 71.2% 

QBS-Prompted (n=1,530) 87.0% 82.2%

Sample Comparisons 

The first thing to note in comparing response rates across the three samples is how, for the quota 

sample, the sample is conditioned upon having completed the ARS. Unlike the random samples, the 

respondents in the quota sample are only eligible for the QBS once the ARS is completed. The random 

sample respondents are selected independent of their ARS completion. Therefore, if an ARS-Prompted 

sample respondent refuses to complete the ARS, they are (unknowingly) refusing to complete the QBS. 

Given that the quota sample requires completion of the ARS, we will compare the three samples 

conditional on completing the ARS. These rates are more similar than the overall rates might suggest: 69
percent for the quota sample, 71 percent for the ARS-Prompted sample, and 82 percent for the QBSPrompted sample (which is statistically higher than the other two).

While respondents in the random sample had three months to complete the QBS, the quota sample was 

collected between 14 and 21 days after contact by the ARS. Since respondents who respond at various 

points in the survey period may differ, the QBS completion rate was calculated for each of the ARSPrompted and the QBS-Prompted samples, conditional on having completed the ARS 14 to 21 days after 

being contacted. This gives samples of size 174 and 92 for the ARS-Prompted and QBS-Prompted 

samples respectively. For just that 7 day window, the QBS completion rate is 67% for the ARS-Prompted 

and 84% for the QBS-Prompted sample. The completion rate increases from 82 to 84%, and is still 

statistically significantly different from the completion rate for the other two samples.

One potential explanation for the lower response rate of the QBS-Prompted sample is that respondents 

are being asked to complete the QBS, a new survey. While respondents may be less prone to take the 

time for a new survey, those that take the time may be a selected sample, who are more willing to 

complete surveys overall, which explains the higher QBS completion rate conditional on completing the 

ARS. Also, the QBS-Prompted sample was specifically asked to complete the QBS online in the 

solicitation materials which could explain the higher completion rate than the ARS-Prompted sample 

that were only asked to complete the QBS after they completed the ARS online. These respondents 

specifically logged into the ARS website to complete the QBS. It is feasible that they would be more 

likely to actually complete the QBS since that was their purpose in logging into the website in the first 

place. However, the actual letters were very similar between the QBS-Prompted sample and the 

standard ARS contact, although there were minor differences. The present study was not designed to be 

able to test what specific features in the mailings could have affected response rates. However, we are 

able to test for response effects by mailing prompt. 

There were 2,032 ARS-Prompted respondents who were in scope for the ARS and received an ARS letter 

by mail, and 947 QBS-Prompted respondents who were in scope for the ARS and received a QBS letter 

by mail. These two groups have similar compositions in terms of industry and size. By comparing 

response rates, we can identify whether the prompting between the two groups had an impact on 

response patterns. Overall, the ARS-Prompted group had a 42-percent completion rate, compared to 24-

percent for the QBS-Prompted group, and this is a statistically significant difference. Conditional on 

completing the ARS, the ARS-Prompted group had a 73-percent completion rate, compared to 88-

percent for the QBS-Prompted group, and this is also a statistically significant difference. There were no 

differences in response patterns for those that completed the QBS. This comparison shows that the 

mailings themselves had a significant impact on inducing respondents to complete the QBS, and this will 

need to be tested further in the next round of the QBS. 

Sample Composition 

In addition to differences in response rates, we consider the types of respondents who are represented 

in each of the samples. Ideally, all samples would have similar distributions across industry and size 

classes, and the distributions would mirror that of the QCEW. Graph 3 shows the industry distribution 
for each of the three samples, in addition to the distribution for the overall ARS 2019 sample as a 

comparison point. 

Looking at the respondents who completed the QBS, we see that there are differences by industry 

across the three sample groups (Graph 3). One notable difference in the industry composition of the 

samples is that the quota sample has a nonzero percentage of unclassified industries, whereas the 

random samples have none that fall into this category. Establishments without a defined industry were 

not chosen to be a part of the random samples, but comprised almost 6 percent of the quota sample.

4

There were statistical differences with QBS-Prompted units more likely to be in Finance and Insurance, 

Professional, Scientific and Technical Services, Real Estate and Rental and Leasing, Wholesale Trade, and 

Information. They are less likely to be in Construction and Accommodation and Food Services. The ARSPrompted sample is also more likely to be in the Utilities industry. 

Looking at the samples by establishment size, we again see differences by sample (Graph 4). One of the 

criteria for being in the QBS-Prompted sample was having 0 to 3 employees (the other was being part of 

a multi-unit). We can see that this was successful, more than half of the QBS-Prompted sample was in 

this category. Looking at the quota sample group, there are fewer large establishments (100 or more 

employees) that responded to the QBS relative to the random samples, but this seems to be consistent 

with the proportions seen in the 2019 ARS sample. However, this is still a small portion of the overall 

sample, which may be an issue if larger establishments are of interest to future surveys. 

 

4

Since conducting this test, the Data Collection Branch (DCB) has identified a way to exclude units in the ARSWeb 

system that do not have a valid 6-digit NAICS from a future QBS.
These differences will be important to keep in mind for future testing and refinement of the quota 

sample to ensure a distribution across all size classes. Looking across the sample groups, the small units 

in the QBS-Prompted sample have slightly different response patterns, both in terms of completing the 

QBS and the answers that they give in the survey. The minor differences in the mailing, and the fact that 

these establishments do not typically receive this survey invitation may influence the response rates. 

Additionally, the small units in this sample differ from the small units in the quota sample and ARSPrompted sample. Employment size here is defined by the average employment over the 3 months in 

the first quarter of 2018, whereas the employment size used to determine whether an establishment is 

in-scope for the ARS is the average of annual employment. Most of the QBS-Prompted small units have 

an annual average employment that is small, whereas the small units in the other samples simply have 

small employment for the quarter, possibly due to downsizing or seasonality5. This is another difference 

that may influence the composition of respondents. 

The other subgroup in the QBS-Prompted sample are establishments that are part of a multi-unit firm. 

There are 1,824 establishments that were selected that fit this criteria. These units tend to be in Retail 

Trade, Health Care and Social Assistance, and Finance and Insurance. These units have a 20% completion 

rate, are more likely to be in the company being surveyed, and are most likely to be in a Human 

Resources department. In the last 6 questions of the survey, they appear to be capable of providing less 

information than average. 

 

5

For example, a summer resort may have low employment in the first quarter of 2018 because they are mostly 

closed, but has larger employment during other months of the year. This unit would be in-scope for the ARS, but 

would be analyzed here as having low employment based on its monthly average for the quarter. 
Before analysis of the complete QBS data could be done, we need to assess whether we can combine 

the three samples and whether the samples lead to the same substantive conclusions. We start by 

looking at the type of respondent who completed the surveys by sample group (Graphs 5 and 6). The

majority of respondents in all three sample groups work for the company being surveyed. Again, there 

were no significant differences by sample group.

There do appear to be some sample group differences when looking at the department the respondent 

works in, looking just at respondents who work directly for the company being surveyed. Most 

respondents in all three samples worked in finance, accounting or payroll. QBS-Prompted respondents 

are more likely to be in Human Resources, and less likely to be in General Management, and these 

differences are statistically significant. 

The differences of establishment size across the three samples could be driving the differences in 

respondent department (Graph 6). Since the QBS-prompted sample had significantly more small units 

and significantly more respondents from human resources, we would expect to see that HR respondents 

are most often from these small units.
Observing Graph 7 however, that does not appear to be the case, and we identified no obvious 

explanation for the difference in respondent department between samples. To identify whether this has 

a significant impact on the results, we tested whether responses to the questions of substance (Q3-Q8) 

were significantly different between the samples, conditional on a respondent being in HR. There are no 

significant differences found, suggesting that the difference in responses to Q2 does not reflect broader 

discrepancies in the samples. 
The substantive QBS questions included in the pilot focused on the type of survey information the 

respondent could provide. The rates at which respondents said they could provide the requested 

information was relatively consistent across the sample groups, though the ARS-Prompted sample 

consistently has a higher rate of saying yes to being able to provide information, and this is a statistically 

significant difference (Graph 8).

For all eight questions, there is a statistical difference between the ARS-Prompted sample and the other 

two samples in how likely a respondent is to answer “Yes” to any of these questions. Overall, ARSPrompted respondents give 4.6 (out of 6) “Yes” responses, compared to 4.2 for both the QBS-Prompted

and quota samples. While the differences are statistically significant, this is due to the large samples and 

precise estimates resulting from the large estimates rather than reflecting an economically meaningful 

difference. 

Although the method of sampling and contact varied across the different sample groups, the survey 

instrument was the same for all the three groups. Thus, we combine the three sample groups for the 

purpose of analyzing the responses to the substance of the questions for the remainder of the paper. 

Impact of Mandatory Status 

Overall, there is a 71% QBS completion rate, out of 12,686 respondents completing the ARS. One 

element that may affect completion rate is whether a respondent is in a state where the law requires 

them to complete the ARS. Since the ARS is required in 29 states, it may be possible that this 

requirement makes someone more likely to also fill out the QBS, despite the fact there is no law 

requiring completion of the QBS. There is not a statistically significant difference in completion rate 

between ARS-required states, conditional on completing the ARS. 

However, this result differs by sample group. For the ARS-Prompted sample, there was a higher QBS 

completion rate in states where the ARS is mandatory (46 percent versus 39 percent). This significant 

difference goes away when conditioning on completing the ARS. This pattern was not found in the QBSPrompted or quota samples. 
Taken together, there is some evidence that being in an ARS-required state makes a respondent more 

likely to complete the ARS. Given that QBS completion is dependent upon ARS completion, we suspect 

that the overall QBS completion increases in ARS-mandatory states. However, once the respondent has 

completed the ARS, there is no impact on QBS completion for being in an ARS-mandatory state. 

QBS Completion 

Overall, 87% of respondents who began the QBS answered all 8 questions, with a fairly uniform 

distribution across answering 1 to 7 questions (Graph 9). Response rates by question show that 98%, 

98%, 94%, 93%, 92%, 91%, 91%, and 91% answered each question, respectively. There is a steady dropoff between question 1 to question 8, but the sharpest decline was between Question 2 and question 3. 

This may be explained by the fact the respondent must first scroll after question 2, at which point the 

respondent can see that there are more questions. 

Survey Timing 

We can also estimate how long a respondent takes to complete the QBS by comparing the finish time of 

the ARS with the finish time for the QBS. Overall, the mean time is 2.4 minutes, with a median of 1.9, 

25th percentile of 1.4 and 75th percentile of 2.6 minutes. The mean is right-skewed because of a 

handful of observations that took much longer, perhaps because the respondent paused to do 

something else in between completing the ARS and the QBS. 

We looked at the number of questions answered, QBS completion rate, and length of time to complete 

the QBS by day of the week and hour that the ARS was completed and found no significant differences; 

responses followed the same pattern regardless of when the QBS was completed. 
Respondent Information 

The majority of respondents (87 percent) work for the company being surveyed for the ARS and QBS.

The second largest group, 8 percent of respondents, work for an outside accounting firm (Graph 10).

Of the respondents who work for the company being surveyed (Question 1), 31 percent report being in 

Finance, accounting, or payroll, 26 percent in General Management, 18 percent report that their 

company does not have departments, 14 percent in Human Resources, and 8 percent report "Other" 

(Graph 11). 
When asked what information they would be able to provide (Question 3), a majority of respondents 

said yes to most of the questions, as shown in Graph 12. The number of layoffs had the highest “Yes” 

rate (75 percent) and the top three products had the lowest (65 percent). 

And finally, we looked to see if respondents tended to be able to provide information for only some of 

the questions, or if they had access to the variety of information asked about (graph 13). We can see 

that almost half of respondents said “Yes” to all six questions, with very few only being able to answer 

one or two. There were more than 10 percent that weren’t able to provide any of the information, 

which suggests that these respondents may have to forward a QBS to a colleague or otherwise work to 

gather the desired information. 
Respondent Information over Collection Period

To determine if respondents who completed the QBS early in the collection period had access to more 

information than those who completed it later, we look at the number of “yes” responses by collection 

day (graph 14). 

Overall, 49 percent of respondents who completed the QBS said they can provide information for all 6 

questions, with 14 percent saying no to all 6 questions, shown in Graph 13. We can see that there is a 

slight decline in ability to provide information as the collection period progresses. Respondents who 

completed the QBS in the first 30 days had an average of 4.5 “Yes” responses, and those who responded

after the first 30 days had an average of 4.2, a statistically significant difference. This may reflect either a 

statistical anomaly, a reduced willingness to provide information if that takes the respondent extra 

effort and time to get them to complete the survey, or a different type of respondent who has less 

information about the sampled unit. If it is one of the latter explanations, then this is important to be 

cognizant of because it lowers the potential value for follow-ups
Ability to Report by Respondent Characteristics 

Responses by Department

Respondents in the company being surveyed had a larger number of "Yes" responses (4.4 out of 6) 

compared to respondents not in the company being surveyed (3.2 out of 6), with the overall average for 

the full sample being 4.2 responses of “Yes.”

Of the respondents in the company being surveyed, there were some slight differences by department 

to the number of ‘“Yes” responses (graph 15). Respondents in “Finance, accounting, or payroll" gave 

fewer "Yes" responses (4.2) than those from other departments. These respondents were more likely to 

be able to provide information to Q7. This is not surprising as Q7 is about 1099 forms likely processed 

within that department. They were also less likely to be able to provide information for Q3 (advertising 

job openings), Q4 (number of job openings), and Q8 (layoffs).

Respondents in "Human Resources" gave fewer "Yes" responses overall (3.8). Respondents in this area 

of the company are more likely to say "Yes" to Q3 (advertising job openings) and Q4 (number of job 

openings), which again is not surprising as this information is relevant to their department. HR 

respondents were less likely to answer “Yes” to Q5 (total revenue), Q6 (top three products), and Q7 

(1099s). Respondents in "General Management" gave more "Yes" responses overall (4.9). 

Reporting that your company does not have departments makes someone more likely to give "Yes" 

responses (4.6). However, as noted later on, this is likely a factor that companies without departments 

are of a small establishment size, and being in a smaller establishment will make it easier to have access 

to the information necessary to answer these questions. 

Responses by Company Size

As shown in graph 16, respondents in the 10-99 size class have the highest completion rate for the QBS 

(36% compared to 22% for the other respondents). Respondents in establishments with employment 

over a hundred are much more likely to be in Human Resources (49 percent) or Finance, accounting or 

payroll (40 percent) compared to other respondents. These respondents in this employment size gave 

the fewer than average "Yes" responses (3.6). 

Respondents in establishments with 0-3 employees were much more likely to report not having 

departments in their company (28 percent compared to 14 percent for the other size classes), and were 

significantly less likely to answer "Yes" to Q3, Q4, and Q8 in the information questions. 
Responses by Industry

There was some variation in the number of “yes” responses by industry (graph 17), though the majority 

of industries were close to the overall average (4.2). Respondents in “Arts, Entertainment and 

Recreation,” “Other Services except Public admin,” and “Construction” gave the most "Yes" responses 

(4.8, 4.5, and 4.4 respectively). 

Respondents in “Mining, Quarrying, and Oil and Gas Extraction”, unknown industry , and “Admin and 

Support and Waste Management and Remediation Services” gave the fewest “Yes” responses with a 

statistically significant difference (3.4, 3.6, 3.9, respectively). 

Conclusions 

The team concludes that this test of the QBS was successful. We found that a majority of ARS 

respondents clicked through and completed the QBS. Additionally, patterns in the responses to the QBS 

show that the responses make sense and therefore provide quality responses. Valuable information 

about ARS respondents was collected, providing useful insight into determining what type of data the 

QBS could reasonably collect. There are three outstanding issues which need to be addressed before a 

long-term implementation plan can be developed: Sampling, response rates and cost. 
 

Sampling Methodology 

The ARS-Prompted sample had the highest response rate, and follows the design originally envisioned 

for the QBS. However, because the ARS does not include all types of establishments, either a 

supplemental sample will have to be drawn (such as the QBS-Prompted sample) and/or the QBS will 

have to be added to the MWR system as well. The supplemental sample had a significantly lower 

response rate (19 percent compared to 42 percent), so if this approach is taken, additional work is 

needed to increase that sample size. 

The quota sample was a successful way to collect a large number of responses very quickly, however we 

recognize that as tested a representative sample was not obtained. The team will work with SMS to 

develop and implement a sampling strategy to further evaluate the validity of using a quota sample. We 

believe there are several options to explore, such as having cell-level quotas by industry and size class. 

The QBS could also be opened and closed at different periods of ARS collection to potentially better 

approximate a random sample, and also in testing, to determine if early respondents differ in 

meaningful ways from those respondents who complete the survey request later in the fielding period.

Response Rates for Out-of-Scope ARS Units

For the random sample, the overall QBS response rate was 25 percent, even after four months and at 

least 2 contacts. Response rates ranged by sample group, with 43 percent of the ARS-Prompted random 

sample respondents responding and only 18 percent of the QBS-Prompted random sample respondents.

Work needs to be done to determine how to collect from both very small units (who may not be a 

traditional business or still active) and specific locations of multi-units (who may not be used to 

completing surveys for themselves).

Additionally, the ARS respondents who received a QBS letter had a significantly lower response rate

than those who received an ARS letter. To the extent that we will always need to have a sample to 

supplement the ARS sample to capture small units and multi-units, this needs to be addressed. The team 

suspects that factors such as the printer (The ARS letter was printed by a contractor while the QBS 

letters were printed at BLS with slightly different letterhead and envelopes), and differences in the 

language and survey appeal led to the decreased response rates. 

QBS Costs

The QBS presents a low cost opportunity for collecting new information. While costs are significantly 

lower than what would typically be expected to initiate a new survey, there are still costs associated 

with fielding a QBS survey both in terms of data collection and labor hours. During the second test of 

QBS collection, the QBS Team will identify and quantify costs for each step of the QBS process. Some 

example of areas with associated costs are below:

1. Data Collection costs
a. Postage and Printing (non-ARS units)

b. Contractor to handle non-response prompting

2. Labor Hours

a. Identifying and vetting topics, preparing survey questions

b. Preparing and submitting OMB Clearance materials

c. Updating and testing database 

d. Updating and testing collection instrument

e. Preparing email blasts, print files and reviewing proofs

f. Managing collected responses and response rates

g. Managing non-response contract

h. Compiling collected responses

i. Preparing estimates and other publication materials

j. Compiling final press release (internal) or materials for client (external)

Identifying a dedicated source for funding each activity will be necessary prior to fielding any QBS in 

production.

Data Validity

Finally, the QBS showed promise in terms of data quality. We saw patterns in the ”Yes” responses by 

department that make logical sense; respondents in financial or payroll positions were more likely to 

answer that they could provide financial/payroll-related information. The first QBS did not actually 

collect substantive data (e.g., we asked if they could tell us how many job openings their company had, 

not for the actual number). A future QBS could ask for actual responses to these questions to evaluate 

the validity of the data collected. 

Next Steps 

To address these points, the team proposes a second field test prior to any production QBS 

implementation. This test would provide necessary information about the outstanding unknowns and 

set the stage to conduct the first QBS in production. We believe that DEWS, ARS and OMB time 

constraints lead to the ability to conduct one QBS per year. 

Test #2 July 2019

Conduct a follow up test with the following three goals:

a. Refine sampling methodology for small units 

b. Refine quota sample methodology

i. IDCF development sprint in January 2019

c. Test new strategies for improving low response rate groups

- Questions will be the same as the first test with the potential to add some specific answers rather 

than YES/NO to some questions.
- New OMB Clearance required – submit package no later than Dec 1, 2018

- Develop a detailed cost model for each component of fielding the QBS 

Production #1 July 2020

Conduct the first production QBS using a topic of interest internally at BLS with the goal of producing 

publishable estimates. 

- Funding source will be internal at BLS, but will need to cover all aspects of QBS collection.

- Track costs to validate estimates. 

- New OMB Clearance required – submit package no later than Dec 1, 2019

Production #2 July 2021

TBD

- New OMB Clearance required – submit package no later than Dec 1, 2020

- Funding source will be internal or external, depending on topic/customer.
Appendix. Small Unit Analysis

To try to understand the difference between the ARS-prompted and QBS-prompted response rates, we 

looked specifically at the smallest units, those that are typically excluded from the ARS. Other BLS 

surveys, such as the CES and OES, also exclude these units from their sample and we suspected that they 

were leading to the lower response rate in the QBS-prompted sample. 

Graph A shows the QBS completion rate, based on rounded average monthly employment (AME) for 

March 2018. The three groups are based on whether the establishment is in scope for the ARS6 and 

whether the establishment received ARS-prompted or QBS-prompted materials. 

This graph shows two important things: 1) even conditional on employment size, ARS-prompting induces 

a higher QBS completion rate (statistically significant difference) and 2) the completion rates are similar 

for the two QBS-Prompted groups (not statistically different), suggesting that differences in response 

rates is not necessarily due to differences in sample compositions between in-scope and out-of-scope 

ARS respondents but rather an artifact of the actual letters. One additional point is that there is no 

statistical difference for establishments with AME=0 between the groups. This makes sense; one 

common reason for nonresponse may be that the establishment has gone out of business, which 

explains the AME of zero. 

 

6 Units without an industry, those marked as ‘unclassified’ are in-scope for the ARS regardless of establishment 

size. That is why there are establishments with AME=0, 1, 2 and 3 that are ‘in-scope for ARS’ 
Graph B shows the sample size for each of the employment-sample group cells. Since small employers 

were a focus of the out-of-scope, QBS-Prompted sample, it is not surprising to see a large number of 

establishments that fall into this category. However, it is notable that such a large proportion of these 

respondents have an AME of zero, reflective of the universe of establishments as no restrictions on 

sampling were specified. Since these establishments have a higher likelihood of being out-of-business, it 

may make sense in future iterations of the QBS to remove this group from the sampling, particularly 

when considering they have such a lower response rate. 

If we remove the establishments with an AME equal to zero in March 2018, the QBS completion rate for 

the entire random sample and the QBS-Prompted sample is 28% and 20%, respectively (compared to 

25% and 18%, respectively when the AME equal to zero units are included).