Design of a Pilot Test of Businesses

to Study the Effects of Omitting Select Survey Processes

on Survey Response and Data Quality

David Kashihara, Agency for Healthcare Research and Quality

Matthew Thompson, U.S. Census Bureau

2023 Federal Committee on Statistical Methodology (FCSM)

Research and Policy Conference

College Park, Maryland
Disclaimers

• The views expressed in this presentation are those of the authors 

and no official endorsement by the Department of Health and 

Human Services, the Agency for Healthcare Research and Quality, 

or the U.S. Census Bureau is intended or should be inferred.

• The Census Bureau has reviewed this data product to ensure 

appropriate access, use, and disclosure avoidance protection of the 

confidential source data [Project No. P-6000376, Disclosure Review 

Board (DRB) approval numbers: CBDRB-FY23-ESMD002-007, 

CBDRB-FY23-ESMD002-012].

2
Presentation Outline

• Introduction to MEPS

• Data collection

• Survey response

• 2021 MEPS-IC pilot test

► Design & Methods

► Results

• Summary & Discussion

3
Medical Expenditure Panel Survey 

(MEPS)

• Household Component (HC)

► Annual survey of households

► Administered by Westat

• Medical Provider Component (MPC)

► Annual sample of providers of services to HC respondents

− Physicians

− Hospitals

− Home health agencies

− Pharmacies

► Administered by RTI International

• Insurance Component (IC)

► Annual survey of

− U.S. business establishments

− U.S. State & local governments

► Administered by the U.S. Census Bureau

4
Overview of the MEPS-IC

• Health Insurance Cost Study (survey name on forms)

• Annual survey

► Over 42,000 business establishments (individual locations)

► Over 3,200 State and local governments

− Not covered in this presentation

• National & State-level estimates

► % of employers offering health insurance

► # of enrolled employees

► Health insurance premiums, copays & deductibles

• Data Collection Modes

► Mail

► Web

► Telephone

5
Pre-COVID MEPS-IC Data Collection

• Research

► Update employer contact information & find best person to contact

• Telephone Prescreener

► Determine if offer insurance. If no - end of survey

• Mailed Survey Forms

► Sent if offer insurance or nonresponse to pre-screener

• Internet (Web)

► Option given in introductory letter of the mailed survey forms packet and on 

the first page of the mailed survey form

• Personal Visit

► Very few – only for the largest non-respondents

• Telephone Follow-up (TFU)

► For establishments with no response via mail or internet

6
2021 MEPS-IC

Data Collection Processes

• Research – update employer contact information

• Prescreener – brief phone call to determine if offer insurance

• Initial Email – email invitation to respond to the survey via web

• Initial Mail – letter invitation to respond to the survey via web

• Follow-Up Procedures

► Reminder Email

► 1st Mail Follow-Up

► 2nd Mail Follow-Up – full questionnaire package mailed

► 1st Email Follow-Up

► Telephone Follow-Up (TFU)

► 2nd Email Follow-Up

► 3rd Email Follow-Up

7
MEPS-IC

Response Mode Percentages

8

0

10

20

30

40

50

60

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Response Percentage

Web Mail TFU
MEPS-IC Private Sector

Overall Response Rate (AAPOR RR2)

9

80.7 82.6 82.7 81.7 79.9

74.6

69.8

66.5 67.6 65.8 67.8

59.2

56.1 56.9

0

10

20

30

40

50

60

70

80

90

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021

Response Rate (%)

Year
2021 Pilot Test - Background

• 2020 – Covid health pandemic forced several 

labor-intensive MEPS-IC survey operations to 

shut down as data collection was about to begin

► Prescreener

► Research

► Survey forms mailout

► Survey forms reception

• Covid mitigation brainstorming

► Make the survey web only?

► Effects of removing labor-intensive processes

− Generally the most costly operations

10
Definitions of Terms

• Establishment – a specific business location

• Firm (Company) – has one or more establishments

► Single Unit – a company with only one business location

► Multiunit – a company with more than one business location

► Single Multiunit – only one establishment of a multiunit 

company sampled into the survey

11
2021 Pilot Test - Design

• Pilot test establishments sampled independently 

from the regular survey (“production”) cases

► Establishments can’t be in both pilot test & production (regular 

survey)

► No overlap of pilot test & production establishments within firms

− Decision made to only sample single unit & single multiunit 

establishments into the pilot test → sample of only smaller firms

− Discarded establishments in multiunits that were selected in the 

production survey to prevent possible TFU overlap → weights 

redistributed within the stratum

• Pilot test cases processed and tabulated separately 

from the production cases

► Data collection methods different

► Pilot test cases should not be used to impute production cases

► Separate tabulations allows for comparison of estimates

12
2021 Pilot Test - Methods

• 5,000 small businesses drawn independently from the 

same frame as the “production” survey

► Establishments in firms with 1-91 employees (stratum bound)

• Data collected concurrently with production cases

► No Research

► No Prescreener

► No Telephone Follow-Up (TFU)

• Data processed separately from production cases

► Separate edit & imputation processes

► Modified nonresponse weight adjustment procedure

► Pilot cases won’t contribute to survey estimates

• Data tabulated separately from production cases

13
2021 Pilot Test

Data Collection Processes

• Research – update employer contact information

• Prescreener – brief phone call to determine if offer insurance

• Initial Email – email invitation to respond to the survey via web

• Initial Mail – letter invitation to respond to the survey via web

• Follow-Up Procedures

► Reminder Email

► 1st Mail Follow-Up

► 2nd Mail Follow-Up – full questionnaire package mailed

► 1st Email Follow-Up

► Telephone Follow-Up (TFU)

► 2nd Email Follow-Up

► 3rd Email Follow-Up

14
2021 Response Rates

Sample Size

(in scope)

Response Rate

(AAPOR RR2)

Overall MEPS-IC 39,935 56.9 %

1 - 91 MEPS-IC Sample 19,108 58.7 %

1 – 91 Pilot Test Sample 4,729 42.9 %

15
2021 Pilot Test – Results

Estimates

• Compared 1 – 91 firm size MEPS-IC establishment to 

Pilot Test estimates at U.S. level

• Establishment-based estimates including:

► % Establishments offering health insurance

► % Women employees

► Total # part-time employees in establishments offering health 

insurance

• Health insurance plan estimates including:

► % Employees enrolled in single & family coverage plans

► Average single & family coverage premiums

► Average individual & family deductibles

16
2021 Pilot Test – Results

Establishment Estimates

Establishment-Level Estimate Pilot Est Pilot SE IC Est IC SE Sig Diff

% Establishments that offer health insurance 32.7 2.10 34.0 0.72 NS

% Women employees 40.4 1.27 44.3 0.46 0.01

% Employees 50+ years of age 25.9 1.03 26.8 0.35 NS

% Employees low wage (< $13.50/hour) 19.8 1.16 19.5 0.43 NS

% Employees offered paid vacation leave 76.8 1.62 78.7 0.55 NS

% Employees offered paid sick leave 65.0 1.69 68.9 0.63 0.05

Total employees in establishments 40,101,016 796,923.1 41,447,071 382,860.9 NS

% Employees in establishments that offer health insurance 57.1 1.80 57.6 0.67 NS

% Employees eligible for health ins in estabs that offer 77.8 1.60 79.4 0.62 NS

% Employees eligible for health ins that are enrolled 69.3 1.57 66.7 0.62 NS

Total part time employees in establishments 10,859,216 547,430.1 11,508,567 221,764.6 NS

% Enrollees in family coverage plans 21.9 1.08 21.5 0.56 NS

% Enrollees in grandfathered plans 13.6 1.81 11.4 0.72 NS

17
2021 Pilot Test – Results

Plan-Level Estimates

Plan-Level Estimate Pilot Est Pilot SE IC Est IC SE Sig Diff

Exclusive providers plan 30.0 2.66 29.0 1.01 NS

Any providers plan 13.4 1.54 12.9 0.71 NS

Mixture of preferred and any providers plan 56.6 2.74 58.1 1.10 NS

Gatekeeper or physician required for specialist referral 32.4 2.55 35.6 1.06 NS

Plan purchased from insurance underwriter 87.3 1.58 83.8 0.86 0.10

Plan self-insured 12.7 1.58 16.2 0.86 0.10

% Employees enrolled in single coverage 61.0 1.63 63.4 0.66 NS

Average single coverage premium 7,489 170.6 7,350 71.8 NS

Average family coverage premium 20,063 544.6 20,442 260.9 NS

Average individual deductible 2,545 72.2 2,444 38.1 NS

Average family deductible 5,175 255.6 4,895 123.6 NS

% Enrollees in plans with a deductible 85.3 1.79 86.8 0.76 NS

18
2021 Pilot Test – Data Quality

Item Nonresponse & Edit Failure Rates

19

Estimate Pilot Test

Item NR

Pilot Test

Edit Failures

Survey

Item NR

Survey

Edit Failures

% Teleworking employees 2.6 1.1 3.7 1.5

% Women employees 3.2 0.0 4.8 0.1

# Active employees enrolled 2.2 0.0 4.4 0.0

Type of provider 5.9 0.0 5.8 0.0

Total single coverage premium 15.3 1.6 19.2 3.3

Total family coverage premium 19.4 1.8 25.5 3.3

Average individual deductible 7.5 11.0 11.8 10.6

Average family deductible 12.6 6.5 17.8 7.3
2021 Pilot Test – Data Quality

Item Nonresponse & Edit Failure Rates

20

Estimate Pilot Test

Item NR

Pilot Test

Edit Failures

Survey

Item NR

Survey

Edit Failures

% Teleworking employees 2.6 1.1 3.7 1.5

% Women employees 3.2 0.0 4.8 0.1

# Active employees enrolled 2.2 0.0 4.4 0.0

Type of provider 5.9 0.0 5.8 0.0

Total single coverage premium 15.3 1.6 19.2 3.3

Total family coverage premium 19.4 1.8 25.5 3.3

Average individual deductible 7.5 11.0 11.8 10.6

Average family deductible 12.6 6.5 17.8 7.3
2021 Pilot Test – Data Quality

Item Nonresponse & Edit Failure Rates

21

Estimate Pilot Test

Item NR

Pilot Test

Edit Failures

Survey

Item NR

Survey

Edit Failures

% Teleworking employees 2.6 1.1 3.7 1.5

% Women employees 3.2 0.0 4.8 0.1

# Active employees enrolled 2.2 0.0 4.4 0.0

Type of provider 5.9 0.0 5.8 0.0

Total single coverage premium 15.3 1.6 19.2 3.3

Total family coverage premium 19.4 1.8 25.5 3.3

Average individual deductible 7.5 11.0 11.8 10.6

Average family deductible 12.6 6.5 17.8 7.3
Summary

• Comparing Pilot Test results to MEPS-IC survey 

estimates (establishments in firms with 1 – 91 

employees) at the U.S. level:

► Estimates: Pilot Test similar to MEPS-IC

► Data Quality: Pilot Test generally better than MEPS-IC

− Item nonresponse rates

− Edit failure rates

• Unit response rate lower for Pilot Test

► 42.9% Pilot Test

► 58.7% MEPS-IC (1 – 91 employees)

22
Discussion

• Pilot Test data quality

► Pilot test mode primarily web

► Web generally better quality than mail or telephone response

− Aided by automated edits in the web instrument (“soft edits”)

• Mode effects (based on a selection of variables used in this 

analysis)

► Pilot test item NR rate from mail increases ~ 40% over web only

► IC (1 - 91 employees) item NR rate from mail & phone increases ~ 60% over 

web only

► Pilot test edit failure rate from mail increases ~ 5% over web only

► IC (1 – 91 employees) edit failure rate from mail & phone increases ~ 15% 

over web only

• Significant cost savings realized from omitting labor-intensive 

processes: Research, Prescreener & Telephone Follow-Up

23
Discussion – Data Collection Methods

• Pilot test vs MEPS-IC survey

► No Research

► No Prescreener

► No Telephone Follow-Up (TFU)

• Best method? Need to consider data quality & cost vs 

unit response

► Pilot test estimates differed from MEPS-IC survey:

− At national level (not so much)

− By firm size (generally a bit more)

− By industry (generally more)

► Bias considerations due to unit & item nonresponse

• Important caveat: pilot test of only smaller businesses

► Larger businesses go through TFU at a much higher rate

► Thus, any conclusions can only be applied to smaller businesses

24
Acknowledgements

• We would like to acknowledge the following people for 

their contributions to this project:

► Brandon (Colby) Flanders, U.S. Census Bureau

► Susanna Winder, U.S. Census Bureau

25
Questions?

David Kashihara

Agency for Healthcare Research and Quality

david.kashihara@ahrq.hhs.gov

26