Using PDF Extraction and Web Scraping Tools to Collect Government Health Insurance Plan Information

Virginia Gwengi, U.S. Census Bureau

FCSM

November 4, 2021

1

Disclaimer: Any views expressed are those of the author(s) and not necessarily those of the U.S Census Bureau. All results have been reviewed and no confidential information has been disclosed. 

Overview

Medical Expenditure Panel Survey-Insurance Component (MEPS-IC)

Motivation

Summary of Benefits and Coverage Form (SBC)

Methods

Web scraping

PDF extraction 

Challenges 

Next steps

2

Medical Expenditure Panel Survey-Insurance Component (MEPS-IC)

Voluntary survey that collects information on employer-provided health insurance, which is different from MEPS-Household Component

Reimbursable survey funded by Agency for Healthcare Research and Quality (AHRQ) with data collection conducted by Census

Conducted every year

Universe: private companies and government entities

Two-part survey

Establishment level questions: number of hours worked to be eligible for health insurance

Plan level questions: premiums, copays, deductibles

3

Motivation

4

We aim to reduce respondent burden, improve data quality, increase timely responses and reduce survey production costs by applying data science methods on existing administrative records (SBC forms)

Summary of Benefits and Coverage Form

5

Mandated as part of Affordable Care Act, the SBC form is the "nutrition facts" of health insurance coverage

Employers with 50+ employees that offer health insurance must provide SBC forms to employees

Many SBC forms can also be found online

Contains information collected in the Medical Expenditure Panel Survey-Insurance Component (MEPS-IC)

Methods

6

Government Master Address File (GMAF)

Get unit name from 

GMAF

Use search API to get URL to start crawling from

OR

To get PDF 

Web

Crawling 

Tool 

PDF Extraction Tool

Retrieve  sample from the frame

Google search API

Scrapy: Python web crawling framework 

Camelot: Python library to extract tables from text-based PDFs

Web scraping tool (1)

7

Motivation

To collect SBC forms:

Many SBC forms can be found online 

Analysts manually search for certainty government insurance forms through web searches

Respondents can upload forms into data collection tool

Manually searching for these forms takes significant time

Policy: Among the first projects to get approval from Census to perform web scraping

Method: Scrapy web crawling framework looks for every pdf which are processed to find the SBC forms

SBC Forms

Web scraping tool (2)

8

Benefits

Save analysts' time in manually searching for forms

Allows for early data collection 

Potentially save respondents' time

Don't need to ask government units to upload forms

Improve data quality: SBC forms may be used to impute missing information or in the edits process, to assist with estimates

PDF extraction tool (1)

Motivation

Analysts manually extract information from SBC forms and hand enter into excel spreadsheets

Data from SBC forms is not exhausted

Method

Step 1: 

Use Camelot python library, which uses visual design to extract table cells from text-based pdfs. These are saved to a json file 

Step 2: 

Reads in the json file and uses regular expressions to identify response items from the table contents

9

PDF extraction tool (2)

10

Benefits

Tool identifies and extracts SBC form in 15 seconds compared to 2-4 minutes in the manual process

Extraction tool can provide additional plan-level information without additional cost 

Identifies PDF SBC forms from the web

Results

Web scraping

For survey year 2021, 1197 SBC forms have been collected which accounts for 22% (205) of the 893 certainty government units

On average, 5 SBC forms per government unit

PDF Extraction

On average, Camelot parses each response item at least 88% of the time

11

Challenges 

12

PDF Extraction Tool

Not all units will have SBC forms (those with fewer than 50 employees, noncompliant employers)

Can only extract SBC forms saved as readable PDFs (not scanned or in Word or plan brochures)

Some forms are trickier than others to extract: inconsistent number of columns, varying patterns of text

Web scraping Tool

Cannot scrape SBC forms on websites that require login information to retrieve forms

Not all units may have web presence

Next steps

13

Employ named entity recognition (NER) to increase the accuracy of the extraction tool

Validate the web scraping tool results against manually searched SBC forms 

Explore how web scraping and pdf extraction tool results can be used in edits and imputation processes

Apply the web scraping and pdf extraction tools to other types of respondents (Non-Certainty government, Private)

14

Thank you

Virginia Gwengi 

martha.v.gwengi@census.gov

# image12.jpeg

# image13.jpeg

# image14.png

# image15.png

# image11.png