Web Scraping in Support of the Census 

Bureau’s Public Sector Programs

Federal Committee on Statistical Methodology Research and Policy Conference

Washington, DC

October 27, 2022

Hector Ferronato1,2 and Brian Dumbacher1

1U.S. Census Bureau

2Reveal Global Consulting

1

Disclaimer: Any opinions and conclusions expressed herein are those of the authors and do not reflect the views of 

the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential 

information and has approved the disclosure avoidance practices applied (Approval ID: CBDRB-FY22-ESMD001-013).
Outline

• Introduction

• Scraping Assisted by Learning (SABLE)

• Quarterly Summary of State and Local Tax Revenue (QTAX)

• State Government Finances (STATEFIN)

• Conclusions and Future Work

2
Introduction

• The Economic Directorate of the U.S. Census Bureau conducts 

various surveys of state and local governments

• Respondent data can often be found on government websites in the 

form of PDF and EXCEL reports.

• Automated data collection from these websites can reduce burden 

for both respondents and Census Bureau analysts.

3
SABLE

4

Supplementary 

files Folders

NLTK

Firewall

External

public

website

Tabula
QTAX

• QTAX collects quarterly tax revenue data from state and local 

governments. 

• Much of this tax revenue data is publicly available on government 

websites on monthly basis.

• Lack of standardization of URLs and report formats

• Using SABLE to automate the data collection

5
QTAX Downloading Option 1: URL Patterns

• The analysts determine what PDFs to download for each state, and 

the researchers then propose candidate URLs that are functions of 

the user-specified year and month. These functions are referred to as 

URL patterns. 

• https://aoa.vermont.gov/sites/aoa/files/revenueeconomy/RevenueRpts/DECEMBER%20-

%20FY20%20Revenue%20Press%20Release%201-

24-20.pdf

6
QTAX Downloading Option 2: Archive Page

• Identify the main archive 

page that the state uses to 

publish the tax reports. In 

this case the Python 

program visits the archive 

and searches for the report 

URL.

• Download with WGET.

7
QTAX Scraping • Analysts tell the researchers 

what taxes to scrape for each 

state. The researchers then 

develop a template that takes 

the TXT version of the PDF as 

input and scrapes the following 

data: • Tax names • Tax values • Time period (e.g., monthly, or 

fiscal year to date)

• Units (e.g., dollars or thousands 

of dollars)

8
QTAX Scraping: Regex Examples

9

m_col = re.search(r"(actual|budget)\s+(budget|actual)", line)

if m_col:

if m_col.group(1) == "budget" and m_col.group(2) == "actual":

col = 2

elif m_col.group(1) == "actual" and m_col.group(2) == "budget":

col = 1

m = re.search(r"estate\s+tax\s+\$?\s*([\d,.()]+)\s+\$?\s*([\d,.()]+)", 

line)
QTAX Results

10

Month Downloading Scraping

Success Caution Problem Success Caution Problem

Jan 2022 23 2 1 22 1 0

Feb 2022 23 2 1 23 0 0

Mar 2022 23 2 1 23 0 0

Apr 2022 23 2 1 23 0 0

May 2022 24 2 0 24 0 0

Jun 2022 24 2 0 24 0 0

Total 140 12 4 139 1 0

Efficacy % 90% 8% 3% 89% 1% 0%

139 scraped out of 140 downloaded = 99.3% scraping efficacy

Classification of results from 26 states in terms of

success in downloading and success in scraping
STATEFIN

• STATEFIN collects a wealth of data on state finances:

• revenue by source, expenditures by function, debt by term, and assets by 

purpose

• Some states provide this data in PDFs published on their websites on 

a yearly basis. 

• The Census Bureau is prototyping using SABLE for web scraping two 

California state finance reports: 

• Legislative, Judicial, and Executive section of the Governor's Budget (report 

0010) 

• Business, Consumer Services, and Housing Agency (report 1000)

11
STATEFIN Downloading

• Each of the State 

Agencies in the 

Budget Detail 

section publishes 

a PDF report of 

the entire 

agency’s budget. 

• Currently, 

analysts manually 

download these 

reports. 

12
STATEFIN PDF-to-EXCEL Conversion

13
STATEFIN Labeling

14

122 labeled objects of interest
STATEFIN Scraping

15

122 labeled objects scraped
STATEFIN Results

• 122 out of 122 (100%) labeled objects were scraped from report 1000

• 193 out of 306 (63%) labeled objects were scraped for report 0010

• These results indicate the scraping program needs to be more flexible 

in order to find the remaining objects and collect their values. 

• Overall, these are positive results for automatic scraping the dollar 

values for each combination of department, program, and object.

16
Conclusions

• The QTAX project involves scraping a relatively small number of tax 

revenue items from 26 different state government websites and reports.

• The STATEFIN prototype involves scraping many financial items from a 

limited number of documents on the California state government website. 

• Results so far for both QTAX and STATEFIN are positive. The downloading 

and scraping methodologies can handle inconsistencies in URLs and 

document layouts. 

• Refinement is needed, but progress has been made towards automating a 

substantial portion of what used to be manual work.

17
Future Work

• Based on the encouraging results obtained so far for QTAX and STATEFIN, 

there is also potential in applying similar methodology to other public 

sector surveys. For example, other surveys of state and local governments 

collect data on public retirement systems and employment and payroll. 

• Lastly, to be transparent with respondents and the greater public, there are 

plans to update the publicly available SABLE GitHub repository with the 

most recent Python code. This repository is located at 

https://github.com/uscensusbureau/SABLE and received its last major 

update after the initial PDF methods for QTAX were developed.

18
Contact Information

• Hector.R.Ferronato@census.gov

• Brian.Dumbacher@census.gov

19