Automated Collection of Publicly Available Data From the Internet

Mike Castro, Sumit Khaneja, Anup Mathur

U.S. Census Bureau

Disclaimer – Any opinions expressed during this presentation are those of the authors and do not necessarily reflect the position of the Census Bureau

1

1

The Use Case

Asking respondents to fill out surveys or answer questions for the Census Bureau is expensive, burdensome, and inefficient.

Stat Agencies are always looking for new sources of data.

Administrative and third-party data.

What about information people, businesses, and governments post about themselves online?

Can we use that information to:

measure the quality of their responses or fill in gaps in our data?

build agile and targeted sampling frames to increase response, and make our surveys less costly to administer?

2

2

The Technology

Automated Web Scraping 

The use of software or code to collect or “mine” specified information from one or more websites. Web scraping is also called Web data extraction, screen scraping, or Web harvesting. 

Web Crawling 

Crawling uses a program to automate Web indexing, a process of mapping the available resources on a website. Web crawlers access one page at a time through a website, identifying any available links on that page and following them until all pages have been indexed. Web crawlers also help in validating HTML code and hyperlinks. Web crawlers are also known as a Web spiders, automatic indexers, bots, or simply crawlers.

Application Programming Interface (API) 

APIs are a tool offered by some data providers that produce structured data outputs in response to queries inputted by the end user. APIs may be open to the general public or may require the use of an authenticator, such as a username/password combination or an API Key, to access the data.

None of these are new – Private sector data aggregators have been using them for years.

3

3

The Challenge

Strike a Balance.

Allow flexibility for research and experimentation.

Very new to the space.

Don’t know what we don’t know.

Use these technologies ethically and responsibly.

Preserve the public trust.

“Data Aggregator” can be a dirty word.

“The Government is Spying on Me.”

4

4

The Context

Some International Stat Agencies have Policies.

UK Office for National Statistics 

https://www.ons.gov.uk/aboutus/transparencyandgovernance/datastrategy/datapolicies/webscrapingpolicy

Statistics Canada

https://www.statcan.gc.ca/en/our-data/where/web-scraping

Not much guidance for U.S. Federal agencies.

GSA Future Focus Blog: https://www.gsa.gov/blog/2021/07/07/gsa-future-focus-web-scraping (not official guidance)

Know some stat agencies are using these methodologies, but no specific guidance for FSS

5

5

Our Solution

September 2020 - Census Bureau’s Data Stewardship Executive Policy Committee (DSEP) 

DS026 - Automated Collection of Data from the Internet

https://www2.census.gov/foia/ds_policies/ds026.pdf

Policy

Governance Structure

6

6

The Policy

Automated collection of data from the Internet is generally permissible if all the following are true:

The data being collected are public information or the Census Bureau has received explicit informed consent from the respondent and/or data provider to collect them; 

The data collection is consistent with the Census Bureau’s mission and done in a way that is legal, ethical, transparent, and does not present a risk to the reputation of the Census Bureau;

Collecting the data does not constitute a disclosure risk for Title 5 data (Sensitive PII), Title 13 data (Confidential Census Data), or Title 26-protected Federal Tax Information (FTI);

The systems and applications used for collection and analysis have a Census Bureau Authority to Operate (ATO) that supports storing Title 13 data. 

7

7

The Governance – AIDCRB

Automated Internet Data Collection Review Board (AIDCRB) 

Provide guidance as well as review and approve automated Internet-based collection activities. 

All current and future research and production projects that use in-scope methods must be reviewed and approved by the AIDCRB before any data collection can occur. 

AIDCRB examines the data collection method to ensure compliance with the policy.

Panel Review and additional SME input (ex. legal) as needed.

Does NOT review “suitability for use.”

Representation:

Technical Areas

Policy/Privacy

Program Areas incl. Decennial, Demo, Econ, R&M

Communications

Disclosure Review Board

8

8

AIDCRB – Additional Functions

AIDCRB also serves as a service and collaboration platform for Census users providing:

Browsable repository of information regarding the projects that have been reviewed, 

Public facing materials re: the Census Bureau’s activities,

Web scraping tools and standards - including User Agent String, Salting and Crawling techniques (more later)

A complete web scraping platform, to assist web scraping projects. 

9

9

Project Review Criteria

10

4 Major Categories:

Privacy Rights of the Respondent 

Rights of the Provider

Protection of Confidential Information

Policy and Sensitivity Considerations

10

**Anup Mathur  (CENSUS/ADRM FED) (AM(F) -**slides 9-14 must be distilled to 2 slides

11

Is the information we’re collecting publicly available?

Publicly Available Information - Data that the Census Bureau can collect and use to support its mission without purchase or entering into of an agreement, and without violation of any applicable access rules, terms of use, or intellectual property rights.

Were the target data made public in a way that is unintentional, illegal, unethical, or contrary to the wishes of the respondent? (don’t scrape a data breach!)

If the project involves the collection of non-public information (involves logging in), is the informed consent obtained by the data provider appropriate, obtained from the correct respondent or representative of the respondent, and does it sufficiently protect the agency from liability?

Do the proposed methods sufficiently limit or ideally eliminate the collection of data that is not essential to the research or stated objectives of the project? 

Privacy Rights of the Respondent 

11

Privacy Rights of the Respondent

Is there a plan in place to account for and dispose of non-relevant data that is consistent with the sensitivity of the data collected (e.g. Personally Identifiable Information or Business Identifiable Information) and applicable records schedules?

Are we being sufficiently transparent with the public about the fact we’re scraping information about or from them?

Public facing privacy compliance documentation.

Census.gov/scraping – Currently under development.

Standardized user agent string to be left on all sites scraped Includes link to census.gov/scraping.

12

12

Rights of the Provider

Will the project respect any restrictions imposed by the robot.txt file of a targeted website provided it is current or has been updated within 5 years from the date that the automated collection will take place?

Will the project take reasonable steps to locate and respect Terms of Use or Conditions of target websites? 

Not standardized, often framed in legal terms

Generally, in place to protect the providers commercial interests, but no statute exempts us from respecting them.

Frequently conflict with robots.txt

Risk-based decision based on the scope and nature of the collection

Manual review vs. machine learning/analysis

Example

API terms of use as well, or terms for account creation 

Sometimes legal needs to review terms, and sometimes you may need to reach out to the provider for permission.	

13

13

Rights of the Provider

Will the project overburden the providers resources?

Scrape off-peak hours, limit queries, etc.

Don’t attempt to subvert any limiters i.e. Captcha, “I’m not a robot,” IP Blocking.

14

14

Disclosure Considerations

Collection Authority Confers Confidentiality Protections.

Does not matter if the data are publicly available – Matters what the data are and why we are collecting them.

Data collected about individuals or establishments for a statistical purpose (our only legitimate reason for collecting such data) are subject to confidentiality and use restrictions at the point of collection.

Does not require comingling and no distinction for research v. production.

Collection must be a one-way street

Data ingest directly into a secure environment.

Ensure targeting doesn’t reveal confidential data.

Protecting your sample and respondents.

Seeding/Salting.

Injecting “noise” into your queries and then discarding afterward.

Disclosure Review Board (DRB) Review of Protocols.

15

15

Other Policy Considerations

Is this project going to make the agency or Federal Government look bad?

Do we need to communicate with other stakeholders or oversight before we engage in the work?

16

16

AIDCRB – Example Case Studies

Reimbursable Projects

Teacher Attrition Survey – DOE

Target – School Websites

Does this teacher still show up on this school’s staff roster?

Vehicle Inventory and Use Survey – BTS

Target - NHTSA Vehicle Product Information Catalogue (vPIC)

API to decode VINs of sample vehicles

Internal Projects

GQs

Are GQs opened or closed? How many people are they supposed to house?

Project Metadata

Target – Academic publication repositories

Outputs of FSRDC Projects, such as papers and other publications 	

17

17

Current Status & Next Steps

Continue to review nascent projects and improve our repository of tools and resources.

Learn what we can from the projects after they’re done.

Monitor the landscape.

Explore ways to begin outreach to promote the legitimate activities conducted by the Census Bureau and other Federal Statistical Agencies, so that data providers can see how these passive collections can benefit them through reduced burden, and the American public through better, more cost-effective statistics.

18

18

Thank You!

Mike Castro - Michael.Castro@census.gov

AIDCRB Chair – Anup Mathur – Anup.Mathur@census.gov

AIDCRB Co-Chair – Sumit Khaneja - sumit.khaneja@census.gov

Policy – census.gov/scraping

19

19

# image2.png

# image3.svg

# image1.png