Working with the Census API
By: Nick Doiron
Tags: Open Government, Open Data, Census Data, Census

Working with the Census API

I’m a software developer at Tufts University, and my main project is a district-drawing website Districtr.org, which covers all forms of school, city, county, and legislative districts. We check that districts have equal population (and allow users to evaluate their plans) through careful use of Census data.

You may be aware of the current Census (please fill it out!) but if you’re a data scientist or political activist, you should know where to find previous Census records for your present-day projects.

Community projects benefit from Census data

This March, my colleagues started using our Census work to map where demographics especially at-risk for coronavirus live — specifically the elderly, group housing, and households that don’t speak English. Knowing the concentration of these populations, versus hospital capacity, would guide our research and advice to policy makers.

Researchers are also comparing other datasets on policing — 911 calls, traffic stops, arrests, police violence, sentencing— to local populations via Census data. For example, we can look at imprisonment at a national level vs. race and ethnicity data:

In June, John Keefe made a series of how-to posts about collecting race and ethnicity data from the 2010 Census to police precincts of multiple cities:

This was an interesting project, but I wanted to make two points:

2010 numbers can be outdated, especially in cities. It might be difficult to identify which neighborhoods are gentrifying, one of the main focuses of study in police bias (article 1, article 2). When I met with a police bias program earlier this year, they didn’t compare their records to Census blocks (a large portion of their arrests, traffic stops, and hiring involved people from outside their city).

If you want to be a Census data scientist, you should be comfortable both with the decennial Census, and the American Community Survey (ACS). As of this writing, one of our best and most recent sources was the 2014–2018 5-year ACS.

Census geo data

Getting the appropriate level of detail

Why don’t we use the more recent ACS for everything?

The decennial Census publishes population counts for every ‘Census block’. The ACS has more recent and more detailed statistics, but we have to zoom out one level to the ‘block group’. For other data points (such as language and national ancestry) we must zoom out again to the ‘tract’ level.

The hierarchy looks like this:

The size of each ‘Census block’ or tract is not consistent when you look at urban or rural or coastal areas. Some blocks cover unpopulated parks, rivers, and airports; some are city blocks or prisons.

Explore blocks and tract maps on NYC’s popfactfinder.planning.nyc.gov

Comparing Census blocks big (in Central Park) and small (Upper East Side, islands)

Download Census shapefiles

You can view map data in GIS software. The leading free/open source application is called QGIS: qgis.org

You can download 2010 blocks or 2018 block groups as shapefiles from

census.gov/cgi-bin/geo/shapefiles/index.php

Census population data

Now that you have the shapes, let’s collect several columns of data from the Census API.

If you can’t install Python and pip, or access the Census website because of your computer or internet settings, you can likely get a free notebook on Google CoLab, or NextJournal.

Get a Census API key

It’s free! https://api.census.gov/data/key_signup.html

Download a helper library?

If you’d like to have some of the API code worked out in advance, try this Python package from DataMade:

There’s probably an R package, too. Leave a comment with your favorite.

Writing a script

This is how I find the columns that I want and download them from the Census API. Your approach can vary from this.

Look up variable names from the Census spec, here for 2010 decennial and here for the recent 5-year ACS. This reference is confusing, so you might want to start with ACS table names on SocialExplorer. Install Python, pip, and any dependencies ( pip install requests ) Edit this Python script with your API key, state, county (or counties), and columns.

If you were using the 2010 Census, what would you change?

column names

API endpoint (http://api.census.gov/data/2010/dec/sf1/)

for=block:*

end of blockid = line should reference block, not block group

Combining population tables and shapefiles

There are two major approaches that I know of:

GeoPandas (the more common one). If you are familiar with Pandas, this will make sense to you. Load the shapefile as a GeoDataFrame, then assign each column based on the data that you downloaded.

GDAL (my usual method). You need a native libgdal installed, then pip install gdal .

Then this code excerpt will work (or you can append it to the end of your downloader script)

Estimating data in one layer from another

Blocks fit inside of block groups which fit inside of tracts, so we can sum them up and calculate populations precisely. If we draw other shapes on the map, such as a walking distance around subway stations, the fit is no longer perfect; we can only estimate the population inside.