Four Simple Steps to Improve U.S. Census Data CSVs
By: Chris Whong
Tags: American Community Survey, Demographics, Data S, Census Data, Open Data

Publishing Standalone Census Data Tables in Qri

We recently published about 160 discrete tables of data from the latest cut of the American Community Survey (ACS) on qri.cloud, in an attempt to make them more easily findable, pullable, and usable. This post outlines some of the challenges of working with exported U.S. Census data “as-is”, and some improvements we sought to make by putting them into our dataset version control system. (Think git for datasets… that’s qri!)

Qri is a database, peer-to-peer network, and version control system build specifically for crafting, building, sharing, and pulling datasets.

We chose to publish state-level data for the most recent 1-year and 5-year estimates provided by the American Community Survey. More specifically, we pulled and published each of the 80 or so subject tables that the census bureau defines. These subjects include things like Types of Computers and Internet Subscriptions, Mean Income, Marital Status, Fertility, School Enrollment, etc. There’s a wealth of information contained within for just about any geographic slice of the United States.

Getting Census Data

Census data is reputed to be difficult to get at, and for years, American Factfinder, a web-based querying interface, was the way in. Factfinder was recently sunset and replaced by data.census.gov to mixed reviews. Regardless of the interface, the most difficult thing about getting at census data is that it can be sliced and diced in so many ways, you need to be really specific about what you want before you begin searching for it. Trying to figure out what you can even ask of the data has a steep learning curve for the uninitiated.

As a simple example, say you are interested in seeing the estimated Median Income in the past 12 months in the U.S.. On data.census.gov, you may end up looking at Subject Table S1903. Bear in mind that what’s in the table is completely dependent on which geographic summary level and data product you choose. The number you’re probably looking for is at the intersection of the “Households” row and the “Estimate” column, $61,937 (plus or minus $94), but it’s presented with a lot of supporting data. The subject tables will give you the big picture number for the geography level you specify, but they also slice things up by other criteria, so we can see the same metric by race, sex, age, number of people in family, nonfamily households, etc. In addition to the additional “slices”, they also present the estimated number of households and the percent distribution, providing more context. There’s a lot of data on the page, but once you get the feel for how things are laid out, it gets easier to find what you’re looking for.

The web UI for subject table S1903 (Median Household Income) on data.census.gov. On the web, hierarchical column headers are used to make the table more readable. These don’t convert well when exporting a CSV, however.

But this two-dimensional table is for a single geography level (The entire United States). What happens when we want to view more than one geography? Using data.census.gov, we could specify that we want to see the same info for each of the 50 States. In this case, the rows remain the same, but the 6 columns of data are repeated for each of the states, making for a wide table.

Four Problems with Census CSVs

The real trouble comes when you try to download the data as a CSV. The familiar layout we see in the web tables is replaced by something more machine-readable, which creates a single row for each geography level. That means all of the information in the two-dimensional table above must be flattened into a single row… a really wide CSV with 6 columns for each row in the table shown above. On top of that, there are a few issues with the CSV as-downloaded:

CSVs downloaded from data.census.gov include a double header row, ambiguous column names, ambiguous column order, and packaged with other files in a zip.

They are…

Multiple Files

When you download, you get a zip file with a data CSV (with a very detailed filename like ACSST1Y2018.S1903_data_with_overlays_2020–05–07T225949.csv), a metadata CSV, and a TXT file (basically a readme) about the data. It’s not necessarily bad to combine the data with supporting files, but it’s one more step to unzip and familiarize yourself with the data.

2. Ambiguous Column Names

The header row in the downloaded CSV is populated with column names like S1903_C03_026M, S1903_C01_027E, S1903_C01_027M. Each has a specific meaning, but won’t make any sense unless you have the answer key. Many users manually replace these with custom column names for only the columns they need at the time.

3. Double Header Row

Probably to help with the ambiguous column names problem above, the downloaded CSV has a second header row that contains a more human-readable string describing the column. It’s really a concatenation of strings delimited by two exclamation points, and it’s exactly the same info that’s in the metadata CSV.

This may throw off third-party tools that import these CSVs since they tend to expect either zero or one header row, not two. Many users end up manually removing one of the header rows in excel before going on to use the data elsewhere

4. Ambiguous Column Order

As you could see in the web UI, the column order is pretty important when a human is looking at the data. Margins of Error are useless without the number they are describing the error of. If you look at the metadata CSV, there is a meaningful order to the columns, but for some reason the CSV isn’t generated with columns in this order.

In the screenshot above, you can see the first column in the CSV is the margin of error of the median income for four-person families (S1903_C03_026M). The estimated income column that we need to make sense of this margin of error is located about 50 columns further to the right (S1903_C03_026E).

Many census data crunchers spend time consolidating multiple files, renaming columns, reformatting double header rows, and reordering columns. Worse yet, they perform the same steps each time they pull down new data.

Cleaning Things Up

We sought to address each of theses shortcomings of ACS data downloads when bringing the data into the Qri dataset version control system. We built a set of scripts that hit the census API, transform the results, and create a Qri dataset for each subject table in the 2018 1-year and 5-year estimates. (Code on Github)

Each dataset has a readme, metadata, and a body, and all those components ride together. Each table is also referenced by a detailed qri dataset name under the username us-census-extracts . For example, the Median Household Income table (S1903) is now published under the qri dataset reference us-census-extracts/acs-2018-5-year-estimate-subject-table-s1903-us-by-state . There’s only one header row, the columns are in the order you’d expect, and the column names include more human-readable delimiters.

Our transformed subject tables lead with the geography name, show the columns in a logical order, cleans up the column names, and removes the “double header row” of raw ACS data downloads.

The resulting columns have pretty long names, but the headers are now cleanly formatted, easier to understand, and ready for analysis. The boring and painful part of the job is over. The fun and interesting part of the job can begin.

So How Did We Do?

Bringing these datasets into qri means they can be easily pushed and pulled, bringing their valuable metadata, readme, and schema information with them in a way that isn’t possible with file-based data. Each table also has its own page on qri.cloud, which can be linked to and discovered in search engines. As we build more collaboration features, users will be able to discuss issues with each table, share projects, and work together on downstream products.

If you use ACS data frequently, we’d love to know what you think about how we chose to transform and publish these tables. You can download Qri today and pull a subject table down to your machine in a few seconds. Give us a yell on twitter or drop into our Discord and let us know what you like and don’t like. Thanks for reading!