Making Census Data Open, Accessible and Useful for Real People
By: Ahmad Barclay
Tags: Data Literacy, Open Data, Topicsindv, Data Visualization, Census

In the case of pre-compiled Javascript frameworks, the logic is that they do away with a “virtual DOM” abstraction layer of frameworks like React, Vue and Angular, and only compile the raw “vanilla” Javascript code your app needs to run. This means that, in most scenarios, they can offer faster performance and a smaller file size. So you get the benefits of a component-based framework — importing all the bits you need from dataviz libraries like D3.js, plus a bunch of built-in features — but at the same time it rarely feels like an overkill approach, even for smaller projects.

2. Back-end

In relation to back-end data delivery, our discussion on the DVS Slack remained fairly superficial, so I thought better here to look at some of the challenges we’re trying to tackle within the ONS, in delivering data on-demand to tens of thousands of concurrent users.

First, in the UK Census, we’re looking at a raw dataset of over 50 million “rows” of data — one for each individual resident — and each containing scores of observations (on age, gender, employment, social class, ethnicity, health, housing, etc.), all of which have the potential to be cross-tabulated. Furthermore, this data always needs to be aggregated in a way that doesn’t disclose data that could identify individual people or households, so can’t just be delivered raw.

Within ONS, there are at least three parallel data architecture solutions being explored for the 2021 Census, with each one playing some kind of balancing act between speed and flexibility (e.g., allowing highly customised queries). I should note that these platforms aim to deliver data for a lot of different purposes, with dataviz apps only being one of them.

At one end of the spectrum, we are investigating pipelines (from simple batch-scripts to cloud-based solutions) to pre-packaged chunks of data into flat files (JSON, CSV, etc.) that would allow us to return common datasets instantly without the need to query an API. At the other end of the spectrum, a custom-built “flexible table builder” would allow people to access pretty much any cut of data they’re interested in — with built-in controls to prevent individual households from being identified — offering maximum flexibility with some inevitable degree of tradeoff on speed.

There are also efforts being invested in solutions somewhere in the middle of this spectrum, for example an API that can flexibly aggregate small chunks of data from a graph database, while attempting to square the performance circle by caching results, so that any common query (like names/codes for geographic areas) can be returned almost instantaneously.

3. Datasets for automation

Datasets for area profiles on Census Reporter are pre-generated to reduce the need for multiple API calls (Credit: Census Reporter)

Another topic that we’re exploring is how to deliver data formats suitable for semi-automated journalism. A typical use case here, to generate data-driven “area profiles” for hundreds or thousands of similar geographies, requires us to produce a data file for each of those places, which might include a series of indicators on population age, density and ethnicity, along with housing, health and labour market conditions. It should also show how these indicators have changed over time, and how they rank alongside all the other similar geographies and/or compare to regional- and national-level indicators.

Producing these datasets will almost certainly require “pipelines” (as referred to above) to pre-package data, since aggregating this data on-the-fly would mean requesting data from many different tables at once for an essentially static output, which simply doesn’t make sense at scale. The Knight Foundation-funded Census Reporter project adopts this kind of approach to formatting datasets for the tens of thousands of areas covered by its area profiles.

The art of the useful

But, to return to the core aim of “making census data open, accessible and useful for real people” — whether they be policy makers, journalists, NGOs or activists fighting for their communities — and, given that time and resources are finite, what kinds of outputs should be prioritised?

Detroit census response rate map (Credit: City of Detroit)

While most of DVS would consider this to be a very simplistic visualization, the overlay that includes internet access SIGNIFICANTLY repositioned the narrative on the digital divide in the City of Detroit. — Mary Aviles

We ran a poll on the Slack discussion, admittedly not the most representative of samples, but the results were interesting. Almost nobody favoured either exploratory platforms — which have long been a staple of “big data” visualisation — and, equally, static “micro-outputs” won little support. Instead, the favourites were “guided experiences” and “personalised/localised outputs”.

I think this response speaks to a growing realisation that there is a relatively low level of data literacy in the public at large, and that it is important to offer accessible stories and insights that can connect with the specific issues and places that matter to individual people.

It is also worth noting here that our early user testing suggests that the expert users capable of engaging with a more open-ended data exploration experience often prefer to download the raw data and visualise it themselves using whichever one of the vast array of dataviz tools on the market best suits their specific needs. Thus, it’s quite possible that custom-built exploratory dataviz platforms and chart builders may be serving an ever-diminishing audience.

However, at the other end of the spectrum, I tend to disagree with the apparent writing off of made-for-mobile/social media dataviz, which I would see as playing an important role in bringing insights to a wider general public audience. In this age of “fake news”, I would see clear and relatable visuals from trusted sources as an essential means to disseminate accurate information, as we have seen during the COVID-19 pandemic.

The future?

I hope this article has helped to highlight the broad spectrum of possibilities in the realm of census dataviz in 2020, and some of the formats and web technologies that we may see becoming more commonplace in the near future.

In relation to my own work, this is only the first step in a pretty long journey. In the UK, the next census is planned to take place in March 2021, with the data to be released sometime in mid 2022, and I think it’s fair to say that we can expect to see important innovations in this space between now and then.

Acknowledgements

A big thank you to everyone who contributed in one way or another to this article. Firstly, my ONS colleagues Henry Lau, Rob Fry and Darren Waters, who helped to frame the questions for the Slack discussion, to Molly Pettit, convener of the DVS “Topics in DataViz” Slack channel, and to Nightingale editors Jason Forrest and Isaac Levy-Rubinett for their patience with this write-up.

And finally, of course, a big thank you to the many DVS members who contributed to the Slack discussion, including Mary Aviles, Alexandra Khoo, Amanda Makulec, Ben Oldenburg, Andrew Whitby, Nicole Edmonds, Darragh Murray, Phil Hawkins, Alok, Karl Tryggvason, Sandy Guberti-Ng, Guillaume Meigniez, Jan Zak, Steffen Kühne, Alan Wilson and Darren Lim.

Ahmad Barclay is an architect and UX designer presently working on census data visualisation with the Office for National Statistics (ONS) in the UK. He was previously a founding partner with the award winning social impact dataviz studio Visualizing Impact and co-initiated the historical mapping and open data project Palestine Open Maps.