Making open data more accessible with datasette
By: Sergio Sánchez
Tags: Towards Data Science, Police Data, Open Data, Datasette, Python

Photo by NASA on Unsplash

California recently released data related to stops made by officers in the eight largest agencies of the state covering the months of July to December of 2018. This was the first wave of data releases that will take effect in the following years. The data covered over 1.8 million stops across the state. While this is a step in the right direction, a single .csv file of around 640 megabytes with 1.8+ million rows and 140+ columns might be intimidating for some folks who would benefit from exploring this data: local leaders, journalists, activists, and organizers, to name a few.

The third principle of the open data charter is that data should be accessible and usable. Providing the data is the first step but to make it usable and accessible for most people we cannot simply publish data; we must take into account user experience and develop ways to facilitate the exploration and use of said data.

In this tutorial, I share one way to do it: using datasette, a python "tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API."

We take California’s recently released police stop data, clean and transform the data, and deploy a datasette instance to Heroku.

Table of contents

— prologue

— about the data

— about datasette

— step-by-step

— — acquiring the data

— — preparing the data

— — serving the data

— deploying to heroku

— automating everything

— github actions

— bonus: using github issues to suggest canned queries

— how long it took

— call to action

prologue

The idea to explore this dataset came from listening to the March 3rd, 2020 episode of Pod Save the People

where they mention this article from The Appeal:

Boudin will announce a second directive today, also reviewed by The Appeal, on what are known as pre-textual stops, in which an officer stops someone for a minor offense or infraction, such as a traffic violation, in order to conduct an unrelated search for items like guns or drugs. According to the new policy, the DA’s office will not prosecute possession of contraband cases when the contraband was collected as a result of an infraction-related stop, “where there is no other articulable suspicion of criminal activity.” Any deviations from the policy should be made in writing and require approval from the DA or a chief of the criminal division. Additionally, the ban includes cases in which a person consented to a search “because of the long-standing and documented racial and ethnic disparities in law enforcement requests for consent to search,” according to the directive.

- https://theappeal.org/san-francisco-da-to-announce-sweeping-changes-on-sentencing-policy-and-police-stops/

In the episode Sam mentions the black and brown folks are being stopped and searched at higher rates and that a lot of these searches are what they call ‘consent searches’ which means that the police actually report no justification at all for searching the person other than asking that person if they can search them and the person allegedly giving consent.

This gross racial disparity is heart-breaking but not surprising.

When I first set out to explore the dataset myself, I found it cumbersome to work with. The dataset size makes it hard for those who are not analyzing data programmatically with paid resources like stata and sas or open-sourced ones like python and R. This information is not designed to be explored easily but there are tools that can help with that.

💡Note: AB-953, the bill that requires agencies to report all data on stops to the Attorney General, does not require the data to be “easily explorable” or “accessible”. The bill requires the data to be collected and reported to the Attorney General. This dataset as it is serves that purpose.

about the data

The dataset is composed of one .csv file of 641.4 MB which contains 1.8 million rows and 143 columns. Each stop instance has a unique DOJ_RECORD_ID and each person stopped has a PERSON_NUMBER . In total there are 1,708,377 stops in this dataset involving 1,800,054 people.

The dataset includes basic information about each stop (like duration, time of day, closest city, agency name), perceived demographic information (race/ethnicity, gender, age, disability), as well as information regarding the reason for stop, for search, for seizure, actions taken during stop, contraband or evidence found, etc. For more detailed information about the dataset you can read the README file the Department of Justice put together and the 2020 Annual Report.

This is Wave I of a series of RIPA data to be released in the following years. The data released this wave pertains to the 8 largest law enforcing agencies (LEAs) in California (those employing over 1,000 officers). These LEAs (and their share of the total observations) are:

README | 2020 Annual report

about datasette

Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API. Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world. It is part of a wider ecosystem of tools and plugins dedicated to making working with structured data as productive as possible.

- datasette.readthedocs.io

datasette is the engine powering this project. In short, it grabs a sqlite database and creates an interactive, explorable website and accompanying API. To prepare the data we used csvs-to-sqlite , another tool from the datasette ecosystem which grabs CSV files and creates sqlite databases from them.

You can find some examples of public datasettes on the GitHub repo’s wiki. Here is one serving the datasets published by FiveThirtyEight (found in their GitHub repo): https://fivethirtyeight.datasettes.com/

The home page shows the data’s license, source, and provides quick access to a few of its tables.

If you click on your database’s name it’ll provide a view of all its tables and provide a text box for you to run custom SQL queries. This is one of the features that makes datasette such a powerful tool.

If you dig into one of the tables you’ll see suggested facets, an easy-to-use filter button, a JSON and CSV link which will provide the table as one of those formats (this means you can use this as an API), and a description for the table. In this case, the description contains an HTML table with headers and definitions.

If you click on the View and edit SQL button you will again get access to a text box to write your own queries on.

datasette is part of a rich ecosystem of tools and plugins. You can add the datasette-vega plugin (repo) to add interactive visualizations of your current table (made with altair)

That is what a basic datasette website looks like but datasette is highly customizable. Take for example The Baltimore Sun’s datasette for people to explore public salary records which is updated yearly.

This is running datasette in the backend but they have added their own templates/layouts, css and javascript files. More on how to customize datasette here: https://datasette.readthedocs.io/en/stable/custom_templates.html

step-by-step

This section is a more technical description of the project. The data preparation and deployment of datasette is pretty straight-forward and it can be divided into three phases:

— Acquiring the data

— Preparing the data

— Serving the data

acquiring the data

The data was retrieved from the Open Justice website of the California Department of Justice: https://openjustice.doj.ca.gov/data. The website provides a link to download the data (as of May 5th, 2020, when I downloaded it for this project). To learn more about the data itself you can read the About the data section on the project’s GitHub repo.

preparing the data

The original CSV file is over 650 MB in size so the very first step was to slice it into 15 CSV files so that each could be uploaded to GitHub in this repository.

💡Why? Just in case it mysteriously disappears from the original data source…

However, because the dataset is too large to serve as a single table (1.8 million rows by 143 columns) it was also broken down into smaller tables. This means we took related variables (based on their suffixes) and extracted them into their own tables. For example, variables related to gender like G_FULL , G_MALE , G_FEMALE , G_TRANSGENDER_MAN , G_TRANSGENDER_WOMAN , G_GENDER_NONCOMFORMING , and G_MULTIGENDER were extracted from the "main" table and were added to a gender table in the database. These can be joined back to the main table using a UNIQUE_ID assigned to them.

Each observation or row of this dataset is assigned a DOJ_RECORD_ID and a PERSON_NUMBER . These are unique to the stop and the person(s) stopped, respectively. This means we could combine them to create a UNIQUE_ID for each row which we could use to join tables together. However, this ends up being a 22 character string which is unnecessarily large. To facilitate things, each row is assigned a numeric id starting at 1,000,000. Starting at one million is completely arbitrary, we could have started at zero but because there are 1.8 million rows, we made the decision to have each numeric id be seven digits. This numeric UNIQUE_ID lets us join tables together and is not a big addition to the database in terms of memory.

Once this UNIQUE_ID is created we can extract columns from the "main" table into their own tables and save those as individual CSV files with the certainty that each observation can be matched across these new tables .

We then use csvs-to-sqlite to create a sqlite database where each CSV is a table. In this step, we also include the Appendix B Table 3.csv file obtained also from the DOJ's website and any other supplemental tables we might have created to accompany the dataset.

csvs-to-sqlite data/processed/*.csv "data/external/Appendix B Table 3.csv" datasette/ripa-2018-db.db

serving the data

After preparing the data and creating the sqlite database we use datasette to serve it as an interactive website and API. This is as easy as running

datasette ripa-2018.db

However, for this project, we customize our datasette instance a bit.

We include a title, description, data source URL, and some extra CSS and JS assets. You can explore datasette/metadata.json to find the specifics.

We also include canned queries; queries included by default in our instance that are displayed in the main page and come with their own URL to facilitate access. These queries are included because they are useful and/or interesting facts found in the data. Some of them are queries that compute specific facts published in the 2020 Annual Report.

We also modify some templates from datasette, specifically base.html and query.html . The first was modified to include some metadata in the <head> (website preview description and such). The second was modified to include a button below the SQL text box to submit the query a user just ran as a suggestion on GitHub to facilitate the sharing of these.

We also change some default options for datasette:

default_page_size:50 - Show only 50 results per page sql_time_limit_ms:30000 - 30 second time limit computing queries (time limit on Heroku) facet_time_limit_ms:10000 - The time limit datasette should spend calculating a facet of your table (it defaults to 200 ms but because our dataset is so large we expand it to 10 seconds)

The resulting code to run is:

datasette ripa-2018-db.db \\

-m metadata.json \\

--extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000"

deploying to heroku

Here’s a high-level overview of the process

So far, we used some python code to process our data (creating a UNIQUE_ID and breaking down the original dataset with 143 columns into various smaller .csv files) and we also used csvs-to-sqlite to build our ripa-2018-db.db database from those .csv files. Until now, we've been interacting with our datasette by running the command datasette ripa-2018-db.db which runs a local server. To make our datasette available to the world we have to deploy it online and luckily it is very, very easy to do as datasette already includes a publish command.

With datasette, you can publish directly to Heroku, Google Cloud Run, or Fly (as of version 0.42). Because I have previous experience deploying to heroku, I found it the easiest of the three but they are all great options and the documentation is easy to follow.

Deploying to Heroku was as easy as running almost the same command used to serve our datasette locally.

datasette publish heroku ripa-2018-db.db \\

--name ripa-2018-db \\

-m metadata.json \\

--extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000"

💡 Notice the --name flag which specifies the name for our heroku app. This means it'll publish to ripa-2018-db.herokuapp.com and it'll overwrite the previous app if there is one.

To run this, you’ll need a heroku account and to install the heroku cli tool. datasette (using the heroku cli under the hood) will ask you to log in and open up a browser window for you to do so. After you log in, it’ll take care of the rest.

¡Listo! At this point we have successfully published our datasette to heroku and we can visit ripa-2018-db.herokuapp.com or ripa-2018.datasettes.cimarron.io/

This will be the end of this process for many people. We’ve successfully grabbed and transformed some data and deployed it to the cloud for others to explore and interact with. We’ve also included an HTML description describing our datasette for users and even some canned queries for people to reproduce some of the facts published in the 2020 Annual Report.

automating everything

While the underlying data your serving might not change often you might find yourself wanting to automate the deployment of your datasette instance for many reasons. For example, when I started this project datasette was in version 0.40 and at the time I’m writing this post it is at version 0.42. Most versions you won’t want to update your datasette instance for but version 0.41 included the ability to create custom pages (changelog).

For example, adding a template file called templates/pages/about.html will result in a new page being served at /about on your instance.

This means we can add much more context in our instance for users. Maybe include a step-by-step guide to help people contribute to the project, other useful links, or a simple page introducing yourself so people using this data learn a bit more about you.

You might also want to include more canned queries or fix a typo on your description. Whatever the reason, automating deployment is easy to implement. Ultimately all you need is to run datasette heroku publish which is a perfect use case for GitHub Actions