How to Best Work with JSON in Python
By: Julia Kho
Tags: Python, Editors Pick, Python Programming, Programming, Data Science

In this article, you will learn how to read, parse, and write JSON files in Python. I will talk about how to best handle simple JSON files as well as nested JSON files. In addition, I will discuss how to access specific values in the data.

What is JSON?

JSON (Java Script Object Notation) is a popular file format for storing and transmitting data in web applications. It is highly likely that you’ll encounter JSON files if you work with data, so you definitely want to learn how to read and write to JSON.

Here is an example of what the structure of a JSON file might look like.

Image by author

The JSON structure looks very similar to Python dictionaries. Notice that the JSON is in the format of “key”: <value> pairs, where key is a string and value can be a string, number, boolean, array, object, or null. To help you visualize that, I’ve highlighted all keys in blue and all values in orange in the image below. Notice that each key/value is also separated by a comma. Go ahead and examine the image below to familiarize yourself with the format. We will discuss the nested structure of the ‘members’ key later on in this article.

Image by author

Importing JSON to Python

Python conveniently has built in functions to help read JSON files. Below are several examples of how to parse JSON files into a Python object.

Before we get started, if you would like to follow along with the examples:

Go to this link. Right click on the page and select Save As. Save the file as a JSON file named superheroes.

Example 1: Loading JSON to Python dictionary

Start by importing the json library. We use the function open to read the JSON file and then the method json.load() to parse the JSON string into a Python dictionary called superHeroSquad. That’s it! You now have a Python dictionary from your JSON file. Pretty simple.

import json



with open('superheroes.json') as f:

superHeroSquad = json.load(f) type(superHeroSquad)

# Output: dict superHeroSquad.keys()

# Output: dict_keys(['squadName', 'homeTown', 'formed', 'secretBase', 'active', 'members'])

One thing of note is that the json library has both load() and loads() . Both do the same thing, but loads() is to create a Python object from a JSON string whereas load() is to create a Python object from a JSON file. You can think of the extra ‘s’ in loads() as “load for strings”.

Example 2: Loading JSON to Pandas Dataframe

Use the method read_json() if you would like to transform the JSON file to a Pandas Dataframe.

import pandas as pd df = pd.read_json(‘superheroes.json’)

Image by author

Note that you are not limited to reading JSON files from your computer. You can pass the URL path to the function as well. This will save you the step of having to download the JSON file.

Example 3: Loading Nested JSON to Pandas

You will likely encounter JSON files that are nested. That usually makes it difficult to work with in Pandas. Nested JSON is similar to the idea of nested dictionaries in python, that is, a dictionary within a dictionary.

Let’s examine the superhero data structure again. Notice that within the ‘members’ key, there are multiple sets of key/value pairs related to the ‘members’ key. The indentation is a helpful indicator illustrating the nested structure.

Image by author

Recall from earlier when we loaded our JSON file into a Pandas dataframe, our ‘members’ column looked like this. Each row contains a dictionary.

Image by author

The column ‘members’ would be much more convenient to work with in Pandas if each of the keys were actually individual columns. I’ll discuss two methods in which we can parse out the data so that each key is broken out into a column.

Method 1

We can use the apply method on the ‘members’ column like this.

df[‘members’].apply(pd.Series)

The four keys within ‘members’ are now broken out to separate columns as shown below. This is much easier for performing data manipulation and transformation.

Image by author

To combine the new columns with the original dataframe, we can use pd.concat .

df = pd.concat([df[‘members’].apply(pd.Series), df.drop(‘members’, axis = 1)], axis = 1)

Method 2

Pandas also has the built-in function json_normalize() that will allow you to flatten nested JSONs. This is a cleaner method to parse the nested JSON.

pd.json_normalize(superHeroSquad, record_path = [‘members’], meta = [‘squadName’, ‘homeTown’, ‘formed’, ‘secretBase’, ‘active’])

We pass in our superHeroSquad dictionary.

record_path contains the column that we want parsed out.

contains the column that we want parsed out. meta is a list of columns we want to keep for the dataframe.

Image by author

One last note: you can also add in the parameter meta_prefix if you would like to add in a particular naming convention to the parsed data.

pd.json_normalize(superHeroSquad, record_path = ['members'], meta = ['squadName', 'homeTown', 'formed', 'secretBase', 'active'], meta_prefix = 'members_')

Image by author

Accessing the data

We can access data anywhere in our JSON file by chaining together the key names and/or indices. For example, let’s say we want to access the secret identity of our second superhero, Madame Uppercut. Note the location of that particular value is highlighted in purple below.

Image by author

In order to access that value, we can use the following code.

superHeroSquad[‘members’][1][‘secretIdentity’]

Starting from the top of the hierarchy and working our way down, the first key that we need is ‘members’ as that is the parent node that our value resides.

Image by author

Inside the ‘members’ key, we are looking for Madame Uppercut, which is the second superhero in the list. Remember that the index for the first items always start at 0, so we want to use 1 to access the second superhero.

Lastly, we are interested in the key ‘secretIdentity’.

Image by author

Combining all of that together gives us this line of code below which returns the value “Jane Wilson”.

superHeroSquad[‘members’][1][‘secretIdentity’]

You might have noticed that I have highlighted two values in blue in the JSON snippet above. Go ahead and try accessing those values as individual exercises. Feel free to share your code in the comments below.

Exporting from Python to JSON

Let’s make a quick edit to the missing secret identity of our last superhero, Eternal Flame, from “Unknown” to “Will Smith” and then export our superHeroSquad dictionary back to the JSON file. We use the method json.dump() to write to the file.

#update secret identity of Eternal Flame

superHeroSquad[‘members’][2][‘secretIdentity’] = 'Will Smith' with open('superheroes.json', 'w') as file:

json.dump(superHeroSquad, file)

If you open your superheroes.json file, it should now have Will Smith as the secret identity of Eternal Flame. Similar to load() and loads() mentioned previously, the json library also has dump() and dumps() . One to store as a JSON file and one to store as a JSON string respectively.

Alternatively, if you are working with the Pandas Dataframe and would like to export to JSON, you can use the to_json() method.

df.to_json(‘superheroes.json’)

Pretty-Printing

You may notice that the JSON file does not look very nice in your output file. It shows up as a single string like below.

Image by author

To make it look prettier, you can use the indent parameter in your json.dump method.

with open('superheroes.json', 'w') as file:

json.dump(superHeroSquad, file, indent = 4)

The output will look like the image below, which is much more readable.

Image by author

Sorting

If required, you may also pass in a sort_key parameter, set to True, in order to sort your keys. Notice that all keys including the nested ones are all sorted.

with open('superheroes.json', 'w') as file:

json.dump(superHeroSquad, file, indent = 4, sort_keys = True)

Image by author

In Summary

JSON data structure is in the format of “key”: <value> pairs, where key is a string and value can be a string, number, boolean, array, object, or null.

pairs, where is a string and can be a string, number, boolean, array, object, or null. Python has built in functions that easily imports JSON files as a Python dictionary or a Pandas dataframe.

Use pd.read_json() to load simple JSONs and pd.json_normalize() to load nested JSONs.

to load simple JSONs and to load nested JSONs. You can easily access values in your JSON file by chaining together the key names and/or indices.

Python objects can be exported back to JSON with pretty printing and sorting.