Cleaning and Extracting JSON From Pandas DataFrames | by Mikio Harman | Towards Data Science

Someone I know came across an interesting problem recently that took me a minute to figure out. They had found this data on Kaggle that contained columns with JSON embedded in them. There were multiple of these columns and they wanted to extract the data from these columns and represent them on the same DataFrame.
Let’s look at the problem in more detail. Here is an example of what the data looked like.
import pandas as pd

df = pd.read_csv("tmdb_5000_movies.csv")
df.head()

# A closer look at the JSON

genres_type = type(df["genres"][0])
print(f"The type of the first row fo the genres column: {genres_type}\n")

genres_sample = df["genres"][0]
print(f"The row is cast as a string even though it looks like a list of dictionaries:\n{genres_sample}")

The type of the first row fo the genres column: <class 'str'>

The row is cast as a string even though it looks like a list of dictionaries:
[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 878, "name": "Science Fiction"}]

JSON (JavaScript Object Notation) is how a lot of information is transferred across the internet. Luckily there is a library called json that comes with the Python standard library. This means that if you already have Python installed then you already have this module. If you're interested in learning more check out the docs!
By importing the json package we can turn all of our JSON objects into their respective Python data types. Python and Pandas will not tell you something is JSON explicitly, but this is usually very easy to determine if you have nested data within curly brackets ({}) cast as a str type.
Here is the code I wrote to decode my JSON into Python:
import pandas as pd
import json # The module we need to decode JSON

df = pd.read_csv("tmdb_5000_movies.csv")

sample_type = type(df["genres"][0])
print(f"The data type before using apply function: {sample_type}\n")

# Column names that contain JSON
json_cols = ['genres', 'keywords', 'production_companies', 'spoken_languages', 'production_countries']


def clean_json(x):
    "Create apply function for decoding JSON"
    return json.loads(x)

# Apply the function column wise to each column of interest
for x in json_cols:
    df[x] = df[x].apply(clean_json)

sample_type2 = type(df["genres"][0])
print(f"The data type after using apply function: {sample_type2}")

The data type before using apply function: <class 'str'>

The data type after using apply function: <class 'list'>

# Look at first row of genres
df["genres"][0]

[{'id': 28, 'name': 'Action'},
 {'id': 12, 'name': 'Adventure'},
 {'id': 14, 'name': 'Fantasy'},
 {'id': 878, 'name': 'Science Fiction'}]
# Look at 2nd element in the list of dictionaries
df["genres"][0][1]

{'id': 12, 'name': 'Adventure'}
# Access "name" key to reveal associated value
df["genres"][0][1]["name"]

'Adventure'
None of what we have done is useful unless we can extract the data from the JSON. To do this I created a function that could be used with the Pandas apply method and is applied by row and not by column (axis=1).
My idea was to one-hot-encode the data so as to maintain a Tidy format. This means that each row represents a single observation and each column represents an attribute of the observation.
I created a new column for each key and value seen in the data and if the key or value was seen in the observation of interest then this was represented with a 1 if it was not seen then a 0 was used instead.
Here is the code:
import pandas as pd
import json # The module we need to decode JSON

df = pd.read_csv("tmdb_5000_movies.csv")

sample_type = type(df["genres"][0])
print(f"The data type before using apply function: {sample_type}\n")

# Column names that contain JSON
json_cols = ['genres', 'keywords', 'production_companies', 'spoken_languages', 'production_countries']


def clean_json(x):
    "Create apply function for decoding JSON"
    return json.loads(x)

# Apply the function column wise to each column of interest
for x in json_cols:
    df[x] = df[x].apply(clean_json)

sample_type2 = type(df["genres"][0])
print(f"The data type after using apply function: {sample_type2}")

The data type before using apply function: <class 'str'>

The data type after using apply function: <class 'list'>

def clean_json2(x):

    # store values
    ls = []

    # loop through the list f dictionaries
    for y in range(len(x[0])):

        # Access each key and value in each dictionary
        for k, v in x[0][y].items():
            # append column names to ls
            ls.append(str(k)+ "_" +str(v))
   
    # create a new column or change 0 to 1 if keyword exists
    for z in range(len(ls)):

        # If column not in the df columns then make a new column and assign zero values while changing the current row to 1
        if ls[z] not in df.columns:
            df[ls[z]] = 0
            df[ls[z]].iloc[x.name] = 1
        else:
            df[ls[z]].iloc[x.name] = 1
    return

print("Original Shape",df.shape)

# Loop over all columns and clean json and create new columns
for x in json_cols:
    df[[x]].apply(clean_json2, axis=1)

print("New Shape", df.shape)

Original Shape (4803, 20)

/Users/mph/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:671: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_with_indexer(indexer, value)

New Shape (4803, 29111)

df.head()

5 rows × 29111 columns
 

Let's explain what the code is doing, first we loop over the length of the list of dictionaries and create a list to store the key:value pairs. We then loop over each dictionary and extract the key and values using .items(). .items() will return a list of tuples where the first element is the key and the second element is the associated value. From here we will append the key and value together with an underscore to our ls object.
# store values    ls = []# loop through the list of dictionaries    for y in range(len(x[0])):# Access each key and value in each dictionary        for k, v in x[0][y].items():            # append column names to ls            ls.append(str(k)+ "_" +str(v))
Next, we are going to create a new column for each combination of key:value pairs seen in the ls object. If the column does not exist then we will create a new column with the associated name and set all values to 0 while changing the value of the current observation to 1.
# create a new column or change 0 to 1 if keyword exists    for z in range(len(ls)):# If column not in the df columns then make a new column and assign zero values while changing the current row to 1        if ls[z] not in df.columns:            df[ls[z]] = 0            df[ls[z]].iloc[x.name] = 1        else:            df[ls[z]].iloc[x.name] = 1    return
x.name returns the index of the current observation. When axis=1 this will return the index values when axis=0 this will return the column name. See the docs here.
# Loop over all columns and clean json and create new columnsfor x in json_cols:    df[[x]].apply(clean_json2, axis=1)print("New Shape", df.shape)
The code returns a DataFrame that has over 29,000 columns and takes a long time to run. You may want to test on a subset of your data before testing on all of your data.
Before using this code you could search each JSON column for how many unique values are present before using the code. You could always cast lesser-seen values as ‘Other’ to help speed up the runtime.
There you have it, working with JSON data in Pandas is not that bad and with a little creativity, you can write code to implement whatever your heart desires. Feel free to adjust the code so it doesn't save certain values or keys. The code provided is meant to serve as boilerplate for whatever JSON related problem you are dealing with.
Find the code for the project here.