How to parse JSON data with Python Pandas?
By: Ankit Goel
Tags: Python, Data Engineering, Programming, Json, Pandas

How to parse JSON data with Python Pandas?

Source: Image by Author

If you are doing anything related to data whether it is Data Engineering, Data Analytics, or even Data Science, you would have surely come across JSONs.

JSON (JavaScript Object Notation) is one of the most used data formats for exchanging data over the web. NoSQL based databases like MongoDB store the data in this format. Although this format works well for storing the data, it needs to be converted into a tabular form for further analysis.

In this story, we will see how easy it is to parse JSON data and convert it into the tabular form. You can download the sample data from the GitHub repository mentioned below. Also, have a look at the notebook for more details about the data and APIs used.

Data Details

I am going to use the data which I generated while working on the Machine Learning clustering problem. No need to worry if this data doesn’t make sense as it is used only for demo purposes. I will use two different JSON-

Simple JSON with no nested lists/dictionaries.

This is already flattened JSON and requires minimal processing.

Sample Record:

{

"Scaler": "Standard",

"family_min_samples_percentage": 5,

"original_number_of_clusters": 4,

"eps_value": 0.1,

"min_samples": 5,

"number_of_clusters": 9,

"number_of_noise_samples": 72,

"adjusted_rand_index": 0.001,

"adjusted_mutual_info_score": 0.009,

"homogeneity_score": 0.330,

"completeness_score": 0.999,

"v_measure_score": 0.497,

"fowlkes_mallows_score": 0.0282,

"silhouette_coefficient": 0.653,

"calinski_harabasz_score": 10.81,

"davies_bouldin_score": 1.70

}

2. JSON with nested lists/dictionaries.

This might seems a little complicated and in general, would require you to write a script for flattening. Later, we will see how it can be converted into a DataFrame with just 1 line of code.

Sample Record:

{

'Scaler': 'Standard',

'family_min_samples_percentage': 5,

'original_number_of_clusters': 4,

'Results':

[

{

'eps_value': 0.1,

'min_samples': 5,

'number_of_clusters': 9,

'number_of_noise_samples': 72,

'scores':

{

'adjusted_rand_index': 0.001,

'adjusted_mutual_info_score': 0.009,

'homogeneity_score': 0.331,

'completeness_score': 0.999,

'v_measure_score': 0.497,

'fowlkes_mallows_score': 0.028,

'silhouette_coefficient': 0.653,

'calinski_harabasz_score': 10.81,

'davies_bouldin_score': 1.70

}

},

{

'eps_value': 0.1,

'min_samples': 10,

'number_of_clusters': 6,

'number_of_noise_samples': 89,

'scores':

{

'adjusted_rand_index': 0.001,

'adjusted_mutual_info_score': 0.008,

'homogeneity_score': 0.294,

'completeness_score': 0.999,

'v_measure_score': 0.455,

'fowlkes_mallows_score': 0.026,

'silhouette_coefficient': 0.561,

'calinski_harabasz_score': 12.528,

'davies_bouldin_score': 1.760

}

}

]

}

Pandas read_json()

This API from Pandas helps to read JSON data and works great for already flattened data like we have in our Example 1. You can download the JSON from here.

# Reading JSON

pd.read_json('level_1.json')

Just reading the JSON converted it into a flat table below.

Flattened data using read_json() by Author

It doesn’t work well when the JSON data is semi-structured i.e. contains nested list or dictionaries as we have in Example 2.

# Example 2 JSON

pd.read_json('multiple_levels.json')

After reading this JSON, we can see below that our nested list is put up into a single column ‘Results’. In the next section, we will see how we can flatten this data.

Source: Image by Author

Pandas json_normalize()

This API is mainly designed to convert semi-structured JSON data into a flat table or DataFrame. You can download the example JSON from here.

# load data using Python JSON module

with open('multiple_levels.json','r') as f:

data = json.loads(f.read()) # Normalizing data

multiple_level_data = pd.json_normalize(data, record_path =['Results'], meta =['original_number_of_clusters','Scaler','family_min_samples_percentage'], meta_prefix='config_params_', record_prefix='dbscan_') # Saving to CSV format

multiple_level_data.to_csv('multiplelevel_normalized_data.csv', index=False)

Flatted data using json_normalize() by Author

Now, let’s understand 1 magical line that I used here to convert JSON to flat-table. This API provides different parameters and you can read more about them here. Here we used-

record_path

record_prefix

meta

meta_prefix

Let’s go through each of them one by one.

record_path

This is the path to the nested array or list which we would like to flatten. In our example, it is ‘Results’ which have multiple nested dictionaries containing algorithm hyperparameters and metrics score.

record_prefix

By default, json_normalize would append a prefix (string) for nested dictionaries based on the parent data like in our example davies_bouldin_score converted to scores.davies_bouldin_score. For nested lists, we can use record_prefix to append to the flattened data. In our example, each value in the ‘Results’ list is related to the DBSCAN algorithm and hence we can append the same(dbscan_). This will convert scores.davies_bouldin_score to dbscan_scores.davies_bouldin_score.

meta

This parameter helps to add metadata with each record into the resulting table. Here the metadata would be which is outside the record_path i.e. Scaler, original_number_of_clusters, or family_min_samples_percentage.

meta_prefix

Similar to record_prefix, meta_prefix would append the string to the metadata defined as part of ‘meta’. In our example, all the metadata is related to configuration parameters for our ML problem and hence we appended ‘config_params_’ to each metadata.