How to parse JSON data with Python Pandas? | by Ankit Goel | Towards Data Science

If you are doing anything related to data whether it is Data Engineering, Data Analytics, or even Data Science, you would have surely come across JSONs.
JSON (JavaScript Object Notation) is one of the most used data formats for exchanging data over the web. NoSQL based databases like MongoDB store the data in this format. Although this format works well for storing the data, it needs to be converted into a tabular form for further analysis.
In this story, we will see how easy it is to parse JSON data and convert it into the tabular form. You can download the sample data from the GitHub repository mentioned below. Also, have a look at the notebook for more details about the data and APIs used.
github.com
I am going to use the data which I generated while working on the Machine Learning clustering problem. No need to worry if this data doesn’t make sense as it is used only for demo purposes. I will use two different JSON-
Simple JSON with no nested lists/dictionaries.
Simple JSON with no nested lists/dictionaries.
This is already flattened JSON and requires minimal processing.
Sample Record:{    "Scaler": "Standard",    "family_min_samples_percentage": 5,    "original_number_of_clusters": 4,    "eps_value": 0.1,    "min_samples": 5,    "number_of_clusters": 9,    "number_of_noise_samples": 72,    "adjusted_rand_index": 0.001,    "adjusted_mutual_info_score": 0.009,    "homogeneity_score": 0.330,    "completeness_score": 0.999,    "v_measure_score": 0.497,    "fowlkes_mallows_score": 0.0282,    "silhouette_coefficient": 0.653,    "calinski_harabasz_score": 10.81,    "davies_bouldin_score": 1.70}
2. JSON with nested lists/dictionaries.
This might seems a little complicated and in general, would require you to write a script for flattening. Later, we will see how it can be converted into a DataFrame with just 1 line of code.
Sample Record:{  'Scaler': 'Standard',  'family_min_samples_percentage': 5,  'original_number_of_clusters': 4,  'Results':   [      {        'eps_value': 0.1,        'min_samples': 5,        'number_of_clusters': 9,        'number_of_noise_samples': 72,        'scores':             {             'adjusted_rand_index': 0.001,             'adjusted_mutual_info_score': 0.009,             'homogeneity_score': 0.331,             'completeness_score': 0.999,             'v_measure_score': 0.497,             'fowlkes_mallows_score': 0.028,             'silhouette_coefficient': 0.653,             'calinski_harabasz_score': 10.81,             'davies_bouldin_score': 1.70            }      },      {        'eps_value': 0.1,        'min_samples': 10,        'number_of_clusters': 6,        'number_of_noise_samples': 89,        'scores':             {             'adjusted_rand_index': 0.001,             'adjusted_mutual_info_score': 0.008,             'homogeneity_score': 0.294,             'completeness_score': 0.999,             'v_measure_score': 0.455,             'fowlkes_mallows_score': 0.026,             'silhouette_coefficient': 0.561,             'calinski_harabasz_score': 12.528,             'davies_bouldin_score': 1.760            }      }  ]}
This API from Pandas helps to read JSON data and works great for already flattened data like we have in our Example 1. You can download the JSON from here.
# Reading JSONpd.read_json('level_1.json')
Just reading the JSON converted it into a flat table below.
It doesn’t work well when the JSON data is semi-structured i.e. contains nested list or dictionaries as we have in Example 2.
# Example 2 JSONpd.read_json('multiple_levels.json')
After reading this JSON, we can see below that our nested list is put up into a single column ‘Results’. In the next section, we will see how we can flatten this data.
This API is mainly designed to convert semi-structured JSON data into a flat table or DataFrame. You can download the example JSON from here.
# load data using Python JSON modulewith open('multiple_levels.json','r') as f:    data = json.loads(f.read())# Normalizing datamultiple_level_data = pd.json_normalize(data, record_path =['Results'], meta =['original_number_of_clusters','Scaler','family_min_samples_percentage'], meta_prefix='config_params_', record_prefix='dbscan_')# Saving to CSV formatmultiple_level_data.to_csv('multiplelevel_normalized_data.csv', index=False)
Now, let’s understand 1 magical line that I used here to convert JSON to flat-table. This API provides different parameters and you can read more about them here. Here we used-
record_path
record_prefix
meta
meta_prefix
Let’s go through each of them one by one.
record_path
This is the path to the nested array or list which we would like to flatten. In our example, it is ‘Results’ which have multiple nested dictionaries containing algorithm hyperparameters and metrics score.
record_prefix
By default, json_normalize would append a prefix (string) for nested dictionaries based on the parent data like in our example davies_bouldin_score converted to scores.davies_bouldin_score. For nested lists, we can use record_prefix to append to the flattened data. In our example, each value in the ‘Results’ list is related to the DBSCAN algorithm and hence we can append the same(dbscan_). This will convert scores.davies_bouldin_score to dbscan_scores.davies_bouldin_score.
meta
This parameter helps to add metadata with each record into the resulting table. Here the metadata would be which is outside the record_path i.e. Scaler, original_number_of_clusters, or family_min_samples_percentage.
meta_prefix
Similar to record_prefix, meta_prefix would append the string to the metadata defined as part of ‘meta’. In our example, all the metadata is related to configuration parameters for our ML problem and hence we appended ‘config_params_’ to each metadata.
I hope this article would help the next time you deal with JSON data. I used to write python scripts to flatten the data or use various packages which would take my entire day figuring out things and now it’s just a matter of 10 seconds. If you are interested in Data Science or would like to learn about some cool python libraries, then check out my other blog- Start your Data Science journey today.
Stay Safe!!! Keep Learning!!!