All Pandas json_normalize() you should know for flattening JSON
By: B. Chen
Tags: Data Analysis, Data Science, Python, Json, Pandas

All nested lists are put up into a single column students and other values are flattened. To flatten the nested list, we can set the argument record_path to ['students'] . Notices that not all records have math and physics, and those missing values are shown as NaN .

pd.json_normalize(json_list, record_path=['students'])

(image by author)

If you would like to include other metadata use the argument meta :

pd.json_normalize(

json_list,

record_path =['students'],

meta=['class', 'room', ['info', 'teachers', 'math']]

)

(image by author)

4. The errors argument

The errors argument default to 'raise’ and will raise KeyError if keys listed in meta are not always present. For example, the math teacher is not available from the second record.

data = [

{

'class': 'Year 1',

'student count': 20,

'room': 'Yellow',

'info': {

'teachers': {

'math': 'Rick Scott',

'physics': 'Elon Mask',

}

},

'students': [

{ 'name': 'Tom', 'sex': 'M' },

{ 'name': 'James', 'sex': 'M' },

]

},

{

'class': 'Year 2',

'student count': 25,

'room': 'Blue',

'info': {

'teachers': {

# no math teacher

'physics': 'Albert Einstein'

}

},

'students': [

{ 'name': 'Tony', 'sex': 'M' },

{ 'name': 'Jacqueline', 'sex': 'F' },

]

},

]

A KeyError will be thrown when trying to flatten the math.

pd.json_normalize(

data,

record_path =['students'],

meta=['class', 'room', ['info', 'teachers', 'math']],

)

(image by author)

To work around it, set the argument errors to 'ignore' and those missing values are filled with NaN .

pd.json_normalize(

data,

record_path =['students'],

meta=['class', 'room', ['info', 'teachers', 'math']],

errors='ignore'

)

(image by author)

5. Custom Separator using the sep argument

By default, all nested values will generate column names separated by . . For example info.teachers.math. To separate column names with something else, you can use the sep argument.

pd.json_normalize(

data,

record_path =['students'],

meta=['class', 'room', ['info', 'teachers', 'math']],

sep='->'

)

(image by author)

6. Adding prefix for meta and record data

Sometimes, it may be more descriptive to add prefixes for the column names. To do that for the meta and record_path , we can simply pass the string to the argument meta_prefix and record_prefix respectively:

pd.json_normalize(

data,

record_path=['students'],

meta=['class'],

meta_prefix='meta-',

record_prefix='student-'

)

(image by author)

7. Working with a local file

Often, the JSON data you will be working on is stored locally as a .json file. However, Pandas json_normalize() function only accepts a dict or a list of dicts. To work around it, you need help from a 3rd module, for example, the Python json module:

import json

# load data using Python JSON module

with open('data/simple.json','r') as f:

data = json.loads(f.read())



# Flattening JSON data

pd.json_normalize(data)

data = json.loads(f.read()) loads data using Python json module. After that, json_normalize() is called on the data to flatten it into a DataFrame.

8. Working with a URL

JSON is a standard format for transferring data in REST APIs. Often, you need to work with API’s response in JSON format. The simplest way to do that is using the Python request modules:

import requests

data = json.loads(requests.get(URL).text) URL = ' http://raw.githubusercontent.com/BindiChen/machine-learning/master/data-analysis/027-pandas-convert-json/data/simple.json' # Flattening JSON data

pd.json_normalize(data)

Conclusion

Pandas json_normalize() function is a quick, convenient, and powerful way for flattening JSON into a DataFrame.

I hope this article will help you to save time in flattening JSON data. I recommend you to check out the documentation for the json_normalize() API and to know about other things you can do.

Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.

You may be interested in some of my other Pandas articles:

More tutorials can be found on my Github