How to work with JSON in Python for Data Analysis | Towards Data Science

In this blog post, we are going to learn about JSON. We are going to learn what JSON is, why it exists, why people use it and finally how to utilise the power of python to help us process it.
As a data scientist, programmer or simply an avid learner of technology, it is important that you understand what JSON is and how to use it. I am certain that you will find numerous applications for it once you understand it.
JSON stands for JavaScript Object notation and it is essentially a way to represent data. JSON follows a format that humans can naturally understand, and its true power comes from its ability to capture complex data relationships.
You can see this in the following example:
{ "usersName": "Costas", "website": {    "host": "www.medium.com",    "account": "costasandreou",    "blogpost": [        {            "type": "title",            "description": {                "main" : "Working with JSON in Python",                "sub" : "the info you need"            }        }    ]  }}
Notice how JSON is structured in key-value pairs, while also holding array objects (notice that “blogpost” has several types underneath it). This is very important to note if you’re coming from a flat data structure background (think Excel or Pandas dataFrames).
JSON was created at the beginning of the millennial to save us from Flash and Java applets. If you happen to not remember those in-browser plugins, consider yourself extremely lucky, as they were the stuff that nightmares are made of! But I digress. JSON was built to provide a protocol for stateless real-time server to browser communication1.
If you are already familiar with XML, you can think of JSON as a lightweight, easier to use, faster alternative.
It is believed that the first major company to begin offering services and therefore popularising the adoption of JSON, was Yahoo in 20051. It is now believed that JSON is the most used data format.
The top 3 reasons people use JSON are:
Very easy to read, write and manipulateIt’s very fast to transfer over the networkSupported by all major browsers, backend tech stacks
Very easy to read, write and manipulate
It’s very fast to transfer over the network
Supported by all major browsers, backend tech stacks
The very first thing you’d want to do when you have to work with JSON is to read it into your Python application. The json library in Python expects JSON to come through as string.
Assuming your data JSON data is already a string:
obj = '{ "usersName": "Costas",  "website": { "host": "www.medium.com", "account": "costasandreou", "blogpost": [ { "type": "title", "description": { "main" : "Working with JSON in Python", "sub" : "the info you need" } } ] } }'import jsonjson_obj = json.loads(obj)print('obj: ',type(obj))print('obj.usersName: ', json_obj['usersName'])
which returns:
obj:  <class 'str'>obj.usersName:  Costas
If on the other hand, if you hold the JSON in a Python Object (like a dictionary), the json library allows you to convert it back to string for further processing.
obj1 = { "usersName": "Costas",  "website": { "host": "www.medium.com", "account": "costasandreou", "blogpost": [ { "type": "title", "description": { "main" : "Working with JSON in Python", "sub" : "the info you need" } } ] } }print('obj1: ',type(obj1))json_obj1str = json.dumps(obj1)print('json_obj1str: ', type(json_obj1str))json_obj1 = json.loads(json_obj1str)print('obj1.usersName: ', json_obj1['usersName'])
which returns:
obj1:  <class 'dict'>json_obj1str:  <class 'str'>obj1.usersName:  Costas
Now that we have seen how to load our JSON data into our Python application, let us look at the options we have to work with JSON.
Extracting information from the JSON is as simple as defining the path we are after and provide the name of the key-value pair. Let’s look at a few examples.
>>> print(json_obj['usersName'])Costas>>> print(json_obj['website']){'host': 'www.medium.com', 'account': 'costasandreou', 'blogpost': [{'type': 'title', 'description': {'main': 'Working with JSON in Python', 'sub': 'the info you need'}}]}>>> print(json_obj['website']['host'])www.medium.com>>> print(json_obj['website']['blogpost'])[{'type': 'title', 'description': {'main': 'Working with JSON in Python', 'sub': 'the info you need'}}]>>> print(json_obj['website']['blogpost'][0]['description']['sub'])the info you need
Once we can extract the data as above, we can easily store them in lists or databases for further processing.
If you are already comfortable in Pandas and you’d like to work with flat data, there is a quick way to get your data into data frames. First up, you can take the first level of your data and add it to a dataFrame.
df = pd.read_json(obj)
As you can see, read_json takes in a string data type and only allows us to see top-level information. In other words, it doesn’t flatten any of the structures. We need a way to navigate through the lower levels of the data.
We can flatten the data by a further level by using the json_normalise pandas method.
obj = json.loads(obj)df1 = pd.json_normalize(obj)
There are certain cases where you simply want to flatten out the data. It certainly can make data analysis quicker at times and can allow for the visual inspection of a large number of records (think Excel type data filtering).
You can do that using a third-party library. First up, install the library:
pip install json-flatten
Then, run the following commands:
>>> import json>>> obj = json.loads(obj)>>> import json_flatten>>> print(json_flatten.flatten(obj)){'usersName': 'Costas', 'website.host': 'www.medium.com', 'website.account': 'costasandreou', 'website.blogpost.0.type': 'title', 'website.blogpost.0.description.main': 'Working with JSON in Python', 'website.blogpost.0.description.sub': 'the info you need'}
All the items we have explored so far focused on processing a single JSON record. However, in the real world, you are likely required to process many JSON documents and be required to aggregate across records.
You can consider any of the following methods for enabling your analysis:
MongoDB to store the data and then run aggregations on top
Flatten the data and use pandas for operations
Flatten the data and export to Excel for further operations
Choose the attributes you are after and load them in a SQL DB for further analysis
There you have it. In exactly 5 minutes you know everything you need to know to get you started working with JSON. I encourage you to spend some time and get intimately familiar with it. It is widely used and it is certainly a must-have on your CV!
{ "References" : [  {    "1" : "https://en.wikipedia.org/wiki/JSON#History"  },  {    "2" : "guru99.com/json-vs-xml-difference.html"  },  {    "3" : "https://pypi.org/project/json-flatten/#description"  }, ]}