Working with JSON and Python in your Data Projects
Tags: Data Science, Python, Jupyter Notebook, Json, Bigquery

Working with JSON and Python in your Data Projects

For many different kinds of projects you will get in touch with data in JSON format — Here is what you need to know to get started.

Photo by XPS on Unsplash

Use cases can variate for example from big data integration projects, where you transfer data from an API to your Data Warehouse to a one-man-data science project where you need some external data to your Jupyter Notebook. How to deal with JSON and to extract the data you need will be explained in the following article with the help of some Python examples.

JSON on the fly

JSON (JavaScript Object Notation) is a data-interchange format. Users can easily read and write this format, and machines can easily parse and generate.

JSON is built on two structures:

A collection of name/value pairs: In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array.

An ordered list of values: In most languages, this is realized as an array, vector, list, or sequence [1].

In the figure below, you can see a typical JSON example. The JSON object begins with a { followed by a name (“company”) and a value (“mycompany”). Often, you will find also objects inside an object. The element “employee” is a classical example of an array which also can have another array within.

Example JSON — Source: techplanet.com [2]

Examples

For the examples I used the API from https://finnhub.io/ which provides paid services besides a lot of free to use financial data.

Example 1: JSON with object in object

With the libraries requests and json you easily get the data from the API:

Jupyter Notebook

After getting the response in a JSON format you can easily flatten it with json_normalize and load it into a Pandas dataframe.

Having the data in a data frame then makes it easy to use the data for Data Science tasks or to load it into a database.

Example 2: Read from JSON array

It is a bit harder to get the data out of an array in an object but also with objects within the array itself — like you can see here:

The idea is to get the period based data for every stock in a relational table structure. Here, we will need some loops:

Example 3: Explicit keys and values

The example above with the goal to get certain keys and values can be realized by:

Here, I get the second item (period November) out of the array and the corresponding buy=18 value.

Example 4: Write dataframe to JSON format

If you need it the other way around, you can simply do it via:

out = df.to_json(orient=’records’)[1:-1].replace(‘},{‘, ‘} {‘)

This is often needed if you want to write your data back into systems like a Data Warehouse or Data Lake. Modern Data Warehouse technologies like Amazon’s Redshift or Google’s Big Query also often support JSON and nested data structures.

Conclusion

When working with data you can’t avoid JSON, especially when working with APIs and free data sources. Also for data integration to your data management systems it could be a good idea to use the JSON format. Many modern technologies are supporting the JSON format.

I hope this article gives you the most important information about JSON and some examples of how you can actually get started to work with this kind of data.

Sources and Further Readings

[1] JSON.org, Introducing JSON

[2] techplanet.com, JSON Example (2019)

Gain Access to Expert View — Subscribe to DDI Intel