How to write ETL operations in Python | by hotglue | Towards Data Science

In this article, you’ll learn how to work with Excel/CSV files in a Python environment to clean and transform raw data into a more ingestible format. This is typically useful for data integration.
This example will touch on many common ETL operations such as filter, reduce, explode, and flatten.
The code for these examples is available publicly on GitHub here, along with descriptions that mirror the information I’ll walk you through.
These samples rely on two open source Python packages:
pandas: a widely used open source data analysis and manipulation tool. More info on their site and PyPi.
gluestick: a small open source Python package containing util functions for ETL maintained by the hotglue team. More info on PyPi and GitHub.
Without further ado, let’s dive in!
This example leverages sample Quickbooks data from the Quickbooks Sandbox environment, and was initially created in a hotglue environment — a light-weight data integration tool for startups.
Feel free to follow along with the Jupyter Notebook on GitHub below!
github.com
Let’s start by reading the data.
This example is built on a hotglue environment with data coming from Quickbooks. In hotglue, the data is placed in the local sync-output folder in a CSV format. We will use the gluestick package to read the raw data in the input folder into a dictionary of pandas dataframes using the read_csv_folder function.
By specifying index_cols={'Invoice': 'DocNumber'} the Invoices dataframe will use the DocNumber column as an index. By specifying converters, we can use ast to parse the JSON data in the Line and CustomField columns.
Let’s take a look at what data we’re working with. For simplicity, I’ve selected the columns I’d like to work with and saved it to input_df. Typically in hotglue you can configure this using a field map, but I've done it manually here.
Let’s clean up the data by renaming the columns to more readable names.
CustomerRef__value -> CustomerIdCustomerRef__name -> CustomerMetaData_LastUpdatedTime -> LastUpdatedMetaData_CreateTime -> CreatedOnCurrencyRef__name -> CurrencyCurrencyRef__value -> CurrencyCode
The Line column is actually a serialized JSON object provided by Quickbooks with several useful elements in it. We'll need to start by flattening the JSON and then exploding into unique columns so we can work with the data.
Again, we’ll use the gluestick package to accomplish this. The explode_json_to_rows function handles the flattening and exploding in one step. To avoid exploding too many levels of this object, we'll specify max_level=1
Here is a snippet from one to give you an idea.
[{    'Id': '1',    'LineNum': '1',    'Amount': 275.0,    'DetailType': 'SalesItemLineDetail',    'SalesItemLineDetail': {        'ItemRef': {            'value': '5',            'name': 'Rock Fountain'        },        'ItemAccountRef': {            'value': '79',            'name': 'Sales of Product Income'        },        'TaxCodeRef': {            'value': 'TAX',            'name': None        }    },    'SubTotalLineDetail': None,    'DiscountLineDetail': None}]
For our purposes, we only want to work with rows with a Line.DetailType of SalesItemLineDetail (we dont need sub-total lines). This is a common ETL operation known as filtering and is accomplished easily with pandas
Look at some of the entries from the Line column we exploded. You'll notice they are name value pairs in JSON.
Let’s use gluestick again to explode these into new columns via the json_tuple_to_cols function. We'll need to specify lookup_keys - in our case, the key_prop=name and value_prop=value
Take a look at the CustomField column. Below is an example of an entry
[{'DefinitionId': '1', 'Name': 'Crew #', 'Type': 'StringType', 'StringValue': '102'}]
You can see this is JSON encoded data, specifying one custom field: Crew # with value 102
To explode this, we’ll need to reduce this as we only care about the Name and StringValue. We can use gluestick's explode_json_to_cols function with an array_to_dict_reducer to accomplish this.
Our final data looks something like below. In this sample, we went through several basic ETL operations using a real world example all with basic Python tools.
Feel free to check out the open source hotglue recipes for more samples in the future. Thanks for reading!