Using Data Science Skills Now: Simple networkx Graphs and Data Lineage.
By: Dawn Moyer
Tags: Data Lineage, Python, Programming, Networkx, Data Science

What we will cover

This article will discuss two separate topics. The first is data lineage — mapping a piece of data from its source to the final data product. The other topic is simple graphing with networkx. Each section is useful on its own, but I wanted to demonstrate how one can apply graphs in everyday work. You will see that a graph is a tool that can validate the completeness of data lineage. Once you understand the inputs, you can apply the technique to other concepts, such as impact analysis or root cause analysis. So let’s go.

We all need good Data Lineage.

Much is available out there on the need for good data lineage. I included a couple of resources at the end of this article. Whether or not data lineage is of particular interest to you is not so important as that it will affect you in some way. When someone on your team cries out, “I can’t find this @! data!” or “The dashboard isn’t working — what’s broken?” it all comes back to data lineage. GDPR has only amplified the need with the requirement to be able to identify all points of personal data within a system.

Ideally, data lineage should be automated because it is a large endeavor, prone to error, and constantly shifting and moving. Many great tools are out there to help organizations get their data mapped and programmatically updated. The full process of onboarding such a tool might take years, so you may have to take on parts of the mapping yourself OR create manual mappings for processes the tools can’t access. Yes, the python script on your laptop that drops a file onto your documents folder, I’m looking at you!

Sometimes you need to manually create some project-level data lineage for another team. Perhaps this team is migrating the process to a new platform.

Let’s walk through an example of creating a data lineage graph for a sample data science project.

Scenario: Move it ALL to the ‘cloud.’

Last year your client requested a dashboard with data analysis. The process was fully automated. Python scripts run daily and update the final .csv file that generates the dashboard. It runs smoothly, and everyone is happy. Happy, until a company directive is in place to move all dashboard processes to the cloud. Could you kindly provide the migration team with a list of all of the data elements required to be moved to the cloud to ensure the dashboard works as intended? Does a request like this sound familiar?

In your first planning meeting, you might provide a visualization such as the one below. You talk about the high-level data movement.

image created by the author

In response, the data engineer asks for a list of all of the individual data points used in the dashboard. All of the ‘gold copy’ data needs to move to AWS as well as the python scripts. Thinking about this, you realize most of the data features used in the dashboard don’t actually ‘exist’ anywhere in the databases. Most are created and formatted, and massaged between the databases and the dashboard. There is no ‘gold copy’ with a 1:1 mapping. This won’t be as simple as just providing a list of the final data points.

The Ask: Provide all of the data points and transformations in your process

In an ideal world, your company would have spent the past couple of years updating an enterprise tool, such as Collibra, with all data lineage information. A simple query, and you would have the results. But alas, you are going to have to build this from scratch.

I created an example scenario. A Tableau Public dashboard visualizes some customer data. While to dashboard itself reads in just one file, the ‘gold copy’ data comes from two different databases. Let’s walk through it.