Visualizing In-App User Journey Using Sankey Diagrams In Python | by Nicolas Esnis | Towards Data Science

TL;DR: Sankey diagrams can display multiple types of data over two dimensions, which makes them a great tool to visualize your application or website’s user journey. This article will get you started to build your own Sankey chart in Python (find the code in this gist).
Staring at a chart wondering what you’re supposed to get out of it can be frustrating. The more appropriate the visualization is, in regard to the data type, the easier it is for the audience to understand the story behind the numbers. And the richer is the information that can be inferred.
As a data analyst at Adikteev, the leading mobile advertising company in gaming apps, I had been looking for a way to display our customers’ in-app user journey, and Sankey diagrams turn out to be a great data visualization to achieve that:
Like any organizational or hierarchy chart, Sankey diagrams can represent an order relation between a set of nodes. And what is a user journey, if not the relations between different steps or phases in a chronological order?
The width of the links between each node of a Sankey diagram being proportional to the flow rate, the viewer can get an idea of the proportion of users who completed a specific journey at a glance.
Sankey diagrams can display different data types over two dimensions: the number of nodes and the links’ width. In the case of user journeys, the number of nodes can relay the in-app events quantity and chronological order information, and the width of the links can display the proportion of users who moved from a specific in-app event to another.
More generally, Sankey diagrams can be used to chart the flow of a website traffic, the energy flows into a factory... Or even the map of Napoleon’s Russian campaign:
When it comes to user journey analysis, these diagrams allow to identify at a glance what are the most frequently fired events, in what order, or what are the different paths from action A to action B. These are information that marketers, decision-makers or your customers are likely to be interested in.
But when building a Sankey diagram with three or more nodes levels, things can get messy. You wouldn’t want to spend hours drawing points and linking them manually, even more if you need to analyze multiple journeys or update the analysis regularly. Python comes really handy when it comes to process big datasets, and two Python libraries can help us here:
Pandas: one of the most famous and most used Python library for data analysis. If you’re not familiar with Pandas but have some notions of SQL, I recommend this great article.
Plotly: a Python graphing library allowing to create interactive, publication-quality graphs.
The rest of this article is a step-by-step guide to making your own user journey analysis in Python using Pandas DataFrames and Plotly’s Sankey graphing object.
Still here? Great!
We’ll plot the user journey of a very simple app. For the sake of clarity, this app contains only 4 types of events: installs, opens, signup and purchase.
This is what it looks like when truncating at the 4th in-app event of the user journey post-install:
If you try to hover the links and the nodes, you’ll see you can get more detailed information about the incoming and outgoing flows, and about the average time from one event to another.
To complete this workshop, you’ll need:
Python 3.6 installed;
The Plotly python package: we’ll use plotly.graph_objects to create the chart;
Pandas: we’ll use Pandas’ DataFrames attributes and some common Pandas functions;
Optionally, you can install Seaborn and Plotly’s chart_studio packages. chart_studio will let you publish your chart, and Seaborn will allow you to generate cool color palettes to custom your chart.
And you’ll need some data. You’ll find here a set of raw data to get you started. The head of this DataFrame looks like this:
We’ll inject the following information into the chart:
Every in-app events of the user journey: the nodes of the chart will stand for all the in-app events, from install to the Nth event. Here, we’ll go up to the 10th event.
The funnel of events of every user: each time we see that a user completed an in-app event, we’ll link this event to his previous event. We’ll look only at installers’ data, so that the 1st event of every user is an install event. The width of the links will increase as we observe more users completing the same sequence of in-app events.
Average time between two events: we’ll also compute the time between every event for each user, to be able to compute the average time between each step of the funnel.
The first step of this tutorial is preparing and cleaning the dataset:
Remove the time_install column and add dummy install events in the event_name and time_event columns;
Compute the rank of each event at the user level in a new rank_action column;
Add the next_action and time_to_next action columns;
In this tutorial, we’ll keep only rows up to the 10th event per user — but feel free to keep more events if you want to!
You should now have a new DataFrame looking like this:
Plotly’s Sankey data structure is made of two Python dictionaries: node and link. The link dict takes four Python lists as parameters:
source: a list of each flow’s source node. To have a source per event flow, we’ll need to append a unique index per event and per event rank to this list.
target: a list of target nodes. We’ll map every source event to its target event, using the next_event column.
value: a list containing each flow’s volume information. We’ll keep count of every time we map a source to a target and append this count to the value list.
label: a list of metadata that will be displayed when hovering each link. Here, we’ll want to know the number of users going from a source event to a target, and the average time it took them to do so.
The node dict takes the two following parameters:
label: a list containing the nodes’ names (in our case, the events’ names);
color: an optional list containing the nodes’ color information.
Have a look at Plotly’s documentation for more insights about each parameter.
First, we’ll create a unique source_index for each event at each step of the user journey, keeping track of every event’s name. A Python dict can achieve that and will look like this:
1: {    'sources': ['install'],     'color': [(80, 190, 151)],    'sources_index': [0]  },2: {    'sources': ['signup', 'purchase', 'reopen'],    'color': [(228, 101, 92), (191, 214, 222), (252, 200, 101)],    'sources_index': [1, 2, 3]  },3: {    'sources': ['reopen', 'purchase', 'signup'],    'color': [(252, 200, 101), (191, 214, 222), (228, 101, 92)],    'sources_index': [4, 5, 6]  }}
The first level keys are the rank of each event. Rank #1 only contains install as a source, since we only look at new installers' data.
We store a unique source_index rather than the events’ names, since the events can occur multiple times within the journey. We also attribute a unique color to each event, and name this dict nodes_dict.
Here’s the script to achieve this:
Then, we need to count, for each step of the funnel (i.e. each rank_event), how many times each user has gone from a source to a target, and keep track of how long it took, using our time_to_next column.
Here again, a Python dict will do the job — we’ll name it links_dict this time. The 1st level keys will be our source indices, and the 2nd level keys the target indices of each flow. The target indices of rank N are retrieved from the source indices of rank N+1:
{  0: { # source index    1: { # target index      'unique_users': 14991,      'avg_time_to_next': numpy.timedelta64(192..., 'ns')    },    2: {      'unique_users': 3879,      'avg_time_to_next': numpy.timedelta64(393..., 'ns')    },    ...  },  1: {    4: {      'unique_users': 3158,      'avg_time_to_next': numpy.timedelta64(495..., 'ns')    },    5: {      'unique_users': 4843,      'avg_time_to_next': numpy.timedelta64(303..., 'ns')    },    ...  },  2: {    5: {      'unique_users': 1426,      'avg_time_to_next': numpy.timedelta64(14524260000000000, 'ns')    },      ...  },  3: {...}
For every user’s sequence of events, we’ll need to:
1) Read in nodes_dict the unique source_index of every event in the sequence.
2) Likewise, read the source_index of each event’s next event (the target indices of rank N are retrieved from the source indices of rank N+1) and store it into a target_index variable.
3) Check if the combination of the source_index and the target_index is already a key of links_dict. If not, we’ll create it. If it is, we’ll increment the count of unique users, and add the time_to_next information. Later, by dividing the time_to_next by the count of unique users, we’ll have the average time from an event to another.
The script looks like this:
Before being able to plot our Sankey diagram, we’ll need to create the targets, sources, values, labels and colors lists from our dictionaries, that will be passed as parameters in the plotting function. This can be achieved easily by iterating over our nodes_dict and links_dict:
All clear! Everything is now ready and we just have to plot the figure. We can add some styling to the nodes with the line and thickness parameters, and make use of hovertemplate to rephrase the hovering metadata and fit our needs. The update_layout Plotly function allows us to play with the chart’s size, title text, font size and background color.
If you want to publish it to the web, you’ll need to create a Plotly account and setup your credentials. You can also render your chart in your notebook or Chrome if you want to keep this offline.
Follow me on Medium if you liked this story! Thanks for reading.