Advanced Data Visualization in Python with HoloViews | by Andrew Riley | Towards Data Science

HoloViews is an open source package that produces high quality and interactive visualizations with minimal code and effort. The project page can found here here along with many examples to understand the capabilities of the package. I have been very pleased with the easy of use and performance relative to the legacy options (Matplotlib, Seaborn, etc... ).
Exploring and learning new packages is more engaging when solving a real world data analysis exercise. This article steps through a cohort analysis of a simplified SaaS business to identify whether there are any emerging trends in the customer base. Such an analysis is useful in a number of contexts including preparing and validating top-line forecasts and assessing the impact of recent changes in the product offerings or pricing. A cohort in this example will be defined as all users who appear in a given month for the first time. The goal of this exercise is to examine how the mix of new business is changing over time and what the change in mix may mean for retention. In this case the mix refers to product tier and subscription duration, in other cases it could be any categorical field available.
The data underlying the analysis are simulated to replicate a pattern observed across multiple companies.
There are two files:
mrr.csv contains the monthly revenue for each subscription over a 2.5 year period. There is no left censoring.
user.csv contains the a user id, subscription id, product tier and a subscription duration
A user may have more than one subscription and they may swap the product tier and subscription duration. For simplicity, retention is calculated on a subscription basis rather than user basis.
The analysis begins with the imports. The point of showing this is to make sure that the reader has all the dependent packages installed. Please note the hvplot package which is built on HoloViews and has among other things a convenient API for plotting with pandas, this requires a separate install from HoloViews.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport osimport hvplot.pandasimport holoviews as hvfrom holoviews import optshv.extension('bokeh')
Side note: The default behavior of datetime formatting in HoloViews is to print the date/time down to the nanosecond. Documentation for controlling this behavior is often hard to find and not entirely clear for beginners. This formatting problem can be solved by creating a DatetimeTickFormatter object and passing it to the graphs as needed.
from bokeh.models.formatters import DatetimeTickFormatterdtf = DatetimeTickFormatter( days = '%m-%Y', months = '%m-%Y' , years = '%m-%Y')
Before plotting the data it needs to be transformed into content that will make the plots meaningful and in the proper format to be passed to HoloVeiws. Some of the basic pandas manipulations are excluded from this article. The full file can be found at github.
Creating the cohorts is the first step in a cohort analysis and there are many ways one can produce this calculation. Given the format of the input data, the following steps are applied here:
### Calculate start period of the subscription## #1 Create cohorts based on the first observation of a month with a positive mrr valuecohort = (mrr > 0).idxmax( axis = 1).rename('cohort')## #2 Add the cohort date to the indextenure =  mrr.join( cohort ).set_index( 'cohort', append = True).stack()tenure = tenure.reset_index( ).rename( columns = {'level_2':'date',## #3 Calculate the number of periods (months) from the cohort date to the mrr date0:'revenue'} )tenure['periods'] =  np.round( (tenure['date'] - tenure['cohort']) / np.timedelta64(1, 'M') ,0).astype(int)
The code in #1 converts the dataframe into values into binary variables that are true when there is positive revenue in a month. Then idxmax is applied to extract the column title of the first occurrence of the maximum value, which will be the first date when revenue is positive. The result is a series with the same index as mrr and the series is appended as a secondary index in #2 to mrr, stacked (this operation unvpivots the data by turning the column headers, monthly dates in this case, into a value field) and saved as a new dataframe called tenure. Step #3 calculates the number of periods (months in this case) that have elapsed between the cohort month and the current month.
Two curves that are often useful to produce in the EDA phase of a cohort analysis that will give a general understanding of the customer retention. Both represent the percentage of business that is remaining at a given point in time relative to the first period that each customer was active. One uses customers as the numerator and denominator and the other uses revenue. These calculations are broken up into two segments as follows:
## #1 Calculate revenue and subs count by cohort by monthgb =  tenure.groupby( [ pd.Grouper( key ='cohort' , freq = 'M') , 'periods'])rev_cohorts = gb['revenue'].sum().unstack()count_cohorts = gb['sub_id'].nunique().unstack()## #2 turn them into a percentage relative to the first monthscaled_revenue_cohorts = rev_cohorts.apply( lambda x : x/x[0] ,axis = 1)scaled_count_cohorts = count_cohorts.apply( lambda x : x/x[0] ,axis = 1)
The calculations in #1 compute the total revenue and unique count of customers for a monthly cohort of customers in each period where there are observations and the unstack command pivots the periods into column headers, which will be useful for plotting with hvplot. The calculations in #2 take advantage of the new data shape and apply a lambda function to divide each period value (revenue or customer count) by the value observed in the first month.
The hvplot extension to HoloViews provides a very quick way to generate interactive visualizations from pandas objects that can help with EDA. The following code, which follows the same API format as the matplotlib, quickly produces line plots for exploration.
### Quick hvplot of both dataframesp1 = scaled_revenue_cohorts[scaled_revenue_cohorts>0].T.hvplot( figsize = [11,7], legend=False , title = 'Revenue')p2 = scaled_count_cohorts[scaled_count_cohorts>0].T.hvplot(figsize = [11,7], legend=False , title = 'Logo Count')### Add the graphs together to display as one output(p1 + p2).cols(1)
There are two things that are easier to implement than would be in HoloViews than in plotting packages. The tooltips allow one to explore the data interactively and are generated automatically. The syntax for adding the graphs together into a single visualization is easier than calling the axes objects from matplotlib.
These graphs are already very informative and help to guide the rest of the analysis. First, notice that there is a mix of different subscription lengths. It will be best to analyze the retention of the different duration options separately since obviously a one year subscription is going to have a different profile than a one month subscription. Secondly, the profile of the retention curves is changing. The shorter lines are the most recent ones since there is simply less data available. These have more curvature than the older ones and also they appear to have a lower retention percentage. Later this article will show a better way to demonstrate this pattern. Before that, we will explore what is changing in the underlying subscription base.
Retention patterns often begin to change when there is a change in the mix of new business. The mix of new business could be changing because of new marketing activities, product offerings, duration options, regional or demographic characteristics, among many other things. In this data set only duration and subscription tiers are available as categorical features. A good way to assess how the mix of any categorical feature changes over time is to plot them in a stacked two bar charts showing the total volume and the relative percentage.
In order to generate the next set of graphs the data will need to be reshaped splitting the cohorts by tier and duration.
## #1 Create product duration cohortsgb = tenure.groupby( ['sub_product','sub_duration','cohort', 'subs_id','periods'])## #2 Divide each periods user count (observations with revenue > 0) by the month 0 countcohorts_scaled = gb['revenue'].sum().unstack().groupby(level=[0,1,2]).apply( lambda x : (x>0).sum()/(x[0]>0).sum())
With the new cohorts data creating the bar charts is fairly easy but does require a few lines:
## #1 mix = cohorts[0].rename('count').reset_index()mix = mix.set_index(['cohort','sub_product','sub_duration'] )mix = mix.sort_index()## #2abs_bars = hv.Bars( mix )abs_bars.opts( stacked = True, xrotation = 45 , width = 700 , height = 400 , xaxis= None , ylabel = 'Conversions')## #3hund_bars = hv.Bars( mix/mix.groupby( level =0 ).sum() )hund_bars.opts( stacked = True, xrotation = 45 , width = 700 , height = 400 , xformatter = dtf, ylabel = 'Relative %')layout = (abs_bars + hund_bars).cols(1)layout.opts(shared_axes=False)
The code under #1 create a multi indexed series that can be passed to the HoloViews bars class. Holoviews requires flat data as opposed to hvplot which can operate smoothly with tabular data. The .opts() method in HoloViews gives the developer very detailed control over almost every part of the visualization.
The bar charts shows that the product tier has not been changing in any clear trend over time. However, repeating this step for duration shows a significant change in the mix of new business.
Six month subscriptions are being replaced entirely by one month subscriptions. This result is almost certainly causing the overall shift in the retention curve. In order to prove that out we will plot the retention curves by product and duration.
The previously shown plots are easy to produce and appealing, but limited in terms of interactivity. One of the nice features of HoloViews is being able to produce dashboard like visualizations. Additionally it produces these visualizations in html so that they may be embedded in a web page. These features require more code than simply passing a dataframe into hvplot. These features are demonstrated with two visualizations are useful for identifying retention trends.
Before producing the visualizations, building some helper functions will keep the code more readable. In the previously shown cohort curve plots, it can be a difficult to see which line belongs to which vintage. In order to make that more clear one can assign a color that fades with the cohort age. The following code block colors the lines blue and has the blue fade to white/gray with age.
def gen_cohort_plot( cohort ):    cohort = cohort[ cohort > 0]## #1 Create curves with hvplot    plot = cohort.T.hvplot( width=700, height=350, legend=False )     d_min = 2015 REPLACE    d_max = 2017 REPLACE## #2 loop through curves to adjust the formatting     for item in plot.items():        date , curve = item        year = pd.to_datetime( date ).year ## #3 interpolate the blue scale bounded between .2 and .8        c = plt.cm.Blues( .8* (year - d_min ) / ( d_max - d_min ) + .2)        curve.opts( hv.opts.Curve( color=c ))    return plot
The function looks intimidating, but the logic is pretty simple. In #1 hvplot is used to generate the same curves as shown previously. These curves are looped through in #2, while #3 linearly interpolates the cohort year cohort year relative to the minimum and maximum cohort year onto the blue color map from matplotlib.
## #1 loop through possible pairs of product and duration ## pd_curve is a helper function that just filters data before ## being passed to gen_cohort_plot()curve_dict_2D = {(d,p):pd_curve(p,d, cohorts_scaled) for p in products for d in durations }## #2 create hmap visualizations hmap = hv.HoloMap(curve_dict_2D , kdims=['Duration', 'Product'])## generate a plot of the number of conversionsconv_curve_dict_2D = {(d,p):conversion_plot( p , d , cohorts ) for p in products for d in durations }conv_hmap = hv.HoloMap(conv_curve_dict_2D, kdims=['Duration', 'Product'])## generate a plot of the average price for a cohorts first monthasp_curve_dict_2D =  {(d,p):asp_plot(p,d, cohort_asp) for p in products for d in durations }asp_hmap = hv.HoloMap(asp_curve_dict_2D, kdims=['Duration', 'Product'])(hmap +conv_hmap + asp_hmap ).cols(1)
The curves are stored in a dictionary that can be passed in different layout options. Shown here is the HoloMap. A HoloMap produces a drop downs selectors based on the dictionary keys shown:
The darker the line the more recent the cohort. After clicking through the different combinations it becomes clear that the one month subscriptions have a worsen retention profile. We’ve previously observed that one month subscriptions are making up a growing portion of the new customer base.
While coloring the the lines different shades aids in visualizing the age of the cohorts, heatmaps provide an alternative method to display the information which makes certain trends more easy to identify. Heatmaps do however require more explanation than the line plot, if it is being presented to parties who are unfamiliar with this format. The tutorial will conclude with a discussion of how to produce and interpret the heatmap view.
Helper functions are created to format the graphs and make the code more manageable.
def heatmap_product_duration( product , duration , data = cohorts_scaled):## #1 filter the data for the product duration pair    idx = pd.IndexSlice    data = data.loc[ idx[product, duration, :] , :].reset_index( level = [0,1], drop = True)            data = data[data>0]        data = data.stack().rename( 'retention').reset_index()    data = data[ data['retention'] > 0]    data.columns = ['cohort','tenure','retention']## #2 Create a heatmap    hm = hv.HeatMap(data , kdims=['tenure','cohort']).sort()## #3 Formatting options note the addition of the hover tool     hm.opts( opts.HeatMap( width = 500 , height = 500  ,colorbar = True, yformatter = dtf , xrotation = 90 ,cmap = 'RdYlGn' ,tools=['hover'], toolbar='above'))    return hmdef left_conv_bar( product , duration, data = cohorts ):## #1 filter the data for the product duration pair    idx = pd.IndexSlice    data = data.loc[ idx[product, duration, :] , :].reset_index( level = [0,1], drop = True)    data = data.loc[:,0].rename( 'conversions').reset_index() ## #2 similar to the previous bar charts, note the invert_axes option    bar = hv.Bars(  data ).opts( invert_axes=True , height = 500 , width = 200 , color = 'Green' , yaxis = None, invert_xaxis=True )    return bar
After building the helper functions, in a way similar to before, the code loops through the product duration pairs and creates a dictionary to store the visualizations before ultimately passing them to a HoloMap.
heatmap_curve_dict_2D = {(d,p):heatmap_product_duration(p,d) for p in products for d in durations }heatmap_hmap = hv.HoloMap(heatmap_curve_dict_2D, kdims=['Duration', 'Product'])left_bar_curve_dict_2D = {(d,p):left_conv_bar(p,d) for p in products for d in durations }left_bar_hmap = hv.HoloMap(left_bar_curve_dict_2D, kdims=['Duration', 'Product'])layout = (left_bar_hmap  + heatmap_hmap ).opts( shared_axes = True)layout
This view shows how the cohorts are aging differently by observing the green in early tenure periods shift to yellow after October 2018 for all tiers of the one month subscription. Note that this shift also corresponds to the period when the number of conversions begins to increase. This phenomenon is fairly common in evolving businesses. Increasing marketing spend or making product/pricing changes will attract more new customers but the incremental new customers are not necessarily the same as the customers in the previous cohorts.
HoloViews allows analysts to produce high quality visualizations that accelerate the time to generate meaningful insights with a high degree of customization. Thanks for reading this far and I hope you check out the library.