Visualize and communicate uncertainties clearly with Python and Plotly | by JP Hwang | Towards Data Science

In this article, I want to show how easy it is to effectively visualise and communicate uncertainties and ranges. More importantly, I would like to demonstrate how helpful doing so can be in gaining an intuitive understanding of data and subsequent decision-making.
Although most of us are aware of uncertainties in every aspect of life, it is not often that we get to see, or work with them. Information or numbers are typically given in averages (means, or medians if we are lucky). Many of us don’t ever get taught statistics and probabilities, and the few that do are mostly taught them in just awful, tedious, academic ways that prioritise formulae over intuitive understandings. As a result, even if you had a good understanding of uncertainties in data, communicating it to most audiences can be tricky. (Try quoting a standard deviation figure in a meeting and count the blank stares that come back.)
Data visualisation can help like no other can in understanding and communicating information related to uncertainties, probabilities and distributions. So, let’s take a look at how uncertainties can be visualised and how to use that data.
For reasons that will become clearer, I will use the framework of basketball, and fantasy basketball in particular, for this article. As always, the focus is on the data and analysis methods, not the sport-specific analysis outputs; so you should be fine even if you don’t care about basketball.
I include the code and data in my GitLab repo here, so you should be able to easily follow along by downloading/cloning the repo if you wish.
I assume you’re familiar with python. Even if you’re relatively new, this tutorial shouldn’t be too tricky, though.
You’ll need pandas and plotly. Install each (in your virtual environment) with a simple pip install [PACKAGE_NAME].
You can use any graphics packages, obviously, but I personally prefer the simplicity and power you get with Plotly.
Imagine that through some random turn of events, someone has asked you to put together a fantasy basketball team.
(You might ask — what is fantasy sports? Simply, in fantasy sports, players compete by forming a team from a pool of all available players in a real league, and these players are assigned points based on their real-life performance.)
How would you go about it? One way would be to simply pick the best available players who expect to produce the most points for your team.
I have prepared a csv file based on the 2018–2019 NBA season. The data comes from basketball-reference.com, and it was obtained using the basketball_reference_web_scraper library. I have pre-processed it to only include players who played over 1000 minutes, and who had not been traded mid-season.
Let’s load the data with:
season_tot_df = pd.read_csv('srcdata/bballref_1819_season_tots_1000plus_mins.csv', index_col=0)
And take a look at it:
season_tot_df.head()
The data includes all of the original columns from the site and a handy fan_ppg column which I created based on this formula.
What does the data look like? A bar graph of points per game can be plotted as follows:
import plotly.express as pxfig = px.bar(season_tot_df, y='fan_ppg', x='name')fig.show()
And with some formatting applied (for brevity, I will not show all my formatting code here, please see the repo if you are interested), we get:
From James Harden to Bruce Brown, this chart plots all 254 players’ fantasy points. So, when it comes to your turn, would it always be best to simply choose the player with the next highest points per game?
I personally don’t think it’s that simple. Here’s why.
The bar chart above plots averages — more specifically, the ‘mean’, which is the total points divided by the number of games. But, as anyone who’s watched JR Smith or Nick Young know, not all averages are made the same.
Some players are just more consistent than others, while others have inconsistencies that will drive you bonkers with their night to night variations.
As an example, let’s compare Khris Middleton with Eric Bledsoe. Both were incidentally on the same team, and have produced nearly identical season average fantasy points:
>>> print(season_tot_df[season_tot_df['name'].isin(['Eric Bledsoe', 'Khris Middleton'])].fan_ppg)45    31.90519549    31.351282
But they do so in wildly different manners, as you will see. To do this, let’s load up our data which includes individual game stats for each of our 254 players:
with open('srcdata/bballref_1819_pl_box.pickle', 'rb') as f:    pl_data_dict = pickle.load(f)
The data is in a dictionary form, where the key value is the slug of a player (identifying string for the player as used by basketball-reference).
Before doing anything, we need to collect Middleton and Bledsoe’s data with this information into one dataframe:
Here, I collect the ‘slugs’ for the two players and use them for the dictionary data for each player. Each player’s data is in a list of dictionaries. They are easily converted into dataframes (temp_df), themselves added to a new list (temp_df_list) and then joined at the matching columns to produce one dataframe (comp_df).
And plotting the data as a bar plot, we see:
fig = px.bar(comp_df, y='fan_pts', color='player', facet_col='player', labels={'fan_pts': 'Fantasy Points', 'date': 'Date'})fig.update_layout(title='Fantasy performance comparison')fig.show()
The entire set from the low end to the upper end in each chart might be referred to as a ‘range of outcomes’. That is, they represent the set of outcomes (I mean, they don’t exactly, but it’s representative, and let’s leave it at that for now).
From these bar charts, it looks like Bledsoe’s data might be slightly more uneven than Middleton. There are more high bars, and also some glaring gaps (low bars). It’s hard to tell for certain from these figures, though. That’s where histograms can help. Histograms plot the spread of data points, where the y-values are counts of values that fit into each x ‘bin’.
fig = px.histogram(    comp_df, x='fan_pts', facet_row='player', color='player',    labels={'fan_pts': 'Fantasy Points', 'count': 'Count'}, nbins=30)fig.update_layout(title='Fantasy performance comparison')fig.show()
That looks like a resounding yes.
These histograms tell us that Bledsoe is clearly more inconsistent than Middleton here for whatever reason. He has higher-scoring games more often, but also lower-scoring games more often. As a result, even though they have the same average, the chance of getting that average score changes drastically with Bledsoe against that for Middleton.
Still, it’s not easy to see here just how much more likely it is for Bledsoe to score 40+ fantasy points vs Middleton. Also, what happens if we want to compare more than two players at a time?
There are a number of tools that make it easy to see distributions in data. Box plot might be one — where it plots distribution as ‘boxes’, indicating percentile breakdowns of datasets. Try this:
fig = px.box(comp_df, x='player', y='fan_pts', color='player', labels={'fan_pts': 'Fantasy Points', 'date': 'Date'})fig.update_layout(title='Fantasy performance comparison')fig.show()
The middle lines indicate ‘median’ values, indicating the most likely outcomes, and the box indicates where half of all outcomes fall into, and the lines where most of the outcomes are expected. The dots indicate ‘outliers’ which is statistic-speak for unusual outcomes.
Plotly also allows the underlying data to be plotted in point form — so let’s do that for further clarity and illustration:
Clearly, having Bledsoe in your team is a high-risk, high-reward strategy.
Violin plot is also a good tool that can give you a closer look at the distribution of data:
fig = px.violin(comp_df, x='player', y='fan_pts', color='player', labels={'fan_pts': 'Fantasy Points', 'date': 'Date'}, points="all")fig.update_layout(title='Fantasy performance comparison')fig.show()
Instead of the simplicity of box plots, where quartiles are marked as lines, violin plots essentially compare histograms, where width of the plot represents how often those values are likely to occur.
This is also a limitation of violin plots. Because width is crucial in deciphering a violin plot, they are not as useful when many data points are shown. As each plot is given less and less width, subtle differences in them becomes more difficult.
Okay, that was probably enough on that.
Let’s go back to our initial question, which was — how could this data help us in choosing players.
Let’s say that we have players left whose average fantasy points production is between 35 and 40, any of whom we could get into our team. Who do we choose? As we have done before, we can put them into histograms, or box plots:
This code is very similar to what we did above for Middleton and Bledsoe, except now we are collecting slugs of players by the fan_ppg value.
We can see here that as the data points increase, the compactness of box plots help it come into its own. Comparing even more players:
Someone like Rudy Gobert, Chris Paul, or Khris Middleton (marked in orange) are what we would call ‘low variance’ players. You would be more certain about how they would perform day-to-day. On the other hand, players like John Wall, D’Angelo Russell, or Victor Oladipo (marked in blue) are much more variable.
So the question becomes — what is more important to you? Maybe you are playing to get into the top few percent of your league, in which case you might want the high variance players.
Or, if your league rewards consistent performance rather than occasional flash performance, the steady players might be better bets. In real life, this may be more like what you might want. Can you imagine having a team of players and never having any idea what you’re getting from them on any given night?
How do we find one group or the other?
One way to sort the player data to identify relative consistency as such is to use measures called ‘standard deviation’, or ‘variance’. Without getting too mathematical, they basically measure how spread out a distribution is. Eric Bledsoe would have a larger standard deviation than Middleton does in the above example.
And to go a step further, we can identify how players of similar average (mean) performance might vary in terms of potential output, by plotting both as a scatter plot as below.
Let’s go through the code briefly. Once again, slugs are collected, except that here the loop is simply over all slug values in the season total dataframe (i.e. every player).
Then a mean and a standard deviation are calculated for each player, based on which a new dataframe is constructed. Lastly, I produce a ratio of standard deviation to the mean, which is used as the colour variable.
(The numbers are rounded for simplicity in display purposes.)
(I have put up an interactive version is here)
The graph shows a good general correlation between standard deviation and the mean, with a large spread. That is, at the same mean values, a large range of standard deviation values exist.
As we discussed previously, there are situations where identifying data points who might have occasional high-output performances are beneficial. Fantasy sports against many players is definitely one of them, where outsized benefits exist for placing towards the top of a large group, than being in the middle of the pack.
The large spread here represents significant choice for the person choosing a player, to optimise their selection towards their goal.
Take a look at the same graph, with some players’ data highlighted.
Karl-Anthony Towns and Paul George in this plot have almost identical means, yet KAT’s standard deviation score is much higher.
On the other hand, Jamal Crawford’s standard deviation is higher than George’s despite having a much lower mean. If a player’s “price” in the fantasy league was based solely on their mean production, Crawford might present a good value opportunity in helping to construct a high-ceiling team.
This is a personal preference, but I will make one note on presenting statistical data. Depending on my audience, I often prefer plain language if possible, rather than jargon. So I might re-label ‘standard deviation’ to something like ‘variability’.
Personally, I found that jargon like ‘mean’ and ‘standard deviation’ often distract (and worse, alienate) the audience sometimes. The statistical significance of a ‘standard deviation’ means nothing in this context. We are simply looking for data points with increased variability, so why not label as such? Take a look at this plot:
I think that this conveys the message better than the one above, where the audience is often trying to remember what a standard deviation is — rather than on the substance of the message.
Of course, for a specialist or a stat-friendly audience, go full throttle on the jargon 😉.
I would like to also talk about the independence of uncertainties. That is when uncertainties stack linearly (and when they don’t, exactly).
An intuitive way to think about it is this. Imagine that two variables with some given uncertainties are being analysed. Would one of them being high, or low, or affect the other?
If it did, the two would be dependent. Practically, imagine that your entire fantasy basketball team included players from the same teams only. This would preclude many of them having career nights, as one performing well would likely mean reduced opportunities for the others.
We can take a look at an example. Let’s compare James Harden’s data with a teammate’s (Chris Paul). I include the code, where we repeat the same thing we did to compare Middleton & Bledsoe’s data. Then, we go on to combine the data into one dataframe so that we can point to them for a scatter plot.
And the resulting plot is below:
The plot shows a classic, L-shaped inverse correlation between the two people’s scores. What does the data look like if we replace CP3 with a similar player who’s not on the same team? Like this, actually:
It is more or less a random distribution. Note also that the highest combined points are higher for the lower plot — as one person’s high score does not preclude the other from doing so.
In other words, correlation is important when we deal with uncertainties of multiple variables, and when we might be looking at data which combines them.
Ok, let’s wrap it up there.
As you might have seen, uncertainties, or ranges, are quite easy to visualise, and a good understanding of them is crucial to be able to make the best of the variability inherent in the world.
It’s not the easiest thing to do, but I personally find that sometimes seeing all of the data (as in our scatter plots), and comparing the distributions (through box plots, histograms or violins) really help me to gain a better understanding of them.
This article uses an example based on sport, but it is applicable to many other domains also. I’d love to hear about the types of uncertainties in your domain.
In one of my upcoming articles, I would like to talk about forecasting (making predictions) and visualising outputs from them. Until then, take it easy!
If you liked this, say 👋 / follow on twitter, or follow for updates. I also wrote this article about my favourite data visualisation books, if you haven’t read it before: