The Quick and Easy Way to Plot Error Bars in Python Using Pandas | by Max Hilsdorf | Towards Data Science

In scientific studies, displaying error bars in your descriptive visualizations is inevitable. Holding information about the variability of your data, they are a necessary complement to your mean scores. However, scientific visualizations tend to be more beautiful on the inside than on the outside.
As data scientists, we are taught to use attractive visualizations to tell stories. Anything that distracts the viewer from the main point we are trying to make is adviced to be removed. This makes perfect sense, as managers and customers usually neither have a good grasp of statistics nor much time and energy to spend on your visualizations.
To interpret error bars, you need:
a basic understanding of descriptive statistics
information on what kind of error bar is displayed (typically standard deviation or standard error)
Both of those presuppositions conflict with the aim of a data storyteller. You want anyone to understand the results without having to read through eight lines of notes.
Data storytellers see error bars as obstacles, because they prevent easy, comfortable, and uniform interpretation of their data. Scientists, on the other hand, are trained to display all necessary information and let the reader interpret the results on their own. Neither of these approaches is superior. Oversimplifying results in technical reports or data team meetings is just as disadvantagious as annoying your managers and customers with spontaneous lectures in statistics.
For anyone willing to justify the “science” in their data science job title, learning when and how to use error bars is inevitable!
When I created the visualizations for my first scientific article, I tried lots of different methods. Let me save you some time and introduce you to the quick and easy method I found. All you need to follow this article and apply the method yourself is some basic knowledge of statistics, python and the pandas library.
There are two main parameters that are suited for error bars.
(2x) Standard DeviationStandard Error
(2x) Standard Deviation
Standard Error
I highly recommend getting familiar with these parameters, so that you can make educated decisions on which parameter to use for your visualizations. In this article by Claudia Clement, the concepts are explained in a perfectly compressed way.
First, we need to import our libraries and load our data. We’ll be using the avocado price dataset from kaggle. You don’t need to know much about the dataset. All we want to know is whether conventional and organic avocados (“type” column) have different prices ( “AveragePrice” column).
# Importsimport pandas as pdimport numpy as np # for calculating standard deviation and meanimport scipy.stats as sp # for calculating standard errorimport matplotlib.pyplot as plt # for improving our visualizations# Read dataavocado = pd.read_csv("avocado.csv")
The easiest way to perform our calculations is by using pandas df.groupby function. This function has many useful applications, but in this case, we‘ll use it for aggregated computations of statistical parameters. Below, you can see the code prototype.
df.groupby("col_to_group_by").agg([func_1, func_2, func_3])
We use df.groupby.agg for a quick and easy way to compute statistical parameters for group comparisons.
We have the dataframe and column to group by, so we need to find the right functions now. Remember, we want the mean, the standard deviation x 2, and the standard error. These are the functions we need:
NumPy
Mean: np.mean
Standard Deviation: np.std
SciPy
Standard Error: scipy.stats.sem
Because the df.groupby.agg function only takes a list of functions as an input, we can’t just use np.std * 2 to get our doubled standard deviation. However, we can just write our own function.
def double_std(array): return np.std(array) * 2
Now, let’s use the prototype code and fill the placeholders.
# df.groupby("col_to_group_by").agg([func_1, func_2, func_3])avocado_prices = avocado.groupby("type").agg([np.mean, double_std, sp.sem])
Nice! Now, let’s select only the “AveragePrice” column, because we don’t need the rest.
avocado_prices = avocado_prices["AveragePrice"]
avocado_prices.head() now gives us the following output:
Great! Now we have all the data we need.
Pandas has a really useful function that allows us get a first visualization quickly without going through the whole matplotlib procedure: df.plot
It’s basically a matplotlib representation within pandas.
To get a first plot without any error bars, we just need one line of code:
avocado_prices.plot(kind = "barh", y = "mean", legend = False, title = "Average Avocado Prices")
Output:
We can see that organic avocados have a higher mean price than conventional avocados. But is this a real difference or just random spread? As I promised in the subtitle, we don’t need a single extra line of code to answer this question using error bars. All we need to do is assign our statistical parameter to the xerr argument. Let’s start with the “double_std” parameter:
avocado_prices.plot(kind = "barh", y = "mean", legend = False, title = "Average Avocado Prices", xerr = "double_std")
Output:
There they are! Even without any deep knowledge of how to interpret these error bars, we can see that the variability in prices is rather high and that the error bars of each bar touch the other bar. Maybe the difference isn’t as clear and large as we expected from seeing just the first plot? Again, I won’t go in depth on the interpretation of standard deviation and standard error since this is a practical guide. Just note that this plot leaves us with a different impression than the previous one.
What happens if we plot the standard error next?
avocado_prices.plot(kind = "barh", y = "mean", legend = False, title = "Average Avocado Prices", xerr = "sem")
Output:
What? The error bars are hardly visible. Standard error is sensitive to sample size, as it is lower in large samples than in small samples. The avocado sample has more than 250k observations, so the results make sense. This third plot leaves as with a completely different impression again!
Whether and how you use error bars makes a huge difference in the “story” your visualization tells. It is crucial to understand the statistics behind error bars to use and interpret them correctly.
So far, we got very good results from just one line of code. If we take some more time to improve our visualization, we can get something like Figure 4.
ax = avocado_prices.plot(kind = "barh", y = "mean", legend = False, title = "Average Avocado Prices", colors = ["steelblue", "seagreen"]) # no error bars added here# Xax.set_xlabel("Price ($)")# Yax.set_ylabel("")ax.set_yticklabels(["Conventional", "Organic"])# Overallfor key, spine in ax.spines.items():    spine.set_visible(False)ax.tick_params(bottom = False, left = False)ax.errorbar(avocado_prices["mean"], avocado_prices.index, xerr = avocado_prices["double_std"],             linewidth = 1.5, color = "black", alpha = 0.4, capsize = 4)
Output:
Creating error bars in python is very easy! Using some advanced pandas functions, we can go from dataframe to a visualization with (or without) error bars in just two lines of code! However, if you haven’t learned the statistics behind error bars yet, you need to do that first. You saw how different the three plots (no error bars vs. dobuled standard deviation vs. standard error) looked. At first this might seem scary, but once you learn it, it’s another very useful tool in your data science skillset.
Thanks for reading!