Data Scientists: STOP Randomly Binning Histograms
By: Bryce Macher
Tags: Data Visualization, Data Science, Statistics

A histogram with 10 bins. Because why not 10? It’s what my data viz instructor told me to do…

Histograms are a crucial part of Exploratory Data Analysis. They save us from being tricked by something like “On average, our widgets reach 1000 people.” That’s a great average! But what’s the range? Distribution? Standard deviation? Variance? These can all be answered through one simple visualization: a histogram.

But there’s a problem with histograms as they’re typically made: Binning. Time for a PSA:

Stop. Randomly. Binning. Histograms. There’s a way to do it. With math.

Let’s take a leap into being better, more rigorous communicators and think about binning our histograms to realistically reflect the quantitative landscape. Below, I start with the concept of histograms and how they’re important. Then I jump into some code and math. Feel free to jump down there if you already know your way around ol’ histy.

What’s in a Histogram?

Histograms answer the first necessary question about quantitative data that stakeholders — customers and colleagues — want to know: what do we have here? Histograms can visualize everything from the power range of Pokemon to the monthly revenue of hotel brands in the month of July.

Let’s assume we have a few hotels, and we want to see how much money they made this month. In millions of USD, our numbers are:

4, 35, 35, 37, 38, 40, 41, 42, 100000

A histogram can visualize some important details that get obscured by pure numbers. Means and averages (11,141.3 thousands USD) are sensitive to outliers. Medians don’t tell you about distribution (38). The range just shows the outliers (4–100000). Bring up standard deviation (#), variance (#), or IQR (# — #), and you’ll have stakeholders daydreaming about the beach. And don’t get me started about how we show them kurtosis and skew…

What’s a data scientist to do? BAM — Histogram time! Let’s see our data!

The Subtle Art of Binning

We’re going to represent a range of values (X axis) and their frequency (y axis) in our data (sorted by X), which will tell us how our data will behave when modeled in various ways. I’ll use the pokemon data set for this example.

Here, I’ve done the typical histogram work.

Ooh, a histogram. With auto-binning.

By default, matplotlib (plt here, and the standard for viz in python) doesn’t bin by anything other than intuition. There are some dangers here…

- Bin too widely, and you won’t be able to detect unusual kurtosis/spikes

- Bin too thinly, and there are a lot of spikes by coincidence

So how do we get it right?

By choosing a mathematically sound binning strategy!

What are your options? How do you choose? Check out my new table of binning! # of bins tells you how many bins there should be, while Width tells you how to calculate the range that the bin should encompass.

Python implementations

One of my favorite exercises with new data scientists is to write out equations in python as functions — it’s a great practice to get into, since most applied/professional/practicing data scientists will be implementing work that isn’t necessarily part of a pre-established package.

For example, if we wanted to calculate bins using Scott’s rule, we would do it as follows:

bins = int(round(

(df.atk.max() - df.atk.min())*

(df.shape[0]**(1/3))/

(3.49 * df.atk.std()

))) plt.hist(df.atk, bins = bins)

What the above returns. Looks a bit different from the default, eh? No random spike here.

OR — we can use matplotlib’s binning choices (only available for some binning methods)

plt.hist(df.atk, bins = "scott")

Where the Scott rule is built into matplotlib (and other viz packages, or you can always use numpy to calculate and assign it to the bin number in your viz package), you’ll need to implement your own Bayesian blocks binning, which gives you bins of uneven widths that better reflect the distribution of data sans a priori assumptions. You can read a great paper on Bayesian Block binning from our friends at NASA here to see why this is a useful method. Below, I just show you the difference based on a custom function (that was too messy to include).

#scotts binned histogram

H1 = plt.hist(

df.atk,

bins = 'scott',

histtype = 'stepfilled',

alpha = .2,

density = True) #blocks binned histogram

H2 = plt.hist(

df.atk,

bins = bayesian_blocks(x),

color = 'black',

histtype = 'step',

density = True)

Here, you see that there is no real ‘even’ distribution of attack, but that there is high sporadicism in the ways pokemon are assigned their power.

Conclusion

Small tangent: our binning problem is really just a symptom to a bigger problem of making data viz an afterthought divorced from the mathematical exploration practice. Data scientists and viz engineers alike should be better advocates for the math behind the psychology behind the art behind data visualization.

The goal of this post has been to make headway on this problem in one little corner of the universe: histograms. Creating a histogram requires an eye for data, but also an understanding how math is a story about that data embedded in the visualization. Please, stop randomly binning and use the math.