Binning Records on a Continuous Variable with Pandas Cut and QCut | by Allison Stafford | Towards Data Science

Today, I’ll be using the “City of Seattle Wages: Comparison by Gender –Wage Progression Job Titles” data set to explore binning — aka grouping records — along a single numeric variable. Find the data here on the Seattle Open Data Portal.
A caution for binners: binning reduces granularity, and is not always helpful. Binning is not typically used in machine learning models.
A caution for binned data consumers: choice of bin edges can have a HUGE effect, especially in small samples. Watch out for people using binning to lie or mislead you. Ask questions about the reasoning, and about results from other binning options.
I use binning to group continuous data into groups for comparison. For example, I might group people into age groups. This allows for statistical comparison and the creation of streamlined visuals.
At the same time, the divisions can be somewhat arbitrary: if we group in 10 year age increments, 30-year-olds are likely more similar to 31-year-olds than the 21-year-olds that fall in their category.
I like to bin in two scenarios:
When there is a real-life or business threshold. For example, in airline miles, when you hit a certain threshold the benefits and incentives change, making logical groupings. For United Airlines: MileagePlus, Premier Silver, Premier Gold, Premier Platinum, Premier 1k.After determining a pattern, when creating visualizations tailored to my audience. In today’s example, I plotted each job’s percent female (what percent of workers with that job title are identified as female in the City report) vs. that job title’s average hourly rate (across all workers with that job level):
When there is a real-life or business threshold. For example, in airline miles, when you hit a certain threshold the benefits and incentives change, making logical groupings. For United Airlines: MileagePlus, Premier Silver, Premier Gold, Premier Platinum, Premier 1k.
After determining a pattern, when creating visualizations tailored to my audience. In today’s example, I plotted each job’s percent female (what percent of workers with that job title are identified as female in the City report) vs. that job title’s average hourly rate (across all workers with that job level):
With the best fit line, we see that as Average Hourly Rate increases, the Percent Female generally decreases at a rate of 0.9% decrease in females per dollar increase in average hourly rate. We have a p-value of 3.6 x 10^-11, therefore this trend is unlikely to be due to sampling. This is a lot to lay on some audiences.
Additionally, with so many titles having few employees, we get these horizontal lines at 0, 50, and 100% female. These stand out in a way that distracts from the takeaway. I wanted to translate this somewhat abstract scatterplot into a quickly digestible visualization. I decided to bin.
There are three main binning decisions: the number of bins, the type of bins, and the labels. I’ll walk through some considerations for each of these decisions one at a time:
Number of bins: A quick internet search can point you to a variety of rules of thumb to help you decide on a bin count. (Try this Wikipedia article.)
Type of bins:
Does your data have natural, meaningful breaks? For example, you might want to investigate differences in behavior across frequent flier classes with set airline miles boundaries. Maybe your want to use the federal retirement age and typical high school graduation to divide people into “youth”, “working age”, “retirement eligible.”Are you looking to split your records by their relative positioning? For instance, you might want to compare the top 1% of income earners to the bottom 10% or bottom 50%, as opposed to having particular income levels in mind.
Does your data have natural, meaningful breaks? For example, you might want to investigate differences in behavior across frequent flier classes with set airline miles boundaries. Maybe your want to use the federal retirement age and typical high school graduation to divide people into “youth”, “working age”, “retirement eligible.”
Are you looking to split your records by their relative positioning? For instance, you might want to compare the top 1% of income earners to the bottom 10% or bottom 50%, as opposed to having particular income levels in mind.
Labels: Again, it’s up to you — what do you want to call your groups? You can also leave them with range labels or integer labels.
Once you’ve decided these three things (or while keeping these decisions in the back of your mind), you’re ready to use cut — if you want divisions based on your independent variable — or qcut — for percent-of-sample splitting.
To group job titles into five groups based on hourly rates, with equal-x-axis-sized bins:
df['pay_grp_cut_n'] = pd.cut(df['total_avg_hrly_rate'], 5)
This adds a column ‘pay_grp_cut_n’ to df where each value is the bin range a record falls into.
Showing the Job Title Count on the y-axis creates a histogram:
We can see that the Hourly pay groups are divided into five $16/hour bins from the minimum average hourly pay ($5) to the maximum average hourly pay($86). This division results in the vast majority of job titles falling into the center and below-center groups.
I can graph the average percent female for these groups to illustrate the change in percent female across hourly pay groups:
# get sums within group - esp for employee countsgrouped = df.groupby(df['pay_grp_cut_n'], as_index=False).sum()# set x and yx = grouped['pay_grp_cut_n'].astype(str)y = grouped['no_female_empl'] / grouped['total_no_empl'] *100# create visualfig, ax = plt.subplots(figsize=(12,8))plt.xlabel('Average Hourly Rate')plt.ylabel('Percent Female')plt.title('Percent Female vs. Average Hourly Rate for Seattle City Jobs')plt.ylim(0,100)plt.bar(x, y, alpha=.5)
This visualization focuses the audience’s attention on the decline in female representation as Job Titles increase in pay. I could use these categorizations to determine the statistical significance of each of these subgroups and/or add error bars depending on the context.
Some other options to optimize cut for your needs (as always, learn more in the docs):
Adjust the labels
df[‘pay_grp_cut_l’] = pd.cut(df[‘total_avg_hrly_rate’]                             ,5                             ,labels=[‘Very Low’, ‘Low’, ‘Medium’                                       ,‘High’, ‘Very High’])
Set the bin boundaries. You might do this for the airline awards levels, or to get whole-number bin breaks.
df['pay_grp_cut_set_boundaries'] = pd.cut(df['total_avg_hrly_rate']                                          ,[5,20,35,50,65,80])
Instead of dividing based on the scalar values, QCut or “quantile cut” splits based on the number of examples falling into each bucket.
If you want five even-sized groups, ordered by hourly pay, you’d use qcut similarly to cut.
df['pay_grp_qcut_n'] = pd.qcut(df['total_avg_hrly_rate'], 5)
Using qcut with even sized groupings, you see that now our bin size varies from approximately 5 to roughly 43. Each bin represents roughly the same number of job titles.
Comparing the number of job titles in each bucket for cut and qcut highlights this difference:
With qcut, you can set the labels the same way you would forcut. df[‘pay_grp_qcut_l’] = pd.qcut(df[‘total_avg_hrly_rate’], 5, labels=[‘Very Low’, ‘Low’, ‘Medium’, ‘High’, ‘Very High’])
Instead of setting the boundaries explicitly, you can set the percent volume in each bucket to create different-sized groupings. For example, if you grade on a curve, then maybe you want to group students into (for me, arbitrary): 10% F, 20% D, 20% C, 40% B, 10% A. You could use qcut to assign grades to students:
df['letter_grade'] = pd.qcut(df,                              [.1, .2, .2, .4, .1],                              labels=['F', 'D', 'C', 'B', 'A']) 
This choice really depends on your use case. Do you have strong business logic or existing classifications that you want to match or that your audience is used to thinking about? That would be a prime reason for cut! Other times, it might make more sense to think about similar-sized groupings, or in percentiles of customers.
Whichever you decide makes sense in your context, you’ll likely want to consider the impact of the path not chosen. With cut the group sizes might be very different — highlight the magnitude of this variation(for yourself and/or your audience). With qcut, the bucket sizes might be extremely variable — forget this at your own peril.
The Fisher-Jenks algorithm finds natural boundaries in one-dimensional data, like k-means, but 1D. Check out jenkspy for a nice implementation. You’ll still need to be thoughtful about the number of bins/breaks you are looking for, but may get more logical groupings, especially if your data set has distinct breaks. For example, heights at a 4-year-old’s birthday party may naturally group into ‘adults’, ‘four-year-olds’, ‘six-year-olds’, ‘two-year-olds’, though the divisions with even cuts or qcuts may be far from these groupings.
Happy Coding! You can find the repo here.