All Pandas qcut() you should know for binning numerical data based on sample quantiles | by B. Chen | Towards Data Science

Numerical data is common in data analysis. Often you have numerical data that is continuous, very large scales, or highly skewed. Sometimes, it can be easier to bin those data into discrete intervals. This is very helpful to perform descriptive statistics when values are divided into meaningful categories. For example, age group instead of the exact age, weight class instead of the exact weight, grade level instead of the exact score.
Pandas has 2 built-in functions cut() and qcut() for transforming numerical data into categorical data.
cut() bins data into discrete intervals based on bin edges
qcut() bins data into discrete intervals based on sample quantiles
In the previous article, we have introduced the cut() function.
towardsdatascience.com
In this article, you’ll learn how to use qcut() to bin numerical data based on sample quantiles. This article is structured as follows:
Discretizing into equal-sized bucketsDiscretizing into buckets with a list of quantilesAdding custom labelsReturning bins with retbins=TrueConfiguring the bin precision
Discretizing into equal-sized buckets
Discretizing into buckets with a list of quantiles
Adding custom labels
Returning bins with retbins=True
Configuring the bin precision
Please check out Notebook for source code.
Visit Github Repo for other tutorials
Similar to Pandas cut(), the simplest usage of qcut() must have a column and an integer as input. It discretizes values into equal-sized buckets:
df = pd.DataFrame({   'age': [2, 67, 40, 32, 4, 15, 82, 99, 26, 30, 50, 78]})df['age_group'] = pd.qcut(df['age'], 3)
Let’s count that how many values fall into each bin.
df['age_group'].value_counts()(1.999, 28.667]     4(28.667, 55.667]    4(55.667, 99.0]      4Name: age_group, dtype: int64
We can see bins have been chosen so that the result has the same number of records in each bin (Known as equal-sized buckets).
In addition, you may notice those interval values are having a round bracket at the start and a square bracket at the end, for example (1.999, 28.667]. It basically means any value on the side of the round bracket is not included in the interval and any value on the side of the square bracket is included (It is known as open and closed intervals in Math).
Now, let’s take a look at the new column age_group.
It shows dtype: category with 3 label values: (1.999, 28.667] , (28.667, 55.667] , and (55.667, 99.0]. Those label values are ordered as indicated with the symbol <. Let’s sort the DataFrame by the column age_group:
df.sort_values('age_group')
One of the problems with the previous approach is that the result always has the same number of records in each bin. In another word, each bin has the same proportion/percentage. For example:
pd.qcut(df['age'], 2) Result: |<- 50% ->|<- 50% ->|pd.qcut(df['age'], 3) Result: |<- 33.3% ->|<- 33.3% ->|<- 33.3% ->|pd.qcut(df['age'], 4)Result: |<- 25% ->|<- 25% ->|<- 25% ->|<- 25% ->|pd.qcut(df['age'], 5)Result: |<- 20% ->|<- 20% ->|<- 20% ->|<- 20% ->|<- 20% ->|
In some cases, you may want to explicitly choose the proportions other than equal-sized buckets, for instance:
10%, 40%, 50%
To do that, we can pass a list of quantiles to the 2nd argument.
df['age_group'] = pd.qcut(df['age'], [0, .1, .5, 1])df.sort_values('age_group')
Note that 4 values [0, .1, .5, 1] are used to produce the 3 proportions/percentages 10%, 40%, and 50%
To verify the result, let’s sort and count the number of values in each bin as follows:
df.sort_values('age_group')['age_group'].value_counts()(36.0, 99.0]    6(5.1, 36.0]     4(1.999, 5.1]    2Name: age_group, dtype: int64
It is more descriptive to label these age_group values as ‘Millennial’, ‘Gen X’, ‘Boomer’, ‘Greatest’. To do that, we can simply pass those values in a list to the argument labels
labels=['Millennial', 'Gen X', 'Boomer', 'Greatest']df['age_group'] = pd.qcut(df['age'], [0, .1, 0.3, .6, 1], labels=labels)
Now, when we looking into the column, it shows the label instead
df['age_group']0     Millennial1       Greatest2         Boomer3         Boomer4     Millennial5          Gen X6       Greatest7       Greatest8          Gen X9         Boomer10      Greatest11      GreatestName: age_group, dtype: categoryCategories (4, object): ['Millennial' < 'Gen X' < 'Boomer' < 'Greatest']
Similarly, it’s showing label when sorting and counting
df.sort_values('age_group')
df['age_group'].value_counts().sort_index()Millennial    2Gen X         2Boomer        3Greatest      5Name: age_group, dtype: int64
There is an argument called retbin to return the bins. If it’s set to True, the result will return the bins. It’s useful when the 2nd argument q is passed as a single number value.
result, bins = pd.qcut(    df['age'],     5,                  # A single number value    retbins=True)# Print out bins valuebinsarray([ 2. , 17.2, 30.8, 46. , 75.8, 99. ])
You may notice all the bin interval values we have made so far have some decimal points, for example pd.qcut(df['age'], 3)
pd.qcut(df['age'], 3)0      (1.999, 28.667]1       (55.667, 99.0]2     (28.667, 55.667]3     (28.667, 55.667]4      (1.999, 28.667]5      (1.999, 28.667]6       (55.667, 99.0]7       (55.667, 99.0]8      (1.999, 28.667]9     (28.667, 55.667]10    (28.667, 55.667]11      (55.667, 99.0]Name: age, dtype: categoryCategories (3, interval[float64]): [(1.999, 28.667] < (28.667, 55.667] < (55.667, 99.0]]
Generally, the age group interval doesn’t use decimal places. Luckily, there is an argument called precision to define how many decimal points to use while calculating bin precision. We can set the precision to 0 to avoid any decimal place.
pd.qcut(df['age'], 3, precision=0)0      (1.0, 29.0]1     (56.0, 99.0]2     (29.0, 56.0]3     (29.0, 56.0]4      (1.0, 29.0]5      (1.0, 29.0]6     (56.0, 99.0]7     (56.0, 99.0]8      (1.0, 29.0]9     (29.0, 56.0]10    (29.0, 56.0]11    (56.0, 99.0]Name: age, dtype: categoryCategories (3, interval[float64]): [(1.0, 29.0] < (29.0, 56.0] < (56.0, 99.0]]
You can also set it to any other value, for instance precision=1 for 1 decimal place.
pd.qcut(df['age'], 3, precision=1)0      (1.9, 28.7]1     (55.7, 99.0]2     (28.7, 55.7]3     (28.7, 55.7]4      (1.9, 28.7]5      (1.9, 28.7]6     (55.7, 99.0]7     (55.7, 99.0]8      (1.9, 28.7]9     (28.7, 55.7]10    (28.7, 55.7]11    (55.7, 99.0]Name: age, dtype: categoryCategories (3, interval[float64]): [(1.9, 28.7] < (28.7, 55.7] < (55.7, 99.0]]
Pandas qcut() function is a quick and convenient way for binning numerical data based on sample quantiles.
I hope this article will help you to save time in learning Pandas. I recommend you to check out the documentation for the qcut() API and to know about other things you can do.
Thanks for reading. Please check out the Notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
A Practical Introduction to Pandas Series
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github