Data Preprocessing with Python Pandas — Part 5 Binning | by Angelica Lo Duca | Towards Data Science

Data binning (or bucketing) groups data in bins (or buckets), in the sense that it replaces values contained into a small interval with a single representative value for that interval. Sometimes binning improves accuracy in predictive models.
Data binning is a type of data preprocessing, a mechanism which includes also dealing with missing values, formatting, normalization and standardization.
Binning can be applied to convert numeric values to categorical or to sample (quantise) numeric values.
convert numeric to categorical includes binning by distance and binning by frequency
reduce numeric values includes quantisation (or sampling).
Binning is a technique for data smoothing. Data smoothing is employed to remove noise from data. Three techniques for data smoothing:
binning
regression
outlier analysis.
You can download the full code of this tutorial from my Github repository.
In this tutorial we exploit the cupcake.csv dataset, which contains the trend search of the word cupcake on Google Trends. Data are extracted from this link. We exploit the pandas library to import the dataset and we transform it into a dataframe through the read_csv() function.
import pandas as pddf = pd.read_csv('cupcake.csv')df.head(5)
In this case we define the edges of each bin. In Python pandas binning by distance is achieved by means of thecut() function.
We group values related to the column Cupcake into three groups: small, medium and big. In order to do it, we need to calculate the intervals within each group falls. We calculate the interval range as the difference between the maximum and minimum value and then we split this interval into three parts, one for each group. We exploit the functions min() and max() of dataframe to calculate the minimum value and the maximum value of the column Cupcake.
min_value = df['Cupcake'].min()max_value = df['Cupcake'].max()print(min_value)print(max_value)
which gives the following output
4100
Now we can calculate the range of each interval, i.e. the minimum and maximum value of each interval. Since we have 3 groups, we need 4 edges of intervals (bins):
small — (edge1, edge2)
medium — (edge2, edge3)
big — (edge3, edge4)
We can use the linspace() function of the numpy package to calculate the 4 bins, equally distributed.
import numpy as npbins = np.linspace(min_value,max_value,4)bins
which gives the following output:
array([  4.,  36.,  68., 100.])
Now we define the labels:
labels = ['small', 'medium', 'big']
We can use the cut() function to convert the numeric values of the column Cupcake into the categorical values. We need to specify the bins and the labels. In addition, we set the parameter include_lowest to True in order to include also the minimum value.
df['bins'] = pd.cut(df['Cupcake'], bins=bins, labels=labels, include_lowest=True)
We can plot the distribution of values, by using the hist() function of the matplotlib package.
import matplotlib.pyplot as pltplt.hist(df['bins'], bins=3)
Alternatively, we can set the edges of each bin manually.
bins = [ 0, 10, 50, 100 ]df['bin_cut_manual'] = pd.cut(df['Cupcake'] , bins=bins, labels=labels, include_lowest=True)plt.hist(df['bin_cut_manual'], bins=3)plt.show()
Binning by frequency calculates the size of each bin so that each bin contains the (almost) same number of observations, but the bin range will vary. We can use the Python pandas qcut() function. We can set the precision parameter to define the number of decimal points.
df['bin_qcut'] = pd.qcut(df['Cupcake'], q=3, precision=1, labels=labels)
Sampling is another technique of data binning. It permits to reduce the number of samples, by grouping similar values or contiguous values. There are three approaches to perform sampling:
by bin means: each value in a bin is replaced by the mean value of the bin.
by bin median: each bin value is replaced by its bin median value.
by bin boundary: each bin value is replaced by the closest boundary value, i.e. maximum or minimum value of the bin.
In order to perform sampling, the binned_statistic() function of the scipy.stats package can be used. This function receives two arrays as input, x_data and y_data, as well as the statistics to be used (e.g. median or mean) and the number of bins to be created. The function returns the values of the bins as well as the edges of each bin.
from scipy.stats import binned_statisticx_data = np.arange(0, len(df))y_data = df['Cupcake']x_bins,bin_edges, misc = binned_statistic(y_data,x_data, statistic="median", bins=2)
Now we should approximate each value of the df['Cupcake'] column to the median value of the corresponding bin. Thus we convert the bin edges to an IntervalIndex, which receives as index the left and right edges of each interval. In our case, the left edges starts from the beginning of the bin edges and do not contain the last value of the bin edges. The right edges instead, start from the second value of the bin edges and last until the last value.
bin_intervals = pd.IntervalIndex.from_arrays(bin_edges[:-1], bin_edges[1:])
We can quantise the Cupcake column by defining a set_to_median() function which loops through the intervals and when it finds the correct interval, it returns the mid value.
def set_to_median(x, bin_intervals):    for interval in bin_intervals:        if x in interval:            return interval.mid
We use the apply() function to apply the set_to_median() to the Cupcake column.
df['sampled_cupcake'] = df['Cupcake'].apply(lambda x: set_to_median(x, bin_intervals))
Now we can plot results. We note the loss of information.
plt.plot(df['Cupcake'], label='original')plt.plot(df['sampled_cupcake'], color='red', label='sampled')plt.legend()plt.show()
Finally, we can plot the median values. We can calculate the y values (y_bins) corresponding to the binned values (x_bins) as the values at the center of the bin range.
y_bins = (bin_edges[:-1]+bin_edges[1:])/2y_bins
Then we plot:
plt.plot(x_data,y_data)plt.xlabel("X"); plt.ylabel("Y")plt.scatter(x_bins, y_bins,  color= 'red',linewidth=5)plt.show()
We can use the package jenkspy, which contains a single function, called jenks_breaks(), which calculates the natural breaks of an array, exploiting the Fisher-Jenks algorithm. We can install the package by running pip3 install jenkspy.
import jenkspybreaks = jenkspy.jenks_breaks(df['Cupcake'], nb_class=3)
Now we can use the cut() function to transform data in labels.
df['bin_cut_break'] = pd.cut(df['Cupcake'] , bins=breaks, labels=labels, include_lowest=True)
And we can plot the histogram.
plt.hist(df['bin_cut_break'], bins=3)plt.show()
In this tutorial I have illustrated how to perform data binning, which is a technique for data preprocessing. Two approaches can be followed. The first approach converts numeric data into categorical data, the second approach performs data sampling, by reducing the number of samples.
Data binning is very useful when discretization is needed.
If you wanted to learn how to perform data preprocessing using the scikit-learn library, stay tuned...
If you have come this far to read, for me it is already a lot for today. Thanks! You can read more about me in this article.
Chris Moffit. Binning Data with Pandas qcut and cut
Chris Moffit. Finding Natural Breaks in Data with the Fisher-Jenks Algorithm
2021/03/31 — added binning by distance and binning by frequency. Updated Sampling.
2021/04/01 — added calculate natural breaks in data and bibliography.