Modality tests and kernel density estimations | by Madalina Ciortan | Towards Data Science

When processing a large number of datasets which can potentially have different data distributions, we are confronted with the following considerations:
Is the data distribution unimodal and if it is the case, which model best approximates it( uniform distribution, T-distribution, chi-square distribution, cauchy distribution, etc)?
If the data distribution is multimodal, can we automatically identify the number of modes and provide more granular descriptive statistics?
How can we estimate the probability density function of a new dataset?
This notebook tackles the following subjects:
Histograms vs probability density function approximation
Kernel density estimations
Choice of optimal bandwidth: Silverman/ Scott/ Grid Search Cross Validation
Statistical tests for unimodal distributions
DIP test for unimodality
Identification of the number of modes of a data distribution based on the kernel density estimation
As explained in this blog post https://mglerner.github.io/posts/histograms-and-kernel-density-estimation-kde-2.html histograms have the drawback of hiding in bins of unsuitable size some of the details of the actual data distribution.
def plotHistogramAndPdf(data, x, pdf):    ax = plt.gca()    plt.hist(data, bins = 4, alpha = 0.4, label = 'histogram of input values');    plt.ylabel('Frequency')    plt.xlabel('x values')    ax2 = ax.twinx()    plt.plot(x, pdf, c = 'red', label = 'probability density function');    plt.ylabel('PDF')    [tl.set_color('r') for tl in ax2.get_yticklabels()]    ax.legend(bbox_to_anchor=(0.4, 1.15))    ax2.legend(bbox_to_anchor=(1.15,1.15))    plt.savefig('figures/hist.jpg', bbox_inches='tight')    plotHistogramAndPdf(data, x, true_pdf)
Kernel density estimations are dependent on an arbitrary bandwidth which governs how smooth is the returned approximation. The example below illustrates the effect of various bandwidth values:
def getKernelDensityEstimation(values, x, bandwidth = 0.2, kernel = 'gaussian'):    model = KernelDensity(kernel = kernel, bandwidth=bandwidth)    model.fit(values[:, np.newaxis])    log_density = model.score_samples(x[:, np.newaxis])    return np.exp(log_density)for bandwidth in np.linspace(0.2, 3, 3):    kde = getKernelDensityEstimation(data, x, bandwidth=bandwidth)    plt.plot(x, kde, alpha = 0.8, label = f'bandwidth = {round(bandwidth, 2)}')plt.plot(x, true_pdf, label = 'True PDF')plt.legend()plt.title('Effect of various bandwidth values \nThe larger the bandwidth, the smoother the approximation becomes');plt.savefig('figures/bw.jpg', bbox_inches='tight')
In order to identify the optimal bandwidth, there are a few types of approaches:
Silverman’s rule of thumb: assumes Gaussian distribution for the unknown density. It is not the most optimal bandwidth selector but is used either as a very fast reasonably good estimator or as a first estimator in multistage bandwidth selectors. More precise solve-the-equation plug-in rules use estimation of integrated squared density derivative functional to estimate the optimal bandwidth. They demand high computations to solve a non-linear equation using iterative methods. They use ROT as a very first estimate
Scott’s rule of thumb:is optimal for random samples of normally distributed data, in the sense that it minimizes the integrated mean squared error of the density estimate.
These 2 methods have the advantage of being computationally fast, but they usually give too few bins and that it is likely to underfit the underlying data distribution. Both methods have been implemented in statsmodels package and illustrated below.
Methods based on cross validation: statsmodels comes with an cv bandwidth parameter. Alternatively, we can implement a grid search cross validation. Unlike the first 2 methods, performing a grid search is likely to be more computationally expensive, especially for larger datasets
from statsmodels.nonparametric.bandwidths import bw_silverman, bw_scott, select_bandwidthsilverman_bandwidth = bw_silverman(data)# select bandwidth allows to set a different kernelsilverman_bandwidth_gauss = select_bandwidth(data, bw = 'silverman', kernel = 'gauss')scott_bandwidth = bw_scott(data)def bestBandwidth(data, minBandwidth = 0.1, maxBandwidth = 2, nb_bandwidths = 30, cv = 30):    """    Run a cross validation grid search to identify the optimal bandwidth for the kernel density    estimation.    """    from sklearn.model_selection import GridSearchCV    model = GridSearchCV(KernelDensity(),                        {'bandwidth': np.linspace(minBandwidth, maxBandwidth, nb_bandwidths)}, cv=cv)     model.fit(data[:, None])    return model.best_params_['bandwidth']cv_bandwidth = bestBandwidth(data)print(f"Silverman bandwidth = {silverman_bandwidth}")print(f"Scott bandwidth = {scott_bandwidth}")print(f"CV bandwidth = {cv_bandwidth}")
As expected, the first Silverman and Scott return larger bandwidth values which lead to bigger bins and thus the loss of information about the data distribution.
Statsmodels allows for an automatic search of the optimal bandwidth based on cross validation and a maximum likelihood operator:
from statsmodels.nonparametric.kernel_density import KDEMultivariatestats_models_cv = KDEMultivariate(data, 'c', bw = 'cv_ml').pdf(x)
plt.figure(figsize= (14, 6))plt.plot(x, true_pdf, label = 'True PDF')kde = getKernelDensityEstimation(data, x, bandwidth=silverman_bandwidth)plt.plot(x, kde, alpha = 0.8, label = f'Silverman bandwidth')kde = getKernelDensityEstimation(data, x, bandwidth=scott_bandwidth)plt.plot(x, kde, alpha = 0.8, label = f'Scott bandwidth')kde = getKernelDensityEstimation(data, x, bandwidth=cv_bandwidth)plt.plot(x, kde, alpha = 0.8, label = f'CV bandwidth')plt.plot(x, stats_models_cv, alpha = 0.8, label = f'Statsmodels CV maximum likelihood')plt.legend()plt.title('Comparative of various bandwidth estimations for KDE');plt.savefig('figures/comp_bw.jpg', bbox_inches='tight')
There are a number of statistical tests addressing the data modality problem:
DIP test
excess mass test
MAP test
mode existence test
runt test
span test
saddle test
Unfortunately, not many have been implemented in python open source libraries.
The following python package https://github.com/BenjaminDoran/unidip provides an implementation of the dip test and also a functionality to ecursively extracts peaks of density in the data utilizing the Hartigan Dip-test of Unimodality.
from unidip import UniDipimport unidip.dip as dipdata = np.msort(data)print(dip.diptst(data))intervals = UniDip(data).run()print(intervals)
Once we have an estimation of the kernel density funtction we can determine if the distribution is multimodal and identify the maximum values or peaks corresponding to the modes.This can be done by identifying the points where the first derivative changes the sign. The method getInflexion points can return by default all inflexion points ( minima + maxima) or just a selection (typeOfInflexion = ‘max’/ ‘min’).The plot below depicts these maximum values which may correspond to multiple modes of the data distribution. The analysis can be continued by setting a threshold based on the height of the peak in order to filter out some less significant values.
def getExtremePoints(data, typeOfExtreme = None, maxPoints = None):    """    This method returns the indeces where there is a change in the trend of the input series.    typeOfExtreme = None returns all extreme points, max only maximum values and min    only min,    """    a = np.diff(data)    asign = np.sign(a)    signchange = ((np.roll(asign, 1) - asign) != 0).astype(int)    idx = np.where(signchange ==1)[0]if typeOfInflexion == 'max' and data[idx[0]] < data[idx[1]]:        idx = idx[1:][::2]            elif typeOfInflexion == 'min' and data[idx[0]] > data[idx[1]]:        idx = idx[1:][::2]    elif typeOfInflexion is not None:        idx = idx[::2]        # sort ids by min value    if 0 in idx:        idx = np.delete(idx, 0)    if (len(data)-1) in idx:        idx = np.delete(idx, len(data)-1)    idx = idx[np.argsort(data[idx])]    # If we have maxpoints we want to make sure the timeseries has a cutpoint    # in each segment, not all on a small interval    if maxPoints is not None:        idx= idx[:maxPoints]        if len(idx) < maxPoints:            return (np.arange(maxPoints) + 1) * (len(data)//(maxPoints + 1))        return idx
We notice that the obtained values correspond to the initial anchors of the generated distribution.