The Kernel Density Integral Transformation
Authors: Calvin McCarter
Venue: Accepted by TMLR ()

Abstract: Feature preprocessing continues to play a critical role when applying machine learning and statistical methods to tabular data. In this paper, we propose the use of the kernel density integral transformation as a feature preprocessing step. Our approach subsumes the two leading feature preprocessing methods as limiting cases: linear min-max scaling and quantile transformation. We demonstrate that, without hyperparameter tuning, the kernel density integral transformation can be used as a simple drop-in replacement for either method, offering robustness to the weaknesses of each. Alternatively, with tuning of a single continuous hyperparameter, we frequently outperform both of these methods. Finally, we show that the kernel density transformation can be profitably applied to statistical data analysis, particularly in correlation analysis and univariate clustering.

Published in Transactions on Machine Learning Research (10/2023)
The Kernel Density Integral Transformation
Calvin McCarter Boston, MA
mccarter. www.terrr.cbin@@@mai...
Reviewed on OpenReview:
https: // openreview. net/ form?? id= 60EcDK2
Abstract
Feature preprocessing continues to play critical role when applying machine learning and statistical methods to tabular data. In this paper, we propose the use of the kernel density integral transformation as a feature preprocessing step. Our approach subsumes the two leading feature preprocessing methods as limiting cases: linear min-max scaling and quantile transformation. We demonstrate that, without hyperparameter tuning, the kernel density integral transformation can be used as simple drop-in replacement for either method, offering protection from the weaknesses of each. Alternatively, with tuning of single continuous hyperparameter, we frequently outperform both of these methods. Finally, we show that the kernel density transformation can be profitably applied to statistical data analysis, particularly in correlation analysis and univariate clustering.
1 Introduction
Feature preprocessing is ubiquitous workhorse in applied machine learning and statistics, particularly for structured (tabular) data. Two of the most common preprocessing methods are min-max scaling, which linearly rescales each feature to have the range [0, 1], and quantile transformation (Bartlett, 1947 Van der Waerden 1952). which nonlinearly maps each feature to its quantiles, also lying in the range [0, 1]. Min-max scaling preserves the shape of each feature's distribution, but is not robust to the effect of outliers, such that output features' variances are not identical. (Other linear scaling methods, such as z-score standardization, guarantee output uniform variances at the cost of non-identical output ranges.) On the other hand, quantile transformation reduces the effect of outliers, and guarantees identical output variances (and indeed, all moments) as well as ranges; however, all information about the shape of feature distributions is lost.
In this papor, we observe that, by computing definite integrals over the kernel density estimator (KDE) Rosenblati 1956: Parzen 1962 Silverman 1986) and tuning the kernel bandwidth, we may construct a tunable "happy medium" between min-max scaling and quantile transformation. Our generalization of these transformations is nevertheless both conceptually simple and computationally efficient, even for large sample sizes. On wide variety of tabular datasets, demonstrate that the kernel density integral transformation is broadly applicable for lasks in machine learning and statistics.
We hasten to point out that kernel density estimators of quantiles have been previously proposed and extensively analyzed (Yamato 1973; Azzalini 1981 Sheather 1990 Kulezycki DaWidowicz 1999). However, previous works used the KDE to yield quantile estimators with superior statistical qualities. Statistical consistency and efficiency are desired for such estimators, and so the kernel bandwidth h is chosen such that h h-> as sample size N oc. However, in our case, we are not interested in estimating the true quantiles the cumulative distribution function (c.d.f.), but instead use the KDE merely construct preprocessing transformation for downstream prediction and data analysis tasks. In fact, as we will see later, we choose h to be large and non-vanishing, so that our preprocessed features deviate substantially from the empirical quantiles.
Our contributions are as follows. First, we propose the kernel density integral transformation as a feature preproccssing method that includes the min-max and quantile transforms as limiting cases. Sccond, we provide computationally-effia.
approximation algorithm that scales to large sample sizes. Third, we demonstrate the use of kernel density integrals in correlation analysis, enabling useful compromise between

---

Published in Transactions on Machine Learning Research (10/2023)
Pearson's and Spearman's p. Third, propose discretization method for univariate data, based computing local minima in the kernel density estimator of the kernel density integrals.
2 Methods
2.1
Preliminaries
Our approach is inspired by the behavior of min-max scaling and quantile transformation, which we briefly describe below.
The min- max transformation can derived by considering a random variable X defined over some known range [U, Vj. In order transform this variable onto the range [0, 1), one may define the mapping S - [o, 1] æ-U defined as x - ((x) == v-t with the upper and lower bounds achieved at U and V, respectively. In practice, one typically observes N random samples ,I,..., XN. which may be sorted into order statistics X1) S X(2) - .... X(N)- Substituting the minimum and maximum for U and V respectively, we obtain the min-max scaling function
z- x
X(1) S S X(N) x x X(1)
X(N) -
Sx(x)
0,
(1)
1,
X(N) SI.
Quantile transformation can be derived by considering random variable X with known continuous and strictly monotonically increasing c.d.f. Fx R - [0, 1]. The quantile transformation (to be distinguished from the quantile function, inverse c.d.f.) is identical to the c.d.f, simply mapping each imput value r to Fx (x) = P[X $ æt. One typically observes N random samples XI...., Xx and obtains an empirical c.d.f.
as follows:
PN(x)
N Σ Hx, <z)== = P[X < æ]. n=1
(2)
The quantile transformation also requires ensuring sensible behavior despite ties in the observed data.
2.2 The kernel density integral transformation
Our proposal is inspired by the observation that, just as the quantile transformation is defined by a data- dcrived c.d.f., the min-max transformation can also be interpreted as the c.d.f. of the uniform distribution over (x),, (n)) We thus propose to interpolate between these via the kernel density estimator (KDE) (Silveraan 1986 Recall the Gaussian KDE with the following density:
X
(x(x)
Σ Kn(x X,,) Nh
K
(3)
where Gaussian kernel K(2) exp(- 22 . /2) 2T and à > 0 is the kernel bandwidth. Consider further the definite integral over the KDE above as function of its endpoints:
ba = S' (a()dd..
(4)
If we replace the empirical c.d.f. in Eq. 2 with the KDE c.d.f. using Eq.
we obtain the following:
# KDD
(x; == Pr.
5
However, we change the integration bounds from (-oc, oo) to (X()). X(M)). such that we obtain (x) = 0 for < X(y) and (xD for X(N) S To do this, while keeping (x+)(() continuous, define the
2

---

Published in Transactions on Machine Learning Research (10/2023)
final version of the kernel density integral (KD-integral) transformation as follows:
(()) S x < (())
(2;h)
0,
x < (() ((N) S x.
(6)
1,
It can be seen that proposed estimator matches the behavior of the min-max transformation and the quantile transformation at the extrema X(1) and X(N)- Also, in practice, we parameterize h adx, where a is the bandwidth factor, and ox is an estimate of the standard deviation of X.
As the kernel bandwidth h + 0o, the KD-integral transformation will converge towards the min-max transformation. Mcanwhile, when h + 0, the KD-integral transformation converges towards the quantile transformation. (Proofs are given in the Appendix.) Furthermore, exactly the formula for computing the kernel density estimator of quantiles. However, in this paper, we propose (for the first time, to our knowledge) choosing a large kernel bandwidth. Rather than estimating quantiles with improved statistical efficiency as in (Sheather 1990. Kulczycki & DaWidowicz 1999), our aim is carry out a transformation that is an optimal compromise between the min-max transformation and the quantile transformation, for a given downstream task. As we will see in the experiment section, this optimal compromise often struck at large bandwidths such as h = 1 1,xx, even as the sample size + co.
noted previously, /N
fRDQ (x) is
2.3 Efficient computation
For supervised learning preprocessing, the KD-integral transformation must accomo- date separate train and test sets. Thus, it is desirable that it be estimated quickly a training set, efficiently (yet approx- imately) represented with low space com- plexity, and then applied quickly to test. samples. For the Gaussian kernel, we compute and store KD-integrals R= minnRmax 1000, N) reference points, equally spaced in [0, 1]. New points are transformed by linearly interpolat- ing these KD-integrals, This approach has runtime complexity of O ORRNrraa)) at training time and Olog(()) Nrest) at test time, and space complexity of O(R)) (R)- While this approach offers tunable control of the error and efficiency at test-time, the tradeoff between precision and training time is not ideal, as shown in Figure D
10-1
interpolated Gaussian (bwfm0.1)
10-2
polyexp (bwf=0 1)
Max abs
Interpolated Gaussian (bwf=1.0)
polyesp (twi=1 0)
10-3_
Interpolated Gaussian (bwf-10.0)
polyexp (bw(=10.0)
10-4
10a2
10-1
10a
201
Figure 1: Precision versus runtime tradeoff for different bandwidth factors a E 10.1,1, 103. The data consisted of Nerain = 10,000 training points sampled from LogNormal(0,, 1) and Niest = 10,000 equally-spaced test points within the range of the training data. The error is computed as the maximum absolute error between estimated values and those from the exact Gaussian KDE edf. We compare against naively approximating the Gaussian KDE via sampling S {30, 100, 300, 3010", 10*} without replacement to form the KDE, followed by interpolating with R = 1000 references as usual. The runtime using exact Gaussian cdf computation is depicted on the x-axis (=*"). Runtime was measured on a machine with 2.8 GHz Core i5 processor.
To address this, we applied the poly(I=|)e-Io) kernel (Hofmeyr 2019) to our sotting. The polynomial-cxpoeettal family of kernels, parameterized by the order к of the polynomial, can be evaluated via dynamic programming with runtime complexity O(kN), eliminating the quadratic dependence on the number of samples. We use order k = 4, which yields a smooth kernel with infinite support that closely approximates the Gaussian kernel with an appropriate rescaling of the bandwidth (Hofmeyr 2019 More details are given in the Appendix.
As shown in Figure I for 10k samples, this results in almost 1000x speedup compared to exact evaluation of the Gaussian KDE edf, across range of bandwidth factors. It also offers 10x to 100x training time
3

---

Published in Transactions on Machine Learning Research (10/2023)
speedup fixed precision level, compared to naively speeding up the Gaussian KDE by subsampling data points then interpolating usual.
Our software package, implemented in Python /Numba with a Scikit-learn Pedregosa et al. 2011 compatible
API, available at https::/github. http:////ao//aa//tto/////o/taata
2.4 Application to correlation analysis
appropriate for measuring the strength of linear relationships, Spearman's rank-correlation coefficient p [Searmaa 1904) is useful for measuring the strength of non-linear yet monotonic relationships. Spearman's 's p may be computed by applying Pearson's 's correlation coefficient after first. transforming the original data to quantiles it is straightforward to extend Spearman's 's by computing the correlation coefficient between two variables by computing their respective KD-integrals, then applying Pearson's 's formula as before. Like Spearman's 's it is apparent that ours is a particular case general correlation coefficient Kendall (1948). For N samples of random variables X and Y, the general correlation coefficient. I may be written as
In correlation analysis, whereas Pearson's r PPaarson 1895
ranks R(x)) == N (x))). Thus,
Σ aiylij
V E of Eii=1 b2
for 0ij == ry -r,, big = 83 81, where now T2 and 8, correspond to the KD-integrals of X, and Y,, respectively.
Because ranks are robust to the effect of outliers, Spearman's p is also useful as a robust measure of correlation; our proposed approach inherits this benefit, as will be shown in the experiments.
2.5 Application to univariate clustering
Here we apply KD-integral transformation to the problem of univariate clustering (a.k.a. discretization). Our approach relies on the intuition that local minima and local maxima of the KDE will tend to correspond to cluster boundaries and cluster centroids, respectively. However, naive application of this idea would perform poorly because low-density regions will tend to have many isolated extrema, causing to partition low-density regions into many separate clusters. When we apply the KD-integral transformation, we draw such points in low-density regions closer together, because the definite integrals between such points will tend to be small. Then, when we form a KDE on these transformed points and identify local extrema, we avoid partitioning such low-density regions into many separate clusters.
Our proposod approach thus comprises three steps:
1. Compute T, =
(DDQ (Xn), Vn E NJ.
2. Form the kernel density estimator (t) for Ti.... Tv. For this second KDE, select vanishing bandwidth via Scott's Rule (h = =N --2.2 Scott 1992)
3. Identify the cluster boundaries from the local minima in fa),,, and inverse- Preess----iitterra--raannooo the boundaries for T, to obtain boundaries for clustering X,.
3 Experiments
In this section, we evaluate our approach on supervised classification problems with real-world tabular datascts, on correlation analyses using simulated and real data, and on clustoring of simulated univariate datasets with known ground-truth. We use the poly([=|)e= kernel with polynomial order к in our experiments.
4

---

Published in Transactions on Machine Learning Research (10/2023)
(A)
(B)
(C)
iotegalerall transform
transform
quantile transtom
1.0
1.0
1.0
class
class
0.5
class
3
2
0.0
0.0
0.0
0.0
2.5
5.0
transform
transtorm
transform
Figure 2: Comparison of (A) min-max scaling, (B) quantile transformation, and (C) KD-integral transforma- tion on the MalicAcid feature in the Wine dataset. The horizontal density plots depict the distribution of the original data, while the vertical density plots show the distribution after each preprocessing step.
3.1 Feature preprocessing for supervised learning
3.1.1 Classification with PCA and Gaussian Naive Bayes
We first replicate the experimental sctup of Raschka 2014), analyzing the effect of feature preprocessing methods on simple Naive Bayes classifier for the Wine datasel. (Forina al. 1988). In addition to min-max scaling, used in (Raschka 2014j, also try quantile transformation and our proposed KD-integra transformation approach. In Figure 2 we illustrate the effect of min-max scaling, quantile transformation, and KD-integral transformation on the KalicAcid feature on this dataset. We see that KD-integral transform in Figure 2 C) is concave for inputs >3, compressing outliers together, while preserving the bimodal shape of the feature distribution. It substantially smoother than the empirical quantile transform in Figure (B).
We examine the accuracy resulting from each of the preprocessing methods, followed by (as in (Raschka) 2014h) principle component analysis (PCA) with components, followed by Gaussian Naive Bayes classifier, evaluated via 70-30 train-test split. For the KD-integral transformation, we show results for the default bandwidth factor of 1, for bandwidth factor chosen via inner 30-fold tross-aalidation. and for a sweep of bandwidth factors between 0.1 and 10. The accuracy, averaged over 100 simulated train-test splits, is shown in Figure 3 A). We repeat the above experimental sctup for 3 more popular tabular classification datascts:
Iris Fisher 1936), Penguins (Gorman et al. 2014), Hawks [Cannon el al. 2019 shown in Figure (-D),
respectively.
Compared to min-max scaling and quantile transformation, tuning-free proposal wins Wine; Iris, it is sandwiched between min-max scaling and quantile transformation; Penguins, it wins over both min-max and quantile transformation; on Hawks, it ties with min-max while quantile transformation struggles. We also compare to z-score scaling; our tuning-free method beats it on Wine and Iris, loses to it on Penguins, and ties with on Hawks. Overall, among the tuning-free approaches, ours comes in first, second, second, and second, respectively. Our approach with tuning is always non-inferior or better than min-max scaling and quantile transformation, but loses to z-score scaling on Penguins; however, note that z-score scaling performs poorly on Wine and Iris.
3.1.2 Supervised regression with linear regression
We compare the different methods two standard regression problems, California Housing Paae & Barry 1997) and Abalone (Nash et al., 1995), with results depicted in Figure 4 Here measure the root-mean- squared error on lincar regression after preprocessing, with cross-validation sctup as before. KD-intcgral, both with default bandwidth and with cross-validated bandwidth, outperformed the other approaches. Even in CA Housing, where quantile transformation outperforms min-max scaling due to skewed feature distributions, there evidently benefit to maintaining a large enough bandwidth to preserve the distribution's shape. In both regression datasets, the performance improvement offered by KD-integral exceeds the difference in performance between linear and quantile transformations. We perform additional experiments CA
5

---

Published in Transactions on Machine Learning Research (10/2023)
(A) Wine (N 178, D = 13)
(B) Iris (N 150, = 4)
0.970
0.92
Accuracy
Accuracy
0.965
0.91
2-score
quantile
0.90
KD-integral (bwf=1)
KD-integral (bwf=CV)
KD-integral
KD-integral bandwidth
KD-integrai bandwidth
(C) Penguins (N = 333, D = 4)
(D) Hawks (N 891, D = =5 5)
0.895
0.890
0.97
Accuracy
Accuracy
0.96
0.865
0.860
0.94
10-1
10a
10a
10-1
10a KD KD-integral bandwidth factor
10a
KD (D-integral bandwidth factor
Figure 3: Accuracy on supervised classification problems for different feature preprocessing methods. Results are shown for Wine (A), Iris (B), Penguins (C), and Hawks (D); higher accuracy is better. Accuracy is shown horizontal line for min-mat max scaling, 2-score scaling, quantile transformation, KD-integral with the default bandwidth factor a 1, and KD-integral with bandwidth selected via CV. In green, we show accuracy as function of KD-integral bandwidth factor a. Error bars depict the standard deviation over 100 simulations
(A) CA Housing (N = 20640, D 8)
(B) Abalone (N = 4177, = 10)
0.76
2.24
0.74
2.22
0.72
2.20
2.18
2.16
10-1
10a
107
10-1
10"
107
KD-Integral bandwidth factor
KD-Integral bandwidth factor
Figure 4: Root mean-squared error (rMSE) supervised regression problems for different feature prepro- cessing methods. Results are shown for California housing (A); and Abalone (B); lower rMSE is better. Performance is shown as horizontal line for min-max scaling, quantile transformation, KD-integral with the default bandwidth factor a = 1, and KD-integral with bandwidth selected via CV. In green, we show rMSE as function KD-integral bandwidth factor a.

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: Performance of preprocessing methods on Small Data Benchmarks, as measured by area under the Receiver Operating Characteristic curve (ROC AUC). The columns display the mean and standard deviation of the ROC AUC, computed over 142 datasets in the benchmark. We also show p-values from one-sided Wilcoxon signed-rane raank test comparisons.
METHOD
Mean (StdDev) ROC AUC p (vs KDI (a=1))
(vs KDI (a=CV))
Min-max
0.864 (0.132)
1.6 10-3
5.1 10-8
Quantile
0.866 (0.131)
7.9 X 10-3
2.9 10
KD-integral (a =
0.868 (0.129)
3.7 10
KD-integral (a = CV)
0.869 (0.129)
1.00
0810100 0390
Relative ROC AUC
0.95
0.90
min-max
0.85
quantile
0.80
KD-integral (bwf=1)
KD-integral (bwf=CV)
102
10a
10a
105
Number of samples
Figure 5: Performance of preprocessing methods on Small Data Benchmarks, plotted against dataset size. For each dataset, we compute the relative ROC AUC for given method its own ROC AUC, divided by the maximum ROC AUC over all preprocessing methods for that dataset.
Housing, with data subsampling, showing that the chosen bandwidth remains constant regardless of sample size; see the Appendix (A.3).
3.1.3 Linear classification on Small Data Benchmarks
We next compared preprocessing methods dataset-of-datasets benchmark, comprising 142 tabular dalasets, each with at least 50 samples. We replicated the experimental setup of the Small Data Benchmarks (Feldman 2021) on the UCI++ dataset repository (Paulo et al.) 2015). In Feldman 2021 the leading linear classifier was a support vector classifier (SVC) with min-max preprocessing, to which we appended SVC with quantile transformation and SVC with KD-integral transformation. As in (Feldman 2021 for each preprocessing method, we optimized the regularization hyperparameter E (10-1, evaluating each method via soeevs-restt-eighhel ROC AUC, averaged over stratified cross-validation folds. For the latter, we measured results with both the default bandwidth factor 1 (so to give equal hyperparameter optimization budgets to each approach), and with pross-validttion grid-search to select C' and E (3-1,30,3" 3. Our results are summarized in Table i We see that KD-integral transfornation provides stauistically-signifeea greater average ROC AUC, with less variance, at the same tuning budget as the other approaches; further improvement is provided by tuning. We further analyzed the performance of the different methods in terms of the number of samples in each dataset Figure 5 Plotting the relative ROC AUC against the number of samples N, we see that our proposed approach particularly helpful in avoiding suboptimal performance for small-N datasets.
10?),

---

Published in Transactions on Machine Learning Research (10/2023)
(A)
(
(C)
Spearman: 0.44
2
Pearson: 0.51 KDi:
0
-10
Spearman:
Spearman: 1 1.00
Pearson: 0.59
Pearson: 80
KDI: 0.72
-20
-2
-2
0.0
0.5
1.0
-2 0
2
X
X
Figure 6: muutstaaio of correlation analysis using Pearson's r, Spearman's P, and our proposed approach, for simulated data. Three scenarios depicted: (A) nonlinear yet monotonic relationship, (B) noisy linear relationship, and (C) linear relationship corrupted by outliers.
(A)
(B)
1.0
1.0
(AveRaams, Mad
Spearman or KD-integral correlation
Sprearman KD-integral correlation
(AveBedrms, AveRooms)
(AveBedrms, AveRooms)
(Population, AveBedms)
0.0
0.0
-0.5
-0.5
Spearman KDHintegral
Spearman KD-integral
-1.0
-1.0
1.00
0
0.0
0.5
10
-100
0.5
0.0
0.5
1.0
Pearson correlation
Pearson corretation
Figure 7: Correlation coefficients dcrived from the California housing datasct. The pink line depicts the gap between Spearman and KD-integral correlations, while the distance from the gray line shows how far each are from the Pearson correlation. Both (A) and (B) contain the same data, but top disagreements between ours and Pearson's s, and ours and Spearman's 's are highlighted separately in (A) and (B), respectively. The error bars in black depict the standard deviation of each correlation, computed from 100 bootstrap simulations.
3.2 Correlation analysis
To provide basic intuition, first illustrate the different methods synthetic datasets shown in Figure replicating the example from (Wikipedial 2009). When two variables monotonically but not linearly related, in Figure ((A), the Spearman correlation exceeds the Pearson correlation. Tn this case, our approach behaves similarly to Spearman's. When two variables have noisy linear relationship, as in Figure BB), both the Pearson and Spearman have moderate correlation, and our approach interpolates betweon the two. When two variables have a linear relationship, yet are corrupted by outliers, the Pearson correlation is reduced due to the outlicrs, while the Spearman corrclation is robust this effect. In this case, our approach also behaves similarly to Spearman's.
Next, perform correlation analysis the California housing dataset, containing district-level features such as average prices and number of bedrooms from the 1990 Consus. Overall, the computed correlation coefficients are lypically close, with only few exceptions, as shown in Figure 7 Our approach's 's top two disagreements with Pearson's 's are on (AverageBedrooms, AverageRooms) and MediamIncome AverageRooms). Our approach's top two disagrcements with Spearman's are on (AverageBedrooms, AverageRooms) and (Population, AverageBedrooms) We further observe that KD-integral-baat correlations typically, but not always, lie between the Pearson and Spearman correlation coefficients.

---

Published in Transactions on Machine Learning Research (10/2023)
Original (0.85)
Full data
Zoom-in
30
2.0
Avededres
Ave Wedress
1.5
value
20-
10-
0.5
0
50
100
10
AveRcoms
AveRaoms
After quantile transformation
Full data
(0.08)
Zoom-in
1.0
Medien house volue (1100K)
0.8
0.8
AveBedrme
AveBedms
0.8
0.6
0.4
0.4
0.0
0.0
1.0
AveRooms
AveRooms
After KD-integral transformation
Full data.
(0.31)
Zoom-in
1.0
1.0
Median house value (S100k)
0.8
Are3edints
Ave Bedimss
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
0.5
1.0
AveRooms
AveRooms
Figure 8: Correlation analysis for (AverageBedrooms, AverageRooms) in the California housing dataset. The rows, from top to bottom, correspond to original data, quantile transformation, and KD-integral transformation. The full dataset shown on the left, while outliers are excluded on the right. Each district is colored by its median house value. In parenthesis above each row are the Pearson (0.85), Spearman (0.08), and KD-integral (0.31) estimated correlations between the variables.
We analyze the correlation disagreements for (AverageBedrooms, AverageRooms) Figure From the original data, it is apparent that AverageBedrooms and AverageRooms are correlated, whether we examine the full dataset exclude outlier districts. This relationship was obscured by quantile transformation (and thus, by Spearman correlation analysis), whereas it is still noticeable after KD-integral transformation.
We repeat the analysis of disagreement for (MedianIncome, AverageRooms) and (Population, AverageBedrooms) in Figure g For the former disagreement, our approach agrees with quantile transformation- based analysis, identifying the typical positive dependence between median income and average rooms, by reducing the impact districts with extremely high average rooms. For the latter disagreement, our approach agrees with original based analysis, identifying the negative relationship one would expect to observe between district population (and therefore density) and the average number of bedrooms.
3.3 Clustering univariate data
In this experiment, we generate five separate synthetic univariate datasets, each sampled according to the following mixture distributions:

---

Published in Transactions on Machine Learning Research (10/2023)
(A)
(B)
Original (0.33)
Orig nol
Full data
Zoam-in
Full data
(-0.07)
Zlom-in
30200
5000-
Nathome
4000
75-
20200
2000
50-
10200
2000
25
1030
10
10
20
1.0
2.0
Medinc:
Medinc
Aweeeccim
AvcBedims
After quantiie transtormation
After cuantile transtormatton
Full diate
10.00 (0.64)
Zoom-in
Full data
(0.03)
Zoom-in
3.0
1.0
1.0
1.0-1
0.8-
6.6
0.4
Pegu
0.4
0.0-
0.1
0.0
0.--1
Modin:
Mesfinc
AvcBcoms
AvcBodima
After KD-intsgra transormazionn
After KD-intacral transformaatin
Full data
(0.64)
Zoomin
Full data
(-0.58)
Zoom
3.0-
10-
1.0-
0.8
0.
08-
0.6
0.4-
0.
0.5
0.0
Medine
Madize
AxnDecimm
AueDedmma
Figure 9: Correlation analysis for (MedianIncome, AverageRooms) (A) and (Population, AverageBedrooms) (B) in the California housing dataset. See Figure for an explanation of the plot.
0.55 * NN = 1 0.75) + 0.30 * N(p 4, 1) + 0.15 Uniffa- = 0, = 20)
+ = 1, = 0.5) + 0.45 4(u=4,0=1) 1)- 0.10 R Unifla 0, b = 20]
0.67 NN( 1, 0.5) 0.33 V(m==400-
0.8 * Exp(A- 1) + 0.2 * Exp(A = =4]
0.5 = Exp(A- = 8) + 0.5 * [100 Exp(A = 5)].
We compare our approach to five other clustering algorithms: KMeans with K chosen to maximize the Silhouette Coefficient (Rousseeuw 1987), GMM with K chosen via the Bayesian information criterion (BIC), Bayesian Gaussian Mixture Model (GMM) with Dirichlet Process prior, Mean Shift cluster-
ing (Comaniciu & Meer 2002). and HDBSCAN (Campello et al., 2013 McInnes et al. 2017 (with
min lll_ccls___sieeee KDE with adaptive sensitivity=0.5 [Abramson 1982 Wang & Wang 2007
caustersselection weteeeepssplo..... and local minima of an adaptive-bandwiid
In Figure I0 we compare the ground-truth cluster identities of the data with the estimated cluster identities from each of the methods, for N = 500 samples. On the top row, we depict. the true clusters, as well as the true number of mixture components. On each of the following rows, we show the distributions of estimated clusters for each the methods. We see that our approach is the only method to correctly infer the true number of mixture components, and that it partitions the space similarly to ground-truth. GMM with BIC performed well on the first three datasots, but not on the last two. Mcanwhile, KMcans performed well on the last three datasets but not on the first two. Bayesian GMM tended to slightly overestimate the number of components, while MeanShift. and HDBSCAN (even after excluding the samples it classified as noise) tended to aggressively overestimate.
We include results for an ablation of our proposed approach, in which we define separate clusters at the local minima of the KDE of unpreprocessed inputs, rather than the KDE of KD-integrals. We see that the
10

---

Published in Transactions on Machine Learning Research (10/2023)
Ground-truth
K-3
K-3
K-2
K-2
100
100
200
100
KMeans
R=2
R-2
R-2
R-2
100
100
100
0
GMM
8-3
R-3
R-2
R-4
R-7
100
100
200
100
Bayesian GMM
R-3
R-4
R-3
R-3
100
100
200
100
MeanShift
R-6
R-6
R-3
R-5
R-6
100
100
200
100
HDBSCAN
R=4
R=4
R-3
R-9
100
100
200
100
0
KDE Local Minima
8-2
R-4
R-2
R-2
100
100
50
200
100
AdaptiveKDE Local Minima
R=2
R=3
R-2
R-2
100
100
50
200
100
0
KD Integra Local Minima
R=3
R=3
R=2
R=2
R=2
100
100
200
300
20
50
100
Figure 10: Clustering performance for five datasets (one dataset per column), generated with N 500 samples. On each of the rows, for the different methods, we indicate the estimated number of components (green if correct, red if incorrect).
ablated method fails the first, second, and fourth datasets, where there is large imbalance between the mixture weights of the cluster components. Using the adaptive-handwidth KDE fixes the second and fourth dataset but not the first.
We repeated the above experimental setup, this time varying the number of samples N e (100, (100,200, 500, 1000, 2000, 5000), and performing 20 independent simulations per each setting of N. For each simulation, recorded whether the true and estimated K number of components matched, as well the the adjusted Rand index (ARI) between the ground-truth and estimated cluster labelings. The results, averaged over 20 simulations, are shown in Figure II Our approach is the only method to attain high accuracy across all settings when N > 1000. Alternative methods behave inconsistently across datascts or across varying sample sizes. For the first two datasets with small N, KD-integral is outperformed by GMM and Bayesian GMM. However, GMM struggles on the first two datasets for large N, and on the fourth and fifth datascts; mcanwhile, Bayesian GMM struggles on all datasets for large N. The only approach that comes close KD-integral using local minima of the adaptive KDE; however, it struggled with smaller sample sizes the first and second datasets.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Acturacy
KMenns
10
10'
10
10
10
10'
10
10
Bayesian GMM
MeanShift
HDBSCAN
AdaptiveKDE Local
KD-integral Local Minima
ARI
0.0
10'
10'
10"
1.0
10' 10'
10'
10'
10'
10*
Figure 11: Performance at clustering univariate features, for varying number of samples N. The top row depicts the fraction of simulations in which the estimated mumber of mixture components equaled the true number. The bottom row depicts the average ARI between the ground-truth clustering and the estimated clusters. Each of the five columns corresponds five mixture distributions described in the text and depicted the top row of histograms.
4 Related Work
As far as we know, the use of kernel density integrals to interpolate between min-max scaling and quantile transformation has not been previously proposed in the literature. Similarly, are not aware of kernel density integrals being proposed to balance between the strengths of Pearson's and Spearman's p.
Our approach is procedurally similar to copula transformations in statistics and finance (Cherubini et al. 2004 Patton 2012). But because we have a different goal, namely generic feature transformation that is to be extrinsically optimized and evaluated, our proposed approach has a markedly different effect. Besides the small adjustment from fKDii (which is procedurally identical to the kernel density copula transformation Gourieroux et al., 2000)) PKDI, our proposal aims something quite different from copula transforms. The copula literature ultimately aims at transforming a particular reference distribution (e-g., uniform Gaussian), with the KDE used in place of the empirical distribution merely for statistical efficiency, thus choosing consistency--velddin
bandwidth (Gourieroux et al., 2000 Fermanian & Scaillet 2003 We depart from this choice, finding that a large bandwidth that preserves the shape of the input distribution is frequently optimal for settings (e.g. classification) where marginal distributions need not have a given parametric form.
Our proposed approach for univariate clustering is similar in spirit to various density-based clustering methods, including mean-shift clustering (Comaniciu & Meer 2002). level-set trees (Schmidberger & Frank 2005 Wang
& Huang 2009 Kent et al. 2013), and HDRSCAN (Campello al. 2013). However, such methods tend to leave isolated points as singletons, while joining points in high-density regions into larger clusters. To our knowledge, our approach for compressing logether such isolated points has not been previously considered.
The kernel density estimator (KDE) was previously proposed Flores et al. 2019 in the context of discrotization-basde preproccssing for supervised lcarning. However, their mothod did not use kornel density integrals as a preprocessing step, but instead employed a supervised approach that, for a multiclass classification problem with C' classes, constructed C different KDEs for each feature.
5 Discussion
Practical Rocommendations We recommend that if one must employ a feature preprocessor without any tuning or comparisons between different preprocessing methods, then KD-integral with bandwidth-factar of is the best one-shot option. If instead tuning is possible, suggest comparing the performance of
12

---

Published in Transactions on Machine Learning Research (10/2023)
KD-integral, with log-space sweep of (0.1,10), in addition to the other popular preprocessing methods. Finally, we recommend that one always estimate our transformation with the order-4 polynomial-exponentia kernel approximation of the Gaussian kernel, and represent it with Rmax 1000 reference points.
Limitations Our approach for supervised preprocessing is limited by the fact that, even though the bandwidth factor is a continuous parameter, its selection requires costly hyperparameter tuning, Meanwhile, our proposed approach would benefit from further thcoretical analysis, well investigation into extensions for multivariate clustering.
Future Work In this paper, have focused per-feature transformation. But, especially in genomics, it is common to perform per-sample quantile normalization (Bolstad et al. 2003 Amaratunga & Cabrera, 2001,, in which features for single sample are mapped to quantiles computed across all features; this would benefit from further study. Also, future work could study whether kernel density integrals could be profitably used in place of vanilla quantiles outside classic tabular machine learning problems. For example, quantile regression (Koenker & Bassett Jr 1978 has recently found increasing use conformal prediction (Romano
et al.] 2019: Liu al. 2022), uncertainty quantification (Jeon al. 2016), and reinforcement learning
(Rowland al 2023
6
Conclusions
In this paper, proposed the of the kernel density integral transformation nonlinear preprocessing step, both for supervised learning scttings and for statistical data analysis. In varicty of experiments sinulated and real datasets, demonstrated that our proposed approach is straightforward to use, requiring simple (or no) tuning to offer improved performance compared to previous approaches.
13

---

Published in Transactions on Machine Learning Research (10/2023)
References
lan S Abramson. On bandwidth variation in kernel estimates-a square root law, The annals of Statistics, pp- 1217 1223, 1982.
Dhammika Amaratunga and Javier Cabrera. Analysis of data from viral dna microchips. Journal of the American Stalistical Association, 96(456):1161 1170, 2001.
Adelchi Azzalini. A note on the estimation of distribution function and quantiles by kernel method. Biometrike, 68(1):326 328, 1981.
Maurice $ Bartlett. The use of transformations. Biometrics, 3(1):39-5) 02, 1947.
Benjamin M Bolstad, Rafael A Irizarry Magnus Astrand, and Terence P. Speed. A comparison of normalization methods for high density oligonucleotide array data based variance and bias. Bioinformatics, 19(2):
185-193, 2003.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-172. Springer, 2013.
A Cannon, G Cobb, B Hartlaub, J Legler, R Lock, T Moore, A Rossman, and J Witmer. Stat2data: Datascts for stat2. R package version, 2(0), 2019.
Umberto Cherubini, Elisa Luciano, and Walter Vecchiato. Copula methods in finance. John Wiley & Sons, 2004.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions pattern analysis and machine intelligence, 24(5):603 619, 2002.
Sergey Feldman. Which machine learning classifiers are best for small datasets? an empirical study. http://hw....oooooooo////////aaa 2021.
David Fermanian and Olivier Scaillet. Nonparametric estimation of copulas for time series. The Journal of Risk, 5(4):25-54 -4,, 2003.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. AnnEug, 7(2):179-188 188, 1936.
Jose Luis Flores, Borja Calvo, and Aritz Perez. Supervised non-parametric discretization based on kernel density estimation. Pattern Recognition Letters, 128:496 504, 2019.
Michele Forina, Riccardo Leardi, C Armanino, Sergio Lanteri, Paolo Conti, P Princi, et al. Parvus extendable package of programs for data exploration, classification and correlation. Elsevier, 1988.
Kristen B Gorman, Tony D Williams, and William Fraser. Ecological sexual dimorphism and environmental variability within a community of antarctic penguins (genus pygoscelis). PloS one, (33):c9008,, 2014.
Christian Gourieroux, Jean-Paul Laurent, and Olivier Scaillet. Sensitivity analysis of values at risk. Journal of empirical finance, 7(3-4):225-24) 2000.
David Hofmeyr. Fast exact evaluation of univariate kernel sums. IEEE transactions on pattern analysis and machine intelligence, 43(2):447-458, 2019.
Soyoung Jeon, Christopher Paciorek, and Michael F Wehner. Quantilo-based bias correction and uncertainty quantification of extreme event altribution statements. Weather and Climale Entremes, 12:24 32, 2016.
Maurice George Kendall. Rank correlation methods. Griffin, 1948.
Brian P Kent, Alessandro Rinaldo, and Timothy Verstynen. Deback: A python package for interactive density-based clustering. arXiv preprint ar
2013.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Sociely, pp. 33-50, 1978.
Piotr Kulezycki and Antoni Leon DaWidowicz. Kernel estimator of quantile. ZESZYTY NAUKOWE- UNIWERSYTETU JAGTELLONSKIEGO-AA. SERTES-, 1236:325-336, 1999.
Meichen Liu, Lei Ding, Dengdeng Yu, Wulong Liu, Linglong Kong, and Bei Jiang. Conformalized fairness via quantile regression. Advances in Newral Information Processing Systems, 35:11561 11572, 2022.
Leland McInnes, John Healy, and Steve Astels. hdbscaa: Hicrarchical density based clustering. J. Open Source Softw., 2(11):205, 2017.
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. DOT: https://doi..rr/11..22222
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Stalistics & Probobilily Letlers, 33(3): 291-297, 1997.
Emamuel Parzen. On estimation of a probability density function and mode. The onmals of mathematical statistics, 33(3):1065 1076, 1962.
Andrew Patton. A review of copula models for economic time series. Journal of Multivariate Analysis, 110: 4 18, 2012.
Luis Paulo, Davi Pereira dos Santos, and Danilo Horta. ucipp 1.1, January 2015. URL https://doi.onnr org/
10. 10.5281/zeoodo. 13748
Karl Pearson. Vii. note regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-3352 1895.
Fabian Pedregosa, Gaăl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. et al. Scikit-lcarn: Machine lcarning in python. the Journal of machine Learning research, 12:2825 2830, 2011.
S Raschka. About feature scaling and normalization and the effect of standardization for machine learning algorithms. sebastian raschka. 2014, 2014.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pp. 832 837, 1956.
Peter J Roussceuw. Silhoucttes: graphical aid to the interpretation and validation of cluster analysis. Journal of compulational and opplied malhemalics, 20:53 65, 1987.
Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyun- yan, Karl Tuyls, Mare G Bellemare, and Will Dabney. An analysis of quantile temporal-difference learning.
arXiv preprint crXi::2301.044 04462, 2023.
Gabi Schmidberger and Eibe Frank. Unsupervised discretization using tree-based density estimation. In European conference on principles of data mining and knowledge discovery, pp. 240 251. Springer, 2005.
DW Scott. Multivariate density estimation. Multivariate Density Estimation, 1992.
Simon J Sheather. Kernel quantile ostimators. Journal of the American Statistical Association, 85(410): 410 416, 1990.
BW Silverman. Density estimation for statistics and data analysis. Monographs on Statistics and Applied Probabilily, 1986.
15

---

Published in Transactions on Machine Learning Research (10/2023)
C Spearman. The proof and measurement of association between two things. Am J Psychol, 15:72-101, 1904.
BL Van der Waerden. Order tests for the two-sample problem and their power. In Indagationes Mathematicae (Proceedings), volume 55, pp. 453 553-458. Elsevier, 1952.
Bin Wang and Xiaofeng Wang. Bandwidth selection for weighted kernel density estimation. orXiv preprint arXiv:0709.1 16110 2007.
Xiao-Feng Wang and De-Shuang Huang. A novel density-based clustering framework by using level set method. JEEE Transactions on knowledge and data engineering, 21(11):1115-133, 2009.
Wikipedia. Spearman's rank correlation coefficient, 2009. [Online; accessed AAugust-2023)
Hajime Yamato. Uniform convergence of an estimator of a distribution function. Bulletin of Mathematical Statistics, 1973.
Qiaochu Yuan. Does bounded section of the normal distribution converge to the uniform distri- bution? Mathematics Stack Exchange, 2023. URL httpss:/matii: htctp:://aah.sstcceeechha / wtw.s//ceeccaaaae.c.o//////////a
(version: 2023-09-17).
16


Reviews:
[{"review_id": "sLcqFJPeNG", "reviewer": "Reviewer_2B75", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Feature engineering is an important issue in machine learning, thus the objective of this paper is relevant. \n\n### Weaknesses \n\n- The novelty of this paper is limited. The KDE with the Gaussian kernel is a quite standard method, and using it for feature preprocessing is also a simple application. Of course, it is acceptable if such a simple approach leads to some unexpected theoretical properties and/or remarkable empirical advantages. However, this paper is not the case.\n- Motivation is not clear. What is the current challenge to be addressed, and for which application is the proposal required?\n- Presentation should be improved. For example, although the kernel bandwidth $h$ is one of the most important parameters in the proposal, it is missing in Equation (3).\n- In experiments, accuracy without any feature preprocessing should be compared as a baseline.\n- In experiments, in Table 1, the proposal is simply not significantly better than other methods. By considering that the proposal requires an additional parameter $h$ (or $\\alpha$), its practical advantage is not convincing.\n- In experiments, clustering on univariate data does not add much information as it is a too simple problem. Trying clustering on multivariate data might be interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use the Gaussian kernel density estimation for feature preprocessing. Each feature value is transformed based on the definite integral over the KDE. The proposal has been empirically evaluated on several scenarios including classification and univariate clustering on synthetic and real-world datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please resolve all the above weaknesses.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "muf6uoSCne", "reviewer": "Reviewer_2qmP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths* \nThe paper is well written, very easy to follow and grasp.\n\n*Weaknesses*\nThe method seems somewhat ad hoc and it is not clear to me from the description and the experiments, what benefits it really brings.\nFor example paragraph 3 states - \"we are not interested in estimating the true quantiles .. instead use the KDE to merely construct a preprocessing step\". This is too vague a statement. Any transformation can be used as preprocessing step. Why is this supposed to be particularly good?\nOn the other hand, kernel methods are well known to outperform classical linear methods when the relationships cannot be expected to be simply linear. I am not clear from reading the paper, to what extent is the addition of the quantile tranformation (on top the kernel transform) really beneficial.\nThe argumentation in favor of the method is largely based on the results of experiments. These are, however, not very convincing to me. This may in part be by not stressing enough where does the strength of the models really come from - mainly from the kernel density estimation or from the qunatile tranform.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a kernel density quantile transformation as a preprocessing step to be used as an alternative to min-max transformation or empirical quantile transformation. The paper briefly introduces the method and then documents on a set of synthetic and classical UCI experiments its properties.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Please address the concerns mentioned under Weaknesses.\n- Equation (4) - are the bounds of the integral swapped? Please check and correct if necessary.\n- Equation (4) - by $P(a, b)$ you mean $P(a \\leq x \\leq b)$? Please check and correct accordingly.\n- On top of page 4 you suggest to estimate the variance $\\sigma_X^2$ from the data and set just the multiple $\\alpha$ as a hyper-parameter. Later on you say that $\\alpha=1$ seems to perform the best for the badwidth. Why is this surprising. Isn't this to be expected (given that you estimate the variance)?\n- Paragraph 2 page 3 - you claim the KD-quantile converges to min-max with $h \\to \\infty$ and to vanilla quantile with $h \\to 0$. Please give the proofs. \n- top of page 4 - Please explain what \"vanishing bandwidth via Scott's Rule\" actually is. I am sorry, but I do not know.\n- section 3.1.1. - I am not clear on the classification pipeline (preprocess, pca, NB). Please elaborate, in particular clarify the purpose of the PCA step.\n- section 3.1.1. - you claim that in Figure 2 we see the reduced distance of outliers - I may not be reading the figure correctly, but where do I see it? Please clarify.\n- end of section 3.1.1 - you say KD is \"non-inferior to the better of the two\". Isn't it worth than min-max in (B) and than z-score in (C). Please check and correct if appropriate.\n- Table 1 - the differences in the mean are negligible (compared to the std). Please comment / clarify.\n- last phrase section 3.1.2 - we see ... I do not see. Please help me see. \n- section 3.3 - Please clarify the motivation for the particular setup of the synthetic datasets. Are these supposed to illustrate something interesting about the method? Why have you chosen particularly these mixtures? Are these somehow particularly challenging for your or the other methods? How? Why?", "broader_impact_concerns": "I do not have any concerns in this respect.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WtbcjTuYj0", "reviewer": "Reviewer_qhJo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The ubiquity of univariate scaling and transformations makes new alternatives to the common approaches of potential use.\n- The intuitive connection to quantile and min-max scaling at the two extremes of the bandwidth makes the approach fairly accessible to non-experts.\n\nWeaknesses:\n- As a contribution the proposal is fairly minimal and lacks theoretical backing.\n- The transformation seems quite similar to a kernel based copula transformation. Clearly since the transformation \"maintains\" some of the shape of the true distribution, where copulas aim for equivalent marginals, they are theoretically distinct. However copula transformations based on an over-smoothed kernel cdf will likely be very similar to the proposal. Ultimately the only difference between the proposal and a kernel based uniform copula is the fact that the proposal forces the transformation to span the interval [0, 1] which, while it may be beneficial, is a minimal contribution.\n- The \"speed up\" using subsampling and a grid of reference points seems very suboptimal when there is considerable literature on efficient and highly accurate univariate kernel smoothing, e.g., [1,2,3].\n- I'm not sure if this is just a typo, but it seems like a poor choice to set the bandwidth proportional to the VARIANCE of the sample, and not the standard deviation. Every piece of theory suggests proportionality to the scale of the variable and not squared scale. I appreciate the authors are distancing their approach from theory, but I presume they would want their method to be scale invariant at the very least.\n- On the experimental side:\n-- although the discussion about the tendency for \"other\" density based clustering methods to create small clusters in low density regions, it is very unlikely in my opinion that popular density based methods would be applied in the standard formulations to univariate data due to how much more simple the clustering problem is. Indeed something like a k-nn based density method or a simple univariate kde with adaptive bandwidth would similarly overcome this \"problem\".\n-- The authors say that it is intuitively the case that population density and average number of rooms in houses are negatively correlated. I don't understand why this would not be opposite. Surely a higher density of people in an area requires more people per domicile, and hence more bedrooms on average? Please correct me if I am overlooking something obvious?\n-- The experiments in general would benefit from some accommodation of sampling variation, using, e.g. bootstrap or some sort of stratified bootstrap when there are outliers present which should be retained.\n\nStrength/Weakness:\n- The additional \"tuning parameter\" being the bandwidth in the kernel cdf may be beneficial as an extra parameter to introduce complexity to simple (e.g., linear) models and potential enhance performance. At the same time there is invariably a cost of an additional hyperparameter, both in terms of model interpretability and computational demands associated with cross-validation or similar.\n\n[1] Raykar, Vikas C., Ramani Duraiswami, and Linda H. Zhao. \"Fast computation of kernel estimators.\" Journal of Computational and Graphical Statistics 19.1 (2010): 205-220.\n[2] Hofmeyr, David P. \"Fast exact evaluation of univariate kernel sums.\" IEEE transactions on pattern analysis and machine intelligence 43.2 (2019): 447-458.\n[3] Fan, Jianqing, and James S. Marron. \"Fast implementations of nonparametric curve estimators.\" Journal of computational and graphical statistics 3.1 (1994): 35-56.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a monotonic univariate transformation based on a min-max scaling of a kernel cdf evaluated at the sample. Traversing the values of the bandwidth from the infinitesimal to infinite, the transformation interpolates between quantile transformation and min-max scaling.\n\nSome heuristic discussion relating to its potential advantages over these other two, as well as other common transformations, plus experiments documenting its potential usefulness in correlation analysis, univariate clustering and as a pre-processing tool for supervised learning are provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Since the proposal is minimal in terms of a methodological and/or theoretical contribution, I believe it lives and dies on the persuasiveness of the experimental results. At the moment, although there are some nice observations coming through, the manner in which the experiments are conducted and the results displayed would benefit from considerably more work. I will note some possibilities below:\n\n- Investigate the usefulness of tuning the bandwidth using, e.g., cross-validation to improve performance of supervised models using the porposed transformation. E.g. in the transformation + PCA + Naive Bayes. Note that I fully appreciate the need to automatic transformations which don't need tuning, at the moment the improvement here is extremely minimal and certainly not outside of what would reasonably be expected due to sampling variation. I am not suggesting you remove the automated (i.e., fixed scaling) results but as mentioned above the results need to be more persuasive to justify and entire publication.\n- Use a log scale for Figure 3 since currently it is hard to see what is going on for small N\n- Add some sort of resampling, e.g. bootstrap, to 3.2 to assess the sampling variation. In the pair of variables to which I directed a query earlier, population density and average bedrooms, the values are very close to zero and so could easily flip signs for different samples. At the moment, even if I am wrong about the intuitive sign of the relationship, the results are not persuasive due to the small magnitude of the correlations.\n- The univariate clustering results are not persuasive. There are some methods designed for univariate clustering and none are included. It also seems unreasonable to simply use standard methods designed for primarily multivariate problems in a univariate setting without sensible modifications. Furthermore, DBSCAN is supposed to \"find noise\" and so saying that identifying very small collections of points which should have been merged into clusters is rather unfair.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision among the reviewers was mixed.  But I feel that the paper has merit, and does a sufficient job to provide evidence that the technique can be useful.  Such pre-processing is common enough, and there is not always a good solution that I think the proposed approach is a useful to consider.
Decision: Accept (poster)