Generalized Oversampling for Learning from Imbalanced datasets and Associated Theory: Application in Regression
Authors: Samuel Stocksieker
Denys Pommeret
Arthur Charpentier
Venue: Accepted by TMLR ()

Abstract: In supervised learning, it is quite frequent to be confronted with real imbalanced datasets. This situation leads to a learning difficulty for standard algorithms. Research and solutions in imbalanced learning have mainly focused on classification tasks. Despite its importance, very few solutions exist for imbalanced regression. In this paper, we propose a data augmentation procedure, the GOLIATH algorithm, based on kernel density estimates and especially dedicated to the problem of imbalanced data.  This general approach encompasses two large families of synthetic oversampling: those based on perturbations, such as Gaussian Noise, and those based on interpolations, such as SMOTE. It also provides an explicit form of such machine learning algorithms.  New synthetic data generators are deduced. We apply GOLIATH in imbalanced regression combining such generator procedures with a new wild-bootstrap resampling technique for the target values. We evaluate the performance of the GOLIATH algorithm in imbalanced regression where we compare our approach with state-of-the-art techniques.

Published in Transactions on Machine Learning Research (06/2024)
Generalized Oversampling for Learning from Imbalanced datasets and Associated Theory: Application in Regression
Samuel Stocksieker
somuel. htt.ullstttottceeeeuunuuuoaoaaa
Laboratoine de Sciences Actuorielle et Finoncière (UR SAF), Université Cloude Bernard Lyjon 1 Institut de Mathématique de Marseille, Aix- Ax--Missellle Université
Donys Pommeret
denus.poomeerrelnuuiiooooo,
CNRS
Centrole Marseille
Institut de Mathématique de Marseille, Aiz Akr-Marsell Université
Arthur Charpentier
sharpentier.arthurruuuue
Département de Mathématique, Université du Québec à Montréal
Reviewed on OpenReview: /ttps://openreeee openreveee..........mmme net/ forum? id- DLaPhQag
Abstract
In supervised learning, it is quite frequent to be confronted with real imbalanced datasets. This situation leads to learning difficulty for standard algorithms. Research and solutions in imbalanced learning have mainly focused on classification tasks. Despite its importance, very few solutions exist for imbalanced regression. this paper, we propose a data augmen- lation procedure, the GOLIATH algorithm, based on kernel density estimates and especially dedicated the problem of imbalanced data. This general approach encompasses two large familics of synthetic oversampling: those based perturbations, such as Gaussian Noisc, and those based on interpolations, such as SMOTE. It provides explicit form common to such machine learning algorithms. and new synthetic data generators can be deduced. We apply GOLIATH in imbalanced regression combining such generator procedures with a new wild-bootstrap resampling technique for the target values. We evaluate the performance of the GOLIATH algorithm in imbalanced regression where we compare our approach with state-of-the-art techniques.
1 Introduction
Many real-world forecasting problems are based predictive models in supervised learning framework and standard algorithms can fail when the target variable is too skewed, Learning from imbalanced data concerns many problems with numerous applications in different fields (Krawezyk) (2016), Fernández et al. (2018a)) ). The major part of such works concerns imbalanced classification (see for instance Buda et al. (2018); Cao et al. (2019). Cui al. 2019), Huang et al.] (2016), Yang & Xu (2020). Branco et al.] (2016b) where many solutions propose a pre-processing strategy especially for the generation of new synthetic data. A large part of such existing methods consist in adapting the well know SMOTE algorithm (Fernández et al. (2018b)). Very few works have addressed the problem of imbalanced regression despite its relevance to many important real-world applications in different ficlds such as economy, metcorology, medicine, or insurance. As for imbalanced classification, some applications focus on predicting rare and extreme values, which can great interest. In the literature, the imbalanced regression corresponds to the correct prediction of rare extreme values of a continuous target variable Fernández al. (2018b) but, contrary to the classification tasks, there no level quantify the imbalance and the labels are continuous. Unlike in a classification context, learning from imbalanced dataset. for regression tasks leads to two additional problems: i) the

---

Published in Transactions on Machine Learning Research (06/2024)
definition of the imbalanced phenomenon and ii) the identification of the observations that are considered minority.
In this paper we propose a very general method, which we shall call GOLIATH (for Generalized Over- sampling for Learning from Imbalanced datasets and Associated THeory) to deal with the imbalanced regression problem. The first step of GOLIATH a synthetic covariates generation based on kernel density estimators. The second step of GOLIATH concerned with the imbalanced regression: a new method based wild-bootstrap procedure is proposed for generating target values given the synthetic covariates. GOLIATH is then two-step algorithm. In this paper, concentrate tabular data rather than images because many applications rely on structured data, and there are still very few solutions available to address them.
Our main contributions can be summarized as follows:
From methodological perspective, providing unified and generalized statistical expression encompassing existing data augmentation techniques, notably the popular and widely utilised SMOTE method (3).
This approach unifies two important families of synthetic data generators: perturbation- based (33 and interpolation-basee
(3.4) methods (see Branco al. (2016b)). Recent rescarch emphasizes the ongoing importance of explaining the properties of SMOTE, and GOLIATH follows this approach by analyzing the structure of the SMOTE algorithm which can be considered as mixture distribution.
ii) Deducing new synthetic data oversampling methods using the large and adaptative expression of GOLIATH 4 more flexible, more suitable and empirically outperform state-of- the-art approaches 6 More preciscly,
Introducing some non-classical kernels density estimates 4.2 to adapt and control the generation of new synthetic data according to the support covariates, especially in discrete or bounded cases. Moreover, the proposed kernels use an efficient estimate of the bandwidth parameter yielding the method data-driven.
Extending the interpolation techniques (SMOTE) with a Beta distribution and then with a Gaussian- distribution extending the generation in the data space rather than onto the segments 4.1
iii) Proposing innovative methodologies to address Imbalanced Regression
Introducing weights for the data generation 5
This is an important issue this topic, unlike for classification tasks. Furthermore, our approach preserves the continuity of the target variable distribution and avoids making discrete or binary, which is generally done in other works.
A new technique to generate the target variable : by combining a a random forest in order to consider its dependencies with the covariates and the new synthetic data.
with
A new way to generate synthetic data:
The 'mix' mode, provides a new solution to the main challenge of synthetic data generation, avoiding overfitting and the introduction of bias.
The paper is organized as follows. In Section B we give general form of our data augmentation proce- dure corresponding to the first step of GOLIATH. We study some standard perturbation and interpolation methods that are included this approach, such as SMOTE and Gaussian Noise. In Section 4 we develop the theory to obtain new generators. In Section 5 we will look more closely at the imbalanced regression, corresponding to the second step of GOLIATH. Numerical results on several applications are presented in Section 0 Finally, discuss the method proposed in Section 7
2

---

Published in Transactions on Machine Learning Research (06/2024)
2 Related works
Learning from imbalanced distributions is topic that concerns several domains and numerous applications Krawezyk (2016). As highlighted in the literature on the subject, especially in the survey by Branco et al.) (2016b), the case of regression has been very little addressed and there are very few solutions available to date. Most of the works addressing regression tasks are based on a relevance function Torgo & Ribeiro p200 The objective of this function, denoted by o, is to associate a relevance value, ranging from 0 to 1, to each value of the target variable Y: o(Y): R - [0, 1], with representing maximum relevance, for extreme values of Y. Using a given threshold 4,, it is possible to divide the dataset into two parts: the subset of minority observations, denoted as D,, and the subset of majority observations, denoted as Dn, such that:
D. = {[x, Vi)i==,..... .n D: o(y:) > f and DN = D \ D,, D being the set of all observations.
Once the support is binarized in this way, it becomes possible to adapt solutions from imbalanced classi- fication as SMOTE, Gaussian Noise, elc. TTogo et al. (2013), Branco et al. (2017), Branco et al. (2019)
Ribeiro & Moniz (202)), Song et al. (2022). Camacho et al. (2022), Branco (2018)). This approach presents the disadvantage age of dividing the continuous distribution of the target variable into binary classes and there- fore involves a loss of information. Indeed, considering an example of a threshold at 0.8, an observation with a relevance value of 0.79 will be considered as majority while an observation with a value of 0.81 will be considered as extreme. Furthermore, this "extreme" observation will have the same importance as an obser- vation with relevance value of 0.99. Furthermore, providing a utility function and an associated threshold is difficult for the user. However, Ribeiro (2011) suggested constructing relevance function. This default function is based on box plot of the distribution, which may not be representative. Additionally, it only considers extremes and not rare values inside the distribution, which are not extreme.
Steininger et al.] (021 proposed a new way of approaching the issue by introducing a cost function. The authors' objective is to individually weight observations based on the rarity of the target value. They there- fore propose to deduce a weight inversely proportional to the probability of occurrence of the target variable. The probability of occurrence is estimated by a classical kernel density estimator. This method is similar to the one propose. The authors use the opposite (1 (1-ch af(() with fly) the estimated density function of the target variable Y) whereas we use the inverse. The use of the inverse (1/f(y) of the estimated density function is also found in Yang et al. (2021) J. However, the authors partition the domain of the target variable, which also leads to information loss. Both approaches use this weighting of observations to redefine the loss function, i.e., with cost-sensitive approach. The main drawback of this approach, while promising, is that the authors modify neural network. However, as indicated below, this type of approach may not be the most effective on tabular data. Our preprocessing solution has the advantage of being able to be associated with any classical learning algorithm, making GOLIATH universal. Indeed, this is what we do in the experiments where an autoMT, with 10 different models is combined with GOLIATH.
Huang al. (2022) built on a similar idea by using the inverse of the kernel density estimator for the tar- get variable to create a utility function. However, similar approaches based on relevance functions, the authors divide the data into subsets of rare and "normal' observations using a threshold. This approach suffers from the same drawbacks as those based on the utility function, except that the relevance function is more intuitive and allows for the treatment of rare non-extreme data. They then suggested optimizing the undersampling of normal values using the relevance function and assessing the impact on prediction (MSE profit & loss). For oversampling, they propose using a conditional VAE to incorporate the concept of trare/nomma as useful labels for data generation.
More recently, other methods have emerged by using deep learning approaches, for dealing with images, such as Sen et al. (2023). Ding et al. (2022), Gong et al. (2022) or Wang & Wang (2024). However, the techniques proposed in this context rely on deep learning. While these approaches are highly effective with images, they often require adjustments to be elfective with (abular data. Indeed, some authors have shown that standard deep learning approaches can be ineffective for certain tasks with tabular data (e.g Shwartz- Ziv & Armon (2022), Ma et al. (2020). [orisov et al. (2022) or (Grinsztajn et al.] (2022)). But other works have shown that these techniques can ultimately be effective adapted specifically to some tabular data (c.g Gorishniy et al. 2021 Katzir et al. (2020) Shhviitt & Segal (2018)
3

---

Published in Transactions on Machine Learning Research (06/2024)
3 A New Kernel-Based Oversampling Formulation
In this section, we present our generic framework and demonstrate that two major families of synthetic data generation can be written in this form.
3.1 Notations and Problem Setting
We consider a sequence of observations { (Zn, }n)}. which are realizations of iid random variables (X,Y), where the target variable Y is univariate and the covariate X is a p=dimensionall random vector. The components of X = (X1, xx)) are supposed to be continuous or discrete and Y is supposed to be quantitative. Let 2ij the variable j for the observation and æj the observed variable j. Write Φ (.i..... [nn the set of all observations and x* new synthetic data. Finally, we designate (+) as the 9th nearest neighbors of an observation F; """",,"""""pp"
As with classification lasks, learning from imbalanced distribution can alter the results of standard algorithms in a regression context, i.e., when the target variable is contimuous. Indeed, conventional methods presuppose a uniform importance of the target variable values Branco et al. (0016b). Furthermore, the metrics used to measure performance, as well as optimization criteria to calibrate algorithms, measure an average error level across all values a uniformly weighted manner Torgo & Ribeiro (2009).
In regression, the continuous and infinite nature of Y introduces two main challenges:
It is not easy to define "rare" values and differentiate them from "frequent" values unlike in classification where this is directly given by the classes;
ii) Therefore, measuring imbalance is not immediate and is complicated to assess unlike in classification where it suffices to compare the classes.
If the proposed solution to the imbalanced regression problem relies on synthetic data generation, then it introduces a third challenge. Indeed, unlike classification, where the labels of Y remain unchanged when creating synthetic data, in regression, it is necessary to generate new relevant values for the target variable. In this way, GOLLATH combines two-step procedure: the first generates X and the second deduces the generation of Y using an innovative procedure based on a wild bootstrap and is based on X*.
Although the problem of imbalanced regression is becoming increasingly recognized, there is no universal formal definition: Ribeiro (2011) and all works associated propose to define it by reducing it to a binary classification problem based on a relevance function (Branco et al. (2016b)) - which we avoid doing. Ren let al. (2022) proposes to define it as follows: "imbalanced regression assumes that the training set and test set draum from different joint distributions, ptrain(s, v) and pbal(x.,) y) respectivcly, where the training set's label distribution ptrain(y) is skewed and the balanced test set's label distribution pbal(y) is uniform. However, we disagree with this idea because, in many real-world applications, the larget variable is naturally distributed asymmetrically. Yang et al. (221)) "formally define the Deep Imbalanced Regression task as learning from imbalanced data with continuous targets, and generalizing to the entire target range". We prefer to formally and simply define the problem of imbalanced regression as follows: learning from continuous distributions that include rare values (including extremes) that are precisely relevant to the studied phenomenon. This definition builds upon the notion of utility of Ribeiro (2011), but we avoid binarizing or discretizing the problem. Since the distribution is continuous, the relevance of values is assumed to be progressive, so binary classification does not seem relevant to us. The scarcity of observations can be due to two factors: either the distribution is inherently asymmetric, and thus the sample will also be theoretically asymmetric, or the distribution is not necessarily asymmetric, but sampling bias generates this phenomenon in the observed sample distribution (as defined in Ren et al. (2022h or more formally in Stocksicker et al. (202)))
The notion of rarily is therefore crucial in defining the phenomenon of imbalance. We propose to define rarity follows (inspired by |Stocksieker et al. (023))): Let Y be the target variable taking values in y R. We denote by fy a kernel density estimator of its probability density function (pdf). We consider that there is an inbalanced situation in one of the following two cases:
4

---

Published in Transactions on Machine Learning Research (06/2024)
Case 1:
When fy significantly deviates from given pdf.
Case 2: When Fr is significantly too small.
In both cases, we write fo for the given pdf (Case 1) or for the uniform pdf (Case 2). We propose the following definition of imbalance: We define a regression problem as (a, 9) 3)-imbalanced if there exists subset X y
such that
J Iv(u)dy [ fo(w)dy
J folu)dy 2B and
<a.
Note that in Case 2, the quantity S₂x fo(y)dy is the Lebesgue measure of X; up to factor. In simpler terms, an imbalanced regression implies that a sample is significantly different from a target distribution (Case 1) or under-represented (Case 2) for at least significant portion of the support of Y. This means that regions with high support in the target distribution would be ignored. The level of imbalance increases notably with larger (a, B) values. As mentioned carlier, this issuc also relics on the sample size. When fy and fo have the same support, the imbalance diminishes with a growing sample size n. Conversely, if certain segments of Y support remain unobserved, the problem may persist even with large n.
3.2 General Formulation of GOLIATH
We define GOLIATH the generalized oversampling procedure based on the form of the following weighted kernel density estimate:
9x (x"xxx"a [+) = -E WiKK(xx", ,)).
(1)
IEI
where (Kt)iz is collection of kernels, (wi)iez is sequence of positive weights with Eiez wi 1, and I represents subset of 21, .n.. Here the index * stands for the synthetic data, In this generalized form of kernel density estimate, a kernel is associated with each obscrvation and takes into account the set of all points T. This is a general form that is specified later depending on the kernels used. We recall that a kernel K is a positive and integrable function such that K(u, 2 0 and f K(u, ē)du = In (1 we propose a general form for the conditional density for the synthetic data generators. The objective is to use the flexibility of the kernels to estimate the density of covariates in order to obtain synthetic data that reflects the distribution of the observations.
We can show that generalizes perturbation- and interpolation--bssed synthetic data oversampling. We give an illustration with the basic algorithms ROSE, Gaussian Noise and SMOTE in Subsections 3.3 and 3.4 where we demonstrate that these methods are particular cases of the generalized form 1 with
corresponding parameters summarized in Appendix A.2 2 Several existing methods can be rewritten in the
form and we give some illustrations in Appendix A.3 In Section we will show that some particularly
I and we will compare some of them to interesting new methods can be derived from the generic form current competitors in the imbalancod regression context.
REMARK 1. The generators in [ are colled smoothed bootstrup methods (see Silverman & Young 1987) Hall et al. [1989), De Angelis & Young (192)). Smoothed bootstrap consists in drawing samples from kernel density estimators of the distribution. It can be decomposed into two steps: first, a seed is randomly drawn and second, a random noise from the kernel density estimator is added to obtain a new sample. In the form 1) the first step is represented by the drawing weight wi and the second by the kernel Kz(x). Convergence properties of smoothed bootstrap are studied in De Mortini & Rapallo (2008) and Falk & Reiss (1989). They proved the consistency of the smoothed bootstrap with classical muultvvriatt kernel estimator and more specifically the convergence in Mallows metric. As described by the authors, the smoothed bootstrap provides better performances Ihan a classical bootstrup when a proper choice of smoothing puramelers is used. Other works have focused on the consistency of the multivariate kernel density estimate and proposed a relevant bandwidth matria, see for instance Silverman (1986), Scott (2015) and Duong 8 Hazelton (2005). All this work has been carried out outside the context of regression models and only with classical kernels. GOLIATH extends this to wabalanced regression using adapted kernels.
5

---

Published in Transactions on Machine Learning Research (06/2024)
3.3 Rewriting Interpolation Approaches
As presented in Fernández al. (0018b), the Synthetic Minority Oversampling Technique (SMOTE) Chawla ct al. (0022) is considered a 'de facto" standard for learning from imbalanced data and has inspired large number of methods to handle the issue of class imbalance. It is also one of the first techniques adapted to imbalanced target values in regression with Torgo et al. (2013). SMOTE algorithm can be summarized as follows: at each step of the data augmentation procedure an observation is randomly selected, which we shall call a seed. Then, onc of its & nearest neighbors among K, which a hyperparamer, randomly drawn. Finally, a new data point generated by interpolating bet ween these two initial observations.
We will denote by æ; the selected seed and by æ.. (e) the (th nearest neighbor of æ4, among its K neighbors. The new data is then generated by linear interpolation between H, and (f). Each observation has a probability 1/n of being the seed, and each of its & neighbours has probability 1/k of being selected. Finally, writing x" the synthetic data we have x² = Aæ; + -1- x),, with A uniformly distributed u([0 1]).
To show that this approach is a particular case of T we proceed in three steps:
1 Conditionally to æ, and (f), the jth component of xx is generated by uniform distribution, with the indicator function:
[m,n (0), (ax (O)) (2e)
SMOTE 9 (z) ai,, a,,
(e)(() Wez
Each component of x" is drawn by the same uniform variable, that 23 = Ax + (1 A)x,, (e) for j = 1, P, and write the multivariate generating density follows:
SMOTE An
(x) (ē Ti.,
a., (f) minn b (f)) max (0)
(e)(e) 2e
where Ust = (x; - ny (f))/ xx, (O)
2 Conditionally to Ki,, x" is generated according to uniform mixture model (UMM) on the segments between a; and its k-nn. The same mixture component used for each component giving:
SMOTE g2
(x [.)
1 h
miiiaa .yy m0xt))) (CD) (255)
[E)(() Fayl
3 More generally, x" is generated according a mixture of UMM as follows:
(1)) (1))))
SMOTE
(x
SMOTE (x
1 x..)
11
(x.)(() xndl
1mal t=1
n
K. KSMOTE (a",
We finally obtain the form 1 with = [0, n), wi = l/n and K,(x", KSMOTE(2* (x" Ø). This new writing represents the conditional SMOTE density given the observation which can be seen as a mixture of Uniform Mixture Model. We can relate this expression to the work of Elreedy al. (2023) and [aahho et al. (2024) in which the authors give an expression of the unconditional SMOTE density, that is integrating the distribution over ă in context of class minority. Other methods derived from SMOTE can be recovered by (see Appendix A.3)
In the original version SMOTE, the seed drawn successively with loop and randomly. These ways are very close when the generated sample size is large
6

---

Published in Transactions on Machine Learning Research (06/2024)
3.4 Rewriting Perturbation Approaches
We illustrate as follows:
by recovering two classical data augmentation procedures, ROSE and Gaussian Noise (GN),
At each step of the ROSE algorithm (see Menardi & Torelli (2014) the seed Tú. is selected randomly. Given Tú. synthetic data is generated with multivariate density
ROSE 9w
(ar a x.)
KROSE (a' - ma.) |/, K (H (12 -m'-.)))).
where K denotes the multivariate Gaussian kernel and H, diag(h,,... Up) is the bandwidth matrix proposed by Bowman Azzalini (1999), with hg = (ip 1/(x+ og 1, P. Finally, a synthetic random variable X is generated with the density
ROSE 9
[+)
TI Σ
ROSE (a* I..) Σ KB
KORSE we KH (x= x...).
Similarly to ROSE, at each step of the Gaussian Noise algorithm (see Lee & Sauchi (2000) a seed is selected and synthetic data is generated. Finally, the generating multivariate density has the form
GN O
(x ())
KGN (a' -x,,)
K(H,, =..)),
H 1/2
where HGN- diag(h1, hp), hg = :g = =1,..... 1.....p.
Both cases are particular cases of 11 with wa = and K, = KH, (x" -x;.), i.e. the same Gaussian kernel for all observations but with a different bandwidth matrix.
3.5 Global criticism
Although there are many extensions of SMOTE or ROSE and Gaussian Noise, such techniques suffer from some drawbacks. For the interpolations techniques, the directions in the data space are limited and deter- ministic because they depend only on the k-nn (nearest neighbors). Moreover, the distance from the seed is also limited because the new sample on the segment with the drawn nearest neighbor. For the perturbation techniques, the directions in the data space are randomly generated and so they can more explore the space. The distance between the new sample and the seed is also unbounded. However, the directions are randomly chosen and do not respect the correlation between the data and their support and the correlations between variables.
4 New Kernel-Based Methods from GOLIATH
Since Goliath is very generic form, its practical interest lies in the choice of a judicious combination of weights and kernels. We proceed in separating the two families interpolation and pertubation, deducing new generators from the form I which allows us to extend the initial approaches.
4.1 Generalized Interpolation Approaches
A general form We propose a particular form of 1 which generalizes the SMOTE algorithm as follows:
g (a (a*())
Σ Wi .)) Σ we
Σ
Teje
gi. (a (a)
VEZ
EEI
REG.
where g (x* is an interpolation function on [m.,, x.,( (f)), To denoting the set of k-nn associated C;...
SMOTE is then a particular case when W; = 1. Teji & and g gint("" (a* (š) 1jows (2) (2)-m: os (#" -2.))
represents a
uniform distribution between the vectors æ; and a; (!).

---

Published in Transactions on Machine Learning Research (06/2024)
Nearest Neighbors Smoothed Bootstrap Since the uniform distribution coincides with the Beta distri- bution with parameters a = 8 = 1, a natural extension of SMOTE to consider general Beta distribution. We find the same idea in Yao al. 2022 within another context. The very flexibility of the Beta distri- bution suggests us to propose x:" Aæs (1 ((); s for i= 1, ...., where follows a generalized Beta
distribution. By abuse of notation, we get the following interpolation function:
ent 9i.e (w (8) T(a+ + 9) (2" -x.))"''aa. (x.. 5")a-' 10..z 10) (f)-a.))
(8)
Extended Nearest Neighbors Smoothed Bootstrap Finally, the previous methods based inter- polation are limited to the 'seed k-nn segments" and therefore do not reach all the data space. To avoid generating bounded discrete support we propose to extend these approaches to any part of the support by adding a Gaussian distribution on the segment as follows:
3 x
Tejs Di, int (x*()),
EEJ
where the extended interpolation function is a Beta Gaussian mixture, that is, Iie "0=i"t(=") (x*[#) is the density of a Gaussian distribution N (0, o²) where 0 is generated by ghe (z)().. To rely on the recent literature, we remark that SASYNO algorithm Gu et al. (2020) is a special case this methodology. This extended version can be viewed as a hybrid method between interpolation and perturbation techniques. It provides a good compromise between the interpolation and perturbation approaches because can generate in the whole data space as the perturbation approach i.e. constraint-free, but assigns distribution to the directions towards the segments, that is orienting the perturbation toward the k-nn.
REMARK 2. We tried to adapt the k-nearest neighbors densily estimate (Biau & Dearoye (2055) that is handwidth-wariahe kernel (also colled balloon kernel) generator but its computation time currently too high to be used.
4.2 Generalized Perturbation Approaches
Classical Smoothed Bootstrap As the ROSE and GN techniques use multivariate Gaussian kernel estimate with a diagonal bandwidth matrix, we can rewrite their associated generating density as follow:
Ix (x= a))
Kny F15)
(2)
20," the univariate gaussian kernel density estimator with smoothing pa- rameter hj- Such kernels clearly not adapted for asymmetrie, bounded discrete variables. This remark is also true for the work of Yang et al. (2021) which uses symmetric kernels to improve learning of imbalanced datasets.
with (u) (2m)-1/2hj1e
Non- Can-Classical Smoothed Bootstrap To fix the drawback of the classical kernel we extend 2 by adapting (1) to the support of æ, considering some non-classical kernels (we refer to some works handling the kernel density estimation for specific distributions inspired from Bouezmarni & Rombouts (2110) Someé
2015), Hayfield & Racine (2008).
Chen (2000). We rewrite
(w
())
Σ
wc II (4j,a.)
where Kn, (u, a univariate kernel adapted to the nature of the jth variable and specifically defined on as follows:
Gaussian kernel for variable defined on R (classical kernel):
t( -(===)²
1
8

---

Published in Transactions on Machine Learning Research (06/2024)
Binomial kernel for discrete variable defined on N:
(x + 1)! (x(u,z) ul(e + u)! Gamma kernel for a positive asymmetric distribution defined on (a, +00|
x h
1 2t1-u
(
u (r-o//
(u, r(1 + (x a)/l)// exp ( ) (w).
Negative Gamma kernel for negative asymmetric distribution defined on b]:
(z-b)/k
Ku x) T(1 (x b)/h)h² 22 b3/h exp 14 (T) 10 Beta kernel for a variable defined on [0, 1]:
Kx(u, 2//h (1 (z/1 (0.11(u).
B 1, 1-2 +1)
à
Truncated Gaussian kernel for variable defined on [a,
kx(u,2) a == [0),
hv22
= (J. - -1-20
Note that if the Dirac kernel (12==;;)) is used, we get the standard bootstrap: includes also the simple oversampling. It is important to note that the GOLIATH algorithm uses an estimation of the smoothing parameter h provided by some specific R-package dedicated to the density estimation (for instance, it uses the Silverman estimation for the Gaussian kernel). Their estimates are based on properties of univariate consistency. Another technique to deal with skewed heavy-tailed distributions to apply transformation of the data in order to use classical kernel density estimation (Charpentier & Flachaire (2015), Charpentier & Oulidi 2010)) but it necessitates proposing a relevant transformation which exceeds the scope of this paper.
REMARK 3.
The use of diagonal bandwidth matrix in 2 does not take into account the correlation between variables. To improve this issue, could consider full (symmetric positive definite) smoothing matriz. In that cose, we would use mallivariate kernel densily estimale: considering the correlation between the variables which would be optimal for generating data. However, the estimation of such a matrix is generally based on the covariance matriz which does not adequately capture non-linear correlations. In practice, it can challenging, even inconsistent, to find a form of a multivariate kernel that adapts to all data and their support. It interesting note that kernel functions can be linked to a wide range of statistical work, such graph-bused learning methods (Sarcheshmehl Pour et al. (2023))
4.3 Goliath Overview
The GOLIATH algorithm is summarized Figure A cartography GOLIATH is also given in Figure The algorithm gives the possibility choose, with the 'mode" parameter, the kind of the returned sample: full synthetic dataset, augmented one, mixed one. The mixed sample is constructed as follows: keep the original observation for the first occurrence of the seed and synthetic data for the next. This mode corresponds to performing an undersampling and an oversampling. More preciscly, to preserve the maximum of information and avoid potential overfitting, we suggest to: keep the initial observation for its first drawing and generating synthetic data from it for the other drawing which the "mix" mode in the GOLIATH algorithm. This technique theoretically helps in reducing bias (by keeping the real data) while avoiding overfitting (by not duplicating the same observations). More details on this option are given in A Appendix B
9

---

Published in Transactions on Machine Learning Research (06/2024)
4.3.1 GOLIATH Overall Algorithm
The GOLLATH algorithm summarized in Figure 4 and Figure I presents its cartogrophy. can be seen that the structure of GOLIATH that of generalized smoothed bootstrap, that is, it utilizes weighted kernel density estimators. GOLLATH encompasses the family of existing perturbation- based generators (such as ROSE and GN) which can be referred to the Classical Smoothed Bootstrap. This also includes what is known the SMOTE Family, which is very broad and encompasses many methods (see) Fernández et al. (0188)). We have the two generalizations of the SMOTE family that we propose: Nearest Neighbors Smoothed Bootstrap and Extended Nearest Neighbors Smoothed Bootstrap. We also find the Non- Classical Smoothed Bootstrap. with specific kernels that we propose in this paper. Finally, we have also added a proposed kernel that seems interesting but appears to be complicated to use practice: the Balloon kernel using the kNN density estimate.
GOLIATH Generalized Smoothed Bootstrop
Extended Nearest Neighbors Smoothed Boostrop
Classical Smoothed Bootstrop
Non Classical Smoothed Bootstrop
Balloon Kernel
KNIN Density Estimate Smoothed Boostrop
Other Clossial Smoothed Bootstrap
Nearest Neighbors Smoothed Boostrop
6amma Smoothed Bootstrop
Beta/truncated Gaussion Smoothed Bootstrop
ROSE
SMOTE Family
Binomial Smoothed Bootstrop
Goussian Noise
Family
Figure 1: GOLIATH algorithm and cartography
5 GOLIATH as Solution for Imbalanced Regression
In this section, we introduce new methodologies to deal with Imbalanced Regression
Using the first generator step of GOLIATH (methods proposed in 2.3.1 and 2.3.2) we generate synthetic covariates X". We then have to generate the target variable Y given such X".
A nonpaaraettric method for drawing weights We propose here to define the drawing weights in 1 as the inverse of the kernel densily estimate for the target variable Y: the more isolated an observation 1s, the higher drawing weight. Our approach comparable to the Inverse Probability Weighted Estimator (IPWE) survey sampling Hernán & Robins (2006). The IPWE is a statistical method used estimate parameters in situations where data come from a population different from the one targeted for inference. IPWE employs weights caleulated from the inverse probabilities of sampling events to adjust observations.
10

---

Published in Transactions on Machine Learning Research (06/2024)
often used to correct sampling biases and extrapolate sample results to the target population. To avoid giving disproportionate weights (lypically too large weight for extreme values), we can use hyperparameter a as below. The weights are normalized to get a sum equal to 1. For i = 1,- n, the normalized drawing weight W; of observation is thus defined as follows:
9i Wi == Σ
1
with qi == (ui)a
(3)
where fy denotes a convergent kernel density estimator.
In the following, we assume a = 1.
REMARK 4. With 1, the generated random variable Y satisfying P(Y* y wi is close uniform distribution. More precisely SStockiicker et al. (2023) generalized result of Sffith & Gelfand (1992) [o(ua] with fo a target probability density function (rdf), then the pdf of by showing that choosing qi fytue)
Y converges to fo- In our case fo is a uniform distribution and we can apply this result as soon as the support of Y bounded. In this way, we oblain a test sample with a distribulion of the largel variable Y approzimately uniform i. balanced. We find the same idea in Ren et al. (2022): 'Therefore, imbalanced regression assumes that the training set and test set are drown from different joint distributions, ptrain(x, y) and pbal(x. y) respectively, where the training set's label distribution ptrainly) is skewed and the balanced test set's i label distribution pbol(y) is uniform.'
In Vang et al. 2 the authors propose also using the inverse of the estimated label density to weight the loss function but apply a discretization of the target variable. The strength of our approach to preserve the continuity of the distribution whereas most other works propose splitting it into bins that involve loss of information. The kernel density is automatically adapted to the distribution of Y, proposed in Subsection
A new approach to generate the target variable as the covariate. Once the covariates are generated from our generator models, we propose to adapt a wild- bootstrap technique with synthetic features (Wul (1986) a The classical Wild-Bootssrrp involves uniformly drawing a prediction error ex to generale new D y1 + egUss where oa random variable.
The target variable is not generated in the same way
The generation of the target variable is performed as follows:
i) Train a Random Forest on the initial sample;
ii) For cach seed F, E Zg: prodict target variable T. associated to x, with the Random Forest; Es being the original observations drawn at step 1.
ili) Obtain the distribution of the absolute residuals on the prediction of the seed: ys D. and draw a randomly residual Ex:
iv) Generate the noise in the Wild-Boolstrap is ~ NN(,0 o), a being a parameter; vs can also be set to to avoid adding noise to the residual.
v) Generate a new y associated the new synthetic x; as follows:
y = y, + lex/v, sign(y. 1
With 1, this form is close to the Wild Bootstrap version with the Rademacher distribution. The idea behind this proposition is to consider taking into account i) the prediction error and ii) the impact of the synthetic covariate on the target variable.
The choice of using a Random Forest is justified by its good predictive performance, its non non- parametriii nature, and the possibility of getting an error distribution for a given target variable value. It represents the second step of GOLIATH. Other interesting methods to generate Y are proposed in Appendix B.2 (but giving lower numerical performances on our illustration, presented in Figure 5a). The algorithm is detailed in Appendix B.2
2The kernel regression (Nadaraya- Watson estimator) was also tested but not selected because high computation time and poor performance
11

---

Published in Transactions on Machine Learning Research (06/2024)
6 Application in Imbalanced Regression
To evaluate the performance of GOLIATH, we focus on the imbalanced regression context because of the natural capacity of the form [ to handle continuous variables.
For each dataset, we construct test sample 10-30% of the initial dataset and use the weighting previously defined: the inverse of the kernel density estimate for the target variable Y. We construct an artificial imbalanced dataset. from the remaining sample. More details on the protocol are given in Appendix C.2 One of the objectives of our approach is to obtain better performance than the imbalanced train dataset. We compare our results to existing methods to deal with imbalancod regression from the UBL R-package, Branco et al. (2016a), classical oversampling, SMOTE, Gaussian Noise, SMOGN, WERCS and ADASYN from the python-package mibbllnneddlaarinnggggeesss their automatic relevance function and the same parameters as GOLIATH if any, in particular k for SMOTE and pert for the Gaussian Noise.
(Wu et al. (2023) ). These techniques are used with
To avoid sampling effects and obtain a distribution of prediction errors we ran 10 train-test datasets. In the same way, to avoid getting results dependent on some learning algorithms we use 10 models from the autoML of the H2O R-package] TeDell & Poirier (2020) among the following algorithms: Distributed Random Forest, Extremely Randomized Trees, Generalized Linear Model with regularization, Gradient Boosting Model, Extreme Gradient Boosting and Fully-connected multi-layer artificial neural network. We present here the aggregated results of the models, a more detailed analysis is available in Appendix [and] D
We then compute the following metrics: RMSE and MAE and weighted-RMME with our weighting function giving more importance to the rare values. Since the test sample is balanced on the target variable, we considered here the RMSE and MAE metrics as relevant to provide an overview of the average error across the whole target variable.
REMARK 5. The basic methods (Gaussian Noise or SMOTE) are different from the UBL version because i) the generation of the largel variable Y is realized with wild boolstrap and considers the new synthelic attributes and üi) the weights w is defined for all samples while UBL uses a relevant function that divides the dataset into rore and frequent sets. Like the ROSE and Gaussian Noise algorithms, GOLIATH takes into accounl a porameter luning the level noise for perturbation approaches (descriplion in Appendir A). It is also possible to use clustering (Gaussian Mixture Model) in GOLIATH in order to apply generation by cluster. All datasets provided by GOLIATH in the opplications were provided with the "mix" mode. Note that the ROSE algorithm did not erist for imbalanced regression.
6.1 Illustrative Application
In order to get reference for predictive performance, we chose a balanced datasct from which we build an imbalanced train dataset. The dataset, named SML2010 Data Set is available on the Machine Learning repository U It is composed of 24 numeric attributes and 4137 instances. The target variable is the indoor temperature (we construct a unique target variable as the mean temperature of dinning-room and the temperature of the room). We train, with the autoML, the following train dataset:
Reference values: Full sample (FTrain), Imbalanced (Imb)
Benchmark: UBL- UBLL-Oversnglling ( (UBL-OS), UBL-SMOTE for regression (UBL-SMOTE), UBL- Gaussian Noise for regression (UBL-GN), UBL-SMOGN for regression (UBL-SMOGN), UBL- WERCS (UBI-WERCS), IRI-ADASYN (IRI-OS)
GOLIATH (step 1): Oversampling (G-OS), Gaussian Noise (G-GN), Gaussian Noise with GMM- clustoring ( (G-GNwCI), ROSE (G-ROSE), ROSE with a GMM-chnstering (G-ROSEWCI), SMOTE (G-SMOTE), Non classical Smoothed Bootsrap with contraints on the distributions (G-NCSB), Classical Smoothed Bootsrap (G-CSB), Nearest Neighbors Smoothed Bootstrap, with Beta distri- bution, ( (G-NNBB), Nearest Neighbors Smoothed Bootstrap with k-NN weights proportionates to the distance from the seed ( (G-NNSBw), Extended Nearest Neighbors Smoothed Bootstrap ( (G-eNNSB).
htttpe///archive.icr uci hwt.aiecccc...cce/aaaaateteeeeee edtp//////daaaasstt//0.0000
12

---

Published in Transactions on Machine Learning Research (06/2024)
Figure 2a shows the results for RMSE on the test sample. The weighted-RMSE and MAE metrics are shown in Appendix g and present similar results. We can observe that the GOLIATH algorithm presents an RMSE smaller than the imbalanced sample and than the benchmark techniques, whatever the generators. The GOLLATH--versampplnng is comparable to the UBL-oversampling which confirms, on this datasct, the relevance of the weighting. The Non-Classical Smoothed Bootstrap (NCSB) is as efficient as the Classical Smoothed Bootstrap (CSB, ROSE, GN). However, provides realistic values for the variables. We can see some examples of inconsistency in Appendix 5b The results show that the clustering seems to improve the performance. The Nearest Neighbors Smoothed Bootstrap (NNSB) and Extended Nearest Neighbors Smoothed Bootstrap(eNNSB) outperform the original SMOTE. It is important to note that the different pa- rameters (the number of nearest neighbors for interpolation techniques and the level of noise for perturbation techniques) are arbitrary and not optimized here. The healmap in Figure 2b shows the robust ness of the methods with their rank by run with respect to the RMSE. We can observe, based on the mean and standard deviation of the rank, that the Classic Smoothed Bootstrap, the Non-Clat -la-Saasic Smoothed Bootstrap, and the Nearest Neighbors Smoothed Bootstrap are the best approaches here.
Run HMSE
30
mean
ed
BASELINE
Ptrain
0,8
0.
0.7
0.0
08
0.7
0.5
0,8
0,9
0.0
0,8
0,1
imb
8
12
13
10
10
18
6
10
Imf
2,19
5,70
1,46
1,5
1,85
2,3
202
1,85
1,30
0.25
RMSE Rank
BENCHMARK
UOI-OS
15
15
17
16
14
UOL-05
2,23
1,80
1,59
1,82
2.12
1,63
2.38
2.22
2,07
1.85
2,02
0.22
UBUSMOTE
13
10
11
9
15
12
13
12
13
12
UN SMCTE
1,87
1,45
1,00
1.53
1.08
1.97
2.17
2.13
1,85
1.77
1.34
0.20
15
11
1
7.74
1,48
1.50
1,61
1.81
0.27
UBL-SMOON
12
10
16
15
15
14
14
14
14
4
15
1,81
1.52
UBL-WERCS
14
16
15
2,43
1,57
1,50
1,59
2.08
1.68
2.24
2.05
2.13
1.86
1,87
0.27
IRL-DAAYNN
38
28
18
37
28
RL AAASYM
2,63
2,26
2,37
2.69
2.58
2.31
60UATH
2.50
2.57
2.44
2.33
2.38
0.20
RMSE Rank
GOUATH
0:08
3,13
3,85
1,67
1,36
2,30
2.36
2.15
1.23
0.20
G:05
12
13
17
G-GN
2,00
1,53
1,22
1,48
1.76
1,49
1,57
1.52
1,61
1.65
1,65
0.25
G-GN
11
10
70
10
3,03
3.03
1,25
1.34
1,59
3,62
0.29
G-ENNN
G-ROSE
5
15
6
3
5
8
G-Rostwcl
3,34
3,00
3,29
3,41
3.43
1,79
1,27
3,72
1,80
1,65
0.22
G.ROSF wCl
2
5-SMOTE
1.85
1,80
1,81
1,50
1,55
1,75
2.25
2.67
1,30
1,83
1,34
0.39
SMOTE
17
G-Ncon
3,19
1,27
1,50
1,40
1,53
1.28
0.26
G-NCSA
13
6
3
5.
11
10
10
1,08
1.72
1.52
6:C3B
G-MMSB
1,67
1,36
1,57
1,30
1.77
1.58
1.79
1.29
1.85
1.00
1,85
0.37
G-NN-B
1
11
5-MASM
1,67
1,87
1,45
1,46
1,84
1
1,51
1.40
1,64
1.74
1,68
0.38
G-NNS5
16
12
GeNN58
2.20
1,79
1,29
1.63
1.48
G-INNSS
(a) RMSE Heatmap
(b) RMSE Ranking
Figure 2: Numerical simulation. RMSE values and ranking for the full sample, the imbalanced sample, 6 competitors, and GOLIATH associated with 11 different generating methods
The RMSE-rank represents the ranking of approaches according to the RMSE for a run: rank corresponds to the training dataset that offers the smallest RMSE on the test sample. We also compared the results using the R-package IRon: Solving Imbalanced Regression Tasks D a useful and relevant package specific to Imbalanced Regression based on Ribeiio & Moniz (2020). The results in Appendix [C6 demonstrate that GOLIATH outperforms UBL approaches, even when considering their performance metrics (weighted MSE, weighted MAE, and SERA).
6.2 Imbalanced Regression Applications
We test our approach on several real data set from repository provided as a benchmark for imbalanced regression probleme and presented in Branco el al. (2019) (descriptions in Appendix Dj. Figures 3a and 3b present RMSE gain (wrt the imbalanced dataset) and the median of the RMSE ranking. We can observe on these datasets that the GOLIATH algorithm empirically outperforms the state-of-tho-art techniques, especially the Non-Classical Smoothed Bootstrap and the Extended Nearest Neighbors Smoothed Bootstrap.
We can see on these several applications, with several runs, several learning algorithms, and several perfor- mance metrics that the GOLIATH approach seems relevant to deal with imbalanced regression. In general GOLIATH gives better results, especially when it is combined with and extended nearest neighbors smoothed bootstrap in its first step of covariates generation.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Dataset
NO2
cpuSm
BOston BENCHMARK
Bank3FM
Abalone
Dataset
NO2
cpu5m
Boston
Bank3FM
Abalone
RMSE gain
RMSE rank
BENCHMARK
UBL-OS
LIB -S
14,0
14,5
13,0
17,0
16,5
UBL-SMOTE
11,5
-53
-7%
-3%
UBL-GN
UBL-SMOGN
-10%
-3%
-10%
29%
-3%
UBL-SMOGN
12,0
12,0
UBL-WERCS
-4%
3%
-1%
57%
-1%
UBL-WERCS
14,0
16,0
15,5
15,0
IRL-ADASYN
10%
2256
-1%
69%
NA
IRL ADASYN
18,0
18,0
16,0
18,0
NA
RMSE gain
GOLIATH
RMSE rank
GOLIATH
G-05
-1%
-2%
-3%
57%
-3%
G-O5
16,0
14,0
14,5
15,5
13,5
11%
18%
99
G-GN
5,5
2,5
6,5
G-GNwCI
-10%
-18%
-14%
G-GNwCI
G-ROSE
G-ROSE
4,0
G-ROSEwCI
G-SMOTE
-2%
-6%
54%
-11%
G-SMOTE
11.5
12,5
4,5
G-NCSB
-23%
-23%
-6%
-9%
G-NCSB
2,0
4,0
5,0
5,5
G-CSB
-12%
-20%
-17%
0%
-9%
G-CSB
3,5
6,0
5,0
6,0
G-NNSB
-33%
-11%
-12%
31%
-14%
G-NNSA
6,0
8,5
8,0
12,0
2,5
G- G-NNSBW
-15%
-11%
-14%
35%
-13%
G-NNSBw
3,0
7,0
6,5
12,0
3,5
G GNNNNSB
-15%
-7%
-19%
-9%%
-21%
G-eNNSB
2,0
10,0
5,5
1.5
1,0
(a) RMSE-gain
(b) Median of the RMSE-rank
Figure 3: Datasets. RMSE values and ranking for 6 competitors, and GOLIATH associated with 11 different generating methods
6.3
Discussion
Based on all our numerical evidences, we strongly recommend the use of GOLIATH when step 1 combines a smoothed Bootsrap with ncarest neighbors. The five versions of GOLIATH involving bootstrap procedures appear to be the best performing and most stable in terms of numerical results (both RMSE gain and rank).
7 Discussion and Perspectives
GOLIATH is algorithm gathering two large families of synthetic data oversampling. Many methods be rewritten as particular cases of it. This approach gets the advantage to obtain general form for the generator which is based both on the theoretical foundations of kernel estimators and classical smoothed bootstrap techniques. It provides general expression for the conditional density of the generator. The use of well-chosen kernels makes it possible to take into account the nature of the covariates: continuous, discrete, lotally or partially bounded. Our approach generalizes the SMOTE algorithum by providing weights and flexible densities for interpolation. We also extend this technique wider support than that of the observations by combining interpolation and perturbation approaches. Numerical applications in imbalanced regression models demonstrate that GOLIATH and its variants are very competitive, especially when the generator used in step 1 is the extended nearest neighbors smoothed bootstrap.
The weights wi (and Tyle in the interpolation case) offer a large flexibility. For instance, it is possible to handle classification tasks by conditioning with the minority class. We could deal with multi-class classification too. It also possible to combine some extensions of SMOTE that propose focus on specific samples in the synthetic data generation (as ADASYN) with a kernel approach in order to perform the methodology.
As perspective, natural extension of this work is to automate the choice of the kernel estimators, the weights, well
parameters according to the data. For example by defining a weights function for the nearest neighbor instead of defining the parameter k. Indeed, the parameter & is sometimes unsuitable and we could suggest. a dynamic weighting depending on the neighborhood. It. is also possible to define a kernel according to the neighborhood into the same dataset. For instance, an interpolation approach is favored within clusters when neighboring points are considered close to the obscrvation. On the other hand, perturbation approach is preferred when the observation appears isolated. Finally, non-standard kernels enable handling specific distributions such bounded or discrete ones.
14

---

Published in Transactions on Machine Learning Research (06/2024)
We also could define w, in order to generate target distribution done in Stocksieker et al. (023 Finally, the perturbation- ion-based approaches, based on kernel density estimators, may find it challenging to accurately capture dependencies between variables. The interpolation approaches consider it but the generation is limited to the segments. The extended-SMOTE proposes a first solution. GOLIATH proposes also an innovalive method to generale Y based on the generated X, regardless of the generator used. It. would also be interesting and potentially effective use multiple generators and capitalize on the strengths of each. The generators could be applied locally based on the data characteristics.
Another research direction would be to better consider i) correlations between variables while respecting their definition domain and ii) mixed data. Finally, it could be interesting to test GOLIATH on image datasets, by combining it with deep-learning model (Deep Imbalanced Regression framework).
Acknowledgment
The authors thank the reviewers for their useful and relevant comments that significantly improved the manuscript. S. Stocksieker would like to acknowledge the support of the Research Chair DIALog under the aegis of the Risk Foundation, joint initiative by CNP Assunance. D. Pommeret would like to acknowledge the support received from the Research Chair ACTIONS under the aegis of the Risk Foundation, an initiative by BNP Paribas Cardif and the Institute of Actuaries of France.
References
Gustavo EAPA Batista, Ronaldo C Prati, and Maria Carolina Monard. A study of the behavior of several methods for balancing machine learning training data, ACM SIGKDD explorations newsletter, 6(1):20 2-2 2004.
Gérard Biau and Lue Devroye. Lectures on the nearest neighbor method, volume 246. Springer, 2015.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEFE Transactions Newral Networks and Learning Systems, 2022.
Taoufik Bouezmarni and Jeroen VK Rombouts. Nonparametric density estimation for multivariate bounded data. Journal of Statisticol Planning and Inference, 140(1):139-155) 152, 2010.
Adrian W. Bowman and Adelchi Azzalini. Applied smoothing techniques for data analysis the kernel approach with s-plus illustrations. Journal of the American Statistical Association, 94:982, 1999.
Paula Branco. btsse Predictive Analytics. PhD thesis, Universidades do Minho, Aveiro e Porto, 09 2018.
Paula Branco, Rita P Ribeiro, and Luis Torgo Ubl: an package for utility-based learning, arXiv preprint arXiv:1604.08077 2016a.
Paula Branco, Luis Torgo, and Rita P Ribeiro. A survey of predictive modeling on imbalanced domains. ACM computing surveys (CSUR), 49(2):1-50, 2016b.
Paula Branco, Luís Torgo, and Rita Ribeiro. Smogn: pre-processing approach for imbalanced regression.
First international workshop on learning with imbalanced domains: Theory and applications, pp. 36 50. PMLR, 2017.
Paula Branco, Luis Torgo, and Rita Ribeiro. Pre-processing approaches for imbalanced distributions regression. Neurocomputing, 343:76 99, 2019.
M. Buda, A. Maki, and M. A. Mazurowski. A systematic study of the class imbalance problem in convolu- tional neural networks. Neural Networrks, 106:249-22 259, 2018.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Chumphol Bunkhumporppat, Krung Sinapiromsaran, and Chidchanok Lursinsap. Safe-level-smote: Safe- level-synthetic minority over-sampling technique for handling the class imbalanced problem. In Advances in Knowledge Discovery and Data Mining: 13th Pacific-Ass Conference, PAKDD 2009 Bangkok, Thailand, April 27-30, 2009 Proceedings 13, pp. 475-482. Springer, 2009.
Luís Camacho, Georgios Douzas, and Fernando Bacao. Geometric smote for regression. Expert Systems with Ayplicalions, pp. 116387, 2022.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distrbbution-awaae margin loss. Aduances nearal information processing systems, 32, 2019.
Arthur Charpentier and Emmanuel Flachaire. Log-transform kernel density estimation of income distribu- tion. L L'Actualité économique, 91(1):1411153 2015.
Arthur Charpentier and Abder Oulidi. Beta kernel quantile estimators of heavy-tailed loss distributions. Statistics and computing, 20(1):35-55, 2010.
Chawla, Bowyer, Hall, and Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321 357, 2002.
Song Xi Chen. Probability density function estimation using gamma kernels. Annals of the Institute of Stalistical Mathemalics, 02:471-4801 2000.
Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and Belongie. Classbalanced loss based on effective number of samples. CVPR, 2019.
Daniela De Angelis and G Alastair Young. Smoothing the bootstrap. International Statistical Review Review/Revue Reevee Internationale de Statistique, pp. 45-56, 1992.
Daniele De Martini and Fabio Rapallo. On multivariate soothhee bootstrap consistency. Journal of statistical planning and inference, 138(6):1828 1835, 2008
Yifei Ding, Minping Jia, Jichao Zhuang, and Peng Ding. Deep imbalanced regression using cost-sensitive learning and deep feature transfer for bearing remaining useful life estimation. Applied Soft Compuling, 127:109271, 2022.
Georgios Douzas, Fernando Bacao, and Felix Last. Improving imbalanced learning through a heuristic oversampling method based on k-means and smote. Information Sciences, 465:1-20, 2018
Tarn Duong and Martin L Hazelton. Cross-validation bandwidth matrices for multivariate kernel density estimation. Scandinavian Journal of Statistics, 32(3):485 506, 2005.
Dina Elreedy, Amir F Atiya, and Firuz Kamalov. A theoretical distribution analysis of synthetic minority oversampling technique (smote) for imbalancod learning. Machine Learning, pp. 1-21, 2023.
M Falk and R-D Reiss. Weak convergence of smoothed and nonsmoothed bootstrap quantile estimates. The Annals of Probability, pp. 362-371, 1989.
Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawezyk, and Francisco Her- rera. Learning from imbalanced data sets, volume 10. Springer, 2018a.
Alberto Fernández, Salvador Garcia, Francisco Herrera, and Nitesh V Chawla. Smote for learning from im- balanced data: progress and challenges, marking the 15-year anniversary, Journal of artificial intelligence research, 61:863-905, 2018b.
Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. arXiv preprint orXiv:2205. 2022.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Newral Information Processing Systems, 118882-18844 2021.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural informalion processing systems, 35:507-520, 2022.
Xiaowei Gu, Plamen P Angelov, and Eduardo A Soares. A self-adaptive synthetic over-sampling technique for imbalanced classification. International Journal of Intelligent Systems, 35(6):923-99) 943,, 2020.
Peter Hall, Thomas J DiCiccio, and Joseph P Romano. On smoothing and the bootstrap. The Annals of Statistics, pp. 692 704, 1989.
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderlino-smote: a new over-sampling mothod in imbal- anced data sets learning. In Advances in Intelligent Computing: International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part. I 1, pp. 578-887, Springer, 2005.
Tristen Hayfield and Jeffrey S Racine. Nonparamctric econometries: The np package. Journal of statistical software, 27:1-32, 2008.
Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on newral networks (IEEE world congress computational intelligence), pp. 1322-1328. IEEE, 2008.
Miguel A Hernán and James M Robins. Estimating causal effeets from epidemiological data. Journal of Epidemiology & Communily Health, 60(7):578 586, 2006.
C. Huang, Y. Li, C. Change Loy, and X. Tang. Learning deep representation for imbalanced classification. CVPR, 2016.
Yang Huang, Duen-Ren Liu, Shin-Jye Lee, Chia-Hao Hsu, and Yang-Guang Liu. A boosting resampling method for regression based on a conditional variational autoencoder. Information Sciences, 590:90 105, 2022.
Liran Katzir, Gal Elidan, and Ran El-iv..v. Net-dnf: Effective deep modeling of tabular data. In Interna- lional conference on learning representalions, 2020.
Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, ((4):221-232) 2016.
Erin LeDell and Scbastien Poirier. H2O AutoML: Scalable automatic machine learning. 7th ICML Workshop on Automated Machine Learning (AutoML), July 2020. URL https://ww. ftttps://wwwauutomm.orog aut.ml. wwwp:owww...//poo-///oooooe contentt
uploads/2020//00 SaatttoM_2220_p___ 20000__ppe___6661111122222222220 6df
Lee and Sauchi. Noisy replication in skewed binary classification. Computational Statistics and Data Analysis, 191, 2000.
Chao Ma, Sebastian Tschiatschek, Richard Turner, José Miguel Hernández-Lobato and Cheng Zhang, Vaem: deep generative model for heterogeneous mixed type data. Advances in Neural Information Processing Systems, 33:1 13:11277-12247, 2020.
Menardi and Torelli. Training and assessing classification rules with imbalanced data. Dala Mining and Knowledge Discovery, 28(1):92 122, 2014.
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 7926-7935. 2022.
Rita P Ribeiro. Ubility-based regression. Ph. D. dissertation, 2011.
Rita P Ribeiro and Nuno Moniz. Imbalanced regression and extreme value prediction. Machine Learning, 109:1803-1888 1835, 2020.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Abdoulaye Sakho, Erwan Scornet, and Emmanuel Malherbe. Theoretical and experimental study of smote: limitations and comparisons of rebalancing strategies. arXiv preprint arXin:2402.03811 0338199 2024.
Yasmin SarcheshmehPour, Yu Tian, Linli Zhang, and Alexander Jung. Clustered federated learning via generalized total variation minimization. IEEE Transactions Signal Processing, 2023.
David W Scott. Mullivariale densily eslimation: theory, practice, and visualization. John Wiley & Sons, 2015.
Snigcha Sen, Krishna Pratap Singh, and Pavan Chakraborty. Dealing with imbalanced regression problem for large dataset using scalable artificial neural network. New Astronomy, 99:101959, 2023.
Tra Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets. Advances Neural Information Processing Systems, 31, 2018.
Ravid Shwartz-Zin and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84 90, 2022.
Bernard W Silverman. Densily estimalion for statistics and dala analysis, volume 26. CRC press, 1986.
BW Silverman and GA Young. The bootstrap: to smooth or not to smooth? Biometrika, 74(3):469 479, 1987.
Adrian FM Smith and Alan E Gelfand. Bayesian statistics without tears: sampling-reamppinn perspective. The American Statistician, 46(2):84-88, 1992.
Sobom Matthieu Someé. Estimalions non paramébriques por noyaur associés mullivariés el applications. PhD thesis, Université de Franche-Comté 2015.
Xin Yue Song, Nam Dao, and Paula Branco. Distsmogn: Distributed smogn for imbalanced regression prob- lems. In Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications, pD. 38 52. PMLR, 2022.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density- based weighting for imbalanced regression. Machine Learning, 110:2187-2211, 2021.
Samuel Stocksicker, Denys Pommeret, and Arthur Charpentier. Data augmentation for imbalancod regres- sion. In International Conference on Artificial Intelligence and Statistics, pp. 7774 7799. PMLR, 2023.
Bo Tang and Haibo He. Kerneladasyn: Kernel based adaptive synthetic data generation for imbalanced learning. In 2015 IEEE congress on evolutionary computation (CEC), pp. 664-671. IEEE, 2015.
Tomek. Two modifications of cnn. IEEE Transactions on Systems, Man, and Cybernetics, SMC-6(11): 769 772, 1976. doi:
Luis Torgo and Rita Ribeiro. Utility-based regression. In PKDD, volume 7, pp. 597 604. Springer, 2007.
Luis Torgo and Rita Ribeiro. Precision and recall for regression. In Discovery Science: 12th International Conference, DS 2009, Porto, Portugal, October 3-5, 2009 12, pp. 332-346. Springer, 2009.
Luís Torgo, Rita P Ribeiro, Bernhard Pfahringer, and Paula Branco. Smote for regression. In Portuguese conference artificial intelligence, pp. 378 389. Springer, 2013.
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via proba- bilistic smoothing. Advances in Neural Information Processing Systems, 36, 2024.
Dennis L. Wilson, Asymptotic properties of nearest neighbor rules using odited data. IEEE Transactions on Systems, Man, and Cybernelics, SMC-2(3):408 421, 1972. doi: 10.1009/TEMC.119244301..
C. F. J. Wu. Jackknife, Bootstrap and Other Resampling Methods in Regression Analysis. The Annals of Statistics, 14(4):1261 1295, 1986. doi: 10.1214/a0s//11665000 URL https:///oi. https://doi.org/10.. org/10. 1214/aos/ 1176350142
18

---

Published in Transactions on Machine Learning Research (06/2024)
Wenglei Wu, Nicholas Kunz, and Paula Branco. Imbbllnccllerrinngggesssoooo python package to tackle the imbalanced regression problem. In Machine Learning and Knowledge Discovery in Datahases: European Conference, ECML PKDD 2022, Grenoble, Prance, September 19 23, 2022, Proceedings, Part VI, pp. 648. Springer, 2023.
Y. Yang and Z. Xu. Rethinking the value of labels for improving dass-imbalanced learning. NeurIPS, 2020.
Yuzhe Yang, Kaiwen Zha, Fing-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression, 2021.
Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving general- ization in regression. Advances in Newral Information Processing Systems, 35:3361-3333 3376, 2022.
19


Reviews:
[{"review_id": "qE1DjK6Hse", "reviewer": "Reviewer_GhSw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a good idea, but the execution and presentation of this idea has serious problems. The paper is not in a publishable state, but with the requisite edits it could be.\n\nStrengths:\n- In terms of theoretical contributions: the paper provides a general framework for thinking about minority oversampling techniques. It shows how popular methods like SMOTE, GN, and ROSE specialize this more general kernel-based framework. The connection with SMOTE is perhaps the least obvious and made precise in Section 4.1.\n  - NOTE for Section 4.1: Notation (regarding the seed S) is inconsistent when rewriting SMOTE in terms of GOLIATH; sometimes S is the index and sometimes it is the point\n- Section 4.3 communicates effectively the different drawbacks between SMOTE and ROSE, and positions GOLIATH as something which, through generalization of the kernel, strikes a compromise.\n- Figures 2 and 3 illustrate that GOLIATH-based methods for the kernel sampling technique work better than SMOTE and other existing methods for the presented cases.\n \nWeaknesses:\n- The overall paper is surprisingly poorly written. As a few examples: What is an imbalanced dataset in the context of regression is not even formally defined, description of interpolation approaches (Section 4.1) is unclear - what is “1” function (usually it is an indicator function but is it used differently, Section 5.3.1 suddenly starts discussing clustering? \n- Section 3 is missing. All that is there is the title, “A New Kernel-Based Oversampling Formulation.”\n- Lack of systematic theory. The paper title says “Associated Theory”, but there are hardly any theoretical results in the paper. Consistency of the proposed technique is handwaived by stating that it “can be considered as smooth bootstrap methods”.\n- Section 5.3.1: The GOLIATH algorithm description is hard to follow. The font is small. The connection to the cartography next to it is unclear. Abbreviations like NCSB and NNSB and eNNSB should have been established in section 5, and it should have been made clearer that GOLIATH chooses one of these as a kernel setting. In fact, choosing between kernel settings seems to be the only thing communicated by this algorithm description: this might be better communicated in an itemized list or plain text. Furthermore, all of these subroutines called in the algorithm description ought to be made more precise somewhere in the text.\n- Lack of detailed/systematic experimental analysis. There is practically no attempt made to understand how the proposed technique perform and scales with key parameters of practical dataset (eg number of training samples, dimensionality, level of ‘imbalance’, relationship between covariates and response, noise levels, etc).\n- Lack of Runtime Analysis: Extending from nearest neighbors to support of the whole dataset potentially could be costly (See Section 5, subtitle “Extended Nearest Neighbors Bootstrap”). Same with training random forest regressor. Runtime of algorithms is only mentioned in Remark 2 and a footnote at the bottom of page 8. \n- Figures 2 and 3 should do a better job of distinguishing between GOLIATH and previous methods. Also, is there any argument in the paper given for why we choose these particular datasets and whether they have any salient differences in size or geometry?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Class imbalance is a common problem in supervised learning. One way to alleviate class imbalance is data augmentation, by adding synthetic data points from the minority class. \n\nThis paper presents a generalization of well-known data augmentation techniques SMOTE (Synthetic Minority Oversampling Technique), ROSE (Randomly Oversampling Examples), and GN (Gaussian noise) in the context of imbalanced regression (i.e. regression in the presence of rare extreme values). The idea is that all of these augmentation approaches involve: picking some “seed” point, identifying its nearest neighbors, and sampling from some distribution which is effectively a (convex) combination of kernel densities from the neighbors. The proposed algorithm GOLIATH (Generalized Oversampling for Learning from Imbalanced Datasets and Associated Theory) optimizes over various choices of kernel and the weighting and achieves better performance in practice. A loose theoretical explanation is provided: GOLIATH finds a sort of middle ground between “interpolation techniques” like SMOTE and “perturbation techniques” like ROSE and GN. \n\nMore specifically, GOLIATH operates as follows:\n- Sample a seed s by looking at the inverse of the kernel value. The smaller this kernel value is, the farther the point is from its neighbors, the more of an isolated observation it is. So we are more likely to oversample these outlier points. \n  - Notably, this method avoids discretizing the support of the target variable, which might be the more “obvious” approach to approaching imbalanced regression. \n- Then generate the corresponding target y*_s by (1) using a random forest regressor to obtain a distribution on the residuals and (2) draw a random residual, scale it by a Gaussian, and add this to the y_s.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the weaknesses that are discussed in the previous section.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sBEpft3fG2", "reviewer": "Reviewer_toEQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Authors tackle a somewhat under-explored topic, i.e., imbalanced regression problems. \n\nMy main concern is the poor quality of presentation and use of language. First of all, the contributions are not explained precisely enough in Section 1. It seems that the main contribution is the GOLIATH algorithm. However, it is not clear what the benefits of this algorithm are, e.g., in terms of computational and statistical aspects. Moreover the GOLIATH algorithm needs to be discussed in much more detail and should also be formulated in more self-contained fashion. \n\nSome statements are quite vague and should be made more precise (e.g., \"..measuring imbalance in regression is challenging because the data is continuous\") \n\nSection 2 is quite short and maybe integrated in Section 1. Section 3 seems to be empty. \n\nEach Section should start with an introduction that provides an overview of its contents and how the section build on/serves other sections. \n\nThe authors should also try to obtain theoretical performance guarantees for some simple probabilistic model for the datasets (similar in spirit to the analysis of clustered federated learning in [Ref1]) \n\nIt might be useful if authors could try to put their work into context of graph-based learning methods. Indeed, the kernel function in (1) could be interpreted as a variant of an adjacency matrix for a graph. \n\n[Ref1] Y. SarcheshmehPour, Y. Tian, L. Zhang and A. Jung, \"Clustered Federated Learning via Generalized Total Variation Minimization,\" in IEEE Transactions on Signal Processing, vol. 71, pp. 4240-4256, 2023, doi: 10.1109/TSP.2023.3322848.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Author introduce a novel method for data augmentation, coined GOLIATH algorithm, that is particularly well-suited for imbalanced regression. The usefulness of the proposed methods is evaluated mainly empirically by numerical experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uXiwdXL3DX", "reviewer": "Reviewer_1Gej", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* Broadly, the paper is well-presented. Aside from some notational confusion (see below), it is easy to follow.\n* Relevant alternatives to parts of the approach (such as k-nn-based KDE) are attempted.\n* Many of the experimental decisions are based on precedent set by prior work.\n\n## Weaknesses\n\n- In Section 1, \"Very few works have addressed the problem of imbalanced regression although many important real-world applications in different fields such as economy, meteorology, or insurance\"--needs refactoring.\n- In Section 2, the authors claim, \"However, the techniques proposed in this context rely on deep learning, which is effective for images but less so for tabular data\", although that seems debated in literature: [1, 4, 5] seem to disagree, while [2-3] agree, though it should be noted that [3] specifically excluded datasets of high dimensionality.\n- Is Section 4 supposed to be Section 3.1?\n- Equation 1 is a bit unclear: since $\\tilde{x}$ represents all the observations, it is unknown at this point whether the kernel is over a matrix (of $n$ observations with $p$ dimensions), or if there should be another summation.\n- Continuing from the previous point, this paper would benefit from a Notation section. For example, in Section 4.1, it is unclear what some terms such as $x_{ij}(\\ell)$ mean. My working assumption is that $x_{ij}$ is the $j$th component of a data point, given that the $i$th data point was randomly chosen, and that $x_{ij}(\\ell)$ is the $j$th component of the $\\ell$th nearest neighbor, given that the $i$th data point was chosen.\n- Some of the links to the appendix seem incorrect. For example, for the evaluation protocol mentioned in Section 7, the paper points to \"Appendix 6\", which likely refers to Appendix D.1.\n- In the protocol for drawing training/test sets, it's not clear to me why the different weighting is necessary (and experimentally valid). Throughout the paper and the appendix, the authors refer to the use of the inverse of the KDE (or its squared) to draw extreme values, and the motivation for doing this is unclear.\n\n## References\n\n[1] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. *Advances in Neural Information Processing Systems*, *34*, 18932-18943.\n\n[2] Borisov, V., Leemann, T., Seßler, K.,  Haug, J., Pawelczyk, M., & Kasneci, G. (2022). Deep neural networks  and tabular data: A survey. *IEEE Transactions on Neural Networks and Learning Systems*.\n\n[3] Grinsztajn, L., Oyallon, E., &  Varoquaux, G. (2022). Why do tree-based models still outperform deep  learning on typical tabular data?. *Advances in Neural Information Processing Systems*, *35*, 507-520.\n\n[4] Katzir, L., Elidan, G., & El-Yaniv, R. (2020, October). Net-dnf: Effective deep modeling of tabular data. In *International conference on learning representations*.\n\n[5] Shavitt, I., & Segal, E. (2018). Regularization learning networks: deep learning for tabular datasets. *Advances in Neural Information Processing Systems*, *31*.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents GOLIATH, a kernel-based generalization of several oversampling methods commonly used in the literature, with the goal of applying it to regression problems, which is less common in the oversampling literature. Starting with a very general form of data synthesis, the authors reframe SMOTE and other approaches in this form. Having shown this, the authors then develop more general versions of these approaches. For regression tasks, the paper proposes several kernels based on each feature's idiosyncrasies. Experiments are provided on 5 datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Some minor editorial fixes (see Strengths & Weaknesses above)\n* More motivation is necessary to justify the inverse KDE rule used in the paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies regression under data imbalance, and presents and evaluates a novel algorithm for this problem. The authors position their algorithm among other oversampling-based methods via a general framework of kernel density estimation. This also allows the authors to motivate the choice of inverse KDE weights for their algorithm. Empirical results evaluate the proposed method on a benchmark of datasets with imbalance. The paper contained minor issues with grammar, terminology and notation, all of which seem to have been addressed. As requested by the reviewers, the authors also included discussions on computation time, presented a better motivation of the algorithm, and clarified the contributions.

The authors yet need to address the concerns raised by one of the reviewers on providing a more detailed/systematic experimental analysis. Concretely, for the final version of the paper, this can be achieved by adding empirical results on how the proposed technique performs and scales with key parameters of the datasets: number of training samples, dimensionality, level of imbalance (potentially through KDE weights), relationship between covariates and response, noise levels. These results are critical in order to ensure that the proposed method is properly evaluated, with an accurate assessment of strengths and weaknesses.
Decision: Accept (poster)