On Embeddings for Numerical Features in Tabular Deep Learning
Authors: Yury Gorishniy
Ivan Rubachev
Artem Babenko
Venue: NeurIPS 2022 Accept (2022)
Keywords: tabular data
deep learning
neural network
architecture
DNN

Abstract: Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with gradient boosted decision trees (GBDT) on some GBDT-friendly benchmarks (that is, where GBDT outperforms conventional DL models). We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL. The source code is available at https://github.com/Yura52/tabular-dl-num-embeddings

On Embeddings for Numerical Features in Tabular Deep Learning
Yury Gorishniy Yandex
Ivan Rubachev HSE, Yandex
Artem Babenko Yandex
Abstract
Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MML,, these architectures map scalar values numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL. which allows constructing more powerful DL models and competing with gradient boosted decision trees (GBDT) on some GBDT-friendly benchmarks (that is, where GBDT outperforms conventional DL models). We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention- based architectures. Overall, we highlight embeddings for numerical features as important design aspect with good potential for further improvements tabular DL. The source code is available at httpt:: // wtt.tt....oootttteeeeeeeeee coom/Yura52/tabul tatplarabaaaaaaaeeeeeeeeaaaaaaaa dl num embeddings
Introduction
Tabular data problems are currently final frontier for deep learning (DL) research. While the most recent breakthroughs in NLP, vision, and speech are achieved by deep models [12], their success in the tabular domain is not convincing yet. Despite large number of proposed architectures for tabular DL 2
I7
31 39
40 the performance gap between them and the "shallow"
ensembles of decision trees, like GBDT, often remains significant [13 (6].
The recent line of works 13
reduce this performance gap by successfully adapting the 45 for the tabular domain. Compared to traditional models, like MLP or ResNet, the proposed Transformer-like architectures have specific way to handle numerical features of the data. Namely, they map scalar values of numerical features to high-dimensional embedding vectors, which are then mixed by the self-attention modules. Beyond transformers, mapping numerical features to vectors was also employed in different forms in the click-through rate (CTR) prediction problems g 14 40]. Nevertheless, the literature is mostly focused on developing more powerful backbones while keeping the design of embedding modules relatively simple. In particular, the existing architectures [13] using quite restrictive parametric mappings, e.g., linear functions, which can lead to suboptimal performance. In this work, we demonstrate that the embedding step has substantial impact on the model effectiveness, and its proper design can significantly improve tabular DL. models.
24
39
Transformer architecture
39 40 construct embeddings for numerical features
"Correspondence to: yuragorishmiiyicclod...
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Specifically, we describe two different building blocks suitable for constructing embeddings for numerical features. The first one a piecewise linear encoding that produces alternative initial repre- sentations for the original scalar values and is based on feature binning, a long-existing preprocessing technique [1]]. The second one relies on periodic activation functions, which is inspired by their usage in implicit neural representations [28] 38 42 NLP 41 45] and CV tasks [25]. The first approach is simple, interpretable and non-differentiable, while the second demonstrates better results on average. We observe that DL models equipped with our embedding schemes successfully compete with GBDT on GBDT-friendly benchmarks and achieve the new state-of-the-arr on tabular DL.
As another important finding, we demonstrate that the step of embedding the numerical features is universally beneficial for different deep architectures, not only for Transformerllkk ones. In particular, we show, that after proper embeddings, simple MLP-like architectures often provide the performance comparable to the state-of-the-an attention-based models. Overall, our work demonstrates the large impact of the embeddings of numerical features on the tabular DL, performance and shows the potential of investigating more advanced embedding schemes future research.
To sum up, our contributions are as follows:
1. We demonstrate that embedding schemes for numerical features are underexplored research question in tabular DL. Namely, we show that more expressive embedding schemes can provide substantial performance improvements over prior models.
2. We show that the profit from embedding numerical features is not specific for Transformer- like architectures, and proper embedding schemes benefit traditional models well.
3. On a number of public benchmarks, we achieve the new state-of-the-an on tabular DL..
2 Related work
Tabular deep learning. During several recent years, the community has proposed large number of deep models for tabular data 2
31 39, 40 46). However, when systematically evaluated, these models do not consistently outperform the ensembles of decision trees, such as GBDT (Gradient Boosting Decision Tree)
3
13
15
17
21
24
7 I9 32). which are typically the top-choice in various ML
competitions [13 36). Moreover, several recent works have shown that the proposed sophisticated architectures are not superior to properly tuned simple models, like MLP and ResNet [3] In this work, unlike the prior literature, we do not aim propose
backbone architecture. Instead, focus on more accurate ways to handle numerical features, and our developments can be potentially combined with any model, including traditional MLPs and more recent Transformer-I like ones.
Transformers in tabular DL. Due to the tremendous success Transformers for different domains 10 45. several recent works adapt their self-attention design for tabular DL. as well [3 17 24 39
Compared to existing alternatives, applying self-attention modules to the numerical features of tabular data requires mapping the scalar values of these features to nigh-dimensional embedding vectors. So far, the existing architectures perform this "scalar" - "vector" mapping by relatively simple computational blocks, which, in practice, can limit the model expressiveness. For instance, the recent FT- Transformer architecture [13] employs only a single linear layer. In our experiments, we demonstrate that such embedding schemes can provide suboptimal performance, and more advanced schemes often lead to substantial profit.
CTR Prediction. In CTR prediction problems, objects are represented by numerical and categorical features, which makes this field highly relevant to tabular data problems. In several works, numerical features are handled in some non-trivial way while not being the central part of the research [8] 40
Recently, however, a more advanced scheme has been proposed in Guo et al. [14]. Nevertheless, is still based on linear layers and conventional activation functions, which we found to be suboptimal in our evaluation.
Feature binning, Binning is a discretization technique that converts numerical features to categorical features. Namely, for given feature, its value range is split into bins (intervals), after which the original feature values are replaced with discrete descriptors (e.g. bin indices or one-hot vectors) of the corresponding bins. We point to the work by Dougherty et al. [[]] which performs an overview of some classic approaches to binning and can serve as entry point to the relevant literature on the topic. In our work, however, we utilize bins in a different way. Specifically, we use their edges to construct lossless piecewise linear representations of the original scalar values. It turns out that this

---

simple and interpretable representations can provide substantial benefit to deep models on several tabular problems.
Recently, periodic activation functions have become a key component in processing coordinates-like inputs, which is required in many applications. Examples include NLP 451. CV [25], implicit neural representations [28] 38 42]. In our work, we show that periodic activations can be used to construct powerful embedding modules for numerical features in tabular data problems. Contrary to some of the aforementioned papers, where components of the multidimensional coordinates are mixed (e.g. with linear layers) before passing them to periodic functions [38 42 we
Periodic activations.
find it crucial to embed each feature separately before mixing them in the main backbone.
3 Embeddings for numerical features
In this section, we describe the general framework for what call "embeddings for numerical features" and the main building blocks used in the experimental comparison in secction 4
Notation. For given supervised learning problem on tabular data, we denote the dataset as {(x), }リ= where y E Y represents the object's label and x'= (x)((wm)) x)(cat)) E X repre- sents the object's features (numerical and categorical). 22 =?((um)) in turn, denotes the i-th numerical feature of the j-th object. Depending on the context, the j index can be omitted. The dataset is split into three disjoint parts: 1, n Strein U Jval U Uzeat, where the "train" part is used for training, the "validation" part is used for early stopping and hyperparameter tuning, and the "test" part is used for the final evaluation.
3.1 General framework
We formalize the notion of "embeddings for numerical features" 2; [(((a:/um)) Rd. where f((x) is the embedding function for the i-th numerical feature, 2; is the embedding of the i-th numerical feature and d; is the dimensionality of the embedding. Importantly, the proposed framework implies that embeddings for all features are computed independently of each other. Note that the function can depend on parameters that are trained some other fashion (e.g. before the main optimization). In this work, we consider only embedding schemes where the embedding functions for all features are of the same functional form. We never share parameters of embedding functions of different features.
part of the whole model in
The subsequent of the embeddings depends the model backbone. For MLP-like architectures, they are concatenated into one flat vector (see Appeeddix A for illustrations). For Transformer-bast architectures, no extra step is performed and the embeddings passed as is, so the usage is defined by the original architectures.
3.2 Piecewise linear encoding
While vanilla MLP is known to be a universal approximator 9 16 in practice, due to optimization peculiarities, it has limitations in its learning capabilities (34]. However, the recent work by Tancik et al. 42 uncovers the case where changing the input space alleviates the above issue. This observation motivates to check if changing the representations of the original scalar values of numerical features can improve the learning capabilities of tabular DL models.
At this point, we try to start simple and turn to "classical" machine learning techniques. Namely, we take inspiration from the one-hot encoding algorithm that is widely and successfully used for representing discrete entities such categorical features in tabular data problems or tokens in NLP. We note that the one-hot representation can be seen as an opposite solution to the scalar representation in terms of the trade-off between parameter efficiency and expressivity. To check whether the one- hot-like approach can be beneficial for tabular DL models, we design a continuous alternative to the one-hot encoding (since the vanilla one-hot encoding is barely applicable to numerical features).
Formally, for the i-th numerical feature, we split its value range into the disjoint set of T intervals Bi, .... By,, which we call bins: Bi = [ b(). The splitting algorithm is an important imple- mentation detail that we discuss later. From now on, we omit the feature index for simplicity. Once the bins are determined, we define the encoding scheme in Equation
3

---

PLB(2) = Jen,
eTl E IR?
bg
61
b2
by
bg
R
0,
< bi- AND > 1
(1)
J
I AND < T
=
otherwise
PLB(x:)
by
bg
where PLE stands for "peicewise linear encoding". We provide the visualization in Figure
e1
Figure 1: The piecewise linear encoding
(PLE) in action for = 4 (see Equation
Note that:
PLE produces alternative initial representations for the numerical features and can be viewed as preprocessing strategy. These representations are computed once and then used instead of the original scalar values during the main optimization.
For 7" = 1, the PLE-representation is effectively equivalent to the scalar representation.
Contrary categorical features, numerical features are ordered; we express that by setting to 1
the components corresponding to bins with the right boundaries lower than the given feature value (this approach resembles how labels are encoded in ordinal regression problems).
The cases (x < ba) and (rr 2 by) are also covered by Equation (which leads to (el S 0) and
(er 2 1) respectively).
The choice to make the representation piecewise linear is itself subject for discussion. We analyze some alternatives in (uuseectionn 5.2]
PLE can be viewed as feature preprocessing, which is additionally discussed in subsection 5.3
models. While the described PLE-representations can be passed MLP- like models as is, attention-base models are inherently invariant to the order of input embeddings, so one additional step is required add the information about feature indices to the obtained encodings. Technically, we observe that it is enough to place one linear layer after PLE(without sharing weights between features). Conceptually, however, this solution has clear semantic interpretation. Namely, it is equivalent to allocating one trainable embedding final feature embedding by aggregating the embeddings of its bins with e, as weights, plus bias to- Formally: fi (I) = Do + E/=1 et 14 = Linear (PLE (z)).
note on attention-
E Rd for each bin B, and obtaining the
In the following two sections, we describe two simple algorithms for building bins suitable for PLE. Namely, we rely on the classic binning algorithms [I]] and one of the two algorithms unsupervised. while another one utilizes labels for constructing bins.
3.2.1 Obtaining bins from quantiles
natural baseline way to construct the bins for PLE by splitting value ranges according to the uniformly chosen empirical quantiles of the corresponding individual feature distributions. Formally, (num)
for the i-th feature: b, = =Q+ (xx((
Jrrai where is the empirical quantile function. Trivial
bins of zero size are removed. In subsection D.I. we demonstrate the usefulness of the proposed scheme the synthetic GBDT-frient SeDT-friendly dataset described in section 5.1 in Gorishniy et al. [13]
3.2.2 Building target-aware bins
In fact, there are also supervised approaches that employ training labels for constructing bins [ Intuitively, such target-aware algorithms aim produce bins that correspond to relatively narrow ranges of possible target values. The supervised approach used in our work is identical in its spirit to the "C4.5 Discretization" algorithm from Kohavi and Sahami 23 In a nutshell, for each feature, we recursively split its value range in greedy manner using target guidance, which is equivalent to building decision tree (which uses for growing only this one feature and the target) and treating the regions corresponding to its leaves the bins for PLE (see the illustration in Figure 4). Additionally,
define b6 minjeJvram of and byp- maxjeJevui. I..

---

3.3 Periodic activation functions
Recall that in subsection 3.2 the work by Tancik al. was used as starting point of our motivation for developing PLE. Thus, we also try to adapt the original work itself for tabular data problems. Our variation differs in two aspects. First, we take into account the fact the embedding framework described in (ubsection forbids mixing features during the embedding process (see subseettoo D.2 for additional discussion). Second, we train the pre-activation coefficients instead of keeping them fixed. As a result, our approach is rather close to Li et al. [25] with the number of "groups" equal to the number of numerical features. We formalize the described scheme in Eqqation
fz) Periodic concat(sin()), cos(v)),
[2TC1 ..... 2TTC22)
(2)
where G are trainable parameters initialized from N\(0,), o). We observe that is an important hyperparameter. Both o and k are tuned using validation sets.
3.4 Simple differentiable layers
In the context of Deep Learning, embedding numerical features with conventional differentiable layers (c.g. linear layers, ReLU activation, etc.) is a natural approach. fact, this technique is already used its own in the recently proposed attention- based architectures [13] 24 39 and in
some models for CTR prediction problems 14 40) However, we also note that such conventional modules be used top of the components described in subsection 3.2 and subsection 3.3 In
section 4] we find that such combinations often lead to better results.
Experiments
In this section, we empirically evaluate the techniques discussed in section 3 and compare them with Gradient Boosted Decision Trees to check the status quo the "DL vs GBDT" competition.
4.1 Datasets
Table 1: Dataset properties. "RMSE" denotes root-mean-sspe square error, "Acc." denotes accuracy.
GE
CH
CA
HO
AD
HI
FB
SA
CO
MI
#objects
9873
10000
20640
22784
48842
61878
98049
197080
200000
581012
1200192
#num. features
32
10
8
16
6
93
28
50
200
54
136
#cat. features
0
0
0
8
0
0
0
0
metric
Acc.
Acc.
RMSE RMSE
Acc.
Acc.
Acc.
RMSE
Acc.
Acc.
RMSE
#cllssess
5
2
2
2
2
7
majority class
29%
79%
76%
26%
52%
89%
48%
We use eleven public datasets mostly from the previous works on tabular DL and Kaggle competitions. Importantly, focus on the middle and large scale tasks, and our benehmark biased towards GBDT-friendly problems, since, as of now, closing the gap with GBDT models on such tasks is one of the main challenges for tabular DL. The main dataset properties are summarized in Table and the used sources and additional details are provided in |Appendix C
4.2 Implementation details
We mostly follow Gorishniy et al. 13 in terms of the hyperparameter tuning, training and evaluation protocols. Nevertheless, for completeness, we list all the details in Appenddxx E the next paragraph, describe the implementation details specific to embeddings for numerical features.
Embeddings for numerical features. If linear layers are used, we tune their output dimensions. The PLE hyperparameters are the same for all features. For caanile-based PLE, we tune the number of quantiles. For target-aware PLE, we tune the following parameters for decision trees: the maximum number of leaves, the minimum number of items per leaf, and the minimum information gain required for making a split when growing the tree. For the Periodic module (see Equation 2). we tune a and k (these hyperparameters are the same for all features).
5

---

4.3 Model names
In the experiments, we consider differ- ent combinations of backbones and em- beddings. For convenience, we use the "Backdhone-Embeeddinn" pattern to name the models, where "Backbone" denotes the backbone (e.g. MLP, ResNet, Transformer) and "Embedding" denotes the embedding type. See Table for all considered embed- ding modules. Note that:
Table 2: Embedding names. See subsection 4.3
Name
Embedding function (f.)
Linear
LR
ReLU Linear
LRLR
BeLU Linear ReLU Linear
PLE,
Q-L
Linear o PLE,
Q-La
ReLU Linear o PLE,
Q LRLR
ReLU o Linear ReLU Linear o PLEç
Periodic is defined in [qquatio 2
PLE:
PLE, denotes the quantile-basee PLE. PLE, denotes the target-aware PLE.
T-L
Linear PLE,
T-LR
ReLU Linear PLE,
T-LRLR
ReLU Linear ReLU Linear o PLE,
Linear- denotes bias-free linear layer.
Periodic Linear Periodic
LReLU denotes leaky ReLUU. AutoDis
PL.
was proposed in Guo et al. [14
ReLU Linear Periodic
"Transformer-L"" is equivalent FT-
PLRLR
ReLU Linear ReLU Linear Periodic
AutoDis
Linear SoftMax Linear- LReLU Linear-
Transformer [13]
4.4 Simple differentiable embedding modules
Table 3:
Results for MLP equipped with simple embedding modules (see subsection 4.3, p. The
metric values averaged over 15 random seeds reported. The standard deviations provided in in[pppendix We consider one result to better than another its mean score is better and its standard deviation is less than the difference. For each dataset, top results are in bold. Notation: corresponds to RMSE, 1 corresponds to accuracy
GE f CH1 CA J HO
AD
OT 1 HI f
FB
1
CO
MI
MLP 0.632 0.856 0.495
3.204
0.854
0.818
0.720 5.686
0.912
0.964
0.747
MI.P-I 0.639
0.861
0.475
3.123
0.856
0.820
0.723
5.684
0.916
0.963
0.748
MIP-LR 0.642 0.860 0.471
3.084
0.857 0.819 0.726
5.625
0.923
0.963
0.746
We start by evaluating embedding modules consisting of "conventional" differentiable layers (linear layers, ReLU activations, etc.). The results are summarized in Table The main takeaways:
first and foremost, the results indicate that MLP can benefit from embedding modules. Thus, we conclude that this backbone is worth attention when it comes evaluating embedding modules. the simple LR module leads to modest, but consistent improvements when applied to MLP.
Interestingly, the "redundant" MLP-L configuration also tends to outperform the vanilla MLP. Although the improvements are not dramatic, the special property of this architecture that the linear embedding module can be fused together with the first linear layer of MI.P after training, which for LRLR and AutoDis, we observe that these heavy modules
completely removes the overhead. do not justify the extra costs (see the results in Appendix F).
4.5 Piecewise linear encoding
In this section, we evaluate the encoding scheme described in subsection 3.2 The results are summarized in Table 4 The main takeaways:
The piecewise linear encoding is often beneficial for both types of architectures (MLP and Transformer) and the profit can be significant (for example, see the CA and AD datasets). Adding differentiable components on top of the PLE can improve the performance, Though, the most expensive modifications such as Q-LRLR and T-LRLR not worth it (see Appendix F

---

Note that the benchmark is biased towards GBDT-friendly problems, so the typical superiority of tree-based bins over quantile-based bins, which can be observed in Table 4 may not generalize to more DI.-friendy datasets. Thus, we do not make any general claims about the relative advantages of the two schemes here.
Table 4: Results for MLP and Transformer with embedding modules based on the piecewise linear The best results are defined
encoding (subsection 3.2). Notation follows Table 3 and Table 2 separately for the MLP and Transformer backbones.
GE
CH f
CA
HO
AD f
OT
HI
FB
SA T
CO f
MI
MI.P
0.632 0.856
0.495
3.204
0.854
0.818
0.720
5.686
0.912
0.964
0.747
MLP-Q
0.653 0.854
0.464
3.163
0.859
0.816
0.721
5.766
0.922
0.968
0.750
MLP-T
0.647 0.861
0.447
3.149
0.864
0.821
0.720
5.577
0.923
0.967
0.749
MLP-Q-LR
0.646 0.857
0.455
3.184
0.863
0.811
0.720
5.394
0.923
0.969
0.747
MLP-T-LR
0.640 0.861
0.439
3.207
0.868
0.818
0.724
5.508
0.924
0.968
0.747
Transformer
0.632 0.860
0.465
3.239
0.858
0.817
0.725
5.602
0.924
0.971
0.746
Transformer- Q-
0.659 0.856
0.451
3.319
0.867
0.812
0.729
5.741
0.924
0.973
0.747
Transformer-Te T-L
0.663 0.861
0.454
3.197
0.871
0.817 0.726
5.803
0.924 0.974
0.747
Transformer Q-LR
0.659
0.857
0.448
3.270
0.867
0.812 0.723
5.683
0.923
0.972
0.748
Transformer-T-LA
0.665 0.860
0. 442
3.219
0.870
0.818 0.729
5.699
0.924
0.973
0.747
4.6 Periodic activation functions
Table 5: Results for MLP and Transformer with embedding modules based on periodic activations (subsection 3.3). Notation follows Table 3 and Table The best results are defined separately for the MLP and Transformer backbones.
GE
CH
CA
HO
AD1
O
HI
FB 4
SA
CO
MI 4
MLP
0.632 0.856 0.495
3.204
0.854
0.818
0.720
5.686
0.912
0.964
0.747
MLP-P
0.631 0.860 0.489
3.129
0.869
0.807
0.723
5.845
0.923
0.968
0.747
MLP-PL
0.641
0.859 0.467
3.113
0.868
0.819
0.727
5.530
0.924
0.969
0.746
MI.P-PLR
0.674 0.857 0.467
3.050
0.870
0.819
0.728
5.525
0.924
0.970
0.746
Transformer-L
0.632 0.860 0.465
3.239
0.858
0.817
0.725
5.602
0.924
0.971
0.746
Transformer-PLR
0.646 0.863 0.464 3.162 0.870 0.814 0.730
5.760
0.924 0.972 0.746
In this section, we evaluate embedding modules based on periodic activation functions as described in suluussection 3.3] The results are reported in Table 5]
The main takeaway: on average, MLP-P superior to the vanilla MLP. However, adding a differentiable component on top of the Peri Periodic module should be the default strategy (which is in line with Li et al. (25)). Indeed, MIP-PIRR and MLP-PL provide meaningful improvements over MIPPP (e.g. see GE, CA, HO) and even "fix" MIP--P where it is inferior to MIP (OT, FB).
Although MLP-PLR is usually superior to MLP-PL, we note that in the latter case the last linear layer of the embedding module is "redundant" in terms of expressivity and can be fused with the first linear layer of the backbone after training, which, in theory, can lead to a more lightweight model. Finally, we observe that MLP-PLRLR and MLP-PLR do not differ significantly enough to justify the extra cost of the PLRLR module (see Appendix F).
4.7 Comparing DL models and GBDT
In this section, we perform big comparison of different approaches identify the best embedding modules and backbones, as well as to check if embeddings for numerical features allow DL models to compete with GBDT on more tasks than before. Importantly, we compare ensembles of DL models against ensembles of GBDT, since Gradient Boosting is essentially an ensembling technique, so such comparison will be fairer. Note that we focus only on the best metric values without taking efficiency into account, only check if DL models are conceptually ready to compete with GBDT.
7

---

We consider three backbones: MLP, ResNet, and Transformer, since they are reported to be repre- sentative of what baseline DL backbones are currently capable of [T3] 24 39 Note that do
not include the attention-based models that also apply attention on the level of objects [24 35 39]. since this non-parametric component orthogonal the central topic of our work. The results are summarized in Table 6)
Table 6: Results for ensembles of GBDT, the baseline DL models and their modifications using different types of embeddings for numerical features. Notation follows Table 3 and Table 2 Due to the limited precision, some different values are represented with the same figures.
GE
CH1
CA
HO
AD
OT
HT
FB 1
SA
CO
MI
Avg.
Rank
CalBond
0.692
0.861
0.430
3.093
0.873
0.825
0.727
5.226
0.924
0.967
0.741
3.6
2.9
XGBoost
0.683
0.859
0.434
3. 152
0.875
0.827
0.726
5.338
0.919
0.969
0.742
4.6
+ 2.7
MLP
0.065
0.856
0.486
3. 3.109
856
0.822
0.727
5.616
0.913
0.968
0.746
8.5
2.6
MLP-LR
0.679
0.861
0.463
3.012
0.859
0.826
0.731
5.477
0.921
0.972
0.744
5.5
2.7
MLP.Q.LR
0.682
0.859
0.433
3,080
0.867
0.818
0.724
5.144
0.924
0.974
0.745
5.1
1.9
MLP-T-LR
0.673
0.861
0. 434
3.099
0.870
0.821
0.727
5.409
0.921
0.973
0.746
5.1
1.7
MLP-PLR
0.830
0.975
0.743
3.0
ResNet
0.690
0.861
0. 0.483
3. 080
0.856
0.821
0.734
5. 5.482
0.918
0.968
0.745
6.7 3.3
ResNet-LR
0.672
0.862
0.450
2.992
0.859
0.822
0.733
5.415
0.923
0.971
0.743
5.6
2.7
RosNet-Q-LR
0.674
0.859
0.4
3. 0G6
0.868
0.815
0.729
5.309
0.923
0.976
0.746
1.7
1.7 2.0
ResNet-T-LR
0.731
0.923
0.975
0.744
ResNei.
0.691
0.861
0.443
3.040
0.874
0.825
0.734
5.400
0.924
0.975
0.743
3.2
1.3
Transformer-L
0.668
0.861
0.455
3.188
0.860
0.824
0.727
5.434
0.924
U.973
0.743
5.9
Transforer-LLR
0.606
0.801
0.446
3.193
0.861
0.824
0. 0.733
5.430
0.921
0.973
0.743
5.2
2.2
Wransfomer--QLL.
0.726
0.924
0.744
Transformer LR
0.686
0.862
0. 0.423
3.149
0.871
0.823
0.733
5.515
0.924
0.976
0.744
3.7
1.771010 2.2
Transformer.P..
0.686
0.864
0.449
3.001
0.873
0.823
0.734
5.581
0.924
0.975
0.743
3.9
The main takeaways for DL models:
For most datasets, embeddings for numerical features can provide noticeable improvements for three different backbones. Although the average rank is not a good metric for making subtle conclusions, we highlight the impressive difference in average ranks between the MLP and MLP-PLR models.
The simplest LR embedding is a good baseline solution: although the performance gains are not dramatic, its main advantage is consistency (e.g. see MLP vs MLP-LR).
The PLR module provides the best average performance. Empirically, we observe a (see Equation 2) to be an important hyperparameter that should be tuned.
Piecewise linear encoding (PLE) allows building well performing embeddings (e.g. T-LR, Q-LR). In addition to that, PLE itself is worth attention because of its simplicity, interpretability and efficiency (no computationally expensive periodic functions).
Importantly, after the MLP-like architectures are coupled with embeddings for numerical features, they perform on par with the Transformer-basee models.
The main takeaway for the DDL vs GBDT" competition: embeddings for numerical features is a significant design aspect that has great potential for improving DI. models and closing the gap with GBDT on GBDT-friendly tasks. Let us illustrate this claim with several observations:
The benchmark is initially biased to GBDT-friendly problems, which be observed by comparing GBDT solutions with the vanilla DL models (MLP, ResNet, Transformer-L)
However, for the vast majority of the "backbone & dataset" pairs, proper embeddings are the only thing needed to close the gap with GBDT. Exceptions (rather formal) include the MI dataset and the following pairs: "ResNet GE", "Transformer & FB". "Transformer & GE", "Transformer & O
Additionally, to the best of our knowledge, it is the first time when DL models perform on par with GBDT the well-known California Housing and Adult datasets.
That said, compared to GBDT models, efficiency can still be an issue for the considered DI. architec- tures. In any case, the trade-off completely depends the specific use case and requirements.
8

---

5 Analysis
5.1 Comparing model sizes
To quantify the effect of embeddings for numerical features model sizes, we report the parameter counts in Table 7 Overall, introducing embeddings for numerical features can cause non-negligible overhead in terms of model size. Importantly, the overhead in terms of size does not translate to the same overhead in terms of training times and throughput. For example, the almost 2000-fold increase in the parameter count for MLP-LR on the CH dataset results in only 1.5-fold increase in training times. Finally, in practice, we observe that coupling MLP and ResNet with embedding modules leads to architectures that are still faster than branse models.
Table 7: Parameter counts for MLP with different embedding modules. All the models are tuned and the corresponding backbones are not identical in their sizes, so we take into account the fact that different approaches require a different number of parameters to realize their full potential.
GR
CH
CA
HO
AD
OT
FB
SA
CO
MI
MLP
2.0M
1.5K
43.5K
3.6M
5.3M
479.9K
25.8K
937.3K
5.8M
3.2M
276.5K
MLP-LR
x2.52
x1931.03
x25.05
x1.28
x0.35
x12.53
x68.16
x4.76
x1.58
x0.72
x15.79
MLP-T
x1.58
x14.13
x7.97
x0.43
x0.04
X2.27
x5.85
x0.47
x0.59
x0.74
x3.85
MLP-TTIR
x1.61
466.555
x6.80
x0.23
x0.16
x2.52
x113.22
x3.43
x0.41
x0.35
x8.47
MLP-PLR
x1.73
x250.24
x 12.94
x1.07
x0.66
x8.05
x110.57
x4.93
x0.64
x0.44
x9.57
5.2
Ablation study
Table 8: Comparing piecewise linear encoding (PLE) with the two variations described in subsec
tion 5.2]
Notation follows Table and Table 2
GE f CH1
CA
HO J
AD
0T 1
HI 1
FB 1
MLP-Q (piecewise linear)
0.653 0.854
0.464
3.163
0.859 0.816 0.721
5.766
MLP-Q (binary)
0.652
0.815
0.462
3.200
0.860 0.810
0.720
5.748
MLP-Q (one-blob)
0.613
0.851
0.461
3.187 0.857 0.808
0.719
5.645
MLP-T
(piecewise linear)
0.647
0.861
0.447
3.149
0.864
0.821
0.720
5.577
MLP-T (binary)
0.639
0.855
0.464
3.163
0.869 0.813
0.718
5.572
MLP-T (onc-blob)
0.622
0.858
0.464
3.158 0.870 0.809 0.724 5.475
In this section, we compare two alternative binning-basede encoding schemes with PLE (see subsee- tion 3.2). The first one ("thermometer" [6]) sets the value instead of the piecewise linear term (see Equation I). The second one is a generalized version of the one-blob encoding (29 (see subsec lion for details). The tuning and evaluation protocols are the same as in ssusecction 4.2 The results in table Table 8 Sindicate that making the binning-based encoding piecewise linear good
default strategy.
5.3 Piecewise linear encoding as a feature preprocessing technique
is known that data preprocessing, such as standardization or quantile transformation, is often crucial for DL models for achieving competitive performance. Morcover, the performance significantly vary between different types of preprocessing. At the same time, PLE-representations contain only values from [0, 1] and they are invariant to shifting and scaling, which makes PLE itself a general feature preprocessing technique potentially suitable for DL models without the need to use traditional preprocessing first.
To illustrate that, for datasets where the quantile transformation was used in snsecciion 4 we reevaluate the tuned configurations of MLP, MLP-Q,, and MLP-T with different preprocessing policies and report the results in Table (note that standardization is equivalent to no preprocessing for models
with PLE).

---

Table 9: Results for MLP and MLP with PLE for different types of data preprocessing. Solutions using PLE are significantly less sensitive to data preprocessing. Notation follows Table and Table 2
GE f CH
CA 4 HO 4 AD f HI H
FB 4
SA
CO MI 4
MLP (none)
0.565 0.796 1.118 5.328 0.808 0.707 13.125
0.911
0.948 0.844
MLP (standard)
0.629 0.855 0.509 3.303 0.855 0.721
5.919
0.912
0.963 0.754
MLP (quantile)
0.632 0.856 0.495 3.204 0.854 0.720
5.686
0.912
0.964 0.747
MLP-Q Q (none)
0,654 0.851 0.463
3.162 0.860
0.721
5.889
0.922
0.968 0.754
MLP-0 Q (quantile)
0.653 0.854 0.464 3.163 0.859 0.721
5.766
0.922
0.968 0.750
MIP-1 (none)
0.644
0.860
0.447
3.175
0.865
0.721
5.598
0.923
0.968 0.749
MLP-T (quantile) 0.647 0.861 0.447 3.149 0.864 0.720
5.577
0.923 0.967 0.749
First, the vanilla MLP often becomes unusable without preprocessing. Second, for the vanilla MLP. it can be important to choose one specific type of preprocessing (CA, HO, FB, MI), which is less pronounced for MLP-Q and not the case for MLP-T (though, this specific observation can be the property of the benchmarks, not of MLP-T). Overall, the results indicate that models using PLE are less sensitive to the initial preprocessing compared the vanilla MLP. This an additional benefit of PLE-representations for practitioners since the aspect of preprocessing becomes less critical with PLE.
5.4 The "feature engineering" perspective
Table 10: The comparison of the effects of Periodic-based modules for XGBoost and MLP
CA 4 HO 4
HI 1
XGBoost
0.436 3.160 0.724 XGBoost with Periodic 0.441 3.184 0.724
MLP
0.495 3.204 0.720
ML.P-I.
0.467 3.113 0.727
At first sight, feature embeddings may resemble feature engineering and should be suitable for all kinds of models. However, the proposed embedding schemes are motivated by DL-specific aspects of training (see the motivational parts of lubssectti 3.2 and subsection 3.3). While our methods likely to transfer well to models with similar training properties (e.g. to linear models since those are special case of deep models), it is not the case in general. To illustrate that, we try adopting the Periodic module for XGBoost by fixing the random coefficients from Equation 2] We also keep the original features instead of dropping them. The tuning and evaluation protocols are the same in subsection 4.2] The results in Table show that this technique, while being useful for DL models, does not provide any benefits for XGBoost.
6 Conclusion & Future work
In this work, have demonstrated that embeddings for numerical features are important design aspect of tabular DI. architectures. Namely, it allows existing DL backbones to achieve noticeably better results and significantly reduce the gap with Gradient Boosted Decision Trees. We have described two approaches illustrating this phenomenon, one using the piecewise linear encoding of original scalar values, and another using periodic functions. We have also shown that traditional MLP-like models coupled with embeddings can perform on par with attention based models.
Nevertheless, we have only scratched the surface of the new direction. For example, it is still to be explained how exactly the discussed embedding modules help optimization on the fundamental level. Additionally, have considered only schemes where the same functional transformation was applied to all features, which may be suboptimal choice.
10

---

References
[1] Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparam- eter optimization framework. In KDD, 2019.
[2] S. O. Arik and T. Pfister. Tabnet: Attentive interpretable tabular learning. arXiv, 1908.07442v5, 2020.
[3] S. Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi. Gradient boosting neural networks: Grownet. arXiv, 2002.07971v2, 2020.
[4] Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.
[5] A. Blackard and D. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3):311-15) 2000.
[6] Buckman, A. Roy, Raffel, and Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference Learning Representations, 2018.
[7] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In SIGKDD,
[8] Covington, J. Adams, and E. Sargin. Deep neural networks for youtube recommendations, In RecSys, 2016.
[9] G. Cybenko. Approximation by superpositions of a sigmoidal function. Math. Control. Signals Syst., 2(4), 1989.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[11] J. Dougherty, R. Kohavi, and M. Sahami. Supervised and unsupervised discretization of continuous features. In ICML, 1995.
[12] Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
[13] Y. Gorishniy, I. Rubachev, Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. In NeurlPS, 2021.
[14] H. Guo, B. Chen, R. Tang. W. Zhang, Z. Li, and X. He. An embedding learning framework for numerical features in CTR prediction. In KDD, 2021.
[15] H. Hazimeh, N. Ponomareva, P. Mol, Z. Tan, and R. Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In ICML., 2020.
[16] K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2), 1991.
[17] X. Huang, A. Khetan, M. Cvitkovic, and Z. Karnin. Tabtransformer:: Tabular data modeling using contextual embeddings. arXiv, 2012.06678v1, 2020.
[18] A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka. Well-tuned simple nets excel on tabular datasets. In NeurlPS, 2021.
[19] G. Ke, Q. Meng, T. Finley, T. Wang. W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146-315 3154, 2017.
[20] R. Kelley Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291-297, 1997.
[21] G. Klambauer, Unterthiner, A. Mayr, and Hochreiter. Self-normalizing neural networks. In NIPS, 2017.

---

[22] R. Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision tree hybrid. In KDD, 1996.
[23] R. Kohavi and M. Sahami. Error-based and entropy-based discretization of continuous features.
In KDD, pages 114-119 119. AAAI Press, 1996.
[24] Kossen, N. Band, C. Lyle, A. N. Gomez, T. Rainforth, and Y. Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In NeurlPS, 2021.
[25] Y. Li, S. Si, G. Li, C. Hsich, and S. Bengio. Learnable fourier features for multi-dimensional spatial positional encoding. In NeurlPS, 2021.
[26] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[27] R. C. B. Madeo, C. A. M. Lima, and S. M. Peres. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, SAC, 2013.
[28] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[29] T. Müller, B. McWilliams, F. Rousselle, M. Gross, and J. Novák. Neural importance sampling. ACM Trans. Graph., 38(5), 2019.
[30] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 28300, 2011.
[31] S. Popov, S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020.
[32] L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin. Catboost: unbiased boosting with categorical features. In NeurlPS, 2018.
[33] T. Qin and T. Liu. Introducing LETOR 4.0 datasets. arXiv, 1306.2597v1, 2013.
[34] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, and A. C. Courville. On the spectral bias of neural networks. In ICML, 2019.
[35] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, T. Adler, D. P. Kreil, M. K. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks all you need. In ICLR, 2021.
[36] R. Shwartz-Ziv and A. Armon. Tabular data: Deep learning is not all you need. arXiv, 21 2106.03253v1,220 2021.
[37] K. Singh, R. K. Sandhu, and D. Kumar. Comment volume prediction using neural networks and decision trees. In IEEE UKSim-AMSS 17th International Conference on Computer Modelling and Simulation, UKSim, 2015.
[38] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020.
[39] G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein. SAINT: im- proved neural networks for tabular data via row attention and contrastive pre-training. arXiv, 2 106.01142vv, 2021.
[40] W. Song, Shi, 7. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, 2019.
[41] D. Sundararaman, S. Si, V. Subramanian, G. Wang. D. Hazarika, and L. Carin. Methods for numeracy-preservee word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
12

---

[42] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich- Keill, N. Raghavan. U. Singhal, R. Ra- mamoorthi, Barron, and Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
[43] R. Turner, D. Eriksson, M. McCourt, J. Kiili, E. Laaksonen, Z. Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. arXiv, https:/arriv.orgaab///10..4 10201 v1, 2021.
[44] Vanschoren, N. van Rijn, B. Bischl, and L. Torgo. Openml: networked science machine learning. arXiv, 1407.7 1407.7722v1, 2014.
[45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention all you need. NIPS, 2017.
[46] R. Wang, B. Fu, G. Fu, and M. Wang. Deep & cross network for ad click predictions. In ADKDD,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the analysis in /subsection 5.1
(c) Did you discuss any potential negative societal impacts your work? [N/A] The work focuses on generic aspect of deep learning models.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] We do not include theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] The supplementary material includes the script used to create data splits. The hyperparameters are either explicitly described in |subsection 4.2) a.2 and
supplementary material, or tuned as described in ssubsectioon 4.2)
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We provide standard deviations in the supplementary material, see Table 18 and see Table
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] The experiment reports included in the supplementary material provide the information about the used hardware and execution times.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix C
(b) Did you mention the license of the assets? [Yes] In the README md file in the supple- mentary material, we refer to the original licenses of the used datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We do provide new datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We use publicly available datasets.
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use publicly available datasets.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not use crowdsourcing. We did not conduct research with human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not use crowdsourcing. We did not conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not use crowdsourcing. We did not conduct research with human subjects.
14


Reviews:
[{"review_id": "gHTVXxhOWfB", "reviewer": "Reviewer_2HxT", "summary": "The paper presents methods for embedding numerical features for tabular deep learning methods. The paper describes the current best practices in the tabular deep learning literature, and then proposes two basic approaches for embedding numerical features and ways to combine and augment them. First, the authors propose Piecewise Linear Encoding (PLE), where a number is mapped to a T-dimenionsal vector by partitioning the range of values into T bins corresponding to the coordinates, where a coordinate is equal to: 1 if the number is below the respective bin, 0 if the number is above the bin, and otherwise the relative location of the number in the given bin, i.e., (x - low) / (high - low). This is effectively a continuous relaxation of treating numeric features as categorical via quantization. Different strategies for how to compute the bins are considered. The second method is based on mapping to a vector with periodic activation functions, i.e., mapping x -> [cos(w_1 * xx), sin(w_1 * x), ..., cos(w_n * xx), sin(w_n * x)], similar to prior proposal for positional embeddings. This paper proposes that the two base feature embeddings can be combined with some sequence of linear and ReLU activation layers before they are fed into the model.\n\nFor evaluating the different embedding methods, the paper considers three base models, a simple MLP, a ResNet, and a Transformers architecture. Various combinations of models and embedding methods are tested on a variety of tabular datasets. The main findings are that with well-chosen numerical feature embedding, deep learning-based approaches can nearly match or exceed the performance of Gradient Boosted Decision Trees. Moreover, it is shown that numerical feature embeddings are important for all model types, and even an MLP can perform exceptionally well.", "questions": "Could the authors explain which method is preferred and under what circumstances? Why would one method perform better on one dataset than on another, or is it all just statistical noise? I would strongly advise to move the bulk of the various variants tested to the appendix and keep only the one or two leading variants (their naming scheme is also quite confusing). It could be worthwhile to display more varied results in the context of an ablation study.\n\nCould the authors please explain their methodology for picking the datasets to evaluate their methods? What does \"GBDT-friendly dataset\" mean?\n\nI would also suggest the authors to include in their related works section a paragraph on the related methods using the NLP literature for embedding numbers.\n\nAs a very minor complaint, I would also recommend the authors ensure each time initials are introduced that their full meanings are also included. For example, GBDT appears in the abstract without explanation. CTR appears in the body without ever explaining what it stands for (Click-through Rate I guess), as well as DL (Deep Learning). Moreover, I would suggest the authors to refrain from using \"dramatic\" wordings (e.g., \"game changer\", \"epic difference\", \"we can't help\") -- let your results speak for themselves.", "limitations": "The authors discussed the runtime costs associated with using deep learning-based methods relative to decision trees, as well as the difficulty in drawing far reaching conclusions based on the selection of datasets and their sample size.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are its comprehensive set of experiments, including not just one variant but many possible combinations of the proposed embeddings. Such a detailed set of results could prove useful for practitioners that might consider adopting one of the proposed methods based on similarity to one of the given datasets in the evaluation. Moreover, tabular deep learning is still a relatively underexplored subject, and any push to establish better baselines could prove important.\n\nHowever, the paper ignores the literature on embedding numerical features found in other domains but effectively addresses the same fundamental problem. Namely, there are many papers on how to treat numerical data found in natural language, and specifically how to embed it in the same space as conventional word embeddings. Many suggestions have been proposed over the years, some are identical to the methods in this paper, e.g., the use of periodic activation functions. While the authors note that it was proposed for positional embeddings, it was also proposed for embedding plain numbers found in the text. See [1] for a survey on the various methods used in natural language processing.\n\nIt is also difficult to discern what the bottom line is from the many experimental results, regardless of whether the proposed embedding methods are novel or not. It is unclear which method is preferred (if any), and why one method would work better than another on a given dataset. There is little attempt to analyze the results or draw any conclusions. The paper would be significantly improved if it was concentrated on one specific method rather than all possible combinations of a set of methods.\n\nFinally, it is unclear why the paper uses the specific datasets and not others. The only explanation given is that these datasets are \"GBDT-friendly\". However, it is unclear what are the qualifications of a dataset to be considered as \"GBDT-friendly\", and why we should ignore other datasets that were previously explored in related literature on tabular deep learning. For instance, in [2] a different set of datasets is considered (though there is some overlap), where they already showed that deep learning methods can beat in most cases GBDT. In my view, this submission lacks a thorough explanation of their methodology for selecting the datasets they analyze and why it is okay to ignore others that have appeared in other papers.\n\n[1] - Thawani et al., Representing Numbers in NLP: a Survey and a Vision, NAACL 2021.\n\n[2] - Gorishniy et al., Revisiting Deep Learning Models for Tabular Data, NeurIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aqjIpGmtm8", "reviewer": "Reviewer_5oG4", "summary": "This paper address tabular data prediction by using deep models. The authors argue that embedding for numerical features are critical but underexplored in previous work. In practice, two embedding modules termed piecewise linear encoding and periodic activations are proposed to improve the performance of the current deep models for tabular data. Extensive experimental results demonstrate the effectiveness of the proposed two embedding schemes.", "questions": "See the weaknesses. (1)The motivation of the module design is still not clear; (2) How about the performance gain of the proposed two modules for more recent proposed deep architecture.", "limitations": "The authors partially discussed the limitations of the current work in the conclusion and future work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The paper is clearly written and easy to follow\n+ The proposed two embedding schemes are straightforward and have practical effectiveness.\n+ Extensive experimental results illustrate the performance of the proposed method on various deep model architectures.\n\nWeaknesses\n- What’s the insight of designing such two embedding modules? It is more like the practical solutions that depend on massive experiments, which makes me wonder about the intuition behind these choices.\n\n- What’s the relationship between these two proposed embedding modules? Should we need to consider them as independent contributions, and why? Are there any different application scenarios for these two modules? Which one is more effective in real applications?\n\n- Is there any more deep architectures for tubular data analysis, and how about their performance by adopting the proposed two modules. If the authors aim to demonstrate the generality of the proposed module, more recent proposed deep networks (but not the simple Resnet or Transformer) should be considered as the baseline models, and their performance improvement by using proposed modules should be clearly reported.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "61rhcDMt8Ra", "reviewer": "Reviewer_GNk8", "summary": "This paper proposes a new encoding scheme for embedding numerical features in neural network models. The authors report that for tabular data in many cases neural methods have failed to improve on Gradient Boosted Decision Trees (GBDT), thus necessitating more research into how to represent the tabular data, in particular features that are numerical as opposed to categorical. The authors propose a variation of a binning/bucketing scheme which they call Piecewise Linear Encoding (PLE), which is more informative than a one-hot encoding scheme, where a numerical feature is simply represented by the embedding corresponding to the bin/bucket id it belongs to. This is clearly a bad scheme for numerical features, since the ordering information is lost: if x_1 > x_2, then there is no ordering relationship between the embedding(x_1) and embedding(x_2), so the model must first learn this implicit relationship. Such a scheme therefore loses some information in the case of numerical data. The authors propose their PLE scheme as follows: for the data x, if it falls between bin b_t and b_{t + 1}, then embedding_i = 0 if i < t, embedding_i = 1 if i > t, and embedding_t = (x - b_t)/(b_{t+1} - b_t). Thus this is a piece-wise linear scheme which explains the naming of the scheme. The authors also consider some variations inspired by the position encoding used in Transformer models for text, where embedding_i also has a sinusoidal component. \n\nThe authors show experimental results using this scheme where they show that on several tabular data-sets this scheme improves on the baseline MLP models using simple linear embeddings and is competitive with decision trees. \n", "questions": "Comments:\n- Line 14: \"Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures\"\n\n- Line 44: \"In particular, we show, that after proper embeddings, simple MLP-like architectures often provide the performance comparable to the state-of-the-art attention-based models.\"\n\nThe statements in line 14 and line 44 should be qualified with a \"for numerical data\", since that is the correct scope of the claim as presented in this work.\n\n- Line 55 \"3. On a number of public benchmarks, we achieve the new state-of-the-art of tabular DL.\" -> \"state-of-the-art on tabular DL\" Typo, please fix.\n\nReferences:\n\nThe piecewise encoding scheme is quite similar to the \"thermometer encoding scheme\" proposed in [1] for representing image intensities, with the main difference being the absence of the piece-wise linear term (x - b_t)/(b_{t+1} - b_t). The motivation was to see whether this representation conferred any adversarial robustness (which it did not), but improvements in standard (vanilla) accuracy of ResNets was observed using this scheme on MNIST, CIFAR-10 and SVHN datasets (see Table 2, 3, 4 corresponding to \"vanilla\" and \"clean\").\n\n[1] @inproceedings{buckman2018thermometer,\n  title={Thermometer encoding: One hot way to resist adversarial examples},\n  author={Buckman, Jacob and Roy, Aurko and Raffel, Colin and Goodfellow, Ian},\n  booktitle={International Conference on Learning Representations},\n  year={2018}\n}\n\nSuggestions: It would make the work more convincing if the authors reported results on larger CTR datasets like CRITEO.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper explores a few different representation schemes to make neural networks work better for tabular data and present empirical results demonstrating that it improves on other simpler representation schemes such as linear or one-hot encoding schemes. The paper is also clearly written and the notation is easy to follow.\n\nWeaknesses: Several of the data-sets used are likely very small scale so it is possible that in the abundant data regime the gains from this method are small. I would also imagine that the reason the decision trees are better than neural networks on tabular data of the kind the authors study is also likely because it seems like a data limited setting. In large(r) CTR datasets like CRITEO, it is not clear that 1) decision trees would be better than MLPs and 2) the encoding scheme would give as big of a \"win\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "03ByRxbdWh", "reviewer": "Reviewer_HiMf", "summary": "The paper proposes several approaches for numerical feature embeddings for tabular deep learning models and experimentally investigates the gains that such approaches provide. Methodologically, simple linear embeddings, piece-wise linear embeddings, and periodic-activation-based embeddings are proposed and experimentally evaluated on a suite of many benchmark tabular datasets. The experimental results showcase an improved performance competitive with gradient boosted decision trees (GBDT) on GBDT-biased datasets.", "questions": "* Line 286: \"However, after coupling DL models with embeddings for numerical features, the situation changes and the MI dataset remains the only one where all DL models lag behind GBDT.\" It is true that all deep models lag behind only on a single dataset. However, this seems like a one-to-many comparison. Do you have recommendations for a single best model or a good rule of thumb? What should a practitioner use? \n\n* Table 6: Simple MLP achieves the best performance (with PLR embeddings). In general, from Table 6 we see that MLP>ResNet>Transformer when numerical embeddings are used. Do you have any explanation/intuition about why?\n\n", "limitations": "The authors stated that the work focuses on a generic aspect of deep learning models and that therefore the negative societal impact discussion does not apply. The limitations have been adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Possible performance improvements from using embeddings for numerical features are showcased\n* The experiments are conducted on a large suite of datasets\n* Good average rank improvements over GBDT on a GBDT-friendly benchmark are achieved by using numerical embeddings\n* The paper is very well-written and easy-to-follow\n* The code is in good shape\n\nWeaknesses:\n* These embeddings for numerical features are essentially feature engineering. Comparison to GBDT with feature engineering would dramatically improve the paper and the experimental results.\n* Methodological contributions of piece-wise linear embeddings are similar to feature binning [1] as the paper mentions in the introduction.\n* I believe automated feature engineering methods such as e.g. [2,3,4] should be mentioned in related work.\n* While the proposed numerical feature embedding approaches are compared to each other, it is unclear if they provide any benefit over e.g. SAINT [5] with their dense layer or small MLP embeddings.\n\n\n[1] Dougherty, J., Kohavi, R. and Sahami, M., 1995. Supervised and unsupervised discretization of continuous features. In Machine learning proceedings 1995 (pp. 194-202). Morgan Kaufmann.\n\n[2] Khurana, U., Samulowitz, H. and Turaga, D., 2018, April. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).\n\n[3] Kaul, A., Maheshwary, S. and Pudi, V., 2017, November. Autolearn—Automated feature generation and selection. In 2017 IEEE International Conference on data mining (ICDM) (pp. 217-226). IEEE.\n\n[4] Khurana, U., Turaga, D., Samulowitz, H. and Parthasrathy, S., 2016, December. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) (pp. 1304-1307). IEEE.\n\n[5] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B. and Goldstein, T., 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept