SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
Authors: Gowthami Somepalli
Avi Schwarzschild
Micah Goldblum
C. Bayan Bruss
Tom Goldstein
Venue: ICLR 2022 Submitted (2022)
Keywords: Transformer
Tabular
Attention
Contrastive Pre-Training

Abstract: Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare.  Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners.  However, recent deep learning methods have achieved a degree of performance competitive with popular techniques.  We devise a hybrid deep learning approach to solving tabular data problems.  Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method.  We also study a new contrastive self-supervised pre-training method for use when labels are scarce.  SAINT consistently improves performance over previous deep learning methods, and it even performs competitively with gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over $30$ benchmark datasets in regression, binary classification, and multi-class classification tasks.

Under review as a conference paper at ICLR 2022
SAINT: IMPROVED NEURAL NETWORKS FOR TABU- LAR DATA VIA Row ATTENTION AND CONTRASTIVE TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved degree performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, per- forms attention over both rows and columns, and it includes an enhanced embed- ding method. We also study a new contrastive self-supervieed pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even performs competitively with gradi- ent boosting methods, including XGBoost, CatBoost, and LightGBM, on average over 30 benchmark datasets in regression, binary classification, and multi-class classification tasks.
INTRODUCTION
While machine learning for image and language processing has seen major advances over the last decade, many critical industries, including financial services, health care, and logistics, rely heavily on data in structured format. Tabular data is unique in several ways that have prevented it from benefiting from the success of deep learning in vision and language. First, tabular data often contain heterogeneous features that represent mixture of continuous, categorical, and ordinal values, and these values can be independent or correlated. Second, there is no inherent positional information in tabular data, meaning that the order of columns is arbitrary. This differs from text, where tokens are always discrete, and ordering impacts semantic meaning. It also differs from images, where pixels are typically continuous, and nearby pixels are correlated. Tabular models must handle features from multiple discrete and continuous distributions, and they must discover correlations without relying on the positional information. Deep learning systems with specialized architectures that embrace these differences have the potential to improve performance beyond what is achieved by classical methods, like linear classifiers and random forests. Furthermore, without performant deep learning models for tabular data, we lack the ability exploit compositionality, end-to-end multi- task models, fusion with multiple modalities (e.g. image and text), and representation learning.
We introduce SAINT, the Self-Attention and INtersample attention Transformer, a specialized ar- chitecture for tabular data. SAINT leverages several mechanisms to overcome the difficulties of training on tabular data. SAINT projects all features categorical and continuous independently into a dense vector space. These projected values are passed as token embeddings into a transformer encoder which uses attention in the following two ways. First, "self-attention" attends to individ- ual features within each data sample. Second, we propose novel "intersample attention", which enhances the classification of a row (i.e., a data sample) by relating it to other rows the table. In- neighst classification, where the distance metric is learned
tersample attention is akin end-to-end rather than fixed. In addition to this hybrid attention mechanism, we also leverage self- supervised contrastive pre- me-training to boost performance for eemi-supervised problems.
We provide comparisons of SAINT to a wide variety of deep tabular architectures and commonly used tree-based methods using diverse battery of tabular datasets. We observe that SAINT,

---

Under review as a conference paper at ICLR 2022
erage, outperforms all other deep learning methods on supervised and semi-supervvied tasks. More importantly SAINT often out-perform boosted trees (including XGBoost (Chen & Guestrin 2016), CatBoost (Dorogush et al. 2018). and LightGBM (Ke et al. 2017)) which have long been in- dustry favorite for complex tabular datasets. Finally, we visualize the attention matrices produced by our models to gain insights into how they behave.
2 RELATED WORK
Classical Models. The most widely adopted approaches for supervised and semi-supervised learn- ing on tabular datasets eschew neural models due their black-box nature and high compute require- ments. When one has reasonable expectations of linear relationships, many modeling approaches are available (Wright 1995 Weisberg, 2005 Starkweather & Moske 2011 McCulloch & Neuhaus 2005). In more complex settings, non- parametric tree-based models are used. Commonly used tools such as XGBoost (Chen & Guestrin 2016), CatBoost (Dorogush et al. 2018), and LightGBM (Ke) (t al. 2017) provide several benefits such as interpretability, the ability to handle a variety of feature types including null values, as well as performance in both high and low data regimes.
While classical methods are still the industry favorite, some recent work brings deep learning to the tabular domain. For example, TabNet (Arik & Pfister 2019) uses neural networks to mimic decision trees by placing importance only few features at each layer. The attention layers in TabNet do not use the regular dot- dot-product self-attention common in transformer- based models, but a sparse layer that allows only certain features to pass through. Yoon et al. (2020) propose VIME, which employs MLPs in a technique for pre-training based on denoising. TA TABERT (Yn et al. 2020), , a more elaborate neural approach inspired by the large language transformer model BERT (Devlin et al. [0188) is trained on semi-structured test data to perform language-specific tasks. Several other studies utilize tabular data, but their problem settings are outside of our scope (Pathak
Deep Tabular Models.
ct al. 2016 Chen et al. 2019 Padhi et al. 2021 Shavitt & Segal 2018 Katzir et al.) 2020).
Transformer models for more general tabular data include Tab Transfosfrre (Huang al. 2020 which uses a transformer encoder to learn contextual embeddings only on categorical features. The continuous features are concatenated the embedded features and fed an MLP. The main issue with this model is that continuous data do not go through the self-attention block. That means any information about correlations between categorical and continuous features lost. In our model, we address the issue by projecting continuous and categorical features to the higher dimensional em- bedding space and passing them both through the transformer blocks. addition, we propose a new type of attention to explicitly allow data points to attend to each other to get better representations.
Axial Attention. Ho et al. (219) propose row and column attention in the context of localized attention in 2-dimensional inputs (like images) in the Axial Transformer. For a given pixel, the attention is computed only on the pixels that are on the same row and column, rather than using all pixels. The MSA Transformer (Rao et al., (021) extends this work to protein sequences and applies both column and row attention across similar rows (tied row attention). TABBIE (Tida et al.) 202T) is adaptation that applies self-attention to rows and columns separately, then averages the representations and passes them as input to the next layer. In all these works, different features from the same data point communicate with each other and with the same feature from a whole batch of data. Our approach, intersample attention, is hierarchical in nature; first, features of data point interact with each other, then data points interact with each other using entire rows/samples.
In a similar vein, Graph Attention Networks (GAT) Veličković et al. 2017) compute attention over neighbors on a graph, thereby learning which neighbor's 's information relevant to a node's prediction. One way to view our intersample attention is as a GAT on complete graph where all rows are connected to all other rows. Yang et al. (2016) explore hierarchical attention for document classification where attention is computed between words in given sentence and then between the sentences, but they did not attempt to compute the attention between entire documents themselves.
Self-Supervised Learning. Self-supervision via 'pretext task' unlabeled data coupled with finetuning on labeled data is widely used for improving language and computer vision models. Some of the pretext tasks previously used include imputing masked values (Pathak et al. 2016; Arik & Phister, [2019] Huang et al. 2020), denoising (Vincent et al. 20088] Yoon et al. (2020), and replaced Inspired by the success of contrastive learning
token detection (Huang al. 2020 lida et al. 2021
2

---

Under review as a conference paper at ICLR 2022
for images (Chen et al. 2020 He et al. 2020 Grill et al. 2020), propose variant of contrastive
pre- pre-training for tabular data. To the best of our knowledge, this is the first work to adopt contrastive learning for tabular data. We couple it with denoising pre-train plethora of datasets with varied volumes of labeled data, and we show that our method outperforms boosting methods.
SELF-ATTENTION AND INTERSAMPLE ATTENTION TRANSFORMER (SAINT)
[ni__: (((())))
Contextual representations batch
Firbwdding layct
SAINT
Projecioo Head
Contrastive
B(-))
PL
S(-)
9t
InfuNCE
Add & Nown
FredForwand
Attention block
Interr pntersample
B
?
p.
S(-)
gh (-)
Projection Tirs
CutMis space
Misup in Jateri spaad
of MLP.s for each faaer cuapee CLS
Malti
Intenampl:
Atention
Ex
Denoising
SELF SUPERVISED PRE-TNAININ
PeedForward
Selt- Self-Attenbil Attention blook
SAIN
Add Noet
Embbeddiza layer EX{)
[Ct.s]
6
MSF (Ragsssis)
Crocr Cross Eneepyy (1 (Uloxileaioo))
Malti-Heal
8())
6%
Self-Atteriion
Embedded inputs of batch [B(x)))
SUPERVISED FINETUNING
Figure 1: The SAINT architecture, including pre-training and training pipelines. (a) Inspired by Vaswani et al. (2077). we use L layers with 2 attention blocks each, one self-attention block, and one our novel intersample attention blocks that computes attention across samples (see Section 3.2) (b) For pre-training, we minimize contrastive and denoising losses between given data point and its views generated by CutMix and mixup (Section 4.. During finetuning/regular training, data passes through an embedding layer and then the SAINT model. We take the contextual embeddings from SAINT and pass only embedding corresponding to CLS token through an MI.P to obtain the final prediction.
In this section, we introduce former (SAINT) and explain its components. Suppose D = {x; yi ) is a tabular dataset with m points, x; is n-dimensional feature vector, and y; is label target. Similar to BERT (De- vlin et al. 2018), we append a (CLS) token with a learned embedding to each sample. Let ,.., ((( be single data-point with categorical continuous features (1) and let E be the embedding layer that embeds each feature into Rdd, Note that may use differ- ent embedding functions for different features. For a given E [R(++)), we get E(x)) E g(n+1)xd
model, Self-Attente and INtersample attention Trans-
X; [[CLS],
Encoding the Data. In language models, all tokens are embedded using the same procedure. How- ever, in the tabular domain, different features can come from distinct distributions, necessitating a heterogeneous embedding approach. Note that tabular data can contain multiple categorical features which may use the same set of tokens. Unless it is known that common tokens possess identical relationships within multiple columns, is important to embed these columns independently. Un- like the embedding of Tab TooTsfforrrrrr (Huang et al. (020). which uses attention to embed only categorical features, we propose also projecting continuous features into dddimeesionall space before passing their embedding through the transformer encoder. To this end, we use separate single fully-connected layer with a ReLU nonlinearity for each continuous feature, thus projecting the --dimensional input into d d--menssional space. With this simple trick alone, significantly improve the performance the TabTransformer model as discussed Section 5. An additional discussion concerning positional encodings can be found in Appendix C
3

---

Under review as a conference paper at ICLR 2022
LayerNorm + Residall
G21 softumax *
202
H01A
M2.3 softmax 2 *
eo softmax
vd
v@
19
V1
k3
93
CONCAT
CONCAT
CONCAT
B(-)
B(-)
E(-)
[CLS]
CLS)
X1
X3
Figure 2: Intersample attention batch of points. is the size of value vectors See Sec.
3.2 for details.
3.1 ARCHITECTURE
SAINT is inspired by the transformer encoder of Vaswani et al. (017)). designed for natural lan- guage, where the model takes a sequence of feature embeddings and outputs contextual represen- tations of the same dimension. A graphical overview of SAINT is presented in Figure Ia). aa).
SAINT is composed of a stack of L identical stages. Each stage consists of one self-attention transformer block and one intersample attention transformer block. The self-attention transformer block is identical to the encoder from Vaswani et al.] (2017). It has a multi-head self-attention layer (MSA) (with h heads), followed by two conneccon ferd-forward (FF) layers with GELU non-linearity (Hendrycks & Gimpel 2016). Each layer has a skip connection (He et al., 2016) and layer normalization (LN) (Ba et al. 2016). The intersample attention transformer block is similar to the self-attention transformer block, except the self-attention layer is replaced by an intersample attention layer (MISA). The details of intersample attention layers are presented below.
The SAINT pipeline, with a single stage (L 1) and a batch of b inputs, is described by the following equations. We denote multi-head self-attention by MSA, multi-head intersample attention by MISA, feed-fon forward layers by FF, and layer norm by LN:
(1) LN(MSA(E(x))) Exi)
x (2) LN(FF (4 (1) + +2 (1)
2
21
(3) = LN(MMSA((2) (2) )))))) 2 (2)
rj LN(FF
(2 (8)) + =1-
(3)
(2)
where ri is SAINT's contextual representation output corresponding to data point Xj. This contex- tual embedding be used in downstream tasks such as self-supervision or classification.
3.2 INTERSAMPLE ATTENTION
We introduce intersample attention (a type of row attention) where attention is computed across different data points (rows of a tabular matrix) in given batch rather than just features of a single data point. Specifically, we concatenate the embeddings of each feature for single point, then compute attention over samples (rather than features). This enables us to improve the representation of point by inspecting others. When feature is missing noisy in one row, intersample attention enables SAINT borrow the corresponding features from other similar data samples in the batch.
An illustration of how intersample attention is performed in a single head is shown in Figure 2 and the pseudo-code is presented in Algorithm [ Unlike the row attention used in Ho et al. (2019); Child et al. (2019); Rao et al. (2021); > Tida et al. I), intersample attention allows all features from different samples to communicate with each other. In our experiments, we show that this ability boosts performance appreciably. In the multi-head case, instead of projecting q, given dimension d, we project them to d/ where is the number of heads. Then we concatenate all the updated value vectors, Us get back vector of length d.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 PyTorch-style oscudo-code for intersample attention. For simplicity, we describe just one head and assume the value vector dimension same as the input embedding dimension.
# b: batch size, number of features, d: embedding dimension
# W q, W k, W are weight matrices of dimension dxd # mm: matrix-matrix multiplication def self attention(x):
# is bxnxd
q, k, v mm (W x), mm (W (W_k, k, x), mm (W v) #q, k, are bxnxd attn softmax (mm
out = mm (attn, np np transpoee (k, (0, 2, / (d)) # bxnxn return out
#out is bxnxd
def intersample, attentionxx). is bxnxd
b, n, = x. shape # as mentioned above x reshape (x, b, (.dd) ) reshape to 1xbx (n+d)
self attention (x) the output is 1xbx (n*d)
out = reshape (x, (b, (D,,,d)) d)) out is bxnxd return out
4 PRE-TRAINING & FINETUNING
Contrastive learning, in which models are pre- pre-trained to be invariant reordering, cropping, or other label-preserving "views" of the data (Chen et al. 2020 He et al. [020)] Pathak et al. 2016 Grill et al. 2020 Vincent al. 2008), is a powerful tool in vision domains that has never (to our knowle ledge) been applied to tabular data. We present a contrastive pipeline for tabular data, visual description of which is shown in Figure [ Existing self-supervised objectives for tabular data include denoising (Vincent et al. 2008), variant of which is used by VIME (Yoon et al. 2020), masking, and replaced token detection as used by TabTransformer (Huang et al. 2020). We find that, while these methods effective, superior performance
be achieved by contrastive learning.
Generating augmentations. Contrastive methods in vision craft different "views" of images via crops and flips. For tabular data, crafting such transformations is non-trivial problem. VIME Yoon eet al., 2020) uses mixup in the non-embedded space as data augmentation, but this is limited to continuous data. We instead propose to CutMix (Yun al.. 2019) to augment samples input space and mixup (Zhang et al. (211)) in embedding space. These augmentations yield a challenging and effective self-supervision task. Assume that of data points are labeled. We denote the embedding layer by E, the SAINT network by S, and projection heads as 91 (+) and 92(+). The CutMix augmentation probability is denoted Peumixx and the mixup parameter a. Given point x;, the original embedding is E(xi), while the augmented representation is generated as follows:
x E
X3
m
① (1 m) X
CutMix in raw data space
(3)
P
+ a) * E(x,)
mixup in embedding space
(4)
where Xa. Xb are random samples from the current batch, X², is the CutMix version of Xb. m is the binary mask vector sampled from Bernoulli distribution with probability Peummix and a is the mixup parameter. We first obtain a CutMix version of every data point in a batch by randomly selecting partner mix with. We then embed the samples and choose new partners before performing mixup.
SAINT and projection heads. Now that we have both the clean pi and mixed p embeddings, we pass them through SAINT, then through two projection heads, each consisting of an MLP with one hidden layer and a ReL.U. A.l.U.10 The use of projection head before computing contrastive loss is common vision (Chen al. 2020 He al. 2020, Grill et al., 2020) and indeed also improves results on tabular data. Ablation studies and further discussion available in Appendix
Loss functions. We consider two losses for the pre-training phase. (i) The first is a contrastive loss that pushes the latent representations of two views of the same data point (2, and z/) close together and encourages different points (2; and 2,, i + 7) to lie far apart. For this, we borrow the InfoNCE loss from metric-learning works (Sohn 2016 Oord al. 2018 Chen et al. 2020 Wu al. 2018); (ii) The second loss comes from a denoising task. For denoising. we try to predict the original data sample from noisy view. Formally, are given ri and reconstruct the inputs x minimize
5

---

Under review as a conference paper at ICLR 2022
the difference between the original and the reconstruction. The combined pre-training loss is:
exp(z; 24/T))
Opre training
log E exp(zi 2x/T)
EE (M))
(5)
==
Contrastive Loss
Denoising Loss
where r; = S((p)) = S(p)), (1 (2 Lj is cross-entropy loss mean squared error depending the jth feature being categorical continuous. Each MLP; is single hidden layer perceptron with a ReLU non-linearity. There are n in number, one for each input feature. Ape is hyper-parameter and J is temperature parameter and both of these are tuned using validation data.
Finetuning, Once SAINT is pre-trained on all unlabeled data, we finetune the model on the target prediction task using the / labeled samples. The pipeline of this step is shown in Figure [b). For a given point X1, we learn the contextual embedding r₁. For the final prediction step, we pass the embedding corresponding only the [CLS] token through a simple MLP with a single hidden layer with ReLU activation to get the final output. We evaluate cross-entropy loss on the outputs for classification tasks and mean squared error for regression tasks.
5 EXPERIMENTAL EVALUATION
We now discuss SAINT variants and evaluate them in both supervised and semi-supervised scenarios on 30 datasets. We also analyze each component of SAINT and perform ablation studies to under- stand the importance of each component. Finally, we probe the behavior of attention maps in SAINT by treating the MNIST dataset (LeCun et al. 1998) tabular data and generating visualizations.
Datasets. We evaluate SAINT on 30 benchmark datasets, with 10 for each of binary classification, multiclass clas- sification, and regression. We chose datasets based on (i) usage in previ- tabular work (Arik & Pfister] 2019 Huang et al., (020) and (ii) availabil- ity the OpenML (Vanschoren al. 2013) platform where the datasets are uniformly processed and accessible | The datasets are diverse, including from
Table I: Configurations of SAINT. The number of stages is de- noted by J., and the number of heads in each attention layer is represented by /. The parameter count averaged over all the datasets with < 100 features and measured for batch size of 256. Time is the cost of 100 epochs of training plus inference on the best model, averaged over 30 datasets across all 3 tasks.
Model
Attention
L
Param xle6
Time (s)
SAINT-s SAINT-i
Self
6
8
5.9
1336
InterSample
2
11.81
684
SAINT
Both
2
12.23
731
452 up to 581,012 samples, with to 100 class labels, and containing from 6 1,777 features both categorical and continuous features. Some datasets are missing data while others are complete, and some are balanced, while others have highly skewed class distributions. dataset IDs and additional details can be found in Appendix B We pre-process each dataset by Z-normalizing all continuous features and by label-encoding all categorical features before data is passed to the embedding layer.
Model variants. The SAINT architecture discussed in the previous section has one self-attention transformer encoder block stacked with one intersample attention transformer encoder block in each stage. We also consider variants with only one of these blocks. SAINT-s has only self-attention, while SAINT-i has only intersample attention. See Table [for an architectural comparison.
Baselines. We compare model to traditional methods like Random Forests (Breiman 2001) Extra Trees (Geurts et al., 2006). and k-NN (Altman| 1992) as well as against powerful boosting libraries XGBoost (Chen & Guestrin 2016). LightGBM (Ke al., 2017]. and CatBoost (Dorogush et al. 2018). We also compare to neural networks, like 2-layer multi-layer perceptrons, Tab (Arik & Plister 2019), and TabTransformer (Huang al.) (2020). For methods that use unsupervised pre- training as preprocessing, we use Masked Language Modeling (MLM) (Devlin et al., (0118) for TabNet and Replaced Token Detection (RTD) (Clark al. 2020) for Tranffosfrrm as in the respective papers. We split the data into 65%. 15%, and 25% for training, validation, and testing, respectively. Hyperparameter tuning is done on validation data and results are reported on test data. Please refer to Appendix D for training details and hyperparameter search spaces for each method.
Datasets available via /tt.ts/s//www..gmm..rg////////o/
for each dataset: id.

---

Under review as a conference paper at ICLR 2022
Table 2: Average rank and the standard error across all 3 tasks individually and together for SAINT and other tabular methods. The average rank in individual task columns is computed over datasets while the overall rank column is computed over 30 datasets. The best performing model has the smallest rank (in bold). Added an additional column for new binary classification ranks after hyperparameter tuning of boosting baselines and additional baseline VIME.
Model Task
Binury
Binary New
Multiclass
Regression
Overall
RandomForest
5.9 + 0.80
5.9 + 0.75
5.5 + 0.56
7.3 1.22
6.2 + 0.52
ExtraTrces
1.07
5.8 = 0.95
7.2 + 0.63
6.7 6.7 1.11
6.6 = 0.56
KNeighborDDit
0.35
1
8.2 + 0.98
10.3 0.63
10.0 0.46
KNeighhorSUmif
0.21
0.47
8.5 +: 1.23
0.54
10.7 + 0.53
LightGBM
0.68
0.60
3.2 + 0.65
0.93
4.3 = 0.45
XGBoost
3.0++
8.67
4.5 + 0.65
5.4 1.4+00.00 0.65
5.0 5.0+0..42
CatBoost
0.50
0.50
5.2 0.73
4.0 0.64
Multi-layered Perceptron
0.61
0.60
6.3 1.00
9.7 0.63
8.1 0.50
VIME
10.8
TabNet
0.53
0.84
10.3 0.58
8.2
0.55
Tranffoforrmrr
0.46
8.6
7.1 1.02
0.71
8.0
SAINT-s
0.75
5.8
5.2 1488
3.8 1.11
4.8 0.66
SAINTi
4.9 0.69
5.1 + 0.62
4.2 + 0.66
4.0 + 0.86
4.4 0.42
SAINT
2.7
0.63
0.48
2.9
0.50
2.7 0.29
Metries. We used different metrics for each of the three tasks we examine in this paper. For binary- classification, we use area under ROC curve (AUROC), accuracy for multi-class classification, and poo-mean-sqqurree error (RMSE) for regression.
Training. We train (including pre-training runs) using AdamW (Loshchilov & Hutter 2017 with 31 = 0.9, 82 = 0.999, decay- = 0.01, and with a learning rate of 0.0001 with batches of size 256 (except for datasets with more than 100 features where vary the batch size depending number of features to fit the batch into a single GPU memory). We split the data into 65%, 15%, and 25% for training, validation, and testing, respectively. We use only validation split for selecting all other hyper-parameters. The final configurations for each of the datasets is presented in Appendix We use CutMix mask parameter Peutmix 0.3 and mixup parameter a 0.2 for all standard pre- training experiments. We use pre-training loss apper-parametrss Apt = 10 and temperature T = 0.7 for all settings.
the
5.1 RESULTS
Supervised setting. In Table 2 we report average ranks of each model on 10 datasets from each binary classification, multi-class classification, and regression task. Note that ranks are computed dataset-wise on a metric. Lower rank represents better performance. In binary and multi-class classification, we measure AuROC and accuracy, and SAINT ranks 2.7 and 2.5 respectively (across 10 datasets for each setting) compared to CatBoost's 2.9 and LightGBM's s 3.2 (the next best method). For regression, we measure RMSE. SAINT ranks 2.9 across 10 datasets compared to the next best method, CatBoost, which ranks 3.9. For complete results on all 30 datasets, see Appendix [E]
Semi-supervised setting. We perform 2 sets of experiments with 50, and 200 labeled data points (the rest are unlabeled). See Table 3 for average ranks across all 30 datasets and standard errors in both settings. We also examine the case when all data is labeled, to see if pre-training still helps. In all 3 cases, the pre-trained SAINT model performs the best. Pre-training had the largest positive impact on the SAINT-s variant with a rank improvement in all cases. Interestingly, we note that when all the training data samples are labeled, pre-training does not contribute appreciably (except for SAINT-s variant), hence the results with and without pre- pre-training are fairly close.
Effect of embedding continuous features. To understand the effect of learning embeddings for continuous data, we perform simple experiment with Tab TabTransformer. We modify Tab Tabrrssffrrrr by embedding continuous features into d dimensions using a single-layer ReLU MLP, just as they use on categorical features, and we pass the embedded features through the transformer block. We keep the entire architecture and all training hyper-parameten its modified version. In binary classification, the average AUROC across 10 datasets increases from 0.82 to 0.85 (an increment of 3.65%), and in multiclass classification, the average accuracy across 10 datasets increases from 0.72 to 0.74 (an increment of 2.77%). In regression, the average RMSE over 10 datasets decreases from 5,07 to 4,998.2 (a decrement of 1.44%). This experiment shows that embedding the continuous data is important and can boost performance significantly.
the same for both TabTransformer and
7

---

Under review as a conference paper at ICLR 2022
When to use intersample attention? In our experiments, we observe that SAINT-i outperforms other variants whenever the number of features is large (> 500) as observed in the cases of MNIST and Bioresponse. Another advantage of SAINT-i is that execution is fast compared to SAINT-s, despite the fact that the number of parameters SAINT-i much higher than that of SAINT-s (see [.. This can be attributed quadratic complexity in terms of number of features to compute attention in SAINT-s while in SAINT-i, this depends on the size of the batch which we can control.
How robust is SAINT to data cor- ruptions? We evaluate the robust- ness of SAINT variants by corrupt- ing the training data via noise and missing values. To simulate noise, we apply CutMix, replacing 10% to 90% of features with values of other randomly selected samples. We ex- amine the resulting trends in binary classification and regression. In tri- als with noise, the drop in mean AUROC mean scaled-RMSE is minimal until 70% of data is cor- rupted, when the performance drops significantly. SAINT and SAINT-i are comparatively more robust than SAINT--. This shows that using row attention improves the model's robustness to noisy training data. However, we find the opposite trend when many features are missing in the training data. In this scenario, SAINT-s and SAINT are quite robust, and the drop in AUROC is not drastic until 70% or more of the data is missing. We conclude that SAINT is reliable for training on corrupted training data. Trend line plots for these scenarios are available in Appendix F
Table 3: Average rank and standard error over 30 datasets under semi-supervised scenarios. Columns vary by number of labeled training samples. In last column we compare how models perform with and without pretraining when all the samples are labeled.
Model \ # Labeled
50
200
All
RandomForest
9.9 1 0.79
8,5 1 0.67
8.1 1 0.58
ExtraTrees
10.2 + 0.75
8.0
8.4 0.63
Deeigh
13.2 0.67
12.4 0.58
KNeighborsUnif
12.3 1.777
13.8 1.48
13.1 1 0.67
LightGBM
0.58
0.69
6.0 0.54
XGBoost
0.73
7.22
6.7 0.50
CatBoost
0.61
6.0
5.7 0.46
MLP
10.9
11.7 1.57
11.1 1 0.95
Tahnet MLM
1.74 0..5
9.9 + 0.61
TabTTrnsf. + RTD
10.3 0.82
0.77
8.8 = 0.83
SAINT-s
6.2
0.64
7.5 1 0.95
SAINT-i
6.0 0.67
6.2
6.1 + 0.43
SAINT
0.72
5.1 0.57
4.2 = 0.36
SAINT-s pre-training
5.5 0.75
6.6 + 0.72
SAINT-i pre-training
5.9 = 0.70
5.6 1: 0.60
6.1 + 0.87
SAINT + pre-training
4.3 + 0.63
4.6 + 0.63
4.3 + 0.77
Figure
Effect of batch size intersample attention performance. As we discuss in Section 3.2 in- tersample attention is computed between batches of data points. We consider multiple mu mllai- classification datasets and examine the impact of batch size by varying it from 8 to 1,024. In most cases, we find that the variation in SAINT-i's performance is small and is comparable to that of SAINT-s. which has no intersample attention component and thus has no dependence on batch size. We note that when the number of training samples is very small (i.e. less than 1,000), the variation in performance increases, but we observe the same trend in SAINT-s as well. We present corresponding plots in Appendix Figure 7
Further discussion and analysis is presented in Appendix
5.2 INTERPRETING ATTENTION
One advantage of transformers is that attention lends itself to interpretability tools as opposed to MLPs, which are hard to interpret. In particular, when we use only one transformer stage, the attention maps reveal which features and which data points are being used by the model to make decisions. We use MNIST (LeCun et al. 1998) data to examine how self-attention and intersample attention behave in our models. While MNIST is not a typical tabular dataset, it has the advantage that its features can easily be visualized as an image.
Figure 3a depicts the attention on each of the pixels/features in a self-attention layer of SAINT. Without any explicit supervision, the model learns to focus on the foreground pixels, and we clearly from the attention map which features most important to the model. The self-attention plots of SAINT-s are similar (Appendix G Figure Sa).
Figures 3b and 3c depict similar visualization a batch of 20 points, two from each class in MNIST. Figure 3b shows intersample attention in SAINT. This plot shows which samples attend to which other samples in the batch. Surprisingly. very few points in batch receive attention. We hypothesize that the model focuses on a few points that are critical because they are particularly difficult to classify without making direct comparisons exemplars in the batch. In Figure 3c
8

---

Under review as a conference paper at ICLR 2022
C
CO
Point in batch (Lahols) www.po--.-.......... www.----asaaaa www.saattt
Points batch (1 (Lahals)
3
3
00 23 6
0 23 6667666677 9
Attended to
Attended to
(a) Self-attn. in SAINT
(b) Intersample attn. in SAINT
(c) Intersample attn. in SAINT-i
Figure 3: Visual representations various attention mechanisms.
Libel
Figure 4: 1-SNE visualization of value vectors in intersample attention layers of SAINT (left) and SAINT- (right). plot 3500 points in each plot, with classes uniformly represented. the left plot, the most attended classes are 1, 2, 3, and 8. But in the right plot, the most attended classes are 0, 2, 6, and
show the intersample attention plot from a SAINT-i model. The same sparse attention behaviour persists here too, but the points being attended to different in this model. Interestingly, find this behavior to be significantly different on the Volkert data (Dataset ID: 41 166, multiclass with 10 classes), where wide range of data becomes the focus of attention depending the input. The intersample attention layer gets dense with the difficulty (to classify) of the datasets. See Appendix for additional MNIST and Volkert attention maps.
Figure F shows the behavior of attention at the dataset (rather than batch) level. We visualize a t-SNE Van der Maaten & Hinton 2008) embedding for value vectors generated in intersample attention layers, and we highlight the points that are most attended to in each batch. In Figure 4 (left), the value vectors and attention are computed on the output representations of self-attention layer. In contrast, value vectors and attention in Figure 4 (right) are computed on the embedding layer output, since SAINT-i does not use self-attention. In these plots, the classes to which the model attends vary dramatically. Thus, the exact classes to which an attention head attends change with the architecture, the trend of using few classes as 'pivot' seems prevalent in intersample attention heads.
6 CONCLUSION
Even though tabular data is an extremely common data format used by institutions in various do- mains, deep learning methods are still lagging behind tree-based boosting methods in production. In this paper, we highlight the limitations of existing deep learning approaches, and the reasons why they do not perform competitively on the tabular datasets. We further introduce three novel improve- ments to counter these limitations (1) intersample attention, (2) contrastive pre-training, (3) and strategy for embedding continuous columns the tabular datasets. Our method, SAINT, performs competitively across large number of tabular datasets with diverse characteristics the tasks of binary classification, multi-class classification and regression. We quantitatively and qualitatively analyse the importance of each of our improvements.

---

Under review as a conference paper at ICLR 2022
7 ETHICS AND REPRODUCIBILITY STATEMENT
The tremendous success of deep learning in vision and language domains is yet to be realized for tabular datasets. In our work, discuss some of the potential limitations of existing deep learn- ing models and how we can mitigate them by deploying not just features but other data points for the inference. We believe that our proposed architecture, along with the self-supervised and data augmentation techniques, will greatly help with representation learning and machine learning appli- cations tabular data domain.
With that in mind, like most deep learning models, SAINT in its current form is prone to capturing and even amplifying the biases in the dataset on which trained on. Our data augmentation strategy might help to mitigate this issue to some extent but by no means is it intended to resolve the data imbalance. We caution the end users to carefully consider their use cases and use our method only after appropriate validation and testing.
Tabular datasets often contain demographics or personally identifiable information (PII). Leaking of information from training data with large deep learning models is topic of great concern and has been demonstrated to a certain extent in the language domain. However, we are not aware of such works in the case of tabular data.
Reprodueibility. We are fully committed to open sourcing our code for reproducibility, as well as for the broader community to build upon it. Please find the code and instructions to use it attached in the supplementary material.
REFERENCES
Naomi S Altman, An introduction to kernel and nearest-neighbor nonparametric regression. Statistician, 46(3):175-18 1992.
The American
Screan O Arik and Tomas Pfister. arXiv: 07442, 2019.
Tabnet: Attentive interpretable tabular Iearning. arXiv preprint
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 2016.
arXiv preprint
Leo Breiman. Random forests. Machine learning, 45(1):5-33 2001.
Jiaoyan Chen, Ernesto Jiménez-Ruiz lan Horrocks, and Charles Sutton. Learning semantic annotations for tabular data, arXiv preprint arXiv: 007811 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. arXiv:2011.1066. 2020.
Exploring simple siamese representation learning.
arXiv preprint
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transform- ers. arXiv preprint arXiv: 10509, 2019.
Kevin Clark, Minh- Taah-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.1 1336, 2018
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data, arXiv preprint NXX::2003300650 2020.
Pierre Geurts, Damien Ernst, and Louis Wehenkel. Extremely randomized trees. Machine learning, 63(1): 3-42, 2006.
10

---

Under review as a conference paper at ICLR 2022
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre II Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self- self-suuee learning. arXiv preprint arXiv:2006.077 07733, 2020.
Kaiming He, Xiangyu Zhang, Shanqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9738, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian emorr linear units (gelus). arXiv preprint arXiv:1606.08415. 2016.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 1rXiv:1912.12180, 2019.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arX6v:20/2..66678 2020.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. Tabbie: Pretrained representations of tabular data, arXiv preprint arXiv:2105.0255 2021.
Liran Katzir, Gal Elidan, and Ran Yaniv. Net-dnf: Effective deep modeling of tabular data. In International Conference on Learning Representations, 2020.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightghm: A highly efficient gradient bonsting decision tree. Advances in neural information processing systems, 30:3 00:3146-3154 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6998 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 1998.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:111..05101. 2017.
Charles E McCulloch and John M Neuhaus. Generalized linear mixed models. Encyclopedia of biostatisties. 4, 2005.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint (rXXiv:807.00744 2018.
Inkit Padhi, Yair Schiff. Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3565-3569. IEEE, 2021.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Picter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021.
Ira Shavitt and Eran Segal. Regularization learning networks: deep learning for tabular datasets, arXiv preprint :.45::8000004400 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1857-1865, 2016.
Jon Starkweather and Amanda Kay Moske. Multinomial logistic regression, 2011.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research. 911), 2008
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49-60, 2013. doi: 10.1 145/2641 190.2641198. URL. http:
doi aam. www.aaccrrgggg wamm..rrg110.. 11452/11,,11//2111111

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need, arXiv preprint arXiv:1706.03762. 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Sanford Weisberg. Applied linear regression, volume 528. John Wiley & Sons, 2005.
Raymond Wright. Logistic regression. 1995.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric in- stance discrimination. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Zichao Yang. Diyi Yang. Chris Dyer. Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics; human language technologiex, pp. 1480-1489, 2016.
Pengeheng Yin, Graham Neubig, Wen-tau Yih, and Sehastian Riedel. Tabert: Pretraining for joint understand- ing of textual and tabular data. arXiv preprint arXiv:2005.0833 083314 2020.
Jinsung Yoon, Yao Zhang. James Jordon, Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Prucessing Syxtems, 33, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 11X 2017.
12

---

Under review as a conference paper at ICLR 2022
Appendix for SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre- Pre-Training
A ADDITIONAL ILLUSTRATIONS
LayerNorm
RRssduul->>
#02,0
* #02,3
0220
122,
#,2,
2,
022
022
Q1
Q2
ko
kj
k2
U2
13
V/
E
E
E(.)
E
2)
{3)
[CLS]
Xi
Figure 5: An illustration of self-attention in point Xj. Inspired by Vaswani al. (2017).
B DATASETS
Data sources, As mentioned in main text, all 30 datasets are downloaded from OpenML Van- schoren al. (013) repository. We present the dataset IDs and some statistics of the datasets in Ta- 4 All datasets can be downloaded via wollttts::/// htttt:s::/www.opemm.l. wrtp: a/<datase__ic
Data preprocessing. each dataset, the categorical features are label encoded, and continuous features z-normalized before passing them into the embedding layer. Each feature (or column) has a different missing value token to account for missing data.
For MNIST, unravel each image into vector of 784 features and consider each image single row. Since each feature is of same type in this dataset, we encode the features into the same embedding space. To distinguish the features, we also positional encodings in the encoding layer.
C COMPLETE TRAINING DETAILS
In each of experiments, single Nvidia GeForce RTX 2080Ti GPU. Individual training runs take between minutes and hours. In total, the experiments this paper account for around 30 GPU days (including semi-supervised experiments and ablation studies).
13

---

Under review as a conference paper at ICLR 2022
Table 4: We present statistics 30 datasets have used in this paper. The first columa is the OpenML ID for each of datasets. regression datasets, we denote the number of classes as N/A.
OpenML
Name (as appears OpenML)
# classes
# features
# continuous features
calegorical festures
Size
Tisk
31
credit-g
2
14
1000
binary classif
44
spombuse
2
58
57
1
4601
binary classit
1017
arthythmia
280
206
74
452
binary classif
KDDCup019 Jepeteeccy
hinary
1487
Lovee-love-Sttr
2
73
72
2534
binary classit
1494
biodeg
41
1055
binary classif
binary classif
4134
Bioresponse
2
1777
1776
1
3751
binary classit
42178
lelco-custonas churn
0
7043
binary classit
42733
prediction small
binary classif
188
eucalyprus
5
20
6
7336
malticlass
1.56
covertype
45
5S1012
multiclass
Diabetes 130US
multiclass
40664
cur-evaluntion
22
1728
multicluss
40685
shuttlc
7
58000
multiclass
multiclass
41975
car volkert okcupid-sten stem
1728
multicluss
41166
10
180
58310
multiclass
41169
42734
18
50789
multiclass
422
topo_2.1
N/A
267
267
0
8881
regression
togresioon
42563
house prices.noming
NIA
37
43
1460
regression
42571
Allstate Claims Severity
NIA
131
15
116
50000
regression
42705
Yolanda
rguession
42724
OnlineNewsPeeularrty
NIA
60
0
39644
regression
42726
ahalone
NIA
4177
regression
colleges
regesssonn
42728
Dirlines,
N/A
50000
regression
42729
N/A
19
10
9 50000
regression
For most of the datasets, with > 100 features we used 4. The variance in the embedding size is only due to the memory constraints of single GPU. We used L 6 layers in the SAINT-s variant and L for SAINT- i and SAINT variants. We use dropout of 0.1 in all attention layers. In feed-forward layers. use dropout of 0.1 in the SAINT-s variant, and use 0.8 SAINT-i and SAINT models. We use attention heads h = 8 for SAINT-s model and h = for SAINT-i and SAINT models. Inside the self-attention layer, the q, and v vectors are of dimension 16, and in the intersample attention layer, they are of size 64.
embedding size d = 32. For MNIST,
d 12, for datasets
Other minor details are shared the code.
Feature Encoding. Transformers for vision and language typically employ same embedding func- tion for each feature, column since each word, pixel, or a patch comes from similar distri- bution. This is not the case with all of the 30 datasets presented in this paper; each feature may be of a different type and drawn from different distribution. Hence we use unique embedding function for each column.
Positional Encoding. tabular datasets, different features, or columns do not have any order. Different feature encodings used to embed different columns is sufficient to provide the model the necessary information about the feature. Hence we skip the positional encodings in all the datasets (except in the case of MNIST, where all columns share the embedding function, and we provide positional encodings instead). Note that the attention operation itself is order invariant both the self-attention and intersample attention cases.
D BASELINES
We use Auto- Auto-Gluor? Erickson et al. 2020 framework to the train the traditional models as well as to perform the hyperparameter sweep. For TabNet, we use the famous Py Torch implementa- tion and for TabTransformer since the official implementation is not available, used the widely
14

---

Under review as a conference paper at ICLR 2022
used PyTorch re-implementatiorf" In the following paragraphs we will give details hyperparam- eter configurations for each baseline. All the baseline models use early stopping on the validation dataset's s performance. All the deep learning methods take embedding for categorical data input and we tuned them with variable embedding sizes from [16, 32, 64].
Random Forest Extra trees Classifiers. We tuned on number of estimators (n.estimatori) from the list [100, 200], on criterion from ['gin', 'entropy'I
Random Forest & Extra trees Regressors. We tuned number of estimators (n.estimators) from list [100, 200), on criterion from I'squared "absoluteeerror])
KNeighbors models. We tuned the we i ghtts parameter from ['uniform' 'distance'), and p value from [1,2]. For large datasets, k-NN model is computationally intractable and hence for datasets with 50000 datapoints, we randomly choose 50k points and train model on the smaller dataset.
LightGBM. We tried with different values of min-dataa in leaf in [10, 20). num leaves in [30,50] and rete in the interval [0.01,0.1 0.1]
XGBoost. We used the values of booster as "gbtree', n.est imators of 200 and tuned over L2 regularization lambda in [1, 10] and the learning the interval [0.01, 0.3).
CatBoost. We used the values of reg lambda 100 and early stoppi ng rounds 500 in the interval 0.1].
and tuned the learn
MLP. We used a 2 Ladden-layered neural network of sizes [200, 100), with 1d BatchNorm opti- mized Adam (Kingma & Ba 2014) with learning rate to 0.001.
TabNet. We have used the hyperparameters suggested by the PyTorch implementation of Tabnet. set to 0.02 and moment um set to 0.3. We used
We used Adam optimizer with learning.rar learning rate scheduler with gamma as 0.95 and step-size 20. The rest of the parameters have tuned based on the ranges provided in the original paper.
We used AdamW optimizer with learning.rate set to 0.001. also tried different levels of feed-forward drop-outs from [0.1, 0.8].
Tranfoosfrrm.r.
E EXTENDED RESULTS
In the main part the paper, we have presented the average ranks of the models across the bench- mark datasets. The supervised results are shared across the following three tables. In Table 5 we share the AuROC scores of all the baselines and SAINT variants for the binary classification task. In Table 6] we share accuracies for multiclass classification. In both these cases, higher the score, better the model. While in case of regression, we share the RMSE scores in Table I0 smaller the value, better the model.
In case of semi-supervised/ pre-training experiments, see the cxhaustive results for the case of 50 labeled samples in TT for 200 samples in Table and all samples labeled in Table [3] Please note that in these 3 tables, we have scores from different tasks together the same table.
F ADDITIONAL ANALYSES
How robust is SAINT to data corruptions? (contd.) As discussed in the main body, we evaluate the robustness of SAINT variants cases data corruption or missing data. We examine the cases of Binary classification and Regression and plot the average scores across the 10 datasets correponding to that task. We present the results Figure
15

---

Under review as a conference paper at ICLR 2022
Table 5: AuROC scores for binary classification datasets. The rows are models while the columns various datasets, represented by their OpenML IDs. Higher the better. Added new baseline VIME. Added additional entries for boosting results with more hyperparameter tuning. Added Average column. Added results for deep learning baselines after additional hyper-parameter tuning for datasets 31,44
Model OpenMI. ID
31
44
1017
1111
1487
1494
1590
4134
42178
42733
Average
RandomForest
0.778
0.986
0.798
0.774
0.908
0.868
0.840
0.670
0.8466
ExtraTrees
0.764
0.986
0,811
0.748
0.921
0.935
0,903
0.856
0.831
0.659
0.841
KNeighibrsDDis
0.722
0.517
0.684
0.808
0.755
0.705
KNeighhors Uniff
0,489
0.847
0.712
0.516
0.734
0.865
0.669
0.790
0.764
0.578
0.696
LightGBM
0.751
0.989
0.807
0,803
0.930
0.860
0.853
LightGBM best
0.752
0.989
0.829
0.815
0.919
0.923
0.930
0.860
0.854
0.683
0.855
X-BBoost
0.781
0.802
0.931
0.864
0.854
0.681
XGBoost.best
0.778
0.989
0.821
0.818
0.919
0,926
0.931
0.864
0.856
0.689
0.859
CatBoost
0.788
0.838
0.818
0.930
0.858
0.856
0.860
CatBoost best
0.788
0.988
0.838
0.818
0.917
0.937
0.930
0.862
0.841
0.686
0.860
MLP
0.745
0.709
0.910
0.818
0.841
0.820
MLP.bost
0.712
0.980
VIME
0.732
0.752
0.653
0.755
0.812
0.892
0.889
0.791
0.784
0.578
0.7638
TabNet
0.472
0.978
0.422
0.718
0.625
0.677
0,917
0.701
0.830
0.603
0.694
TabNet_hest
0.7361
0.9788
Tab Transformer
0.764
0.980
0.729
0,763
0.884
0,913
0.907
0.809
0.841
0.638
0.823
TabT Traanffrrmereeee chest
0.771
0.982
SAINT
0.774
0.982
0.781
0.804
0.906
0.933
0,922
0.819
0.858
0.656
0.843
SAINT-i
0.774
0.981
0.759
0.816
0.920
0.934
0.919
0.845
0.854
0.662
0.846
SAINT
0.790
0.991
0.843
0.808
0.919
0.937
0,921
0.853
0.857
0.676
0.859
Table 6: Accuracy scores for multiclass classification datascts. The rows are models while the columns are various datasets, represented by their OpenML IDs. Higher the better.Added Average column
Model \ OpenML ID
188
1596
4541
40664
406885
40687
40975
41166
41169
42734
AAverage
RonsomForessaatr
0.653
0.953
0.607
0.951
0.999
0.697
0.967
0.671
0.358
0.743
0.760
ExtraTreesEntr
0.653
0.946
0.595
0.951
0.999
0.697
0.956
0.648
0.341
0.736
0.752
KNeighhoesDist
0,442
0.965
0.49]
0.925
0.997
0.720
0.893
0.620
0.205
0.685
0.694
KNeigghoestnif
0.422
0.963
0.489
0.910
0.997
0.739
0.887
0,605
0.189
0.693
0.689
LighiGBM
0.667
0.969
0.61
0.984
0.999
0.716
0.981
0.721
0.356
0.754
0.776
XGBoost
0.612
0.928
0.611
0.984
0.999
0.730
0.984
0.707
0.356
0.752
0.766
CalBoost
0.667
0.871
0.999
0.730
0.962
0.692
0.376
0.763
MLP
0,388
0.915
0.597
0.992
0.997
0.682
0.984
0.707
0.378
0.733
0.737
TabNet
0.259
0.744
0.997
0.275
0.599
Tab Transformer
0.660
0.715
0.601
0.947
0.999
0.697
0.965
0.531
0.352
0.744
0.721
SAINTS
0.680
0.735
0.607
0.981
0.999
0.735
0.992
0.582
0.726
SAIVT-
0.646
0.937
0.598
0.995
0.999
0.735
0.981
0.713
0.380
0.747
0.773
SAINT
0.680
0,946
0.606
1.000
0.999
0.735
0.997
0.701
0.377
0.752
0.779
Table 7: RMSE values for regression datasets. The rows are models while the columns are various datasets, represented by their OpenMI. IDs. Lower the Rester.Results are shown in decimal places instcad of
Model \ OpenML
422
541
42563
42571
427U5
42726
42727
42728
42729
RandeenForestMSE
0.027
17.814
37085.577
1999.442
16.729
12375.312
2.476
0.149
13.700
1.767
ExurTTrcesMS
0.027
19,260
35049.267
1961.928
15.349
12505.090
2.522
13.578
1,849
DieighhorsDist
0.029
25.054
46331.144
2617.202
14.496
13046.090
2.501
0.167
13.692
2. 2.100
KNeightorsUail
0.029
24.698
47201.343
2629.277
18.397
12857.449
2.592
0,169
13.703
2.109
LightGBM
0.027
19.871
32870.697
11639.594
13.468
1.958
XCBoost
0.028
13.791
36375.583
1903.027
12.311
11931.235
2.452
0.145
13.480
1844
CatBoost
0.027
14,060
35187.381
1886.593
11,890
11614.567
2.405
0,142
13.441
1.883
NeuuraNeeF
0.028
22.756
42751.432
1991.774
15.892
11618.684
2.500
0.162
13.781
3.351
TabNet
0.028
22.731
200802.769 769
1943.091
11.084
11613.275
2.175
0,183
16.665
2.310
TabTransfommr
0.028
21.600
37U57.6886
1980.696
11618.356
2.494
3.259
SAINTS
0.027
9.613
193450.70 703
1937.
10.034
11.580835
2.145
0.158
12.603
1.833
SAINT
0.028
12.564
33992.508
1997.111
11.513
11612.084
2.104
0,153
12,534
1.867
SAINT
0.027
11.661
33112.387
1953.391
10.282
11577.678
2.113
0.145
12.578
1.882
Effect of batch size on intersample attention performance. (cont.) As discussed in the main body, we examine the affect of batch size on different SAINT variants in Figure 7 We pick multiclass classification datasets with varying numbers of features and samples. In all cases, that the variance in Accuracy is minimal when varying the batch size from to 1024. In all the plots, see logg (batchsize) plotted on the X-axis with accuracy on Y-axis.
16

---

Under review as a conference paper at ICLR 2022
Table 8: Scaled RMSE values for regression datasets. The models while the columns various datasets, represented by their OpenML Generated by dividing smallest RSME value across models for a given dataset. Lower the better.New table
Model OpenML ID
422
42563
42571
42705
42724
42726
42727
42728
42729
Average
RundomForesMSF
1.00
1.85
1.13
1.06
1.67
1.07
1.18
1.05
1.09
1.00
1.21
EXtraTreeMMS
1.00
2.00
1.07
1.04
1.53
1.08
1.20
1.04
1.08
1.05
1.21
DNcighbersDist
107
1.41
1.44
1.19
1.09
1.37
KNeighhorsUnif
1.07
2.57
1.44
1.39
1.83
1.11
1.23
1.19
1.09
1.19
1.41
LightGBM
1.00
1.07
1.17
XGBoost
1.04
143
1.11
1.01
1.23
1.03
1.17
1.02
1.08
1.05
1.12
CatBoost
1.00
1.07
1.14
NeuralNetFastAA
1.04
2.37
1.30
1.06
1.58
1.00
1.19
1.14
1.10
1.90
1.37
TabNet
1.44
1.03
1.33
1.76
TabTranstomer
1.04
2.25
1.13
1.05
1.56
1.00
1.19
1.14
1.04
1.84
1.32
SAINT
SAINT4
1.02
1.31
1,03
1.06
1.15
1.00
1.00
1.08
1.00
1,06
1.07
SAINT
1.00
SAINT
SAINT-i
SAINT-s
binary/ missing
binary noise
1.00
mean AUROC
0.75
0.25
regression missing
regression noise
mean scaled s RMSE
1.00
0.50
0.00
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
% of corruption
% of corruption
Figure 6: Robustness of SAINT's variants to data corruptions
F.1 PRE-TRAINING ABLATIONS
In Table D we study various configurations of pre-training components. We perform 3 primary studies: we vary (1) projection head, (2) pre-training loss, and (3) data augmentation method. Note, the final result in all 3 studies refers to the same experiment (hence the row is repeated), which is the final chosen configuration for our model.
Effect of projection heads. As described in Section we use two different projection heads, 91 (-) and 92(-). ). to project the contextual representations to lower dimensions and then compute contrastive losses. We study three different options for the heads: (1) distinct projection heads (2) heads with weight sharing, and (3) no projection heads at all. Table shows that using distinct projection heads performs best.
Varying pre-training loss. We train SAINT's variants with different loss functions, as shown in Study of T We try denoising and contrastive losses, in addition to cosine similarity loss positive pairs (inspired by Grill et al. (2020); Chen & He (2020). The combination of contrastive and denoising consistently yields the best results in all SAINT variants.
Varying the pre-training augmentations. We also try to understand how important it is to use CutMix and mixup generate augmented embeddings in the pre-training pipeline. We tinker with
17

---

Under review as a conference paper at ICLR 2022
SAINT
SAINT-
SAINT-s
dset_ie pet_id- 188
dset id = 40864
1.0
Accuracy
0.8
0.6
dset_ id = 40685
dset id 40687
1.0
Accuracy
0.8
0.6
10
8
log2(batchsize)
log2(batchsize)
Figure 7: Trend lines of Accuracy with varying training batch size. Results shown for datasets
Table 9: Ablation studies the pre-training pipeline of SAINT. We break
down the effect method. We report average AuROC over 10 datasets for the case where only 200 points in the dataset are labeled.
projection head, pre-training loss, and augmentation
Study
Variation
SAINT-s
SAINT-i
SAINT
no proj. head weight sharing head
0.752
0.757
0.768
0.780
0.784
0.790
w. diff proj. head
0.789
0.794
0.802
pre-training
0.774
0.788
0.792
contrastive
0.776
0.785
0.793
denoising cosine similarity
0.770
0.773
0.786
0.758
0.762
0.775
contra. denois.
0.789
0.794
0.802
CutMix
0.762
0.768
0.778
3
mixup
0.766
0.775
0.779
CutMix mixup
0.789
0.794
0.802
various configurations in Study of Table 9 and we observe that using these two augmentations in unison results in the best performance across all SAINT variants.
G ADDITIONAL INTERPRETABILITY PLOTS
In Figure Sa
we show self-attention plot for the SAINT-s variant (with L 1) on MNIST. The self-attention in one stage SAINT-s model behaves similar to one stage SAINT model. However, when there are more stages, the attention in the last stage is not quite as interpretible.
In Figure 8b
show the intersample attention between batch of points from different classes in SAINT model on the Volkert dataset (OpenML dataset ID: 41 166). Similarly in Figure we show intersample attention in the SAINT-i variant the same batch of points from the Volkert dataset.
18

---

Under review as a conference paper at ICLR 2022
As mentioned in the main body, the intersample behaviour is not quite sparse that of MNIST.
We hypothesize that the sparsity the intersample attention layer depends on how separable the classes in the dataset are. (Volkert is harder dataset than MNIST).
In Figure 9 we show the t-SNE plots on value vectors for SAINT and SAINT-i variants on Volkert. Unlike MNIST, all the classes attended equally.
C
G
Points in batch (Labels) 00 Ln wobtebooo www.
Points batch (I (aablls).
3
Attended to
Attended to
(a) Self-attn. in layered SAINT-s MNIST dataset
(b) Intersample attn. in SAINT in Volkert dataset
(c) Intersample atin. in SAINT-i in
Volkert dataset
Figure 8: Visual representations of various attention mechanisms. (a) Self-attention in SAINT-s on MNIST (b,c) Intersample attention in SAINT and SAINT-i on the Volkert dataset.
Jabch
-30
Figure 9: A t-SNE visualization of value vectors in intersample attention layers of SAINT (left) and SAINT- (right) the Volkert dataset. We plot 3000 points in each figure, with classes uniformly represented. Unlike MNIST, all classes are uniformly attended to this dataset.
Table 10: Runtime results for Binary classification. Time is the cost 100 epochs plus inference on the chosen Newel.Newly added table.
Model
Time(s)
LightGBM
298
XGBoost
136
CatBoost
423
MLP
213
VIME
458
TabNet
514
Tab Transsoorrr
1100
SAINT-s
1144
SAINT-i
487
SAINT
523
19

---

Under review as a conference paper at ICLR 2022
Table 11: Semi Semi-superree -wweruepeiieeee learning with 50 labeled samples case: Actual scores for each of the model for a given dataset. First 10 datasets correspond to multiclan lass
classification, hence the scores are accuracies, the next 10 belong to binary classification, so the scores are AuROCs, and the last 10 columms are regression type, hence they carry RMSE values. Removed the negative sign in RMSE scores
Table 12: Semi-supervised learning with 200 labeled samples case: Actual scores for each of the model for a given dataset. First 10 datasets correspond to mulliclass classification hence the scores are accuracies, the next 10 belong binary classification, so the scores are AuROCs, and the last 10 columns are regression type, hence they carry RMSE values. Removed the negative sign in RMSE scores
270
2.0
230
255
2.00
3.13
3.00
240
2.70
4272
2.20
2.3
2.00
200
235
222
7.38
5:11
500
2.0
230
200
3..0
300
3000
2035
90.5
28.23
16.00
1620
1633
16360
10.37
10.35
41728
2005
200
200
3500
300
2966
29
32.00
1958
415 WAAR
1936
1036
1635
4797
123
0250
0.2
015
011
025
0250
017
1T5
O1
115
405
015
22
2.59
2.34
43736
2.5
2.5
5.0
505
3.0
233
2358
2.25
55400000000
1NS0200000
1999200110
Roymssie
113500
113311
1101154
HIGTT
11070
11001
1007
Megneeien
42734
1061 1601/%
13
11001
183511
15022
1835
187201
130
1100
11.00
42
1.ST
122
1120
10A1
1000
ORS
11200
1120
1.2
3453112
3300
240000
200030
2407.3
43311
3051 301100
30%
2000
20000
20522S
3813S7
4600
20120
505030
603740
J0545
4173331
359060
30
4558
1502
20
200065
2727177
51
4000
72
35
25
52
541
2410
3075
3010
19.00
30.
19
10.00
1200
1932
2005
1114
0.75
016
42
100
000
610
11.1
1744
1052
0.2
1000
11
1000
11
020
055
42738
11.%%
11.55
1155
1255
1255
1055
115
1054
16:50
16:30
10.0
4EPN
000
03
0.05
0.02
0.52
0.5
0.11
0.5
0000
0.5
03
0.5
0.0
120
025
055
055
414
0.75
0%
0.0
070
027
0%
03
025
0.2
033
054
052
032
0.75
0.76
0.2
0.00
0.0
0.
Biurry Chadliicaioo tien
1890
0.3
035
0.55
0.55
033
03
0338
0-00
0338
1144
11
130
110
200
1300
1494
110
008
0191
015
R00
R91
114
Rinary
035
0.76
055
100
030
00
0.5
take
0.50
0000
0.00
0.47
0.5
0.5
0.00
0.04
0:50
0.5
0.4
0.0
0:5
0.5
060
100
000
0.0
0.46
0.45
000
000
020
050
055
141
0%5
0%0
00
027
030
030
0.56
0.2
1200
105
104
132
125
106
44
60%
10S
099
0.74
0.76
176
622
0.2
0.00
31
674
0.74
055
005
020
605
155
152
150
42%
000
040
000
011
071
071
011
000
0006
071
871
000
600
070
41
000
000
000
000
0000
200
100
5.00
011
0.10
<116
2014
001
0076
0076
001
060
200
200
2U7
41166
044
033
036
0721
0241
035
0751
076
0748
0.00
0239
0.50
032
0/9
049
040
044
020
022
014
010
4995
170
630
6.5
600
622
677
0.60
000
000
035
680
682
Malts Chaniilicatin
1.5
0.77
000
06
0.75
000
000
000
1000
0.70
1000
022
100
100
0002
0.91
0/6
Mainccaon
4963
000
001
100
005
100
180
46664
01:0
0765
035
075
0329
1.30
0.89
0.77
0330
0.50
0.74
0.74
0.2
0:35
0:35
022
023
1035
10.09
0.20
0.36
1000
1000
12.00
10.00
030
0.55
00.5
00.5
0.00
0.31
0.51
0.51
4641
0.38
0.5
0.1
0.30
0.07
035
0.54
0.5
1721
1122
0.50
1.50
11:00
11:00
1037
1030
1061
1053
055
1662
1156
1100
1000
1007
1000
0445
0.57
0.1
0:00
0112
0030
003
005U
0.30
040
188
005
0550
022
1040
040
0.52
01
0.90
000
000
SAINT po-taminng
TiTTund Comerr KID
pe maning
te tmaling
Mevica ristaee
MLM
Midial Medial/tatttt datuet
Tubrer MLM
Acttooa
SAING
SAINE
SAING
SAINE
Bank/
Esto Eane
Achloot
SUNTA
NLP
20

---

Under review as a conference paper at ICLR 2022
Table 13: Semi supervissd learning with ALL samples labeled case: Actual scores for each of the model for a given dataset. First 10 datascts correspond to multiclass classification, hence the scores are accuracies, the next 10 belong to binary classification, so the scores are AuROCs, and the last columns are regression type, hence they carry RMSE values. .R Removed the negative sign in RMSE scores
4275
1.7
LAS
2,70
2.1
1.00
03
13
135
ES
1.50
4224
11.00
1358
13.0
1320
13A
13
11
1530
1376
11
1335
1233
1235
1200
1351
120
4272"
01.5
01.
11
1.7
012
015
110
110
110
100
4726
246
232
230
230
225
24
120
230
2.50
210
211
210
200
47724
110.00
TON
TOEESY
TAXe
1
10020
1330
113002
122
42705
16.1
1430
LTN
11
125
100
3X3
1131
103
102
21571
19944
100
201220
30022
1000
19639
LM0
1901
MEEEE 1905S
WAINE 1940
150.11
1883
307
2014
4200
30005
300022
47003
5502231
541
1701
25.05
24.00
1030
13.0
14.
1529
22
2231
505
14.00
11.6
501
11.56
1200
422
0753
073
073
073
073
072
0725
172
1725
1723
0723
0723
1723
1723
172
4223
0/0
112
153
155
110
112
102
179
105
105
100
105
100
100
100
IN
321%
30.05
30.05
022
0036
000
11.
022
0.36
02
0.2
0.2
0.2
0.2
LOS
1590
0.0
030
0.26
020
035
005
000
030
000
UST
UO2
Biuary Elanllation
12%
000
000
003
0055
0022
002
000
000
000
100
100
1ON
1000
100
140
00
002
074
072
000
000
000
000
000
100
1111
1122
112
1720
1200
LOX
LIX
LIKE
100
0.:00
02.1
02.24
02.71
0231
00.00
12
10.00
44
030
6:00
020
0255
000
020
120
1200
LOSK
1X0
31
0.6
0.6
030
0.2
0.5
0.6
0.9
C22
030
000
4273
0.44
0.7
0.20
0220
02.5
075
025
022
022
0.
021
02
000
0.0
0.0
411e0
0.56
0.1
0.30
0.00
0.66
0.
035
00
000
35
100
21106
022
0X
002
00
072
011
020
031
021
035
011
020
DES
02
Chelliation
4075
0.6
039
0.0
0.0
000
1.0
7.01
40007
0.30
0.30
012
012
023
L22
L23
Malliclas
400s
150
100
20104
072
002
002
000
005
000
005
03
000
105
100
244
0221
0220
0.20
0120
021
0200
0230
0220
1330
1220
1221
122
JEN
15%
0.23
0.03
0.0
0.00
0.9
000
033
0.76
000
000 00T
00T
0074
000
1.0
108
0001
0001
0041
004
000
0001
000
0225
0030
300
0003
000
000
0000
27D
SAN aInE
SANTE gi0e 1 JUAIN
SANN 2-S BAG
Minker MnnEIraatee Inteet
Mank minunn
Flner MLM
TaskAA
Earr Lans
SINT4
SANT
SANT
MLF
21


Reviews:
[{"review_id": "aH45NlYvhsv", "reviewer": "Reviewer_bykr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Introduction does not cite literature sufficiently when discussing general tabular data ML.\n\n- It is unclear why encoding continuous features with embeddings improves the performance. Although the experimental results support that, more motivations and analysis on this should be presented.\n\n- The major concern with benchmark methods is that their hyperparameters are not fully tuned. For deep learning based methods, tuning learning rate and the number of hidden units is very important for high accuracy. For ensemble decision trees, tuning the number of decision trees is very important. Without full-scale tuning, the outperformance claims of SAINT are not convincing. I suggest carefully looking at how the corresponding methods tune the hyperparameters in the original papers and mimic the same process. Otherwise, the outperformance claims of SAINT are not credible. \n\n- The impact of augmentation on the results is unclear. More results with different augmentations methods should be added to validated the CutMix choice. \n\n- Explainability capabilities are not presented in a very convincing way. Is there a ground truth for feature importance to quantify the accuracy of the explanations? Maybe such an experiment on synthetic data would add value. Merely showing the sparse activations does not prove whether the attention maps are accurate.  \n\n- Beyond showing the average improvement results, it would be great to focus the discussions on on the specific tasks and datasets where SAINT shows the biggest improvements. \n\n- Overall, the novelty of the paper is not high. Most of the constituent ideas (e.g. sample-wise attention, transformer modules, contrastive learning etc.) were published in other papers. This paper combines them in a judicious way and shows important improvements, but most of the contributions are empirical. \n\n\n", "summary_of_the_paper": "The paper introduces SAINT, a deep learning model for structured data that utilizes attention between rows, as well as contrastive pre-training. Performance improvements in supervised and semi-supervised settings are demonstrated, along with robustness and explainability capabilities. ", "summary_of_the_review": "Overall, the paper has important contributions, especially in improving semi-supervised learning for tabular data. However, there are different issues in the content listed above, that need to be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WT9bIgu3kPs", "reviewer": "Reviewer_Qd8Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed model reflects the characteristics of tabular data well, and easy to follow.\n- The authors conduct an extensive set of experiments with 30 different datasets, which is much appreicated.\n\nWeaknesses:\n- Debatable prediction performance of SAINT: The main evaluation result in Table 2 is reported in ranks, not the actual relevant metrics such as AUROC, AUPRC, Accuracy, MSE, R^2. This makes it very hard to trust the results, since the reader has no idea how well SAINT is doing compared to baselines. And when you look at the actual performance metrics in the Appendix (Table 5, 6, 7), it is evident that SAINT's performance against strong baselines is comparable at best. Furthermore, the standard deviations of the ranks in Table 2 are quite high, therefore making the highest ranks of SAINT statistically insignificant. \n- The efficacy of inter-sample attention: The efficacy of the proposed inter-sample attention is questionable. The authors claim that inter-sample attention is helpful when the number of features is large (i.e. many columns in the table), but this claim is not backed up by data in TAble 5, 6, and 7 in the Appendix. For datasets with more than 100 features, SAINT-i's performance metrics are inconsistent. Furthermore, the interpretation of the inter-sample attentions in Section 5.2 and Section G do not fully shed light on why such behavior is shown by SAINT, but simply states guesswork.\n- Missing baselines: Relevant methods are missing in the baseline such as VIME [1], TABBIE[2]. Is there any reason why they are not part of the baselines?\n\n[1] Yoon, J., Zhang, Y., Jordon, J. and van der Schaar, M., 2020. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33.\n[2] Iida, H., Thai, D., Manjunatha, V. and Iyyer, M., 2021. TABBIE: Pretrained Representations of Tabular Data. arXiv preprint arXiv:2105.02584.\n", "summary_of_the_paper": "This paper proposes SAINT, a neural network model for handling tabular data with both continuous and discrete values. SAINT uses both self-attention among variables and inter-sample attention among different samples. Using both InfoNCE and denoising objectives for pre-training strategies, SAINT was able to outrank all baseline methods in experiments with 30 different tabular datasets, consisting of binary-classification, multi-class classification, and regression.", "summary_of_the_review": "The proposed model, SAINT, seems like a well-motivated model, but the presentation of the experiment results and related analysis is either misleading or incomplete, therefore making it difficult to accurately evaluate the work in its current form. Reading the supplementary material further decreases the reviewer's confidence in this work's claimed contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U93phjQuKXT", "reviewer": "Reviewer_WEK7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents interesting techniques and addresses an important problem. The main concern is regarding the validity of the main claim, regarding outperforming existing classical algorithms. According to Appendix D, the hyperparameter search for these algorithms included a small set of arbitrarily selected hyperparameters, which might not reflect the actual performance they could provide. For example, the XGBoost search is only over two possible values of lambda and a small range of learning rate values, without changing the number of trees, the max-depth or any of the other hyper-parameters. LightGBM also has just two different configurations checked for num_leaves and min_data_in_leaf, with a small learning rate search range (0.01,0.1). In CatBoost, the search is only on the learning rate in that range, without changing any other hyperparameter. The CatBoost result for dataset 1596, for example, which is about 10% below LightGBM, may reflect missing tuning. \nOn the other hand, the proposed SAINT model has a carefully architectured network with various settings of hyperparameters. The paper says that hyperparameter search was done for each dataset on a validation set, but the specifics of this search per dataset are not specified for the SAINT model (e.g., how many options were checked).\nRather than comparing SAINT to many models that might not be well-tuned, this model should be compared to at least one or two well-tuned models, e.g., XGBoost and CatBoost. A reasonable tuning would include all the key hyperparameters (e.g., in XGBoost the number of trees ranging between 50 to 500 and their depth ranging between 1 and 20). It should perform the same number of iterations for each model, e.g., 100 or 1000 iterations with a standard Bayesian optimization package. When different models have a different number of hyperparameter tuning iterations, as reported in this paper, the comparison would not be apples-to-apples.  \n\nAnother issue is the presentation of the results in ranks rather than actual values. While this simplifies things, it may be confusing, since in several cases the differences in accuracy were very small (even 0.1% or less, as reported in Appendix E). It would therefore be good to additionally note each model’s average accuracy or AUC for the classification tasks, and some normalized value of the average RMSE.\n\nAn additional issue to check is why the regression task on dataset 422 had RMSE of exactly 0.03 with all the models (Table 7). Also, when using just 50 labels (Table 9), some of them had RMSE < -30 (both the sign and scale may need checking). The negative RMSE in all the entries in Tables 9-11 looks like a typo.\n", "summary_of_the_paper": "The paper presents a new deep-learning network architecture for tabular data classification and regression problems, called SAINT. While deep-learning provided significant improvements in many domains, tabular data problems are still dominated by classical algorithms like XGBoost and Catboost. The new architecture is based on a new attention mechanism, applying attention both between samples (rows) and features (columns). It also embeds continuous features rather than embedding just categorical ones. The new model is studied on 30 diverse datasets (10 for binary classification, 10 for multiclass classification and 10 for regression), and compared to 10 previous types of models, including both classical models and recently proposed deep-learning models. The authors show that the average rank of SAINT across the various datasets is the best compared to these 10 algorithms (average rank of 2.7, vs. average rank of 4.0 for Catboost). Ablation studies are performed to evaluate the impact of each direction of attention and the impact of the embedding, and the explainability is demonstrated on MNIST. Additionally, the paper presents a contrastive learning approach for pre-training on unlabeled data and fine-tuning on a small number of labels, and it show that pre-training improves the rank for these small numbers of labels (on average by 1.4 for 50 labels and by 0.5 for 200 labels, not improving the rank when all labels are available).", "summary_of_the_review": "This paper considers an important problem and presents interesting technical contributions which extend what was done in previous works. However, the experimental evaluation was not done rigorously enough to support the main claim. Different models had a different number of hyperparameter tuning iterations, and the tuned hyperparameters were not necessarily the most important ones. Therefore, based on the current data, the paper does not meet the criteria of ICLR. I would reconsider it if the authors provide the required data along the lines described above (even if the updated results would show a smaller improvement, its significance and soundness could be much higher).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ONiHtKGYgY", "reviewer": "Reviewer_ufhZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe authors address an important problem setting since deep learning has not yet met its expectations on tabular data and the submission is one step further to this goal.\nThe paper is well and clearly written and the different components of the SAINT architecture are described in detail.\nThe importance of each introduced component is verified by several ablations and comparisons with strong baselines for tabular data show that the suggested architecture performs competitively across several datasets.\nThe performance increase of the suggested pre-training method is well-documented in the case of unlabeled data.\n\nConcerns / Questions:\nDatasets: The authors state that the datasets were selected based on their usage in TabNet and TabTransformer, and their availability in OpenML. Why was the UCI Repo here excluded? It is well-known and datasets are easily accessible such that datasets like “dota2games” or “htru2” could have been used from TabTransformer. While it is understandable that due to the separation in binary/multiclass classification and regression tasks not all datasets from TabNet and TabTransformer could be used it seems a lot of them have been replaced by other datasets from OpenML. I would kindly ask the authors to explain why this is the case and to communicate the underlying substitution rule. For example “poker hand”, “Higgs”, “Mushroom”, “KDD Churn”, “KDD Upselling”, “KDD Census Income” from TabNet and “albert”, “1995_income”, “bank-marketing”, “insurance_co (coil2000)”, “jannis”, “jasmine”, “online shoppers”, “philippine”, “seismic bumps”, “sylvine” from TabTransformer could have been used since they are in OpenML. Please also state the rule by which additional datasets for the respective task were selected from the vast number of datasets in OpenML.\n\nSelf-supervised Pre-Training: While the authors experimentally confirm the advantage of their pre-training method one could still raise the question whether it is really necessary. One could argue that by the similarity-based approach in intersample attention, SAINT could use the feature information of the unlabeled data during training (where a loss is only backpropagated on data for which labels exist) without the need for pre-training. It would have been interesting to see investigations in this direction. A similar approach was followed in [1].\nIn the submission, it is stated that pre-training is done on a plethora of datasets. On how many and which datasets was this done and what is the additional compute time needed for pre-training? Why are the results in the supervised setting (Table 2, last 3 lines, last column) different to the semi-supervised setting without missing labels without pre-training (Table 3, “middle” 3 rows, last column)?\nOne can see that the results for some datasets are (nearly) identical between different methods or are particularly bad (40685, 41169, 422, 42563, 42571, 42724 for supervised) Was it double-checked that there do not exist issues with the datasets themselves and what do the negative error values in the regression task in the semi-supervised setting mean (Tables 9-11)? \n\nHyperparameter-Settings: What are the exact hyperparameter search runs for each deep learning method (including MLP), how much effort (number of runs) was put into the hyperparameter selection for each? Were some of the ablations already part of the selection process?\n\nComputational Cost: Table 1 shows the computational cost for the different SAINT architectures. What are the values for the other deep learning architectures and how do the runtime estimates compare to the classical machine learning methods?\n[1] Kossen et al. Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning. ArXiv, 2106.02584, 2021\n", "summary_of_the_paper": "The paper provides (main contributions) a new deep learning architecture (SAINT) for tabular data that performs attention over both samples and features. For datasets with missing labels, the authors also analyse a  new contrastive self-supervised pre-training approach. The paper also introduces a new embedding method for continuous attributes.\n", "summary_of_the_review": "Overall, the paper contributes important ideas where the core ideas are well-supported by experiments and the strengths outweigh my concerns. So I vote for accepting the submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a deep-learning network architecture for (semi)-supervised tabular data classification and regression problems based on a new attention mechanism between samples (rows) and features (columns).
The model is compared to 10 sota methods, studied on 30 diverse datasets (10 for binary classification, 10 for multiclass classification and 10 for regression).
contrastive learning approach for pre-training on unlabeled data and fine-tuning on a small number of labels
Explainability capabilities are not presented in a very convincing way. 
While the reviewers find the problem relevant, they criticise novelty and, in particular, the experimental comparison.
Concerns about hyperparameter tuning of own vs. comparison methods voiced by the reviewers.
While these concerns have partially been addressed in the author response, the reviewers still doubt the fairness of comparison.
Decision: Reject