Year: 2021

Abstract:
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even performs competitively with gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over 30 benchmark datasets in regression, binary classification, and multi-class classification tasks.

heading: 1 INTRODUCTION
text: While machine learning for image and language processing has seen major advances over the last decade, many critical industries, including financial services, health care, and logistics, rely heavily on data in structured format. Tabular data is unique in several ways that have prevented it from benefiting from the success of deep learning in vision and language. First, tabular data often contain heterogeneous features that represent a mixture of continuous, categorical, and ordinal values, and these values can be independent or correlated. Second, there is no inherent positional information in tabular data, meaning that the order of columns is arbitrary. This differs from text, where tokens are always discrete, and ordering impacts semantic meaning. It also differs from images, where pixels are typically continuous, and nearby pixels are correlated. Tabular models must handle features from multiple discrete and continuous distributions, and they must discover correlations without relying on the positional information. Deep learning systems with specialized architectures that embrace these differences have the potential to improve performance beyond what is achieved by classical methods, like linear classifiers and random forests. Furthermore, without performant deep learning models for tabular data, we lack the ability to exploit compositionality, end-to-end multitask models, fusion with multiple modalities (e.g. image and text), and representation learning.
We introduce SAINT, the Self-Attention and INtersample attention Transformer, a specialized architecture for tabular data. SAINT leverages several mechanisms to overcome the difficulties of training on tabular data. SAINT projects all features – categorical and continuous – independently into a dense vector space. These projected values are passed as token embeddings into a transformer encoder which uses attention in the following two ways. First, “self-attention” attends to individual features within each data sample. Second, we propose a novel “intersample attention”, which enhances the classification of a row (i.e., a data sample) by relating it to other rows in the table. Intersample attention is akin to a nearest-neighbor classification, where the distance metric is learned end-to-end rather than fixed. In addition to this hybrid attention mechanism, we also leverage selfsupervised contrastive pre-training to boost performance for semi-supervised problems.
We provide comparisons of SAINT to a wide variety of deep tabular architectures and commonly used tree-based methods using a diverse battery of tabular datasets. We observe that SAINT, on av-
erage, outperforms all other deep learning methods on supervised and semi-supervised tasks. More importantly, SAINT often out-performs boosted trees (including XGBoost (Chen & Guestrin, 2016), CatBoost (Dorogush et al., 2018), and LightGBM (Ke et al., 2017)), which have long been an industry favorite for complex tabular datasets. Finally, we visualize the attention matrices produced by our models to gain insights into how they behave.

heading: 2 RELATED WORK
text: Classical Models. The most widely adopted approaches for supervised and semi-supervised learning on tabular datasets eschew neural models due to their black-box nature and high compute requirements. When one has reasonable expectations of linear relationships, many modeling approaches are available (Wright, 1995; Weisberg, 2005; Starkweather & Moske, 2011; McCulloch & Neuhaus, 2005). In more complex settings, non-parametric tree-based models are used. Commonly used tools such as XGBoost (Chen & Guestrin, 2016), CatBoost (Dorogush et al., 2018), and LightGBM (Ke et al., 2017) provide several benefits such as interpretability, the ability to handle a variety of feature types including null values, as well as performance in both high and low data regimes.
Deep Tabular Models. While classical methods are still the industry favorite, some recent work brings deep learning to the tabular domain. For example, TabNet (Arik & Pfister, 2019) uses neural networks to mimic decision trees by placing importance on only a few features at each layer. The attention layers in TabNet do not use the regular dot-product self-attention common in transformerbased models, but a sparse layer that allows only certain features to pass through. Yoon et al. (2020) propose VIME, which employs MLPs in a technique for pre-training based on denoising. TABERT (Yin et al., 2020), a more elaborate neural approach inspired by the large language transformer model BERT (Devlin et al., 2018), is trained on semi-structured test data to perform language-specific tasks. Several other studies utilize tabular data, but their problem settings are outside of our scope (Pathak et al., 2016; Chen et al., 2019; Padhi et al., 2021; Shavitt & Segal, 2018; Katzir et al., 2020).
Transformer models for more general tabular data include TabTransformer (Huang et al., 2020), which uses a transformer encoder to learn contextual embeddings only on categorical features. The continuous features are concatenated to the embedded features and fed to an MLP. The main issue with this model is that continuous data do not go through the self-attention block. That means any information about correlations between categorical and continuous features is lost. In our model, we address the issue by projecting continuous and categorical features to the higher dimensional embedding space and passing them both through the transformer blocks. In addition, we propose a new type of attention to explicitly allow data points to attend to each other to get better representations.
Axial Attention. Ho et al. (2019) propose row and column attention in the context of localized attention in 2-dimensional inputs (like images) in the Axial Transformer. For a given pixel, the attention is computed only on the pixels that are on the same row and column, rather than using all pixels. The MSA Transformer (Rao et al., 2021) extends this work to protein sequences and applies both column and row attention across similar rows (tied row attention). TABBIE (Iida et al., 2021) is an adaptation that applies self-attention to rows and columns separately, then averages the representations and passes them as input to the next layer. In all these works, different features from the same data point communicate with each other and with the same feature from a whole batch of data. Our approach, intersample attention, is hierarchical in nature; first, features of a data point interact with each other, then data points interact with each other using entire rows/samples.
In a similar vein, Graph Attention Networks (GAT) (Veličković et al., 2017) compute attention over neighbors on a graph, thereby learning which neighbor’s information is relevant to a node’s prediction. One way to view our intersample attention is as a GAT on a complete graph where all rows are connected to all other rows. Yang et al. (2016) explore hierarchical attention for document classification where attention is computed between words in a given sentence and then between the sentences, but they did not attempt to compute the attention between entire documents themselves.
Self-Supervised Learning. Self-supervision via a ‘pretext task’ on unlabeled data coupled with finetuning on labeled data is widely used for improving language and computer vision models. Some of the pretext tasks previously used include imputing masked values (Pathak et al., 2016; Arik & Pfister, 2019; Huang et al., 2020), denoising (Vincent et al., 2008; Yoon et al., 2020), and replaced token detection (Huang et al., 2020; Iida et al., 2021). Inspired by the success of contrastive learning
for images (Chen et al., 2020; He et al., 2020; Grill et al., 2020), we propose a variant of contrastive pre-training for tabular data. To the best of our knowledge, this is the first work to adopt contrastive learning for tabular data. We couple it with denoising to pre-train on a plethora of datasets with varied volumes of labeled data, and we show that our method outperforms boosting methods.

heading: 3 SELF-ATTENTION AND INTERSAMPLE ATTENTION TRANSFORMER
text: (SAINT)
In this section, we introduce our model, Self-Attention and INtersample attention Transformer (SAINT) and explain its components. Suppose D = {xi, yi}mi=1 is a tabular dataset with m points, xi is an n-dimensional feature vector, and yi is a label or target. Similar to BERT (Devlin et al., 2018), we append a [CLS] token with a learned embedding to each sample. Let xi = [[CLS], f {1} i , f {2} i , .., f {n} i ] be a single data-point with categorical or continuous features f {j} i , and let E be the embedding layer that embeds each feature into Rd. Note that E may use different embedding functions for different features. For a given xi ∈ R(n+1), we get E(xi) ∈ R(n+1)×d. Encoding the Data. In language models, all tokens are embedded using the same procedure. However, in the tabular domain, different features can come from distinct distributions, necessitating a heterogeneous embedding approach. Note that tabular data can contain multiple categorical features which may use the same set of tokens. Unless it is known that common tokens possess identical relationships within multiple columns, it is important to embed these columns independently. Unlike the embedding of TabTransformer (Huang et al., 2020), which uses attention to embed only categorical features, we propose also projecting continuous features into a d−dimensional space before passing their embedding through the transformer encoder. To this end, we use a separate single fully-connected layer with a ReLU nonlinearity for each continuous feature, thus projecting the 1−dimensional input into d−dimensional space. With this simple trick alone, we significantly improve the performance of the TabTransformer model as discussed in Section 5.1. An additional discussion concerning positional encodings can be found in Appendix C.

heading: 3.1 ARCHITECTURE
text: SAINT is inspired by the transformer encoder of Vaswani et al. (2017), designed for natural language, where the model takes in a sequence of feature embeddings and outputs contextual representations of the same dimension. A graphical overview of SAINT is presented in Figure 1(a).
SAINT is composed of a stack of L identical stages. Each stage consists of one self-attention transformer block and one intersample attention transformer block. The self-attention transformer block is identical to the encoder from Vaswani et al. (2017). It has a multi-head self-attention layer (MSA) (with h heads), followed by two fully-connected feed-forward (FF) layers with a GELU non-linearity (Hendrycks & Gimpel, 2016). Each layer has a skip connection (He et al., 2016) and layer normalization (LN) (Ba et al., 2016). The intersample attention transformer block is similar to the self-attention transformer block, except the self-attention layer is replaced by an intersample attention layer (MISA). The details of intersample attention layers are presented below.
The SAINT pipeline, with a single stage (L = 1) and a batch of b inputs, is described by the following equations. We denote multi-head self-attention by MSA, multi-head intersample attention by MISA, feed-forward layers by FF, and layer norm by LN:
z (1) i = LN(MSA(E(xi))) +E(xi) z (2) i = LN(FF1(z (1) i )) + z (1) i (1)
z (3) i = LN(MISA({z (2) i } b i=1)) + z (2) i ri = LN(FF2(z (3) i )) + z (3) i (2)
where ri is SAINT’s contextual representation output corresponding to data point xi. This contextual embedding can be used in downstream tasks such as self-supervision or classification.

heading: 3.2 INTERSAMPLE ATTENTION
text: We introduce intersample attention (a type of row attention) where attention is computed across different data points (rows of a tabular matrix) in a given batch rather than just features of a single data point. Specifically, we concatenate the embeddings of each feature for a single point, then compute attention over samples (rather than features). This enables us to improve the representation of a point by inspecting others. When a feature is missing or noisy in one row, intersample attention enables SAINT to borrow the corresponding features from other similar data samples in the batch.
An illustration of how intersample attention is performed in a single head is shown in Figure 2 and the pseudo-code is presented in Algorithm 1. Unlike the row attention used in Ho et al. (2019); Child et al. (2019); Rao et al. (2021); Iida et al. (2021), intersample attention allows all features from different samples to communicate with each other. In our experiments, we show that this ability boosts performance appreciably. In the multi-head case, instead of projecting q, k, v to a given dimension d, we project them to d/h where h is the number of heads. Then we concatenate all the updated value vectors, vi, to get back a vector of length d.
Algorithm 1 PyTorch-style pseudo-code for intersample attention. For simplicity, we describe just one head and assume the value vector dimension is same as the input embedding dimension.
# b: batch size, n: number of features, d: embedding dimension # W_q, W_k, W_v are weight matrices of dimension dxd # mm: matrix-matrix multiplication def self_attention(x):
# x is bxnxd q, k, v = mm(W_q,x), mm(W_k,x), mm(W_v,x) #q,k,v are bxnxd attn = softmax(mm(q,np.transpose(k, (0, 2, 1)))/sqrt(d)) # bxnxn out = mm(attn, v) #out is bxnxd return out
def intersample_attention(x): # x is bxnxd b,n,d = x.shape # as mentioned above x = reshape(x, (1,b,n*d)) # reshape x to 1xbx(n*d) x = self_attention(x) # the output x is 1xbx(n*d) out = reshape(x,(b,n,d)) # out is bxnxd return out

heading: 4 PRE-TRAINING & FINETUNING
text: Contrastive learning, in which models are pre-trained to be invariant to reordering, cropping, or other label-preserving “views” of the data (Chen et al., 2020; He et al., 2020; Pathak et al., 2016; Grill et al., 2020; Vincent et al., 2008), is a powerful tool in vision domains that has never (to our knowledge) been applied to tabular data. We present a contrastive pipeline for tabular data, a visual description of which is shown in Figure 1. Existing self-supervised objectives for tabular data include denoising (Vincent et al., 2008), a variant of which is used by VIME (Yoon et al., 2020), masking, and replaced token detection as used by TabTransformer (Huang et al., 2020). We find that, while these methods are effective, superior performance can be achieved by contrastive learning.
Generating augmentations. Contrastive methods in vision craft different “views” of images via crops and flips. For tabular data, crafting such transformations is a non-trivial problem. VIME (Yoon et al., 2020) uses mixup in the non-embedded space as data augmentation, but this is limited to continuous data. We instead propose to use CutMix (Yun et al., 2019) to augment samples in input space and mixup (Zhang et al., 2017) in embedding space. These augmentations yield a challenging and effective self-supervision task. Assume that l of m data points are labeled. We denote the embedding layer by E, the SAINT network by S, and 2 projection heads as g1(·) and g2(·). The CutMix augmentation probability is denoted pcutmix and the mixup parameter is α. Given point xi, the original embedding is pi = E(xi), while the augmented representation is generated as follows:
x′i = xi m+ xa (1−m) CutMix in raw data space (3) p′i = α ∗E(x′i) + (1− α) ∗E(x′b) mixup in embedding space (4)
where xa, xb are random samples from the current batch, x′b is the CutMix version of xb, m is the binary mask vector sampled from a Bernoulli distribution with probability pcutmix, and α is the mixup parameter. We first obtain a CutMix version of every data point in a batch by randomly selecting a partner to mix with. We then embed the samples and choose new partners before performing mixup.
SAINT and projection heads. Now that we have both the clean pi and mixed p′i embeddings, we pass them through SAINT, then through two projection heads, each consisting of an MLP with one hidden layer and a ReLU. The use of a projection head before computing contrastive loss is common in vision (Chen et al., 2020; He et al., 2020; Grill et al., 2020) and indeed also improves results on tabular data. Ablation studies and further discussion are available in Appendix F.
Loss functions. We consider two losses for the pre-training phase. (i) The first is a contrastive loss that pushes the latent representations of two views of the same data point (zi and z′i) close together and encourages different points (zi and zj , i 6= j) to lie far apart. For this, we borrow the InfoNCE loss from metric-learning works (Sohn, 2016; Oord et al., 2018; Chen et al., 2020; Wu et al., 2018); (ii) The second loss comes from a denoising task. For denoising, we try to predict the original data sample from a noisy view. Formally, we are given r′i and we reconstruct the inputs as x ′′ i to minimize
the difference between the original and the reconstruction.The combined pre-training loss is:
Lpre-training = − m∑ i=1 log exp(zi · z′i/τ)∑m
k=1 exp(zi · z′k/τ)︸ ︷︷ ︸ Contrastive Loss
+λpt m∑ i=1 n∑ j=1
[Lj(MLPj(r′i),xi)]︸ ︷︷ ︸ Denoising Loss
(5)
where ri = S(pi), r′i = S(p ′ i), zi = g1(ri), z ′ i = g2(r ′ i). Lj is cross-entropy loss or mean squared error depending on the jth feature being categorical or continuous. Each MLPj is a single hidden layer perceptron with a ReLU non-linearity. There are n in number, one for each input feature. λpt is a hyper-parameter and τ is temperature parameter and both of these are tuned using validation data.
Finetuning. Once SAINT is pre-trained on all unlabeled data, we finetune the model on the target prediction task using the l labeled samples. The pipeline of this step is shown in Figure 1(b). For a given point xi, we learn the contextual embedding ri. For the final prediction step, we pass the embedding corresponding only to the [CLS] token through a simple MLP with a single hidden layer with ReLU activation to get the final output. We evaluate cross-entropy loss on the outputs for classification tasks and mean squared error for regression tasks.

heading: 5 EXPERIMENTAL EVALUATION
text: We now discuss SAINT variants and evaluate them in both supervised and semi-supervised scenarios on 30 datasets. We also analyze each component of SAINT and perform ablation studies to understand the importance of each component. Finally, we probe the behavior of attention maps in SAINT by treating the MNIST dataset (LeCun et al., 1998) as tabular data and generating visualizations.
Datasets. We evaluate SAINT on 30 benchmark datasets, with 10 for each of binary classification, multiclass classification, and regression. We chose datasets based on (i) usage in previous tabular work (Arik & Pfister, 2019; Huang et al., 2020) and (ii) availability on the OpenML (Vanschoren et al., 2013) platform where the datasets are uniformly processed and accessible.1 The datasets are diverse, including from
452 up to 581,012 samples, with 2 to 100 class labels, and containing from 6 to 1,777 features – both categorical and continuous features. Some datasets are missing data while others are complete, and some are balanced, while others have highly skewed class distributions. dataset IDs and additional details can be found in Appendix B. We pre-process each dataset by Z-normalizing all continuous features and by label-encoding all categorical features before data is passed to the embedding layer.
Model variants. The SAINT architecture discussed in the previous section has one self-attention transformer encoder block stacked with one intersample attention transformer encoder block in each stage. We also consider variants with only one of these blocks. SAINT-s has only self-attention, while SAINT-i has only intersample attention. See Table 1 for an architectural comparison.
Baselines. We compare our model to traditional methods like Random Forests (Breiman, 2001), Extra Trees (Geurts et al., 2006), and k-NN (Altman, 1992) as well as against powerful boosting libraries XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017), and CatBoost (Dorogush et al., 2018). We also compare to neural networks, like 2-layer multi-layer perceptrons, TabNet (Arik & Pfister, 2019), and TabTransformer (Huang et al., 2020). For methods that use unsupervised pretraining as preprocessing, we use Masked Language Modeling (MLM) (Devlin et al., 2018) for TabNet and Replaced Token Detection (RTD) (Clark et al., 2020) for TabTransformer as in the respective papers. We split the data into 65%, 15%, and 25% for training, validation, and testing, respectively. Hyperparameter tuning is done on validation data and results are reported on test data. Please refer to Appendix D for training details and hyperparameter search spaces for each method.
1Datasets available via https://www.openml.org/d/<dataset_id> for each dataset id.
Metrics. We used different metrics for each of the three tasks we examine in this paper. For binaryclassification, we use area under ROC curve (AUROC), accuracy for multi-class classification, and root-mean-squared error (RMSE) for regression.
Training. We train (including pre-training runs) using AdamW (Loshchilov & Hutter, 2017) with β1 = 0.9, β2 = 0.999, decay = 0.01, and with a learning rate of 0.0001 with batches of size 256 (except for datasets with more than 100 features where we vary the batch size depending on the number of features to fit the batch into a single GPU memory). We split the data into 65%, 15%, and 25% for training, validation, and testing, respectively. We use only validation split for selecting all other hyper-parameters. The final configurations for each of the datasets is presented in Appendix C. We use CutMix mask parameter pcutmix = 0.3 and mixup parameter α = 0.2 for all standard pretraining experiments. We use pre-training loss hyper-parameters λpt = 10 and temperature τ = 0.7 for all settings.

heading: 5.1 RESULTS
text: Supervised setting. In Table 2, we report average ranks of each model on 10 datasets from each binary classification, multi-class classification, and regression task. Note that ranks are computed dataset-wise on a metric. Lower rank represents better performance. In binary and multi-class classification, we measure AuROC and accuracy, and SAINT ranks 2.7 and 2.5 respectively (across 10 datasets for each setting) compared to CatBoost’s 2.9 and LightGBM’s 3.2 (the next best method). For regression, we measure RMSE. SAINT ranks 2.9 across 10 datasets compared to the next best method, CatBoost, which ranks 3.9. For complete results on all 30 datasets, see Appendix E.
Semi-supervised setting. We perform 2 sets of experiments with 50, and 200 labeled data points (the rest are unlabeled). See Table 3 for average ranks across all 30 datasets and standard errors in both settings. We also examine the case when all data is labeled, to see if pre-training still helps. In all 3 cases, the pre-trained SAINT model performs the best. Pre-training had the largest positive impact on the SAINT-s variant with a rank improvement in all cases. Interestingly, we note that when all the training data samples are labeled, pre-training does not contribute appreciably (except for SAINT-s variant), hence the results with and without pre-training are fairly close.
Effect of embedding continuous features. To understand the effect of learning embeddings for continuous data, we perform a simple experiment with TabTransformer. We modify TabTransformer by embedding continuous features into d dimensions using a single-layer ReLU MLP, just as they use on categorical features, and we pass the embedded features through the transformer block. We keep the entire architecture and all training hyper-parameters the same for both TabTransformer and its modified version. In binary classification, the average AUROC across 10 datasets increases from 0.82 to 0.85 (an increment of 3.65%), and in multiclass classification, the average accuracy across 10 datasets increases from 0.72 to 0.74 (an increment of 2.77%). In regression, the average RMSE over 10 datasets decreases from 5,071.30 to 4,998.2 (a decrement of 1.44%). This experiment shows that embedding the continuous data is important and can boost performance significantly.
When to use intersample attention? In our experiments, we observe that SAINT-i outperforms other variants whenever the number of features is large (> 500) as observed in the cases of MNIST and Bioresponse. Another advantage of SAINT-i is that execution is fast compared to SAINT-s, despite the fact that the number of parameters in SAINT-i is much higher than that of SAINT-s (see Table 1). This can be attributed to quadratic complexity in terms of number of features to compute attention in SAINT-s while in SAINT-i, this depends on the size of the batch which we can control.
How robust is SAINT to data corruptions? We evaluate the robustness of SAINT variants by corrupting the training data via noise and missing values. To simulate noise, we apply CutMix, replacing 10% to 90% of features with values of other randomly selected samples. We examine the resulting trends in binary classification and regression. In trials with noise, the drop in mean AUROC or mean scaled-RMSE is minimal until 70% of data is corrupted, when the performance drops significantly. SAINT and SAINT-i are comparatively more robust than SAINT-s. This shows that using row attention improves the model’s robustness to noisy training data. However, we find the opposite trend when many features are missing in
the training data. In this scenario, SAINT-s and SAINT are quite robust, and the drop in AUROC is not drastic until 70% or more of the data is missing. We conclude that SAINT is reliable for training on corrupted training data. Trend line plots for these scenarios are available in Appendix F, Figure 6.
Effect of batch size on intersample attention performance. As we discuss in Section 3.2, intersample attention is computed between batches of data points. We consider multiple multi-class classification datasets and examine the impact of batch size by varying it from 8 to 1,024. In most cases, we find that the variation in SAINT-i’s performance is small and is comparable to that of SAINT-s, which has no intersample attention component and thus has no dependence on batch size. We note that when the number of training samples is very small (i.e. less than 1,000), the variation in performance increases, but we observe the same trend in SAINT-s as well. We present corresponding plots in Appendix F, Figure 7. Further discussion and analysis is presented in Appendix F.

heading: 5.2 INTERPRETING ATTENTION
text: One advantage of transformers is that attention lends itself to interpretability tools as opposed to MLPs, which are hard to interpret. In particular, when we use only one transformer stage, the attention maps reveal which features and which data points are being used by the model to make decisions. We use MNIST (LeCun et al., 1998) data to examine how self-attention and intersample attention behave in our models. While MNIST is not a typical tabular dataset, it has the advantage that its features can easily be visualized as an image.
Figure 3a depicts the attention on each of the pixels/features in a self-attention layer of SAINT. Without any explicit supervision, the model learns to focus on the foreground pixels, and we clearly see from the attention map which features are most important to the model. The self-attention plots of SAINT-s are similar (Appendix G, Figure 8a).
Figures 3b and 3c depict a similar visualization on a batch of 20 points, two from each class in MNIST. Figure 3b shows intersample attention in SAINT. This plot shows which samples attend to which other samples in the batch. Surprisingly, very few points in a batch receive attention. We hypothesize that the model focuses on a few points that are critical because they are particularly difficult to classify without making direct comparisons to exemplars in the batch. In Figure 3c, we
show the intersample attention plot from a SAINT-i model. The same sparse attention behaviour persists here too, but the points being attended to are different in this model. Interestingly, we find this behavior to be significantly different on the Volkert data (Dataset ID: 41166, multiclass with 10 classes), where a wide range of data becomes the focus of attention depending on the input. The intersample attention layer gets dense with the difficulty (to classify) of the datasets. See Appendix G for additional MNIST and Volkert attention maps.
Figure 4 shows the behavior of attention at the dataset (rather than batch) level. We visualize a t-SNE (Van der Maaten & Hinton, 2008) embedding for value vectors generated in intersample attention layers, and we highlight the points that are most attended to in each batch. In Figure 4 (left), the value vectors and attention are computed on the output representations of a self-attention layer. In contrast, value vectors and attention in Figure 4 (right) are computed on the embedding layer output, since SAINT-i does not use self-attention. In these plots, the classes to which the model attends vary dramatically. Thus, the exact classes to which an attention head attends change with the architecture, but the trend of using a few classes as a ‘pivot’ seems prevalent in intersample attention heads.

heading: 6 CONCLUSION
text: Even though tabular data is an extremely common data format used by institutions in various domains, deep learning methods are still lagging behind tree-based boosting methods in production. In this paper, we highlight the limitations of existing deep learning approaches, and the reasons why they do not perform competitively on the tabular datasets. We further introduce three novel improvements to counter these limitations - (1) intersample attention, (2) contrastive pre-training, (3) and a strategy for embedding continuous columns in the tabular datasets. Our method, SAINT, performs competitively across a large number of tabular datasets with diverse characteristics on the tasks of binary classification, multi-class classification and regression. We quantitatively and qualitatively analyse the importance of each of our improvements.

heading: 7 ETHICS AND REPRODUCIBILITY STATEMENT
text: The tremendous success of deep learning in vision and language domains is yet to be realized for tabular datasets. In our work, we discuss some of the potential limitations of existing deep learning models and how we can mitigate them by deploying not just features but other data points for the inference. We believe that our proposed architecture, along with the self-supervised and data augmentation techniques, will greatly help with representation learning and machine learning applications in tabular data domain.
With that in mind, like most deep learning models, SAINT in its current form is prone to capturing and even amplifying the biases in the dataset on which it is trained on. Our data augmentation strategy might help to mitigate this issue to some extent but by no means is it intended to resolve the data imbalance. We caution the end users to carefully consider their use cases and use our method only after appropriate validation and testing.
Tabular datasets often contain demographics or personally identifiable information (PII). Leaking of information from training data with large deep learning models is a topic of great concern and has been demonstrated to a certain extent in the language domain. However, we are not aware of such works in the case of tabular data.
Reproducibility. We are fully committed to open sourcing our code for reproducibility, as well as for the broader community to build upon it. Please find the code and instructions to use it attached in the supplementary material.

heading: Appendix for SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

heading: A ADDITIONAL ILLUSTRATIONS

heading: B DATASETS
text: Data sources. As mentioned in main text, all 30 datasets are downloaded from OpenML (Vanschoren et al., 2013) repository. We present the dataset IDs and some statistics of the datasets in Table 4. All datasets can be downloaded via https://www.openml.org/d/<dataset_id>
Data preprocessing. In each dataset, the categorical features are label encoded, and continuous features are z-normalized before passing them into the embedding layer. Each feature (or column) has a different missing value token to account for missing data.
For MNIST, we unravel each image into a vector of 784 features and consider each image as a single row. Since each feature is of same type in this dataset, we encode all the features into the same embedding space. To distinguish the features, we also use positional encodings in the encoding layer.

heading: C COMPLETE TRAINING DETAILS
text: In each of our experiments, we use a single Nvidia GeForce RTX 2080Ti GPU. Individual training runs take between 5 minutes and 6 hours. In total, the experiments in this paper account for around 30 GPU days (including semi-supervised experiments and ablation studies).
For most of the datasets, we use embedding size d = 32. For MNIST, we use d = 12, for datasets with > 100 features we used d = 4. The variance in the embedding size is only due to the memory constraints of a single GPU. We used L = 6 layers in the SAINT-s variant and L = 1 for SAINTi and SAINT variants. We use dropout of 0.1 in all attention layers. In feed-forward layers, use dropout of 0.1 in the SAINT-s variant, and we use 0.8 in SAINT-i and SAINT models. We use attention heads h = 8 for SAINT-s model and h = 4 for SAINT-i and SAINT models. Inside the self-attention layer, the q, k, and v vectors are of dimension 16, and in the intersample attention layer, they are of size 64.
Other minor details are shared in the code.
Feature Encoding. Transformers for vision and language typically employ same embedding function for each feature, or a column since each word, or pixel, or a patch comes from a similar distribution. This is not the case with all of the 30 datasets presented in this paper; each feature may be of a different type and drawn from different distribution. Hence we use a unique embedding function for each column.
Positional Encoding. In tabular datasets, different features, or columns do not have any order. Different feature encodings used to embed different columns is sufficient to provide the model the necessary information about the feature. Hence we skip the positional encodings in all the datasets (except in the case of MNIST, where all columns share the embedding function, and we provide positional encodings instead). Note that the attention operation itself is order invariant in both the self-attention and intersample attention cases.

heading: D BASELINES
text: We use Auto-Gluon2 (Erickson et al., 2020) framework to run the train the traditional models as well as to perform the hyperparameter sweep. For TabNet, we use the famous PyTorch implementation3, and for TabTransformer since the official implementation is not available, we used the widely
2https://auto.gluon.ai/stable/index.html 3https://github.com/dreamquark-ai/tabnet
used PyTorch re-implementation4. In the following paragraphs we will give details on hyperparameter configurations for each baseline. All the baseline models use early stopping on the validation dataset’s performance. All the deep learning methods take embedding for categorical data as input and we tuned them with variable embedding sizes from [16, 32, 64].
Random Forest & Extra trees Classifiers. We tuned on number of estimators (n estimators) from the list [100, 200], on criterion from [‘gini’,‘entropy’].
Random Forest & Extra trees Regressors. We tuned on number of estimators (n estimators) from the list [100, 200], on criterion from [‘squared error’,‘absolute error’].
KNeighbors models. We tuned the weights parameter from [‘uniform’,‘distance’], and p value from [1,2]. For large datasets, k-NN model is computationally intractable and hence for datasets with > 50000 datapoints, we randomly choose 50k points and train the model on the smaller dataset.
LightGBM. We tried with different values of min data in leaf in [10, 20], num leaves in [30, 50] and learning rate in the interval [0.01, 0.1]
XGBoost. We used the values of booster as ’gbtree’, n estimators of 200 and tuned over L2 regularization lambda in [1, 10] and the learning rate in the interval [0.01, 0.3].
CatBoost. We used the values of reg lambda as 100 and early stopping rounds as 500 and tuned the learning rate in the interval [0.01, 0.1].
MLP. We used a 2 hidden-layered neural network of sizes [200, 100], with 1d BatchNorm optimized on Adam (Kingma & Ba, 2014) with learning rate set to 0.001.
TabNet. We have used the hyperparameters suggested by the PyTorch implementation of Tabnet. We used Adam optimizer with learning rate set to 0.02 and momentum set to 0.3. We used learning rate scheduler with gamma as 0.95 and step-size as 20. The rest of the parameters we have tuned based on the ranges provided in the original paper.
TabTransformer. We used AdamW optimizer with learning rate set to 0.001. We also tried different levels of feed-forward drop-outs from [0.1, 0.5, 0.8].

heading: E EXTENDED RESULTS
text: In the main part of the paper, we have presented the average ranks of the models across the benchmark datasets. The supervised results are shared across the following three tables. In Table 5, we share the AuROC scores of all the baselines and SAINT variants for the binary classification task. In Table 6, we share accuracies for multiclass classification. In both these cases, higher the score, better the model. While in case of regression, we share the RMSE scores in Table 10, smaller the value, better the model.
In case of semi-supervised/ pre-training experiments, see the exhaustive results for the case of 50 labeled samples in Table 11, for 200 samples in Table 12, and all samples labeled in Table 13. Please note that in these 3 tables, we have scores from different tasks together in the same table.

heading: F ADDITIONAL ANALYSES
text: How robust is SAINT to data corruptions? (contd.) As discussed in the main body, we evaluate the robustness of SAINT variants in cases of data corruption or missing data. We examine the cases of Binary classification and Regression and plot the average scores across the 10 datasets correponding to that task. We present the results in Figure 6.
4https://github.com/lucidrains/tab-transformer-PyTorch
Effect of batch size on intersample attention performance. (cont.) As discussed in the main body, we examine the affect of batch size on different SAINT variants in Figure 7. We pick 4 multiclass classification datasets with varying numbers of features and samples. In all cases, we see that the variance in Accuracy is minimal when varying the batch size from 8 to 1024. In all the plots, we see log2 (batchsize) plotted on the X-axis with accuracy on Y-axis .
F.1 PRE-TRAINING ABLATIONS
In Table 9, we study various configurations of pre-training components. We perform 3 primary studies: we vary (1) projection head, (2) pre-training loss, and (3) data augmentation method. Note, the final result in all 3 studies refers to the same experiment (hence the row is repeated), which is the final chosen configuration for our model.
Effect of projection heads. As described in Section 4, we use two different projection heads, g1(·) and g2(·), to project the contextual representations to lower dimensions and then compute contrastive losses. We study three different options for the heads: (1) distinct projection heads (2) heads with weight sharing, and (3) no projection heads at all. Table 9 shows that using distinct projection heads performs best.
Varying pre-training loss. We train SAINT’s variants with different loss functions, as shown in Study 2 of Table 9. We try denoising and contrastive losses, in addition to a cosine similarity loss on positive pairs (inspired by Grill et al. (2020); Chen & He (2020)). The combination of contrastive and denoising consistently yields the best results in all SAINT variants.
Varying the pre-training augmentations. We also try to understand how important it is to use CutMix and mixup to generate augmented embeddings in the pre-training pipeline. We tinker with
various configurations in Study 3 of Table 9, and we observe that using these two augmentations in unison results in the best performance across all SAINT variants.

heading: G ADDITIONAL INTERPRETABILITY PLOTS
text: In Figure 8a, we show a self-attention plot for the SAINT-s variant (with L = 1) on MNIST. The self-attention in one stage SAINT-s model behaves similar to a one stage SAINT model. However, when there are more stages, the attention in the last stage is not quite as interpretible.
In Figure 8b, we show the intersample attention between a batch of points from different classes in SAINT model on the Volkert dataset (OpenML dataset ID: 41166). Similarly in Figure 8c, we show intersample attention in the SAINT-i variant on the same batch of points from the Volkert dataset.
As mentioned in the main body, the intersample behaviour is not quite as sparse as that of MNIST. We hypothesize that the sparsity of the intersample attention layer depends on how separable the classes in the dataset are. (Volkert is a harder dataset than MNIST).
In Figure 9, we show the t-SNE plots on value vectors for SAINT and SAINT-i variants on Volkert. Unlike MNIST, all the classes are attended to equally.
Ta bl
e 11
:S em
isu
pe rv
is ed
le ar
ni ng
w ith
50 la
be le
d sa
m pl
es ca
se :A
ct ua
ls co
re s
fo re
ac h
of th
e m
od el
fo ra
gi ve
n da
ta se
t. Fi
rs t1
0 da
ta se
ts co
rr es
po nd
to m
ul tic
la ss
cl as
si fic
at io
n, he
nc e
th e
sc or
es ar
e ac
cu ra
ci es
,t he
ne xt
10 be
lo ng
to bi
na ry
cl as
si fic
at io
n, so
th e
sc or
es ar
e A
uR O
C s,
an d
th e
la st
10 co
lu m
ns ar
e re
gr es
si on
ty pe , he nc e th ey ca rr y R M SE va lu es .R em ov ed th e ne ga tiv e si gn in R M SE sc or es
Ta sk
/5 0
M ul
tic la
ss C
la ss
ifi ca
tio n
B in
ar y
C la
ss ifi
ca tio
n R
eg re
ss io n M od el /d at as et 18 8 15 96 45 41 40 66 4 40 68 5 40 68 7 40 97 5 41 16 6 41 16 9 42 73 4 31 44 10 17 11 11 14 87 14 94 15 90 41 34 42 17 8 42 73 3 42 2 54 1 42 56 3 42 57 1 42 70 5 42 72
4 42
72 6
42 72
7 42
72 8
42 72 9 R an do m Fo re st 0. 45 0. 45 0. 48 0. 76 1. 00 0. 74 0. 79 0. 42 0. 08 0. 57 0. 78 0. 88 0. 60 0. 54 0. 53 0. 83 0. 76 0. 57 0. 81 0. 51 0. 04 24 .9 9 54 35 8. 57 31 26 .3 4 12 .7 2 11 80 6. 27 2. 88 0. 20 30 .2 5 2. 83 E xt ra Tr ee s 0. 37 0. 41 0. 54 0. 69 0. 97 0. 40 0. 69 0. 42 0. 07 0. 71 0. 68 0. 87 0. 46 0. 46 0. 78 0. 84 0. 82 0. 63 0. 79 0. 44 0. 03 29 .6 5 50 78 9. 30 29 43 .0 5 10 .9 0 11 73 5. 62 2. 73 0. 19 29 .8 2 2. 69 K N ei gh bo rs D is t 0. 31 0. 22 0. 48 0. 68 0. 92 0. 57 0. 70 0. 37 0. 07 0. 60 0. 54 0. 70 0. 46 0. 56 0. 55 0. 81 0. 54 0. 61 0. 61 0. 52 0. 03 23 .9 5 64 04 9. 40 31 63 .3 4 11 .4 1 11 78 4. 33 3. 01 0. 19 30 .1 0 2. 80 K N ei gh bo rs U ni f 0. 31 0. 21 0. 48 0. 68 0. 87 0. 48 0. 68 0. 36 0. 07 0. 69 0. 54 0. 70 0. 46 0. 56 0. 54 0. 77 0. 52 0. 61 0. 61 0. 52 0. 03 23 .8 8 65 21 0. 60 31 36 .6 2 11 .4 1 11 77 5. 15 2. 94 0. 19 30 .0 7 2. 81 L ig ht G B M 0. 40 0. 47 0. 49 0. 78 0. 98 0. 67 0. 78 0. 47 0. 08 0. 67 0. 78 0. 95 0. 69 0. 58 0. 77 0. 91 0. 82 0. 63 0. 81 0. 53 0. 03 24 .0 5 48 56 3. 42 26 04 .7 6 11 .1 5 11 67 0. 46 3. 40 0. 19 30 .0 0 2. 54 X G B oo st 0. 42 0. 45 0. 47 0. 79 0. 99 0. 73 0. 78 0. 47 0. 08 0. 66 0. 80 0. 94 0. 69 0. 57 0. 69 0. 90 0. 81 0. 61 0. 80 0. 51 0. 03 24 .4 4 49 67 5. 84 27 04 .1 3 11 .5 4 11 75 1. 20 3. 57 0. 18 30 .0 9 2. 58 C at B oo st 0. 41 0. 44 0. 46 0. 76 1. 00 0. 72 0. 76 0. 48 0. 08 0. 71 0. 78 0. 95 0. 74 0. 54 0. 64 0. 88 0. 79 0. 62 0. 79 0. 51 0. 03 24 .7 0 47 90 9. 49 26 28 .7 2 10 .8 6 11 73 5. 08 2. 59 0. 19 29 .8 1 2. 62 M L P 0. 34 0. 50 0. 35 0. 69 0. 95 0. 65 0. 69 0. 41 0. 07 0. 49 0. 73 0. 92 0. 76 0. 46 0. 74 0. 88 0. 73 0. 63 0. 75 0. 49 33 .1 8 29 .5 9 46 18 0. 42 25 04 .7 9 10 .9 0 37 03 08 30 .0 0 2. 60 0. 21 29 .8 4 2. 67 Ta bn et + M L M 0. 34 0. 51 0. 35 0. 70 0. 95 0. 65 0. 69 0. 41 0. 07 0. 49 0. 74 0. 92 0. 77 0. 47 0. 74 0. 88 0. 73 0. 63 0. 75 0. 49 35 .0 9 30 .6 4 42 84 7. 30 24 57 .1 2 10 .8 4 35 48 86 80 .0 0 2. 63 0. 22 30 .9 5 2. 82 Ta bT ra ns fo rm er + R T D 0. 38 0. 41 0. 54 0. 69 0. 98 0. 41 0. 69 0. 43 0. 07 0. 72 0. 78 0. 89 0. 60 0. 54 0. 53 0. 83 0. 76 0. 57 0. 82 0. 52 0. 03 26 .5 0 49 54 5. 39 47 94 .6 2 13 .5 0 13 18 0. 53 2. 70 0. 22 20 .2 2 3. 13 SA IN Ts 0. 38 0. 62 0. 54 0. 78 0. 90 0. 70 0. 85 0. 41 0. 11 0. 71 0. 70 0. 95 0. 68 0. 52 0. 85 0. 91 0. 85 0. 65 0. 80 0. 55 0. 03 23 .5 1 41 75 3. 81 37 00 .1 8 11 .4 7 11 69 1. 84 2. 39 0. 19 16 .8 9 2. 70 SA IN Ti 0. 48 0. 62 0. 54 0. 80 0. 97 0. 73 0. 77 0. 44 0. 08 0. 71 0. 66 0. 95 0. 74 0. 51 0. 82 0. 89 0. 82 0. 67 0. 80 0. 54 0. 11 25 .3 1 36 43 7. 50 25 43 .7 4 15 .8 3 11 71 0. 47 2. 35 0. 19 16 .8 9 2. 45 SA IN T 0. 39 0. 65 0. 52 0. 79 0. 92 0. 68 0. 78 0. 42 0. 08 0. 71 0. 74 0. 96 0. 69 0. 55 0. 85 0. 90 0. 85 0. 69 0. 77 0. 58 0. 05 22 .2 2 40 75 3. 81 24 00 .8 0 12 .5 8 11 68 1. 22 2. 28 0. 21 16 .9 5 2. 75 SA IN Ts + pr etr ai ni ng 0. 50 0. 56 0. 54 0. 78 0. 96 0. 73 0. 81 0. 39 0. 09 0. 71 0. 72 0. 95 0. 77 0. 55 0. 86 0. 90 0. 86 0. 65 0. 78 0. 54 0. 03 23 .6 8 41 76 3. 81 40 90 .3 8 11 .6 0 11 67 8. 75 2. 29 0. 20 16 .9 6 2. 62 SA IN Ti+ pr etr ai ni ng 0. 39 0. 63 0. 54 0. 78 0. 91 0. 67 0. 71 0. 43 0. 11 0. 71 0. 72 0. 96 0. 73 0. 55 0. 82 0. 86 0. 82 0. 67 0. 77 0. 61 0. 15 22 .9 9 37 90 6. 08 24 07 .7 3 11 .6 0 11 68 0. 45 2. 19 0. 21 16 .8 7 2. 70 SA IN T + pr etr ai ni ng 0. 48 0. 64 0. 54 0. 79 0. 96 0. 63 0. 82 0. 46 0. 10 0. 71 0. 69 0. 96 0. 74 0. 50 0. 85 0. 91 0. 85 0. 69 0. 82 0. 57 0. 04 21 .5 8 39 24 4. 90 25 82 .7 4 12 .3 8 11 63 2. 98 2. 34 0. 18 16 .8 7 2. 58
Ta bl
e 12
:S em
isu
pe rv
is ed
le ar
ni ng
w ith
20 0
la be
le d
sa m
pl es
ca se
:A ct
ua ls
co re
sf or
ea ch
of th
e m
od el
fo ra
gi ve
n da
ta se
t. Fi
rs t1
0 da
ta se
ts co
rr es
po nd
to m
ul tic
la ss
cl as
si fic
at io
n, he
nc e
th e
sc or
es ar
e ac
cu ra
ci es
,t he
ne xt
10 be
lo ng
to bi
na ry
cl as
si fic
at io
n, so
th e
sc or
es ar
e A
uR O
C s,
an d
th e
la st
10 co
lu m
ns ar
e re
gr es
si on
ty pe , he nc e th ey ca rr y R M SE va lu es .R em ov ed th e ne ga tiv e si gn in R M SE sc or es
Ta sk
/2 00
M ul
tic la
ss C
la ss
ifi ca
tio n
B in
ar y
C la
ss ifi
ca tio
n R
eg re
ss io n M od el /d at as et 18 8 15 96 45 41 40 66 4 40 68 5 40 68 7 40 97 5 41 16 6 41 16 9 42 73 4 31 44 10 17 11 11 14 87 14 94 15 90 41 34 42 17 8 42 73 3 42 2 54 1 42 56 3 42 57 1 42 70 5 42 72
4 42
72 6
42 72
7 42
72 8
42 72 9 R an do m Fo re st 0. 50 0. 58 0. 48 0. 88 1. 00 0. 70 0. 90 0. 48 0. 16 0. 68 0. 74 0. 96 0. 80 0. 60 0. 80 0. 73 0. 87 0. 75 0. 81 0. 56 0. 03 17 .1 2 36 36 7. 86 25 23 .5 3 12 .5 7 11 80 3. 10 2. 53 0. 19 29 .9 5 2. 47 E xt ra Tr ee s 0. 52 0. 62 0. 54 0. 86 1. 00 0. 73 0. 85 0. 48 0. 14 0. 71 0. 74 0. 96 0. 79 0. 55 0. 85 0. 86 0. 87 0. 76 0. 83 0. 52 0. 03 24 .7 6 38 66 7. 70 23 82 .5 1 10 .4 3 11 66 1. 06 2. 76 0. 17 29 .6 6 2. 30 K N ei gh bo rs D is t 0. 31 0. 50 0. 47 0. 83 0. 97 0. 68 0. 77 0. 40 0. 14 0. 64 0. 54 0. 76 0. 70 0. 54 0. 64 0. 65 0. 55 0. 70 0. 69 0. 53 0. 03 24 .7 5 55 57 8. 23 30 31 .6 9 11 .8 5 12 10 2. 29 3. 20 0. 20 29 .9 5 2. 68 K N ei gh bo rs U ni f 0. 29 0. 50 0. 44 0. 82 0. 93 0. 65 0. 76 0. 39 0. 14 0. 66 0. 55 0. 76 0. 67 0. 54 0. 63 0. 64 0. 55 0. 69 0. 68 0. 53 0. 03 24 .4 3 56 55 2. 06 30 32 .3 0 11 .8 6 12 11 0. 19 3. 19 0. 20 29 .9 4 2. 69 L ig ht G B M 0. 52 0. 57 0. 51 0. 86 1. 00 0. 67 0. 85 0. 51 0. 16 0. 71 0. 76 0. 97 0. 77 0. 57 0. 87 0. 87 0. 87 0. 77 0. 83 0. 55 0. 03 19 .4 6 36 20 6. 57 23 86 .7 7 10 .7 4 12 44 4. 40 3. 05 0. 17 29 .7 9 2. 29 X G B oo st 0. 56 0. 59 0. 50 0. 83 0. 99 0. 69 0. 83 0. 50 0. 16 0. 71 0. 75 0. 97 0. 81 0. 59 0. 87 0. 91 0. 87 0. 78 0. 81 0. 55 0. 03 20 .1 4 37 27 1. 72 23 32 .0 3 10 .6 1 11 80 3. 80 3. 16 0. 18 29 .6 9 2. 27 C at B oo st 0. 52 0. 61 0. 53 0. 86 1. 00 0. 70 0. 85 0. 52 0. 15 0. 71 0. 75 0. 97 0. 82 0. 58 0. 85 0. 77 0. 88 0. 80 0. 83 0. 54 0. 03 19 .2 0 34 45 9. 98 23 52 .1 0 10 .3 9 11 71 3. 59 2. 85 0. 18 29 .6 6 2. 38 M L P 0. 40 0. 53 0. 35 0. 89 0. 99 0. 66 0. 81 0. 49 0. 14 0. 66 0. 70 0. 94 0. 76 0. 48 0. 81 0. 90 0. 79 0. 75 0. 76 0. 54 0. 09 25 .1 4 40 23 4. 38 24 52 .2 5 10 .8 0 11 85 52 10 0. 00 2. 52 0. 18 29 .7 2 8. 71 Ta bn et + M L M 0. 40 0. 53 0. 35 0. 89 0. 99 0. 67 0. 82 0. 49 0. 14 0. 66 0. 71 0. 95 0. 76 0. 48 0. 82 0. 91 0. 80 0. 76 0. 77 0. 54 0. 10 23 .7 3 44 06 5. 53 26 13 .5 7 9. 85 11 90 84 90 0. 00 2. 37 0. 17 32 .4 9 8. 03 Ta bT ra ns fo rm er + R T D 0. 52 0. 62 0. 54 0. 86 1. 00 0. 73 0. 85 0. 48 0. 14 0. 72 0. 75 0. 97 0. 80 0. 61 0. 80 0. 73 0. 88 0. 76 0. 82 0. 56 0. 03 18 .2 2 45 05 1. 72 46 07 .7 7 12 .1 4 13 99 4. 30 2. 71 0. 20 19 .5 8 3. 02 SA IN Ts 0. 53 0. 66 0. 55 0. 90 0. 99 0. 72 0. 89 0. 44 0. 18 0. 71 0. 76 0. 97 0. 79 0. 60 0. 88 0. 92 0. 88 0. 76 0. 81 0. 58 0. 03 15 .6 4 38 10 7. 56 40 99 .7 2 11 .0 0 11 67 5. 51 2. 31 0. 18 16 .9 0 2. 58 SA IN Ti 0. 64 0. 67 0. 54 0. 90 0. 99 0. 72 0. 87 0. 49 0. 18 0. 71 0. 74 0. 95 0. 78 0. 61 0. 87 0. 91 0. 87 0. 79 0. 80 0. 56 0. 03 18 .2 5 35 33 2. 53 23 19 .9 4 13 .1 7 11 71 6. 38 2. 17 0. 18 16 .8 8 2. 51 SA IN T 0. 59 0. 66 0. 54 0. 86 0. 99 0. 72 0. 90 0. 49 0. 16 0. 71 0. 77 0. 97 0. 72 0. 60 0. 88 0. 91 0. 88 0. 80 0. 82 0. 58 0. 03 16 .6 8 35 70 5. 18 22 91 .8 9 11 .0 0 11 66 7. 23 2. 27 0. 18 16 .9 8 2. 69 SA IN Ts + pr etr ai ni ng 0. 65 0. 66 0. 56 0. 89 0. 99 0. 73 0. 91 0. 44 0. 16 0. 71 0. 76 0. 97 0. 69 0. 63 0. 88 0. 90 0. 88 0. 76 0. 82 0. 58 0. 03 13 .6 6 37 50 7. 56 38 31 .7 1 11 .7 6 11 67 5. 68 2. 21 0. 19 16 .9 2 2. 53 SA IN Ti+ pr etr ai ni ng 0. 64 0. 67 0. 55 0. 87 0. 99 0. 71 0. 85 0. 49 0. 16 0. 71 0. 75 0. 97 0. 75 0. 64 0. 88 0. 90 0. 88 0. 79 0. 81 0. 58 0. 08 18 .3 2 35 29 9. 48 22 59 .7 6 11 .7 6 11 67 2. 01 2. 20 0. 19 16 .9 2 2. 44 SA IN T + pr etr ai ni ng 0. 50 0. 66 0. 55 0. 91 0. 99 0. 68 0. 89 0. 49 0. 17 0. 71 0. 74 0. 96 0. 77 0. 66 0. 89 0. 92 0. 89 0. 80 0. 82 0. 59 0. 03 20 .6 5 37 10 7. 56 22 64 .7 9 11 .6 7 11 66 7. 22 2. 26 0. 17 16 .8 3 2. 62
Ta bl
e 13
: Se
m i-
su pe
rv is
ed le
ar ni
ng w
ith A
L L
sa m
pl es
la be
le d
ca se
: A
ct ua
l sc
or es
fo r
ea ch
of th
e m
od el
fo r
a gi
ve n
da ta
se t.
Fi rs
t 10
da ta
se ts
co rr
es po
nd to
m ul
tic la
ss cl
as si
fic at
io n,
he nc
e th
e sc
or es
ar e
ac cu
ra ci
es ,
th e
ne xt
10 be
lo ng
to bi
na ry
cl as
si fic
at io
n, so
th e
sc or
es ar
e A
uR O
C s,
an d
th e
la st
10 co
lu m
ns ar e re gr es si on ty pe ,h en ce th ey ca rr y R M SE va lu es .R em ov ed th e ne ga tiv e si gn in R M SE sc or es
Ta sk
/A ll
M ul
tic la
ss C
la ss
ifi ca
tio n
B in
ar y
C la
ss ifi
ca tio
n R
eg re
ss io n M od el /d at as et 18 8 15 96 45 41 40 66 4 40 68 5 40 68 7 40 97 5 41 16 6 41 16 9 42 73 4 31 44 10 17 11 11 14 87 14 94 15 90 41 34 42 17 8 42 73 3 42 2 54 1 42 56 3 42 57 1 42 70 5
42 72
4 42
72 6
42 72
7 42
72 8
42 72 9 R an do m Fo re st 0. 65 0. 95 0. 61 0. 95 1. 00 0. 70 0. 97 0. 67 0. 36 0. 74 0. 78 0. 99 0. 80 0. 77 0. 91 0. 93 0. 91 0. 87 0. 84 0. 67 0. 03 17 .8 1 37 08 5. 58 19 99 .4 4 16 .7 3 12 37 5. 31 2. 48 0. 15 13 .7 0 1. 77 E xt ra Tr ee s 0. 65 0. 95 0. 60 0. 95 1. 00 0. 70 0. 96 0. 65 0. 34 0. 74 0. 76 0. 99 0. 81 0. 75 0. 92 0. 93 0. 90 0. 86 0. 83 0. 66 0. 03 19 .2 7 35 04 9. 27 19 61 .9 3 15 .3 5 12 50 5. 09 2. 52 0. 15 13 .5 8 1. 85 K N ei gh bo rs D is t 0. 44 0. 97 0. 49 0. 92 1. 00 0. 72 0. 89 0. 62 0. 20 0. 69 0. 50 0. 87 0. 72 0. 52 0. 74 0. 87 0. 68 0. 81 0. 76 0. 58 0. 03 25 .0 5 46 33 1. 14 26 17 .2 0 14 .5 0 13 04 6. 09 2. 50 0. 17 13 .6 9 2. 10 K N ei gh bo rs U ni f 0. 42 0. 96 0. 49 0. 91 1. 00 0. 74 0. 89 0. 61 0. 19 0. 69 0. 49 0. 85 0. 71 0. 52 0. 73 0. 86 0. 67 0. 79 0. 76 0. 58 0. 03 24 .7 0 47 20 1. 34 26 29 .2 8 18 .4 0 12 85 7. 45 2. 59 0. 17 13 .7 0 2. 11 L ig ht G B M 0. 67 0. 97 0. 61 0. 98 1. 00 0. 72 0. 98 0. 72 0. 36 0. 75 0. 75 0. 99 0. 81 0. 80 0. 91 0. 92 0. 93 0. 86 0. 85 0. 68 0. 03 19 .8 7 32 87 0. 70 18 98 .0 3 13 .0 2 11 63 9. 59 2. 45 0. 14 13 .4 7 1. 96 X G B oo st 0. 61 0. 93 0. 61 0. 98 1. 00 0. 73 0. 98 0. 71 0. 36 0. 75 0. 76 0. 99 0. 78 0. 80 0. 90 0. 92 0. 93 0. 86 0. 85 0. 68 0. 03 13 .7 9 36 37 5. 58 19 03 .0 3 12 .3 1 11 93 1. 23 2. 45 0. 15 13 .4 8 1. 85 C at B oo st 0. 67 0. 87 0. 60 0. 99 1. 00 0. 73 0. 96 0. 69 0. 38 0. 75 0. 79 0. 99 0. 84 0. 82 0. 91 0. 93 0. 93 0. 86 0. 86 0. 69 0. 03 14 .0 6 35 18 7. 38 18 86 .5 9 11 .8 9 11 61 4. 57 2. 41 0. 14 13 .4 4 1. 88 M L P 0. 25 0. 76 0. 50 0. 57 0. 90 0. 28 0. 84 0. 51 0. 21 0. 64 0. 49 0. 98 0. 38 0. 64 0. 54 0. 63 0. 80 0. 74 0. 88 0. 49 0. 02 18 .9 9 16 95 27 .1 5 15 67 .1 6 11 .2 9 10 43 5. 57 1. 91 0. 16 15 .3 0 1. 88 Ta bn et + M L M 0. 39 0. 91 0. 60 0. 99 1. 00 0. 68 0. 98 0. 71 0. 38 0. 73 0. 70 0. 98 0. 74 0. 71 0. 91 0. 93 0. 91 0. 82 0. 84 0. 65 0. 03 22 .7 6 42 75 1. 43 19 91 .7 7 15 .8 9 11 61 8. 68 2. 50 0. 16 13 .7 8 3. 35 Ta bT ra ns fo rm er + R T D 0. 67 0. 71 0. 59 1. 00 1. 00 0. 73 0. 94 0. 57 0. 39 0. 75 0. 74 1. 00 0. 79 0. 77 0. 90 0. 93 0. 89 0. 86 0. 84 0. 66 0. 03 22 .3 1 40 44 0. 32 19 93 .5 7 14 .6 9 12 38 1. 85 2. 36 0. 16 11 .8 4 2. 94 SA IN Ts 0. 62 0. 71 0. 65 0. 89 1. 00 0. 69 0. 97 0. 63 0. 19 0. 71 0. 75 1. 00 0. 75 0. 87 0. 95 0. 89 0. 91 0. 77 0. 82 0. 68 0. 03 9. 05 17 41 32 .3 3 19 74 .9 7 9. 58 11 79 5. 60 1. 95 0. 16 13 .3 8 1. 74 SA IN Ti 0. 65 0. 94 0. 60 0. 99 1. 00 0. 73 0. 98 0. 71 0. 38 0. 75 0. 77 0. 98 0. 76 0. 82 0. 92 0. 93 0. 92 0. 84 0. 85 0. 66 0. 03 12 .5 6 33 99 2. 51 19 97 .1 1 11 .5 1 11 61 2. 08 2. 10 0. 15 12 .5 3 1. 87 SA IN T 0. 68 0. 95 0. 61 1. 00 1. 00 0. 74 1. 00 0. 70 0. 38 0. 75 0. 79 0. 99 0. 84 0. 81 0. 92 0. 94 0. 92 0. 85 0. 86 0. 68 0. 03 11 .6 6 33 11 2. 39 19 53 .3 9 10 .2 8 11 57 7. 68 2. 11 0. 15 12 .5 8 1. 88 SA IN Ts + pr etr ai ni ng 0. 68 0. 74 0. 61 0. 98 1. 00 0. 73 0. 99 0. 58 0. 19 0. 76 0. 77 0. 98 0. 78 0. 80 0. 91 0. 93 0. 92 0. 82 0. 86 0. 66 0. 03 9. 61 19 34 30 .7 0 19 37 .1 9 10 .0 3 11 58 0. 83 2. 14 0. 16 12 .6 0 1. 83 SA IN Ti+ pr etr ai ni ng 0. 59 0. 95 0. 64 1. 00 1. 00 0. 75 0. 89 0. 76 0. 37 0. 76 0. 84 0. 97 0. 81 0. 84 0. 90 0. 90 0. 99 0. 88 0. 84 0. 65 0. 03 11 .5 6 31 14 1. 51 20 77 .8 4 12 .1 1 12 23 1. 31 2. 10 0. 16 13 .5 1 1. 77 SA IN T + pr etr ai ni ng 0. 62 1. 00 0. 57 1. 00 1. 00 0. 68 1. 00 0. 72 0. 41 0. 76 0. 79 1. 00 0. 91 0. 77 0. 95 1. 00 0. 99 0. 82 0. 89 0. 73 0. 03 12 .0 3 31 92 5. 26 20 72 .6 4 9. 42 11 55 4. 45 2. 09 0. 15 12 .6 1 1. 96

Review summaries:
The paper provides (main contributions) a new deep learning architecture (SAINT) for tabular data that performs attention over both samples and features. For datasets with missing labels, the authors also analyse a  new contrastive self-supervised pre-training approach. The paper also introduces a new embedding method for continuous attributes. 

This paper proposes SAINT, a neural network model for handling tabular data with both continuous and discrete values. SAINT uses both self-attention among variables and inter-sample attention among different samples. Using both InfoNCE and denoising objectives for pre-training strategies, SAINT was able to outrank all baseline methods in experiments with 30 different tabular datasets, consisting of binary-classification, multi-class classification, and regression.

The paper presents a new deep-learning network architecture for tabular data classification and regression problems, called SAINT. While deep-learning provided significant improvements in many domains, tabular data problems are still dominated by classical algorithms like XGBoost and Catboost. The new architecture is based on a new attention mechanism, applying attention both between samples (rows) and features (columns). It also embeds continuous features rather than embedding just categorical ones. The new model is studied on 30 diverse datasets (10 for binary classification, 10 for multiclass classification and 10 for regression), and compared to 10 previous types of models, including both classical models and recently proposed deep-learning models. The authors show that the average rank of SAINT across the various datasets is the best compared to these 10 algorithms (average rank of 2.7, vs. average rank of 4.0 for Catboost). Ablation studies are performed to evaluate the impact of each direction of attention and the impact of the embedding, and the explainability is demonstrated on MNIST. Additionally, the paper presents a contrastive learning approach for pre-training on unlabeled data and fine-tuning on a small number of labels, and it show that pre-training improves the rank for these small numbers of labels (on average by 1.4 for 50 labels and by 0.5 for 200 labels, not improving the rank when all labels are available).