Year: 2021

Abstract:
Handling tabular data with deep learning models is a challenging problem despite their remarkable success in vision and language processing applications. Therefore, many practitioners still rely on classical models such as gradient boosting decision trees (GBDTs) rather than deep networks due to their superior performance with tabular data. In this paper, we propose a novel hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer). The proposed SALT consists of two blocks: Transformers and linear layer blocks that take advantage of shared attention matrices. The shared attention matrices enable transformers and linear layers to closely cooperate with each other, and it leads to improved performance and robustness. Our algorithm outperforms tree-based ensemble models and previous deep learning methods in multiple benchmark datasets. We further demonstrate the robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios.

text: Handling tabular data with deep learning models is a challenging problem despite their remarkable success in vision and language processing applications. Therefore, many practitioners still rely on classical models such as gradient boosting decision trees (GBDTs) rather than deep networks due to their superior performance with tabular data. In this paper, we propose a novel hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer). The proposed SALT consists of two blocks: Transformers and linear layer blocks that take advantage of shared attention matrices. The shared attention matrices enable transformers and linear layers to closely cooperate with each other, and it leads to improved performance and robustness. Our algorithm outperforms tree-based ensemble models and previous deep learning methods in multiple benchmark datasets. We further demonstrate the robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios.

heading: 1 INTRODUCTION
text: In the fields of vision and natural language processing, deep networks such as CNN, RNN, LSTM and Transformer have gained great popularity with its impressive performance. In particular, Transformer(Vaswani et al., 2017) designed as language model, improves the performance of lots of deep learning models in various domains, so that there are many powerful models based on Transformer (e.g. Devlin et al. (2019), Brown et al. (2020) and Dosovitskiy et al. (2021)).
Although the deep networks are powerful and used in natural language processing and vision, they are sub-optimal for other types of real-world problems that require to use tabular data, such as fraud detection(Luo et al., 2019), product recommendation(Guo et al., 2017) and disease prediction(Koppu et al., 2020). Tabular data has different characteristics from other data. Unlike text data that include vocabularies and words, and image data that include RGB values, the tabular data are usually mixed with different types of complex variables. For example, the tabular data contain continuous variables such as such as age, height,and weight have different ranges of values, and categorical variables which are independent from one another like gender and nationality.
For these reasons, deep learning models were not quite successful with tabular data despite its strength in natural language processing and vision fields. Instead of deep learning models, classical models such as tree-based ensemble models are mainly used for tabular data. However, these classical approaches have some limitations. Continuous learning with real-time data is quite difficult using these classical methods. When the tabular data is high-dimensional sparse, the performance of tree-based methods is degraded. Also, the tree-based ensemble methods do not perform well for multi-modality learning and end-to-end systems.
Therefore, it is an interesting to study a deep learning models for tabular data. Many studies have been attempted not only to overcome the shortcomings of the classical model with a deep learning model but also to overcome its performance. (Arik & Pfister, 2020), (Huang et al., 2020) and (Somepalli et al., 2021).
In this paper, we propose a new hybrid deep learning model architecture, named as SALT (Sharing Attention between Linear layer and Transformer). We summarize the contributions of our paper as follows :
• SALT shares the attention matrices between two blocks. Sharing the attention matrix allows to learn two blocks strongly and effectively. We demonstrate that sharing attention matrices improves the performance better.
• SALT introduce the improved embedding method for continuous variables. We demonstrate that this method performs better than others.
• SALT has four types of variants depending on which block is used and in what direction the attention matrix is shared. Each variant shows strength in different data.
• SALT outperforms the other models on average over a variety of benchmark datasets. It also performs well even in small data environments via semi-supervised and self-supervised learning

heading: 2 RELATED WORK

heading: 2.1 TREE-BASED MODELS
text: Decision trees (Quinlan, 1986) are well-known for their high predictive performance compared to computational complexity. In addition, it has the strength of having explanatory power in units of variables with most statistical information gain. However, decision trees are likely to work well only on specific data because their decision boundaries are perpendicular to the data axis. To improve this drawback and performance, there are many ensemble models of decision trees such as Random forest (Breiman, 2001) and GBDTs(Gradient Boosting Decision Trees). Especially, GBDTs methods such as XGBoost(Chen & Guestrin, 2016), LightGBM(Ke et al., 2017) and CatBoost(Dorogush et al., 2018) are commonly used with powerful performance in lots of machine learning competitions and industrial sites.

heading: 2.2 DEEP LEARNING MODELS FOR TABULAR DATA
text: There are several studies on deep networks to overcome the limitations of tree-based models and outperform the performance of GBDTs. Especially there are deep learning models based on attention mechanism. TabNet(Arik & Pfister, 2020), for example, is designed to learn similarly to a decision trees, and it has interpretability with the attentive layer. It shows better performance than GBDTs in some dataset. TabTransformer(Huang et al., 2020) is designed based on Transformer and has contextual embedding values. However, it embeds only categorical variables, so that there is a limitation for continuous variables. There are some models that improve the limitation of TabTransformer(Song et al., 2018),(Somepalli et al., 2021). They have the contextual embedding values of not only categorical variables but also continuous variables. Especially a model called SAINT(Somepalli et al., 2021) introduces inter-sample self-attention method and shows powerful performance. But it has a fatal disadvantage that training costs are high.

heading: 2.3 TRANSFORMER
text: Self-attention is the core module of Transformer(Vaswani et al., 2017). The self-attention consists of three parameter matrices : K(keys), Q(queries), and V (values). Formally, input embedding values X ∈ Rn×d of n features of dimensions d, are projected using WQ ∈ Rd×dq , WQ ∈ Rd×dq , and WQ ∈ Rd×dq to extract feature representations Q, K, and V . With Q, K, and V , self-attention can be written as,
Q = XWQ,K = XWK , V = XWV (1)
Attention(Q,K, V ) = Softmax( QKT√ dk )V (2)
MHA(Multi Head Attention) is having multiple attention heads. Multi head allows the attention matrix to have the abundant representations(Michel et al., 2019),(Voita et al., 2019). Each head attention has different Q, K, and V weight matrices and calculates the attention values with the
equation 2. All heads are then concatenated and multiplied by the weight matrix to generate the final output of the layer.
MHA(Q,K, V ) = Concat(head1, ..., headh)WO (3) where headi = Attention(Q,K, V ) and WO is weighted matrix for final output. The dimension dh of each head is typically given as d/Nh.

heading: 2.4 GATING MECHANISM
text: The gating mechanism is an effective method of learning by controlling the information flow path of the LSTM (Hochreiter & Schmidhuber, 1997). However, the more layers of these mechanisms are stacked, the more likely the gradient is vanishing. As an improved mechanism, gate linear units(GLU) is introduced by (Dauphin et al., 2017). GLU has been used in many deep learning models and shows the better performance (Arik & Pfister, 2020),(Shazeer, 2020). To briefly explain the GLUs, they divide the input in half, take an activation function on one side, and multiply by element with the other. Therefore, the output dimension is half the input dimension.
h(x) = (W1X + b1)⊗ σ(W2X + b2) (4) The gating mechanism is used to control the information flow slightly similar to the self-attention mechanism. The main difference between gating mechanism and self-attention is that gating mechanism controls only the bandwidth of a single element, while self-attention considers information of two different elements.

heading: 3 SALT : SHARING ATTENTION BETWEEN LINEAR LAYER AND TRANSFORMER

heading: 3.1 SALT ARCHITECTURE
text: In this section, we introduce our model, SALT (Sharing Attention between Linear layer and Transformer) shown in Figure 1. SALT uses the input embedding values obtained from the embedding
layer. The embedding values have the shape of ’features(n)× embedding dimensions(d)’. The main body of SALT has two blocks: Transformers block inspired by (Vaswani et al., 2017) and Linear layer block inspired by (Liu et al., 2021). Each block has subblocks by feature-wise and dimensionwise. These two subblocks allow communications between different features and different embedding elements and makes the model robust(Tolstikhin et al., 2021). The output values from two blocks become the contextual embedding values. SALT performs fin-tuning and pre-training with contextual embedding values.

heading: 3.2 SALT FOR LEARNING
text: The learning process of SALT is as follows. Let D = {xi, yi}mi=1 be tabular dataset of m samples. The feature(n) variables xfeatures ∈ Rn include categorical xcat and continuous variables xcont. SALT adds the special token [cls] to the feature variables and takes them as an input values like BERT (Devlin et al., 2019). So, xi = [[cls], xcat, xcont] is the input values consisting of the feature variables and a special token [cls]. The embedding layer E(·) converts the input values xi ∈ R(n+1) into d-dimensional values E(xi) ∈ R(n+1)×d. Let Transformer be Transformer(·) and Linear layer block be Linearlayer(·). SALT consists of an L stack of these two blocks. Each block returns the output value z and the sharing attention matrices, sf and sd by feature-wise and dimension-wise as the following equations :
z (1) t , s (1) tf , s (1) td = Transformer1(E(xi)) (5)
z (1) l , s (1) lf , s (1) ld = Linearlayer1(E(xi)) (6)
z (1) t and z (1) l are the output values of Transformer block and Linear layer block from the first stack, respectively. s(1)ft and s (1) dt are the feature-wise attention matrix and dimension-wise attention matrix from Transformer block. s(1)fl and s (1) dl
are from Linear layer block. The matrices from blocks are calculated in the head direction by function S(·) and the calculated matrix becomes the sharing attention matrix for the next blocks.
Sf (sft , sfl) = concat(sft , sfl)Wf (7)
Sd(sdt , sdl) = concat(sdt , sdl)Wd (8)
Wf ,Wd ∈ Rh×2h are weight for projection in the head direction. The two attention matrices obtained from the above equation are sent to the blocks of the next stack. This procedure is repeated until the last stack as the following equations.
z (i) t , s (i) ft , s (i) dt = Transformeri(z (i−1) t , s̃ (i−1) f , s̃ (i−1) d ) (9)
z (i) l , s (i) fl , s (i) dl = Linearlayeri(z (i−1) l , s̃ (i−1) f , s̃ (i−1) d ) (10)
s̃ (i) f = Sf (s (i) ft , s (i) fl ) s̃ (i) d = Sd(s (i) dt , s (i) dl ) (11)
Finally, the output values of the last stack, zLl and z L t are added as the output of SALT. This value also becomes the contextual embedding valuesẼ(xi) = [ẽ([cls]), ẽ(x1), · · · , ẽ(xn)]. The cls token, ẽ([cls]) is used for fine-tuning, and the other values, ẽ(xi) are used for pre-training using MLM(Masked Language Model)(Devlin et al., 2019).

heading: 3.3 SHARING ATTENTION
text: The main idea of SALT is the sharing attention matrices between two blocks. The sharing attention matrices, s̃f and s̃d are obtained from the function S(·) defined as Equation 7 and 8. The blocks add these matrices to the attention matrix QK T
√ dk of the self-attention module. This flows similar with (He
et al., 2020). The self-attention module returns a new sharing attention matrix QK T
√ dk + s̃ obtained by adding the attention matrix and the sharing attention matrix s̃. The self attention module works the remaining operations with this new matrix.
Attention(Q,K, V, s̃) = softmax( QKT√ dk + s̃)V (12)
while Transformer has a multi head self-attention module, Linear layer has a single head selfattention module. For calculating the sharing attention matrix, Linear layer repeats the single head attention matrix QK T
√ dk with the number of heads of the sharing attention matrix.

heading: 3.4 LINEAR LAYER BLOCK
text: Linear layer block has two sub blocks. The one sub block is forward by feature-wise, the other sub block is by dimension-wise of embedding values. The sub blocks are defined as :
attn, s = Attention(x, s̃) (13)
z = σ(wx+ b) z̃ = g(z, attn) y = wz̃ + b (14) where σ is an activation function, g(·) is gating function and attn is the attention value from selfattention module(Liu et al., 2021). The gating function g(·) has two arguments, z and attn. To compute with attn which has multi-head form, z ∈ Rn×d needs to be converted shape same as attn ∈ Rh×n×d. Gating function is defined as :
u, v = split(z) (15)
vnorm = LayerNorm(v) (16) vout = wvvnorm (17)
z̃ = (vout + attn) u (18) The one of the divisions of z, v becomes vout through a normalization and multiplication weights wv . Then, vout is added with the attention value attn. Lastly, it calculates element-wise multiplication with u, the other of divisions. Linear layer block can capture representations of relationships using the gating function with attention values. Because gating mechanism controls the flow of information and works similar with self-attention. It is expected that the attention value will help capture more representations by adding it to v, one of the divided values before element-wise multiplication.

heading: 3.5 EMBEDDING LAYER OF SALT
text: SALT introduces the improved embedding layer of continuous variables of tabular data. In our knowledge, there are two methods of embedding for continuous variables. The one is using the embedding matrix m ∈ Rh×d, projection layers f(xconti) ∈ R1×h and softmax function (Guo
et al., 2021b). The other method is the embedding layer that converts f(xconti) ∈ R1×d into Econti(xconti) ∈ R1×d using only the projection layer used in Somepalli et al. (2021). The first method uses different embedding matrices for each variable to obtain an embedding value. SALT improves this method to use the same embedding matrix between categorical variables and continuous variables. As shown in 3 (a), this embedding layer E(·) works as follows.
E(xconti) = softmax(fi(xconti))⊗ e (19) E(xcati) = e[x̃cati ] (20)
where e ∈ Rhd is embedding matrix, fi(·) is projection into the h-dimensional space R1×h and x̃cat is the value for the index of embedding matrix e. Figure 3 (b) shows the tendency to increase of transformed the embedding value as the original input value increases. The value is the age variables of income dataset which we evaluate in our experiments. We demonstrate that our embedding layer of continuous variables works well and perform better than other methods. See Table 2.

heading: 4 EXPERIMENTS

heading: 4.1 BASELINE MODELS
text: We compare the proposed SALT architecture to the tree-based models such as Decision tree, Random Forests, XGBoost and LightGBM. We also evaluate the deep networks, simple MLP, TabNet and SAINT for comparing. TabNet and SAINT are the deep learning models studied for tabular data learning. SAINT has three variants SAINT, SAINT-i and SAINT-s. Therefore we evaluate the nine models for comparing with our model SALT.

heading: 4.2 VARIANTS OF SALT
text: As shown in Figure 4, there are four variants of SALT. The first is two-way sharing between two blocks. The second and third variants are designed only with Transformer and Linear layer blocks,
respectively. The last is one way sharing from Transformer block to Linear layer block. We evaluate all of them in the various experiments.

heading: 4.3 DATASETS AND METRIC
text: We evaluate the models including SALT on six binary classification datasets which are the various size of samples and features. The size of data samples varies from 1,055 to 45,211 samples. The number of features ranges from 11 to 57. The datasets are divided into 65%, 15% and 25% for training, validation and test sets. The datasets are publicly available in UCI(Dua & Graff, 2017), AutoML(Guyon et al., 2019) and Kaggle. We use the auroc as metric to measure the performance of binary classification. With the six datasets, we evaluate a variety of experiments, including supervised learning, unsupervised learning, and semi-supervised learning as well as the embedding method experiment.
4.4 PARAMETERS
We set default values for hyper-parameters as follows. Embedding size is 32, attention dimension is 16, the number of heads is 8 and the depth of stacks is 6. We also set the hyper-parameters of SAINT-s with these values. However, for SAINT and SAINT-i, we set the values following (Somepalli et al., 2021). Because the inter-sample attention of SAINT requires heavy resources for learning, as shown in 1 As optimizer, we use AdamW(Loshchilov & Hutter, 2019) with β1 = 0.9, β2 = 0.999, decay = 0.01 and learning rate = 1e−4. We set the 100 epochs, 30 early stopping count and 256 size of batch. For training, we use one GPU of Nvidia GeForce RTX 2080Ti.
5 RESULTS
5.1 EMBEDDING LAYERS
We evaluate the three types of embedding layer on six datasets. As mentioned in Section 3.5, the first method is using the multiplication with embedding matrices and values projected by layers. The second is using simple mlp for projection. Our proposed embedding method improves the first method and outperforms the others. See Table 2.

heading: 5.2 SHARING ATTENTION MATRIX
text: We visualize the attention matrices in Figure 5. It shows the effect of sharing attention matrices. (a) shows the attention matrices, the matrix of the Transformer block on the left, the matrix of the Linear layer block in the middle, and the matrix of the sharing function S(·) on the right. On the other hand, (b), (c) and (d) are models without the sharing attention matrices. They are
non-share mode of SALT, SALT-former, and SALT-linear, respectively. They show that each block learns certain parts more intensively. However, (a) shows that the sharing attention matrix helps each block learn more diversely. The matrices are from the experiments of online shopper datasets. The model of (a) has the best performance with an AUROC score of 0.927. (b), (c) and (d) perform 0.925, 0.924, and 0.925 with AUROC scores, respectively.
5.3 SEMI-SUPERVISED LEARNING & PRE-TRAINING
We evaluate SALT, SALT-variants and SAINT under the semi-supervised learning scenario where small-labeled samples are available. We compare five models under four conditions according to the number of labeled samples. The five models are four variants of SALT and SAINT. See Table 4 for results of the AUROC score. In the condition of using all samples, SALT performs the best. Pre-trained SALT and pre-trained SALT-oneway perform best under different conditions with few datasets of 50, 100, and 300 labeled samples. Other variants also show the better performance with pretraining in small data scenarios. We demonstrate that SALT performs well on small data environments with semi-supervised learning.

heading: 5.4 SUPERVISED LEARNING
text: In Table 5, we report the results on the six binary classification datasets. The models perform with five trials of different seeds. The one of SALT variants outperforms the most models on a variety of datasets. SALT has the best score in four of the six datasets, a majority, compared to other models including GBDTs. It also showed the best performance in the datasets except for one dataset compared to the deep learning models. SALT showe the best performance with an average score in all six datasets.

heading: 6 LIMITATIONS AND CONCLUSIONS
text: We propose SALT, a novel and hybrid deep learning architecture for tabular learning. SALT introduces the improved embedding method and the sharing attention matrix between two blocks. Furthermore, SALT can be improved for training resource with efficient transformers (Tay et al., 2020). In addition, variants of SALT showed superior performance in four out of six datasets compared to other models including GBDTs. SALT also shows benefits of unsupervised pre-training. With good performance and various benefits of attention matrix, SALT shows the attention mechanism works on tabular datasets too. However, our model still shows lower performance than GBDTs in some data. Therefore, continuous research on tabular data modeling is needed. The code is available at https://github.com/Juseong03/SALT.

Review summaries:
This paper proposes a hybrid deep network, named SALT (Sharing Attention between Linear layer and Transformer). The SALT consists of two blocks: Transformer and linear layer blocks that take advantage of shared attention matrices. They compare SALT with tree-based ensemble models and previous deep learning models on multiple benchmark datasets. It furher shows robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios.   The main body of SALT has two blocks: Transformers block inspired by (Vaswani et al., 2017) and Linear layer block inspired by (Liu et al., 2021). Each block has subblocks by feature-wise and dimension- wise. These two subblocks allow communications between different features and different embed- ding elements and makes the model robust (Tolstikhin et al., 2021). The output values from two blocks become the contextual embedding values. SALT performs fin-tuning and pre-training with contextual embedding values.  For supervised learning setting, the mean AUROC score of the proposed SALT improves upon the treebased LightGBM by mere 0.09%, and it improves upon the best deep learning based model SAINT by 0.29%

This paper proposed a hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer).  There are two blocks in SALT: Transformers and linear layer blocks. And sharing attention matrices are introduced to promote cooperation between these two blocks.

This paper introduces a Transformer-based architecture for tabular datasets. This architecture combines a Transformer with a gating MLP (gMLP) by sharing the attention matrices. The authors also proposed a new approach to encode continuous variables. The proposed model is evaluated on six binary classification tabular datasets. The proposed model is evaluated in a supervised learning context, but also in a semi-supervised context.