SALT : Sharing Attention between Linear layer and Transformer for tabular dataset
Authors: Juseong Kim
Jinsun Park
Giltae Song
Venue: ICLR 2022 Submitted (2022)
Keywords: Tabular data
Attention matrix
Transformer
Deep learning

Abstract: Handling tabular data with deep learning models is a challenging problem despite their remarkable success in vision and language processing applications. Therefore, many practitioners still rely on classical models such as gradient boosting decision trees (GBDTs) rather than deep networks due to their superior performance with tabular data. In this paper, we propose a novel hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer). The proposed SALT consists of two blocks: Transformers and linear layers blocks that take advantage of shared attention matrices. The shared attention matrices enable transformers and linear layers to closely cooperate with each other, and it leads to improved performance and robustness. Our algorithm outperforms tree-based ensemble models and previous deep learning methods in multiple benchmark datasets. We further demonstrate the robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios.

Under review as a conference paper at ICLR 2022
SALT SHARING ATTENTION BETWEEN LINEAR LAYER AND TRANSFORMER FOR TABULAR DATASET
Anonymous authors Paper under double-blind review
ABSTRACT
Handling tabular data with deep learning models a challenging problem despite their remarkable success in vision and language processing applications. There- fore, many practitioners still rely on classical models such as gradient boosting decision trees (GBDTs) rather than deep networks due to their superior perfor- mance with tabular data. In this paper, we propose a novel hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer). The proposed SALT consists of two blocks: Transform- ers and linear layer blocks that take advantage of shared attention matrices. The shared attention matrices enable transformers and linear layers to closely coop- erate with each other, and leads to improved performance and robustness. Our algorithm outperforms tree-based ensemble models and previous deep learning methods in multiple benchmark datasets. We further demonstrate the robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios.
INTRODUCTION
In the fields of vision and natural language processing, deep networks such CNN, RNN, LSTM and Transformer have gained great popularity with its impressive performance. In particular, Trans- former Vaswani et al., (017) designed language model, improves the performance of lots of deep learning models in various domains, so that there are many powerful models based on Transformer (e.g. Devlin et al. 2019). Brown al. (220) and Dosovitskiy et al. (201)).
Although the deep networks are powerful and used in natural language processing and vision, they sub-optimal for other types of worl-world problems that require to tabular data, such as fraud detectiongLuu et al. 2019), product recommendation er al.] 2017) and disease predic- Koppu al., (020). Tabular data has different characteristies from other data. Unlike text data that include vocabularies and words, and image data that include RGB values, the tabular data are usually mixed with different types of complex variables. For example, the tabular data contain continuous variables such as such as age, height,and weight have different ranges of values, and categorical variables which are independent from one another like gender and nationality.
For these reasons, deep learning models were not quite successful with tabular data despite its strength in natural language processing and vision fields. Instead of deep learning models, clas- sical models such tree-based ensemble models are mainly used for tabular data. However, these classical approaches have some limitations. Continuous learning with real-time data quite difficult using these classical methods. When the tabular data is high-dimensiioal sparse, the performance of tree-based methods is degraded. Also, the brse- ensemble methods do not perform well for multi-modality learning and end-to-end systems.
Therefore, it is an interesting to study deep learning models for tabular data. Many studies have been attempted not only to overcome the shortcomings of the classical model with deep learning model but also to overcome its performance. (Arik Phister 2020), (Huang et al.] 2020) and (Somepalli al. 2021

---

Under review as a conference paper at ICLR 2022
In this paper, we propose new hybrid deep learning model architecture, named SALT (Sharing Attention between Linear layer and Transformer). We summarize the contributions of our paper as follows:
SALT shares the attention matrices between two blocks. Sharing the attention matrix allows to learn two blocks strongly and effectively. We demonstrate that sharing attention matrices improves the performance better.
SALT introduce the improved embedding method for continuous variables. We demon- strate that this method performs better than others.
SALT has four types of variants depending which block is used and in what direction the attention matrix is shared. Each variant shows strength in different data.
SALT outperforms the other models on average over a variety of benchmark datasets. also performs well even in small data environments via semi-supervised and sulf-supervvsed learning
2
RELATED WORK
2.1 TREE-BASED MODELS
Decision trees (Quinlan 1986) are well-known for their high predictive performance compared to computational complexity. In addition, it has the strength of having explanatory power in units of variables with most statistical information gain. However, decision trees are likely to work well only on specific data because their decision boundaries are perpendicular to the data axis. To improve this drawback and performance, there are many ensemble models of decision trees such as Random forest (Breiman 2001) and GGrasiGr Boosting Decision Trees). Especially, GBDTs meth- ods such as XGBoost(Chen & Guestrin 2016), LightGBM(Ke et al. 2017) and CatBoost CatBBosst(oroguu (orogush eet al. 2018) are commonly used with powerful performance in lots of machine learning competitions and industrial sites.
2.2 DEEP LEARNING MODELS FOR TABULAR DATA
There are several studies on deep networks overcome the limitations of tree-based models and outperform the performance of GBDTs. Especially there deep learning models based on at- tention mechanism. TabNet(Arik Plister 2020), for example, designed to learn similarly to decision trees, and it has interpretability with the attentive layer. It shows better performance than GBDTs some dataset. TabTTaassfrmeerHHunn et al., 2020) is designed based on Trans- former and has contextual embedding values. However, embeds only categorical variables, that there is limitation for continuous variables. There are some models that improve the limitation of RabTransformer(SSne (Songegrrnnng et al., 2018).(Somepp) et al.) (02T). They have the contextual embedding values of not only categorical variables but also continuous variables. Especially model called (Aoep(So et al. 202T) introduces inter-sample self-attention method and shows powerful performance, But it has fatal disadvantage that training costs are high.
2.3 TRANSFORMER
Self-attention is the core module of Transformer Vaswani et al. 2017). The self-attention consists of three parameter matrices: K(eey)), Q(queries), and (vvalues). Formally, input embedding values X E R2 of n features of dimensions d, are projected using Wo E Rd We E Rd and We E Rdxd, to extract feature representations Q, K, and V. With Q, K, and V, self-attention can be written as,
Q = XWQ, K XWW, V = XW
(1)
QKT V Vdk
Attention (Q, (2 V) Softmax
MHA(Multi Head Attention) is having multiple attention heads. Multi head allows the attention matrix to have the abundant representations(Michne et al, 2019). et al., 2019). Each head attention has different Q K, and V weight matrices and calculates the attention values with the
2

---

Under review as a conference paper at ICLR 2022
equation All heads then concatenated and multiplied by the weight matrix to generate the final output of the layer.
MHA(Q, K, V) = Concat(headd, heada) Wo
(3)
where head, = Attention Q, K, V) and WO weighted matrix for final output. The dimension de of each head is typically given as d/Np.
2.4 GATING MECHANISM
The gating mechanism is an effective method of learning by controlling the information flow path of the LSTM (Hochreiter & Schmidhuber, 1997). However, the more layers of these mechanisms are stacked, the more likely the gradient is vanishing. As an improved mechanism, gate linear unit) is introduced by (Dauphin al. 2017). GLU has been used in many deep learning models and shows the better performance (Arik & Prister 2020), (Shazeer) (020). To briefly explain
the GLUs, they divide the input in half, take an activation function on one side, and multiply by element with the other. Therefore, the output dimension is half the input dimension.
h(z) = (W1X + 61) @ (W2X + 62)
(4)
The gating mechanism is used control the information flow slightly similar to the self-attention mechanism. The main difference between gating mechanism and self-attention is that gating mech- anism controls only the bandwidth of single element, while self-attention considers information of two different elements.
3 SALT SHARING ATTENTION BETWEEN LINEAR LAYER AND TRANSFORMER
3.1 SALT ARCHITECTURE
Rea Turlng
PIt Tralting
Cedinial
(NTVE PPSL-PP Rill
Treedeess
TRAMSOOA
Figure 1: The Architecture of SALT that has two blocks, Transformers and Linear layers. SALT train with [CLS] token for fine-tuning and with contextual embedding values for Semi-supervised learning and Pre-training.
In this section, we introduce our model, SALT (Sharing Attention between Linear layer and Trans- former) shown in Figure SALT uses the input embedding values obtained from the embedding
3

---

Under review as a conference paper at ICLR 2022
layer. The embedding values have the shape of (eatures(()) x embedding dimensions(d)). The main body of SALT has two blocks: Transformers block inspired by (Vaswani et al. (017)) and Linear layer block inspired by (Liu et al. 202T). Each block has subblocks by feature-
and dimension- wise. These two subblocks allow communications between different features and different embed- ding elements and makes the model blocks become the contextual embedding values. SALT performs fin-tuning and pre-training with contextual embedding values.
al., (02T). The output values from two
3.2 SALT FOR LEARNING
The learning process of SALT is as follows. Let D = { V: be tabular dataset of m samples. The feature(n) variables I features E Iº include categorical Toot and continuous variables Zcont- SALT adds the special token [cls] to the feature variables and takes them
input values like BERT (Devlin et al. 2019). So, 24 = [[c1s], &eats Teons] the input values consisting of the feature variables and special token [cls]. The embedding layer E(-) converts the imput values F, E R(++1) into d-dimensional values E(x;) R(x+1)xd Let Transformer be Transformer(-) and Linear layer block be Linearlayer(-)) SALT consists of an stack these two blocks. Each block returns the output value and the sharing attention matrices, 8f and sa by feature-wise and dimension- wise as the following equations:
2t Byg 8ta
(1) (1) (1)
Transformer: (F((z))
(5)
(1) (1) (1)
11 = LLiearlayer, (E(a;))
(6)
2t and 2 are the output values of Transformer block and Linear layer block from the first stack,
(1)
(1) respectively. ff and dd, are the feature-wise attention matrix and dimension-wise attention matrix
(1)
(1) from Linear layer block. The matrices from blocks from Transformer block. ff. and d
(1)
calculated in the head direction by function S(.) and the calculated matrix becomes the sharing
attention matrix for the next blocks.
S((5f.88)) s5,
Sa(sa, sd.)) = concat(s2,.1 )d)) Wa
(8)
Wa E R2x2h are weight for projection in the head direction. The two attention matrices obtained from the above equation are sent to the blocks of the next stack. This procedure is repeated until the last stack as the following equations.
de
Transformer, (2) (f-1) ((--1)
.88 "0-"",,(-1) ;/(
(9)
(1-1) (1-11 5(1-1)
de
Linearlayer,
(2)
.88
(10)
Sy
(() 5d
-S
Sa(s 8
(11)
Finally, the output values of the last stack, =/ and af are added as the output SALT. This value also becomes the contextual embedding values E(x,) = [e([c1s)) ((x1),... ē(xx)). The cls token, ē([c1s]) is used for fine-tuning, and the other values, ē(xx)) are used for pre-training using MLM(Masked Language Model)(Devinn et al. 2019).
3.3 SHARING ATTENTION
The main idea of SALT is the sharing attention matrices between two blocks. The sharing attention matrices, & and & are obtained from the function S(-) defined as Equation [nd The blocks add
these matrices to the attention matrix º of the self-attention module. This flows similar with (He vor
al. 2020). The self-attention module returns new sharing attention matrix ºKT + obtained by
vdz adding the attention matrix and the sharing attention matrix The self attention module works the remaining operations with this new matrix.
QKT dah
Attention(() Q, K, V, softmax
+ 3)V
(12)
4

---

Under review as a conference paper at ICLR 2022
while Transformer has multi head self-attention module, Linear layer has single head self- attention module. For calculating the sharing attention matrix, Linear layer repeats the single head attention matrix QK" with the number of heads of the sharing attention matrix. vod
3.4 LINEAR LAYER BLOCK
Azti Nom
Fredharsand
spit
Starng Anttthon searo Base 3]
Sirst head ball Allantaan
ritenttion
Mazi
mmm
(a)
(b)
Figure 2:
The blocks of SALT. The left (a) sub block of Linear layer block. The right (b) is sub block of Transformer block.
Linear layer block has sub blocks. The sub block is forward by feature-wise wise, the other sub block is by dimension-win of embedding values. The sub blocks are defined as
alin, Attention(z, 5)
(13)
= [wr + b) I = g(2, attn) y w +
(14)
where a is an activation function, g(.) gating function and alln the attention value from self- attention module(Liuu et al. (021). The gating function g(.) has two arguments, and altn. To compute with attn which has multi-head form, z nn needs to be converted shape same as attn jxxxxxd Gating function is defined as
u, U = split(2)
(15)
Unorm LayerNorm())
(16)
Vout Pnorm
(17)
(Vout attn) O
(18)
The of the divisions of 2, v becomes Vout through normalization and multiplication weights Wy. Then, Vout is added with the attention value attn. Lastly, it calculates element-wise multipli- cation with u, the other of divisions. Linear layer block can capture representations of relationships using the gating function with attention values. Because gating mechanism controls the flow of information and works similar with self-attention. is expected that the attention value will help capture more representations by adding it to one of the divided values before element- wise mul- tiplication.
3.5 EMBEDDING LAYER OF SALT
SALT introduces the improved embedding layer of continuous variables of tabular data. In knowledge, there are two methods embedding for continuous variables. The one is using the embedding matrix E Rixdd projection layers f( IR1x and softmax function (Guo
5

---

Under review as a conference paper at ICLR 2022
Vode
AGE
(a)
(b)
Figure The improved embedding layer introduced by SALT.
al. 20 The other method is the embedding layer that converts [(Eront Rixd into
Boont, (cooss)) E IR 1xd using only the projection layer used in Somepalli et al. (022). The first method uses different embedding matrices for each variable to obtain embedding value. SALT improves this method to use the same embedding matrix between categorical variables and continu- ous variables. As shown in (a), this embedding layer E(-) works as follows.
E( = softmax (fa (f (zeont,)) 8
(19)
E( = e[ãcat,]
(20)
where Rhd is embedding matrix, f.(-) is projection into the h-dimensional space 1x and Zeat
the value for the index of embedding matrix e. Figure 3 (b) shows the tendency to increase of
transformed the embedding value the original input value increases. The value is the age variables
of income dataset which we evaluate in our experiments. We demonstrate that our embedding layer of continuous variables works well and perform better than other methods. See Table
4 EXPERIMENTS
4.1 BASELINE MODELS
We compare the proposed SALT architecture to the tree-based models such as Decision tree, Ran- dom Forests, XGBoost and LightGBM. We also evaluate the deep networks, simple MLP, TabNet and SAINT for comparing. TabNet and SAINT are the deep learning models studied for tabular data learning. SAINT has three variants SAINT, SAINT-i and SAINT-s. Therefore we evaluate the nine models for comparing with our model SALT.
4.2 VARIANTS OF SALT
(b)
(c)
Figure 4: The variants of SALT. (a) SALT (b) SALT-linear (c) SALT-former (d) SALT-oneway
As shown in Figure 4 there are four variants of SALT. The first is two-way sharing between two blocks. The second and third variants are designed only with Transformer and Linear layer blocks,

---

Under review as a conference paper at ICLR 2022
SALT variants
share mode share
SALT
SALT-oneway
SALT-former
SALT-Linear
0.9094
0,9075
0.9069 0.9065
0,9066
Non share
0.9089
0.9071
0.9055
Table 3: The results of the comparison between share mode and Non share mode with SALT variants.
respectively. The last one way sharing from Transformer block to Linear layer block. We evaluate all of them in the various experiments.
4.3 DATASETS AND METRIC
We evaluate the models including SALT six binary classification datasets which are the various
size of samples and features. The size of data samples varies from 1,055 to 45,211 samples. The number of features ranges from 11 to 57. The datasets divided into 65%, 15% and 25% for training, validation and test sets. The datasets are publicly available UCIiDua & Graff (017)). AutoML/GGuyo al., 2019) and Kaggle. We the auroc metric measure the performance of binary classification. With the six datasets, we evaluate a variety of experiments, including su- pervised learning, unsupervised learning, and semi sumi-super learning as well the embedding method experiment.
4.4 PARAMETERS
We set default values for hyper-parameters as follows. Embedding size is 32, attention di- mension is 16, the number of heads is 8 and the depth of stacks is 6. We also set the hyper-parameters of SAINT-s with these val- ues. However, for SAINT and SAINT-i, we set the values following (Somepalli et al. (202T). Because the inter-sample attention of SAINT requires heavy resources for learning, shown in As As optimizer, we use Adam W (Loshchilov & Hutter 2019) with B1 0.9, f2 0.999, decay 0.01 and learning rate = 1e-4. We set the 100 epochs, 30 early stopping count and 256 size of batch. For training, we use one GPU of Nvidia GeForce RTX 2080Ti.
Model
Batch
L
Param le6
SAINT
256
196. 16
SAINT-i
256
1
195.87
SAINT-s
256
6
53.48
SALT
256
77.65
SALT-oneway
256
6
77.64
SALT-former
256
6
77.49
SALT-linear
256
6
77.32
Table The parameters comparing SAINT on
six datasets
RESULTS
5.1 EMBEDDING LAYERS
We evaluate the three types of embedding layer on datasets. As mentioned in Section 3.5) the first method is using the multiplication with embedding matrices and values projected by layers. The second is using simple mlp for pro- jection. Our proposed embedding method improves the first method and outperforms the others. See Table 2
Methods
Mean AUROC
Guo et al. (2021a
0.9066
Simple MLP
0.9079
Ours
0.9094
Table 2: The Mean AUROC for the embedding methods with SALT six datasets.
5.2 SHARING ATTENTION MATRIX
We visualize the attention matrices in Figure It shows the effect of sharing attention matrices. (a) shows the attention matrices, the matrix of the Transformer block on the left. the matrix of the Linear layer block in the middle, and the matrix of the sharing function S(-) on the right. On the other hand, (b), (c) and (d) models without the sharing attention matrices. They

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
Figure 5: Visualization of the (feature-wise) attention matrices. For each stack in the model we plot the row. (a) and (b) show the effect of share mode and non-share mode, respectively. (c) is from SALT-foremr and (d) is from SALT-linear.
non-share mode SALT, SALT-former, and SALT-linear, respectively. They show that each block learns certain parts more intensively. However, (a) shows that the sharing attention matrix helps each block learn more diversely. The matrices are from the experiments of online shopper datasets. The model of (a) has the best performance with an AUROC score of 0.927. (b), (c) and (d) perform 0.925, 0.924, and 0.925 with AUROC scores, respectively.
5.3 SEMI-SUPERVVISED LEARNING & PRE-TRAINING
We evaluate SALT, SALT-variants and SAINT under the sumi-super learning scenario where small-labeled samples available. We compare five models under four conditions ac- cording the number of labeled samples. The five models are four variants of SALT and SAINT. See Table 4 for results of the AUROC score. In the condition of using all samples, SALT performs the best. Pre-trained SALT and pre-trained SALT-oneway perform best un- der different conditions with few datasets of 50, 100, and 300 labeled samples. Other vari- ants also show the better performance with pre- training in small data scenarios. We demon- strate that SALT performs well on small data environments with semi-supervised learning.
Labeled
Model name
50
100
300
ALL
SAINT
0.7935
0.8235
0.8549
0.9065
SALT
0.7983
0.8251
0.8647
0.9094
SALT-oneway
0.7934
0.8225
0.861
0.9075
SALT-former
0.7909
0.8219
0.8582
0.9069
SAT.T-inear
0.7934
0.8168
0.8587
0.9066
SAINT+pt
0.8025
0.827
0.8591
SALT
0.8068
0.8284
0.8656
0.9084
SoL.T-newwy-+t
0.8036
0.8288
0.866
0.9079
SALT-former+pl
0.8012
0.8225
0,8588
0.9066
SALT-lineartpe 0.7981
0.817
0,8589
0.9059
Table 4: The results under semi-supervised learn- ing, varying by the number of labeled training samples. The pt means pre-training.
5.4 SUPERVISED LEARNING
In T we report the results on the six binary classification datasets. The models perform with five trials of different seeds. The one of SALT variants outperforms the most models variety of datasets. SALT has the best score in four the six datasets, a majority, compared to other models including GBDTs. It also showed the best performance in the datasets except for one dataset compared to the deep learning models. SALT showe the best performance with an average score in all six datasets.
8

---

Under review as a conference paper at ICLR 2022
Sample size
4,601
10,000
7,043
1,055
32,561
45,211
Feature size
57
20
16
Mcan
Model dataset
Spambase
Shrutime
Blastchar
Qsar bio
Income
Bank marketing
Decision Tree
0.9643
0.6698
0.8277
0.92
0.7204
0.8051
0.8179
Random Forest
0.9851
0.8527
0.8196
0.9269
0.9028
0.9254
0.9021
XGBaost
0.987
0.8479
0.8119
0.9216
0.9198
0.928
0.9027
LightGBM
0.9882
0.8611
0.8252
0.9224
0,9236
0.9307
0.9085
MLP
0.9758
0,4846
0.792
0.9238
0.6226
0.7347
0.7556
TabNet
0.9777
0,8274
0.7923
0.8251
0.9066
0.9196
0.8748
SAINT
0.9835
0,8638
0.8339
0,9198
0.9113
0.9264
0.9065
SAINT-i
0.9833
0.8617
0.8327
0,9260
0,9098
0.9259
0,9066
SAINT-s
0.9821
0.8584
0.8335
0,9121
0.9124
0.9302
0.9048
SALT 0.985*
0.8675
0.8341
0,9272
0.9113
0,9311
0.9094
SALT-oneway 0.9829
0.866
0.8347
0.9221
0.9T09
0.9286
0.9075
SALT-former 0.9835
0.8634
0.8329
0.9221
0.9104
0.9288
0.9069
SALT-linear
0.9832
0.8643
0.8333
0.9199
0.9104
0.9287
0.9066
Table 5: The Mean AUROC of evaluating models with six datasets. Bold means the best perfor- mance in the entire models. means the best performance in the deep learning models.
6 LIMITATIONS AND CONCLUSIONS
We propose SALT, a novel and hybrid deep learning architecture for tabular learning. SALT intro- duces the improved embedding method and the sharing attention matrix between two blocks. Fur- thermore, SALT can be improved for training resource with efficient transformers (Tay et al.) 2020). In addition, variants of SALT showed superior performance in four out of six datasets compared to other models including GBDTs. SALT also shows benefits of unsupervised pre-training. With good performance and various benefits of attention matrix, SALT shows the attention mechanism works on tabular datasets too. However, our model still shows lower performance than GBDTs in some data. Therefore, continuous research on tabular data modeling is needed. The code available at
REFERENCES
Sercan O. Arik and Tomas Pfister. TabNet:
Attentive Interpretable Tabular Learning. www.orggg/ab///oooeee
arXiv: 1908. 07442 [cs, stat), December 2020. 07442 arXiv: 1908.07442.
URL
Leo Breiman. Machine learning, volume 45, number springerlink. Machine Learning, 45:5-32, 10 2001. doi: 10.1023/A:10109333000
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners arXiv:2005. 14165 arXiv: 14165.
[cs], July 2020. URL htt.aatt://axx.v.or///bbaa
14165
Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 785-794, August 2016. doi: 10. 011144/2299967222997333 URL http:///xxxxvvx.x//a./. wrt.:/srg////////
0275.02754
arXiv: 1603.02754.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolutional Networks. 1:Xiv:1612.0800 [cs], September 2017. URL httt:://arxi: arXiv: 1612.08083.
org/abs/
/tcabb///.... 612. 08083
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. arXiv: 1810.04805.
URL htRt:/tt:///axt wwwpiivvorgga//a//aeeee 04805
04805

---

Under review as a conference paper at ICLR 2022
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. CatBoost: gradient boosting with categorical features support. 1800: 18363 [cs, stat], October 2018. URlhttp://aaxix
org/abs/ 18100//1111. 11363
arXiv: 1810.11363
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recogmition at Scale. arXiv:2010. 1929 arXiv: 2010.11929.
[cs], June 2021. URL http:hhtp::ttree. www.orrgggbb.o. www.orrggbbb///./.o/a
Dheeru Dua and Cascy Graff. UCI machine learning repository, 2017. ics. uci. meiieedumml
2017. URL httt:://tccciivee
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiugiang He. Deepfm: A factorization-
machine based neural network for CTR prediction. CoRR, abs/1703.04247, 2017. URL http:
//arxiv.org abs / 1703.1700 04247
Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. An
embedding learning framework for numerical features in ctr prediction. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, Aug 2021a. doi: 10.114573444548834477777 URLhttp::/ doi. http:///g/g............. 1el.11114444545 01444447588..777777777 3467077
Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. An Embed- ding Learning Framework for Numerical Features in CTR Prediction. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2910-2918, August org 08986
2021b. doi: 10. 441444445548.34677777 URL arXiv: 2012.08986.
Isabelle Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengy- ing Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michéle Sebag, Alexander Statnikov, Wei- Wei Tu, and Evelyne Viegas. Analysis of the automl challenge series 2015-2018. In AutoML,
Springer series on Challenges in Machine Learning. 2019. URL https: /www. automl. oml. org/ www.ecootttttupllaacc..... 2018 8/ 09/ chapter chatteero-cchallle wew.lleenee.... pdf
Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. RealFormer: Transformer
Likes Residual Attention. arXiv:2012.11747 [cs], December 2020. URL http://arxiv.
oxq/abs:2002. 11747 arXiv: 2012.11747.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8):
1735-1780, 1997.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. arXiv:2012. 06678 [cs], December 2020. /ttttt
//arxiv. //arxiv.org/aat wtt.://orrraabb//2..0.555 06678
arXiv: 2012.06678.
Guolin Ke. Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Q. Ye, and Tie- Yan
Liu. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, 2017.
Srinivas Koppu, Praveen Kumar Reddy Maddikunta, and Gautam Srivastava. Deep learning disease prediction model for use with intelligent robots. Computers Electrical Engineering, 87:106765, 2020. ISSN 0045-7906. doi: httpa//etee...tteeeeeeeeeeeeeeaa
URL htths:ss:
// www. sc incceediiiicccccccccc rect
Hanxiao Liu, Zihang Dai, David R. So, and Quoc Le. Pay Attention MLPs. arXiv:2 [cs], June 2021.
08050
arXiv: 2105.08050.
Nya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. 1 fes, math), Jamuary 2019, URL
/11, 111.171111111111111111 arXiv: 171 1.01001.
Yuanfei Luo, Mengshuo Wang, Hao Zhou, Quanming Yao, Wei- Wei Tu, Yuqiang Chen, Qiang Yang, and Wenyuan Dai. Autocross: Automatic feature crossing for tabular data in roal-world applications. CoRR, abs/1904. 128577 2019. htRl:htt:://aart http://aaaav..og//ab/////b/. www./bo//////ooeo..oooooooe//./r /0oobs///94.... 12857
10

---

Under review as a conference paper at ICLR 2022
Paul Michel, Omer Levy, and Graham Neubig, Are Sixteen Heads Really Better than One? TrXiv:/90S.10650 [cs], November 2019. URL pttt:://arxiv.org/aaaa.
arXiv: 1905.10650.
R. Quinlan. Induction of decision trees. MACH. LEARN, 1:81-106, 1986.
Noam Shazeer. GLU Variants Improve Transformer. 2xXi::2002.05222 [cs, stat), February 2020.
URL http::tttpp.l.....////ap wttp:/iii.rr///oooee abs//////2222 05202]
arXiv: 2002.05202.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-
Training. arXiv:2 106.01442 [cs, stat], June 2021. URL httt:://axxiv..org/bbs///22
01342 arXiv: 2106.01342.
Weiping Song. Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. CoRR, abs/18100. 19211, 1992, 2018. URL
org/ 1t1.//81...1222
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient Transformers: A Survey. arXiv:2009. 06732 [cs], September 2020. URL. http:::/arxw..org////a/t. arXiv: 2009.06732.
llya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MIP-Mixer: An alI-MIP Architecture for Vision. arXiv:2 105.01601 [cs], June 2021.
01600
arXiv: 2105.01601.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. cember 2017. URL http::://axi...rrrree/e wttp://wwwrraa/////ee////// 03762
/eeo:000603777 [cs], De- arXiv: 1706.03762.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi- Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. arXiv: arXiv: 9995009418 [cs], June 2019. URL http:://aatiiii hww.l/.org/abbbbb/ 905. 09418
arXiv:
1905.09418.


Reviews:
[{"review_id": "jJP87vyPFd7", "reviewer": "Reviewer_mutc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of the paper is marginal improvement over the SOTA deep learning model SAINT for supervised setting scenario. \n\nThe main weakness is in lack of justification for the proposed choice of SALT model, and inadeqate explanation of the SALT architecture. It is not clear what is dmension-wise attention. There is no motivation/justification for the Transformers and Linear layers with attention sharing. Further, why is that there is only one layer of feature-wise attention and linear layer, and then L layers of dimension-wise attention and linear layer. Neiher has been given any good justification for sharing attention between Transformer and Linear layer. Further, for the most important supervised learning scenario, the SALT model barely beats the tree-based LightGBM model.", "summary_of_the_paper": "This paper proposes a hybrid deep network, named SALT (Sharing Attention between Linear layer and Transformer). The SALT consists of two blocks: Transformer and linear layer blocks that take advantage of shared attention matrices. They compare SALT with tree-based ensemble models and previous deep learning models on multiple benchmark datasets. It furher shows robustness of the proposed SALT with semi-supervised learning and pre-training with small dataset scenarios. \n\nThe main body of SALT has two blocks: Transformers block inspired by (Vaswani et al., 2017) and Linear layer block inspired by (Liu et al., 2021). Each block has subblocks by feature-wise and dimension- wise. These two subblocks allow communications between different features and different embed- ding elements and makes the model robust (Tolstikhin et al., 2021). The output values from two blocks become the contextual embedding values. SALT performs fin-tuning and pre-training with contextual embedding values.\n\nFor supervised learning setting, the mean AUROC score of the proposed SALT improves upon the treebased LightGBM by mere 0.09%, and it improves upon the best deep learning based model SAINT by 0.29%", "summary_of_the_review": "Though the numerical exeriments show that the proposed model SALT marginally improves over the sota deep learning model SAINT, it barely beats the tree-based LightGBM model. The architectural choices of SALT have no motivation/justification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e0CrrbNMzTH", "reviewer": "Reviewer_eW8f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have serious concerns about the paper, as listed below: \n\n- The paper writing is very poor. There are numerous language and grammar errors. At many places, the expressions are not clear. The quality of the figures is low. \n\n- Overall, the novelty is low. Transformer architecture is modified in a straightforward way. Even the semi-supervised learning approach is the simple adaption of MLM-like unsupervised pre-training. \n\n- Benchmarking is employed and presented very poorly. For comparison models like TabNet, XGBoost etc., it is not clear how the hyperparameter tuning is done and what parameters are included in the search space. The authors should clearly describe which parameters were tuned and what validation reward is used. Otherwise, the outperformance conclusions are not convincing at all.\n\n- How do you do hyperparameter tuning in semi-supervised regime? It is unclear how semi-supervised validation data is used.\n\n- The results are presented in AUROC but what is the training objective? If the training objective is not AUROC, how do you ensure the metric mismatch is not dominating? \n\n- What is the significance of Fig. 5? How can you convince the readers that SALT has learned attention patterns that are meaningful?\n\n- The difference between supervised learning results is very low across different models. It is unclear whether the results are statistically significant. \n\n- No ablation studies are presented for the major constituents of the claims, such as the benefit of sharing attention layers. ", "summary_of_the_paper": "The paper proposes a new tabular deep learning architecture based on sharing attention matrices enable transformers and linear layers. Comparisons with other tabular learning models on various benchmarks are demonstrated. ", "summary_of_the_review": "I suggest substantially revising the paper and submitting to another venue. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VSrZ9Qk09jc", "reviewer": "Reviewer_A8EH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- I think it is interesting to develop deep models for tabular data. Learning on tabular datasets is still an under-explored research area.\n- The idea of sharing attention matrices looks interesting. \n- The authors show that combining a Transformer-like architecture with a gMLP-like one improves the performances. (I assume the SALT results are significantly better than SALT-former and SALT-linear, see my comment later).\n- The authors analyzed different variants of their model.\n\n**Weaknesses**\n\n- Section 3.5: “SALT improves this method to use the same embedding matrix between categorical variables and continuous variables.” I am not sure what is the intuition behind this idea. The authors should motivate this claim.  \n- The model is presented to work on any tabular datasets. However from experience, a lot of tabular datasets contain at least one date/time column. I wonder if SALT can work on date/time columns. \n- The model is evaluated only for binary classification tasks. I think adding experiments for other tasks (e.g. multi-class classification, multi-label classification, regression) can increase the quality of the paper. \n- The authors should explain how they do the semi-supervised learning in section 5.3. \n- For most of the tables, I recommend showing the variance for different random seeds or some other confidence interval measures. For a lot of results, the difference between two models looks small and it is difficult to know if the difference is significant. \n- It is difficult to understand Table 3 without additional information. I recommend adding information about the dataset or metric used in the caption. This Table is not mentioned in the text.\n- The authors visualize the attention matrices for different variants of their model. However, these visualizations are not commented on so it is difficult to understand the message from this analysis. \n- The authors should comment on the scalability of their model. The SAINT model was evaluated on a larger dataset. I wonder why SALT was not evaluated on this dataset. I think the authors should also compare the training time/inference time of their model with existing models. \n- The authors should add information about the Transformer architecture used. Is it the original Transformer architecture also called PostNorm? Or is it more recent architecture like PreNorm or ReZero?\n- The last sentence of the conclusion breaks the anonymity of the paper: “The code is available at https://github.com/Juseong03/SALT” \n", "summary_of_the_paper": "This paper introduces a Transformer-based architecture for tabular datasets. This architecture combines a Transformer with a gating MLP (gMLP) by sharing the attention matrices. The authors also proposed a new approach to encode continuous variables. The proposed model is evaluated on six binary classification tabular datasets. The proposed model is evaluated in a supervised learning context, but also in a semi-supervised context.", "summary_of_the_review": "Overall, I feel it is quite difficult to understand the potential impact of this paper. The idea of sharing attention matrices looks interesting but there are some questions about the scalability of the proposed approach and if it can work on any tabular datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ODX7CeUiutA", "reviewer": "Reviewer_6yF7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- This manuscript attempts to propose a deep learning model for tabular data, a problem with practical application value.\n- An interesting hybrid network architecture is presented, the main idea of which is good and reasonable.\n\nCons:\n\n- The novelty is limit. The key block SHARING ATTENTION shares similar idea with existing works, RealFormer.\n- The writing of the manuscript can be polished. Typos in Equation 7 and 8 are confusing.", "summary_of_the_paper": "This paper proposed a hybrid deep network architecture for tabular data, dubbed SALT (Sharing Attention between Linear layer and Transformer). \nThere are two blocks in SALT: Transformers and linear layer blocks. And sharing attention matrices are introduced to promote cooperation between these two blocks.", "summary_of_the_review": "The proposed SALT, in my view, should be technically correct. \nBut I think the novelty is limit. \nConsequently, although both the problem and the method is okay, this manuscript is a borderline work.\nI recommend rejecting it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to use Transformers with tabular data by sharing attention. Reviewers raise significant concerns about the motivation, writing and experimental results. Author's did not submit a response. Hence I recommend rejection.
Decision: Reject