MET : Masked Encoding for Tabular data
Authors: Kushal Alpesh Majmundar
Sachin Goyal
Praneeth Netrapalli
Prateek Jain
Venue: Submitted to ICLR 2023 (2023)
Keywords: Tabular Data
Self Supervised Learning
Masked Auto-Encoder

Abstract: We propose  $\textit{Masked Encoding for Tabular Data (MET)}$ for learning self-supervised representations from $\textit{tabular data}$. Tabular self-supervised learning (tabular-SSL) -- unlike structured domains like images, audio, text -- is more challenging since each tabular dataset can have a completely different structure among its features (or coordinates), which is hard to identify a priori. $\textit{MET}$ attempts to circumvent this problem by assuming the following hypothesis: the observed tabular data features come from a latent graphical model and the downstream tasks are significantly easier to solve in the latent space. Based on this hypothesis, $\textit{MET}$ uses random masking based encoders to learn a positional embedding for each coordinate, which would in turn capture the latent structure between coordinates. Through experiments on a toy dataset from a linear graphical model, we show that $\textit{MET}$ is indeed able to capture the latent graphical model. Practically, through extensive experiments on multiple benchmarks for tabular data, we demonstrate that $\textit{MET}$ significantly outperforms all the baselines. For example, on Criteo -- a large-scale click prediction dataset -- $\textit{MET}$  achieves as much as $5\%$ improvement over the current state-of-the-art (SOTA) while purely supervised learning based approaches have been able to advance SOTA by at most $2\%$ in the last six years. Furthermore, averaged over $\textit{nine}$ datasets, $\textit{MET}$ is around $3.9\%$ more accurate than the next best method of Gradient-boosted decision trees -- considered as SOTA for the tabular setting.

Under review as a conference paper at ICLR 2023
MET: MASKED ENCODING FOR TABULAR DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
We propose Masked Encoding for Tabular Data (MET) for learning self- supervised representations from tabular data. Tabular self-supervised learning (tabular-SSI.) unlike structured domains like images, audio, text more chal- lenging since each tabular dataset can have a completely different structure among its features (or coordinates), which hard identify priori. MET attempts circumvent this problem by assuming the following hypothesis: the observed tab- ular data features come from latent graphical model and the downstream tasks are significantly easier to solve in the latent space. Based on this hypothesis, MET uses random masking based encoders to learn a positional embedding for each coordinate, which would in turn capture the latent structure between coordinates. Through experiments on a toy dataset from a linear graphical model, we show that MET is indeed able to capture the latent graphical model. Practically, through extensive experiments on multiple benchmarks for tabular data, we demonstrate that MET significantly outperforms all the baselines. For example, Criteo a large-scale click prediction dataset - MET achieves as much as 5% improve- ment over the current state-of-the-art (SOTA) while purely supervised learning based approaches have been able to advance SOTA by at most 2% in the last six years. Furthermore, averaged over nine datasets, MET around 3.9% more accu- rate than the next best method of Gradient -bossted decision trees considered as SOTA for the tabular setting.
INTRODUCTION
Self-supervised pre-training (SSL.) followed by supervised fine-tuning has emerged as the state of the art approach for multiple domains such as natural language processing (NI.P) (Devlin et al.)
2019). computer vision (Chen et al.) 2020b) and speech/audio processing (Baevski et al. [2020) However, despite presence of an extensive amount of raw and unlabeled data in a variety of critical tabular-heavy domains like finance, marketing, etc., it has been challenging extend SSL based pre-training approaches to tabular data.
Broadly speaking, there are two dominant approaches to SSL: (i) reconstruction of masked inputs, also known as contrastive learning. Most of the existing tabular-SSL. methods (Verma et al., 2020 Ucar al. 2021) have adopted the second approach of contrastive learning. The underlying structure and semantics of specific domains such as images remain somewhat static, irrespective of the dataset. So, one can design generalizable domain specific augmentations like cropping, rotating, resizing etc. However, tabular data does not have such fixed input vocabulary space (such as pixels in images) and semantic structure, and thus lacks generalizable augmentations across different datasets. Consequently, there are only limited number of augmentations that have been proposed for the tabular setting such as mix up, adding random (gaussian) noise and selecting subsets of features (Verma et al. 2020) Ucar et al.) 2021).
and (ii) invariance to certain augmectaation///aasffommatiiii
In this paper, we hypothesize the following: for any tabular dataset, (i) there is latent (i.e., unknown/unobserveed) graphical model that captures the relations between different coordi- nates/features, and (ii) classification easier in the latent space. For example, in the CovType dataset where the task is to predict the type of forest (e.g., deciduous, alpine etc.) given features such as elevation, soil type extensive research in mountain and forest science has established that there are very specific relations among different features (Catry et al. 2009; Badía et al.) 2016), and leveraging and learning these relations could yield significant improvements in classification accuracy of machine learning models.

---

Under review as a conference paper at ICLR 2023
Positional Embeddings (PB)
Masked Token Represgntation
Input Data (z)
Txask
Reconstruction Loss
Masking (%9
Masked Input (x=))
Input Embedding $
Encoder (To
Encoder Output Tha (f( )
Concat
Concet
Reconstructed Input hu( Tex (Ta... (M(="),<<))
Projection Head hw
Decoder Output
Decoder
Decoder Input (Th. (**="))
Te ( (V (V(="))
(To
Figure I: MET Framework for tabular-SSL. Given an input z, we mask y fraction of it's coordinates to get the masked input z². We then concatenate the coordinate value x² with a learnable positional encoding PE to obtain its input embedding )x=²). The obtained input embedding is then passed through the encoder Tome The encoder output To.. )1)((z)) is then concatenated with the masked token representations 77S, and given as the input to the decoder Tos.- Finally, the decoder output is projected back to the input space using projection head h.,, obtain the reconstructed input é. Reconstruction loss then optimized end-to-end.
Based on this hypothesis, we propose masking-based reconstruction approach for self- self-supervised learning for tabular datasets. More concretely, for every unlabeled data point, we randomly choose fraction of the coordinates, mask their values, and then train model to predict the values of these masked coordinates using the remaining unmasked coordinates. We use a transformer architecture with learnable (positional) embeddings for each coordinate, which capture the relations between different coordinates. While masked reconstruction task with a transformer architecture has been successfully used for SSL in computer vision (He al. 2021) and NLPP (Devlin al. 2019), the best of our knowledge, this is the first work to successfully apply this paradigm to tabular datasets.
In particular, we demonstrate through experiments on simple toy tabular setting, how the position embeddings in a transformer, learned with masked reconstruction task, can capture the dependency structure across features. Further, on a real-world dataset of forest cover type classification, we show that features with the most correlated positional embeddings indeed have meaningful relations between them, as corroborated by extensive works in the forest science literature.
We evaluate the performance of MET through extensive experiments on nine datasets spanning a wide range in number of examples, classes and difficulty. Our experiments show that MET out- performs current SOTA tabular-SSL methods like DACL (Verma et al. (020). SubTaab (Ccar et al.) 2021I, VIME (Yonn et al. 2020), as well as SOTA tabular supervised algorithms such as gradient boosted decision trees (GBDT) (Mason et al. 1999) on all of these datasets. MET gives an average accuracy improvement of 3.9% over the second best algorithm, which GBDT. For example, on Criteo a popular large scale dataset for click through rate prediction with 43 million examples MET achieves 5% improvement in AUROC over the current SOTA (Wang et al., 2021). To put this in perspective, the SOTA Criteo has improved by less than 2% over the last six years Leaderboard (002). Furthermore, on some datasets, MET trained with about 20% of the labeled train-set is as effective as standard supervised learning methods trained with all the labeled points in the train-set.
To summarize, in this paper, we propose MET, which is masking based reconstruction approach with a transformer architecture, and demonstrate its effectiveness for tabular-SSL. Conceptually, through experiments on a toy setting, we show that this approach can learn the relations between different coordinates in the dataset, which helps in downstream classification. Practically, we show through extensive experiments on several popular tabular datasets that MET significantly outper- forms all the current SOTA tabular-SSL. baselines as well as SOTA supervised approaches.
RELATED WORK
Self-Supervised Learning (SSL): SSL has shown promising results not only in the regimes where the labelled training data is scarce but has also shown great empirical success in training large-scale models across various domains like Natural Language Processing and Computer Vision. SSL can be broadly classified into two categories: Pretext task based approaches and contrastive learning based approaches. Pretext based SSL approaches solve a a"pretext" task like reconstruc- tion from a masked or noisy input, to learn the underlying distribution of the unlabeled data.
2

---

Under review as a conference paper at ICLR 2023
Wav2 Wav2Veo(Baevskt et al. 2020) efficiently trains a large scale speech-to-tex model using masked re- construction. Similarly, Jiang et al. (2019); Hsu et al. (2021) have used masking for learning speech representations. Masked language modelling has been extensively studied in literature (Devlin et al. 2019 Radford & Narasimhan 2018), and has shown quite promising results. Motivated by the suc- cess of masking based approaches, recent paper He al. (202T) proposed similar approach for visual representation learning. Prior to this, Li et al. (2021) also proposed masking, although in feature space. In this paper, motivated by the constraints in tabular setting along with the success of masking based approaches, we build a purely reconstruction based approach for tabular-SSL.
concurrent line of work in SSL learns representations using instance level separation tasks as dis- cussed in Chen et al. (2020b); [Chen & He (2021): (Zbontar et al. (2021)) He et al. (2020). However, such approaches require domain knowledge to create positive-negative sample pairs. Recently, do- main agnostic augmentations for creating separation tasks have been proposed (Verma et al. 2020) We compare MET against such approaches in our empirical study (Section
SSL for Tabular Data Reconstruction based SSL has been previously explored in SubTab Ucar al. (202I), which treats as a multi-view representation learning problem. They try to learn repre- sentations for multiple croppings of the input data and inference time aggregate the representations of the croppings (multiple-views). Note that MET performs random masking over the input space only during training. at inference the representation is given by passing all the coordinates through the encoder and hence does not consider the problem as a multi-view representation learning. Yoon al. 2020) uses combination of predicting the masked tokens and reconstruction. Both Yoon al. (2020); Ucar et al. (202)) add gaussian noise to prevent indentity map learning by the auto-encoder. MET efficiently searches for noise using adversarial search to learn better representations. Another work Huang et al. (2020) uses transformers to solve a pretraining task of predicting the class of missing categorical features using cross-entropy. However, MET uses masked reconstruction and doesn't 't make any assumption on the type of features i.c. categorical or continuous.
Adversarial Self Supervised Learning While adversarial SSL has been explored in the context of contrastive learning (Kim al., 2020), it seems to be less explored for reconstruction based SSL. Our method MET proposes a novel framework where we try to find adversarial points in the input manifold which have a high reconstruction loss. Chen et al. (2020ah have also proposed adversarial learning, although to learn robust pre-trained models. MET instead explores the use of adversarial search over input manifold to learn better separable representations for higher accuracy on down- stream classification. Shi et al. (2022) proposes to find an adversarial mask which maximizes the distance between the representations of input and its masked (adversarial) counterpart.
Learning Graphical Models There is large body of work unsupervised Iearning of prob- abilistic relations between different coordinates/features. as expressed by a graphical model (Chow & Liu 1968 Bach & Jordan 2002 Bresler et al. 2008 Netrapalli al. 2010 Klivans & Meka 2017). . A popular approach in these works is to predict the value of any single coordinate using all the remaining coordinates. This can however be computationally challenging since this procedure needs to be repeated for every coordinate. The masking based reconstruction task with a random subset of coordinates masked in every step can be seen as a computationally efficient way of do- ing the same thing. To the best of our knowledge, this is the first work to connect masking based reconstruction approaches for SSL with the work on learning graphical models.
3 PRELIMINARIES
Notation: We use E Rdd to denote an input datapoint and x² to denote the 5th coordinate of x. Every coordinate in x i.e. x R can be either a categorical or a non-categorical value, without being explicitly specified. Let S denote the set of masked co-ordinates for and 5 = [d] S be the set of mon-masked coordinates. S is constructed by randomly sampling [rd] coordinates from the set [d]. where ? is the masking ratio hyper-parameter. Note that we sample different S in every training iteration. Let S denote the masked input, i.e. S consists of only the non-masked coordinates/features x Vj E 5.
Self-Supervised Representation Learning: Consider access to corpus of unlabelled data given by D., (a where each input x; E Rdd The general goal of SSL is to lear parameterized
3

---

Under review as a conference paper at ICLR 2023
mapping To..: R - R between the input 2; and its representation To... (2,) E R", such that the representations are well suited for a downstream task as described next.
Evaluation of Learned Representations: In this paper, evaluate the quality of learned repre- sentations through accuracy on a downstream classification task. More concretely, we have access to labelled training dataset D Puain { 9e) ) where y/ E I and each (Iz ys) is drawn inde- pendently and identically (i.i.d) from some underlying distribution D on Rd x R The task is to learn classifier Cs Rd + R which minimizes (x [x(c y)}, where é is a loss function such loss or cross entropy loss etc. Given the learned encoder network To..: : R I", train shallow classifier 9μ. Rm IR (we 2-hidden layer MLP in our default setting) and use the resulting accuracy to evaluate the quality of learned representations from To i.e., we set (x) gu (T (Ir) and train only 9μ-
METHOD
As motivated in Section 1 our proposed approach MET is based on the hypothesis that there is an underlying graphical model which captures the relation between the various coordinates (features) of the tabular data. MET tries to learn this graphical model, using a masked reconstruction based technique. More concretely, this generally involves having encoder To... : RR - RT and decoder Tex. R + Rd. Encoder's 's task is to take noisy version of the input x (c.g.. where some coordinates of æ are masked) and map it to some latent representations (z). The decoder then maps the representations back to the input space to get reconstructed input å. The reconstruction error ||x - 2 is then generally used as a learning objective (although MET also involves an adversarial objective explained later). Note that this masked reconstruction task, interpreted through the lens of learning graphical models, can learn the underlying latent structure among different coordinates (Klivans & Meka 2017). thereby making downstream classification tasks easier.
Our proposed model to get the reconstructed input é is given by:
== hww (To ([Te.. (w(x 1-),)
s.t., [2 )1s = Concat(x' PE*), Vj E 5, p0 Coonca((Imms,PP)) Vj S. (1)
Parsing the above Equation given an input x and its set of masked coordinates S, we obtain its input embedding )x(x) by concatenating the coordinate value zí with a learnable positional
encoding PE for every unmasked coordinate E S. The obtained input embedding then passed through the encoder To... The encoder output To... (1b(x²)) then concatenated with the masked token representations v. and given as input to the decoder Tex Finally, the decoder output is projected back the input space using a projection head hw. to obtain the reconstructed input å. We explain each of the components in detail below:
1. Input Embedding: : Given an input æ and it's set of masked coordinates S, recall that x denotes the masked input. Then (x E (i dduaee denotes the input embedding corresponding to the masked input. It is constructed in per-coordinate fashion by concatenating the coordinate value x E IR with a learnable positional encoding PF E Teddetedd -1 Vjj E S.
) Concat( PE*) R Vj S.
2. Encoder: Given that transformers (Vaswani et al. 2017b) explicitly capture the relation between different coordinates through positional embeddings and attention mechanism, they are a natural choice for the encoder Te... and the decoder Te.. architecture. Later in Section 4.3 we show that the learnt positional embeddings PE in MET, indeed capture the underlying graphical model, indicating why MET is successful various downstream tasks.
The representations for the masked input i.c. the encoder output now given by:
To (x TR. d
3. Masked Token Representations: Before we pass the encoder output to the decoder, we concate- nate it with the masked token representations i.e. the representations for the coordinates which
4

---

Under review as a conference paper at ICLR 2023
masked, given by 2 Rddmm Again, 2/ is constructed in a per-coordinate fashion, by concatenating a fixed learnable "mask token" Emask E R with PE
[ Concat(tmask PE E R Vj E S.
(4)
Note that we do not pass the masked token representations T through the encoder.
4. Decoder And Projection Head The concatenated encoder output and masked token represen-
tation i.e. [To... (v)()))) (3-5 is passed as input to the decoder Tow Finally, the decoder output is mapped back to the input space using linear projection head hw 12dxdunn + R. to obtain the reconstructed input as given by Equation
4.1 MET-S: ONLY RECONSTRUCTION BASED ORJECTIVE
First, simplified version of our proposed approach, which we call MET-S, uses just the reconstruc- tion error as the learning objective. Given corpus of unlabeled dataset D. {x; M the learning objective for MET-S is given by:
N₂ x3 -0,
LMET-S (fenee O W, PE, mmask)
1.1
(5)
where é; denotes the reconstructed input for z,, as given by Equation
At inference time (downstream classification), we don't t mask the input i.e. S and S [d]. Also, we drop the decoder Te.. and simply use the "flattened" encoder output as learnt representa- tions. More concretely, given Fd we pass the whole imput through the encoder and "flatten" the encoder output get the learnt representations (xx) Te.. (V(xld)) IR where m d dembed.
4.2 MET: ADVERSARIAL RECONSTRUCTION
In the context of supervised learning, several papers have demonstrated that adversarial training can yield more robust features Tsipras et al. 2018) that are better for transfer learning (Salman al.) 2020). While an adversarial loss function has been observed to encourage learning of robust features contrastive SSL (Chen et al. 2020a Kim al. 2020), to the best our knowledge, does not seem to have been explored in the context of masked reconstruction. In this work, we demonstrate that an adversarial version the reconstruction loss works better than standard reconstruction loss. More concretely, the adversarial reconstruction loss is given by:
Caaly (Denc: Odcey W. PE, mmask)
||: &:
max
where a. denotes the reconstruction corresponding to perturbed input x; + in Equation I (replac- ing with + in Equation D}. In this paper, constrain the adversarial noise in radius L2 norm ball around the input data point æi, where is chosen from a grid-search in {2, 4, 6, 10, 12, 14).
Finally, MET minimizes the sum of loss functions given in Eqn. 5 and Eqn. (6)
MMET LMETSS X.
We fix À = 1 in all our experiments. The overall algorithm for MET, which minimizes Eqn. 7 is For consistency notation, we present a non-bate non-batch version of the algorithm.
given Algorithm[] [
4.3 ANALYSIS ON TOY DATASET
As explained previously in Section T proposed approach MET relies on learning the graphical model that defines the relation between various features (coordinates) of the tabular dataset. We first show that MET is indeed able learn this latent graphical model by presenting some interesting results on 10-dimensional toy dataset.
Let each datapoint æ 10 be sampled from linear graphical model as follows (recall that xd e R denotes the j&h coordinate of :):
0 NO 1); x =
=xxx (- n [1. where ny N((,
(8)
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: MET' Masked Encoding Tabular data
Input
Unlabelled data .., {a; ) masking ratio [0, 1]. N iterations, Encoder
Tom. Decoder Te mask token muskk projection head h positional encodings PE, data embedding vi(x;)), mask token embedding v projection radius weight of adversarial loss
for iteration 0, 1, N do
for x E Du do
Sc [d]
s.t. [S| = Lyd] /* Random subset of coordinates to mask +/
à = (Te Bes
(Te.. (1b(2) 1), 1)
/* Reconstruct from masked input +/
Lice |æ - 213
/* Standard reconstruction loss. */
/* Initialize adversarial perturbation. for steps in 1, 2...... ..v stvssteps do
8~ S~,,OO,aa/vA
x* Find adversarial perturbation å maximize reconstruction loss using gradient ascent.
1 (Tom TT.., (v((x + +))5))
Lex 6) = la 1
8 = 8 TVannll
V.Cm
8 = |8
where a = ||8|| 1[] < <e]+ c 1[]6] 2
6 = 1 (Tomm To (vb(x + 1) 1))
Cady = 10 10
/N Adversarial reconstruction loss. */ /+ Final loss is a sum of standard and adversaria, serraarllll reconstruction losses. */
Lioal = Cad + x Lady
Ω = Ω VLCual where Ω = {feme: Olee, W, mmskk PE} / Gradient descent. on the set of all learnable parameters Q. */
Note that each coordinate x² is distributed according to N 1). For downstream classification, the label is given by y 1 " n² + (0.5)) where 4-1 denotes the inverse CDF for chi-square distribution with 9 degrees of freedom. This choice is so that we have a balanced set of positive and negative examples.
0.89
1.0
0.91
0.67
0.46
0.25
0.89
0.05
0.03
6.13
0.57
0.67
0.82
1.0
0.94
0.7
0.57
0.22
0.2
0.01
Co-ordinates
First, on the downstream task, our proposed approach MET (96.7 accuracy) outperforms all the baselines, most competitive being Gradient Boosted Decision Trees (88.4 4 accuracy); Appendix A.4 Furthermore, MET is
0.92
0.25
0.35
0.7
0.89
3.0
0.9
0.7
0.31
0.06
0.06
0.05 0.05
0.22
0.46
0.7
0.9
1.0
0.79
0.65
able to identify the relation between consecutive coordi- nates and hence the entire underlying graphical model, as demonstrated by the high cosine similarity scores be- tween learned positional embeddings of consecutive co- ordinates in Figure 2 As the downstream task is fairly simple once the graphical model learned (i.e. in n; space), MET is able to provide significantly more ac- curate solution. vation real world tabular datasets, where MET is able to identify the meaningful relations among all features.
0.41 0.13 0.1
0.01
0.02
0.06
0.41
0.65
0.92
1.0
Co-ordinates
Figure 2: Cosine Similarity score for the learnt positional embeddings the toy dataset by MET.
Section 5.4 we make a similar obser-
5 EXPERIMENTS
We now present extensive empirical study of our proposed approach MET against various tabular- SSL methods, along with other classical supervised-learnine baselines as described below. We experiment with standard tabular-SSI. benchmarks namely permuted MNIST, permuted Fashion- MNIST and permuted CIFAR-10 10 that are used by existing works Verma et al. (2020); |Carr et al.) 2021 well as several real world datasets namely CovType, Adult- Obesity, Arrhythmia,

---

Under review as a conference paper at ICLR 2023
Table Downstream classification accuracy four standard multi-class tabular datasets, comparing MET against various baselines. MET uses adversarial training + masking while MET-S only uses masking. MET either matches or outperforms the baselines across the all the datasets, giving an average accuracy improve- ment of 2% compared to the next best method of GBDT. Standard deviations are mentioned in appendix for completeness, although they statistically insignificant.
Type
Methods
FMNIST
CIFARTO
MNIST
CovType
Average
MLP
87.62
16.50
96.95
65.47
66.64
RF
88.43
42.73
97.62
71.37
75.04
Supervised Baseline
GBDT
88.71
45.7
100
72.96
76.84
RF-G
89.84
29.32
97.65
71.57
72. 10
MET-R 88.84
28.94
97.44
69.68
71.23
VIME
80.36
34.00
95.77
62.80
68.23
Seff-Superviee Methods
DACL+
81.40
39.70
91.40
64.23
69.18
SubTab
87.59
39.34
98.31
42.36
66.90
MET-
90.94
48.00
99.01
74.11
78.02
Our Method
MET
91.68
47.82
99.19
76.71
78.85
Thyroid. Finally, we consider Criteo, which an important large scale click-through rate prediction dataset with 45 million training samples. A detailed description of the datasets is in AppendixA...
5.1 BASELINES AND EXISTING METHODS
VIME Yoon et al. (020)) A SSL. approach which uses a combination of masked token prediction
and reconstruction loss. Note that MET proposes masked input reconstruction.
SubTab Ucar et al. (2021) SubTab views SSL multi-view representation learning problem,
where representations from multiple croppings are aggregated at test time.
DACL Verma et al. 2020) domain agnostic contrastive learning baseline which uses mixup We specifically use DACL+ which uses geometric mean based mixup.
augmentation.
MLP: We also compare against this natural baseline, wherein we train MLP over the raw tabular data (and not the learnt representations) with the available labeled samples.
Random Forest (RF): We train random forest with 100 decision trees the tabular data.
Gradient Boosted Decision Trees (GBDT) Friedman 2001): GBDT one of the most successful approaches on tabular data general.
Random Featurization (MET-R): We fine-tune an MI.P over the representations from randomly initialized encoder, checking the effectiveness of learnt representations in MET.
Random Gaussian Featurization (RF-G) Here, we compute random kitchen sink (Rahimi & Recht 2008) style features, i.c., (xx) Rr is the embedding of point x RRd Random features are known to be asymptotically an accurate approximation of the RBF kernel, which in turn is known to be a highly accurate and in fact, a "universal" classifier for tabular data. Note that we fix the embedding dimension of RF-G to be same as that of MET.
5.2 IMPLEMENTATION DETAILS
We use transformers Vaswani et al. 701 as the backbone for both the encoder and the decoder. Embedding dimension for the encoder and the decoder is chosen from a gridsearch in (64, 100, 128). feedforward dimension from {64, 100, 128). encoder and decoder depth from {1, 3, 6} and the num- ber of heads from {1, 3). For the tree based baselines (RF and GBDT), we do a gridsearch over the step size in { {1.0, 0.3, 0.1, 0.01, 0.001, 0.0001), maximum depth in {2, 5, 10, 20, auto) and min- imum samples leaf node in {1,22 2,5), 5). The exact hyper-parameee for all the experiments are in appendix.
5.3 DOWNSTREAM CLASSIFICATION
In this section, we compare the quality of representations obtained from MET against various base- lines mentioned in Section 5.1 We compare the downstream classification accuracy using

---

Under review as a conference paper at ICLR 2023
Table 2:
Downstream accuracy and AUROC scores on five popular tabular datasets for binary classification, for
MET and other baselines. MET outperforms the baselines on both accuracy and AUROC metries. Overall, MET achieves an average accuracy (AUROC) improvement of 5.40% (4.64%) over the next best method of GBDT. Standard deviations (statistically insignificant) are mentioned in appendix.
Datasets
Metric
MLP
RF
GBDT
RF-G
MET-R
DACL+
VIME
SubTab
MET
Accuracy
58.1
65.99
67.19
58.39
58.8
62.34
59.23
67.48
74.38
Obesity
AUROC
52.3
64.36
64.4
54.45
53.2
61.18
57.27
64.92
71.84
Accuracy
84.36
85.88
86.01
85.59
75.51
84.46
82.23
84.43
86.25
Income
AUROC
89.39
91.53
92.5
90.09
83.48
89.01
87.37
88.95
93.85
Accuracy
74.28
71.09
72.03
74.62
73.57
69.82
68.78
73.02
78.49
Criteo
AUROC
79.82
77.57
78.77
80.32
79.17
75.32
74.28
76.57
86.17
Arrhy-
Accuracy
59.7
68.18
69.79
60.6
51.67
57.81
56.06
60.1
81.25
thmia
AUROC
72.23
90.63
92.19
74.02
58.36
69.23
67.03
69.97
98.75
Accuracy
50
94.94
96.44
50
57.42
60.03
66.
59.9
98.1
Thyroid
AUROC
62.3
99.62
99.34
52.65
82.03
86.63
94.87
88.93
99.81
Accuracy
65.29
77.22
78.30
65.84
63.39
66.89
66.48
68.99
83.69
Average
AUROC
71.21
84.74
85.44
70.31
71.25
76.27
76.16
77.87
90.08
MLP over the learnt representations in two cases : (a) when we use the entire labeled training data and (b) when we use only a fraction of the labeled training data.
Multi-Class Classification Benchmarks TTalle compares accuracy of MET with downstream classification against tabular-SSL methods and supervised learning baselines. Both MET (standard + adversarial reconstruction) and MET-S (only standard reconstruction) outperform the baselines across all the datasets. For example, the permuted Fashion MNIST dataset, MET achieves an accuracy of 91.68%, outperforming all the other tabular SSL baselines like SubTab (87.59%) and also supervised baselines like GBDT (88.71 71% (1) and RF-G(899.8 84%). Similarly, on CovType, MET is about 10% more accurate DACL+ and in fact about 34% more accurate than SubTab, perhaps due lack of semantics in neighbouring columns in the dataset. Here again, MET is around 4% better then the state-of-the-an classical baseline GBDT. Overall, MET establishes itself as a new state-of-the-art (SOTA) approach for self supervised learning on tabular data.
presents results on five real-world binary classi- fication tabular datasets. MET again outperforms all other methods by a significant margin. In particular, on Criteo, which is a large-scale (45 million training sample) click prediction dataset, it achieves AUROC of 86. 17% which around 7.5% higher than that of GBDT and around 6% higher than the next best method RF-G. To put this in context, the SOTA AUROC on this dataset is R 81% [Wang et al. 202T) and has increased by less than 2% over the last six years. Overall, MET achieves an average AUROC (accuracy) improvement of 5. 5.40% (4.64%) over the next best method of GBDT.
Binary Classification Benchmarks Table 2
Accuracy With A Fraction Of Labeled Training Data: Next we do evaluation when only a frac- tion of labeled data is available for supervised training. Specifically, we vary the fraction of labeled data from 20% to 100% and compare the obtained downstream classification accuracies with the baselines. Figure Jshwws the variation of accuracy with the fraction of labeled data for MET, com- paring against four of the strong baselines. We observe that MET outperforms the baselines for all the choices of fraction of labeled data used for supervised learning.
Fashion MNIIST
Forest CoverType
Adult Income
MLP META
15.3
HIT
Accuit
MCS4
MLP
MLP
87.0
NET
30
Proction train data(%)
Proction of train datai%)
Prattsn of train datart%
We compare the performance (downstream classification accuracy) of MET with various haselines as the fraction of labeled data used for training the downstream classifier is varied. Observe that MET consistently outperforms the haselines even when a smaller fraction of labeled data is used for downstream classifier training.
Figure 3:
Effect of Adversarial Reconstruction: Next, we analyze the effect of using adversarial reconstruc- tion loss, along with the standard reconstruction loss i.e., MET vs MET-S. In Table 1 observe
8

---

Under review as a conference paper at ICLR 2023
that MET always outperforms or matches MET-S (without adversarial noise), and in some cases like CovType gives up to 2.5% improvement compared to MET-S.
5.4 ANALYSIS AND ABLATION STUDIES
Here, we investigate whether MET is able learn the underlying graphical model and also perform an ablation study to understand the effect of masking ratio and how the masked tokens are used.
MET Learns The Underlying Graphical Model On CovType: Similar to results in Section 4.3 we observe that the learned positional embeddings from MET indeed capture meaningful relations among the coordinates on real-world data too. For example, the position embedding of "Elevation" feature is maximally correlated (i.e., has highest inner product) with that of "Soil Type" feature. Indeed, several works in the forest science literature e.g., (Badfa et al. 2016) have established the strong relation between elevation and soil type. MET is able to learn this relation through the masked reconstruction task. Similarly there is high correlation between the position embeddings of "Slope" and "Wilderness" features, and "Distance to roadways" and "Nearest wildfire ignition points" fea- tures, which again has been investigated in literature Mert& & Yalcinkaya (017);; Francisco Moreira & Bacao (01O). More details the features and correlations between their positional embeddings can be found in Appendix A.3
Further, note from Table [and Table 2 MET is much more accurate than RF-G. While random features asymptotically approximate an RGF kernel itself, MET performs much better as it is able capture the underlying graphical model using the reconstruction task.
Masking Ratio We observe that in general a high masking ratio (7) of 50% 70% seems to give the best downstream accuracy and hence good separable representations. Figure 4 in Appendix A.1.I shows detailed plots of change in accuracy masking ratio is varied across four datasets.
Mask Token Recall that MET uses a learnable mask token, which passed to the decoder directly. Note that the mask token is kept same for all the masked coordinates (although positional encoding would be different) in MET. We tried a variant MET, in which we used a separate mask token (tmask) for each coordinate observe that using a separate mask token didn't gave any performance gain. Similarly, passing the learnable masked token through the encoder didn't 't lead to any performance gain, despite the fact that this increases the compute requirement too.
the FMNIST and CovType datasets. In both cases, interestingly we
Lastly, in the Appendix A.I inference time (Section 4.1 compared to averaging. We also show that downstream accuracy with MET representations not sensitive the depth of MLP over the representations.
study the effect of flattening (concatenating) the representations
CONCLUSION AND LIMITATIONS
In this paper, proposed a reconstruction based approach, MET, for self-supervised representa- tion learning on tabular datasets. MET is based on the hypothesis that there exists latent (i.e.. underlying/unooberved)) graphical model which captures the relations between different coordinates of the tabular data. Conceptually, through experiments on an interesting toy setup. we show that MET is indeed able learn the underlying graphical model, and hence gives much higher down- stream classification accuracy. Practically, we show that over a suite of nine standard and real-world datasets, MET substantially outperforms a set of strong and diverse baselines, including other SOTA tabular-SSL methods as well SOTA supervised learning algorithms such as gradient boosted deci- sion trees.
While reconstruction-basee based SSL has been shown to learn powerful representations across various domains such as text Devlin et al. (2019). vision He et al. (202T) as well as tabular (this paper), a thorough understanding of why reconstruction loss capable of learning such good representations is still missing in literature. In this paper, we take a step in this direction by connecting masking- based reconstruction with the long line of work on learning graphical models. Our results motivate further investigation into the relation between graphical models and downstream classification tasks, as well as the efficacy of other graphical model learning approaches (e.g., those based on information theory) for self-supervised representation learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
UCI adult income dataset. https://archive www.scc............... hcc..eeeb/mmm/aactsstt///aaaa
Francis Bach and Michael Jordan. Learning graphical models with mercer kernels. Advances in Neural Information Processing Systems, 15, 2002.
David Badía, Alberto Ruiz, Antonio Girona, Clara Martí, José Casanova, Paloma Ibarra, and Raquel Zufiaurre. The influence of elevation on soil properties and forest litter in the siliceous moncayo massif, sw europe. Journal of Mountain Science, 13(12):2155-2169 2016.
Alexei Baevski, Yuhao Zhou, Abdeirahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural In- formation Processing Systems, volume 33, pp- 12449-12460. Curran Associates, neurips po/ppaper/220//ile/
H. Larochelle,
Inc., 2020. URL ptttp:://prrceeeeee htw.o//1900022111100000000000000 Papp Papprr.--ppppp...
Guy Bresler, Elchanan Mossel, and Allan Sly. Reconstruction of markov random fields from sam- ples: Some observations and algorithms. In Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, pp. 343-356. Springer, 2008.
Filipe X Catry, Francisco C Rego, Fernando L Bação, and Francisco Moreira. Modeling and map- ping wildfire ignition risk in portugal. International Journal of Wildland Fire, 188199921, 2009.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 699-708, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference Machine Learning, pp. 1597-1607, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750- 15758, June 2021.
CKCN Chow and Cong Liu. Approximating discrete probability distributions with dependence trees. IEEE transactions on Information Theory, 1403):444444:4444
UCI covtype dataset. ref bef-datanews. 10/
Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. pp. 4171-4186 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 1/.18863//1/N11114223
Francisco Rego Francisco Moreira, Filipe X. Catry and Fernando Bacao. Size-dependent pattern of wildfire ignitions in portugal: when do ignitions turn into big fires?, 2010.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 189-):119911233 2001. ISSN 00905364. URL http:///
cestor.orge
stable/22 2699986
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked au- toencoders are scalable vision learners. CoRR, abs/21 0ts/2111.00637 11.06777, 2021. URL https: ///arxiv. org/abs//20 211 06377
10

---

Under review as a conference paper at ICLR 2023
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units, 2021. URL Ott.://ppss//arr.....o///////222
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin, Tabtransformer: Tabular data mod- 06678
eling using contextual embeddings, 2020. URL.nttpsss/arziiv.orr////
Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, and Xiangang Li. Improving transformer-bssed speech recognition using unsupervised pre-training, 2019. URL https:// 1910. 09932
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial sulf-supervised contrastive learning.
Advances in Neural Information Processing Systems, 33:2983- 2994, 2020.
Adam Klivans and Raghu Meka. Learning graphical models using multiplicative weights. 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pp. 343-354. IEEE, 2017.
PaperswithCode Criteo Leaderboard. Leader board for Critco. https:/ httpppppppssttosee...ppp com/ sot poota/click thhta//cccc-throughhhh rate cerediction-on--on cessed 22- 22-September-2022]
2022. [Online; ac-
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng. Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang. Mst: Masked self-supervised transformer for visual representation, 2021. URL /ttpps://arxvv.orrtoe
2106 05656
Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms gradient descent. Advances in neural information processing systems, 12, 1999.
Ahmet Mert and Berna Yalcinkaya. Relationship between some wild mammals and forest structural diversity parameters. Journal of Environmental Biology, 38:879-888 884, 10 2017. doi: 10.22438/ jeb/38/5(SI/GM--02.
Praneeth Netrapalli, Siddhartha Banerjee, Sujay Sanghavi, and Sanjay Shakkottai. Greedy learn- ing of markov network structure. In 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1295-1302 IEEE, 2010.
UCI obesity dataset. http:://sttha...oo/oool//////lee
DODS-arrhythnin
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Neurips, 2008
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3553 3545,,35,,,0 2020.
Yuge Shi, N. Siddharth, Philip H. S. Tom, and Adam R. Kosiorek. Adversarial masking for self- supervised learning, 2022. URL http::s://aari...oorra////2.00..
OODS thyroid dataset.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 1rX1v:1805.12152 2018.
Talip Ucar, Ehsan Hajiramezanali. and Lindsay Edwards. Subtab: Subsetting features of tabular data for sulf-super representation learning. Advances Neural Information Processing Systems, 34, 2021.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. httlhtttss:::pooeeeeeee wew.inng...inneeeeeeee cct./aaper/2217//11ele
pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention you need. Advances neural informa- fion processing systems, 30, 2017b.
Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le Towards domain- agmostic contrastive learning. CoRR, abs/2011.04419 2020. URL https://axxiv.. org/
04419
Zhiqiang Wang. Qingyun She, and Junlin Zhang. Masknet: Introducing feature-wise multiplication to ctr ranking models by instance-guide mask. ArXiv, abs/2102.07619, 2021.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the suc- cess of self-and semi-supervised learning to tabular domain. Advances Neural Information Processing Systems, 33:11033-111 1043, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021. URL /tt.https://rrxii.oor 0003
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
ABLATION STUDY
In this section, we discuss various ablations and discuss the sensitivity of MET various hyper- parameters.
A.1.1 MASKING RATIO
We discussed the variation in performance of MET as we vary the fraction of input tokens masked (?) Section 5.4 Our general observation is that MET learns the best representations (in terms of downstream classification accuracy) when the masking ratio is high (50%-70%) Figure 4 shows the variation of downstream classification accuracy as the masking ratio varied [30, 50, 70, 80, 90) on four different datasets. On FMNIST, MNIST and Adult-Income a masking ratio of 70% works the best whereas on CovType a masking ratio of 50% gives the best results.
Fashion MNIST
MNIST
91
99.0
MET Accuracy
MET Accuracy
90
98.8
89
98.6
98.4
88
40
60
80
40
60
80
Masking Percentage
Masking Percentage
Forest CoverType
Adult Income
86.25
MET Accuracy
MET Accuracy
86.00
74.0
85.75
73.5
85.50
73.0
85.25
60
80
40
Masking Percentage
Masking Percentage
Figure study variation downstream accuracy as masking ratio varied in MET, four tabular classification datasets. We observe a high masking ratio (50%-70%) generally works best.
A.1.2 CONCATENATED VS AVERAGED EMBEDDINGS
explained that at inference time, for input x, the learnt representations given by O To.. (1)() E Rm where m ddembed- Note that we made the choice of the learnt representations for each coordinate E [d] the encoder output. Here we perform an ablation by averaging the representations (instead of concatenation) across the coordinates to get the final learnt representation. We try both these approaches on FMNIST and CovType. For both datasets, concatenation significantly outperforms averaging. For CovType, concatenation and averaging obtain 76.71% and 61.87% accuracies respectively and for FMNIST, they obtain 91 .8%% and 88.64% respectively.
In Section
concatenating/latten
13

---

Under review as a conference paper at ICLR 2023
Table 3: Downstream task accuracy with varying number of hidden layers in the downstream MLP classifier with the learnt representation being fixed. We observe that for most datasets or 3 hidden layers achieve the best performance, with relatively low sensitivity to the choice of the number of hidden layers.
Num hidden layers Accuracy
2
3
87.16%
90.59%
90.86%
90.82%
90.65%
90.62%
A.1.3 ACCURACY VS DEPTH OF MLP OVER LEARNT REPRESENTATIONS
Here we analyze how sensitive the downstream task performance is to variation in the depth of the MLP over the learnt representations. We fix the learnt representations for the FMNIST dataset and vary the number of hidden layers in the MLP. Table 3 shows the variation of accuracy with the hidden layers. We find that for most of the datasets, a depth of two to three hidden layers works the best. Although, on the same note, to the choice of the number of hidden layers. For example, on FMNIST the best accuracy is 90.86% with 2 hidden layers whereas with 5 hidden layers, the accuracy is 90.62%.
mention that the downstream accuracy is not much sensitive
A.2 MEAN AND STANDARD DEVIATION FOR DOWNSTREAM CLASSIFICATION RESULTS
In Section 5.3
we compared the downstream classification accuracy for various baselines and MET over nine tabular datasets (Table I Table 2). For completeness, we add the results with mean and standard deviations, computed over five independent runs of each algorithm in Table 4 and Table
Table Downstream classification accuracies over four commonly used tabular benchmarks. MET uses adversarial training + masking for reconstruction-bet -aaioon----ddddee- self-supervised learning, whereas MET-S is an ablation where only masking is used. We observe that MET outperforms the baselines over all the datasets, establishing itself as new state-of-the-art.
Type
Methods
FMNIST
CIFARIO
MNIST
CovType
MLP
87.57 = 0.13
16.47 1:447:000 0.23
96.98 = 0.1
65.45 + 0.09
Supervised
RF
87.19 = 0.09
36.75
+ 0.17
97.62 + 0.18
64.94 + 0.12
Baselines
GBDT
88.71
1 0.07
45.7 0.27
100 1 0.0
72.96 1 0.111
RF-G
89,84
0.08
29.28
1 0.16
97.63 1 0.03
71.53 1 0.06
METER
88.81
0.12
28.97
1: 0.08
97.43:0:00 0.02
69.68 1 0.07
VIME
80.36
0.02
34.00
0.5
95.74
1: 0.03
62.78 1 0.02
SSL
DACL+
81.38
0.03
39.7
1 0.06
91.35
1 0.075
64.17 1 0.12
Basclines
Sub
87.58
= 0.03
39.32 0.04
98.31
+ 0.06
42,36 - 0.03
MET
MET-S
90,90
0.06
47.96
0.1
98.98 = 0.05
74.13 = 0.04
(Our Method)
MET
91.68 + 0.08
47.92 + 0.13
99.17 + 0.04
76,68 + 0.12
Table 5: Downstream accuracy and AUROC scores on five popular tabular datasets for binary classification, for MET and other baselines. MET outperforms all the baselines both accuracy and AUROC metries.
Datasets
Metrie
MI.P
GBDT
RF-G
MET-R
DACL
VIME
SobTTbb
MRT
Obestty
annc
52.3 0.12
64.36
0 0.07
64.4
0.05
58.30 0.17
54.45
0.08
5322 0:18
58.
62.34 1
61.18 + 0.07
57.27 - 0.21
0.17
64.92
0.03
0.06
74.36
71.54
Incomse
Acc.
84.35
0.11
846 0.2
16,01
+ 0.06
0.13
75,50
0044
83.99
+ 0.44
54.46 - 0.05
84.41
+ 0.06
16.21 + 0.05
89.39
0.32
+ 0.08
9009
83.48
89,01
+
0.07
88,95
93.85
0.33
Criteo
aurce
Acc.
70.3
1128
0.17
1 0.3
71.09
77.57
0.1
0.05
76.77
72.03
0.04
0.00
80.30
1.60
0.16
0.08
79.17
73.57=
0.17
0.12
15.30
6932
1 0.27
000
63.78
74.28
- 0.39
1: 0.13
76.57
73.02
- 0.05
1: 0.00
78,49
86.17 1: 0.2
6005
Arrby-
Ace.
59.7
1 0002
68.18
0.02
69.79
1
0066
1 0.05
31.87
1: 0.1
57.81
1: 0.7
50.06
3: 0.04
60.1 1: 0.1
$1.35
6.12
thunia
aurce
72.33
006
90.63
92.19
74
0.12
58.36
0.52
69.23
+ 0.98
67.03 + 0.27
69.97
98.75
Thyroid
Acc.
50:0:00
94.94
1: 0.1
96.44
2: 0.07
30
1 0.0
57.42
0.37
60.03 1 0.05
66.1
1: 0.19
59.9 1:: 0.16
98.1
1: 0.08
623
99.62
0.03
99.34
52.65
0.13
82.03
0.26
86.63
0.1
94.87 1: 0.03
88,93 0.12
99.81 0.09
A.3 CORRELATION ACROSS FEATURES FOR COVTYPE
In Section 5.4 coordinates on the CovType dataset. Figure 5 shows the cosine similarity of the learnt positional encoding of feature/coordinate with the learnt positional encodings of other features/coordinats.. Note that we show the average similarity across different runs of the MET on Cov to reduce the variance. We observe that the feature "Flevation's positional encodings are maximally correlated to
we discussed that MET is indeed able to capture meaningful relation among the
14

---

Under review as a conference paper at ICLR 2023
Cosine Similarity Score
Cosine Similarity Score
Elevation
Horiz. dist. to Roadways
0.5
0.0
Aspect
Slope
Horiz dist. Hydrology
Vert. dist. to Hydrology
Horiz. dist. to Roa Roadways
Hillshade@9am
Hillshade@
Hilishade@ 3pm
Horiz, fire points
Wilderness area
Soil type
0.0
Elevation
Aspect
Slope
Horiz dist. to Hydrology
Vert. dist. to Hydrology
@illshaee@ Noon
Hillshadee@ 3pmm
Horiz. dist. fire points
Wilderness area
Soil type
Hillshade@
Figure Comparing the cosine similarity of the learnt positional encoding feature/coordinate with the learnt positional encodings of other features/coordinates. The maximally correlated features are indeed mean- ingful.
the encodings of the feature "Soil Type", as has been corroborated in Badía et al. (2016) in the forest science literature. Similarly, with "Distance to nearest ignition point" as has been talked about Francisco Moreira & Bacao 2010
observe that "Distance to roadways" has the most correlation
A.4 RESULTS FOR TOY DATASET
we discussed results on toy dataset from a linear graphical model. We observed that MET is able to capture the graphical model, shown by the high correlation between the positional embeddings for the consecutive coordinates in Figure 2 Consequently, MET is able to perform the downstream task with high accuracy. For completeness, share the accuracy and AUROC numbers for all the baselines Table
In Section 4.3
Table 6: We compare the performance of MET the toy dataset from linear graphical model with all the baselines. MET is able capture linear graphical model as shown correlation plot Figure and outperforms all the baselines both accuracy and AUROC.
Type
Methods
Accuracy
AUROC
MLP
68.65
75.4
RF
82.25
90.16
Supervised Baseline
GBDT
88.4
95.35
RF-G
66.6
73.62
MET-R
73.48
80.29
DACL+
74.65
82.92
Self-Supervised Methods
VIME
62.65
71.19
SubTab
63.28
73.52
Our Method
MET
96.7
99.61
A.5 DATASETS
M It is 10-class classification handwritten digit recognition dataset. Input images of size 28 x 28 are flattened to get a 784-dimensional vector as the input. The flattened dataset is then permuted (i.e. the pixel/coordinate order shuffled) to get the final dataset. A split of 54, 000 samples as train set, 6000 samples as validation set and 10, 000 samples as test set is used.
consists of 28 x 28 dimensional images, and is a 10-class classification dataset. The data is flattened get 784 coordinates in tabular form and then shuffled
FMNIST:
The data normalized and shuffled for all datascts.
15

---

Under review as a conference paper at ICLR 2023
mentioned above. A split of 54, 000 samples as train set, 6000 samples validation set and 10, 000 samples as test set is used.
is a colored dataset with images of size 32 x 32 x 3 belonging to ten different classes. We flatten it to get 3072 coordinates in tabular form and then shuffle mentioned previously. A split of 45, 000 samples as train set, 5000 samples as validation set and 10, 000 samples as test set is used.
CIFAR-10:
Forest Cover Type: The covtype dataset is 7 class classification UCI dataset where the task is predict the forest cover type. The dataset has in total 54 features, out of which there is a one hot vector of length 4 representing the wilderness. We replace this one hot vector with a single integer denoting the index of non-zero entry. Similarly there is a one hot vector of length 40 for soil type, which we again replace with a single integer denoting the index of the non-zero entry. In summary, we process the 54-dimensional dataset a much harder 12-dimensional dataset since the categorical features are now represented as integers instead of one-hot representations" A split of 11,340 samples as train set, 3780 samples as validation set and 565, 892 samples as test set is used.
The adult income dataset is a UCI dataset where the prediction task is to determine whether person makes over $50K year based on the census data. It consists of mix of six continuous and eight categorical fields. Similar to CoverType dataset, we use integers instead of one-hot representation for the categorical features and get 14 features in tabular form. A split of 27, 146 samples as train set, 3016 samples as validation set and 15, 060 samples as test set is used.
Income:
The obesity dataset has a task of predicting obese vs non-obese based on the human gut metagenomic sample. The input has 465 features. The features normalized and shuffled before feeding it to the model. It is relatively small dataset of 253 samples, hence we use a 80-10-10 train-val-test split with 10-fold gross-validation for evaluation purposes.
Obesity:
Criteo: Display advertising a billion-dollarr industry and an important use case of machine learning. Criteo consists of one-week data from CriteoLabs for toock-through-rate(d
prediction summing up 45M samples with 39 features each. Out of the 39 features, 26 are categorical, some of which have as many as 5M distinct values in the form of anonymized string and the other 13 are real-valued fields. A split of 34, 380, 462 samples as train set, 3, 438, 047 samples as validation set and 9, 168, 124 samples test set is used.
Arrhythmia: It is standard outlier detection (one class classification) benchmark for tabular dataset, available on OODS -rrh-tarhytmmial repository, which we convert into a binary classification one. The dataset has set of inlier samples and set of outlier samples which much less in number compared to the inlier samples. We split the outlier samples equally between the train and the test set. We keep 1 1 ratio between the inlier outlier classes in the test and use the remaining inlier samples in the training set. A split of 348 samples as train set, 38 samples as validation set and 66 samples as test set used.
Again, it is standard outlier detection benchmark for tabular dataset, which we convert into a binary classification one. It is available on the thyroid (atasse] We follow the same procedure the Arrhythmia dataset above for splitting the inlier and outlier samples into train-test set. A split of 6000 samples as train set, 666 samples as validation set and 534 samples as test set is used.
Thyroid:
A.6 HYPER-PARAMENE
In this section, we share the hyper- parameters of MET for replicating the results on all the nine tabular datasets. Note that Encoder Depth refers to the number of transformer layers the en- coder and Decoder Depth refers to the number of transformer layers in the decoder. Adversarial
Consequently, our accuracy numbers are not directly comparable standard results on this dataset.
16

---

Under review as a conference paper at ICLR 2023
Learning Rate (Irav) refers to the step size for gradient ascent in adversarial training and Ir refers to the learning rate for gradient descent on reconstruction loss. We perform grid-search for em- bedding dimension((emtee)) and feed-forward dimension(fw) in {64, 100, 128), number of heads in the transformer architecture in {1,2, 3), encoder and decoder depth {1,3, 6), learning rate(Ir) in 1e 1e-3 1e-4, 1e-"). masking ratio(y) in (30, 50, 80,80,00000 90), adversarial steps(advsepp)) in 2,2,4, radius of L2-norm ball(e) in {2, 6, 10, 12, 14) and step size for gradient ascent in (Iradv) in {0.1, 0.01). The optimal set of parpee-aaameters for various datasets obtained mentioned in Table and Note that all the ablation studies are conducted using these parameters.
Table 7: parameerssseeerss for replicating the results with MET.
Embedding
Feed-ford forward
Number of Encoder
Decoder Depth
Masking
lr
Dimension(demheen
Dimension
Heads
Depth
Ratio(7)
FMNIST
64
64
6
le
0.70
CIFARIO 10
100
64
3
le
0.70
MNIST
64
64
6
le
0.70
CovType
100
64
1
le
0.50
Obesity
64
100
6
le
0.70
Income
64
64
3
le-3
0.80
Criteo
64
100
6
le
0.50
Arrhythmia
64
64
1
le=3
0.30
Thyroid
100
128
2
3
6
le" 3
0.80
Table 8: Hyper-parametee parameters for replicating the results with MET.
Adversarial
1.2
Norm Ball Adversarial
Steps(adv seps)
Radius(c)
(Iraay)
FMNIST
le
CIFAR 10
3
14
1e-2
MNIST
2
12
le
CovType
5
le
Obesity
5
1e
Income
le
Arrhythmia
12
1e-2
Thyroid
2
le
A.7 COSINE-SIMILARITY AND LATENT GRAPH
In this section, we show how cosine-similarity is co-related with the ability to capture the latent graph structure. The experimental setup tries reconstruct given coordinate only using other coordinates. Given a set of coordinates z;'s we fix a coordinate = æ; that needs to be reconstructed. Now by using the other of coordinates Zyt=i one at time, reconstruct and measure the mean-absolute error across all samples of the toy datase 4.3 As shown in Figures 6 7 and & we observe that features with high cosine similarity scores show lower mean absolute error while reconstruction. Note that we min-max normalize the cosine similarity score and the mean absolute errors for plotting them on the same scale.
17

---

Under review as a conference paper at ICLR 2023
Figure 6: Higher cosine similarity leads to better reconstruction of features and hence able to capture the latent graph structure. The following graphs are coordinates 1,2,3,4.
Figure 7: Higher cosine similarity score leads to better reconstruction of features and hence able to capture the latent graph structure. The following graphs are for coordinates 5,6,7,8.
Figure Higher cosine similarity score leads reconstruction features and hence to capture the latent graph structure. The following graphs are coordinates 9,10.
18


Reviews:
[{"review_id": "ccMbm3kXNw", "reviewer": "Reviewer_F52Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new model for self-supervised learning on tabular data which is based on transformers and auto-encoding via masked reconstruction. To encode feature types, the model uses learnable \"positional\" embeddings which can also capture relations between features. The method is evaluated on a variety of tabular and non-tabular datasets where it shows promising results.", "summary_of_the_review": "The paper explores interesting ideas for SSL on tabular data and shows promising results. However, short-comings in terms of clarity, reproducibility would require attention prior to publication.", "strength_and_weaknesses": "The paper addresses an interesting problem, i.e., how to extend the success of self-supervised representation learning to tabular data. The proposed approach is reasonable and seems to provide promising experimental gains. The experimental evaluation is performed on a wide range of datasets what surpasses prior work in this area. Due to the use of learnable positional embeddings, the methods gains also some form of interpretability (i.e., similarity relations between features) what is a nice additional benefit of the approach.\n\nHowever, in its current form, the paper has some shortcomings that would require further attention, most importantly with regard to clarity, novelty, and reproducibility. I will detail these aspects further in the next section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GN1BwV6kVfk", "reviewer": "Reviewer_wvLH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission contributes a masking based reconstruction approach with a transformer architecture for self-supervised learning based on quadratic reconstruction error with adverserial perturbation of the input.\n\nThe method is benchmarked on a set of classic datasets where it is shown to predict better than standard methods including gradient boosted trees.", "summary_of_the_review": "This manuscript contributes an interesting idea, but it is not clear that the empirical evidence is entirely robust.\n", "strength_and_weaknesses": "Transformers are clearly powerful architectures and it is interesting to mix them with masked loss for tabular data.\n\nThe datasets benchmarked in table 1 are not tabular data (4 out of 5 are image data). Grinsztajn et al \"Why do tree-based models still outperform deep learning on typical tabular data?\" give a clear definition of properties of tabular data and a list of openly-available data to use.\n\nWhat is the train-test-validation strategy used? Reading the code submitted withthe manuscript seems to reveal that the left-out data set where the performance of the algorithm is evaluated was also used to set the hyper-parameters.\n\nHow are categorical columns dealt with in the reconstruction loss?\n\nHow are categorical inputs dealt with in the perturbation of the input in eq 6?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-nXeRyVLZGu", "reviewer": "Reviewer_twHD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised learning (SSL) framework for tabular data, Masked Encoding for Tabular Data (MET). MET employs masked auto-encoding with Transformer architectures. This paper shows that MET outperforms existing SSL methods for tabular data and classical machine learning approaches. In addition, this paper empirically verifies that the learned representations well capture the relationship between features.\n", "summary_of_the_review": "Although the proposed method is not new, the empirical results are strong. Since the tabular domain is yet under-explored, I think this paper's contribution is somewhat meaningful in this field. Hence, I vote for weak accept.\n", "strength_and_weaknesses": "Strengths\n- The proposed method, MET, outperforms recent SSL methods (VIME, DACL, SubTab) and classical ML approaches (RF, GBDT, RF-G) by a large margin.\n- This paper empirically verifies that the learned positional embeddings well capture the relationship between features; in other words, MET is highly interpretable.\n\nWeaknesses\n- I feel the lack of methodological novelty in this paper. I think this paper is a simple application of the existing MAE approach into the tabular domain with no modification.\n- Most multi-classification benchmarks are vision datasets, not tabular: FMNIST, CIFAR10, and MNIST. Other tabular benchmarks should be tested rather than vision datasets.\n- How to select the hyperparameters in Table 7? Which split (train, validation, test) is used for the selection?\n- How to handle categorical features? This should be described in the main manuscript.\n  - I found that this paper uses categorical features as integers instead of one-hot vectors for CovType. Why use this approach?\n- The transformer architecture is necessary? Since the number of input features is constant, one can use MLPs (or other architectures) as VIME did. I cannot find why the transformer architecture is important.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject