SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
Authors: Talip Ucar
Ehsan Hajiramezanali
Lindsay Edwards
Venue: NeurIPS 2021 Poster (2021)
Keywords: representation learning
self-supervised learning
contrastive learning
tabular data
multi-view learning

Abstract: Self-supervised learning has been shown to be very effective in learning useful representations, and yet much of the success is achieved in data types such as images, audio, and text. The success is mainly enabled by taking advantage of spatial, temporal, or semantic structure in the data through augmentation. However, such structure may not exist in tabular datasets commonly used in fields such as healthcare, making it difficult to design an effective augmentation method, and hindering a similar progress in tabular data setting. In this paper, we introduce a new framework, Subsetting features of Tabular data (SubTab), that turns the task of learning from tabular data into a multi-view representation learning problem by dividing the input features to multiple subsets. We argue that reconstructing the data from the subset of its features rather than its corrupted version in an autoencoder setting can better capture its underlying latent representation. In this framework, the joint representation can be expressed as the aggregate of latent variables of the subsets at test time, which we refer to as collaborative inference. Our experiments show that the SubTab achieves the state of the art (SOTA) performance of 98.31% on MNIST in tabular setting, on par with CNN-based SOTA models, and surpasses existing baselines on three other real-world datasets by a significant margin.

SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
Talip Uçar, Ehsan Hajiramezanali, Lindsay Edwards
Respiratory and Immunology, R&D, AstraZeneca ehsan haj iramezanal,, lindsay edwards
{talip. ucar, (talip.ucar,
Abstract
Self-supervised learning has been shown to be very effective in learning useful representations, and yet much of the success is achieved in data types such as images, audio, and text. The success is mainly enabled by taking advantage of spatial, temporal, or semantic structure in the data through augmentation. However, such structure may not exist in tabular datasets commonly used in fields such healthcare, making it difficult to design an effective augmentation method, and hindering similar progress in tabular data setting. In this paper, introduce new framework, Subsetting features of Tabular data (SubTab), that turns the task of learning from tabular data into a multi-view representation learning problem by dividing the input features to multiple subsets. We argue that reconstructing the data from the subset of its features rather than its corrupted version in autoencoder setting can better capture its underlying latent representation, In this framework, the joint representation can be expressed as the aggregate of latent variables of the subsets at test time, which we refer to as collaborative inference. Our experiments show that the SubTab achieves the state of the art (SOTA) performance of 98.3 % on MNIST in tabular setting, on par with CNN-based SOTA models, and surpasses existing baselines on three other real-world datasets by a significant margin.
Introduction
In recent years, the sule-suue learning has successfully been used learn meaningful represen- tations of the data in natural language processing (34
10 9 A similar success has 13 8 This progress is mainly enabled by taking advantage of spatial, semantic, or temporal structure in the data through data augmentation [7]. pretext task generation [11] and using inductive biases through architectural choices (e.g. CNN for images). However, these methods can be less effective in the lack of such structures and biases in the tabular data commonly used in many fields such as healthcare, advertisement, finance, and law. And some augmentation methods such cropping, rotation, color transformation etc. are domain specific, and not suitable for tabular setting. The difficulty in designing similarly effective methods tailored for tabular data is one of the reasons why self-supervised learning under-studied in this domain [46).
11
288
been achieved image and audio domains 15 37
The most common approach in tabular data is to corrupt data through adding noise 4 An autoen- coder maps corrupted examples of data to a latent space, from which it maps back to uncorrupted data. Through this process, it learns representation robust to the noise in the input. This approach may be as effective since it treats all features equally as if features are equally informative. However, perturbing uninformative features may not result in the intended goal of the corruption. A recent work takes advantage of self-supervised learning in tabular data setting by introducing a pretext task 46), in which de-noising autoencoder with classifier attached to representation layer is trained on
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

X
ii) Reconstruction Loss
X 2 ++ x1
X or 82 +
overlep
overap
X or 3g
E
iii) Contrastive & Distance loss
X2
hs hz
21 Z2, Z3.
Z2 X3
Dividing features into subsets (Analogous to cropping images)
h2
81 22 Z3
Rs hg
SubTab framework: i) Dividing the features into subsets (similar feature bagging, or cropping images), ii) Reconstruction of either subsets of features (Zg,ūz,0gg) or complete feature space (X.,.),.X3), which are used to compute reconstruction loss. iii) Generating projections used compute contrastive and distance loss. Encoder, D Decoder, G Projection.
Figure I:
corrupted data. The classifier's task is to predict the location of corrupted features. However, this framework still relies noisy data in the input. Additionally, training a classifier imbalanced binary mask for a high nigh-dimensional data may not be ideal to learn meaningful representations.
In this work, we turn the problem of learning representation from hingle-view of the data into the one learnt from its multiple views by dividing the features into subsets, akin to cropping in image domain or feature bagging in ensemble learning, generate different views of the data. Each subset can be considered a different view. forces the encoder to learn better representation than the ones learned through the existing methods such as adding noise. We train our model a self-supervised setting and evaluate it on downstream tasks such classification, and clustering. We five different datasets; MNIST in tabular format, the cancer genome atlas (TCGA) 42 human gut metagen-one omic samples of obesity cohorts (Obesity) 336 26). UCI adult income (Income) 24). and UCI BlogFeedback (Blog)
show that reconstructing data from the subset of its features
SubTab can: i) construct a better representation by using the aggregate of the representation of the subsets, process that we refer as collaborative inference ii) discover the regions of informative features by measuring predictive power of each subset, which useful especially high-dimensional data iii) do training and inference in the presence of missing features by ignoring corresponding subsets and iv) use smaller models by reducing input dimension, making less prone to overfitting.
2 Method
The augmentation methods such adding noise, rotation, cropping etc. commonly used in image domain. Among them, the cropping is shown to be the most effective technique [7]. Inspired from this insight, propose subsetting features of tabular data.
Figure I presents SubTab framework, which we have an encoder (E), a decoder (D), and an optional projection (G). For the purpose of this paper, will refer h as latent, representation, projection, E. and X as the reconstruction of subset, and whole data respectively. Small letters are associated with subsets while capital latters are associated the whole set of features. Moreover, throughout this work, when we say that representation is "good", we refer to its performance in classification task using a linear model.
In SubTab framework, we divide tabular data to multiple subsets. Neighbouring subsets can have overlapping regions, defined as percentage dimension of subset. Each of the subsets fed to the same encoder (i.e. parameter sharing) to get their corresponding latent representation. A shared decoder used reconstruct either the subset fed to the encoder, or full tabular data (i.e. reconstructing all features from the subset of features). We chose the latter in experiments since

---

Neighbor columns
ftg
PS
ht
hz
NS
Random columns
E
h
CL
RL
Aggregation (pooling)
DL
X3
Random features
(a) Push-Pull forces
(b) Feature selection
(c) Representation at test time
Figure 2: a) Push-Pull forces applied by each loss. PS / NS: Positive/Negative sample; CL/RL/DL: Contrastive, Reconstruction, Distance losses b) Column feature selection strategies for adding noise to each subset. Top: Selecting a block of neighbouring columns; Middle; Selecting columns randomly; Bottom: Selecting random features per row c) Latent variables from each subset is aggregated at test time. The mean (default), sum, max, or min aggregation can be used.
it is more effective in learning good representations. We should also note that, in the latter case, the autoencoder cannot learn the identity, eliminating the constraint the dimension of the bottleneck (i.e. representation). We compute one reconstruction loss term per subset.
optionally add contrastive loss to objective by using all combination of pairs of projections from all subsets. For example, given three subsets as Figure | there are three Iाī = 3 For four subsets, would pairs of combination, and so on. We can add one more loss term, referred as distance loss, to reduce the distance between the pairs of projections of the subsets by using loss function such three loss terms apply a pulling force on positive samples while contrastive loss also applies a push force between positive and negative samples shown in Figure 2
Moreover,
combinations of two: (%) 2 =
squared error (MSE). All
Once the dataset is divided into subsets data preparation step, a process that is similar to feature bagging in ensemble learning, their location is fixed. Thus, don't 't change the relative order of features in a subset during training since standard neural network architectures are not permutation invariant. This is to ensure that same features fed to the same input units of neural network. However, our method can be extended to permutation invariant setting as a next step.
2.1 Strategies for adding noise
Our framework complementary to other augmentation techniques used in tabular data setting. Thus, we experimented with adding noise to randomly selected entries in each subset by using three types of noise: i) adding Gaussian noise, ((0 o²). ii) overwriting the value of selected entry with another value randomly sampled from the same column, referred as swap-noise, iii) zeroing-out randomly selected entries, referred as zero-out noise.
Moreover, we use three different strategies when selecting the features to add noise to, shown in Figureph i) random block of neighboring columns (NC), ii) random columns (RC) iii) random features per each sample (RF). To add noise, we create binomial mask. m. and a noise matrix, with same shape as the subset, in which the entries of the mask assigned to I with probability p. and to 0 otherwise. The corrupted version, 1ıc, of subset is generated as following:
3ic = (1 (1mm) + + m O
(1)
2.2 Training
Our objective function is:
C + LLa
where L,, Lr, L. and La are total, reconstruction, contrastive, and distance losses, respectively. i) Reconstruction loss: Given subset, denoted by Tk. reconstruct either the same subset, Th the entire feature space Xς. Then, compute the reconstruction loss for ttth subset by

---

computing mean squared error using either (æk: Tge,, or (X,X,) pair as shown in Figure chose the latter since it was more effective. Overall reconstruction loss:
|
We
E x (x(1-Xx ) N
1
C
Sk. where Sk
(3)
where K is the total number of subsets, N is the size of the batch, Sy is the reconstruction loss for ht subset, and L, is the average of reconstruction loss over all subsets.
ii) Contrastive loss: If the dataset rich in the number of classes such that chances of sampling negative samples are high, we can use a projection network (G) to get projections, z's, of rep- resentations, h's. Samples at the same rows of two subsets, ø1 and 22, can be considered as positive pairs while remaining rows in the subsets can be considered as negative to those sam- ples.This allows us to compute the contrastive loss for each pair of projections using a loss function such as the normalized
menperature-scales cross entropy loss (NT-Xent) 77. For three subsets, [1, I2, 23 ). we can compute such a loss for every pair {za: 2b} of total three pairs from the set S {[21, 22), {a1, 23), 22 }3)}. Overall contrastive loss is:
N I(z (+) ,0 20 + 1(z0 (1) za (0)
L.
1
Σ
p(za: zb), where p(za: zb)
(4)
J
2N
exp(sim((z (8) 26 (i) TT)
I(za (6) Zb (8)) -log
(5)
Σ km K exp(sim(za (1) 2b (
where J is the total number of pairs in set S, p(za; 2b) is total contrastive loss for a pair of pro- jection {za, } 1zz, (*), 22 (1) is the loss function for corresponding positive pairs of examples (za (i) 26 (+) in subsets {=a, }b), and Le is the average of contrastive loss over all pairs.
iii) Distance loss: We can also add mean-squared error (MSE) loss for pairs of projections of subsets since the corresponding samples in subsets should be close to each other. Hence, we can compute an overall MSE loss as:
N
La Σ
p(za; 2b). where p(za; 2b) = [za.En]ES
1 N
2 )
(6)
The pseudocode of algorithm can be found Algorithm[Tin in the Appendix. We should note that both L. and La in equation (2) are optional, and we used them only in some experiments.
2.3 Test time
At test time, feed the subsets of test set to the encoder, and get the aggregate of the representations of all available subsets as shown in Figure 2g Please note that we can use mean, sum, min, max, or any other aggregation method get joint representation, which is analogous to pooling in Computer Vision, or the aggregation of neighbouring nodes in graph convolutional networks [23]. We used mean aggregation in all our experiments, but did compare different aggregation methods in Appendix
F4 Our experiments show that we can use the representations of only one, or few subsets and still achieve good performance at test time. For example, we could use only h1. or aggregate of h and h2 rather than aggregating over all subsets (h1, f22, h2) in Figure 2g This allows the model to infer from the data even the presence of missing features, in which case we can ignore the subset with missing features. We can also design an aggregation function that computes weighted mean of the representations of subsets since some subsets might be more informative than others:
K
K
h Σ
mh hg. and Z
where K is number of subsets, and 7) is the weight for kis subset. 17 can be a learnable parameter in semi-supervised, or supervised setting by using an attention mechanism. We can also use ID convolution in equation (7) by treating representations of subsets as separate channels during training. We left these ideas as future work and used the mean aggregation (i.e. ч 1) throughout our experiments, unless explicitly stated. A comparison of different aggregation methods can be found in Table A3 in the Appendix.

---

3 Experiments
We conducted various experiments on diverse set of tabular datasets including MNIST 27 tabular format, the cancer genome atlas (TCGA) 442, human gut metagen-omic samples of obesity cohorts (Obesity) [6, 26), UCI adult income (Income) (24], and UCI BlogFeedbacc (Blog) to demonstrate the effectiveness of the SubTab framework. We compare our method to autoencoder baseline with and without dropout, other self- superviee methods such as VIME-self [46], Denoising Autoencoder (DAE) [43]. and Context Encoder (CAE) [39] as well sully-supervvsed models such logistic regression, random forest, and XGBoost [6]. For each dataset, once we decided on a particular autoencoder architecture, we used it for all models compared (i.e. VIME-self, DAE, CAE, and our model). We tried both ReLU and leakyReI.U as activation functions for all, and both performed equally well. The code for SubTab is provided The summary of model architectures and hyper- parameters are in Table Al in the Appendix. We should note that we ran more experiments using; i) Synthetic datasets and ii) OpenML-CC18 datasets [2] in Appendix G and respectively.
3.1 Data
MNIST: We flattened 28x28 images, and scaled them by dividing all with 255 as it is done in [46] We split training set into training and validation sets (90- 10% split) when searching for hyper-parameters, and then used all of training set to train the final model. The test set is used only for final evaluation.
The Cancer Genome Atlas (TCGA): TCGA is public cancer genomics dataset characterized over 20,000 primary cancer and matched normal samples that holds information over 38 cohorts. The task is classify the cancer cohorts from the reverse phase protein array (RPPA) dataset. It includes 6671 1 samples with 122 features, which we divided to 80-10-109 10-000%0000- valii--iiittionn-ie sets. Once hyper-parameters is found, we trained the models on combined training and validation set.
Obesity: The dataset consists of publicly available human gut metagen-omic samples of obesity cohorts 36). It is derived from whole-genome shotgun metagenomic studies. The dataset consists of 164 obese patients and 89 non-obese controls and has 425 features 22 We scaled the dataset by using min-max scaling. Since it small dataset, we evaluated the model by using 10 randomly drawn training-test (90-109 splits, for each of which we used 10-fold cross-validation.
UCI Adult Income: It is well-known public dataset extracted from the 1994 Census database [24]. includes the details such as education level and demographics to predict whether the income of a person exceeds S50K/yr. The data consists of six continuous and eight categorical features. After one-hot encoding of categorical features, there are total 101 features. The pre- pre-processing steps can be found in Section B. of Appendix.
UCI BlogFeedback: The data originates from blog posts, and is originally used for regression task of predicting the number of comments in the upcoming 24 hours. Similar to Yoon et al. [46]. we turned into a binary classification task of predicting whether there is comment for post or not. There are 280 integer and real valued features, and separate training and test datasets are provided. Further information can be found in Section of Appendix.
3.2 Evaluation
For self-supervised models, once the models are trained, evaluate them by training a logistic regression model on the latent representations of training set, and testing it on the latent representation of the test set. For SubTab, the joint latent representation is obtained by using the mean aggregation of embeddings of the subsets for both training and test sets. We use the performance on a classification task as a measure of quality of the representation as it is usually done in the self-supervised learning. MNIST has 10, TCGA has 38, and the rest (i.e. Obesity, Income, and Blog) has 2 classes each.
3.3 Results
MNIST: We used a simple three-layer encoder architecture with dimensions of [512, 256, 128], referred as the base model, in which the last layer is linear layer. During training of the base model, we used both reconstruction and contrastive losses. Additionally, we trained our model under three
5

---

0.98
0.97
+
4.3
0.95
0.94
6-6
0.93
Overlap between subsets
0.92
0.25
0.5
0.75
(c)
Figure 3: a) Test accuracy on MNIST dataset over different number of subsets and varying levels of overlaps. b-c) I-SNE plots for training (b) and test (c) sets of MNIST for the case of using subsets with 75% overlap between neighboring subsets.
100
Train
Test Accuracy on MNIST
model
Classification Accuracy
97.5
98
Test
97
96
95
92.5
94
80
90
&
IMAT
87.5
Number of subsets aggregated
MLP-tased model applled tatuler appted to image
(a)
(b)
Figure 4: a) After training the base model (latent dimension= on four subsets with 75% overlap, we test its performance using different number of subsets. The performance improves as we start increasing number of subsets involved in prediction. b) Comparing model CNN-based SOTA models trained on 28x28 MNIST in image format (please see Section 3.4 for details).
conditions: i) without any noise in the input data, ii) with noise in the input data and iii) same as (ii). but we also added distance loss computed for pairs of projections {zi, z......
For SubTab, we trained our base model multiple times without noise at the input. For each training, used different number of subsets with different levels of overlap between neighbouring subsets (Figure 33). For small number of subsets (e.g. 2 or 3), the performance monotonically decreases when we increase the overlap between subsets. But, for higher number of subsets, the performance generally improves as we increase the number of shared features between the neighbouring subsets. In general, our results show that K with 75% overlap, and K 7 with 50% overlap perform the best in MNIST dataset, where K refers to the number of subsets. Figure 3 also shows t-SNE plots of training and test sets for K= with 75% overlap, which proves the high quality clustering, while Table I summarizes the classification accuracy of all models on the test set. Our base model without noise outperforms autoencoder baselines and other self supervisddd models with the same architecture. We experimented with three noise types for all self-supervised models, and observed that adding swap-noise at the input pushes the performance higher. For SubTab, adding distance loss and increasing the dimensions of the last layer from 128 to 512 help improve the performance even further. Moreover, we conducted three additional experiments (details Section C.3 of Appendix):
In the first experiment, for the optimum case of K = 4 with 75% overlap, we trained and tested accuracy of a linear model by using the joint representations obtained from the varying number of subsets. Starting with single subset of the data, we plot the training and test accuracy of the model (Figure 4.) The linear model able to achieve 87.5% test accuracy using the representation of single subset. As we start adding latent representations of remaining subsets, both the training and test accuracy keep increasing, eventually achieving top accuracy when subsets are used. The evolution of clusters corresponding to Figure 4p can be seen in Figure A7 in Appendix. This experiment indicates that we can achieve a good performance using only small subset of features when we don't 'tt have access data other features.

---

100
all 7 subsets
95.14
100
all subsess
Classiftcaioon Accuracy
80
90
3, 4,5 5
4,
2, 4, 5.6
89.37
Clasification Accuracy
90
80
4.
2,3,4,5,6
4,5
83.53
100
95
No training
all 7 subsets
70
75.24
60
NG training
60
training
85
50
50
80
30
30
Number subsets aggregated
Subset used for prediction
Subsets used for aggregation
20
70
3
7
3,4,5
2,2,
(a)
(b)
(c)
Figure 5: a) The test accuracy using the mean aggregation of the latent representations of subsets, starting with the first subset, and keep adding new subsets sequentially. b) The test accuracy of individual subsets. c) Comparing the test accuracy by aggregating the representations of different set of subsets at test time for two versions of the model; untrained and the one trained on all subsets.
In the second experiment, we evaluated SubTab under the condition of missing features dur- ing training (Figure (5)). To do so, we first sliced the unshuffled features of MNIST to seven subsets with no overlap (the case corresponding to the legend "7" at zero overlap in Figure 3D. Each subset corresponds to four rows in 28x28 image, starting from top four rows (subset 1) to the bottom ones (subset 7). Then, we trained the base model on five different sets of subsets; {4), (4, 5), f3, 4, 4, {2, models. Please note that we selected the sets such that we expand out from the most informative middle regions of the image (i.e. subset 4) to the least informative top and bottom areas.
3, 4, 5, 6), and (1, 2,3, 3, 4, 5, 6, 7}, resulting in five different trained SubTab
In order compare the performance of five models, we followed the following steps for each trained model: 1) We first obtained the embeddings of all seven subsets for both training and test sets; 2) We then trained and evaluated logistic regression model by using the joint embedding of each of the following seven sets: {1}, [1. 2), (1, (1.2,3)..... {1, 2, 3, 4, 5, 6, 7} i.c. starting from the first subset, we kept adding new subsets sequentially to increase the information content in the sets. For example, for the set {1, 2.33. we first trained a logistic regression model by using joint embedding of subset 1, 2 and 3 from training set, and evaluated by using the joint embedding of same subsets from test set. The joint embedding of a set is obtained by using mean aggregation of embeddings of subsets in the set. In addition five models, we initialized a sixth SubTab model, but kept untrained and followed the same procedure described before to use it as baseline. The results are shown in Figure
In this experiment, we observe that even when the model is trained on single subset (subset 4, or the blue line in Figure 55), aggregating the representations of all seven subsets including the subsets not used in training does improve the results. This is because the encoder able to map samples of different classes to different points in latent space even if it is not trained on them. Since we use the mean aggregation over diffferent views (i.e. subsets) of the same class, we can still make each class in the data distinguishable from the rest in the latent space. We also note that when the model is trained on more and more subsets, its performance keeps improving. As a baseline, we also conducted the same test using untrained model (red line in the plot), and observed similar behaviour in which the test accuracy generally improves as we use more subsets when constructing the joint latent representation. Moreover, we measured the test accuracy of individual subsets to see how informative each subset is (p). The result as expected since we kept the features unshuffled in this experiment, and know that the subsets corresponding to the mid-region of the images (i.e. subsets 3, 4, and 5) should be more informative than the ones corresponding to the top and bottom regions (i.e. subsets 1, and 7). We repeated the same experiment using 28 subsets to get higher resolution and added the result in Figure AS in Appendix. From this experiment; i) we see that joint representation improves as we include more subsets (i.e. sub-views) at training and/or test time, ii) we can identify the informative subsets of features using SubTab framework.
In the third experiment, we evaluated SubTab on handling missing features at test time. Specifically, we used the model trained on subsets, and compared it to the untrained model (i.e. our baseline). For each model, we obtained the joint embedding for training set by using mean aggregation over embeddings of all seven subsets, and then trained a linear model. The test accuracy of the linear model is measured by using: i) only subset 4, ii) aggregate of the most informative subsets (3,4.5),
7

---

Table 1: Accuracy scores for all models for various datasets. The abbreviations in the table; NC:
Neighbour columns used, RF: Random features used, G: Gaussian noise used, St Swap noise used.
Type
Models
MNIST
Income
Blog
Obesity
TOGA
Supervised
Logistic Regression
92.60100003
84.68:0.005
84.1510.12
62,351-4002
36,981 1.25
baseline
Random Forest
9..96.10.06
84.62.10.07
83.6110.115
67.45:12.23
61.62.1. 1.02
XGBoost
98,02-+0.066
86.11=0020
84.29+4..3
64.05444.52
12.11-1.31
Autoencoder
AE
92.77+0.32
84.67=0.07
84.06:20.24
61,964:3.28
56.16=0.75
baseline
AE w/ Dropout (p=0.2)
94.31:::28
85.00=0.100
84,18:0.20
62,74114.38
56.87=2.26
(RF)
96.3010.14:7S)
84.3710..36 (G)
84.12.0.29 (G)
56.43 (G)
54.31 (G)
CAF (NC)
96.39--0.20 (5)
84.24+-0 18 (G)
84.3+-0.1 (G)
62.26-5.01 (G)
14.20+1.17 (G)
VIME sell
95.23.10.17 (S)
84.43.10.08 (G)
84.1110:227 (G)
66.45.14.54 (G)
55.11 (G)
Self-
SubTab with:
supervised
Base model (No noise)
97.26=0.2
85.31=00088
84.29:4.26
68.01+3007
57,02= 50
*Noise
97.47-0.18 (S)
85.34+00.07 (G)
8447-0.15 (G)
7113444,08 (G)
58.25- (G)
+Distance loss +1 S11
97.52=0. 14 (S)
85.35:20.06 (G)
84.64=0.19 (G)
69.25:4419 (G)
58 (G)
97.86--0.07 (5)
iii) aggregate of (2,3,4.5,6) excluding the least informative subsets, and iv) all seven subsets of the test set (Figure 5p).
The results indicate that SubTab can accommodate missing features at test time, and can still perform well. This might also indicate that working with subsets can give us a way to deal with uncertainty better when there are missing features at test time. As the model collects more information in the form of more features, its prediction improves (see Figure Sp) We can also train the model when there are missing subsets during training. and it still performs well (e.g. see legend "4", corresponding to the model trained only subset 4, in Figure 5j.. Our experiments simulate a practical scenario. For example, in healthcare, we might not have access to some features in one hospital while we might have them in another. So, method would be beneficial in this type of cases.
Overall, we can make the following observations from our experiments: i) the less informative subsets can add value to the overall representation, or at least does not harm the performance (see the aggregate over [3,4,5) versus "All" in Figure(5))) ii) untrained model can be used to analyze which subsets can be potentially more informative, iil) once model is trained on a subset, the performance of the individual subset does not change whether it is trained together with other subsets or not (for example, compare the performance of subset 3, 4, and 5 across all models in Figure PP)) iv) general idea behind our framework works even for untrained model, and v) we may not need to impute data in framework since can simply ignore them missing subsets, which is good since imputation generally distorts data, and the results.
TCGA: We used an encoder architecture with three layers [1024, 784, 784], where the third layer is linear. For VIME-self, DAE, CAE, and our model, we experimented with three noise types (Gaussian, swap, and zero-out noise) at the different % levels of masking ratio p. We observed that p = [0. (0,15,00 0.3) range worked well for all models. For Gaussian noise, we used a distribution with zero mean, and different levels of standard deviation (o). Among all three noise types, Gaussian noise with o = 0.1 worked the best for all models. Please note that VIME-self uses swap-noise in its original implementation, but swap-noise does not work well on this dataset. For SubTab, similar to MNIST, we used four subsets with 75% overlap. Sub performs better than other self-supervised models with a significant margin and almost doubles the performance of logistic regression model trained on raw data as shown in [
Obesity: We used two-layer encoder with [1024, 1024] dimensions. Second layer is linear layer. Gaussian noise 0.3) and masking ratio p 0.2 works well across all models. Six subsets (K = 6) with 0% overlap performed the best for the SubTab. We note that this dataset has 164 obese patients out of 253 total patients. So, the bascline accuracy is 164/253- = 64.82%. Based on this fact, we can say that all models, except ours, did not perform well on this dataset. Our model with added Gaussian noise results in accuracy of 71.13 + 4.08%, which is well above all models, including supervised ones. It means that our model was able to learn useful representation from the data. We should also note that the performance of our model is much better than what Oh and Zhang (36) reported (66 + 3.2%) even though they trained a DAE on the same data, and reported their results using a random forest, a non-linear model, on the learned representations rather than a linear model.
UCI Adult Income& & BlogFeedback: For these two datasets, we used the same architecture as in Obesity. For Income dataset, the best performance obtained using 5 subsets with 25% overlap whereas we used 7 subsets with 75% overlap for Blog dataset. For the base model, we only used
8

---

reconstruction loss. Adding Gaussian noise to the input and distance loss to the objective improves the performance for both datasets. SubTab outperforms other self-supervised models in both datasets.
The choice of hapaee-aramettrs and other details for all experiments can be found in Table A1 in Section C.1 of Appendix.
3.4 Ablation study
We conducted a comprehensive ablation study us- ing MNIST. Table 2 summarizes our experiments. The first thing to note is that the performance the our base model is already good with only construction loss. Hence, we can argue that the reconstruction of original feature space from sub- set of features a very effective way of learning representation. By adding noise the input data, we can improve the performance. In the case of MNIST, swap-noise is very effective. Also, by adding additional losses such as contrastive, and distance losses as well increasing the dimension of representation layer from 128 to 512, we can further improve the results. Moreover, we shuf- fled the features of MNIST to make sure that we don't have any gains from unintentional spatial correlations between neighboring features. We kept all parameters and random seeds same for the comparison. As shown in the table, our model's performance does not change much. We also tried concatenating latent variables of subsets rather than aggregating them when testing the performance. Comparing last two rows in the table, the aggregation shown to work much better. Please note that we compared different aggregation functions in Appendix F.4 showing that mean aggregation worked the best.
Table 2: Ablation study using MNIST with subsets with 75% overlap. Abbreviations are; RL: Reconstruction Loss, CL: Contrastive Loss, DL: Distance Loss, SF: Shuffled Features, LD: Latent Dim, Agg: Aggregating embeddings.
Agg
Accuracy
128
97.13
128
97.11
128
97.26
Zero-ou
128
97.25
Gaussian
128
97.25
Swap
128
97.47
Swap
128
97.52
Swap
128
97.2
Swap
512
95,92
Swap
Finally, we compared the performance of SubTab on shallow and deep architecture choices. We trained and tested very shallow architectures for SubTab (referred as shallow SubTab), and compared them to relatively deeper SubTab models used in Table [ (referred as deep SubTab). We used one-layer encoder and decoder with 784 dimension each for MNIST while using 1024 dimension for other datasets. Shallow SubTab is trained and evaluated under the same conditions as the deeper ones. As shown Table shallow SubTab significantly improves results in MNIST and TCGA,
placing our model performance on par with CNN-based SOTA models (20 19 in Figure 4b. Obesity is the only dataset which exploits the deeper architecture.
25 22
32
as shown
Related works
We refer the reader to the introduction section that lists some of the recent noticeable works in self-supervised learning. Since our work focuses on tabular data, we will review some of the recent work done in tabular data in self-supervised framework. The most recent work is mostly based on solving a pretext task. For example, Yoon et al. (46) uses de-noising autoencoder with classifier attached its representation layer. A random binary mask is generated to mask and overwrite portion of entries in the tabular data, and the corrupted data is given as input to the encoder. The classifier is used to predict the mask while decoder used re-construct the uncorrupted original input similar to de-noising autoencoder [43]. Although the proposed method is shown to work well in the experiments, there are couple drawbacks to this approach. Firstly, this approach might not work well in very high-dimensional, small and noisy data sets since the model might easily become pver-parameterized and be prone overfitting the data. Secondly, training a classifier in this
Table 3: Comparing shallow and deep SubTab architectures.
Model
MNIST
Income
Blog
Obesity
TCGA
Deep SubTab
97.86-0.09
84 84.6410.11 8.005
4.013
58,25+t 1.36
Shallow SubTab
98.31-0.06
8..341.003
84. 84.64:1..09
66. 1,535
61.41::1.11

---

setting can be challenging since it needs to predict very high dimensional, sparse, and imbalanced binary mask, similar the problems observed when training a model on imbalanced, binary dataset. In similar way, TabNet [1] and TaBERT [45 also tries to recover original data from corrupted one.
5 Conclusion
In this work, we show that a simple MLP-based autoencoder trained on MNIST in tabular format can perform par with the CNN-based SOTA models trained MNIST images in unsupervised/self- supervised framework. SubTab achieves SOTA in MNIST dataset in tabular setting. We also tested our approach other commonly used tabular datasets, and proved its benefits. In SubTab, the main performance gain comes from two parts of the model: i) reconstruction of all features from the subset of features, and ii) learning the joint representation by aggregating the embeddings of the subsets.
Using subsets features may obviate the need for data imputation during training, and allows inference using subsets of features at test time. It might open the door to distributed training of high-dimensional data since the models can be trained on different subsets of features at the same time. We can also potentially take advantage of different datasets with common features by assigning those features to same subsets (i.c. transfer learning). We should note that the subsets shared the same autoencoder in experiments although could use separate autoencoders for different subsets if some of the features are drastically different than the rest.
SubTab is computationally scalable when we use only reconstruction loss during training. However, using contrastive, and/or distance losses requires the combinations projections, which makes the computational complexity quadratic during training and limits the number of subsets we can use to divide the data. In this case, computational complexity is still linear at test time since we need to compute only the aggregate of the representations of the subsets. Also, when we divide the features into subsets, we keep the location of features in each subset same throughout training and test time since neural networks are not permutation invariant. As possible solution, to permutation invariant architectures by treating collection of features as a set. We also showed that SubTab framework resolution. A hierarchical version of SubTab might be used for identifying individual important features, but we leave it as a future work.
extend our work
be used to discover most informative subsets of features with limited
Finally, although the primary focus of this work is tabular data setting, SubTab can be extended to other domains such as images, audio, text and so on. We leave the extensions and applications of SubTab as a future work.
Broader Impact
Tabular data is a commonly used format in healthcare, finance, law and many other fields. Despite its broad usage, the most of the research in deep learning, especially with regards to unsupervised representation learning, has been on other data types such as images, text and audio. Our paper tries to close this gap by introducing a new framework to learn good representations from tabular data in mmsupervisedd/sef--uperyv widespread applications of tabular data in other areas such as transfer learning, distributed learning, and multi-view learning, in which we can combine knowledge such demographics and genomics from tabular data with those in images, text and audio. However, we should be aware of the shortcomings of such data integration in terms of biases and privacy issues that it might introduce.
setting. The progress in this line of research will open doors to
7
Acknowledgements
We thank the anonymous reviewers for their helpful and constructive feedback on the paper. We would also like to thank the entire Respiratory and Immunology AI team and are grateful for general support from other organizations within AstraZeneca.
10

---

References
[1] Serean Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. arXiv preprint 1rXiv:1908.07442, 2019.
[2] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint arXiv: 7708. 1708807731 2017.
[3] James P Bridge, Sean B Holden, and Lawrence C Paulson. Machine learning for first-order theorem proving. Journal of automated reasoning, 53(2):141-172, 2014.
[4] Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pages 145-152. Springer, 2014.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
[6] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '16, pages 785- 185-794, New York, NY. USA, 2016. ACM. ISBN 978- 1-4003-4232--. doi: 2939785
10.1 (4.144///33972.29999778
URL http://ddd acm. org/ 10. 11111119399981
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 20X2:22003.04297, 2020.
(9] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160-167, 2008
[10] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ou,, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint 1911:2911 2019.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Ming-Wei
[12] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:///://rrchitv ics.uci wuiiedeuie cdu/ml
[13] William Falcon and Kyunghyun Cho. A framework for contrastive self-supe learning and designing approach. arXiv preprint arXiv:2009.00104 2020.
[14] Ananda L Freire, Guilherme A Barreto, Marcus Veloso, and Antonio T Varela. Short-term memory mechanisms neural network learning robot navigation tasks: A case study. In 2009 6th Latin American Robotics Symposium (LARS 2009), pages 1-6. IEEE, 2009.
[15] Jean- Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
[16] Michael Harries and New South Wales. Splice-2 comparative evaluation: Electricity pricing. 1999.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 2020.
11

---

[18] Irwin Hersey. Textures: A photographic album for artists and designers by phil brodatz. Leonardo, 1(1):91-92, 1968.
[19] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International Conference on Machine Learning, pages 1558 1567. PMLR, 2017.
[20] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9865-9874, 2019.
[21] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint wrXiv:160..0177 2016.
[22] Lyes Khacef, Laurent Rodriguez, and Benoit Miramond. Improving self-organizing maps with unsupervised feature extraction. In International Conference Neural Information Processing, pages 474-486. Springer, 2020.
[23] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint /609.0/609.00000 2016.
[24] Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree tree hybrid. In Kdd, volume 96, pages 202-207, 1996.
[25] Adam R Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. arXiv preprint arXiv: 06818, 2019.
[26] Emmanuelle Le Chatelier, Trine Nielsen, Junjie Qin, Edi Prifti, Falk Hildebrand, Gwen Falony, Mathieu Almeida, Manimozhiyan Arumugam, Jean-Michel Batto, Sean Kennedy, et al. Richness of human gut microbiome correlates with metabolic markers. Nature, 500(7464):541-546, 2013.
[27] Yann LeCun. The mnist database of handwritten digits. http://yaan.. lecun. com/exdb/nnistl/ 1998.
[28] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint /907. 2019.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arX1v:1711.05101, 2017.
[30] DD Lucas, Klein, Tannahill, D Ivanova, S Brandon, D Domyancic, and Y Zhang, Failure analysis of parameter-induced simulation crashes in climate models. Geoscientific Model Development, 1577:11 2013.
[31] Renata CB Madeo, Clodoaldo AM Lima, and Sarajane M Peres. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pages 46-52, 2013.
[32] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders. arXiv preprint 15Xiv:1511.050 2015.
[33] Merz and PM Murphy. Pima indians diabetes dataset. UCI Repository, 1996.
[34] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint
2013.
[35] Michiel O Noordewier, Geoffrey Towell, and Jude w Shavlik. Training knowledge-based neural networks to recognize genes in dna sequences. In Advances in neural information processing systems, pages 530-536, 1991.
[36] Min Oh and Liqing Zhang. Deepmicro: deep representation learning for disease prediction based on microbiome data. Scientific reports, 10(1):1-9, 2020.
12

---

[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint wrXiv:1807.03744 2018.
[38]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. Utt.http://ppperrrrre http::/s....errr..c....cccooocee cc/paper/
9015-pytorch-an- http://www....aaae......eeeaaaaa pdf
[39] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016.
[40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[41] Jeffrey Pennington. Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[42] Katarzyna Tomczak, Patrycja Czerwińska, and Maciej Wiznerowicz. The cancer genome atlas (tega): an immeasurable source of knowledge. Contemporary oncology, 19(1A):,68, 2015.
[43] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008.
[44] I-Cheng Yeh, King-Jang Yang, and Tao-Ming Ting. Knowledge discovery on rfm model using bernoulli sequence. Expert Systems with Applications, 36(3):5866-5871, 2009.
[45] Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding textual and tabular data. arXiv preprint arXiv:2005.083 /4, 2020.
[46] Jinsung Yoon. Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning tabular domain. Advances in Neural Information Processing Systems, 33, 2020.
[47] Kun Zhang and Wei Fan. Forecasting skewed biased stochastic ozone days: analyses, solutions and beyond. Knowledge and Information Systems, 14(3):299-326, 2008
13


Reviews:
[{"review_id": "sOSzBuc2P4", "reviewer": "Reviewer_2zYu", "summary": "The paper proposes SubTab, a self-supervised (pre-training) method for learning good representations of tabular data. The method generates multiple views for each (unlabeled) example by selecting different (possibly overlapping) subsets of features. The model is trained with a three term loss, each averaged over all pairwise comparisons between the views -- a reconstruction term that teaches the model to reconstruct the full example from the cropped example, a contrastive term that encourages different crops of the same example to be closer together than crops of different examples (within the minibatch, infoNCE loss), and a distance term that encourages crops of the same example to be close together in L2 distance. Optionally, the input example can be corrupted by noise. After pre-training a linear classifier on the latent representation is trained using supervision. In this step, the latent embedding used is actually an average of those arising from feeding in several different random crops of the example. The authors test the method against VIME (Neurips 2020), context encoders, and denoising autoencoders on 5 datasets, which were used in the VIME paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has merits:\n1) The motivation of the paper is good -- indeed, self-supervised learning for tabular data is relatively understudied.\n2) The idea of generating views using random subsets of features is somewhat novel (though \"Self-supervised Learning for Large-scale Item Recommendations\" (https://arxiv.org/pdf/2007.12865.pdf) proposes using random feature masking for recommenders)\n3) The authors compare against recent baselines and seem to outperform VIME.\n4) Ablations are conducted.\n\nHowever, there are concerns that push me to lean against acceptance:\n1) More than just 5 datasets are needed to be convincing, especially given that both the datasets and model sizes are small so training should be quick (compared to say large language models). I would recommend benchmarking against 69 datasets from OpenML-CC18 -- as done in https://arxiv.org/abs/2106.15147\n2) During test-time, the model applies a linear classifier on the average of the multiple latent embeddings (stemming from different crops of the original example). Averaging or \"ensembling\" in this way almost always boosts performance. If I understand correctly, this wasn't done for the baselines?\n3) Even more baselines are needed. In particular, dropout -- both \"feature dropout\", where entries of the example are zeroed, and \"model dropout\", where dropout is applied to the intermediate neurons. Ensembling can be tested out here too. Also, I would be interested in seeing a comparison of SubTab with SCARF (https://arxiv.org/abs/2106.15147) -- though I understand this is concurrent work.\n4) The method has quite a few hyper-parameters -- one for each term in the loss and also the type of noise to apply. Given that different datasets have different optimal hyper-parameters, it's not entirely clear that the gains over baselines aren't just a consequence of having more tunable hyper-parameters.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "fW9G4d_7YeV", "reviewer": "Reviewer_TMPY", "summary": "This paper proposes a self-supervised framework for tabular data. The idea is to split tabular data into multiple subsets of features, embed them into a latent space, and compute a joint representation by averaging the subsets’ latent variables. The approach outperforms existing self-supervised tabular baselines and reaches performances that are on par with CNN models on MNIST.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method is simple, elegant, and conceptually appealing. The approach has many interesting applications such as learning better representations in tabular datasets (e.g. for healthcare) and distributed training.\n- The approach outperforms existing self-supervised methods in a number of tabular datasets. It also reaches performances that are on par with CNN models on MNIST.\n- The authors analyse the performance of the model across different numbers of subsets, overlap rates, types of noise, and trained/untrained models, with interesting insights and practical advice (e.g. the untrained model can be used to detect informative subsets).\n\nWeaknesses:\n- It is mentioned that “all subsets are fed to the same encoder to get their corresponding latent representation” (L65-66) and that “we don’t change the relative order of features in a subset since standard neural network architectures are not permutation invariant” (L80-81). It is unclear how this happens - do the authors mask out features of other subsets (e.g. by setting them to 0)? If that’s the case, on what basis do they claim that SubTab allows to “use smaller models by reducing input dimension, making it less prone to overfitting“ (L53)? Or do they reuse the same input neurons for different input features?\n- The claim that SubTab can “do training and inference in the presence of missing features by ignoring corresponding subsets” (L52-53) is not well supported in the paper. Can the authors demonstrate this? While SubTab might work well in some scenarios (e.g. when the variables are missing completely at random; MCAR), in many real-world scenarios the MCAR assumption does not hold. For example, for single-cell RNA-seq, it is well known that dropout patterns are informative of cell-type (e.g. see https://www.nature.com/articles/s41467-020-14976-9) and thus ignoring this information can result in degraded performance.\n- How does the proposed approach compare to a plain autoencoder with the same encoder/decoder architectures? How does it compare to self-supervised TabNet? (see https://arxiv.org/abs/1908.07442)\n\nMinor comments/questions:\n- “We shuffle the order of subsets in every batch during training to avoid introducing any unintentional bias“ (L79-80). How exactly would the order introduce a bias? The forward pass of SubTab appears to be invariant to subset permutations.\n- Equations 4 and 5 might be incorrect and can potentially be simplified. 1) The indices a and b are undefined - don’t the authors loop over the subset indices a and b? (similar for Equation 6), 2) isn’t the overall contrastive loss computed by averaging the individual positive (pairs of subsets within the same sample) and negative (pairs of subsets within different samples) losses? Equation 4 seems to imply that the loss is computed only on contiguous samples within the batch (in which case the order of subsets might indeed result in an unintentional bias).\n- How did the authors optimise the hyperparameters of the baselines? To avoid biases in the comparison, all methods should be optimised in the same way.\n- Results for some baseline methods are taken from Yoon et al. “Extending the success of self-and semi-supervised learning to tabular domain”. Did the authors ensure that the train/validation/test splits are the same?\n- The untrained model achieves better than random performances. How did the authors initialise the weights? How sensitive is the initialisation? It would be helpful to provide the standard error in Figure 5 (and all figures in general).\n- Intuitively, it seems that SubTab should work well with highly redundant datasets where several sets of features are informative of the underlying latent variables (e.g. MNIST or TCGA). How does the model perform when this is not the case? Do the claims from L49-L53 still hold?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SVQ4XIHwJ4G", "reviewer": "Reviewer_pA7F", "summary": "This work presents a self-supervised method for the tabular domain, where structural information is not typically available (in contrast to images/text). The proposed method suggests partitioning input rows into subsets of features, where these subsets can have a certain overlap percentage as controlled by a hyperparameter and mapped to the latent space via an encoder where they define their reconstruction loss (authors provide two options 1—reconstructing the original subset from the subset latent representation 2—reconstructing the complete row from the subset representation), an optional contrastive loss, as well as a distance loss term in the projection embedding space where the negatives are chosen from subset latent representations from other rows. Orthogonal to their work, they leverage various noise strategies commonly used in the tabular domain, such as swap noise (imputing a feature from the same column from a different row), gaussian noise, and zero imputation. The authors compare with the previous SOTA method (VIME) on various tabular datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Subsetting features (local view) to predict a global view is somewhat similar to random cropping and zooming in images, and I believe its application in the tabular domain is novel. \n\nIn section 2, as the authors used fully connected layers, by subsetting features, different columns will be fed to the same neuron at the input layer; this may lead to representations collapse to the same latent point. For example, consider a tabular data with six columns, and the rows 000101, 101000 then as per the method, these would map to similar latent representations if we consider subset features of length 3, but in fact, they may correspond to different manifolds when taken as a whole. This would also cause problems for the contrastive loss term/distance loss.\n\nIn section 2, l. 79, does random shuffling help performance? It is not clear how this could get rid of bias as a subset of features is being fed to the network, and from my understanding, ordering of subsets does not seem to play a role in the proposed method.\n\nSection 2, l. 105 It would be great if authors could give intuition behind the contrastive/distance loss terms and why they make sense for their proposed augmentation\n\nSection 3. How was the $\\alpha_k$ in eqn. 3) set?\n\nIn subsection 3.1 it is unclear whether authors use the complete labelled training sets or treat only a subset of the training set as labelled. The ambiguity occurs when authors make a comparison with VIME[1], specifically, l. 196. VIME considers 10% of the training set as labelled and treats the rest as unlabeled in their results.\n\nIn subsection 3.3, the authors consider an [512-256-128] architecture, whereas VIME results use a single encoder layer of size 784 and 100-100 MLP with ReLU activation for their predictor, achieving 95.75 % or 94.55 % for VIME-self. There's confusion as authors claim to use the values from [1] when reporting results in Table 1, but in l.138 claim to use the same encoder size achieving lower accuracy values than the simpler architecture used in VIME. One may argue that this makes the comparison unfair/unclear. If authors tune architecture/hyperparameters for their method, the same should be done for other methods.\n\nIn table 1. Authors borrow values from VIME [1], and report the variant of VIME Self-SL only, is there a reason for this? VIME proposes an unsupervised pre-training step using swap noise (does not use labels), and enforces a consistency loss in their predictor (10% training labelled set) hence the semi-supervised. As a result, I believe their complete VIME method should be compared. Further, following my previous comments, it is unclear if the training setup is fair, specifically the architecture, and perhaps the preprocessing steps (using 1/255 normalization versus z normalization or min-max for MNIST may provide 1-3% accuracy gain in low capacity networks). Ideally, other methods should be implemented in the same framework with similar data pipelines and then compared. VIME MNIST results are reproducible and can be used to verify implementation errors.\n\n\nL.204, the figure being referenced is the curve for the Blog dataset; I believe 5a) was the intended figure. \nIn this paragraph, does “train” refer to the pre-training step or training the predictor? In l. 214, If the encoder hasn’t been trained on the subsets, why can they be mapped to different points in the latent space? Following my previous example, it would seem representations for different rows could collapse to the same point. Why is subset 1 excluded (due to white pixels In MNIST?) ? Overall this paragraph is confusing.\n\nThere’s some minor inconsistencies e.g.  l. 227 (3, 4, 5) l.231 [3, 4, 5]\n\n \n[1] Yoon, Jinsung, et al. \"Vime: Extending the success of self-and semi-supervised learning to tabular domain.\" Advances in Neural Information Processing Systems 33 (2020).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seems adequate.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MR2n7Z8veP", "reviewer": "Reviewer_b2iX", "summary": "The authors propose a novel self-supervised learning for tabular data.\nThe authors utilize the reconstruction of the entire data using the subsets of the features as the pretext task.\nThe authors show some consistent performance improvements in comparison to SOTA SSL in tabular representation learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Hyper-parameters-\n It seems like the proposed method introduces multiple hyper-parameters: alpha, beta, gamma, p, etc.\n- How to optimize those hyper-parameters? What is the objective function to optimize them?\n- It would be good if the authors can show some sensitive analyses about those hyper-parameters. \n- If eta is a learnable parameter, how can we learn the optimal eta? It would be good if the authors can describe that clearer.\n\n2. Neighbor columns\n- In image data, neighbor pixels should share the similar patterns.\n- On the other hand, in tabular data, there is no reason to share some patterns across neighbor features. \n- At that point, what is the difference between neighbor columns and random columns? We can still make some overlapping in random columns case.\n\n3. Ensemble\n- How can this work be related to the ensemble works?\n- It seems like using weighted sum among the different representations can be interpreted as the ensemble approach for representation learning.\n- As the authors said, eta_k = 1 for all k. In that case, it seems like a similar idea with bagging.\n\n4. Discovering the informative features\n- It seems like the proposed model only determines which subset of features is more important.\n- However, it cannot show a feature level importance.\n- It would be good to describe this kind of limitation.\n- Also, it would be good if the authors provide some qualitative analyses on this feature subset importance in the experiment section.\n\n5. Etc\n- How much performance differences between ReLu and LeakyReLu? Also, can the authors show the performances of VIME-self with LeakyReLu?\n- How is the performance of the proposed model with ReLU? (for fair comparison with VIME)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors need to illustrate the limitation of the proposed model explicitly.\nFor instance, the proposed method cannot provide the feature level importance.\nThere is no negative social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have addressed most of the reviewer concerns during the review process, and I now suggest the acceptance of the paper. As the reviewers have acknowledged, the performance improvements are notable with an elegant and conceptually appealing method. 

There is significant content in the author responses that need to be integrated into the paper prior to publication, especially on the new results on extra benchmark datasets, improved literature review, numerous clarification points on the method explanations, hyperparameter tuning, ablation studies, and description of limitations. 
Decision: Accept (Poster)