Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes SubTab, a self-supervised (pre-training) method for learning good representations of tabular data. The method generates multiple views for each (unlabeled) example by selecting different (possibly overlapping) subsets of features. The model is trained with a three term loss, each averaged over all pairwise comparisons between the views -- a reconstruction term that teaches the model to reconstruct the full example from the cropped example, a contrastive term that encourages different crops of the same example to be closer together than crops of different examples (within the minibatch, infoNCE loss), and a distance term that encourages crops of the same example to be close together in L2 distance. Optionally, the input example can be corrupted by noise. After pre-training a linear classifier on the latent representation is trained using supervision. In this step, the latent embedding used is actually an average of those arising from feeding in several different random crops of the example. The authors test the method against VIME (Neurips 2020), context encoders, and denoising autoencoders on 5 datasets, which were used in the VIME paper.
main_review: The paper has merits:
1) The motivation of the paper is good -- indeed, self-supervised learning for tabular data is relatively understudied.
2) The idea of generating views using random subsets of features is somewhat novel (though "Self-supervised Learning for Large-scale Item Recommendations" (https://arxiv.org/pdf/2007.12865.pdf) proposes using random feature masking for recommenders)
3) The authors compare against recent baselines and seem to outperform VIME.
4) Ablations are conducted.

However, there are concerns that push me to lean against acceptance:
1) More than just 5 datasets are needed to be convincing, especially given that both the datasets and model sizes are small so training should be quick (compared to say large language models). I would recommend benchmarking against 69 datasets from OpenML-CC18 -- as done in https://arxiv.org/abs/2106.15147
2) During test-time, the model applies a linear classifier on the average of the multiple latent embeddings (stemming from different crops of the original example). Averaging or "ensembling" in this way almost always boosts performance. If I understand correctly, this wasn't done for the baselines?
3) Even more baselines are needed. In particular, dropout -- both "feature dropout", where entries of the example are zeroed, and "model dropout", where dropout is applied to the intermediate neurons. Ensembling can be tested out here too. Also, I would be interested in seeing a comparison of SubTab with SCARF (https://arxiv.org/abs/2106.15147) -- though I understand this is concurrent work.
4) The method has quite a few hyper-parameters -- one for each term in the loss and also the type of noise to apply. Given that different datasets have different optimal hyper-parameters, it's not entirely clear that the gains over baselines aren't just a consequence of having more tunable hyper-parameters.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents a self-supervised method for the tabular domain, where structural information is not typically available (in contrast to images/text). The proposed method suggests partitioning input rows into subsets of features, where these subsets can have a certain overlap percentage as controlled by a hyperparameter and mapped to the latent space via an encoder where they define their reconstruction loss (authors provide two options 1—reconstructing the original subset from the subset latent representation 2—reconstructing the complete row from the subset representation), an optional contrastive loss, as well as a distance loss term in the projection embedding space where the negatives are chosen from subset latent representations from other rows. Orthogonal to their work, they leverage various noise strategies commonly used in the tabular domain, such as swap noise (imputing a feature from the same column from a different row), gaussian noise, and zero imputation. The authors compare with the previous SOTA method (VIME) on various tabular datasets.
main_review: Subsetting features (local view) to predict a global view is somewhat similar to random cropping and zooming in images, and I believe its application in the tabular domain is novel. 

In section 2, as the authors used fully connected layers, by subsetting features, different columns will be fed to the same neuron at the input layer; this may lead to representations collapse to the same latent point. For example, consider a tabular data with six columns, and the rows 000101, 101000 then as per the method, these would map to similar latent representations if we consider subset features of length 3, but in fact, they may correspond to different manifolds when taken as a whole. This would also cause problems for the contrastive loss term/distance loss.

In section 2, l. 79, does random shuffling help performance? It is not clear how this could get rid of bias as a subset of features is being fed to the network, and from my understanding, ordering of subsets does not seem to play a role in the proposed method.

Section 2, l. 105 It would be great if authors could give intuition behind the contrastive/distance loss terms and why they make sense for their proposed augmentation

Section 3. How was the $\alpha_k$ in eqn. 3) set?

In subsection 3.1 it is unclear whether authors use the complete labelled training sets or treat only a subset of the training set as labelled. The ambiguity occurs when authors make a comparison with VIME[1], specifically, l. 196. VIME considers 10% of the training set as labelled and treats the rest as unlabeled in their results.

In subsection 3.3, the authors consider an [512-256-128] architecture, whereas VIME results use a single encoder layer of size 784 and 100-100 MLP with ReLU activation for their predictor, achieving 95.75 % or 94.55 % for VIME-self. There's confusion as authors claim to use the values from [1] when reporting results in Table 1, but in l.138 claim to use the same encoder size achieving lower accuracy values than the simpler architecture used in VIME. One may argue that this makes the comparison unfair/unclear. If authors tune architecture/hyperparameters for their method, the same should be done for other methods.

In table 1. Authors borrow values from VIME [1], and report the variant of VIME Self-SL only, is there a reason for this? VIME proposes an unsupervised pre-training step using swap noise (does not use labels), and enforces a consistency loss in their predictor (10% training labelled set) hence the semi-supervised. As a result, I believe their complete VIME method should be compared. Further, following my previous comments, it is unclear if the training setup is fair, specifically the architecture, and perhaps the preprocessing steps (using 1/255 normalization versus z normalization or min-max for MNIST may provide 1-3% accuracy gain in low capacity networks). Ideally, other methods should be implemented in the same framework with similar data pipelines and then compared. VIME MNIST results are reproducible and can be used to verify implementation errors.


L.204, the figure being referenced is the curve for the Blog dataset; I believe 5a) was the intended figure. 
In this paragraph, does “train” refer to the pre-training step or training the predictor? In l. 214, If the encoder hasn’t been trained on the subsets, why can they be mapped to different points in the latent space? Following my previous example, it would seem representations for different rows could collapse to the same point. Why is subset 1 excluded (due to white pixels In MNIST?) ? Overall this paragraph is confusing.

There’s some minor inconsistencies e.g.  l. 227 (3, 4, 5) l.231 [3, 4, 5]

 
[1] Yoon, Jinsung, et al. "Vime: Extending the success of self-and semi-supervised learning to tabular domain." Advances in Neural Information Processing Systems 33 (2020).

limitations_and_societal_impact: Seems adequate.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a self-supervised framework for tabular data. The idea is to split tabular data into multiple subsets of features, embed them into a latent space, and compute a joint representation by averaging the subsets’ latent variables. The approach outperforms existing self-supervised tabular baselines and reaches performances that are on par with CNN models on MNIST.
main_review: Strengths:
- The proposed method is simple, elegant, and conceptually appealing. The approach has many interesting applications such as learning better representations in tabular datasets (e.g. for healthcare) and distributed training.
- The approach outperforms existing self-supervised methods in a number of tabular datasets. It also reaches performances that are on par with CNN models on MNIST.
- The authors analyse the performance of the model across different numbers of subsets, overlap rates, types of noise, and trained/untrained models, with interesting insights and practical advice (e.g. the untrained model can be used to detect informative subsets).

Weaknesses:
- It is mentioned that “all subsets are fed to the same encoder to get their corresponding latent representation” (L65-66) and that “we don’t change the relative order of features in a subset since standard neural network architectures are not permutation invariant” (L80-81). It is unclear how this happens - do the authors mask out features of other subsets (e.g. by setting them to 0)? If that’s the case, on what basis do they claim that SubTab allows to “use smaller models by reducing input dimension, making it less prone to overfitting“ (L53)? Or do they reuse the same input neurons for different input features?
- The claim that SubTab can “do training and inference in the presence of missing features by ignoring corresponding subsets” (L52-53) is not well supported in the paper. Can the authors demonstrate this? While SubTab might work well in some scenarios (e.g. when the variables are missing completely at random; MCAR), in many real-world scenarios the MCAR assumption does not hold. For example, for single-cell RNA-seq, it is well known that dropout patterns are informative of cell-type (e.g. see https://www.nature.com/articles/s41467-020-14976-9) and thus ignoring this information can result in degraded performance.
- How does the proposed approach compare to a plain autoencoder with the same encoder/decoder architectures? How does it compare to self-supervised TabNet? (see https://arxiv.org/abs/1908.07442)

Minor comments/questions:
- “We shuffle the order of subsets in every batch during training to avoid introducing any unintentional bias“ (L79-80). How exactly would the order introduce a bias? The forward pass of SubTab appears to be invariant to subset permutations.
- Equations 4 and 5 might be incorrect and can potentially be simplified. 1) The indices a and b are undefined - don’t the authors loop over the subset indices a and b? (similar for Equation 6), 2) isn’t the overall contrastive loss computed by averaging the individual positive (pairs of subsets within the same sample) and negative (pairs of subsets within different samples) losses? Equation 4 seems to imply that the loss is computed only on contiguous samples within the batch (in which case the order of subsets might indeed result in an unintentional bias).
- How did the authors optimise the hyperparameters of the baselines? To avoid biases in the comparison, all methods should be optimised in the same way.
- Results for some baseline methods are taken from Yoon et al. “Extending the success of self-and semi-supervised learning to tabular domain”. Did the authors ensure that the train/validation/test splits are the same?
- The untrained model achieves better than random performances. How did the authors initialise the weights? How sensitive is the initialisation? It would be helpful to provide the standard error in Figure 5 (and all figures in general).
- Intuitively, it seems that SubTab should work well with highly redundant datasets where several sets of features are informative of the underlying latent variables (e.g. MNIST or TCGA). How does the model perform when this is not the case? Do the claims from L49-L53 still hold?

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel self-supervised learning for tabular data.
The authors utilize the reconstruction of the entire data using the subsets of the features as the pretext task.
The authors show some consistent performance improvements in comparison to SOTA SSL in tabular representation learning.
main_review: 1. Hyper-parameters-
 It seems like the proposed method introduces multiple hyper-parameters: alpha, beta, gamma, p, etc.
- How to optimize those hyper-parameters? What is the objective function to optimize them?
- It would be good if the authors can show some sensitive analyses about those hyper-parameters. 
- If eta is a learnable parameter, how can we learn the optimal eta? It would be good if the authors can describe that clearer.

2. Neighbor columns
- In image data, neighbor pixels should share the similar patterns.
- On the other hand, in tabular data, there is no reason to share some patterns across neighbor features. 
- At that point, what is the difference between neighbor columns and random columns? We can still make some overlapping in random columns case.

3. Ensemble
- How can this work be related to the ensemble works?
- It seems like using weighted sum among the different representations can be interpreted as the ensemble approach for representation learning.
- As the authors said, eta_k = 1 for all k. In that case, it seems like a similar idea with bagging.

4. Discovering the informative features
- It seems like the proposed model only determines which subset of features is more important.
- However, it cannot show a feature level importance.
- It would be good to describe this kind of limitation.
- Also, it would be good if the authors provide some qualitative analyses on this feature subset importance in the experiment section.

5. Etc
- How much performance differences between ReLu and LeakyReLu? Also, can the authors show the performances of VIME-self with LeakyReLu?
- How is the performance of the proposed model with ReLU? (for fair comparison with VIME)
limitations_and_societal_impact: The authors need to illustrate the limitation of the proposed model explicitly.
For instance, the proposed method cannot provide the feature level importance.
There is no negative social impact.
review_title: null

Meta-review: The authors have addressed most of the reviewer concerns during the review process, and I now suggest the acceptance of the paper. As the reviewers have acknowledged, the performance improvements are notable with an elegant and conceptually appealing method. 

There is significant content in the author responses that need to be integrated into the paper prior to publication, especially on the new results on extra benchmark datasets, improved literature review, numerous clarification points on the method explanations, hyperparameter tuning, ablation studies, and description of limitations. 
Decision: Accept (Poster)