Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. They basically convert each row into a feature vector and then apply stacked transformers for feature encoding. There are several advantages of this encoding: (1) it can deal with the tables that have different number of columns; (2) it is easier to transfer the knowledge learned from different columns. They conduct experiments on one clinical dataset and several public datasets under four different settings: supervised learning, feature incremental learning, transfer learning, and zero-shot learning. The empirical results show that the proposed approach outperform the baselines in the literature. They also showed that in the zero-shot learning scenario, they can almost match the performance of pretraining plus fine-tuning.
strengths_and_weaknesses: - Originality: The main idea in this paper is a good combination of several ideas proposed in the literature. With modifications and adaptations, it worked and yield promising results on several datasets.

- Quality: The proposed approach is technically sound and the experimental results showed that it outperformed several strong baselines for tabular data prediction. Although the results are impressive, I have several comments:
  - One of the main advantages advertised in the paper is that the proposed method could easily extend to feature incremental learning, pretraining+finetuning, and zero-shot inference. In the paper "existing works only cover vanilla supervised learning and fixed-table pretraining due to the fixed-column assumption." In my point of view, this is overclaiming. Not all existing works only cover vanilla supervised learning. For example, those transformer-based architectures like TabTrans, FT-Trans, can be easily adapted to those settings. 
  - The zero-shot performance in Table 5 seems surprising to me. How do you split the table into three distinct sets? Do you do random split and how many random seeds have you tried? I would imagine a split that during the training, the model mostly sees Categorical and Binary features while during test it mainly sees Numerical features. In this way, I don't think the model is able to do zero-shot transfer. Moreover, can you try a setting that you manually control the number of categorical, binary, and numerical feature in both training and testing and see how does the model generalize?  
  - In addition to the quantitative results, I would also like to see some qualitative analysis of the transferability of the model. What does the data look like and why the model is able to do the transfer?
  - How does the self-supervised pretraining affect the performance? I would like to see an ablation study where you only training the model with the direct supervision signals. This could help understand how much of the improvement is from the architecture design and how much improvement is from the self-supervised pretraining.

- Clarity: In general, this paper is well-organized and easy to follow. I think section 2.4 could be better explained: what's the definition of $v_i^k$ in line 133? How do you compute $\psi$ in equation 4?

- Significance: This paper achieved strong results across a range of different datasets. Although the experiments are not comprehensive enough for the readers to understand every aspect of the system, I think it still sets a strong baseline and a good reference for the future work in this direction.
questions: see strengths and weaknesses for details
limitations: This paper has sufficiently addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a tabular learning framework that covers transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and finetuning. This approach does not assume that columns in the table are fixed and work even with variable column tables. The authors propose two Contrastive Learning-based pre-training approaches by vertically partitioning the tables. This pre-training approach is feasible since the columns can vary across tables, making self-supervised and supervised pre-training possible. The transformer model proposed performs significantly better in all the claimed settings (transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and fine-tuning). In addition, the authors also introduce *clinical trial mortality prediction* tabular dataset.
strengths_and_weaknesses: **Pros**
* The proposed contrastive learning methods are computationally cheaper.
* This variable column approach is really useful when the tables have too many columns and encoding them will be difficult in current existing transformers for tabular data (e.g. TaBERT).

**Cons**
* Setting for *Feature incremental learning* and *Transfer learning* seems very similar. (Dividing the dataset into three sets containing an equal number of columns and first training on set 1&2 then train on set 3 vs the transfer learning setting in the paper)
* line 214-215 is confusing (incomplete)
* In Feature incremental learning, no comparisons on how the performance on set1 after training on set1+set2; set1, set2, set1+set2 after training on set1+set2+set3. Will the performance on previous sets decrease? If yes, How to mitigate that?
questions: 1. How the partitions of set 1,2,3 are created in transfer learning, zero-shot inference, and feature incremental learning settings?
2. Are these sets created by randomly partitioning for every seed? Or the partition is fixed for all seeds?
3. Are subsets in feature incremental learning and zero-shot inference settings the same?
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the transferability of tabular data classification methods. It proposes three novel settings to evaluate the model transferability in terms of columns: column overlapping, column increment, and zero-shot. It also proposed a novel method combining self-supervised and supervised pre-training. 
strengths_and_weaknesses: ### Strength
* Three novel settings to evaluate the model transferability on tabular data classification. Transferability is an important research topic.
* A novel method based on (self-)supervised pre-training for tabular data classification which is more accurate and transferable.

### Weakness
* Incorrect claim in line 109: $E$ is not contextualized. To get contextualized embedding, the input embeddings should interact with each other, but not simply concatenization.
* Feature incremental learning setting is unclear.
* No baseline results (e.g. VIME and SCARF) for zero-shot setting.
* This paper assumes tables are matrix-like and column types are given, which hiders its transferability. Many papers in the NLP community have explored to process tables under a more flexible setting:
  * Wang et al. [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://arxiv.org/pdf/2205.03972). NAACL 2022
  * Yang et al. [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://arxiv.org/pdf/2203.00274.pdf). ACL 2022
  * Wang et al. [Retrieving complex tables with multi-granular graph representation learning](https://arxiv.org/pdf/2105.01736). SIGIR 2021
questions: * line 98: Why not assign an embedding vector when $x_b \neq 1$? Do you compare them?
* line 105: There is no value limit for $x_u$. Do you encounter any problem without scaling $E_{u}$?
* line 214: (5) -> (2) in Fig1?
limitations: * It would be meaningful to compare the methods focusing on numerical tables with those focusing on text tables.
review_title: null

Meta-review: metareview: This work introduces and evaluates a general scheme to feature-ize tabular data, and methods for (self-supervised) pre-training over the same, with a focus is on learning transferable representations. 

Reviewers were unanimous that the approach proposed constitutes a flexible, practical approach that borrows and brings together existing SOTA techniques. Some questions about the specific settings concerned in the evaluation (and distinctions between them) were sufficiently addressed during the response period. Empirical results show consistent gains over the baselines on the tasks considered.

An additional suggestion: one might naively anticipate that transfer learning for tables is not particularly promising given the very different semantics two arbitrary tables might have. However, the scenarios considered here involve settings in which transfer seems a priori reasonable; I might suggest the authors address this upfront, and explicitly outline the conditions under which transfer learning for tables is anticipated to work (and what assumptions are necessary for such cases), and where it is not. 
confidence: Certain
Decision: Accept