TransTab: Learning Transferable Tabular Transformers Across Tables

Abstract: Tabular data (or tables) are the most widely used data format in machine learning (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required to merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How to learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining on multiple distinct tables? How to train an ML model which can predict on an unseen table? 

To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to a generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input to a gated transformer model. The other insight is to introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, incremental feature learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3\% AUC lift on average over the supervised learning.

Reviews and author responses:

 Most of my queries/doubts/concerns are answered and I also increased the rating to 7.

 Thanks for addressing my questions and concerns and I am happy to raise my score to 7.

 We thank Reviewer THBY for the  helpful feedback. Besides our general response above, please see our specific response below.

**Line 109 claim**

> Incorrect claim in line 109: E is not contextualized. 

It is correct that $E$ is not contextualized in terms of feature level when applying interactions like attention. However, *TransTab* treats each feature in two parts: column name and cell value, e.g., value *20* under the column *age*. Here, when we concatenate *20* with *age*, the value is contextualized. Thanks for the suggestion and we rephrase this part in the new version.



**Feature incremental learning setting**

> Feature incremental learning setting is unclear.

We clarified the settings in the new version. Please refer to the response to **Reviewer teon**: **Feature incremental learning v.s. transfer learning**, **Line 214-215**, **Experiments regarding feature incremental learning**, and **Establishment of subsets**. We have updated the experiment descriptions in the new version.



**Baselines in zeroshot learning**

> No baseline results (e.g. VIME and SCARF) for zero-shot setting.

Although VIME and SCARF are all self-supervised methods, they are unable to deal with zero-shot prediction because they are trained either with reconstruction or with the contrastive objective. After pretraining, these models need an additional classification head and further supervised fine-tuning. These methods do not handle the case when we only have limited  labeled samples because pretraining and supervised learning are infeasible. Moreover, they need to be retrained every time when table structure varies.



**Related papers in NLP**

> This paper assumes tables are matrix-like and column types are given, which hiders its transferability. Many papers in the NLP community have explored to process tables under a more flexible setting:

Thank  for providing the literature on table retrieval and table-to-text generation. We will add them to the related works. Nonetheless, these works are on quite different tasks from ours. And it is unclear if they contribute to superior tabular prediction performances. Specifically, both table-text generation and table retrieval methods encode the whole table to an embedding while *TransTab* encodes each row of tables. We agree it is interesting future work to extend *TransTab* to more settings, e.g., nested table.



**Line 98 binary feature**

> line 98: Why not assign an embedding vector when $x_b \neq 1$? Do you compare them?

 We did consider the implementation that adds an indicator embedding for True or False in binary features. We compared it with the current method and found that did not achieve better results. Moreover, our current method significantly reduces computational cost for high-dimensional sparse tables because all False binary columns are not included in encoding.




**Line 105 value limit**

> line 105: There is no value limit for $x_u$. Do you encounter any problem without scaling $E_u$?

In practice, we scale $x_u$ by standardization or normalization such that $E_u$ is numerically stable. We added this detail in our new version on Line 105.



**Line 214 typo**

>line 214: (5) -> (2) in Fig1?

Thanks, we fixed this typo in the new version.



**More baselines**

> It would be meaningful to compare the methods focusing on numerical tables with those focusing on text tables.

This is an interesting future work, which probably needs additional methods to handle different data types and their semantic relations.
We could draw ideas from the literature on table generation, table retrieval, and table semantic parsing to tackle this new setting. But it is probably beyond the scope of this paper.


 We thank Reviewer teon for the helpful feedback. Besides our general response at the beginning, we provided additional details to address some specific comments.



**Feature incremental learning v.s. transfer learning**

> Setting for Feature incremental learning and Transfer learning seems very similar.

It is true that both applications are conceptually similar.  However, they are still different:

- For *feature incremental learning* when we have set1, set2, and set3 which built incrementally (set2 includes all columns in set1, set3 include all in set2), the object is to involve all three in the same round of training (w/ supervised loss) and enhance the prediction for set3. 
- For *transfer learning* when we have three equal-sized subsets (columns are assigned randomly into the three sets), the training process has two stages: pretraining + finetuning. In the first stage, the model is trained on set1+set2 using contrastive pretraining, namely VPCL in our paper (supervised or self-supervised); in the second, the model is trained on set3 only (supervised).

We discriminate between these two settings because they apply to different scenarios. Transfer learning based on VPCL is for learning from a wide range of data to build a foundation model good for adapting to downstream tasks; Feature incremental learning is for making the best of all data from the same domain.



**Line 214-215**

> line 214-215 is confusing (incomplete)

We rephrased and extended that part on the new version on Line 213-216. We split the raw dataset into three subsets: set1, 2, and 3. Baseline methods apply to two scenarios: (1) learning from all data that only have features of set1 and (2) learning from data from set3 only. We report the best of the two. *Transtab* applies to learning from all three subsets.



**Experiments regarding feature incremental learning**

> In Feature incremental learning, no comparisons on how the performance on set1 after training on set1+set2; set1, set2, set1+set2 after training on set1+set2+set3.

It is feasible to add those additional experiments, but the primary object of feature incremental learning is to enhance the performance on set3. We do not train *TransTab* stepwise on set1, 2, and then 3. Instead, all three sets are used in one training round simultaneously. The reviewer's proposal fits the target of transfer learning: improving performance on each dataset by learning across datasets. We refer the reviewer to Table 4 in the paper which illustrates the results of transfer learning and Table 6 that shows the average improvement led by contrastive pretraining on all datasets.



**Establishment of subsets**

> How the partitions of set 1,2,3 are created in transfer learning, zero-shot inference, and feature incremental learning settings? Are these sets created by randomly partitioning for every seed? Or the partition is fixed for all seeds? Are subsets in feature incremental learning and zero-shot inference settings the same?


For experiments in Sec. 3.2, 3.3, 3.4, we create subsets randomly with a fixed seed, respectively. That is, the subsets vary across these sections.

- Feature incremental learning. The columns are split into three distinct parts ${v_1,v_2,v_3}$. Set1 contains $v_1$, set2 contains $v_1,v_2$, and set3 has $v_1,v_2,v_3$. Three sets have an equal number of samples.
- Transfer learning. The columns are split into two parts $v_1,v_2$ where $v_1$ and $v_2$ have 50% of elements overlapped. Two sets have an equal number of samples.
- Zeroshot learning. The columns are split into three distinct parts ${v_1,v_2,v_3}$. Set1 contains $v_1$, set2 contains $v_2$, set3 contains $v_3$. Three sets have an equal number of samples.

We add this explanation to appendix D of the new version.

 **Ablation on self-supervised pretraining**

> How does the self-supervised pretraining affect the performance? I would like to see an ablation study where you only training the model with the direct supervision signals.

In Section 3.1, our method uses the direct supervision signals without a pretraining step, which might be the requested ablation study. We further discussed the benefit of pretraining in Section 3.5. In particular, the proposed VPCL pretraining generally leads to better performance on clinical trial mortality prediction datasets. We clarified these settings in the experiment section, on Line 203.



**Clarity**

> I think section 2.4 could be better explained: what's the definition of vik in line 133? How do you compute ψ in equation 4?

As defined in Line 133, $v$ are vertical partitions of the table, e.g., $v_1$ are rows under column 1, $v_2$ are rows under column2, and so on. In Line 137, $\psi$ is the cosine similarity of two vectors. We improved Sec 2.4 to address these writing issues in the new version.


 We thank Reviewer MTvW for the helpful feedback to our work. We addressed most of Reviewer MTvW's comments in our general response above. And we will provide additional responses to specific comments below.



**Justification of phrases**

> In the paper "existing works only cover vanilla supervised learning and fixed-table pretraining due to the fixed-column assumption." In my point of view, this is overclaiming. Not all existing works only cover vanilla supervised learning.

We agree that our original sentence may cause misunderstanding. We intend to illustrate that due to the fixed-column assumption, most existing works only handle supervised learning or pretraining on the same-structure tables. We rephrased that in the new version.



**Zeroshot prediction** 

> The zero-shot performance in Table 5 seems surprising to me. How do you split the table into three distinct sets? Do you do random split and how many random seeds have you tried? ... Moreover, can you try a setting that you manually control the number of categorical, binary, and numerical feature in both training and testing and see how does the model generalize?

In the zero-shot learning experiments, we split the table columns into three equal-sized subsets where columns are 50\% mutually overlapped (e.g., set1 and set2 have 50\% same columns, set2 and set3 have 50\% same columns). Likewise, each subset has 1/3 number of samples. We follow the setting specified in Table 2 caption. All the results are averaged over 10 runs with different random seeds. During this process, the subset columns are fixed but train/test splits are changed. We add more experiments to dive deep into the zero-shot prediction capability of our method. To be specific, we split the dataset into two subsets with no sample overlaps and test zero-shot performance with column overlap ratio varying from 0 (non-overlap) to 1. 
The corresponding figure is added in the new version (Fig. 6 in Appendix). Nonetheless, we agree with ZSL that the problem is very challenging when the training tables and testing tables are highly mismatched. 


| AUC\overlap ratio | 0      | 0.2    | 0.5    | 0.8    | 1.0    |
| ----------------- | ------ | ------ | ------ | ------ | ------ |
| credit-g          | 0.5584 | 0.5740 | 0.6241 | 0.6441 | 0.7612 |
| credit-a          | 0.8118 | 0.8128 | 0.8583 | 0.8621 | 0.8697 |
| dress-s           | 0.5640 | 0.5663 | 0.5847 | 0.5740 | 0.7011 |
| cylinder-b        | 0.5279 | 0.5461 | 0.6657 | 0.6509 | 0.6550 |



**Qualitative analysis of the transferability**

> In addition to the quantitative results, I would also like to see some qualitative analysis of the transferability of the model. 

We added some case studies and discussed why transfer learning and zero-shot prediction across them are feasible.  Two clinical trial datasets are illustrated below. We observe several shared columns across the two datasets. Moreover, there are columns named differently but sharing similar meanings, e.g., "adverse effect: infection" and "adverse effect: infection without neutropenia(specify)". *TransTab* can be effective in both scenarios.   

|      | adverse effect: nausea | adverse effect: vomiting | adverse effect: asthenia | adverse effect: infection |
| ---- | ---------------------- | ------------------------ | ------------------------ | ------------------------- |
| 0    | 0                      | 0                        | 0                        | 0                         |
| 1    | 0                      | 0                        | 0                        | 0                         |
| 2    | 0                      | 0                        | 0                        | 0                         |

|      | adverse effect: febrile neutropenia | adverse effect: infection (documented clinically) | adverse effect: infection without neutropenia(specify) |
| ---- | ----------------------------------: | ------------------------------------------------: | -----------------------------------------------------: |
| 0    |                                   0 |                                                 0 |                                                      0 |
| 1    |                                   0 |                                                 0 |                                                      0 |
| 2    |                                   0 |                                                 0 |                                                      0 |




 We thank the reviewers for their thoughtful and constructive reviews. Most comments center around the settings and experiments of **feature incremental learning** and **zero-shot learning**. In response, we have updated our manuscript to include a detailed explanation of the experiment settings. We also ran additional experiments to test the sensitivity of zero-shot learning w.r.t. the overlapping ratio of table columns.

Here is the change summary:

- Sec. 2.2, L105: added a footnote to explain how to avoid the numerical issue when encoding numerical features.
- Sec 2.2, L109: rephrased the concept w.r.t. the cell embedding contextualization to avoid misunderstandings.
- Sec 2.4, L133: further explained $v_i^k$ in $x_i$.
- Sec 2.4, L137: further explained the similarity function $\psi$.
- Sec 3.1, L203: further explained the supervised learning setting.
- Sec 3.2, L213-L216: rephrased the experiment settings of feature incremental learning.
- Sec 3.4, L242-L244: added an experiment to check the sensitivity of zero-shot learning w.r.t. the overlapping ratio of columns of two tables.
- Sec 4, L287: added related works suggested by Reviewer THBY and discussed the difference to our method.
- Appendix D, L592: added an introduction on how to build the subsets for three experiments (Sec 3.2, 3.3, 3.4).
- Appendix, Fig.6: added figures for the sensitivity evaluation of zero-shot prediction.

 This paper proposed to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. They basically convert each row into a feature vector and then apply stacked transformers for feature encoding. There are several advantages of this encoding: (1) it can deal with the tables that have different number of columns; (2) it is easier to transfer the knowledge learned from different columns. They conduct experiments on one clinical dataset and several public datasets under four different settings: supervised learning, feature incremental learning, transfer learning, and zero-shot learning. The empirical results show that the proposed approach outperform the baselines in the literature. They also showed that in the zero-shot learning scenario, they can almost match the performance of pretraining plus fine-tuning. - Originality: The main idea in this paper is a good combination of several ideas proposed in the literature. With modifications and adaptations, it worked and yield promising results on several datasets.

- Quality: The proposed approach is technically sound and the experimental results showed that it outperformed several strong baselines for tabular data prediction. Although the results are impressive, I have several comments:
  - One of the main advantages advertised in the paper is that the proposed method could easily extend to feature incremental learning, pretraining+finetuning, and zero-shot inference. In the paper "existing works only cover vanilla supervised learning and fixed-table pretraining due to the fixed-column assumption." In my point of view, this is overclaiming. Not all existing works only cover vanilla supervised learning. For example, those transformer-based architectures like TabTrans, FT-Trans, can be easily adapted to those settings. 
  - The zero-shot performance in Table 5 seems surprising to me. How do you split the table into three distinct sets? Do you do random split and how many random seeds have you tried? I would imagine a split that during the training, the model mostly sees Categorical and Binary features while during test it mainly sees Numerical features. In this way, I don't think the model is able to do zero-shot transfer. Moreover, can you try a setting that you manually control the number of categorical, binary, and numerical feature in both training and testing and see how does the model generalize?  
  - In addition to the quantitative results, I would also like to see some qualitative analysis of the transferability of the model. What does the data look like and why the model is able to do the transfer?
  - How does the self-supervised pretraining affect the performance? I would like to see an ablation study where you only training the model with the direct supervision signals. This could help understand how much of the improvement is from the architecture design and how much improvement is from the self-supervised pretraining.

- Clarity: In general, this paper is well-organized and easy to follow. I think section 2.4 could be better explained: what's the definition of $v_i^k$ in line 133? How do you compute $\psi$ in equation 4?

- Significance: This paper achieved strong results across a range of different datasets. Although the experiments are not comprehensive enough for the readers to understand every aspect of the system, I think it still sets a strong baseline and a good reference for the future work in this direction. see strengths and weaknesses for details This paper has sufficiently addressed the limitations.

 This paper presents a tabular learning framework that covers transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and finetuning. This approach does not assume that columns in the table are fixed and work even with variable column tables. The authors propose two Contrastive Learning-based pre-training approaches by vertically partitioning the tables. This pre-training approach is feasible since the columns can vary across tables, making self-supervised and supervised pre-training possible. The transformer model proposed performs significantly better in all the claimed settings (transfer learning across tables, zero-shot inference, feature incremental learning, pre-training, and fine-tuning). In addition, the authors also introduce *clinical trial mortality prediction* tabular dataset. **Pros**
* The proposed contrastive learning methods are computationally cheaper.
* This variable column approach is really useful when the tables have too many columns and encoding them will be difficult in current existing transformers for tabular data (e.g. TaBERT).

**Cons**
* Setting for *Feature incremental learning* and *Transfer learning* seems very similar. (Dividing the dataset into three sets containing an equal number of columns and first training on set 1&2 then train on set 3 vs the transfer learning setting in the paper)
* line 214-215 is confusing (incomplete)
* In Feature incremental learning, no comparisons on how the performance on set1 after training on set1+set2; set1, set2, set1+set2 after training on set1+set2+set3. Will the performance on previous sets decrease? If yes, How to mitigate that? 1. How the partitions of set 1,2,3 are created in transfer learning, zero-shot inference, and feature incremental learning settings?
2. Are these sets created by randomly partitioning for every seed? Or the partition is fixed for all seeds?
3. Are subsets in feature incremental learning and zero-shot inference settings the same? NA

 This paper focuses on the transferability of tabular data classification methods. It proposes three novel settings to evaluate the model transferability in terms of columns: column overlapping, column increment, and zero-shot. It also proposed a novel method combining self-supervised and supervised pre-training.  ### Strength
* Three novel settings to evaluate the model transferability on tabular data classification. Transferability is an important research topic.
* A novel method based on (self-)supervised pre-training for tabular data classification which is more accurate and transferable.

### Weakness
* Incorrect claim in line 109: $E$ is not contextualized. To get contextualized embedding, the input embeddings should interact with each other, but not simply concatenization.
* Feature incremental learning setting is unclear.
* No baseline results (e.g. VIME and SCARF) for zero-shot setting.
* This paper assumes tables are matrix-like and column types are given, which hiders its transferability. Many papers in the NLP community have explored to process tables under a more flexible setting:
  * Wang et al. [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://arxiv.org/pdf/2205.03972). NAACL 2022
  * Yang et al. [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://arxiv.org/pdf/2203.00274.pdf). ACL 2022
  * Wang et al. [Retrieving complex tables with multi-granular graph representation learning](https://arxiv.org/pdf/2105.01736). SIGIR 2021 * line 98: Why not assign an embedding vector when $x_b \neq 1$? Do you compare them?
* line 105: There is no value limit for $x_u$. Do you encounter any problem without scaling $E_{u}$?
* line 214: (5) -> (2) in Fig1? * It would be meaningful to compare the methods focusing on numerical tables with those focusing on text tables.

Meta-review: This work introduces and evaluates a general scheme to feature-ize tabular data, and methods for (self-supervised) pre-training over the same, with a focus is on learning transferable representations. 

Reviewers were unanimous that the approach proposed constitutes a flexible, practical approach that borrows and brings together existing SOTA techniques. Some questions about the specific settings concerned in the evaluation (and distinctions between them) were sufficiently addressed during the response period. Empirical results show consistent gains over the baselines on the tasks considered.

An additional suggestion: one might naively anticipate that transfer learning for tables is not particularly promising given the very different semantics two arbitrary tables might have. However, the scenarios considered here involve settings in which transfer seems a priori reasonable; I might suggest the authors address this upfront, and explicitly outline the conditions under which transfer learning for tables is anticipated to work (and what assumptions are necessary for such cases), and where it is not. 

Decision: Accept