Transfer Learning with Deep Tabular Models
Authors: Roman Levin
Valeriia Cherepanova
Avi Schwarzschild
Arpit Bansal
C. Bayan Bruss
Tom Goldstein
Andrew Gordon Wilson
Micah Goldblum
Venue: ICLR 2023 poster (2023)
Keywords: tabular data
transfer learning
tabular models
representation learning

Abstract: Recent work on deep learning for tabular data demonstrates the strong performance of deep tabular models, often bridging the gap between gradient boosted decision trees and neural networks. Accuracy aside, a major advantage of neural models is that they are easily fine-tuned in new domains and learn reusable features. This property is often exploited in computer vision and natural language applications, where transfer learning is indispensable when task-specific training data is scarce. In this work, we explore the benefits that representation learning provides for knowledge transfer in the tabular domain. We conduct experiments in a realistic medical diagnosis test bed with limited amounts of downstream data and find that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods. We further compare the supervised and self-supervised pretraining strategies and provide practical advice on transfer learning with tabular models. Finally, we propose a pseudo-feature method for cases where the upstream and downstream feature sets differ, a tabular-specific problem widespread in real-world applications.

Published as a conference paper at ICLR 2023
TRANSFER LEARNING WITH DEEP TABULAR MODELS
Roman Levin 1
Valeriia Cherepanova" Avi Schwarzschildd 12
Arpit Bansal
C. Bayan Bruss
Tom Goldstein Andrew Gordon Wilson*
Micah Goldblum
University of Washington
University of Maryland
SCapital One
NNew York University
ri rilevineuw. edu, rcherepa@umd.eed
ABSTRACT
Recent work on deep learning for tabular data demonstrates the strong performance of deep tabular models, often bridging the gap between gradient boosted decision trees and neural networks. Accuracy aside, major advantage of neural models is that they are easily fine-tuned in new domains and learn reusable features. This property is often exploited in computer vision and natural language applications, where transfer learning is indispensable when task-specific training data is scarce. In this work, explore the benefits that representation learning provides for knowledge transfer in the tabular domain. We conduct experiments in a realistic medical diagnosis test bed with limited amounts of downstream data and find that transfer learning with deep tabular models provides definitive advantage over gradient boosted decision tree methods. We further compare the supervised and self- supervieed pre- trainiriing strategies and provide practical advice on transfer learning with tabular models. Finally, we propose a pseudo-feature method for cases where the upstream and downstream feature sets differ, a tabular-specific problem widespread in real-world applications.
INTRODUCTION
Tabular data is ubiquitous throughout diverse real-world applications, spanning medical diagnosis (Johnson et al. (2016), housing price prediction (Afonso et al. 2019). loan approval (Arun et al.) 2016).
and robotics (Wienke et al. 2018). yet practitioners still rely heavily on classical machine learning systems. Recently, neural network architectures and training routines for tabular data have advanced significantly. Leading methods in tabular deep learning (Gorishniy et al.] [02TT] 2022] [Somepall] et al.] 2021 Kossen et al. (2021) now perform par with the traditionally dominant gradient boosted decision trees (GBDT) (Friedman] 20001: Prokhorenkova et al.] 2018 [hen and Guestrin 2016) Ke et al. 2017). On top of their competitive performance, neural networks, which are end-to-end differentiable and extract complex data representations, possess numerous capabilities which decision trees lack; one especially useful capability is transfer learning, in which representation learned on pre-training data is reused or fine-tuned on one or more downstream tasks.
Transfer learning plays central role in industrial computer vision and natural language processing pipelines, where models learn generic features that are useful across many tasks. For example, feature extractors pre-trained on the ImageNet dataset can enhance object detectors (Ren et al. 2015). and large transformer models trained on vast text corpora develop conceptual understandings which can be readily fine-tuned for question answering language inference (Devlin al. 2019). One might wonder if deep neural networks for tabular data, which are typically shallow and whose hierarchical feature extraction unexplored, can also build representations that are transferable beyond their pre-training tasks. In fact, a recent survey paper on deep learning with tabular data suggested that efficient knowledge transfer in tabular data an open research question (Borisov et al., 202T). In this work, we show that deep tabular models with transfer learning definitively outperform their classical counterparts when auxiliary upstream pre-training data is available and the amount of downstream data is limited. Importantly, we find representation learning with tabular neural networks to be more powerful than gradient boosted decision trees with stacking a strong baseline leveraging knowledge transfer from the upstream data with classical methods.
Equal contribution. This work was completed prior to the author joining Amazon

---

Published as a conference paper at ICLR 2023
Some of the most common real-world scenarios with limited data medical applications. Accumu- lating large amounts of patient data with labels is often very difficult, especially for rare conditions or hospital-speciin
tasks. However, large related datasets, e.g. for more common diagnoses, may be available in such cases. We note that while computer vision medical applications are common (Irvin et al. 2019; [Santa Cruz et al., [202T] Chen al.) [2018)] Tuubéé al.
2021). many medical
datasets are fundamentally tabular (Goldberger et al. |0000 Johnson et al., 2021 2016 Law and Liu 2009). Motivated by this scenario, we choose a realistic medical diagnosis test bed for our experiments both for its practical value and transfer learning suitability. We first design a suite of benchmark transfer learning tasks using the MetaMIMIC repository (Grzyb et al.) 2021 Wožnical et al. 2022) and use this collection of tasks to compare transfer learning with prominent tabular models and GBDT methods at different levels of downstream data availability. We explore several transfer learning setups and lend suggestions to practitioners who may adopt tabular transfer learning. Additionally, we compare supervised pre- pre-training and self-supervised pre-traue pre-training strategies and find that supervised pre-training leads to more transferable features in the tabular domain, contrary to findings in vision where a mature progression of self-supervised methods exhibit strong performance (He et al. 2020).
Finally, we propose a pseudo-featurr method which enables transfer learning when upstream and downstream feature sets differ. As tabular data is highly heterogeneous, the problem of downstream tasks whose formats and features differ from those of upstream data is common and has been reported to complicate knowledge transfer (Lewinson 2020). Nonetheless, if our upstream data is missing columns present in downstream data, we still want to leverage pre-training. Our approach uses transfer learning in stages. In the case that upstream data is missing a column, first pre-train a model on the upstream data without that feature. We then fine-tune the pre-trained model on downstream data to predict values in the column absent from the upstream data, Finally, after assigning pseudo-values of this feature to the upstream samples, we re-do the pre-training and transfer the feature extractor to the downstream task. This approach offers appreciable performance boosts over discarding the missing features and often performs comparably models pre-trained with the ground truth feature values.
Our contributions are summarized follows:
1. We find that recent deep tabular models combined with transfer learning have a decisive advantage over strong GBDT baselines, even those that also leverage upstream data.
2. We compare supervised and self-supervised pre-training strategies and find that the super- vised pre-training leads to more transferable features in the tabular domain.
3. We propose pecudo-feature method for aligning the upstream and downstream feature sets in heterogeneous data, addressing a common obstacle in the tabular domain.
4. We provide advice for practitioners on architectures, hyperparameter tuning, and transfer learning setups for tabular transfer learning.
2 RELATED WORK
Deep learning for tabular data. The field of machine learning for tabular data has traditionally been dominated by gradient-boosted decision trees (Friedman, 2001 Chen and Guestrin 2016 Ke et al.) 2017: Prokhorenkova et al. 2018). These models are used for practical applications across domains ranging from finance to medicine and are consistently recommended as the approach of choice for modeling tabular data (Shwartz-Ziy Ziv and Armon 2022
An extensive line of work on tabular deep learning aims challenge the dominance of GBDT models. Numerous tabular neural architectures have been introduced, based on the ideas of creating differen- tiable learner ensembles (Popov et al., 2019 Hazimeh et al.) 20200 Yang et al.) 2018 Kontschieder
et al. 2015 Badirli et al.) 2020). incorporating attention mechanisms and transformer architectures
(Somepalli et al.] 202 Gorishniy et al. 2021 Arik and Pfister 2021 Huang et al.
2020 Song
et al. 2019 Kossen et al. 2021). as well as a variety of other approaches (Wang et al.
2017 2021
Beutel et al. 22018] K Klambauer et al. 2017 Fiedler 2021: Schäfl et al.
2021). }. However, recent
systematic benchmarking of deep tabular models (Gorishniy et al. 2021 Shwartz-Ziv and Armon
2022) shows that while these models are competitive with GBDT on some tasks, there is still no
universal best method. Gorishniy et al. (021) show that transformer-based models are the strongest
2

---

Published as a conference paper at ICLR 2023
alternative to GBDT and that ResNet and MLP models coupled with strong hyperparameter tuning routine (Akiba et al. 2019) offer competitive baselines. Similarly, Kadra et al. (202I) and Fiedler (202I) find that carefully regularized and slightly modified MLPs are competitive well other regularized architectures (Yang et al. 2022). In a follow-up work, Gorishniy et al. (2022) show that transformer architectures equipped with advanced embedding schemes for numerical features bridge the performance gap between deep tabular models and GBDT.
Transfer learning. Transfer learning (Pan and Yang 2010 Weiss al. 2016 Zhuang al., 2020
has been incredibly successful in domains of computer vision and natural language processing (NLP).
Large fine-tuned models excel on variety of image classification (Dosovitskiy et al., 2020 Dai
et al.
202T and NLP benchmarks (Devlin et al.) 2019; Howard and Ruder 2018). ImageNer (Deng
jt al.
2009) pre-trained feature extractors are incorporated into the complex pipelines of successful
object detection and semantic segmentation models (Chen et al. 201 Ren et al.] [015: Redmon
and Farhadi 2018 Redmon et al. 2016). Transfer learning is also particularly helpful in applications
with limited data availability such as medical image classification (Alzubaidi al. 2021 Heker and
Greenspan 2020 Chen et al. 2019 Alzubaidi et al. 2020).
In the tabular data domain, recent review paper (Borisov et al. 2021) finds that transfer learning is underexplored and that the question of how to perform knowledge transfer and leverage upstream data remains open. In our work. seek to answer these questions through systematic study of transfer learning with recent successful deep tabular models.
Multiple works mention that transfer learning in the tabular domain is challenging due the highly heterogeneous nature of tabular data (Jain et al. [021]]]] Lewinson 2020). Several papers focus on converting tabular data images instead (Sharma et al. 2019 Zhu al. [021 Sun et al. 2019) and
leveraging transfer learning with vision models (Sun et al. 2019). Other studies explore designing
CNN-like inductive biases for tabular models (Joffe 2021). transferring XGBoost hyperparameters
(Wožnica et al. [2222] Grzyb et al. (202I) and transferring whole models (Fang et al. 2019 Al-Stouhi and Reddy 2011 Li al. 2021) in the limited setting of shared label and feature space between the
upstream and downstream tasks. Additionally, a recent work by Wang and Sun (2022) proposes a variable-column tabular neural network architecture and applies it to learn information from multiple tables with overlapping columns. While TransTai Tab operates tables with shared label space, our work focuses on exploring how representation learning can be used to transfer knowledge between related, but different tasks.
Stacking could also be seen as a form of leveraging upstream knowledge in classical methods (Wolpert 1992. Ting and Witten 1997b.
Self-supervised learning. Self-supervised learning (SSL) aimed at harnessing unlabelled data through learning its structure and invariances has accumulated large body of works over the last few years. Prominent SSL methods, such as Masked Language Modeling (MLM) (Devlin et al. (2019) in NLP and contrastive pre-training in computer vision (Chen et al., (020) have revolutionized their fields making SSL the pre-training approach of choice (Devlin et al. [219 Lan et al., 2019 Liu
et al. 2019 Lewis al. 2019 Chen al., 2020 [e et al.) [020 Caron al.) [020 Bardes et al.
2021
Misra and Maaten 2020 In fact, SSL pre-training in vision has been shown to produce more
transferable features than supervised pre-training on ImageNet (He et al. 2020).
Recently, SSL has been adopted in the tabular domain for semi-supervised learning Yin et al.
2020: Yoon et al., 2020 Ucar et al. [002]] Somepalli et al., 202T Huang et al., 2020). Contrastive
pre- tre-training on auxilary unlabelled data (Somepalli et al.) (021) and MLM-like approaches (Huang
ct al. 2020) have been shown to provide gains over training from scratch for transformer tabular architectures in cases of limited labelled data. Additionaly, Rubachev et al. (2022) investigate benefits of supervised and unsupervised pre-training when applied to the same data without transferring knowledge between tasks. Finally, self-supervised pre-training successfully applied to time- series data in medical domain (Yèche et al. 2021 McDermott et al. 202I).
3 TRANSFER LEARNING SETUP IN TABULAR DOMAIN
To study transfer learning in the tabular domain, need to choose benchmark tasks and training pipelines. In this section, we detail both our upstream -ownssteam datasets as well as the tools we use to optimize transfer learning for tabular data.
3

---

Published as a conference paper at ICLR 2023
Pre-training
Upstream Data
Pulse
Weight
Mulli-Label Classification
Pattent
Heart
Anemia
130
80
155
2000
78
165
2009
0
Fine-tuning
Downstream Data
Patieal
Pulse
Weight
Patint
Perpara
37
143
New Target
74
136
145
180
Figure 1: Tabular transfer learning pipeline with MetaMIMIC. We pre-tral pre-train deep tabular neural networks on abundant upstream patient data with 11 diagnosis targets via multi-label classification. Then, fine-tune the pre-trained models limited downstream data with similar features predict the new target diagnosis.
3.1 METAMIMIC TEST BED FOR TRANSFER LEARNING EXPERIMENTS
MetaMIMIC repository. As medical diagnosis data often contains similar test results (i.e. features) across patients, and some diseases (i.e. tasks) are common while others are rare, this setting is a realistic use-case for our work. We thus construct a suite of transfer learning benchmarks using the MetaMIMIC repository Grzyb et al. (2021); Wožnica et al. (2022) which is based on the MIMIC-IV Johnson et al. (2021); Goldberger et al. (2000) clinical database of anonymized patient data from the the Beth Israel Deaconess Medical Center ICU admissions. MetaMIMIC contains 12 binary prediction tasks corresponding different diagnoses such as hypertensive diseases, ischematic heart disease, diabetes, alcohol dependence and others. It covers 34925 patients and 172 features, including one categorical feature (gender), related to the medical examination of patients hand-selected by authors to have the smallest number of missing values (Grzyb al. [021 Wožnica et al. [2022). The features include mean, maximum and minimum statistics lab test results as well as general features such height, weight, age and gender. The 12 medical diagnosis targets related tasks of varied similarity and make MetaMIMIC a perfect test bed for transfer learning experiments.
Upstream and downstream tasks. A medical practitioner may possess abundant annotated diagnosis data corresponding to number of common diseases and want to harness this data to assist in diagnosing another disease, perhaps which is rare or for which data is scarce. To simulate this scenario, we create transfer learning problems by splitting the MetaMIMIC data into upstream and downstream tasks. Specifically, we reserve targets for the upstream pre- traininiinn tasks and one target for the downstream fine-tuning tasks, thus obtaining 12 upstream-downstreenn downstrean splits one for each downstream diagnosis. Additionally, we limit the amount of downstream data to 10, 20, 100, and 200 samples corresponding to several scenarios of data availability.
In total, we have 60 combinations of upstream and downstream datasets for our transfer learning experiments. We pre-train our models as multi-label classifiers the upstream datasets with 11 targets and then transfer the feature extractors onto the downstream binary diagnosis tasks, Figure presents diagram illustrating the pipeline.
3.2 TABULAR MODELS
We conduct transfer learning experiments with four tabular neural networks, and we compare them to two GBDT implementations.
For neural networks, we use transformer based architectures found to be the most competitive with GBDT tabular approaches (Gorishniy al.] [222]] HHung et al.) 2020 Gorishniy et al. 2022). The specific implementations we consider include the recent FT- Prannsosfrrmer (Gorishniy et al. [202]] and
Tabnforsfrrmrr (Huang et al. 2020). We do not include implementations with inter-sample attention
4

---

Published as a conference paper at ICLR 2023
(Somepalli al. 2 Kossen et al. 2021) in experiments since these do not lend themselves to scenarios with extremely limited downstream data. In addition, we use MLP and ResNet tabular architectures as they known to be consistent and competitive baselines (Gorishniy et al. 2021).
The Tranfoosfrrm architecture comprises of an embedding layer for categorical features, a stack of transformer layers and multi-layer perceptron applied concatenation of processed categorical features and normalized numerical features. In contrast, FT-Transformer architecture transforms all features (including numerical) embeddings and applies stack of Transformer layers the embeddings.
For GBDT implementation, libraries (Chen and Guestrin 2016).
the popular Catboost (Prokhorenkova et al. 2018) and XGBoost
3.3 TRANSFER LEARNING SETUPS AND BASELINES
In addition to range of architectures, we consider several setups for transferring the upstream pre-trained neural feature extractors onto downstream tasks. Specifically, we use either single linear layer or a two-layer MI.P with 200 neurons in each layer for the classification head. We also either freeze the weights of the feature extractor or fine-tune the entire model end-to-end. To summarize, we implement four transfer learning setups for neural networks:
Linear head atop frozen feature extractor
MIPP head atop frozen feature extractor
End-to-end fine-tuned feature extractor with a linear head
End-to-end fine-tuned feature extractor with an MLP head
We compare the above setups to the following baselines:
Neural models trained from scratch on downstream data
CatBoost and XGBoost with and without stacking
We use stacking for GBDT models to build stronger baseline which leverages the upstream data (Wolpert 1992; Ting and Witten 1997p. To implement stacking, we first train upstream GBDT models to predict the 1I upstream targets and then concatenate their predictions to the downstream data features when training downstream GBDT models.
3.4 HYPERPARAMETER TUNING
The standard hyperparameter tuning procedure for deep learning is to randomly sample validation set and use to optimize the hyperparameters. In contrast, in our scenario we often have too little downstream data to afford sizeable validation split. However, upstream data and leverage hyperparameter transfer which is known to be effective for GBDT (Wožnica al. 2022 Grzyb al. 2021).
make use of the abundant
We tune the hyperparameters of each model with the Optuna library (Akiba et al., 2019) using Bayesian optimization. In particular, for GBDT models and neural baselines trained from scratch, we tune the hyperparameters on a single randomly chosen upstream target with the same number of training samples available in the downstream task, since hyperparameters depend strongly on the sample size. The optimal hyperparameters are chosen based on the upstream validation set, where validation data is plentiful. We find this tuning strategy to be especially effective for GBDT and provide comparison with default hyperparameters in Appendix C The benefits of this hyperparameter tuning approach are less pronounced for deep baselines.
For deep models with transfer learning, we tune the hyperparameters on the full upstream data using the available large upstream validation set with the goal to obtain the best performing feature extractor for the pre-training multi-target task. We then fine-tune this feature extractor with small learning rate on the downstream data. As this strategy offers considerable performance gains over default hyperparameters, we highlight the importance of tuning the feature extractor and present the comparison with default ayperparameters in Appendix C as well as the details on hyperparameter search spaces for each model.
5

---

Published as a conference paper at ICLR 2023
Fi-Transformer
ResNet
MLP
Catboost
XGBoost
200
11.1
5.2
15.8
15.2
8.2
16.0
11.2
10.2
8.6
16.1
9.8
9.8
11.3 9,9
Num sumples
100-
11.9
5.5
6.1
14.8
6.2
13.1
12.3
6.3
15.1
6.2
12.2
13.3
8.9
15.0
7.8
10.2
6.4
9.2
7.7
8
20
11.6
7.8
10.3
7.4
13.2
12.1
8.8
11.6
5.9
13.5
10.1
5.5
10.8
6.1
13.9
10.8
10.8
10.0
10
11.8
8.2
9.2
13.2
10.4
7.4
10.7
7.6
15.7
9.5
4.4
9.2
4.8
13.4
10.6
12.2
11.7
12
14
4-
10.9
7.4
7.9
9.6
9.2
13.9
9.0
7.5
10.2
8.2
14.7
8.0
3.8
10.8
4.6
11.5
10.0
15.2
15.0
FS
LH-EZ
MLP-EZE
LH
MLP
FS
LI-ED
ML. MILP-EZE
LH
MLP
FS
LHEEE
MLP-E3E
LH
MLP
Fi
Stacking
PS
Stacking
16
Figure 2: Average model ranks across all downstream tasks. Deep tabular models and GBDT performance is presented the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to-end training. Rank is averaged across all downstream tasks. FI-Transformer fine-tuned end-to-end outperforms all GBDT models, including GBDT with stacking, at all data levels. MLP is highly competitive in low data regimes.
RESULTS FOR TRANSFER LEARNING WITH DEEP TABULAR MODELS
In this section, we compare transfer learned deep tabular models with GBDT methods varying levels of downstream data availability. We note that here we present the aggregated results in the form of the average rank of the models across all of the twelve downstream tasks at each data level. We choose this rank aggregation metric since it does not allow a small number of high-variance tasks to dominate comparisons, unlike, for example, average accuracy. Ranks are computed taking into account statistical significance of the performance differences between the models. We further report the detailed results for all the models on datasets and results for Tabrransformer in Appendix D
Figure 2 presents average model ranks on the downstream tasks heatmap where the warmer colors represent better overall rank. The performance of every model is shown on the dedicated panel of the heatmap with the results for different transfer learning setups presented in columns. First, noting the color pattern in the Catboost and XGBoost columns, we observe that deep tabular models pre-trained on the upstream data outperform GBDT all data levels and especially in the low data regime of 4-20 downstream samples.
We emphasize that knowledge transfer with stacking, while providing strong boosts compared to from-scratch GBDT training (see Stacking and FS columns of GBDT), still decisively falls behind the deep tabular models with transfer learning, suggesting that representation learning for tabular data is significantly more powerful and allows neural networks to transfer richer information than simple predictions learned on the upstream tasks.
We summarize the main practical takeaways of Figure 2 below:
Simpler models such MLP with transfer learning are competitive, especially in extremely low data regimes. More complex architectures like FT-TTranfformer offer consistent perfor- mance gains over GBDT across all data levels and reach their peak performance in higher data regimes.
Representation learning with deep tabular models provides significant gains over strong GBDT baselines leveraging knowledge transfer from the upstream data through stacking. The gains are especially pronounced in low data regimes.
Regarding transfer learning setups, find that using MLP head with trainable or frozen feature extractor is effective for all deep tabular models. Additionally, a linear head with an end-to-end fine-tuned feature extractor is competitive for FT-Transformer..

---

Published as a conference paper at ICLR 2023
FT-Transfomer+Li--B22
FT-Transfomerr+MLPP
200
2.2
3.9
1.8
1.1
1.8
3.8
2.1
1.6
Num samples
100
2.5
3.8
1.9
1.2
2.2
3.7
2.2
1.4
Rank
20
2.9
3.1
1.9
1.5
2.8
2.8
1.9
1.3
10
2.0
1.9
1.9
1.8
2.4
2.0
1.5
2.5
1.9
2.3
1.8
1.8
2.0
2.0
2.2
FS
MLM
Contastivv
Supervised
FS
MLM
Contrastive
Supavised
Figure 3: Comparison of supervised and sulf-supe pre-training strategies for FT- Transformer The left panel illustrates end-to-end fine-tuning with linear head and the right plot illustrates end-to-end fine-tuning with MLP head, the two most effective strategies for FT- TT-rrssfformer. Within each panel, columns represent pre-training strategies and rows correspond to the number of available downstream samples. FS stands for training from scratch (without pre-training on upstream data), MLM and Contrastive denote pre-training strategies, and Supervised denotes supervised pre-training on upstream data. Warmer colors indicate better performance. Contrastive pre-training outperforms while MLM pre-training is not effective. Supervised pre-training outperforms all self-supervised pre-training strategies our experiments.
and contrastive self-supervised
trained models,
To verify that transfer learning with deep tabular models works well beyond the medical domain, we conduct experiments other datasets: Yeast functional genomics data from the biological sciences domain (Flisseeff and Weston (001) and Emotions data from the music domain (Trohidis et al.) 2008). Both datasets are multilabel, and treat each classification label separate task. Similarly to the experiments with MetaMIMIC, we split the tasks into downstream and upstream by reserving 1 labels for the upstream task and the 1--h label for the downstream task. We report results of these experiments in Figures I1 in Appendix D.6 We observe similar trends and in particular that deep tabular models mre-trained on upstream data and finetuned with MLP head outperform deep baselines trained from scratch and Catboost models leveraging stacking.
SELF-SUPERVISED PRE-TRAINING
In domains where established SSL methods are increasingly dominant, such as computer vision, self-supervised learners are known to extract more transferable features than models trained on labelled data (He et al.) 2020 (02T). In this section, we compare supervised pre-training with unsupervised pre-training and find that the opposite true in the tabular domain. We use the Masked Language Model (MLM) pre- pre-training recently adapted to tabular data (Huang et al., 2020) and the tabular version of contrastive learning (Somepalli et al., 2021). Since both methods were proposed for tabular transformer architectures, The inferior performance of self-supervised pre- pre-training might be a consequence of the fact that SSL is significantly less explored and tuned in the tabular domain than in vision or NLP.
conduct the experiments with the FT-Transformer model.
5.1 TABULAR MLM PRE-TRAINING
Masked Language Modeling (MLM) was first proposed for language models by Devlin et al. (0199) as powerful unsupervised learning strategy. MLM involves training model to predict tokens in text masked at random so that its learned representations contain information useful for reconstructing these masked tokens. In the tabular domain, instead of masking tokens, random subset of features is masked for each sample, and the masked values are predicted in a multi-target classification manner randomly selected feature for each sample,
(Huang et al.) 2020). In our experiments, asking the network to learn the structure of the data and form representations from n features that are useful in producing the value in the n-th feature. For more detail, see Appendix
mask
B
7

---

Published as a conference paper at ICLR 2023
Upstream data
Upstream pseudo-feature
Paticat
Pulse
Labels
2.5
1.2
1.25
93
1.50
30
96
Num samples
100
2.5
1.3
1.3
1.75
m
2000
78
91
20
2.0
1.7
1.5
Pre-training
10
2.2
1.4
1.5
2.25
1.5
Predicting
Missing
Picadd
Original
2,50
feature for upstreum data
Downstream pscudo-feature
2.3
1.7
1.2
1.25
Num samples
100
2.3
1.8
1.2
1.50
Fine-tuning
Patical
Pulse
Oxygen
20-
2.0
1.6
1.5
11.55
Kink
1.3
2.00
2.2
2.0
1.2
2.25
Missing
Paeado
Onggnal
Downstream data with new feature
Figure 4: Pseudo-Feature method for aligning upstream and downstream feature sets. Left: Diagram illustrating strategy for handling mismatches between the upstream and downstream features. When upstream data missing a feature present in downstream data, is predicted with model pre-trained upstream data and fine-tuned predict the new feature the downstream data. When a feature is missing in the downstream data, is predicted with a model trained to predict this feature on the upstream data. Right top: Scenario with missing feature in the upstream data. Comparison of ranks of FT-Trranffrrmrr model trained on data with missing feature, with pseudo-feature and with the original feature. Right bottom: Scenario with missing feature in the downstream data. Comparison of ranks of FT-Traasformrr model trained and fine-tuned on data with missing feature, fine-tuned with pseudo and with original feature. both scenarios, using the pseudo-feature is better than training without the feature but worse than worse than the original ground truth feature values.
5.2 CONTRASTIVE TRE-TRAINING
Contrastive pre-training uses data augmentations to generate positive pairs, or two different augmented views of a given example, and the loss function encourages a feature extractor to map positive pairs to similar features. Mcanwhile, the network also trained to map negative pairs, or augmented views of different base examples, far apart in feature space. We utilize the implementation of contrastive learning from Somepalli et al. (202T). I). In particular, we generate positive pairs by applying two data augmentations: CutMix ( Yuun al.] 2019) in the input space and Mixup (Zhang et al. (017) in the embedding space. For more details, see Appendix
5.3 COMPARING SUPERVISED AND SEPE-SUPERVISED PRE-TRAINING
While self-supervised learning makes for transferable feature extractors in other domains, our experiments show that supervised pre-training consistently better than the recent SSL pre-training methods we try that are designed for tabular data. In Figure 3 we compare supervised pre-training with contrastive and MLM pre-training strategies and show that supervised pre-training always attains the best average rank. Contrastive pre-training produces better results than training from scratch on the downstream data when using a linear head, but it is still inferior to supervised pre- training.. Tabular MLM pre- pre-training also falls behind the supervised strategy and performs comparably to training from seratch in the lower data regimes but leads to a weaker downstream model in the higher data regimes.
8

---

Published as a conference paper at ICLR 2023
6 ALIGNING UPSTREAM AND DOWNSTREAM FEATURE SETS WITH
While so far we have worked with upstream and downstream tasks which shared a common feature space, in the real world, tabular data is highly heterogeneous and upstream data having different set of features from downstream data is realistic problem (Lewinson) 2020). In our medical data scenario, downstream patient data may include additional lab tests not available for patients in the upstream data. In fact, the additional downstream feature may not even be meaningful for the upstream data, such medical exams only applicable downstream patients of biological sex different from the upstream patients. In this section, we propose pseudo-feature method for aligning the upstream and downstream data and show that the data heterogeneity problem be addressed more effectively than simply taking the intersection of the upstream and downstream feature sets for tabular transfer learning, which would throw away useful features. The proposed pseudo-featuee feature method is related to missing data imputation Zhang et al. (2008); Sefidian and Daneshpour (2019); Yoon et al. (2018) and self-training ideas Tanha et al. (2017). particular, misalignment of feature sets in upstream and downstream data can be seen as an extreme scenario with features having all
values missing.
Suppose our upstream data (X,,: Y.) is missing a feature new present in the downstream data (Xa, Ya). We then use transfer learning stages. As the diagram on the left panel of Figure 4 shows:
1. Pre-train feature extractor f : Xx, Yu on upstream data (Xu, Y.) without feature Tnew-
2. Fine-tune the feature extractor f on the downstream samples X to predict Trew as the target and obtain a model f: X. \ { + Knew-
3. Use the model f assign pseudo-values &nw of the missing feature to the upstream samples: nnew = f(x.) thus obtaining augmented upstream data (X,, J [Grnww }, Yu).
4. Finally, we can leverage the augmented upstream data (X,, U {&new), Y.)) to pre-train a feature extractor which we will fine-tune on the original downstream task (Xa, Ya) using all of the available downstream features.
Similarly, scenarios with a missing feature downstream data, we can directly train a feature predictor on the upstream data and obtain vaeuue- for the downstream data.
This approach offers appreciable performance boosts over discarding the missing features and often performs comparably to models pre-trained with the ground truth feature values as shown in the right panel of Figure mi The top heatmap represents the experiment where downstream data has an additional feature missing from the upstream data. The bottom heatmap represents the opposite scenario of the upstream data having an additional feature not available in the downstream data. To ensure that the features experiment with meaningful and contain useful information, for each task we chose important features according to GBDT feature importances. We observe that in both experiments, using the pscudo feature is better than doing transfer learning without the missing feature at all. Additionally, we observe that in some cases, the pscudo-feature approach performs comparably to using the original ground truth feature (10-100 samples on the top heatmap and 20 samples on the bottom heatmap). Interestingly, the pseudo-featur method is more beneficial when upstream features are missing, which suggests that having ground-trut -truh values for the additional feature in the downstream data is more important.
DISCUSSION
In this paper, we demonstrate that deep tabular models with transfer learning definitively outperform strong GBDT baselines with stacking in realistic scenario where the target downstream data is limited and auxillary upstream pre-training data is available. We highlight that representation learning with neural networks enables more effective knowledge transfer than leveraging upstream task predictions through stacking. Additionally, we present pseudo-feature method to enable effective transfer learning in challenging cases where the upstream and downstream feature sets differ. We provide suggestions regarding architectures, hyperparameter tuning, and setups for tabular transfer learning and hope that this work serves as guide for practitioners.

---

Published as a conference paper at ICLR 2023
8 ETHICS STATEMENT
We conduct experiments with the MetaMIMIC medical dataset (Grzyb et al. 2021 Wožnica et al.
2022), which based on the publicly accessible MIMIC database (Goldberger et 1000:Johnson
et al. 2016 202T). Regarding the patient consent to collect this data, as stated in Johnson et al. "The project was approved by the Institutional Review Boards Beth Israel Deaconess Medical Center (Boston, MA) and the Massachusetts Institute of Technology (Cambridge, MA). Requirement for individual patient consent was waived because the project did not impact clinical care and all protected health information was deidentified". The MIMIC database is freely available to any person upon completion of the credentialing process found at the following link: https //mimic- iv mi E. edu docs / access/ and is distributed under Open Data Commons Open Database License v1.0, please see the following link for details: https: https///pyyiiien...em content/mimin c iv demo- ceeec-ooopvvieee 1i www.eseeennn....ooooo
2016
REPRODUCIBILITY STATEMENT
We include the code for reproducing our results in the supplementary materials. We also describe strategies for data preprocessing, training the models, choosing the best epoch as well hyperparam- eter ranges in Appendix B and
10 ACKNOWLEDGMENTS
This work supported by the Office of Naval Research, the National Science Foundation (IIS- 2212182), and Capital One Bank.
REFERENCES
B. Afonso, L. Melo, W. Oliveira, S. Sousa, and L. Berton. Housing prices prediction with deep learning and random forest ensemble. In Anais do XVI Encontro Nacional de Inteligência Artificial Computacional, pages 389-400. SBC, 2019.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
Al-Stouhi and C. Reddy. Adaptive boosting for transfer learning using dynamic updates. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 60-75. Springer, 2011.
L. Alzubaidi, M. A. Fadhel, O. Al-Shamma, J. Zhang, and Y. Duan. Deep learning models for classification of red blood cells in microscopy images to aid in sickle cell anemia diagnosis. Electronics, 93):427, 2020.
L. Alzubaidi, M. Al-Amidie, A. Al-Asadi, A. J. Humaidi, O. Al-Shamma, M. A. Fadhel, J. Zhang, Santamaría, and Y. Duan. Novel transfer learning approach for medical imaging with limited labeled data. Cancers, 13(7):1590, 2021.
O. Arık and T. Pfister. Tabnet: Attentive interpretable tabular learning. AAAI, volume 35, pages 6687,6687,2000
K. Arun, G. Ishan, and K. Sanmeet. Loan approval prediction based on machine learning approach. IOSR J. Comput. Eng, 18(3):18-21, 2016.
S. Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi. Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07971, 2020.
D. Bahri, H. Jiang, Y. Tay, and D. Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15// 2021.
10

---

Published as a conference paper at ICLR 2023
A. Bardes, J. Ponce, and Y. LeCun. Vicreg: cottame--nvarrane--ccvaiiiiini regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
A. Beutel, P. Covington, S. Jain, C. Xu, J. Li, V. Gatto, and E. H. Chi. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 46-54, 2018.
V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:21 2021.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 12-9942-9924, 2020.
L.-C. Chen, Y. Zhu, G. Papandreou, Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 20.
S. Chen, A. Qin, D. Zhou, and D. Yan. U-net-generated generated synthetic ct images for magnetic resonance imaging-only prostate intensity-modulate radiation therapy treatment planning. Medical physics, (155))))-696665 201
S. Chen, K. Ma. and Y. Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv preprint arXiv: 00625, 2019.
Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
7. Dai, H. Liu, Q. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint arXiv:2 1at.::2000004333 2021.
J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2009. doi: 10. L199/CVPR.2099550663.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini arXiv:2010.1 1929, 2020.
A. Elisseeff and J. Weston. A kernel method for multi-labelled classification. Advances in neural information processing systems, 14, 2001.
W. Fang, C. Chen, B. Song, L. Wang, J. Zhou, and K. Q. Zhu. Adapted tree boosting for transfer learning, In 2019 IEEE International Conference Big Data (Big Data), pages 741-750. IEEE, 2019.
J. Fiedler. Simple modifications to improve tabular neural networks. arXiv preprint 0rX1v:2108.03214, 2021.
H. Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: components of a research resource for complex physiologic signals. circulation, 101(23)e215-e222) 2000.
11

---

Published as a conference paper at ICLR 2023
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. arXiv preprint 2tti::2000..19159 2021.
Y. Gorishniy, I. Rubachev, and A. Babenko. On embeddings for numerical features in tabular deep learning, arXiv preprint arXiv:2203.05555 2022.
M. Grzyb, Trafas, K. Wožnica, and Biecek. metamimic: analysis of hyperparameter trans- ferability for tabular data using mimic-iv database, 2021. URL https::/ggttub.... com/ Mode
.df
H. Hazimeh, N. Ponomareva, P. Mol, Z. Tan, and R. Mazumder. The tree ensemble layer: Differen- tiability meets conditional computation. In International Conference Machine Learning, pages 4138-4148, PMLR, 2020.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:21 100::2//1.0000 1.06377, 2021.
M. Heker and H. Greenspan. Joint liver lesion segmentation and classification via transfer learning. arXiv preprint arXiv:2004.12352 2020.
Howard and S. Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.061400 2018.
X. Huang, A. Khetan, M. Cvitkovie, and Z. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:20/2.06678, 2020.
J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ileus, Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
V. Jain, M. Goel, and K. Shah. Deep learning on small tabular dataset: Using transfer learning and image classification. In International Conference Artificial Intelligence and Speech Technology, pages 555-568. Springer, 2021.
L. Joffe. Transfer learning for tabular data. 2021.
A. Johnson, L. Bulgarelli, T. Pollard, S. Homn, L. A. Celi, and R. Mark. Mimic-iv, 2021. URL http::///ppssstotttor/ooooooooo wttp:/ttttttttiiiiiim....m.
A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark. Mimic-iii, freely accessible critical care database. Scientific data, (1):1-9, 2016.
A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprint arXiv:2 1899, 2021.
G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Kontschieder, M. Fiterau, Criminisi, and R. Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference computer vision, pages 1467-1475, 2015.
J. Kossen, N. Band, C. Lyle, N. Gomez, T. Rainforth, and Y. Gal. Self-attention between datapoints: Going beyond individual onput-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2023
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint 1.09:19999.11422. 2019.
M. Y. Law and B. Liu. Dicom-rt and its utilization in radiation therapy. Radiographics, 29(3): 655-667, 2009.
E. Lewinson. Python for Finance Cookbook: Over 50 recipes for applying modern Python libraries to financial data analysis. Packt Publishing Limited, 2020.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L.. Zettle- moyer. Bart: Denoising sequence-to-sequeene pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:I910.1 13461, 2019.
Z. Li, D. Ding, X. Liu, P. Zhang, Y. Wu, and L. Ma. Tinet: Tabular transfer network for few-samples prediction. In /EEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, pages 293-301, 2021.
Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:1907.1100 2019.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50 (0-60, 1947.
M. McDermott, B. Nestor, E. Kim, W. Zhang, A. Goldenberg, P. Szolovits, and M. Ghassemi. A comprehensive ehr timeseries pre-training benchmark. In Proceedings of the Conference on Health, Inference, and Learning, pages 257-278, 2021.
I. Misra and L. d. Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-13 1359, 2010. doi: 10.1 10.1109/TKDE.2209.199
S. Popov, S. Morozov, and Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint UrXiv:/909.06311 2019.
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: (xXvv:880400276: 2018.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
Ren, K. He, R. Girshick, and Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91-99, 2015.
I. Rubachev, Alekberov, Y. Gorishniy, and A. Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv preprint 2=Xxi:2207.03208 2022.
B. G. Santa Cruz, M. N. Bossa, Sölter, and A. D. Husch. Public covid-19 x-ray datasets and their impact on model bias-a systematic review of significant problem. Medical image analysis, 74: 102225, 2021.
B. Schäfl, L. Gruber, A. Bitto-Nemling, and S. Hochreiter. Hopular: Modern hopfield networks for tabular data. 2021.
13

---

Published as a conference paper at ICLR 2023
A. M. Sefidian and N. Daneshpour. Missing value imputation using novel grey based fuzzy means, mutual information based feature selection, and regression model. Expert Systems with Applications, 115:68-95 94, 2019.
A. Sharma, E. Vans, D. Shigemizu, K. A. Boroevich, and T. Tsunoda. Deepinsight: A methodology to transform a non-image data to an image for convolution neural network architecture. Scientific reports, (1))1177,015
R. Shwartz-Ziv and A. Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022. ISSN 1566-2535. doi: https://dd..nnn///..ooooiiiiiiid
URLhttpa /www.sccieett /www.sclenced. rect. com/ wtt.c///c....in.........aaaaa/aa 0023600
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2 063444 2021.
W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang. Autoint: Automatic feature inter- action learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference Information and Knowledge Management, pages 1161-1170, 2019.
B. Sun, L. Yang. W. Zhang, M. Lin, P. Dong, C. Young, and Dong. Supertml: Two-dimensional word embedding for the precognition on structured tabular data. In Proceedings of the IEEESCVF Conference on Computer Vision and Patlern Recognition Workshops, pages 0-0, 2019.
Tanha, M. Van Someren, and H. Afsarmanesh. Semi-supervised self-training for decision tree classifiers. International Journal of Machine Learming and Cybernetics, 8(1):355-370, 2017.
K. M. Ting and I. H. Witten. Stacked generalization: when does it work? 1997.
K. Trohidis, G. Tsoumakas, G. Kalliris, I. Vlahavas, et al. Multi-label classification of music into emotions. In ISMIR, volume pages 325-330, 2008.
V. Turbé, C. Herbst, T. Mngomezulu, S. Meshkinfamfard, N. Dlamini, T. Mhlongo, T. Smit, V. Cherepanova, K. Shimada, J. Budd, et al. Deep learning of hiv field-based rapid tests. Nature Medicine, 27(7):1155-1170 2021.
T. Ucar, E. Hajiramezanali, and L.. Edwards. Subtab: Subsetting features of tabular data for self- supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021.
R. Wang, B. Fu, G. Fu, and M. Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pages 1-7. 2017.
R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning rank systems. In Proceedings the Web Conference 2021, pages 1785-1797, 2021.
Z. Wang and J. Sun. Transtab: Learning transferable tabular transformers across tables. arXiv preprint arXiv:2205.09328 2022.
K. Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big data, 3(1): 1-40, 2016.
Wienke, D. Wigand, N. Koster, and S. Wrede. Model-based performance testing for robotics software components. In 2018 Second IEEE International Conference on Robotic Computing (IRC), pages 25-32. IEFE, 2018.
Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-8)) 83, 1945. ISSN 00994987. URL http:/ttp////c jwwor.sttr.coo http://stoor.orggaaaaleee/ee/eee
D. H. Wolpert. Stacked generalization. Newral networks, 5(2):241 networrk,,5(2:2211-222 1992.
K. Wožnica, M. Grzyb. Z. Trafas, and P. Biecek. Consolidated learning-a domain-specift. model-free optimization strategy with examples for xgboost and mimic-iv, arXiv preprint arXiv:2201.11 11815, 2022.
14

---

Published as a conference paper at ICLR 2023
J. Yang, O. Lindenbaum, and Y. Kluger. Locally sparse neural networks for tabular biomedical data. In Interaational Conference on Machine Learning, pages 25123-25155 PMLR, 2022.
Y. Yang, I. G. Morillo, and T. M. Hospedales. Deep neural decision trees. arXiv;1806 06988, 2018.
arXiv preprint
H. Yèche, G. Dresdner, F. Locatello, M. Hüser, and G. Rätsch. Neighborhood contrastive learning applied to online patient monitoring. In International Conference on Machine Learning, pages 11964 11944. PMLR, 2021.
Yin, G. Neubig, W.-t. Yih, and S. Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08. 00 2020.
Yoon, J. Jordon, and M. Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pages 5689-5698. PMLR, 2018.
J. Yoon, Y. Zhang, J. Jordon, and M. van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 15:33-0--104, 2020.
S. Yun, D. Han, S. Oh, S. Chun, Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032. 2019.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0944 2017.
S. Zhang, J. Zhang, X. Zhu, Y. Qin, and C. Zhang. Missing value imputation based on data clustering.
In Transactions on computational science I, pages 128-138. Springer, 2008
Y. Zhu, Brettin, F. Xia, A. Partin, M. Shukla, H. Yoo, Y. A. Evrard, J. H. Doroshow, and Stevens. Converting tabular data into images for deep learning with convolutional neural networks. Scientific reports, 11(1):1-11, 2021.
F. Zhuang, Z. Qi. K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):33-76, 2020.
15

---

Published as a conference paper at ICLR 2023
A LIMITATIONS AND IMPACT
In this section we discuss limitations of our work. We note that transfer learning with deep tabular models requires more computational resources than training XGBoost or CatBoost, especially when the hyperparameter tuning is used thus incurring additional costs on practitioners adopting approach. In addition, while we handle the cases of differing upstream and downstream feature sets thus allowing for transfer learning with heterogeneous data, our approach relies on having reasonably similar (but not identical) upstream and downstream feature sets and related upstream and downstream tasks and do not propose foundation model for tabular data.
B EXPERIMENTAL DETAILS.
B.
HARDWARE
We ran our experiments NVIDIA GeForce RTX 2080 machines.
B.2
IMPLEMENTATION LICENSES
For the model implementations and training routines we adapt the code from the following publicly available repositories and libraries:
RTDL under MIT License
Tab Transformer under
under MIT License
2
Catboost
under Apache License
XGBoost under Apache License
B.3
DETAILS METAMIMIC TARGETS
The MetaMIMIC Repository (Grzyb et al. 2021 Wožnica et al. 2022) is based on the publicly accessible MIMIC database (Goldberger et al. 2000 Johnson et 2016 202I) which contains health related data from patients admitted to the ICU. MetaMIMIC contains health records for 34925 unique patients, who are at least 15 years old, were diagnosed with at least one considered diseases, and did not stay in hospital for more than 60 days. The target diagnoses correspond the 50 most common conditions, which are also found in both ICD-9 and ICD-10 codes (International Classification of Diseases), which results in 12 unique diagnoses. Please, Table for selected targets, their ICD codes and frequency in the considered cohort (adapted from Table in Wožnical al. (2022)).
To understand how related the MetaMIMIC tasks are, we compute pair-wise correlations between targets and present them as a heatmap Figure We find that the maximum correlation of 0.36 is between ischematic and hypertensive diagnoses, while the average absolute pair-wise correlation is 0.08. We conclude that although certain tasks are slightly correlated, such as targets corresponding to cardiovascular diseases, the average pair-wise correlation between all the targets is low.
B.4 DETAILS THE UPSTREAM-DOWNSTERR
SPLITS AND NUMBER SEEDS
We reserve 6985 patients (20% of the MetaMIMIC data) for the downstream test set, and use 22701 patients (65% of the MetaMIMIC data) for training and 5239 patients (15% of the MetaMIMIC data) validation set for hyperparameter tuning of the upstream feature extractors. We further simulate scenarios with different levels of data availability by sampling from the training data downstream patients for fine-tuning. We note that these splits do not change across seeds. We use similar splits Yeast and Emotions datasets.
https://catboost..
https://xgboost.al)
16

---

Published as a conference paper at ICLR 2023
Condition
ICD-9
ICD-10
Frequency
Hypertensive diseases
401-405
110-116
59.8%
Disorders of lipoid metabolism
272
E78
40.3%
Anemia
280-285
D60-D64
35.9%
Ischematic heart disease
410-414
120-125
32.8%
Diabetes
249-250
E08-E13
25.3%
Chronic lower respiratory diseases
466, 490-496
J40-J47
19.5%
Heart failure
428
I50
19.4%
Hypotension
458
195
14.5%
Purpura and other hemorrhagic conditions
287
D69
11.9%
Atrial fibrillation and flutter
427.3
I48
10.5%
Overweight, obesity and other hyperalimentation
278
E65-E68
10.5%
Alcohol dependence
303
F10
7.7%
Table 1: Selected targets with corresponding ICD codes and frequency the considered cohort.
1.0
diabetes
0.23
0.19
0.15
0.17
0.05
0.04
0.01
0.21
0.05
0.01
hypertensive
0.23
1
0.25
0.15
0.1
0.05
0.05
0.05
0.36
0.11
0.01
0.07
ischematic
0.19
0.25
0.27
0.06
0.08
0.05
0.08
0.32
0.12
0.03
heart
0.15
0.15
0.27
0.07
0.09
0.14
0.05
0.09
0.18
0.06
0.06
overweight
0.17
0.1
0.06
0.07
0.03
0.08
0.03
0.1
0.05
0.01
0.03
0.6
anemia
0.05
0.05
0.08
0.09
0.03
0.02
0.14
0.03
0.08
0.22
0.04
respiratory
0.04
0.05
0.05
0.14
0.08
0.02
0.03
0.02
0.03
0.01
0.01
hypotension
0.01
0.05
0.08
0.05
0.03
0.14
0.03
0.06
0.07
0.05
0.01
0.4
lipoid
0.21
0.36
0.32
0.09
0.1
0.03
0.02
0.06
0.11
0.01
0.11
0.05
0.11
0.12
0.18
0.05
0.08
0.03
0.07
0.11
0.08
purpura
0.01
0.01
0.03
0.06
0.01
0.22
0.01
0.05
0.01
0.08
0.12
0.06
0.07
0.1
0.06
0.03
0.04
0.01
0.01
0.11
0.12
diabetes
hypertensive
ischematic
heart
onerweeiht
anemia
respiratory
hypotension
lipoid
atrial
purpura
alcohol
Figure 5: Correlations between MetaMIMIC targets. The heatmap represents absolute pair-wist wise
correlations between MetaMIMIC targets.
We run each experiment with 10 random seeds for the experiments on transfer learning with deep tabular models in Section 4 and Appendix D, and with 5 random seeds for the experiments with self-supervised pre-training and feature imputation in Sections 5, and Appendix D.
B.5 DATA PREPROCESSING
We preprocess numerical features with quantile transformation with standard output distribution for neural networks and we use original features for GBDT models. Quantile transformer is first fit on upstream data and then applied to downstream data to preserve similar feature distribution in upstream and downstream data. We note, that using the same normalization for upstream and downstream data is very important step for transfer learning. We impute missing values with mean values for numerical features and with category for categorical features.
17

---

Published as a conference paper at ICLR 2023
B.6 TRAINING DETAILS
B.6.1 SUPERVISED PRE-TRAINING
All deep models are trained with Adam optimizer (Loshchilov and Hutter, 2017). We pre-train models upstream data for 500 epochs with patience 30, meaning that we continue training until there are 30 consecutive epochs without improvement on the validation set, but no longer than 500 epochs. Since we assume limited data availability for downstream task, do not sample validation set for early stopping. Instead, we fine-tune/train models from scratch for 200 epochs and choose an "optimal" epoch as discussed in Section [.6.3] We use learning rate le for training from scratch on downstream data and learning rate 5e 5 for fine-tuning pre-trained models. For pre-training, learning rate and weight decay are tunable hyperparameters for each model and each pre-training dataset (collection of upstream tasks). Batch size was set to 256 in all transfer-lerrnng experiments.
B.6.2 SELF-SUPERVISED PRE-TRAINING
MLM Pre-training
To implement MLM pre-training for tabular data, for each training sample in a batch we randomly select feature to replace with a masking token (in the embedding space), which unique for every feature. We also initialize a distinct fully connected layer, or head, for each column which is used to predict the masked values of that feature. We compute the appropriate loss, cross-entropy for categorical features and MSE for numerical ones, using the output from the head corresponding to the masked feature.
Our masked pre-training routine has very no additional hyperparameters. The only deviations form the pre-training hyperparameters listed above are that we use larger batch sizes of 512 and we do not employ patience. In this setting pre-training for full 500 epochs.
Since MLM pre- pre-training requires training additional classification heads for every feature in the data, it dramatically increases the memory requirements. Because of that we limit the experiments with self-supervised pre-training to using the default FTlttranfformrr configuration, including the setups compare to, that is training from scratch and using supervised pre-training.
Contrastive Learning
We adapt the implementation of contrastive learning for tabular data from Somepalli et al. 2021 The pre-training loss consists of two components, the first is the InfoNCE contrastive loss, which minimizes the distance between the representation of two views of the same data point (original and augmented samples), while maximizing the distance in the feature space between different samples. The second component is denoising loss, which is used to train an MLP head to predict the original sample from noisy view of that sample. For a detailed explanation of contrastive pre-training for tabular data we refer the reader to SSoepalli et al. (202T). To construct positive pairs for contrastive loss and noisy samples for denoising loss we use two data aumentations: CutMix in the input space and Mixup in the embedding space Yun et al. (2019); Zhang et al. (2017).
Formally, a batch of data represented by = {2;) where each x, has q features. Let m denote binary mask (over the features of any x;) with each entry being with probability p. Then CutMix augmentation of a sample in the input space is computed as
CutMix m x I, + (1 m) x Iz
where j is a random index chosen from [0, r2). To define Mixup in the embedding space, let X CutMix be the set of the embeddings of the entire CutMix-ed batch. Then, Mixup augmentation is computed as convex combination
Mixup
CuiMix + (1 - p)
where k is again a random index chosen from [0, m].
18

---

Published as a conference paper at ICLR 2023
The hyperparameters are similar supervised pre-training with only several modifications. First, use smaller batch sized of 200 and we train for a full 500 epochs (no patience). Also, contrastive learning has additional hyperparameters and μ4, both of which we set to 0.9.
B.6.3 EPOCH SELECTION
Since we work with limited downstream data, sampling a validation set for early stopping is not always possible. Therefore, select the number of fine-tuning epochs for deep models follows. For the data levels of and 10 samples, we simply select 30 fine-tuning epochs. For more data of 20 samples, we select 60 fine-tuning epochs. In the larger data levels of 100 and 200 samples, sample 20% of the data as a validation set to perform early stopping with the flexible end-to-end fine-tuned transfer learning setups prone to overfitting. For early stopping, terminates training no improvement in the validation score is observed for more than 30 epochs. In the less flexible transfer learning setups with frozen feature extractor, select 100 fine-tuning epochs for the MLP head atop a frozen feature extractor and 200 fine-tuning epochs for the linear head atop frozen feature extractor. Finally, for the deep baselines with the hyperparameters tuned on a small subsample of the upstream data, we select the best epoch from the small upstream subsample.
B.6.4 STACKING FOR GBDT MODELS
For XGBoost models incorporate stacking by training 11 additional XGBoost classifiers upstream tasks and stack their predictions as additional features for downstream data. For Catboost we apply the same strategy, but we train single multi-label Catboost classifier predicting all 11 upstream labels.
B.6.5 STATISTICAL SIGNIFICANCE
We compute ranks taking into account statistical significance of the performance differences between the models; on a given task, top models without statistically significant performance difference all receive rank 1. To compute statistical significance use the one-sided Wilcoxon Rank-Sum test Wilcoxon (1945); Mann and Whitney (1947) with 0.05. We run each experiment with 10 random
seeds for the experiments in Section and random seeds for the experiments Sections 5 6
C HYPERPARAMETER TUNING
In this section we provide hyperparaeter search spaces and distributions for Optuna for each model, which are adapted from the original papers. For Catboost and XGBoost models we use search spaces proposed in Gorishniy et al. 2222T. We run 50 Optuna trials to tune hyperparameters for each model. In our experiments we tune the hyperparameters on full upstream data for deep tabular models with transfer learning, and upstream data of the same size downstream data for deep baselines trained from scratch and for GBDT models.
C.O.1 FT-TRANSFORMER
The hyperparameter search space and distributions as well as the default configuration are presented Table 2 The number of heads is always set to as recommended the original paper.
C.0.2 RESNET
The hyperparameter search space and distributions as well as the default configuration are presented in Table
C.0.3 MLP
The hyperparameter search space and distributions as well as the default configuration are presented in Table
19

---

Published as a conference paper at ICLR 2023
Table Optuna hyperparameter search space and default configuration for FT-Transformer
Parameter
Search Space
Default
Number of layers
UniformInt] 1, 4]
Feature embedding size
UniformInt (64, 512
192
Residual dropout
{0, Uniform [0, 0.2]]
0.0
Attention dropout
Uniform[0, 0.5
0.2
FFN dropout
Uniform [0. 0.5
0.1
FFN factor
Uniform (2/3, 8/3)
4/3
Learning rate
LogUniform [e 5, le
-
1e-4 4
Weight decay
LogUniform [e le
1e-
Table 3: Optuna hyperparameter search space and default configuration for ResNet
Parameter
Search Space
Default
Number of layers
Uniformint] [1
5
Category embedding size
Uniformint 64, 512]
128
Layer size
Uniformint 512]
200
Hidden factor
Uniform 1, 4]
3
Hidden dropout
Uniform 0, 0.5]
0.2
Residual dropout
{0, Uniform 00, 0.5]]
0.2
Learning rate
LogUniform[le [le 5, le 2]
le - 4
Weight decay
{0, LogUniform(le: 6, 3]}
0.0
Table 4: Optuna hyperparameter search space and default configuration for MLP
Parameter
Search Space
Default
Number of layers Category embedding size
UniformInt
3
UniformInt 64, 512]
128
Layer size
UniformInt 512]
[300, 200, 300]
Dropout
{0, Uniform [0, 0.5]
0.2
Learning
LogUniform [e le 2]
1e 4
Weight decay
(0, LogUniform 1e 6, le 3}}
1e 5
C.0.4 TABTRANSFORMER
The hyperparameter search space and distributions as well as the default configuration are presented in 5
Table Optuna hyperparameter search space and default configuration for TabTransformer
Parameter
Search Space
Default
Number of heads
UniformInt 8]
Number of layers
UniformInt(1, 12]
Category embedding size
UniformInt (8, 128)
32
Attention Dropout
Uniform[0.0. 0.5]
0.0
FF Dropout
Uniform (0.0rmm0000.0m 0.5]
0.0
Learning rate
LogUniform [e le
le
Weight decay
LogUniform 1e 6, le 3
le
C.0.5 CATBOOST
The hyperparameter search space and distributions are presented in Table For default configuration use default parameters from the Catboost library (Prokhorenkova al. 2018
20

---

Published as a conference paper at ICLR 2023
Table 6: Optuna hyperparameter search space for Catboost
Parameter
Search Space
Iterations
UniformInt(2. 2, 1000]
Max depth
Uniformint(3 33 10]
Learning rate
Uogurrform le - 5, 1]
Bagging temperature
Uniform (0,
L2 leaf reg
Uniffrrmm 1, 10]
Leaf estimation iterations
Uniformint[1, 10]
C.0.6 XGBooST
The hyperparameter search space and distributions as well as the default configuration are presented in Table 7 For default configuration use default parameters from the XGBoost library (Chen and Guestrin 2016).
Table Optuna hyperparameter search space for XGBoost
Parameter
Search Space
Num estimators
UniformInt 2, 1000]
Max depth
UniformInt (3, 10]
Min child weight
LogUniform[LL 8, 1c5]
Subsample
Uniform (0fform00.5,11
Learning rate
LogUniform(1e 5, 1]
Col sample by level
Uniform (0.5, 1]
Col sample by tree
Uniform 00.5,
Gamma
{0, LogUniform Le 8, 1e2]
Lambda
(0, LogUniform le 8, le2
Alpha
{0, LogUniform le 8, lc2]
C.I TUNING GBDT MODELS
In Table 8 explore the effectiveness of our hyperparameter tuming strategy for GBDT models. In particular, we compute average ranks for models with tuned and default configurations. Recall, because of the limited data availability in downstream tasks, data of the same size as the downstream data using full-size validation set. We find that using upstream data for tuning hyperparameters is effective for XGBoost model while for catboost tuned configuration outperforms default configuration at three out of five data availability levels.
the hyperparameters
upstream
Table Comparison of tuned and default configurations of GBDT models. The table displays ranks computed pair-wise for default/tuned configurations of XGBoost and Catboost models.
Num samples
10
20
100
200
XGBoost Tuned
1.17
1.17
1.25
1.33
1.42
XGBoost Default
1.42
1.83
1.67
1.50
1.58
Catboost Tuned
1.33
1.08
1.17
1.25
1.42
Catboost Default
1.25
1.25
1.42
1.33
1.33
C.2 TUNING DEEP BASELINES
In Table 9 we evaluate hyperparameter tuning routine for deep baselines, i.e. deep neural networks trained from scratch. We find that unlike GBDT models, hyperparameters tuned upstream data do not transfer to downstream tasks at lower data regimes (i.e. 4-20 samples) for deep models. However, tuning helps deep baselines in higher data regimes.
21

---

Published as a conference paper at ICLR 2023
Table Comparison of tuned and default configurations of deep tabular baselines. The table displays ranks computed pair-wise for default/tuned configurations of deep tabular neural networks.
Num samples
4
10
20
100
200
FT- FT-arassformer Tuned
1.33
1.25
1.00
1.33
1.33
FT Traansfforerr Default
1.00
1.08
1.58
1.417
1.50
ResNet Tuned
1.25
1.17
1.33
1,25
1.75
ResNet Default
1.00
1.08
1.25
1.00
1.00
MLP Tuned
1.42
1.67
1.58
1.17
1.33
MLP Default
1.00
1.08
1.33
1.67
1.50
TahTransformerr Tuned
1.25
1.33
1.17
1.08
1.17
Tabrransformer Default
1.17
1.25
1.33
1.50
1.67
C.3 TUNING DEEP MODELS FOR TRANSFER LEARNING
In Table TO we evaluate the hyperparameter tuning strategy for deep tabular neural networks with transfer learning. Recall, we tune the hyperparameters on the full upstream data which is then used for pre-training. We find this strategy helpful for most models, especially for FT-Transformer and Tranffrsformer, and for most transfer learning setups.
22

---

FI-Transformer + MLP Tuned Published as a conference paper at ICLR 2023
Table 10: Comparison of tuned and default configurations of deep tabular neural networks with transfer learning. The table displays ranks computed pair-wies wise for default/tuned configurations of deep tabular models fine-tuned with 4 different transfer-learning setups.
Num Samples
4
10
20
100
200
FT-Transformer + LH-E2E Tuned
1.00
1.00
1.00
1.08
1.17
FT-Transformer + LH-E2E Default
1.25
1.25
1.17
1.42
1.17
FT-Transformer MLP--22 Tuned
1.00
1.00
1.08
1.00
1.08
FT-Tranffrrmer + MLP-F2E Default
1.08
1.00
1,25
1.33
1.58
FI-Tranffrmmer + I.H Tuned
1.00
1.00
1.00
1.00
1.00
Fr-raasffommrr + L.H Default
1.17
1.25
1.33
1.58
1.67
1.08
1.00
1.00
1.00
1.00
FT-Transformer + MLP Default
1.17
1.08
1.33
1.58
1.75
ResNet + LH-E2E Tuned
1.00
1.17
1.33
1.25
1.33
ResNet + LH-E2E Default
1.08
1.00
1.00
1.17
1.25
ResNet + MLP-E2E Tuned
1.00
1.00
1.08
1.08
1.00
ResNet + MLP-E2E Default
1.17
1.00
1.08
1.58
1.42
ResNet + LH Tuned
1.00
1.00
1.08
1.00
1.00
ResNet + LH Default
1.17
1.25
1.25
1.50
1.50
ResNet + MLP Tuned
1.00
1.00
1.00
1.00
1.08
ResNet + MLP Default
1.17
1.17
1.33
1.58
1.75
MLP + LH-E2E Tuned
1.17
1.25
1.42
1.42
1.17
MLP + LH-E2E Default
1.08
1.08
1.08
1.33
1.50
MLP+MLP-ED MLP-EEE Tuned MLP + MLP-E2E Default
1.00
1.08
1.33
1.25
1.42
1.08
1.00
1.25
1.25
1.33
MLP+ LH Tuned
1.00
1.25
1.17
1.17
1.08
MILP L.H Default
1.08
1.08
1.17
1.17
1.25
MLP + MLP Tuned
1.08
1.08
1.08
1.00
1.00
MLP+ + MLP Default
1.00
1.00
1.17
1.17
1.42
Tab TahTrasfforeer + L.HF22E Tuned
1.00
1.00
1.08
1.08
1.00
TabTransformer + LH-E2E Default
1.17
1.25
1.25
1.50
1.25
TabTransformer + MLP--2E Tuned
1.00
1.00
1.08
1.08
1.00
PabTransformer + MLP-E2E Default
1.17
1.25
1.25
1.42
1.25
TabTransformer + LH Tuned TabTransfornan + LH Default
1.00
1.00
1.00
1.00
1.00
1.25
1.17
1.25
1.25
1.17
Tab BabTransformer- MLP Tuned Tab Transformer + MLP Default
1.00
1.00
1.00
1.00
1.00
1.25
1.33
1.25
1.25
1.25
23

---

Published as a conference paper at ICLR 2023
D ADDITIONAL RESULTS
D.1 RESULTS FOR TABTRANSFORMER
Figure o equivalent Figure 2 but also includes the results for TabTransformer model, a tabular neural network consisting transformer block for categorical features and an MLP block on top for numerical features. We find that Tabfrrnsformer performs poorly compared other deep tabular neural networks, which might be explained by the fact that the original paper Huang et al. (2020) only suggests tune the hyperparameters of transformer block, but not the MLP part, while Meta-MIMIC data has only one categorical features and 171 numerical features.
FT-Transfomnr
ResNet
MLP
Tab Rab-Transformer
Catboost XGBoost
200
Num samples
100
13.5
3.6
6.2
16.8
6.7
14.6
13.0
7.6
17.4
6.6
13.2
15.0
9.9
17.0
8.3
17.5
17.3
17.2
19.2
21.0
10.9
6.4
9.8
8.2
10
15.5
12.5
15
10-14.5
9.6
10.7
11.0 11.8
16.6 12.5
8.9
13.0 8.9
19.2
14
5.9
11.2
6.2
16.8
12.2
12.3
11.5 14.8
16.3 12.8
15.0 14.4
20
4-13.8
10.0
11.11111 11.3
17.2
11.2
9.5
12.6 10.6
18.2
4.9
13.8
5.8
16.2
10.6
10.3
13.4 13.5
14.5 12.6
19.0 18.7
FS
LH-E2E
MLP-EZE
LH
MLP
FS
LH-EDE
MLP.EDE
LH MIL
LH-EZE MLP-EZE ERE
LH MLP
PS
LM-E2E
MLP.E3E
LH
MLP
FS Stocking
F8 Stecking
Figure 6: Average model ranks including Tab Taaansforrer.. Deep tabular models and GBDT performance is presented the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training upstream data), LH and MIP denote linear and MI.P heads correspondingly. E2E denotes end-to-end training. Rank is averaged across all downstream tasks.
D.2 RANKS STANDARD ERROR
Figure presents standard errors for ranks in Figure computed across tasks.
FT- Transfsrerr
ResNei
MLP
Catboost
XGBoost
200
1.5
1.0
1.0
0.7
1.5
1.1
1.2
1.0
0.7
1.4
1.6
0.9
1.7
0.7
1.3
1.4
1.0
1.9
1.9
Rank Standard Eiror
Num samples
100-
0.7
0.9
0.8
1.3
0.5
20-
1.3
1.2
1.2
1.7
1.7
1.3
L11
1.5
1.9
1.6
1.0
1.2
1.4
1.7
1.8
1.4
1.4
2.0
1.9
10-
1.0
1.8
1.4
1.9
1.7
1.7
1.0
1.5
1.4
0.8
2.0
1.3
1.3
2.0
1.8
1.3
1.4
2.0
1.4
1.2
1.0
1.0
2.0
FS
LH-E2E
MLP-E3E
1H
MLR
FS
LH-E2E
MLP-EZE
LH
MLP
LR-EZE
ELP-EZE
LH
MLP
FS
Stacking
Sucking
Figure 7: Standard error for model ranks computed across all downstream tasks. This figure
complements Figure
Darker colors indicate lower standard errors.
D.3 MODEL-WISH RANKS
In Figure g we compare different transfer learning setups for each deep tabular model.
D.4 AVERAGE ROC-AUC IMPROVEMENT OVER CATBOOST
In Figure 9 we display ROC-AUC improvements Catboost baseline averaged across all down- stream tasks. We observe trends similar to ones with ranks; MLP model performs best in low data regimes, while Tr-arfosfoerrr offers consistent gains across all data levels.
24

---

Published as a conference paper at ICLR 2023
FT- Transformer
ResNet
MI.P
Tab- Transformer
Cathoost
XG
200
3.2
2.1
2.0
4.8
2,4
4.1
2.3
1.4
4.6
2.2
3.2
2.5
2.1
4.5
2.7
2.3
1.3
1.2
3.8
4.8
1.
1.1
14
1.0
Num Samples
100
2.3
2.0
4.4
1.9
3.5
3.4
2.0
4.1
3.1
2.3
4.0
2.0
2.0
1.9
2.5
3.9
1.8
1.1
1.6
1.0
2.9
3.2
3.9
3.5
1.0
3.5
2.7
2.3
3.0
2.8
3.8
3.2
2.4
3.1
2.1
4.2
3.0
1.8
3.2
1.8
3.0
1.8
2.0
2.1
3.5
1.7
1.1
1.1
1.0
4
3.5
2.7
2.3
2.9
2.8
4.0
2.5
2.2
2.9
2.5
4.3
2.7
1.6
3.6
2.0
3.7
1.9
1.8
2.8
2.8
1.5
1.0
1.0
1.2
FS
LH-EZE
MLP-ECE
LH
MLP
FS
LH-E2E
MLP-EZE
LH MLP
FS
LHHEZE
MLP-E2E
LH
MLP
FS
LH-EJE
MLP-E2E
LH
MLP
FS
Stacking
FS Stacking
Figure Model-wise ranks for GBDT and neural networks. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training upstream data). LH and MLP denote linear and MLP heads correspondingly, E2E denotes end- to-end training. Rank is computed across training setups for each model and is averaged across all downstream tasks.
FT-Transformer
ResNet
MLP
Tab Transforner
Catboost XGBoost
200 4.01 0.02 0.02
0.02-00000000
0.000.00
0.06-0..1
0.01
0,000
-0.44-0.2-000
40,05 -005--4.3--.0----00 0.03--.122011
0.00 D.0
4.04-4033
0.050
Num Samples
100-0.02000
0.00
0.03-0.3300
4007
0.02
100000000000 0.00
4007 40.05 -.----0------------
0.03
0.025
0.000
20-0.01
0.05
0.05
0.02
0.05
0.00
0.03
0.05
0.01
0.06
0.00
0.03
0.07
0.02
0.06
00001
0.02
0.02
0.01
-0.01
0.00
0.03
0.03
0.03
0.025
10 10-0.01
0.05 0.04
0.04
0.03
0001 0001000000
0.05 0.03
0.05
00033 0.04
0.07
0.04 0.07
40.01
0.04
0.04 0.04
0.02
0.00 0.03
0.01 0.01
0,050
4+0.00 0.06
0.04
0.03 0.02
0.02
0.03
0.02
0.93
0,00
0.04
0.02
0.06
0.00
0.04
0.04
0.02
0.00
0.02
0.04 0.0440.04
0.075
FS LIKEE
ELP-EDE
LH
MLP
FS
LH.EZE
MLP-E3E
LH
MLP
FS
LH-EDE
MLP-ZE
LH
MLP
FS
LH-E2E
ML MLREE
LH
MLP
F$
Stacking
FS Stacking
0,,000
Catboost baseline. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to-end training. ROC-AUC improvement is computed difference in ROC-AUC with Catboost model and is averaged across all downstream tasks.
Figure 9: Average ROC-AUC improvements
D.5 DATASET-LEVEL RESULTS
we report ROC-AUC measurements for each model on each downstream task. We include the results for GBDT models, neural baselines and neural networks with transfer learning.
Tables
13
D.6 EXPERIMENTS ON ADDITIONAL DATASETS
To expand the experimental evaluation of tabular transfer learning beyond the medical domain, include experiments on two additional multi-label datasets: Yeast functional genomics data Blisseeff) and Weston (2001) and Emotions data Trohidis et al.) (2008). Yeast contains micro-array expression data and phylogenetic profiles of genes with labels corresponding to different gene functional classes. Each gene associated with a set of functional classes, therefore each task aims to predict gene is associated with particular gene functional group. The dataset contains 2417 samples (genes) with 103 numerical features and 14 labels for each sample. Emotions data aims to detect emotions in music, it contains rhythmic and timbre features of 593 songs and labels corresponding In our experimental setup, we treat each classification label as a separate task. Similarly to the experiments with MetaMIMIC, labels for the upstream task and the n-th label for the downstream task. We further limit the amount of downstream data to 4, 10, 20, 100 and 200 samples. This strategy results in 70 combinations of
emotions.
split the tasks into downstream and upstream by reserving
25

---

Published as a conference paper at ICLR 2023
FT-Tranne Transformer
MLP
Catboost
200
9.0
4.6
4.1
9.4
2.4
6.8
7.2
2.5
8.7
1.8
6.0
Nurn samples
100-
8.0
5.2
4.5
8.2
2.6
8.6
6.6
2.7
2.0
7.9
Rank
5.2
2.6
1.6
7.4
10
4.6
3.5
4
6.6
4.4
6.3
3.5
6.0
2.9
2.7
6.5
FS
LH-E2E
MLP-EZE
LH
MLP
FS
LH-EZE
ML MLP-EZE
LH
MLP
Stacking
Figure 10: Average model ranks across downstream tasks in Yeast data. Deep tabular models and Catboost performance is presented on the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly. E2E denotes end-to-end training. Rank is averaged across all downstream tasks.
FT-Transformer
MLP
Catboost
200
6.5
4.3
3.7
9.3
2.0
9.3
5.8
3.0
10.5
4.8
3.2
2
Num samples
100-
6.7
3.7
4.8
8.7
1.8
9.3
6.3
2.8
10.7
8
4.2
Rank
20-
8.2
4.2
5.0
6.8
1.0
7.5
10.0
2.7
9.5
3.0
6.2
66
7.7
4.2
4.8
6.0
3.2
7.0
8.2
3.5
7.0
5.0
6.0
8
6.7
4.5
4.7
6.3
3.8
8.0
6.5
4.3
6.7
4.2
5.8
10
FS
LH-E2E
MLP-EZE
LH
MLP
FS
LH-E3E
MLP-EZE
LH
MLP
Stacking
Figure 11: Average model ranks across downstream tasks in Emotions data. Deep tabular models and Catboost performance is presented the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to-end training. Rank is averaged across all downstream tasks.
upstream and downstream datasets constructed from the Yeast database and 30 combinations from the Emotions database. We conduct experiments with FT-Transformer, MLP, and Catboost models which demonstrated the best results in our previous experiments. We tune the hyperparameters for deep models and Catboost described in Appendix C [Tg present average model ranks of two tabular neural network and Catboost models on downstream tasks as a heatmap with warmer colors representing better results.
We observe that deep tabular models pre-trained on upstream data and finetuned with MLP head outperform models trained from scratch and Catboost models leveraging stacking. Interestingly, MLP model performs better on the Yeast dataset, while FT-Transformer performs better on the Emotions dataset.
D.7 EXPERIMENTS WITH FEWER PRE-TRAINING TASKS
To further validate the effectiveness of transfer learning in scenarios where fewer pre-training tasks available repeat experiments with the first 5 targets from Meta-MIMIC dataset: Diabetes,
26

---

Published as a conference paper at ICLR 2023
FT- Transformer
MLP
Catboost
200
8.0
2.5
1.3
9.8
3.0
7.5
5.2
3.8
10.5
8.5
3.0
Num samples
100-
9.5
1.5
4.8
7.8
3.2
5.8
4.8
8.5
6.5
3.8
Rank
2.8
4.8
6.5
4.5
3.5
4.8
7.2
10
4.8
7.2
10.0
3.5
3.2
7.5
2.2
2.0
2.2
8.2
FS
LH-E2E
MLP-EZE
LH
MLP
FS
LH-EZE
MLP-EZE
LH
MLP
Stacking
Figure 12: Average model ranks for models pre-trained and finetuned the first five MetaMIMIC targets. Deep tabular models and Catboost performance is presented on the corre- sponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly. E2E denotes end-to-end training. Rank is averaged across 5 downstream tasks.
Hypertensive, Ischematic, Heart, Overweight. In particular we pre-train on 4 available targets and finetune the remaining task. We limit the number of downstream tasks to 5 for consistency of the number of pre-training targets across downstream tasks. Figure [2 presents the results this experiment, observe similar trends to the original experiments with all targets. In particular, find that deep neural networks pre-trained even on fewer upstream targets definitively outperform Catboost with stacking with MLP model performing better in low-data regimes and FT-Transformer performing better when more downstream data is available.
D.8 WHEN IS TRANSFER LEARNING MOST EFFECTIVE? CASE STUDY
One might wonder when deep neural networks are effective for transferring knowledge between tasks. We hypothesize that transfer learning is most effective in scenarios where upstream and downstream tasks are related. In the previous experiments with pre-training fewer tasks (see Appendix (D77) we found that transfer learning is effective when transferring knowledge between the first five tasks, which include few cardiovascular-related conditions (hypertensive, ischematic, heart), diabetes and overweight conditions. From the correlation matrix in Figure 5 we can see that these tasks are slightly more correlated with each other than with other conditions such as anemia, respiratory, hypotension. purpura, alcohol. We notice that the latter set of tasks is also less related to cardiovascular conditions. To evaluate our hypothesis that transfer learning is less effective when pre-training and downstream tasks are less related, we run experiments with pre-training the model on the first 5 related tasks and transferring knowledge to the set of "less related" tasks. The results for this experiment are presented in Figure 13 we observe that although pre-trained MLP model still provides performance boosts in small data regimes compared to Catboost, the trends are less pronounced compared to results for transferring knowledge to "more related tasks" in Figure 12
D.9 EXPERIMENTS WITH HIGHER DATA REGIMES IN DOWNSTREAM DATA
To expand the experimental evaluation to higher data regimes downstream data we repeat our experiments with 400, 1000 and 2000 samples available for finetuning in the downstream data. The results for this experiment are presented in Figure 14 We find that even in higher data regimes pre-training upstream data provides significant boost to FT- Transsforrerr model compared to Catboost with stacking. At the same time, MLP performs on par with Catboost when there is more downstream data available, which aligns with our previous results.
27

---

Published as a conference paper at ICLR 2023
FT- Transformer
MLP
Catboost
200
3.8
3.8
6,6
10.0
7.0
4.8
4.8
5.0
10.2
7.6
2.2
Numn samples
100-
6.2
4.2
4.0
9.6
6.0
6.6
3.8
10.0
6.4
3.2
Rank
4.8
5.8
6.2
3.4
5.6
10
5.6
7.0
3.8
4
4.4
6.2
8.2
5.6
7.4
3.0
4.8
FS
LH-EZE
MLP-EZE
LH
MLP
FS
LH-EZE
MLP-EZE
EZE
LH
MLP
Stacking
Figure 13: Average model ranks for transferring knowledge to "less related" tasks. Deep tabular models and Catboost performance is presented on the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond the number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly; E2E denotes end-to-end training. Rank is averaged across "less related tasks".
FT- Transformer
MLP
Catboost
2000
5.9
3.1
2.6
8.2
9.2
6.1
4.5
3.8
10.1
7.3
4.5
Num samples
Rank
1000
7.4
3.0
4.2
8.3
7.2
6.0
5.0
4.5
10.2
6.4
3.4
400
7.0
3.1
4.4
9.2
5.5
6.5
5.2
3.8
10.2
5.3
4.2
FS
LH-E2E
MLP-E2E
LR
MLP
FS
LH-E2E
MLP-EJE
LH
MLP
Stacking
Figure 14: Average model ranks in hither-data regimes for downstream tasks. Deep tabular mod- els and Catboost performance is presented on the corresponding panels. Within each panel, columns represent transfer learning setups, and rows correspond to number of available downstream samples. Warmer colors indicate better performance. FS denotes training from scratch (without pre-training on upstream data), LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to-end training. Rank is averaged across all downstream tasks.
D.10 ADDITIONAL EXPERIMENTS WITH ESEUDO-FEATURE METHOD
We further verify the effectiveness of our pseudo- feature method in scenarios with multiple features misaligned between upstream and downstream data. We repeat the experiment from Section 6 but instead of single feature remove three features. In Figure 15 the top left heatmap represents the experiment where the downstream data has three additional features missing from the upstream data. The bottom left heatmap represents the opposite scenario of the upstream data having three additional features not available in the downstream data. To ensure that the features we experiment with are meaningful and contain useful information, we chose important features according to GBDT feature importances. In Figure 15 we include the average ranks of FT- Fr--rsasffomer models pre-trained with missing features, pre-trained with imputed features and with the original features, which we assume unavailable. Additionaly, in the right panel present standard errors for model ranks computed across all downstream tasks. From these experiments, we see that pre- pre-training on data with imputed features significantly outperforms pre-training with missing features and that our feature imputation method enables effective transfer learning in scenarios where upstream and downstream feature sets differ in multiple features.
28

---

Published as a conference paper at ICLR 2023
Upstream -acuto-feature
Upstream pscudo-feature
0.20
200
2.5
1.3
200
0.3
0.3
Rank Standard Error
1.5
Num sumples
Num samples
100
2.2
1.4
1.6
100
0.2
0.3
Rank
0.25
20
2.1
1.7
1.6
20
0.3
0.3
2.0
10
2.2
1.6
0.3
0.30
2.1
1.9
2.5
Missing
Pseudo
ginial
Missing
Pseudo
Original
Downstream
pscudo-feature
feature
Downstream pscudo pscudo-fcature
0.20
200
2.4
1.7
1.4
1.50
200
0.3
0.2
0.3
Rank Standard Emror
Num samples
Num samples
100
2.2
1.5
1.4
100
0.3
0.2
0.3
1.75
0.25
20
2.1
1.8
1.4
20
0.3
0.2
0.3
2.00
10
2.2
1.7
1.6
10
0.3
0.2
0.3
0.30
2.1
1.8
1.6
2.25
0.3
0.2
0.3
Missing
Pscudo
Original
Missing
Pseudo
Original
Figure 15: Pseudo-Feature method for aligning upstream and downstream feature with misaligned features. Left: Comparison of ranks of Franrfosfrrmer model trained on data with missing features, with pseudo-features and with the original features. Right: Standard error for ranks computed across all tasks. Figures in the top row correspond to the experiments with missing features in the upstream data, while figures the bottom row present experiments with missing features in the downstream data.
D.11 ADDITIONAL EXPERIMENTS WITH CONTRASTIVE SELF-SUPERVISED LEARNING
We extend our comparison of supervised pre-training and contrastive pre-training strategies by including two additional corruption hyperparameters À = 0.7, 0.8 for contrastive pre-training used in experiments in Section 5 (which used À = 0.9). We also include SCARF contrastive pretraining strategy proposed in (Bahn et al. 2021). For SCARF method we generate a random mask over features, then for each masked feature we sample uniformly from the joint distribution of that feature (in the training set). Then we compute the InfoNCE loss using these corrupted inputs and the clean inputs for contrastive pretraining. We use the default corruption hyperparameter À = 0.6 from the SCARF paper (Bahri et al. 2021). We compare contrastive pre-training strategies and supervised pre-training in Figure 16 We find that increasing the level of corruption (i.e., decreasing λ) does not lead to improved performance on downstream tasks for contrastive pre-training Also, while the SSL methods outperform training from scratch, especially in the lower data regimes, supervised pre-training consistently better than all considered SSL. methods.
29

---

Published as a conference paper at ICLR 2023
FT- LL-nsformer+L--EAE
FT MrL E2E
200-
3.4
3.2
2.4
3.0
4.7
1.6
3.2
3.5
3.1
2.9
5.3
2.2
Num samples
100-
3.2
2.6
3.9
3.8
4.7
1.5
3.1
2.9
3.4
2.9
3.9
2.0
Rank
20
4.5
2.8
2.3
2.8
3.2
2.8
2.5
10
3.2
2.7
2.9
3.7
2.5
2.8
2.9
3.5
3.1
2.7
4-
2.9
2.3
3.5
3,7
3.8
2.5
2.2
2.2
3.6
3.5
3.6
2.6
FS
0.8
SCARF
Supervised
Contrastive A 0.8
FS
Contrastive A =077
SCARF
Supervised
Contrastive =
Contrastive
Contrastive
Contrastive
Figure 16: Comparison of different pre-training strategies for FT-Transformer. The left panel illustrates end-to-end fine-tuning with linear head and the right plot illustrates end-to-end fine-tuning with MLP head. Within each panel, columns represent pre-training strategies and rows correspond the number of available downstream samples. FS stands for training from scratch, À denotes the corruption parameter in contrastive pre-training, SCARF corresponds to contrastive pre-training strategy from (Bahri al. 202I) and Supervised denotes supervised pre-training on upstream data.
30

---

Published as a conference paper at ICLR 2023 (
200
0,74-
0.00
000
0.759
0.002
00002
002 0.286
0.006
0.807660
1807-66001
0.726-0.002
0.002
0813 001
0.726
0.03 003
8.0002
0,788
001
0.814 00000001
0.7.2-0.000 000
0,818 18800000003
180_0..000
0.001
0,761 00.003
0.785_0.003
0811=0.00 000000
0.699-0.00
0.807-0.000
1.807_0.00
00.003
0010 0.739-0.002
0,746 0.01100001
0.772
0.770 0002 0002
0664- 0.000
0.639_0.
0.00
0.066
0.734-0
0.004
1.0._.00000 0.002
0,752_0.000 .000
0.776_0.00 0.0
0,752
0.728 -0 000 0.760
0.807
0.766.
100
0.678 003
0.740
o08
0.771
00.002
0.716
000002
0.792 001
00700=0000
0.707
20.003 003
0.736
1.00000000 00.002
0.767
=0 001
0,786
000
0.707
0.707=0.00
=007=00000 002
0.799
-00000000
0.729
0030033
0.726
7726=0.002
0,788 =0 001
00.003
0.789-0.000
0.789
-0.00000 00001
0,711
0.72)
001
0.708
1.0-0.000 00.004
0,705
Lo 004
0. 0,668
ond
0.642
006
0.735-o
00.00100 0.0001
0.768
_ 000022
0.747 001
0.770=0.0m 001
Table 11: ROC-AUC scores for all models for "Diabetes", , "Hypertensive" and "Ischematic" downstream tasks. FT denotes FT- FT-Transformer, Tab denotes
Tab TabTransformer. The first two rows in each section correspond to training from scratch, where FS corresponde to deep baseline architecture (wned on subsample of upstream data), and FS-2 shows the results for the same architecture as one used with transfer learning, LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to to dol-tt--nddd end training.
Ischenatic
20
0.644=0.011
0.596=0.011
=100000000 0.002
00.004
0.688_0.00 20.001
0.740
0. 0000
0.63 =0 200000
100000
=0 0.0
0.715 000
0.678=0.000
0,752 20.000
1..00
0.668 0.668
20.0000000 000
0L689
20.002
0,768
1. 0.0
000002
0.758
20.000
0.620=0.00
0.57 20.000
0.675
00.004
0.75 0675_0.001
0.655
0o0
0.628 .006
0.651 0.65-m0.00000
10.000
0.734_o.001
L0.001
0.699
0.659 0oo 0.722
0.644
0.701
0.708
0.688
0.63
0
0.066
0.675
0.75%
0,734
10
0.6562 .004
0.6566 .008
0.664
.00
0.704++0
.000000
0.679
0002
0.707 00T
0.622
000
0.6466+00
000
0,663
002
0,724
0000
0.6362200000
0.00
0.715
0,643 1. 000
0,667
.000
0.662
00222-00000
0,748
10004
0004
000000222
0.7.3
2:0 0000
0.659
n.00
0.632200011
031
0.688 0..000
0,668 Lo cos
0,652
.00000
0.624
0008
0.641 +..000
0,688
1...oo.00 03000
0.6562 o00
4
0.62010 000
0.655
002
0.675
004
0,702+
0.637
00s
0.60 10 006
0.608
1 008
0.627
00.000
0.63 Lo 003
0.677 008
0.58 0.58
+ 00s
0.663 0.663-1..00
0.648 L 00t
0.639
004
0.657
0.713
00L
0,626+
0.715
0.461
0.472
003
0.664
aos
000 0,664
0.625
000
0.625 0000
0.657+0.000
004
0.671
000
117110.00
0.61920
0.628 007
0.
0.
200
0,722 .00
0.701=0
.004
0.760 00002
0.768-
0.716.20000 000000
0,767
0.001
0.689
00000
0.724+n0.00
0.762.1 001
0,767 00000001
0.738 0011
01.688 nn
0.699 L 000
0.692 .0000022
1.753-2n-001
0,753
0002
0.724+0..00
0.764200000
7764::00000 0011
0.701 20.002
0.659.n.00 000
0,727
0.727 0000 oo0
0.664 0004
0.654 .004
0.738 0000
0.756
0.000
0.75622-00
o00
0.759
0,68820 nos
0.66620 nos
0,746 a. 802
0.714g nos
0,760 003
0.06620 uOS
Li DO2
0,757 DoS
0.729 001
0,670 Lo noz
0.68820 nog
0,692 n02
0,753 002
0,762 n01
0,667 nos
0.364 nOT
0.59 nos
0.659 Lo DOS
0.66620 noG
0.66620.0o n04
0.721 no1
0.732 001
0.5142 no4 0.733 no1
0,740=a.00 n01
0,718-0.00)
100
0,759 +0
0.664
0,757
0.697 001 0.68
Hypertensive
0.528 nc0
no4
002
now
003
0.568 u04
0,71620 002
0,711940 n01
0.578LL n01
0.5 1818 005
002
n02
0,55810 nE3
0.685 nos
neg
004
n04
0.491 n01
0.576.1 03
20
0.525
0.707
0.709++
0.708
0,725
0.538++
0.739
(16824
0,752
0.704
0.739
0.570
0.685L0
0.678
0.657
0.517
0.47. 007
0.482 007
4. 008
008
004
012
0.194 nos
LO 002
DOS
0.719 002
0.492 011
002
002
0022
n04
011
to. 004
0ot 0.665 Lo 001
004
004
0.501 +0. 804
0.602 Lo D0S
0.514 nos
0.826 001 0.48 003 0.51010.00 004
10
0,713
0,721
0.713
0.724
0.92.
0,719
0,730
0.442 0 003 0,70110
0.674
0,736
0,711
0.738
0.465
0.496
0.658
0.683
0.738
0.665
0,684
0.453 0110
0,464
0.00
000
0.669
0069-0000 005
0.488-
0658- 0..00
0648 012
0.444
011
0.500
0664-
00t
0.704
000
0.686-6.00
0.06
0,704
0.500 010
0,661
D661-0.008
0.706
0.000
0.638
0.706-0.0m0
0.706
90.00
0.496-0.00
0513_0..0
005
0.68-
0.004
0658
0.53
005
0.63=
000
0.524
0.00s
0.04
0.012
0.46 onx
200
0.802 010
0.779 oss
0.821
008
0.830
0.7022
000
0.755
001
0.677
ooe
0.776
002
0.789 1. 001
0.775 002
0.709920.00
002
0.752
0.700 Lo 002
0.671 000
0.76622-.000
0.623
000
0.707
002
0.744
001
0.691
200000
000
0,71320.00
003
0.727 00s
0.725 Lo co5
0,705 3.0 cod
0.7022
oo8
0.797 001
0.826 Lo 001
0.824
oon
0.
100
0.51 0.540 0.540 004 0. 0.567= 004 0.596620 009
003 0,666200
008
0,770
000
+0.0000000
0.700 ons
0,736.0
002
2. 009
0.67740.000
003
Lo 002
0.633 009
0,700 002
0,72010.000
002
065210
003
0,550 012
0,69
002
0.688
10.000
+0.000 002
0.719 003
0.61620.00
008
0.56310.012
012
20. 000
0,704
005
0,705
005
0,703 008
0.782 001
Lo..0zz 002
0,805
001
0,806 000
0.627
0,205
0,701
0.704
0,807
Diabetes
0.5770=00000
300 004
000
0.654-00.012
20.005
0:002.0.011
011
(50))...00
0.530 0.538-00.00
Lo 002
0.61620..0 007
0.692=0.00 002
0.635
0.515 002
0.543.0.00
004
20.002 2012
0.626.20.0m
004
0.697
0012
0.62722.000
ons
0.517
0.54120.004
on
20.000
0.6944 169410000
003 005
ons
0.703
1355000000 -0.00
onn
1223100000
0.5982n.o00
onn
0.59).0.0m
20
0.710
0.704
0.50)
0.696
01689
0.699
0.703=0
0.623
0.5
00550=0.0006
0.022
20.000
0.524+0.001 0.606 =0.000
0.689 10.004
10.011
0.481 10.011
0.514+0.0mp
10.000
0.445=0.00 000 0. 34220.000
0.61220.0m
0.54622000
20.0m
0.503 OUT 0.491 Lo.00s
0.52920.000
000
20.002
0.589 10.00
0.663 +0.002
0. 20.0000000
1..882n.007 0.487 20.000
0,507 0.50710.000
10.004
0.692#0.000 20.00s
0,692 0.69210.000 Lo.00s
0,691 0.011 10.000
0.698 =0.000
0.514+0.000
Lo.00e
20.0mm
20.004
10
0.0P2
0.660
0. 0.546.2:
0.682
0
0.555
0.525
0,529
0.5
00.1142..0m 0088
0.661 007
1.6902200000 000
0.534 0012 012
00.498500001 4.4
0.484400.00 .00
0.643 ood
(167220 000
0.4572n0007
0.50720.00 000
n.6752n.00
0.536 000
0.685-00002
0.533200.0m
0.077
0.693 000
0.693 000
0.698 000
01698 .008
00,95 0..004
1..07.0.001
n.5322n.om
0.524a0.011 011
Target Task
Num Samples
FT-FS
FT-FS-2
FR-LH-E2E
FFMLP-EZE F2EE
FELI
FT-MPP
ResNer-PS
ResNot-IS-2
Re-Net- EDE
ResNer--MLP-R MLE
RosNet-LHt
ResNet-MLA MLP
MLP.DS
MLP-FS-2
MLP-LII-E2E
MLP-MLP-E2E
MLP-LH
MLP MLPMLP MLP
Tab Tab-FS-2
Tab Tab-Lit LIEE
Tab-Mi. MLP-E2E
Tab-LH
Tab- AdL-
Calboost
Cathoost stacking
XGBBrst
XGBoost-stacking
11

---

Published as a conference paper at ICLR 2023 (
200
0,722
0.07
0.7446.002
0.002
0.762
0.0220
002
00.022 0.773-6.00
0617-0.00
0.000
0.122 003
0,767 0.022
000001
0748-00
7748-0000
001
0.767
000
0.639-0.00
000
0.739-0.00
0.001
0.790
0.000
0.749-0.003
0.758 0.001
0.699_0.omd
00.022
10.002 0.770_0.000
0.779_0.000
0.742-0.00 00.00
0.745 -.50000000 0003
0.590
0.03
0.574=0
0.04
0004
0.778-
0.08-00000
7783_000000
0.283_0. 0,0000
0.783 0,000
- 000 0.796
0.600
0.787
Table 12: ROC-AUC scores for all models for "Heart", "Overweight" and " "Anemia" downstream tasks. FT denotes FT--Trasffommer Tab denotes TabTransfor The first two rows in each section correspond to training from scratch, where FS corresponds to deep baseline architecture (wnedd on subsample of upstream data), and FS-2 shows the results for the same architecture as one used with transfer learning, LH and MLP denote linear and MLP heads correspondingly, E2E denotes end -o--tt--nddd end training.
0.764 003
001
-0 001
-0.000 0.000
003
0,750 003
0.757-0.000
00.00
-0 001
0000
on8
0.739 0.739-0.000
190000000
0000001
TM=00.02
0,757 =0 0010
0.597 -0. 00.003
20.2-20002
20 003
000
L0 001
000004
004
00.00000 0.0001
1 00000
=n onn
000
100
0.782
0.756
0.750
0,610=0
0.705
0.736
0,700
0.635
0.763
0.768
0.706
0.703-6.
0.703
0.762
0.727
0.701=0
0.698
0,592
0.576
0.775
0.781
0,788
0.788
0.
Anenia
20
0,74 =0 0o0
0.735=o.com
000006
20.0000000
-0.000
0.597-0.000
0.687 =0 000
0.698 20.000000000 000004
0,747
000002
00g 0.665
10 0.0
0,727
000000
0.596==0.0m
0.710=20.044
10.000
1.73820.0m 000
0.657
20.002
0.725
20 000
00000
0.694 20.003
0,708
0.729
0000
0.650=0.000
0.000
0.60-0000
0.591 0
= 000
0.572=0.000
0.000
0.760-
0.00000000
0.76310.000 10.001
0,773_00000
0.754 000 0,772=0.002
0.665
0.725
0.597
0.708
0.592
0.694
0.650
0.763
0.7
10
0,709 .o0
0.749 .002
0.646-::
000
0.682
0.5M0
000
0,642.0
0oG
0.653
000
0.752+0..00
0,652
0,697 .0.00
0.593 0000
0.666
0.689 LO 000
0,753
.00000
0.615
0.695 2:00000000 004
0.567
.000
0.666 0000
0.677
0.68P 0.0000
0.600
0.6001-00000
00.004
0,603.0
001
0,567 0.000
0.561 0006
0.765
1765+0.000 0.000
0.754
000
10000
0.7542..0m
4
0.64340.01 013
0.728
0.06
0.639
000
0.662+0.002
0.557 003
0.610
013
0.553 019
0.614+0.019
0,607
003
0,0084 000
0.5633#0.00 004
0.582
0.66 1,64110 Lo 00s
0.733
000
0.61020.000
0.600
00L
0.551
0.655 00s
0.548220.00
000
0.602 000
0.594 0.0004
0,504
0000
0.561 0.561:0 005
#0
0.5600+ 000
0.557 00.000
0
0.504
000
0
0.56
0.
200
0.822
000
0.679:20.00
.004
0,818
8: 00.00
0.788-
0.674
0000
0.718 000
0.665
10.55.00000 007
0.729++0.000
000
0.770 002
0,765
.0000
0668= 0011
(.72622..00)
0.698 Lo 002
0.646 .00000
1.71822-000
0.626.00 0.000
0.686--0.00 0.002
0.689
000000
0.654gn..0m
0.612.2..om
.000
0,726 27.00
0.733 000 oo0
0.679
00.000
0.676220 0.04 .004
0.765 -0.000
0.797 10.001
0.76622.00
0000
0.767
0.
0,582 015
0,638= nos
20.00.000 015
0.675 D02
0.656 DOG
0.642.0 BOS
0.666 1.00-000000 00.004
0,714 001
0,674 n00
0,687 001
0.651 Lo Dos
0.566 DoG
0,683 Do2
0.614 004
00.002
0.646 nos
0,578 n02
0.364 032
0.684 nos
0.684 Lo 003
0.679 noG
0.676 n04
0,764 -..4-0.00 n02
0,789 n01
0,789 n01
0,720=0.00
100
0.669
0.685
0.759
0,792
Overweight
0.623 oce
0.610 no4
t0 012
002
016
DOG
0.700.0 D02
013
n02
0.6510 nea
0.573ao 00a
004
no4
0.547 n04
0.6844 no4
nes
l: 004
no4
0.625 no4
0.672.00 005
0.715 aYo
20
0.728
0.581++
0.682.10
0.507
0.642.
U.615+0
0.568
0.679:30
0.525
(1697
0.609
U.685
0.582.
0L582an
0,68410
0.68 1,68l10
0.674
0,711
0,506 no3
0.490200. nos
BOS
0.675 1 005
016
0.502 011
0.504 n08
do 003
0,545 011
n08
0.544
0.499 Lo Dos
10 003
0.675 002
0.57410 BOS
+0.001
mos
DOS
N07
0.679 004
0.679 Lo 001
0,681 1: 004
0.674 004
0.499 +... nos
007
n04
0.538 D03
10
0.620
0.469
0.40
0.683
0.576
0,506
0.499
0.539
0.500 ons 0.540
0.691
0.489
0.6.8
4
0516-0.000 .012
0.491 00.08 008
0.63
004
0.500-6.00
0669
0.9--.0002
0.002
0.3220 013
0.515-0.00 013
0.495
000 0.699
_0 003
0547
0.698
0.08
0.548
0.533_ -. 00t
0.528
000
0.680
0,599
0.000
0.755
0.597
197_0..00
0510
009
8.00000
0.666
0.004
0676
001
0673
004
0663
0004
0.605
-0.0000000
0614
2614_0.000 0.000
0511=6.005
0.5
0.746 coa
0.748 003
o 002
003
0022
001
oon
0.772 002
001
008
0.765 Lo 001
0.751 001
001
003
001
0.741 000
0,73621.000 004
0.760 004
0.75710 001
0.63620 GoT
0.617 001
0.744 001
0.777 001
0.757 001
0.767 001
002 0.76622-.00
200
0.782
0.7M4
0.7112
0.790.20
0.7 0
0.7601.
0.789
0.677
0.783
0,780..0.co
0.679
0.767
100
0,687 oos
0,762 002
0,770
1770000000
002
0.768+0..00
0.689 200 ons
0,777
001
10.000 004
0.750+0.00
004
065611 002
003
0.661 #o 00s
0,768
20.002
0,7 00t
0,7561.
003
0.67
0.719.0
003
+0.
20. 002
0,731 20, oNY
0,709
770910.000 005
to 20.000
0,705 Lo 005
0,631 10 000
0.61620.00 00r
0.712
0,756
003
0.735 001
0,742 001
0.683
0,761
0.755
0.561
Heart
0.653 ons
00.01 01
300 .02
0.67222.000
22.2000000 001
0:667 003
0.644 10. onG
0044
Lo 002
0.64 0.641 .044
0.668 1.668200002
0.683-000m
20.0m3
0.646 Lo 002
0. 0.669 ona
0.66520.00
001
0.08520.000 20.002
15.00000002 00.002
1.68420.00 20.002
0.645
0.614200000
0n4
0.636 20.0000000
0.636 1061100000 004
0.621 2661 ond
0.600
007
102200.00 001
2774100000
0.64622..0
ons
0.64920..oz
20
0.6
0.630
0.644
1.683
0.085
0.684
0.652
0.674
0.5322 0.557 00s
0.589 20.944
10.000004
0.542+0..00 0.642 0.612=0.007
0.664
0.516 0.66 016 1.640
00.221
+0.010
0.60910.000
10.004
0.61820.006 00,00
0.657=00.0m
0.627 0.627
20.00s
0.426 OUT 0.51210.0100
0.64320.000 000
n.62622..om 0.64420.00
10.000
0.630 0.680=0.002
0.681 20.003
(..012n..0m 0. 0.54520.002
0.56510.000
10.006
0.613
0.613
0.5 0.599 00T 01920.000
0.595=0.008
+0.004
0.6241o.0m
Lo.000
0.56420.0m
0.567 2.56720.00 20.004
10
0.0P2
0.642
0.609
0.565
0.581
0.624
0.517=00.00 0005
0.589 002
0.626.90.00 000
0.489 0.9200.028 028
0.4977 020
0.587 0os
0.497 011
(2612 000
(0.4882.-012
0.570 010
0.629 10 000007
0.610=00.00
n.62029..om
0.517200000 000
0.6022 0002::0.00m
0.602.0 oo0
00.599 0077
0.554 000
0.555L.0.0 0005
0047722n.00
0.458tn.004
Dataset
Num Samples
FT-FS
FT-FS-2
FR-LH-E2E
FF-MP-EEE FZEE
FELI
FT-MPP
ResNer-PS
ResNet-FS-2
Re-Net- LH- EDE
ResNer--MLP-E MLE
RosNet-LIt
ResNet-MLA MLP
MLP.DS
MLP-FS-2
MLP-LIH-EEE
MLP-MLP-E2E
MLP-LH
MLP M.P.LP MLP
Tab Tab-FS-2
Tab Tab-Lit LIEE
Tab- MEEE
Tab-LH
Tab- MLP
Calboost
Cathoost stacking
XGBBrst
XGBoott-stakknng
32

---

Published as a conference paper at ICLR 2023 (
200
0.755
0.00
0.660
0.004
0.0044
0,745
1745-0.002
00002
0.737-6.00
0.724_0.00
0.000
0.740 001
0.69 0.022
0.000
0725-0
001
0,74
00000001
0.71466000
0002
0,746
-0 000 0.59-.
0.000
0665
0.003
0.713-0.003
0,741 001
0.707 0.0001
0.743_0.00
00000
0656-00002
1556_20000
0.600-0.00
0.668 168_0.0000 0.000
0651 0.000
0.635=0. 0007
0.693-0
1203-000000
111__0.000 0.002
0.695 000
0.723-0..mmm 0000
0.664
0.561
0.718.
100
0.640
2.64000.012
0.598
0.598=0.000 008
0.727
=0 002
0.714-1
00.004
0.707
10.000
0.735=0
002
0.595 =0. 004
0.523
00.018
0.635
-0 002
0.7M
000 0.697=
1897=0.00 001
0.742
20:0013
0.598
0.633
003
0.624
122-0.00
0.708
00.00
000033
0.740=0.000
0.740
-0.0000
0000
0.538
000007
0.513 009
0.625
1.5500.000 00000
0.625
_0 000
0.647
o00
0.634 007
0.618 -.8--..00 0.o
0.669
_ 00.005
1.55820.0 002
0.576=00.0
0000
Table 13: ROC-AUC scores for all models for "Respiratory", "Hypotension and "Lipoid" downstream tasks. FT denotes FT-Transforme Tab denotes
Tab TabTransformer The first two rows in each section correspond to training from scratch, where FS corresponds to deep baseline architecture (wnned on subsample of upstream data), and FS-2 shows the results for the same architecture as one used with transfer learning, LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to to -ol--t--nddd end training.
Lipoid
20
0.564 coc
0.574=0.00m 0..00
10 000
-0.000
0.655
20.00
0.689
002
0.581
000
0.574-0.008
0610
0.0
0.657=0.c00 000000
0.620=0.000
0.684 20.000
0.54
0.000
0.54
000
0.624-
0.684
20 0.0004
0.644-0.008
00000
0.701
20.004
0.505=
4487200000
=0.0mm
0.6 0
1.000000
0.619
000
0.621 6621=0.000
0.536 .000 0.547=0.000
0..00
0.555_0.00
L0.000
0.579.0.00
0.545
0.630
0.633
0.581
0,574
0.644
0.701
0.487
0.61
0.62)
0.555
10
0,55220.00
0.530
.002
2:0 000
0.475
0,582
000
0.53
011
0,574
000
0.551+0.000
000
0.571
0oS
0,545
010
0.55220.000
0.551
0.531
015
0,577
00000
1.617
10000000000
0,592
00T
0.577
0044
0.594
0000
0.5102n.000
0.489.0.000
000
0.614
00000
0.614.0
006
0,595
000000
0.618
.008
0.53610.002 002
0.48320.004
0.487 000
0.565
0 0.502
000
0.520
004
0.584
000
0.429+
0.63100000
000
0.415 009
0.503
007
0005+0.000
1o 00t
0.44 0,442 000
0.563420.00 000
0.442
Lo 0os
0.51010
003
0.00220.000
0.430 009
0.608
0.429
000
0.53622 002
0.537 003
0.615
00s
0.615 0.615 000
0.617
000
0.617 008
0.523+00014
0.515
0.390
nos
0,394 000
4
0.505
0.561
0.507
0
200
0.644 000
0.611
0011
0.635
000
0.677-00000
0.553.20.00
.0000
0.637 001
0.578
0111
0.628
0,590 001
0,624 DOS 0.638 00000011
0.524 DOS 0.535=n.00m 000
0.649...000
0.648 Lo 002
0.682
.0.00001
0.59522n.0
0.654.0.000
0.535
8.645200.00 .000
8.6522..0
0.6322n.00 001
0.577=00.00 0000
0,544.0 003 0.572.10.002 001
0,537 000
0.526 0044
0.635 0000
0.643
00000
0.5722n000 on0
0.5162n
0.54640 nce 0,001 no0
0.516 011
0.392 002
D02
001
DOG
0,547 DOS
0.62 Lo 001
20 007
0.45 001
n01
0,56 DOS
0.366 nos
0.540 nos 0.544 0.54420.0m nos
0,533 noG
0.523 no4
0.552+ no1
0,583 001
0.539 nm
0.543 n01
100
0.627
0,538
0.612
0.572
0.589
0,626
0,5632n.00
0.548-0.000
0.628
0,57
Hypotension
012
002
002
003
013
D03
004
no8
0.529 nos
0.587 002
D01
063
0.54010 D02
nea
0.537 004
0.520 no4
0.531 nos
0.54510 nos
0.553 no1
20
0.53930
0.584.10
U.612+
0.545.1
0,604
0.506.0
0.539
0.53110
0.601
0,527
0.621
015603-
0.633
U.557
0.631
(15444
0.54011
0.553
0.518 009
0.552 nos
1. UOS
004
002
007
10.....000 011
806
004
0.543 004
0.603
D0S
003
0.530 002
LO 002
002
0022
noS
to. 004
001 0.524
004
004
804
1.50 10 001
0.509 no2
0.511 n03
10
0.564
0.595
0.533
0.593
0.535
0.535
0.00
0.51310
0,545
0,628
0.550
0.629
0.6629
0.535
0.525
0.523
0.534
0.512
0.490
0.501
4
0.526 0008
0.574 o03
0,564
0.002
002
0.595
0,511
00002
0,500
003
0.516
0os
0.552
0..10
0521
003
0.614
000
0.503
004
0.666
0.504
0. 00t
0.568 003
0.582-0.002
0.608
004
0.133
0611-0.000
0,111
000004
0.547
0.51
0.004
0.526
0.004
0.566
00t
0511
004
0.511
000
0.523
0.000
0.568
00.000000 00002
0.507 004
0,588 001 0,499 ood 0.00= 000
0,561 004
0.577 008
0,585 3:0 001
004
0022
001
004
003
001
002
001
001
002
003
002
002
002
002
003
004
004
002
001
0.496 non
200
0.562
0.564
0.56910
0.550
0.586
0.581
0,578
0.568
0.566.20000
0.576
0.574
0.591
0,583
0.567
0.575
0.567
0.526.1.00
0.5
0.582.0
0,562
0.558
0.572
0,584
0,572 009
0.562 010
0,578 30. 002
00.
0.575 002
004
001
0,583 003
0.570 001
003
004
001
002
002
003
00s
20.004 004
001
004
004 0.558
0.554 002
0.592 002
0.57 20 001
100
0,576+0.00
0,565
0.564
0,363 0333340.00
+0.0000000
0,577.0
0,573
0,564
0,54210
0,57820.00
0,583
0.67740.00
0.577
0.52520.0m
0,49110.00
0.551
0,552
0,56210
Respiratory
0.5 (45 ono
000
30. 002
0.54320.000
002
0.51820.00
004
0.520
210
+0.004
0.517.10 002
0.5 0,511 on3
0.550=20.00:
0.53022000.
LO 006
on3
0.531
0,535 003
+0.001
0.55422.000
20.00o
ons
0.47722.00
0.458 188..0000 o02
20.00000000 ons
Lo 003
004
0.552
004
0003100000
0.49420.0m
0.497=0..0
20
01:84
0.517
0.406
0.503
0,458
0.553
0.504
2.504
0,551
0.49%
1.503
10
0.475 o0G
0.469220.om 20.00
0.498
0.498
20.004
0.521 0.462 #0.007
0.536
10.0000.008
0.500 0.50010.000
0.460 +0.000
0.53010.002
#00000
0,521
0.551 20.001
0,532 0.532
20.0m
0.494 L0.004
0.447 20.000 000
0.447
0.5452n.002 0.51520.001
0.53310.000
0.531+00.00 0.514=0.002
0.545
0.545
20.00s
20.0920.00
0.45410.000
0.
0.522 +0.200
0.522
$22.100000
0.548
0.548 20.000000 000
0.547
#0.005
0.492
+02+00
0,498
Lo.002
0.4981..00
0,495 20.00
0. 0.500=0 cos
10.9922..0m 0005
0.545.0.00 0014
0.556.20.000 0002
0.547 0.01 001
0.499 00T
0.4779+0.000
0.536.10 ood
0.524 000
0.552 0022
0.52002..com
0.492 ooe
0.500 000
0.56410.000 003
n.571:0000m .00
(..6622n.0m
0:461 0.46120.004
0.531=00.004
0.531 001
0.550 000
0.5500 0005
00.422 0001
0.488L 0.00000
0.4932n.0m o0
0.499
Dataset
Num Samples
FT-FS
FT-FS-2
FR-LH-E2E
FF-LP-EZE FZEE
FELI
FT-MPP
ResNer-PS
ResNet-IS-2
Re-Net- EDE
ResNer--MLP-ER MLP-EB
RosNet-LIt
ResNet-MLA MLP
MLP.DS
MLP-FS-2
MLP-LIH-E2E
MLP-MLP-E2E
MLP-LH
MLP MLPMLP MLP
Tab.FF Tab-FS-2
Tab Tab-Lit LIEE
Tab- MLP-EDE
Tab-LH
Tab- MLP
Calboost
Cathoost stacking
XGBBrst
XGBoott-stakkngg
33

---

Published as a conference paper at ICLR 2023
200
0.44000
0.00
000
0.633=0.00 0..00
0.05
0.000
0,700
000
0.700-6.000
0.475_0.00
0.0004
0,703 0.03
0,700-00 0.000
0.003
0,704.
000400000
003
0.720 0000
0.463=6.00
000
0.689_0.00
0632
1.2__...000
0.003
0.556
0.05
06.3-000022
0.03
0.658-0
000
0.454-6.00
0.590_0.0m
1900-..000 0.000
0672-0.008
0.666 0.05
0.005
0.666-0..
0.000
- 0.000
0.489
0.000
0,478=0.
0.08
0008
0.706-0. 0.001
1692_00000 0.001
0,502_ 0002
0.505 0.0000
0.745
0.68
0.692
Table 14: ROC-AUC scores for all models for " "Atrial", "Purpura" and "Alcohol" downstream tasks, FT denotes FI-Transforme frrns, Tab denotes TabTrrasforrmer. The first two rows in each section correspond to training from scratch, where FS corresponds to deep baseline architecture (tuned on subsample of upstream data), and FS-2 shows the results for the same architecture as one used with transfer learning. LH and MLP denote linear and MLP heads correspondingly, E2E denotes end-to-end training.
100
0.606
000
0.635
o00
0.667
=0 003
-0 00.004
0,477=0.00
=17 0..004
003 0.683
0.680
-0.000000000 ons
0.702
1020-00000 00.00a
0.603
=0 003
0.689
0000
0.461
on8
0.66) 0.66)=00000
=000000000
0.618
-0 002
0.626
0000
0,492
002220002
0.581=0.000
004
00.002 0,580
0.580=0.000
000
0.536=0.012
0.538
23882.000 023
0.614
00.000
0.613_0 000
0,404
009
0.481
008
0.6440-
00.000.00222 0.002
0.625
_ 00.05
0.639
100=0.00 001
0.659 0000
0.
Alcohol
20
0.600 coo
0.588=0.000 0000
0.666
0.622
00.000
0.513
0.666
000
0.61=0..00
000
0.616
00.006
0,576
000
0.630
0.0
0.488=0.00m
0.649
20.000
0,584
1.54410.000
0.0000
0.53
000
0.496=0.000
0.555
000
000002
0.439-6.002
0.523=0.00m
0.523
0.552=0.016
0.459
0.476=0.000
0.476
20.00
0.476
1047600.000
0.489 0.000000000000000...
0.467
0000
0.611
1611=0.004
0.599
L0.005
0.613
0.613=0.com
0,570 .110
0.581 .008
2: 00400044
000
011
004
.00
003
000000
1.77100000m
10440-000 031
0.0000
0.473
000
.002
0.501 0000
0.5.362--.00
0.57320.010 030
000000
L0 006
0.00000
0000
+0. .0.000
0.566 00000000007
00000000
0.5482..0m nm
10
0.573
0.509
0.520
0.6066
0,550
0.573++
0.504
0,621
0.617
0.504
0,508
0.517
0,473+
0.501
0.495
0,502
0.474
0.568
0.623 0,5466
4
( 0.5034.0 0o8
0.603
004
1..5 000
0.658
0.4622 003
0.085
0.0
0.56010.002
013
0,567
1367740.011
0.473
00L
0,715 007
0.43620.000
nos
0,719
0.518.Lo
0.0
0.514 008
0.08320.003
0.639 00s
0.460+
0.642
00s
0.626-20.00
0.62310.000
007
0.506
our
0,506
0.4733 007
0.473
008
008
0.541+
002
1.54$10.00 1..00z
0.61820
0.585
0.545
0.62
.00
0,852 .006
0.777 000
0000
0,723 000
0.002
0.775++0.000
0.715 002
0,761 0000004
0.54622.00
0004
0.751
0.749 L ood
.000
1.723-2n.002
002 0.772 001
0.563-+0..0 00000
0.738:50.00 238.:.00000
1.6722..0mm
0.723 00d
0.680200.00
0.000 0000
0.552 .00
0.534 .005
0.875 0000
1.0000
1.877 ood
200
0.552.20.00
0.659
0,750
0.687
01.778
0.851
018788
0,737 o05 0.870 n07
0.778 004
005
DoS
DOG
0.755 8OT
0,769 0.769+0.000
DO2
0,729 DOG
0.512 004
O, 0,740
0.68910 noT 0,754 Lo DOZ
0,71620 noG
0,505=n.n0 nOS
0,505+a 103--0.000 -0.004
000
0,6732n.01
0.654 803
0.617 no4 0.5% 0.599 no0
0,599 Lo DoS
0,557 nos
0.537 nos
0.344 -0 nos
0.804 n01
0,80220.00 n01
100
0,784
0.629
0,561
0.715
0.692
0.728
061810 002 0,687
0,836
012
004
002
008
022
0.697 n00
0.535 no8
0.733 003
D02
0.662.10 nos
0.672 n14
0.66 0.66110 LEo ne4
noa
0.5624o 005
0.535 nos
0.677 nee
0.680.10 now
0.762an nox
Purpura
20
0.45
0.664
U.695
0.543.2
0.615
0.689
0.730
0.659.0
(1.642
0,725
U.564++
0.67710
0.763.5
0,703 010
0.745 nos
LO 002
002
014
Lo NO7
+0. 006
do Dot
0.652 011
0.560 nos
0.617
022
no6
0.595 002
0,688 LO uG3
007
L1 009
0.579 n05
0.579 Lo 065
0.565 1: n0S
0.529 008
0.618 nos
0.64 1. D6S.
0.646-2 no2
0.652 002
10
0.641
0.672
0.538
0.563
0,708
0,702
0.599
0.570
0,696
0.561+
0.647
0.697
0,681
0,711 007
0,763 o0s
0.002
00202
010
007
0.010
0.562 005
0.687 .00
0.533 0333=60008
0. 026
003
0.594_0.0m
0.004
0.688 0.000
0.649 0.003
003
0.000
000
0500 0,05
0.00
0.651 111-0..000
1...000 000
0.572 000
0568-6 004
0.43
0.708
0.511
0647
0.643
0.665
0.53p
0,764
0.699
0.003
0.600
0.572
0.572
0,508
0681
0.690 cos
0.7033 001
0,788 o. 002
002
0022
001
0.625 i.0. 010
002
1. 001
0, 0,748 cod
002
0.707 10 001
001
001
004
002
0.692.1..00 1. 002
0.716 00s
0.720 cos
0.584 cos
0.567 oo8
0.705 + 002
Lo 002
0.583 001
0.574 cos
200
0.752
0.612.
0.772.0
0.749
0.595
0.766
0.7262..00
0.7332n.000
0,753
0.593
0.740
0.625
0.758
0.696
0.749
0.6 007 0,702 &0 oos
0os
10. 002
002
002
0.691 1o 004
0,703 +0. 002
Lo 001
0,732 005
0.587 002
001
0,577 010
ons
001
004
003
011
10. 002
t0. 00s
Lo 00s
008
008
0624+0.000
001
003
001
0.627 001
100
0.665
0,725
0,753+0
051020
0,766.0
0.88
0,770
0,704
0.64620.00
0.7 0.71420.00
0,508
0.745
20.52000000
0,00%
0.688
0.607
0.07
0,587
0.568
0.68
Aurial
20
0.382.00.00 ons
0 0.604
30. 004
+0.005
0.577 002
0.667 20 007
0.599 onG
0.623
0.587
L0 002
0 0,650 ons
0.520=20.022
0.677
20.004
0.603 Lo 003
0.635
ona
0.606620.00
0.681 0.081 100.001
0.588
000
2.679-20.000
0.679
20.002
0.60820.0
0.58320.002
002
0.591
=100000000
1.591.0-000
003
0.5 0.575 007
0.557=0.000
008
0.541 3411=0.00
0.555 000
0.58622.001 001
0.58420.00
10
0.594 01
0.628=0.00s o.0
0.003:1..000
20.000
0.627 +0.000
0.56820.000
0.637
00.595 0005000007 0..00000000
0.573 +0.0000000
0.609 ooe 0.523.0.00
0.627
0.557=00008
0.625.0.0m
20.0mm
Lo.010
0.594
29420.000 000
0) 0.58420.00
20.04
0.689
10.000
+0.004
0.682 0.682
20.003
0.5932n.007 0 0.56320.007
0.547.0.000
10.004
20.007
L0.000
0,578 000
0..55=0.008
0.598- 0011 0.588+00000
0.610...007
0.63220.0m
00.3640.004
0.603
00.699 0.625
0.529
0.593
0.547
0.573
0.5
0.60320 0oe
0.606g20000 0005
0.579 1799.00.00 .00
01605 0.000000
0.543.90000 000
0.587 0.15 015
0.585+00.016
0.671 013
(8.582 0077
0.535 015
0.588 0044
n.5632n.004
0.66220.00 007
0.525+00.00
n.653g0000 0077
0.530200004 000
0.561=:0.0m 000
0.561 00e
0.540 000
0..44 0088
(.610....o 001
0.573gn.om
0.5844...om
Num Samples
FR-LH-E2E
FFALP-EZE FZEE
ResNer-PS
ResNot-1S-2
Re-Net- LH- EDE
ResNer--MLP-E MLP
RosNet-LLH
ResNet-MLA MLP
MLP-FS-2
MIP-LII-E2E
MLP-MLP-E2E
MLP MLP.LP MLP
Tab Tab-Lit LIEE
Tab-Mi. MLP-EDE
Cathoost stacking
XGBoott-stakknng
Dataset
FT-FS
FT-FS-2
FELI
FT-MPP
MLP.DS
MLP-LH
Tab Tab-FS-2
Tab-LH
Tab- MLP
Calboost
XGBBrst
34


Reviews:
[{"review_id": "uSN5mwIxtm4", "reviewer": "Reviewer_1RK9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies transfer learning on deep tabular models. Authors create a transfer learning benchmark from the MetaMIMIC repository, and then conduct transfer learning experiments on it. Specifically, authors find that fine-tuning pre-trained FT-Transformer consistently outperforms classical models such as XGBoost and Catboost with different training data sizes. Authors also find self-supervised pre-training is inferior to supervised pre-training on tabular data. Additionally, to deal with applications with missed features, authors propose to use pseudo values for feature alignments. ", "summary_of_the_review": "Overall, this paper presents useful comparisons of transfer learning between deep Transformers and tree models on tabular data, though the findings are not so surprising. My major concerns are, 1) when (on which kinds of data) shall we use deep Transformers and when to use tree models, and 2) whether is the conclusion still right with fewer pre-training tasks? ", "strength_and_weaknesses": "Strength\n1. Deep learning with tabular data is an important and unsolved problem. \n2. This paper is clearly written. \n3. The proposed approach of pseudo-value completion seems interesting and effective. \n\nWeaknesses\n1. Previous studies (such as FT-Transformer) have already shown that DNNs can generally outperform XGBoost/CatBoost. Therefore, the main findings of this paper sound less surprising. Authors provide no insights about when DNNs transfer better than tree models and when not. \n2. All experiments use 11 splits as the upstream task and 1 as the downstream task. This may not be representative enough for various applications. I wonder whether using less splits in pre-training still results in the same conclusion. \n3. A minor typo: “with without”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dAkpqUuFXEj", "reviewer": "Reviewer_ADx7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies transfer learning for tabular data. The authors take MetaMIMIC for experiments, where there are 12 medical targets. They train on 11 targets and test on the leaveout target. They find that transfer learning provides definitive advantages over gradient boosted decision trees. They further compare self-supervised learning and supervised pretraining on tabular and find that supervised pretraining is more helpful. Finally, they propose a new feature imputation method to learn tackle missing values.", "summary_of_the_review": "1. The chosen targets may be too similar.\n2. No analysis about the targets used. \n3. Missing comparison with previous work. Unclear about the novelty.", "strength_and_weaknesses": "Strengths:\n\n1. The paper studies transfer learning on 12 medical targets.\n\n2. The paper demonstrates the benefits of using supervised pretraining for tabular data.\n\n3. A method for feature imputation is proposed. \n\n\nWeaknesses:\n\n1. All the experiments are performed in the medical domain. The 12 selected targets are closely related. Although the authors also validate on other datasets including Yeast functional genomics data and Emotions data, the targets in each dataset are too similar. Does the author test on other domains? For example, the authors can train on MetaMIMC and validate on Yeast functional genomics data.\n\n2. Similarly, the authors should provide some analysis about the 12 targets of MetaMIMC. How correlated these targets are?\n\n3. There are several works studying transfer learning for tabular data, e.g. TransTab (https://arxiv.org/abs/2205.09328). The novelty of this paper is ok but not significant. Although the paper cites the TransTab, it should not be regarded as concurrent work anymore because the paper is already on arxiv in May. The paper may need to carefully compare the differences with respect to these previous work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XpjXcF7oCvA", "reviewer": "Reviewer_HXuq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "## Summary\n\n* This paper investigates whether deep learning methods are effective for transfer learning on tabular data. In comparison to domains such as NLP and CV, tabular data transfer learning is not as well studied.\n\n* The overall contributions of the paper are: (1) a thorough empirical evaluation of transfer learning methods on tabular data (considering gradient boosted decision trees, training deep models from random initialization, supervised pre-training, and self-supervised pre-training); (2) a strategy for transfer learning with tabular data when the pretraining and finetuning datasets differ in what features are measured; (3) a distilled set of recommendations for empirical work in this space.\n\n\n## More details \n* The paper outlines an empirical testbed for understanding the benefit of transfer learning with tabular data. The authors adopt a medical intensive care unit dataset, MIMIC-IV, for the primary evaluation. This dataset has structured tabular data for a number of patients together with 12 downstream labels of interest. The evaluation setup for supervised pretraining involves: (1) conducting pre-training on 11 of the labels; (2) fine-tuning on the 1 remaining label, with different finetuning dataset sizes, to gauge performance. \n\n* There is also an evaluation on two non-medical datasets in the appendix. \n\n* In this experimental evaluation, the authors find that supervised pre-training outperforms training strong gradient-boosed decision trees (GDBT) baselines  with tuned hyperparameters (Xgboost and Catboost). This is an interesting result, since it demonstrates that neural models can be effective for transfer learning on tabular data. \n\n* The paper evaluates a variety of different neural network strategies in the experimental evaluation, including recently proposed strategies for tabular data such as FT-Transformer and TabTransformer, in addition to simpler methods such as MLPs and ResNets. A variety of different finetuning strategies are also considered for these models (e.g., linear evaluation, end2end training with a linear head/MLP head).\n\n*  The second part of the experimental evaluation compares different pre-training strategies, namely supervised pre-training as compared to self-supervised strategies such as masked imputation. Overall, supervised pretraining strategies perform better than self-supervised pretraining methods.\n\n* The paper then studies a setting where the pretraining and finetuning sets may contain different features, so straightforward transfer learning is not as readily appropriate. To tackle this, the paper suggests a process to impute the missing feature using an additional trained model, before re-running pretraining/finetuning. They find that this strategy improves over just excluding the missing feature (though the evaluation in this setting appears less complete). \n\n* The paper presents detailed information about experimental setups, including considerations of: how to handle the small data regimes, how to do hyperparameter search, ranges of considered hyperparameters, how early stopping applies in such extreme small data regimes. ", "summary_of_the_review": "Overall, I think the experimental evaluation in this paper is thorough and has useful insights for practitioners. I would therefore vote to accept it. ", "strength_and_weaknesses": "## Strengths\n\n* The empirical evaluation in the paper is thorough and clear. The range of investigated methods (state of the art deep learning methods for tabular data, popular tree-based models, self-supervised methods) is very helpful in understanding trends and how these different strategies perform. Additionally, evaluation on multiple datasets of different types helps strengthen the empirical findings.\n\n* The range of experiments and domains studied, together with distilled findings, is useful for practitioners to understand how best to apply these strategies in practice.\n\n* The related work is fairly comprehensive as far as the ML space goes. There are other works that would be helpful to refer to, which are mentioned later in these comments.\n\n## Neutral \n\n* One could argue that the main methodological novelty in this paper comes from the last section, which concerns how to handle situations when the pretraining and finetuning tasks differ in what features they consider. Personally I think the strength of the empirical study and evaluation is valuable as a standalone contribution, even if this is the only 'new' methodological aspect.\n\n## Weaknesses\n\n### Details on empirical study\nGiven the work's main contribution is the empirical study, there are some more details that would be good to include for full completeness.\n1.  A clearer discussion of how many different seeds/splits were considered in experiments.\n2. Why were these particular low data regimes considered, as opposed to other values? What happens if we scale up to larger dataset sizes (even a single run with a large data scale would help round off the discussion)? \n3. Was splitting into train/val/test on the upstream and downstream tasks on a per-patient level? What were upstream dataset sizes on average?\n4. Dataset details are particularly crucial for the medical application domain -- could you provide a 1-2 sentence description of the tasks, and the label prevalences?\n5. What do error bars on the ranks look like (even for a subset of the experiments)?\n\n### Other related works\nAlthough the related work surveys the ML field well, there are some specific relevant works in the medical ML literature that would be worth mentioning somewhere. They deal with structured data timeseries rather than prediction from a single vector, but they also investigate contrastive pretraining, masked imputation pretraining, and supervised pretraining: \n* https://proceedings.mlr.press/v139/yeche21a.html (ICML 2021)\n* https://dl.acm.org/doi/pdf/10.1145/3450439.3451877 (CHIL 2021).\n\n### Other things\n* In the contrastive experiments, how were the corruption hyperparameters chosen? These might significantly impact performance, so should they also be incorporated in the hyperparameter search?\n* This ICLR paper from this year: https://openreview.net/forum?id=CuV_qYkmKb3 studied contrastive learning on tabular data, and performed comprehensive experiments. This might be a better augmentation strategy to compare with, rather than the one studied.\n* In the pseudo feature experiments, what happens if more than one feature needs to be imputed? How does this impact performance? Do the experiments consider removing different features in each run? What does the variance in performance/rank look like?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NuLMe8V7zW", "reviewer": "Reviewer_gnrQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the highly important problem of transfer learning with DNNs on tabular data. This avenue has huge potential in many practical applications. The problem and DNNs for tabular data are generally underexplored so this contribution could be important for the community. They propose a way to use DNNs for transfer tasks in several settings. The proposed schemes are compared to strong tree-based methods on many datasets, indicating the potential of transfer learning on tabular data.", "summary_of_the_review": "Overall I enjoyed reading the paper. I think new solutions to challenges in tabular data are really important for the ML community. The authors did a good job of analyzing and presenting their work. The proposed solution is simple yet seems to be effective even when the number of labeled samples in the target distribution is not high. This can open the door to many follow-up works. I believe that the paper should be accepted.", "strength_and_weaknesses": "Strengths: The paper is well-written and easy to follow. The approach is simple but seems to work in many datasets and across different settings. Expireneltal evaluation is strong and demonstrates the potential of the approach.\nWeaknesses: some related work on regularizations for tabular data is missing; for example\n\n[1] Kadra et al. Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data 2021\n\n[2] Yang et al. Locally Sparse Neural Networks for Tabular Biomedical Data 2022\n\n[3] James Fiedler, Simple modifications to improve tabular neural networks\n\nTo improve the presentation, it would be worthwhile to remind the reader what are MLM and FS in the caption of figure 3.\nAlso, please explain the procedure for downstream pseudo features in the caption of figure 4.\n\nDo you have intuition why this procedure is beneficial? Was this done (or analyzed) in any prior work? This seems related to the procedure commonly used to impute missing values.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster