Revisiting Pretraining Objectives for Tabular Deep Learning
Authors: Ivan Rubachev
Artem Alekberov
Yury Gorishniy
Artem Babenko
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Tabular Data
Pretraining

Abstract: Recent deep learning models for tabular data currently compete with the traditional ML models based on decision trees (GBDT). Unlike GBDT, deep models can additionally benefit from pretraining, which is a workhorse of DL for vision and NLP. 
For tabular problems, several pretraining methods were proposed, but it is not entirely clear if pretraining provides consistent noticeable improvements and what method should be used, since the methods are often not compared to each other or comparison is limited to the simplest MLP architectures.

In this work, we aim to identify the best practices to pretrain tabular DL models that can be universally applied to different datasets and architectures. 
Among our findings, we show that using the object target labels during the pretraining stage is beneficial for the downstream performance and advocate several target-aware pretraining objectives. 
Overall, our experiments demonstrate that properly performed pretraining significantly increases the performance of tabular DL models, which often leads to their superiority over GBDTs.

Under review as a conference paper at ICLR 2023
PRETRAIN THEN FINETUNE: GETTING THE MOST FROM YOUR LABELED DATASET IN TABULAR DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Similarly to computer vision and NLP, deep learning models for tabular data can benefit from pretraining, which is not the case for traditional ML models, such as gradient boosted decision trees (GBDT). Although the pretraining techniques for tabular data are actively studied, the existing works mostly focus on the unsu- pervised pretraining, implying the access to a large amount of unlabeled data addition to the labeled target dataset. By contrast, pretraining in the fully super- vised setting, when the available data is fully labeled and directly represents the downstream tabular task, receives significantly less attention. Moreover, the exist- ing works on pretraining typically consider only the simplest MLP architectures and do not cover the recently proposed tabular models.
In this work, we aim to identify the best practices for pretraining in tabular DL that can be universally applied to different datasets and architectures in the fully supervised setting. Among our findings, we show that using the object target labels during the pretraining stage is beneficial for the downstream performance and advocate several target-aware pretraining objectives. Overall, our experiments demonstrate that properly performed pretraining significantly increases the perfor- mance of tabular DL models on fully supervised problems.
INTRODUCTION
Tabular problems are ubiquitous in industrial MI. applications, which include data described by a set of heterogeneous features, such as learning-to-rank, click-through rate prediction, credit scoring, and many others. Despite the current dominance of deep learning models in the ML literature, for tabular problems, the "old-school" decision tree ensembles (e.g., GBDT) are often the top choice for practitioners. Only recently, several works have proposed the deep models that challenge the supremacy of GBDT in the tabular domain (Arik & Pfister 2020 Gorishniy al. 2021 2022) and
suggest that the question "tabular DL or GBDT" is yet to be answered.
An important advantage of deep models over GBDT is that they can potentially achieve higher performance via pretraining their parameters with a properly designed objective. These pretrained parameters, then, serve as better than random initialization for subsequent finetuning for down- stream tasks. For computer vision and NLP domains, pretraining is a de facto standard and is shown be necessary for the state-of-the-art performance (He al. 2021 Devlin al. 2019). For tabular problems such a consensus is yet to be achieved as well as the best practices of tabular pretraining are be established. While large number of prior works address the pretraining of tabular DL
models YYon et al. 2020 Bahri et al. 2022: Ucar et al. 2021 Darabi et al. 2021). it is challeng- ing to make reliable conclusions about pretraining efficacy tabular DI. from the literature since experimental setups vary significantly. Morcover, most evaluation protocols assume the unlabeled data is abundant and only use a small subset of labels from each dataset during finetuning for eval- uation demonstrating pretraining efficacy, but somewhat limiting the performance of supervised baselines on these datasets. Such protocols of "unsupervised pretraining" are common in vision or NLP domains, where huge "extra" data is available on the Internet.
In contrast, in our work, we focus on the setup with fully labeled tabular datasets to understand if pretraining helps tabular DI. in fully supervised setting and compare pretraining methods to

---

Under review as a conference paper at ICLR 2023
the strong supervised baselines. We argue that this setup is reasonable in practice, since for many tabular problems there is no publicly available relevant data to pretrain on. In this setup, we perform systematic experimental evaluation of several pretraining objectives, identify the superior ones, and describe the practical details of how to perform tabular pretraining optimally. Our main findings, which are important for practitioners, summarized below:
Pretraining provides substantial gains over well-tuned supervised baselines in the fully su- pervised setup.
Simple self- pref-prediotion based pretraining objectives are comparable to the objective based on contrastive learning. To the best of our knowledge, this behavior was not reported before in tabular DL.
The object labels can be exploited for more effective pretraining. In particular, we de- scribe several "target-aware" objectives and demonstrate that they often outperform their "unsupervised" counterparts.
The pretraining provides the most noticeable improvements for the vanilla MLP architec- ture. In particular, their performance after pretraining becomes comparable to the state-of- the-art models trained from scratch, which is important for practitioners, who are interested in simple and efficient solutions.
The ensembling of pretrained models is beneficial. It indicates that the pretraining stage does not significantly decrease the diversity of the models, despite the fact that all the models are initialized by the same set of parameters.
Overall, our work provides a set recipes for practitioners interested in tabular pretraining. which results in higher performance for most of the tasks. The code of our experiments is available online
2 RELATED WORK
Here we briefly review the lines of research that are relevant to our study.
Status quo in tabular DL. A plethora of recent works have proposed a large number of deep mod- els for tabular data (Klambauer et al., 2017:
Popov et al. 2020 Arik & Pfister 2020) Song et al.]
2019 Wang et al., 2017: Badirli et al., 2020 Hazimeh et al.) 2020, Huang et al. [0200] Gorishmiy
et al.,.. 2021 Kossen et al.) 2021). Several systematic studies, however, reveal that these models typi- cally do not consistently outperform the decision tree ensembles, such as GBDT (Gradient Boosting Decision Tree) (Chen & Guestrin 2016; Prokhorenkova et al.] 2018 Ke et al., 2017). which are typically the top-choice in various ML competitions (Gorishniy et al. 202 Shwartz-ZZiv Armon 2021]. Additionally, several works have shown that the existing sophisticated architectures are not consistently superior to properly tuned simple models, such as MLP and ResNet GGorishni et al., 2021 Kadra et al. 2021). j. Finally, the recent work (Gorishniy et al. 2022) has highlighted that the appropriate embeddings of numerical features in the high-dimensional space are universally ben- eficial for different architectures. In our work, experiment with pretraining of both traditional MLP-like models and advanced embedding- based models proposed in Gorishniy et al. 2022)..
Pretraining in deep learning. For domains with structured data, like natural images or texts, pre- training is currently an established stage in the typical pipelines, which leads to higher general performance and better model robustness (He et al. 2021 Devlin al. 2019) Pretraining with the auto-encoding objective was also previously studied as regularization strategy helping in the optimization process without large scale pretraining datasets (Brhan et al. 2010; El-Nouby et al.,
2021 Krishna et al.) 2022). During the last years, several families of successful pretraining meth- ods have been developed. An impactful line of research on pretraining based on the paradigm of contrastive learning, which effectively enforces the invariance of the learned representations to the human-specified augmentations (Chen et al., 2020 He et al. 2020). Another line of methods ex- ploits the idea of self- prediction, i.e., these methods require the model to predict certain parts of the input given the remaining parts (He et al. 2021] Devlin et al., 2019). In the vision community, the seff-prediction based methods are shown to be superior to the methods that use contrastive learning
'Code: htttps://aaonymmouuss wwtpppp...iinneer//..e//
2

---

Under review as a conference paper at ICLR 2023
objectives (He et al., 2021). In our experiments, we demonstrate that seff-prediction based objectives are comparable to the contrastive learning ones on tabular data, while being much simpler.
Pretraining for the tabular domain. Numerous pretraining methods were recently proposed in several recent works on tabular DL (Arik & Plister, 2020, Yoon et al.) 2020; Darabi et al. 202T
Ucar et al. 2021 Somepalli al. 2021 Kossen al. 2021 Padhi al. 2021). However, most of these works do not focus on the pretraining objective per se and typically introduce it as a com- ponent of their tabular DL pipeline. Moreover, the experimental setup varies significantly between methods. Therefore, it is difficult to extract conclusive evidence about pretraining effectiveness from the literature. To the best of our knowledge. there is only one systematic study on the tabular pretraining (Bahri et al. (002)). but its experimental evaluation is performed only with the simplest ML.P models, and we found that the superiority of the contrastive pretraining. reported (Bahri et al. (2022), does not hold for tuned models in our setu, where contrastive objective is comparable to the simpler self- self- ceff-prediction objectives.
REVISITING PRETRAINING OBJECTIVES
In this section, we evaluate the typical pretraining objectives under the unified experimental setup on the number of datasets from the literature tabular DL. Our goal is to answer whether pretraining generally provides significant benefits over tuned models trained from scratch and to identify the pretraining objectives that lead to the best downstream task performance.
3.1 EXPERIMENTAL SETUP
We mostly follow the experimental setup from Gorishniy et al. (2021) and describe its main details here for completeness.
Notation. Each tabular dataset (X, Y) is represented by a set of pairs ((2,, y:) where æ; (z)...., x² E X are the object's 's features (both numerical and categorical) and ys E the target variable. The downstream task is either regression Y = E or classification Y {1,..., k}. Each model has the backbone (x((0) that followed by two separate heads: a pretraining head h ()) and downstream task head {=([)), with learnable parameters 0, μt, respectively, and f(x denotes the output of the backbone for an input object .r.
curated set of eleven middle to large scale datasets used in prior literature on tabular deep learning. The benchmark is biased towards tasks, where tuned MLP models were shown to be inferior GBDT (Gorishniy et al. 202T) since aim to understand if pretraining can help the deep models to beat the "shallow" ones. The datasets represent diverse set of tabular data problems with classification and regression targets, The main dataset properties are summarized in Table
Datasets. We evaluate the pretraining methods
We report ROC-AUC for all binary classification datasets, accuracy for multi-class classification datasets and RMSE for regression datasets, with OT being the one exception, where we report log- loss, as it was used as a default metric in the corresponding Kaggle competition. We use the quantile- transform from the Scikit-learn library (Pedregosa et al., 2011) 1) to preprocess the numerical features for all datasets except OT, where the absence of such transformation was shown to be superior Additional information about the datasets is provided in Appenddix A
(Gorishniy et al. 2022
GE
CH
CA
НО
OT
HI
FB
AD
WE
CO
MI
#objects
9873
10000
20640
22784
61878
98049
197080
48842
397099
581012
1200192
mrum.features.
32
10
8
16
93
28
50
123
54
136
featurrsss
0
1
0
0
0
1
8
0
0
0
metric
Acc.
AUC
RMSE RMSE Log. Loss
AUC
RMSE
AUC
RMSE
Acc. 7
RMSE
#classes
Table I: Datasets used for the experiments.
3

---

Under review as a conference paper at ICLR 2023
Models. We use MLP simple deep baseline to compare and ablate the methods. Our imple- mentation of MLP exactly follows Gorishniy et al. (2021), the model is regularized by dropout and weight decay. As more advanced deep models, evaluate MLP equipped with numerical feature emheddings, specifically, target-aware piecewise linear encoding (MLP-T-LRR) and embeddings with periodic activations (MI.P-P.R) from Gorishniy al.] (2022). These models represent the current tate-of-theearr solution for tabular DL (Gorishniy et al., (022)) and are of interest as most prior work on pretraining in tabular DL focus on pretraining with the simplest MLP models in evaluation. The implementation of models with numerical embeddings follows Gorishniy et al. (2022). We use AdamW (Loshchilov & Hutter 2019) optimizer, do not use learning rate schedules and fix batch sizes for each dataset based on the dataset size (seelAppendix A).
Pretraining is always performed directly on the target dataset and does not exploit additional data. The learning process thus comprises two stages. On the first stage, the model parameters are op- timized w.r.t. the pretraining objective. On the second stage, the model is initialized with the pre- trained weights and finetuned on the downstream classification or regression task. We focus on the fully-supervised setup, i.c., assume that target labels are provided for all dataset objects. Typically, pretraining stage involves the input corruption: for instance, to generate positive pairs in contrastive- like objectives or to corrupt the input for reconstruction in self-prediction based objectives. We use random feature resampling as a proven simple baseline for input corruption in tabular data (Bahri et al. 2022 Yoon et al. 2020), we tune the corruption probability in all experiments. Learning rate and weight decay are shared between the two stages (see Table 15 for the ablation). We fix the maximum number of pretraining iterations for each dataset at 100k. On every 10k-th iteration, we compute the value of the pretraining loss using the hold-out validation objects for early-stopping on large-scale WE, CO and MI datasets. On other datasets we directly finetune the current model every 10k-th iteration and perform early-stopping based on the target metric after finetuning (we do not observe much difference between early stopping by loss or by target metric, see Table 16).
Hyperparameters & Evaluation. Hyperparameter tuning is crucial for a fair comparison, therefore, we use Optuna (Akiba et al.) (019) to optimize the model and pretraining hyperparameters for each method on each dataset. We use the validation subset of each dataset for byperparameter tuning. The exact search spaces for the hyperparameters of each method are provided in Apppendix B
We run the tuned configuration of each pretraining method with 15 random seeds and report the av- erage metric the test splits. When comparing to GBDT, we obtain three ensembles by splitting the fifteen single model predictions into three disjoint subsets of five models and averaging predictions within each subset. Then, we report the average metric over the three ensembles.
3.2 COMPARING PRETRAINING ORJECTIVES
Here we compare the contrastive learning and self-prediction objectives from prior work in the de- scribed setup. For contrastive learning, we follow the method described in Bahri et al. (0022): use InfoNCE loss, consider corrupted inputs è as positives for x and the rest of the batch as negatives. For self-prediction methods, we evaluate two objectives: the first one is the reconstruction of the original x, given the corrupted input è (the reconstruction loss is computed for all columns). the sec- ond one the binary mask prediction, where the objective predict the mask vector m indicating the corrupted columns from the corrupted input át. The results of the comparison are in Table 2 We summarize our key findings below.
Contrastive is not superior. Both the reconstruction and the mask prediction objectives are prefer- able to the contrastive objective. The two self self-prediction objectives have the advantage of being conceptually simpler, and easier to implement, while also being less resource-intensive (no need for the second view of augmented examples in each batch, simpler loss function). We thus recommend the self- deff-rreiition based objectives as a practical solution for pretraining in tabular DL.
Pretraining is beneficial for the state-of-the-arr models. Models with the numerical feature em- beddings also benefit from pretraining with either reconstruction or mask prediction demonstrating the top performance on the downstream task. However, the improvement is typically less noticeable compared to the vanilla MLPs
There is no universal solution between self-prediction objectives. We observe that for some datasets the reconstruction objective outperforms the mask prediction (OT, WE, CO, MI), while
4

---

Under review as a conference paper at ICLR 2023
GE f CH1 CA
HO
OT L
HI
FB 4
AD1
WEL
CO
MI
MLP
no pretraining
0.635 0.849 0.506
3.156
0.479 0.801 5.737 0.908
1.909
0.963
0.749
contrastive
0.672
0.855
0.455
3.056
0.469 0.813
5.697 0.910
1.881
0.960
0.748
rec
0.662
0.853
0.445
3.044
0.466
0.805
5.641
0.910
1.875
0.965
0.746
mask
0.691 0.857
0.454
3.113
0.472
0.814 5.681
0.912
1.883
0.964
0.748
MLP-PLR
no pretraining
0.668 0.858 0.469
3.008 0.483 0.809 5.608 0.926
1.890
0.969
0.746
rec
0.667 0.852
0.439
3.031 0.472 0.808 5.571
0.926
1.877
0.971
0.745
mask
0.685 0.863 0.434 3.007 0.477 0.818
5.586 0.927
1.911
0.970
0.748
MLP-T-LR
no pretraining
0.634 0.866 0.444 3.113 0.482 0.805 5.520 0.925
1.897 0.968
0.749
rec
0.652 0.857 0.424 3.109 0.472 0.808 5.363 0.924 1.861
0.969
0.746
mask
0.654 0.868 0.424 3.045 0.472 0.818 5.544 0.926 1.916 0.969 0.748
Table 2: Results for pretraining deep models with different objectives. We report metries averaged over 15 seeds, bold entries correspond to results that are statistically significantly better (we use Tukey HSD test). The comparisons are separate for different models. t corresponds accuracy and ROC-AUC metrics, corresponds to RMSE and log-loss for OT. "no pretraining" stands for the supervised baseline, initialized with random weights.
on others the mask prediction is better (GE, CH, HI, AD). We also note that the mask prediction objective sometimes leads to unexpected performance drops for models with numerical embeddings (WE, MI), we do not observe significant performance drops for the reconstruction objective.
The main takeaway: simple pretraining strategies based on self-prediction lead to significant im- provements in the downstream accuracy compared to the tuned supervised baselines learned from scratch across different tabular DL models and datasets. In practice, we recommend trying both re- construction and mask prediction tabular pretraining baselines, either one might show superior performance depending on the dataset being used.
4 TARGET-AWARE PRETRAINING OBJECTIVES
In this section, show that exploiting the target variables during the pretraining stage further increase the downstream performance. Specifically, we evaluate several strategies to leverage infor- mation about targets during pretraining, identify the best and compare them GBDT. Below we describe a list of target-aware pretraining objectives that we investigate.
Supervised loss with augmentations. A straightforward way to incorporate the target variable into the pretraining is by using the input corruption as an augmentation for the standard supervised learning objective. An important difference of this baseline in our setup the one in Bahri et al. (022) is that we treat learning corrupted samples as a pretraining stage and finetune the entire model on the full uncorrupted dataset afterwards (we ablate this (subsection 5.3).
Supervised loss with augmentations self-prediction. We evaluate a natural extension to the above baseline: a combination of the supervised objective with the unsupervised self-prediction. Note, that during the pretraining stage both losses are calculated on corrupted inputs, while the finetuning is performed on the non-corrupted dataset. For the self-prediction objectives we evaluate both the reconstruction and the mask prediction. We use different prediction heads for supervised and self predictionn objectives. We sum supervised and self predicreditiion losses with equal weights.
Target-aware pretraining. An alternative the approaches described above is the modification of the pretraining task itself. An example of this approach is supervised contrastive learning (Khosla et al. 2020), where the target variable is used to sample positive and negative examples. We intro- duce the target variable into the self- aeffprrediction based objectives with two modifications.
First, condition the mask prediction the reconstruction head on the original input's target by concatenating the hidden representation from the backbone network = f() with the tar- get variable representation before passing it to the pretraining head obtain predictions p
5

---

Under review as a conference paper at ICLR 2023
GE f
CH1 CA
HO 4
OT 4
HI
FB
AD
WE
CO
MI
Avy. Rank
ML.P
pretraining
0.635
0.849
0.506
3.156
0.479
0.801
5.737
0.908
1.909
0.963
0.749
5.5 + 1.4
mask
0.691
0.857
0.454
3.118
0.472
0.814
5.681
0.912
1.883
0.964
0.748
3.8
- 1.4
rec
0.662
0.853
0.445
3.044
0.466
0.805
5.641
0.910
1.875
0.965
0.746
3.6 - 1.5
sup
0.693
0.856
0.441
3.077
0.459
0.814
5.689
0.914
1.883
0.968
0.748
3.0 + 1.0
mask larget
0.683
0.857
0.434
3.056
0.468
0.819
5.633
0.914
1.876
0.965
0.748
2.9
2.9 13
rec + target
0.659
0.853
0.454
3.044
0,463
0.806
5.636
0.909
1.884
0.965
0.745
3.7 1.
mask + sup
0.693
0.857
0.436
3.099
0.458
0.817
5.685
0.915
1.873
0.967
0.748
2.7 1.2
rec + sup
0.684
0.854
0.436
3.012
0.456
0.815
5.672
0.911
1.862
0.967
0.747
2.6 - 1.5
MLP-PLR
no pretraining
0.668
0.858
0.469
3.008
0.483
0.809
5.608
0.926
1.890
0.969
0.746
3.5 1.7
mask
0.685
0.863
0.434
3.007
0.477
0.818
5.586
0.927
1.911
0.970
0.748
2.8 + 1.7
rec
0.667
0.852
0.439
3.031
0.472
0.808
5.571
0.926
1.877
0.971
0.745
2.6
+ 1.2
sup
0.710
0.859
0.433
3.136
0.479
0.811
5.521
0.924
1.873
0.971
0.748
2.5 1.2
mask target
0.694
0.862
0.425
3.023
0.474
0.821
5.537
0.929
1.911
0.969
0.749
2.5 1.9
rec + target
0.688
0.860
0.445
3.064
0.475
0.812
5.507
0.927
1.887
0.971
0.748
2.7 1.3
mask sup
0.711
0.866
0.441
3.129
0.480
0.813
5.480
0.925
1.875
0.969
0.745
2.5
1.4
sup
0.709
0.858
0.433
3.059
0.465
0.807
5.571
0.927
1.865
0.971
0.745
1.9
1.2
MLP-T-LR
no pretraining
0.634 0.866
0.444
3.113
0.482
0.805
5.520
0.925
1.897
0.968
0.749
3.9 + 1.7
mask
0.654
0.868 0.424
3.045
0.472
0.818
5.544
0.926
1.916
0.969
0.748
2.8 - 1.7
0.652 0.857 0.424
3.109
0.472
0.808
5.363
0.924
1.861
0.969
0.746
2.5 1.4
sup
0.682 0.860 0.430
3.135
0.471
0.807
5.525
0.927
1.893
0.971
0.747
2.8 1.5
mask + target
0.649
0.865 0.421
3.058
0.474
0.820
5.644
0.929
1.924
0.969
0.749
2.
rec + target
0.668 0.864 0.440
3.113
0.473
0.806
5.493
0.927
1.862
0.969
0.746
2.5
- 1.4
mask sup
0.676
0.858 0.429
3.199
0.468
0.814
5.510
0.926
1.869
0.971
0.748
2.5 0.8
rec + sup
0.678 0.865 0.437
3.112
0.462
0.807
5.516 0.927
1.862
0.970
0.748
2.4 1.2
Table Variations of the target-aware pretraining schemes. Notation follows Table 2 Bold results indicate statistically significant winners across all models and methods. "+ target" denotes target- conditioned pretraining, "+ sup" denotes auxiliary supervised head.
h(concat(2, yl). For classification datasets we encode y with one-hot-encoding, for regression targets we use the standard scaling.
Second, we change the input corruption scheme by sampling the replacement from the feature target conditional distribution where a target is different to the original. Intuitively, corrupting the input object x in the direction of the target different the original makes the pretraining task more cor- related with the downstream target prediction. Concretely, given an object- strect-tagge pair (æ;, Vi), we sample a new target ÿE from a uniform distribution over the set {y y A y: then each feature xi is replaced with sample from the p(r² (ix) distribution, instead of p(x)).
4.1 COMPARING TARGET-AWARD OBJECTIVES
Here we compare the strategies of incorporating the target variable into pretraining. The results of the comparison are in Table 3 Our key findings are formulated below.
Supervised loss with augmentations another strong baseline for MLP. Pretraining with the supervised loss corrupted data consistently improves supervised training from scratch for the MLP. This objective is strong baseline along with the self prediccin based objectives. However, for models with numerical embeddings the supervised objective with corruptions is less consistent and sometimes is inferior to training from scratch, thus for these models we recommend the self- prediction objectives alone as baselines.
For regression problems, when the target variable is continuous, we preliminarily discretize it into n uni- form bins, where chosen according the Freedman Diaconns rule (Freedman & Diaconis 1987).

---

Under review as a conference paper at ICLR 2023
GE 1
CH 1
CA 4
HO 4
OT
HI f
FB
AD
WE 4
CO
MI /
Avg. Rank
CatBoost XGBoost
0.692
0.864
0.430
3.093
0.450
0.807
5.226
0.928
1.801
0.967
0.741
2.6 1 1.4
0.683
0.860
0.434
3.152
0.454
0.805
5.338
0.927
1.782
0.969
0.742
3.0 = 1.5
MLP
pretraining 0.656
0.852
0.482
3.055
0.467
0.805
5.666
0.910
1,850
0.968
0.747
4.8 111
mask target 0.709
0.860
0.414
2.949
0.457
0.828
5.551
0.916
1.809
0.969
0.746
2.8
1.2
rec + sup 0.709
0.859
0.419
2.951
0.442
0.817
5.531
0.913
1.801
0.973
0.745
2.5
1.2
MLP-PLR
pretraining 0.695 0.864
0.454
2.953
0.470
0.814
5.324
0.928
1.835
0.974
0.744
2.6 1.2
mask target 0.719
0.866
0.407
2.952
0.458
0.828
5.373
0.930
1.849
0.973
0.745
2.1 1.2
rec + sup
0.737 0.862
0.424
2.964
0.449
0.811
5.124
0.929
1.813
0.974
0.744
2.0 1.0
MLP-T-LR
pretraining 0.662 0.868
0.437
3.028
0.472 0.808
5.424
0.927
1.850
0.972
0.747
3.7
mask target 0.673 0.868 0.410 2.894
0.460
0.827
5.458
0.930
1.849
0.972
0.746
2.4 144
rec + sup 0.705 0.866 0.425 3.057 0.444 0.814 5.422
0.927
1.811
0.974
0.746
2.5
+ 1.1
Table 4: Comparison of pretrained models GBDT. Notation follows ensembles of models. Bold entries correspond to the overall statistically significant best entries.
Table 2 Results represent
Target-aware objectives demonstrate the best performance. Both the supervised loss with self- prediction and modified self-prediction objectives improve over the unsupervised pretraining base- lines across datasets and model architectures. For the objective with the combination of supervised and self- prediction losses the variation with the reconstruction loss is the most consistent across models and dataset with performance drops below the pretraining- free baseline. The variant with mask prediction shows similar performance and stability for the MI.,, but not as good as recon- struction for models with numerical embeddings. For the target-aware self-prediction objectives, the modified mask prediction delivers significant improvements over its unsupervised counterpart. Modified reconstruction objective, however, does not improve over its unsupervised counterpart.
Targer-aware objectives help further increase the downstream performance, im- proving upon their unsupervised counterparts. For the reconstruction based self-prediction baseline the addition of the supervised loss is most beneficial ("rec + sup" from Table 4). for the mask pre- diction objective it's target-aware modification provides more improvements ("mask + target" from Table 4). A simple MLP model pretrained with those "target-aware" objectives often reaches or surpasses complex models with numerical embeddings trained from seratch. In practice, we recom- mend first trying the baseline pretraining objectives ("rec", "mask", "sup" from Tablle 4). choosing the suitable baseline for the dataset and improving it accordingly: supervised loss for the reconstruc- tion and target- aware modification for the mask prediction.
Main takeaways:
4.2 COMPARISON GBDT
Here compare MLPs and MLPs with numerical feature embeddings pretrained with the super- vised loss with reconstruction and the target-aware mask prediction objectives to the GBDTs. Ta- 4hoows the results of the comparison.
We observe that both pretraining with target aware objectives and using numerical feature embed- dings consistently improve the performance of the simple MLP backbone. In particular, MLP cou- pled with target aware pretraining starts to outperform GBDT on datasets (GE, CA, OT, HI). Com- bined with numerical feature embeddings, pretraining improves MLP performance further, making it superior to GBDT on the majority of the datasets, with two exceptions in WE and MI.
5 ANALYSIS
5.1 INVESTIGATING THE PROPERTIES OF PRETRAINED MODELS
In this section, we provide a possible explanation of why the incorporation of the target variable into pretraining lead to better downstream task performance. We do this through the experiments on
7

---

Under review as a conference paper at ICLR 2023
Figure 1: The decodability of ob- ject feature from the intermedi- ate representations computed by the pretrained models and the models trained from scratch. The pre- trained models decently capture the information about all the features, while the randomly initialized mod- els capture the most informative features and suppress the others.
4x
MLP
MLP (mask)
30-10
MLP (mask target)
RMSE
2 1x11
10-1
6x 10
Importance rank
GE 1
CH1
CA
HO 1
OT 1
HIt
FB L
AD
WE CO 1
MI
single
0.683 0.857 0.434 3.056 0.468 0.819 5.633 0.914 1.876 0.965 0.748
standard ensemble 0.709 0.860
0.414 2.949 0.457 0.828
5.551 0.916 1.809 0.969 0.746
efficient ensemble 0.702 0.861
0.411 2.967 0.461 0.825
5.590 0.917 1.820 0.969 0.746
Table 5: Efficient ensembling for MLP mask + target. Efficient ensemble consists of models finetuned from one pretrained checkpoint, standard ensemble consists of models from different pretraining runs (random seeds).
the controllable synthetic data. Here we describe the properties and the generation process of the data and our observations on the differences of the pretraining schemes.
We follow the synthetics generation protocol described Gorishniy et al. (202T) with a modifi- cation that allows for the manual control of the feature importance for the particular prediction task. Concretely, we generate the objects features { as samples from the multivariate Gaus- sian distribution with zero mean and covariance matrix Σ with identical diagonal and constant e = 0.5 everywhere else. Additionally, we sample a vector p E R from the Dirichlet distribution p ~ Dir(Im) and let p define the influence of the objects features on the target by sampling features for tree splits with probabilities defined by this vector. Intuitively if a particular feature is often used for splitting in the nodes of decision tree, it would have more influence on the target variable. In- deed, we find the feature importanced correlate well with the predefined vector p. For more details on the synthetic dataset construction see (ppenddix C
For each generated dataset, we then check whether the finetuned models capture the information about object features their representations. Specifically, we train an MLP to predict the value of the i-th object feature given the frozen embeddings produced by finetuned network initialized from (a) random initialization, (b) mask prediction pretraining, (c) target-aware mask prediction pretrain- ing. The separate MLP is used for each feature and the RMSE learning objective is used. Then we report the RMSE on the test set for all features E [0, m] along with their importance rank in the dataset on Figure (we use CatBoost feature importances). Here, the lower ranks correspond to the more important features. Figure reveals that the target-aware pretraining enables the model to capture more information about the informative features compared to the "unsupervised" pretraining and, especially, to the learning from scratch. The latter one successfully captures the most informa- tive feature from the training data, while suppressing the less important, but still significant features. We conjecture that this is the source of superiority of the target-aware pretraining.
5.2 EFFICIENT ENSEMBLING
Here we show, that is possible to construct ensembles from one pretraining checkpoint (pretrained with the target conditioned mask prediction objective). To this end, we finetuning with 15 different random seeds starting from the one pretrained checkpoint. Table shows the results.
Both ensembling the models from a shared pretrain checkpoint and ensembling multiple independent pretraining runs produces strong ensembles, which shows that it is sufficient to pretrain once and create ensembles by several independent finetuning processes. This is important in practice since finetuninig is typically cheaper (i.e. requires fewer iterations), and still is able to produce diverse models from the one pretraining checkpoint for ensembles of comparable quality.
Comppted with the CatBoost method "get feature importance
8

---

Under review as a conference paper at ICLR 2023
GE f CH1 CA HО 4 OT HIT FB L. AD1 WE
CO
MI
no pretraining
0.635 0.849 0.506 3.156 0.479 0.801 5.737 0.908 1.909 0.963 0.749
aug
0.693 0.856 0.441 3.077 0.459 0.814 5.689 0.914 1.883 0.968 0.748
aug no finetune
0.674 0.853 0.464 3.251 0.461 0.808 6.167 0.910 1.938
0.958 0.752
rec + sup
0.684 0.854 0.436
3.012
0.456 0.815 5.672 0.911
1.862
0.967 0.747
rec + sup no finetune 0.683 0.853 0.467 3.232 0.474 0.811 6.044 0.907 1.901 0.956 0.752
Table 6: Finetuning MLP on clean data versus using the model trained on corrupted inputs only.
GE
CH1
CA 4 HO + OT 4
HII 1
FB 4
AD 1
WE CO
MI 4
CatBoost
0,683 0.864 0.433 3.115 0.457 0.806 5.324
0.927
1.837
0.966
0.743
34s
214s
32s
169s
1869s
1317s
MLP (no pretraining)
0,635
0.849 0.506 3,156 0,479 0.801
5.737
0.908
1.909
0.963
0.749
159s
12s
57s
352s
211s
MLP (rec + sup 50k) 0.679
0.857
0.441
3.064
0,462
0.813
5.650
0.910
1.879
0.966
0.747
4038
242s
5938
292s
MLP (ree + sup 150k) 0.692 0.859
0.435
3.012
0.456 0.816
5.629
0.910
1.866
0.968
0.746
1069s
927s
665s
Table Training times for CatBoost and MLP. Second row in each group reports average time spent training one model in seconds on an A100 GPU for MLPs and on CPU for CatBoost.
5.3 ON IMPORTANCE OF FINETUNING ON CLEAN DATA
Here we show, that the second stage of finetuning the model on the entire dataset without input cor- ruption often necessary for the best downstream performance. To this end we compare finetuning the models clean data with using models right after pretraining for two objectives: supervised loss and supervised loss with the reconstruction objective.
Across all datasets for both methods finetuning on uncorrupted data with the supervised loss proves to be essential for the best performance. Sometimes excluding the second finetuning stage degrades the performance below the tuned supervised baseline of training from scratch.
Results shown in Table 6
5.4 DOES PRETRAINING REQUIRE MORE COMPUTE?
In this section we investigate how much more time is spent pretraining, compared to training the models from scratch. We run pretraining with "rec + sup" objective with 50k and 150k pretraining iterations thresholds (carly-stopping, in theory, could make 150k iterations equivalent to the 50k, but in practice it was not the case). We report downstream performance along with average time spent to pretrain and finetune a model in Table (we also add CatBoost training times to the table).
Pretraining often requires by an order of magnitude more compute, it is especially apparent on smaller scale datasets like GE, CH, CA, HO, OT, HI, AD. However, the absolute time spent pretraining is still acceptable, as the original training from scratch takes seconds on small datasets. Generally, the more iterations you use for pretraining, the better downstream quality you get.
CONCLUSION
In this work, we have systematically evaluated typical pretraining objectives for tabular deep learn- ing. We have revealed several important recipes for optimal pretraining performance that can be universally beneficial across various problems and models. Our findings confirm that pretraining can significantly improve the performance of tabular deep models and provide additional evidence that tabular DL. can become strong alternative GBDT.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, 2019.
Sercan O. Arik and Tomas Pfister. Tabnet: 1908.07442v5, 2020.
Attentive interpretable tabular learning. arXiv,
Sarkhan Badirli, Xuanqing Liu. Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S.
Keerthi. Gradient boosting neural networks: Grownet. arXiv, 2002.07971vv, 2020.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. International Conference on Learning Representations, 2022.
URL
net ftr///orumt//// si=iiid_eoae qYkmKb3
Baldi, Sadowski, and Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.
Jock A. Blackard and Denis Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3)131-151, 2000.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In SIGKDD, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Sajad Darabi, Shayan Fazeli, Ali Pazoki, Sriram Sankararaman, and Majid Sarrafzadeh. Contrastive mixup: Self-and semi-supervised learning for tabular domain. arXiv preprint arXiv:2108. 12296, 2021.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 1810.04805v2. 2019.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint 1rX1v:2112.10740. 2021.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11: 625-660, mar 2010. ISSN 1532-4435.
David Freedman and Persi Diaconis. On the histogram as a density estimator:1 theory. z. Wahhsscheinlichkketttthero verw Gebiete, 57(4):453-476 December 1981.
Yura Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning, arXiv preprint arXiv:2203.05556. 2022.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurlPS, 2021.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiabilityy meets conditional computation. ICML, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2/1 11.03777. 2021.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv, 2012.06678v1, 2020.
10

---

Under review as a conference paper at ICLR 2023
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In NeurIPS, 2021.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. Advances neural information processing systems, 30:3146-3155 2017.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-297, 1997.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673 2020.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NIPS, 2017.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision-tree hybrid. In KDD, 1996.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. In NeurlPS, 2021.
Kundan Krishna, Saurabh Garg, Jeffrey P Bigham, and Zachary C Lipton. Downstream datasets make surprisingly good pretraining corpora. arXiv preprint arXiv:2209. 14389, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Renata C. B. Madeo, Clodoaldo Ap. M. Lima, and Sarajane Marques Peres. Gesture unit segmen- tation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, SAC, 2013.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark John Francis Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Mariya Shmatova, Panos Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. ArXiv, abs/2107.07455 2021.
Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 35669,566,,, IEEE, 2021.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning tabular data. In ICLR, 2020.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurlPS, 2018.
Tao Qin and Tie-Yea Yan Liu. Introducing LETOR 4.0 datasets. arXiv, 1306.2597v1. 2013.
Ravid Shwartz-Ziv and Amitai Anmon. Tabular data: Deep learning is not all you need. arXiv, 1,06.03253v1, 2021.
Kamaljot Singh, Ranject Kaur Sandhu, and Dinesh Kumar. Comment volume prediction using neural networks and decision trees. In IEEE UKSim-AMSS 17th International Conference on Computer Modelling and Simulation, UKSim, 2015.

---

Under review as a conference paper at ICLR 2023
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: improved neural networks for tabular data via row attention and contrastive pre- tetiiiiinng... arXiv, 2106.01342v1, 2021.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, 2019.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances Neural Information Processing Systems, 34, 2021.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo. Openml: networked science in machine learning. arXiv, 1407.7722v1, 2014.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In ADKDD, 2017.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self- and semi-supervised learning to tabular domain. In NeurlPS, 2020.
A DATASETS
We used the following datasets:
Gesture Phase Prediction (Madeo et al. (2013)
Churn Modeling
California Housing (real estate data, Kelley Pace & Barry (997))
House 16H
Adult (income estimation, Kohavi (196))
Otto Group Product Classification
Higgs (simulated physical particles, Baldi et al. (2014); we use the version with 98K sam-
ples available at the OpenML repository Vanschoren et al. (2014)
Facebook Comments (Singh al. (2015))
Covertype (forest characteristics, Blackard & Dean. 2000))
Microsoft (search queries, Qin & Liu 2013).. We follow the pointwise approach to
learning-to-rann and treat this ranking problem as regression problem.
Weather (temperature, Malinin al. 2021 We take 10% of the dataset for experi- ments due to the its large size.
B HYPERPARAMETERS
B. CATBOOST
We fix and do not tune the following hyperparameters:
sarly-stopping-rounaa = 50
od-pval = 0.001
iterations ions 2000
For tuning on the MI and CO datasets, we set the task type parameter to "GPU". In all other cases (including the evaluation these two datasets), set this parameter to "CPU".
12

---

Under review as a conference paper at ICLR 2023
Parameter
Distribution
Max depth
UniformInt [1, 10]
Learning rate
LogUniform (0.001,
Bagging temperature
Uniform [0, 1]
L2 leaf reg
LogUniform [1, 10]
Leaf estimation iterations
Uniformint [1, 10]
Iterations
100
Table 8: CatBoost hyperparameter space
B.2 XGBoOST
We fix and do not tune the following hyperparameters:
booster "gbtree"
early-stooping-rouneee = 50
n n-estimators 2000
Parameter
Distribution
Max depth
UniformInt [3, 10]
Min child weight
LogUniformi (0.0001, 100]
Subsample
Uniform (0.5, 1]
Learning rate
LogUniform(0.o00 (0.001, 1]
Col sample by tree
Uniform (0.5, 1]
Gamma
{0, Log 0 00000, 100]} {0, LogUniform(0.. 10)
Lambda
# Iterations
100
Table 9: XGBoost hyperparameter space.
B.3 MLP
We fix and do not tune the following hyperparameters:
Layer size 512
Head hidden size = 512
Parameter
Distribution
# Layers
UniformInt(11 8]
Dropout
(0, Uniform(0, 0.5]
Learning rate
Log MogUniformmmeec 0.005]
Weight decay
{0, LogUniformmIee4 1e-3))
Corrupt Probability
Uniform 0.8]
# Iterations
100
Table 10: MLP hyperparameter space.
13

---

Under review as a conference paper at ICLR 2023
B.4 EMBEDDING HYPERPARAMETERS
We fix and do not tune the following hyperparameters:
Layer size 512
Head hidden 8 i ze = 512
The distribution for the output dimensions of linear layers is Uniformint([, 128].
PLR, T-LR. We share the same hyperparameter space for models with embeddings across all datasets.
For the target-aware embeddings (tree-based) T-LR, the distribution for the number of leaves is UniformInt(2, 256). the distribution for the minimum number of items per leaf is Uniformint(1, 128] and the distribution for the minimum information gain required for making split is LogUniform| [le-9, 0.01].
For the periodic embeddings PLR. The distribution for & is Uniformint(1, 128), the distribution for the a parameter is LogUniform|| 100]
C SYNTHETIC DATASET DETAILS
To generate the corresponding objects targets {3))= for randomly sampled objects features {zr)i=1 and importance vector E Rn we build an ensemble of 10 random oblivious decision trees {T)())) 10 of depth 10, where on each tree level we sample feature ~ Cat(p) and threshold & ~~ Unif(min(x' max(²")) for decision rule. For each tree leaf, we sample a scalar 1 (0, representing a logit in binary classification. We define the binary targets follows:
((x) T
We set the size of the dataset 50.000 and the number of features m 8 and generate 50 datasets with different feature importance vectors for the analysis.
10
(xxx) })-
D PRETRAINING THE FT- TR-NSSSSFRR MODEL
Here we evaluate the recommended pretraining objectives for pretraining the FT- Tr-ansfformer model from Gorishniy et al. (202T). We evaluate pretraining for the FT-Trrasformer on subset of datasets for the reconstruction, mask prediction, and their supervised variations ttarettarget and rec+sup). The results are the Table
GE CH CA HO J OT
HI AD f Avg. Rank
MLP-PLR
no pretraining
0.668 0.858
0.469
3.008 0.483 0.809
0.926
2.7 + 1.2
mask
0.685
0.863
0.434
3.007 0.477
0.818
0.927
2.1 + 1.1
0.667
0.852
0.439
3.031 0.472
0.808
0.926
2.1 + 1.1
+ supervised
0.709
0.858
0.433
3.059
0.465
0.807
0.927
1.7 1 1.2
mask target
0.694 0.862 0.425 3.023 0.474 0.821
0.929
1.9 1.1
FT-Transformer
pretraining
0.631 0.860 0.470 3.240
0.479 0.809
0.912
3.7 + 1.5
mask
0.696 0.864 0.429 3.113
0.466 0.845 0.916
1.4
rec
0.687 0.860 0.438 3.120
0.478
0.840
0.917
2.6
+ 1.3
rec + supervised
0.718 0.862 0.430
3.078
0. 0.463 0.825
0.917
1.9
+ 1.1
mask target
0.694 0.861 0.436 3.083
0.467 0.842 0.918
2.0
+ 0.9
Table 11: Results for pretraining the FT-Transformer model, along with the results for pretrain- ing the MLP-PLR, Bold accounts for the best pest-performing model between both MLP-PLR and FT- Transformer.
We can see that pretraining provides performance improvements for the FT-Transformer model comparable those for MI.P-PIR, with MI.P-P.R remaining state-of-the-art with without pre-
14

---

Under review as a conference paper at ICLR 2023
training. Including transformers does not reveal new patterns behavior, but provides additional evidence that pretraining does help neural networks on tabular data.
E COMPARISON TO THE LEARNING RATE SCHEDULERS
Since in our setup pretraining is technique which helps in the optimization process, we compare to various commonly used learning rate schedules which are also used as an aid in the optimiza- tion. The results for various learning rate schedulers along with two self-training based pretraining baselines are reported Table 12
GE f CH
CA
HO L
OT
1
AD f
WE L
CO
MI L
Avg.
Rank
pretraining
0.635 0.849 0.506
3.156
0.479
0.801
5.737
0.908
1.909
0.963
0.749
2.7 + 0.6
cyelic
0,616 0,850
0.498
3.148
0.479
0.801
5.835
0.908
1.920
0.962
0.749
3.2
warmup
0,655
0.854 0.491
3.123
0.478
0.799
5.662
0.908
1.912
0.964
0.749
2.5 2.5 1.1
warmup + lin
0.669 0,851 0.489
3.123
0.480
0.801
5.669
0.908
1.912
0.968
0.748
2.2
warmup + cos
0.639 0.851 0.494
3.107
0.478
0.801
5.823
0.908
1.893
0.962
0.748
2.6 0.6
mask rec
0.691 0.857 0.454
3.113
0.472
0.814
5.681
0.912
1.883
0.964
0.748
1.5 0.7
0.662 0.853 0.445 3.044
0.466
0.805
5.641 0.910
1.875
0.965 0.746
1.5 1 0.5
Table 12: Comparison of self- self-prediction based pretrains to learning rate schedulers. All the results are for the MLP model
We can see that the learning rate schedules provide some improvement and help the optimization process, however, the effect is almost negligible compared to the one from pretraining. We conjec- ture that exhaustive hyperparameter tuning diminishes the benefits of using the advanced learning rate schedules.
F DATASETS WITH MORE CATEGORICAL FEATURES
Additionally, we experiment with datasets that contain more categorical features. We add 4 datasets, the information about the datasets is presented in Table 13
Abbr
Name
Train Validation Test Num # Cat Task type
Batch size
DI
Diamond
34521
8631
10788
Regression
128
BL
Black Friday
106764
26692
33365
5
Regression
128
BR
Brazilian houses
6842
1710
2136
8
Regression
128
BA
Bank
28934
7234
9043
Binclass
128
Table 13: Datasets with categorical features
The results for the MLP and MLP-PLR with self- seff-prediction-bee pretrains along with their super- vised variations presented in Table 14
We can see that results on the datasets with more categorical features align with the results in the main text: pretraining strategies based self-prediction improve upon the supervised baselines and target-aware objectives help further increase the downstream performance.
G SHARE OR SPLIT LEARNING RATE AND WEIGHT DECAY BETWEEN PRETRAINING AND FINETUNING?
Here we demonstrate that tuning and using the same learning rate and weight decay for both pre- training and finetuning results in similar performance to tuning these parameters separately for the two stages. We opt for sharing the learning rate and weight decay for pretraining and finetuning in all the experiments in the paper.
15

---

Under review as a conference paper at ICLR 2023
DI
BI.
BR 4
BA f
CatBoost XGBoost
0.134
0.682
0.407
0.932
0.136
0.680
0.407
0.930
MLP
no pretraining
0,140 0.697
0.406
0.927
rec
0.140
0.695 0.400 0.926
mask
0.138
0.691 0.385 0.929
rec supervised
0.135
0.689
0.394 0.930
mask + target
0.135
0.691
0.376 0.933
MLP-PLR
pretraining
0.134 0.689
0.399
0.929
rec
0.133 0.684
0.390
0.931
mask
0.131
0.685
0.392 0.931
rec supervised
0.133
0.681
0.393
0.928
mask target
0.132 0.685 0.385 0.933
Table 14: The results for single models with various pretraining objectives on datasets with categor- ical features.
GE
CH
CA 4
HO
OT
HI f
FB +
AD
WE 4
CO f
MI 1
shared wd/ shared Ir 0.683+1e2
0.857+2e3
0.434+7e3 3.056=4e2 0.468-2e3
0.819+2e3
2.
0.914+1e-3
1.876#5e3
0.965=7e-4
0. D.748 1e-4
shared wd/ / split Ir split wd/ / split Ir split wd/ / split Ir
0.697+9e3
03857+3e-
0.4
3.032=3e2 0.469=2e4
0.819+2e3
5.647+4e2
0.915+9e-4
1.934+8e3 0.964-9e-4
0. 0.748=4e de
0.6881963
0.85613e3 0.43014e3 3.04624e2 0.471
27
5.73415-2
0.914
1.891
0.964_1e3
0.748=3e-4
0.69421e22
0.8581263
0.431 1603 3.06623e22 0.468_2-33
0.82112e5
5.63214<2 0.9141163
1.87814es
0.966_1es
0.748 36-4
Table 15: Results for single models with MLP mask + target pretraining
H EARLY-STOPPING CRITERIONS
Here we demonstrate that early stopping the pretraining by the value of the pretraining objective on the hold-out validation set comparable the early stopping by the downstream metric on the hold-out validation set after finetuning. See Table 16 for the results. This is an important practical observation, as computing pretraining objective much faster than the full finetuning of the model, especially on large scale datasets.
GE CH t CA 4 HO OT 1 HI FB J AD 1
finetune early stop 0.683 0.857 0.434 3.056 0.468 0.819 5.633 0.914 pretrain early stop 0.674 0.855 0.434 3.031 0.469 0.818 5.738 0.914
Table 16: Results for single models with MLP mask + target pretraining
I APPLYING AUGMENTATIONS TO GBDTs
We adopt the augmentations used in pretraining to GBDT's by augmenting the datasets before GBDT tree construction, effectively enlarging datasets with random column resampling to match the num- ber of pertrubations the neural network is exposed to during pretraining (tens of millions of randomly corrupted samples). We report results in Table 17 We observe that adding noisy augmented samples to the datasets deteriorates GBDT performance significantly.
J SUMMARY OF PRETRAINING METHODS
For summary of the pretraining methods mentioned in the paper see Table 18
16

---

Under review as a conference paper at ICLR 2023
GE t CH 1 CA & HO AD HIT
CatBoost
0.682 0.864 0.433 3.115 0.927 0.806
CatBoost (aug) 0.660 0.838 0.474 3.561 0.923 0.791
Table 17: Comparison of CatBoost and CatBoost on data augmented with random column resam- pling-
Abbr.
Description
Uses target variable
contrastive
Contrastive loss with resampling as augmentation Prediction of masks for the resampled columns
mask
rec
Reconstruction original values of resampled columns Mask prediction combined with supervised loss
mask + sup
mask + target
Mask prediction with modified column resampling and loss Reconstruction combined with supervised loss
+ sup
rec + target
Reconstruction with modified column resampling and loss
Table 18: Summary of the objectives studied in this work
K EXTENDED TABLES WITH EXPERIMENTAL RESULTS
The scores with standard deviations for single models and ensembles are provided in 19 and 20 respectively.
17

---

Under review as a conference paper at ICLR 2023
MI J
0.743 0e-4
0.742*1.5e- 1.5e4
0.749
6
0.748
1e
0.746
Se- 3e-
0.960=1.2xm 1.2033 0.748
Ged
0.748 nod
0.747 2.44
0.748 Se-4
0.747 9-e.
0.748 2.4.5.4
1111e 0.748
0.745 1. 13-9
0.746
1. 9.-9
0.746+3.55 Te
8.1e- 0.748 9e
0.808 0.808-1.244 1.2-3 5.571#122d1 0.926 6.4e-4 1.877 0.971= 1.8e- 0.745 1:33::3666 Ge-d
1.5ed 0.748
1e-4 0ed
0.745 4e-
0.745
0. 0.749 122
0.748
Se-1
0.748
0.745 Se
-.00-11 0.74915
2nd1 2e-1
0.748 Godd
0.969-7.3044 0.74624.8
0.971-5.8e 0.747+331e
0.748 Se
0.748
0.749
4.9e-4 1.74945114
0.749 1.4ea
0.746
de-d
0.7.61.3
1.837 1e-1 0.966 3.2e-
1e.4 0.969=6.1e- 100-001144
0.963 0.963-8.7e4
8.7e4
9.3e-1
0.965_5-1e-
5-de-d
1.883 663 2.268
4.00-4
-6,0-4
1.1e-1
0.969 -.00
0.969_6.0ed
0.971
4.404
5.86e
3.9-4
339e4
1e 0.970=144
16-3 0.969 4.604
0.971-3.9ed 3.9644
0.970-5.0e-
5.0e-4
0.969-4.064
4.00-4
6.5e-1
7.5e-d
CO
1. 1.802 0e-3 0.9 0.969
+2 9--3 0.964
0.965
0.966-
0.967
0.967
0.965
0.964 0.964
0.965.
1.87713.3e 0.966-
0.970-
0.971=
0.971
0.969
1.906 Se-8 0.970
0.971
0.968
0.971
0.970
0.969-
0.969
0.969
0.968
1. 1.909+44.0 64.
187513-4e3
14. 50-3
1.873
+8.3+5.1ea
8c-3
1.876 14-
12-3-8
1.800+5.0e
1.87322 063
1.875 2-3
16-8
2--3
1.887
14.5-3
50-3
1.803 42.
+2 2.
1.924
66-8
7e-3
13.55..
5e-3
WE 4
1.883
1.875
1.881
1.873
1.862
1.887
1.884
1.911
1.865
1,911
1.876
1.897
1.916
1.861
1,869
1.862
1.874
1.862
1.86713.
13.
1.
AD f
0.927 0.927-3.1e 3.1ed
0.927
9.5e 5.737 1e 0.908
3.0ca
1e2 0.912 8.0e-1
5.641 2e-2 0.910
1.26-
0.46912.6e- 0.813-1363 5.007 0n-2 0.910
0.814 3.68912.1e4 1--2 0.914
0.915-6.ed
0.911=1.4e-
5.633 0.914-11164
0.912-67e-4
9.0e-4
0.910-1.1e
0.926 6.11e
4e 0.927- 8.1e44
5.52114.66 6e-2 0.921_1.563
0.925
1.003
0.927=5.86-
4e-2 0.929
3.3e-4
0.92855662
0.812-1.1e3 5.507 0.927
-39644
0.812=5.5e4 5.502+9 se 2 0.927=3.4e
6.88-1
0.926_7.1m4
0. 0.808-1.064 5.36326..e 6o-2 0.921
2 0.927
6e 0.926
8.7e-4
3.7e-4
0.929-a.5e-
8.5e-4
0.928-
0.927-5.6-1
5-0ee1
5.39113.000 0 0.927 0.927=4.1e4
4.1e-
0.915
1-22 0.909
6-22 0.925
0.927
0.929
0.927
5.324 00-2
5.383
8e-2
5.681
5.680 40.2
5.685
5.672
de
5.607
5.64011.9w4
9--2
0. 0.809 2333 5.608
5.586+2
5.480 60-2
5.571
5.537
5,440s8 1e-2
5.520
5.54413 50-2
5.525+2.3.2
5.510+3
0.807-20 22003 5.516+2 20-2
1.2ea 5.64444.602
1.6ea 5. 5.492+
20-2
1.003 5. 5.493 4o-2
FB
1.5--3 5. 5.6361s
0.806_. 16660016e..... 1.533 5.391
5.
3.115 Se-2 0. 0.457:1.360 36-3 0. 0,806_. 0.806_3.4ee 3.3e-d
0 0,804= 0.804_1.5ea
0.814-1.763
0.505 11.33
0.806-7.3e4
0.817
0.815- 0.815
0.819 1.6
1.765
1.33
0 1.818- 01818-8.4e4 s.ddd
0.811=100r
0.813_8224
0.807=622--
0.821=1.1e
0.805 0036-1
0.818-1.86
0.807 0.807-6.264 6.204
0.814- 0.814-8.2e4
8.244
0.820-1.240
0.806 0.806_1.0e3
0.819
HI
0.801-
0.806_
0.806_1
0.821=1
0. 0.820
0.
0.
MLP-PLR
0.
MLP-T-LR
0. 0.436=22.e.3 3. 1,1060=c.. Be- 0. 06,0-
MLP
0.479*1.4e
3
3.113 1.3e2 0.472:3.0e3
3.014 3.3 0.466 1.113
0.69321.1e2 0.856_1.66 1.8 25.413 3.077= 22.002 0.45912.1e3
3. 3.108-2.66. 2.0 0. 0.480+1.9c
1993
0.758+
4.0e22 0.456+1.9e3
3.056-4.0ess 4.0e22 0.468 1.003
1146821.902
1.-3
0.463 1.66-3
0.853_3.444 3. te-3 0 14551166-- 3.046 2. 4-22 0.4633 0.46312.0e3 2.0033
2.36 0. 0..88+1.662
0.477+25e
7.2-3 1.852-7223 0.39: 3.031 3,031-386 8e2 0.472+3.0-3
3.136 le2 0.47911.963
3.129 3.129-4.1es le2 0. 0.48011.5 18-3 86-3
3.059 3.602 0.465122263
3.023 3.2 0.4 0.474=2003
0.47922.0-3
0. 0.475+199c
0.471+2.1e3
1 1.5222 0. 0.48211.763
0.47212563
0.472=2.ma3
0.471+1.2+4
2299-2266 4e 0. 0.468+9..c
2.23
0.474#1.9es
1.933
0.476
0.473 1.333
0.4701222-3
OT J
3.019 3.019-3.66d 2. 0.4688
0. 0.462+2.00
3 3. 3.156-2.ee 2
3.056 2.222 202
0.436+6696-- 3.099 2.099-24455
4.-3 3.012 3.012-1.022
1004-1.800 1.0e2
3.008 3.008-2.06
3.007 -007-1662 1.02
2.971=1 1.lee
0.688 0. 0.860 0. 0.445x2.7.3 6.064-3.42
3.008-2.7e.
0.42422463 3.015_3.722
0.4212310a 3.109-a.7ed
0.682+5.1c4 0.860-4.1 4.1ea 0.430 3 3.
(.058-4.900
4.302
5e-2
2.0
HO 4
0.43427-204 3.056
0. D.659L L. 6--3 0. 0.853 3.20-3 0.45416.76-3 3.044
3.113
7.8ea 0.429+1.cc 3 3. 1.19922
3.112
0.421 3.058
3.050
3.113
3.104_
CA
0.68314.7e- 0.864 0.86118.1e4 8.14- 0.433 0.433
1.6ea 0. 0.506+8.6cc
2.53 0.454
2.2 0.445 166211-0-22 0. 0.853
0.672214422 4. 1.855_20004 0033 0.45514.563
0.454+4.26-
-.44-..66 1.00a 0.4366+444c.
9..80 O 0.857 1857-3.1e4 331eed 0 4.5
0.668 0.858-4 a O. 0.460+5.22
0. D.685+5.6e-3 O. 0.863- 0.863-1883 1.803 0.434+4.3.3
0.710111663 0 1.859-11es3 43-33366-3
0.71147.1e3 0. 0.866_2.00 00- 0.4412-196-
0. 0.858=1.9m 1.0 0.433
0.69429.1e3 0. 0.862=1.76 763 0. 1.25244263
0. 0.421+1.5e3
target sampling O, 0.687*7.96- 3 8.80 0. 3.
0.44411.8. 18-3
0.437+2.0e-
3-2- 0.417
0.44019.5r-3
1333-3
O. 0.43213
0.693+8.26-3 0.857-2 2.00 0.
0.63416.90-3 0. 0.866_ 1.30-3 0.
C
0.6782.4.8c-3 0 0.858
0. 0.635+1.3c- 2 O. 0.849-1
0.691#1002 1.022 0.857=2
0.853_2.222
0.666+1.462 0.850-2.24
O. 0.854
0.683 -0-2 0.857
2.14
0.8 0.853
0. 0.864-2.8s5
0. 0.868_: 1.03
0.857=4.40 4.461
0.676+7.763 0. 0.858-
O. 0.865 1.865-11e3
0.64927.86-- 0.865-1.
0.861-33
0.864
1.763
2.022 0. 0.859
2.733
0.684+7.76-
164115.666
0.709
0.69010.66t
0.6542644-3
0.6522744-3
0.678+0.7e-
0.6499s..3c-
0.668
target sampling 0.667.1 1867.1.0022
GE 1
target sampling
target sampling
target sampling
target sampling
no pretraining
mask target
pretraining
mask target
no pretraining
mask target
contrastive
mask + sup
ree + target
mask sup
rec target
mask + sup
rev + target
CalBoust
XGBoost
rec + sup
rec sup
rec + sup
supcon
mask
mask
mask
rec
sup
rec
sup
rec
sup
Table 19: Extended results for si models
18

---

Under review as a conference paper at ICLR 2023
MI 4
0.741 2..41=1.4e0 4e-4
0.742 3.33
0.968+2.5e 0.747=8 B. B
0.746=7. 7.7
0.97212.0e. 0.744 7--8
0.96412.1e-4 0.746
0.746 1e-4
0.745=8.3e5
0.747
-1.30-
0e-6 0.745=
1745=1.1ed
0.746
1.
0.746
0.743 113.1.0 1222-
0.744 7.66-
0.744 1.2e
0,974+23c 0.745-
8.
0.743=2.1e4
0.746
1e-4
0.974 0.744 0.05
0.744
0.745
2.54 Se-4
0.744= De0-
0.745 2.744
0.744=6.36
0.97211 1.-4 0.747
1 7.66-5
0.745
0.7.4=
0.746-1.44. 40-5
0.747=8.2ed 8.22.
0.746
3. Be-4
0.746
0.747= D..044
0.744
1e-4
0.744 1.744_3.0e0 13
C
0.967 1133--4
0.960
Sc-5
0.907+11e4
0.97323.60-4
0.970+2.1e4
0.973+3.955
0.973+2.06
0.96915.22 1.22
0,968
8c-44
0.97216.14
0.97212.7e-4
0,974+222
0.975+2.9cr 2.--
0.975
66-4
6e-4
0.97412.2e-
1.973
0.97
0,974+3.46 4e.
0.974+3
6e
0.97318.8e5
0,97324.065
1.849- 4e 0.975+288c4
0,975+1 4e-4
0.974+1.1ed
0.972+2.5-4
0.973+1.1c4
0.97211.1-4
0.97215.5es
1.801 20-3
1.850-1 De
1 1.0e3
1e-4
8033
4.7
4c44
0:01=4400 4.0e3
1.809 3e-4
Desd
1.825. 2. 2.5e-
1.82216.6 6e-4
3e3
3e3
3e-4
1.848=1 1.7
1883
6e-3
1.860= 3.0
6e 1.811=1.56 Se3
1.850 D. De-e-
1.836_2.7e6 2.33
1.803 3e3t
1.824-3.88 De
1.849 4.203
1.840 2ed
46-3
1.0033
1.82824.7AM
1.849-
WE
1.782=4
1.828
1.811
1.804
1.827
1.810-2
1.837
1,835-1
1.850
1.818
1.834
1.813
1.833
0.927 1.811=1.111
1.830
1.809
1.
1.801
AD
0.928 1.34
0.927
0.91042.7e4
0.913
0.91217555
0.912 1.34
0.91627.0ea
0.912+1.264
0.916+2.064
0.913+5.2e4
0.91614.66-
0.91412.6e4
0.91012
0.913124e-1
0.928+7.765
0,928+100
1.0-4
5.124 2e 0.928 2e-4
3.407 8e- 0.9261-13ed
0.92712.1e4
0.92911.7e4
0.930 1.0c4
0.980
0.928 1.004
0, 0.928
0 0.92712.8e
0.927
0.9251122.4
0.928+114e4
0.927+5.1e4
0.930
1. 0. 0.929
0.928
1.4.-4
0.928
119281
0.
0.930
0.45013.6e4 0.807 5.226 22-2
5.338
1.8222
0.467 +2 3 0. 0.805+2.9e 9 5, 5.606+2.8e3
5.578
8-3
5.571
0.82014.5 5.63421442
2.6022
1. 0. 0.809+1.14 1e 5.579+1.46
5.601+4.7e3
5.531
+3.0e3
3.55117204
5.601 62.0222002
5.555.5.885
15-8
5 11:e7
5.324+1262
5.444
5.36611.1e2
5.124
5.373
8-2
5.175 5e.2
5.096
1.02
5.173+2.7e2
5.424
2--2
5.370
5.039
0.809 4e-4 5.442+1.622
5.407+3.364
5. 5.422
5.458
5.384
5.344
5.083
14.7es
FB
5. 5.555
5.
4.4 5.
5.
5.
5.
H
0.454 0. 0.805xs.366 30.4
0.451+1.66 8e-8 0. 0.824 5e-4
1--2-- 1e-3 0.81212.5e2
0.81621.400
0.819+1.ed
0.817
de-4
0.457 De-4 0 0.828
0.456+3.8e.. S.-44 0. 0.822
1.13
0.816.15.664
0.81517.00-4
0.814+7.964
0.827+11e4
0.454 3 3 0.813
0.465 1.333 0.81611-464
0.817
46d
0.81125.364
0.458 2--4 0. 0.828
0.46649 de-4 0. 0.828 5e-4
0.816*1774
0.815+3.3e
0.808
0 0.826 36-3
0.81222.704
0.818+2.8cd 8..4
0.814+6.84
8c14
0.827
4--4
0. 0.826x
2044
0.812
led
0.81115.1e4
MLP-PLR
MLP-T-LR
MLP
OT
D.453
a 16-1 1e44
0.443 0e-8
0.465+1..0
0.443
1.5e3
2.951+1.50 91. 0.442
Ue-4
0.448
4e-1
0.451
11.22-3
2.953+7.4e 0.470+7.5e
0.457 de-4
0.468
8--4
0.449
3e-4
0.465
0.450
0.472
14.77-
0.150
0.15023.4044
0.463
+3.5e.
3.088 +..84..114 1e 0.460+4.6c
3.057 +1.002 0.444+1.46
4e.3
0.460
4c-8
0.462++444
3.045 1e-2 0.45612
4e-8
111...... 4.-2 0.454
8
0.451
43333 2. 2.933+2.00 de 0.
HO. J
0.430 3.093 1e33
3. 3,152+1.224 22.3
3,055+8..d 3
0.43748 3.026
3e3
2.967 16. 96-3
2.95214.403
3.022 9022
3.014+6.0c
de
3. 3,066+1.
1.
2.9 2.949 9.-2
2. 2.955+2.1e.
2. 2.926 12.5 3e-2
100011.111 10-2
2.92
2.95448.1ed
3.05411.5e3
3e-2
3.055
2. 2.96442.80
303
0 0.407 2--4 2. 2.952 5-3
2.8 2.877 2
2. 2.983+1.20 22
2-12 3.028 11 80-2
2. 2.93041.2. 202
3.035 1..521.902 99-2
0.424+11ca 3.1077 +8.80 8
2. 2.894+1.8
88-2
2. 2.938 2.938 +1.81..82 Me-2
3. 3.029
2--1 3.003
2.951
2.
CA
0.434+7.004
0.482+2.8
3
1.424
0.4.444
0m-3
0.71712.203 0.85727.4c4 0.424 404
0.434+3.0e
0.418
22-3
8e3 0.419+21e-3
0.414:111e3
- larget sampling 0. 7776+.4.9.22 3 0.860+1.1c4 1e-3 0.110#1.5e-3
0.433
0.43518
0.454+1
0.42++17e3
5.098 5. 1418+112-
0.421 3e.0
0.424 0ed
0.424
0.403+1
0.431
0.437
0.41311
0. 13.04 0.86121 0.41421.500
1ee 0.421+8.0
0.425+5.1e4
0.410+7.864
1e- 0.397+a.7ed
0.432 1e-4
0.423
19.
CH
0. 0.69221. 8.-83 8o-3 0.86416.88 Me-s
0. 0.683 1:303 0. 0.8602-4.3e.
0. 0.05645..95 3 0,852 4
0. 0.722 6e. 0.859 de-1
0.67912.003 0.85612.8--
0.70814.4x3 0.85718.8.d
0. 0.686+55263 0.85148.4e4 464
0.716+5.7e: 0.859+1.1e3
0.859+1.8e3
0. 4.3-3 0.860
66-3
0.677 14.66- 0.857 9--4
- target sampling 0. 0.669115 13.7033 1.85811-46 -1-3
0. 7..95 0.86447.0c
0. 0.725+4.e. 0.865+7.004
0.733 26-3 0.86719 de-
0. 0.73212.0e3 D.- 0 0.86913.54
504
0.737 0.86221
0.71913553 5e. 0. 0.66644364
3e4
0.86711.0e3
O. 0.862+2 0e4
0. 0.86049.3e4 3e
0. 0.662 17.663 0.86818
15 0--4
0.67914.763 0.868123.4
0. 0.698+3.763 0.865+7.0e4
0. 0.698+4.0ea a O. 0.866+1.1e4
0. 0.705 de.3 0.866+4446
0 0.86814.66- 4444
0.866+1.1e-
0.693 5e-3 0 0.86613.1 1-4
target sumpling 0. 0.700 12--3 0.86311.
3--3
0. 43.09
0.724
0.705+2.3e3
0.712+2333
0.673+1.0--
target sampling 0.67 0.677+5.12
GE
0.
target sampling
target sampling
no pretraining
mask larget
no pretraining
mask+ target
no pretraining
mask target
contrastive
mask sup
rec + target
mask sup
rec + target
mask sup
rec + target
CatBoost
XGBoost
rec + sup
rec + sup
rec + sup
supeon
mask
mask
mask
rec
sup
rec
sup
rec
sup
Table 20: Extended results for ensemble models
19


Reviews:
[{"review_id": "T0DvYavT-kE", "reviewer": "Reviewer_QuBL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the pretraining practice on tabular data. It shows that using the object target labels during the pretraining stage is beneficial for the downstream performance. Several target-aware pretraining objectives are proposed. Experiments on various datasets validate the claim.\n", "summary_of_the_review": "The paper is clear and well-written. Several pretraining strategies are investigated in the paper. Experiments support the claim of the paper. There are some suggestions, and please see the weakness part.\n", "strength_and_weaknesses": "The paper compares various supervised pretraining objectives on tabular data. Some helpful conclusions are drawn from the experimental results. \n\nHere are some suggestions:\n1. The main assumption of the paper is that \"Pretraining is always performed directly on the target dataset and does not exploit additional data\", which is not always the case in the deep learning field. It seems the authors need to point out revisiting the \"supervised\" pretraining. The reviewer has two questions:\n    1.1 There are some self-supervised learning strategies on tabular data, such as SubTab. Will they help get better performance?\n    1.2 Could we pre-train on another dataset and then apply the pre-trained model to the target dataset? \n2. The authors need to make the formulation of the pretraining and fine-tuning stages clear. Which learnable parameters are shared between the two stages? Some readers may not be familiar with the tabular model.\n3. The pretraining objectives introduce a lot of augmentation strategies. Will they also help GBDT methods?\n4. If we use different deep models like ResNet or FT-Transformer, will we get similar results?\n5. The authors validate that pretraining is a good trick to boost the ability of deep tabular models. Does it mean that pretraining becomes a basic step when we use deep models on tabular data? Then how to compare various deep models in a fair manner? Different methods may need different pretraining strategies.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jnhlkf15qk", "reviewer": "Reviewer_S291", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Deep learning has become an important method for tabular data. Even though pretraining based DL has been demonstrated successful in computer vision and NLP problems, it is challenging to make reliable conclusion about pretraining efficacy in tabular DL. This work provides a fully labeled tabular datasets to understand if pretraining helps tabular DL in a fully supervised setting and compare pretraining methods to the strong supervised baselines. \n\nUsing the target dataset, the method pretrain the model parameters with pretraining objectives, including generative positive pairs in contrastive-like objective or to corrupt the input for reconstruction in a self-prediction based objectives. In the second stage, the model is finetuned on the downstream classification or regression task. ", "summary_of_the_review": "The paper is insufficient for a publication in ICLR, as explained in the weakness section. \n\nPlease provide more detailed explanation on the experimental setup, particularly:\n- The pretraining dataset, dataset size.\n- Downstream tasks. tasks losses, labeled dataset size, etc. \n\nPlease also formulate the problem properly with the pretraining losses and downstream task prediction losses. \n\nIf the downstream tasks have different prediction losses, how should we pick the pretraining recipe?", "strength_and_weaknesses": "Strengths:\n- The paper generates some useful insights around how to pretrain a MLP model for tabular data and finetune the model to achieve stronger results on downstream tasks. There are also interesting discussions round pretraining objective. Self reconstruction and mask prediction are useful pretraining objectives, as compared to contrastive loss. \n\n- Combing supervised loss and unsupervised loss in the pretraining helps further increase the downstream performance. \n\nWeaknesses:\n- The paper is not clear about its experimental setup: what is the pretraining dataset and what is the downstream datasets? Are they the same but with different losses? \n\n- As the paper pointed out that adding supervised loss and unsupervised loss during pretraining enhances downstream task performance. But the paper does not provide any evidence if adding target task prediction loss during pretraining would benefit generalization? Whether adding the prediction of some tasks would hurt the performance on some other tasks with different loss functions?\n\n- If there is not universal solution between self-prediction objective, when applied in real, how should we pick the optimal objective, given some mixture of downstream tasks?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IaWw8JqRWm", "reviewer": "Reviewer_MXDS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore different pretraining objectives for tabular datasets, apply them to vanilla and embedding-based MLP architectures, evaluate their relative downstream predictive performance, and compare their performance with gradient-boosted decision trees (GBDTs, namely CatBoost and XGBoost). Specifically, the authors explore unsupervised pretraining objectives, target-aware pretraining objectives, and combinations of both.\n\nExperiments on 11 datasets (6 classification and 5 regression tasks) suggest several findings: 1) deep learning (DL) models generally benefit from pretraining in terms of downstream performance (this includes both simple MLP and transformer-based architectures, although there was no improvement of transformer-based models over simple MLP architectures); 2) target-aware pretraining objectives performed better than unsupervised objectives; 3) combining target-aware and unsupervised pretraining objectives tends to work best, and perform similarly or better than GBDTs on downstream tasks.", "summary_of_the_review": "The paper builds on work by Gorishniy et al. (2021) and expands the evaluation of pretraining objectives for tabular data. However, the paper lacks a more thorough comparison to GBDTs and AutoML methods, and the clarity of the paper could be improved.", "strength_and_weaknesses": "Strengths\n---\nPretraining is an important part of the significant success of DL models on downstream vision and NLP tasks, and thus exploring various approaches of pretraining for tabular data is an important and potentially significant endeavor.\n\nThe insight of self-prediction generally outperforming contrastive learning is interesting and significant.\n\nExperiments are performed on a wide range of tabular classification and regression datasets.\n\nWeaknesses\n---\nThere should be a more detailed comparison to GBDTs. The performance of the best pretraining methods are often similar or marginally better than the GBDT methods. An empirical runtime comparison of the training and inference times between GBDTs and the MLPs would provide significant value. Runtime results in addition to model size comparisons would give readers and machine learning practitioners a better sense of the tradeoffs between these methods and ultimately decide if the marginal increase in predictive performance is worth the additional complexity of different pretraining methods.\n\nNo comparisons to AutoML methods such as AutoGluon, which often outperform simple GBDT methods for tabular tasks.\n\nThe writing could be more precise in places. For example, what percentage of the inputs are corrupted in the description of the contrastive approach described in Section 3.2? Also, in Tables 3 and 4, the caption describes the bolded results as statistically significant across all models; however, there are multiple bolded numbers in the same column, thus I'm not sure what the bolded results are significant to. There are also a number of grammatical errors; consider using a service like Grammarly to fix these issues.\n\nI think a more detailed discussion of the differences between this work and the work by Gorishniy et al. (2021) would greatly benefit readers. Also a table summarizing all of the pretraining methods discussed in this paper would also help readers significantly.\n\nMinor Weaknesses\n---\nThe definitions of \"standard ensemble\" and \"efficient ensemble\" in Table 5 are not clearly stated. There are also no standard errors or significance tests for Table 5.\n\nConsider using \"\\cite\" when passively citing previous work.\n\nWhen open quotes are used, they are facing the wrong direction.\n\nTables 1, 6, and 7 are shown in the middle of a paragraph, consider moving them to either before or after the paragraph, or to the top or bottom of the page.\n\nMissing period in the captions of Tables 1, 2, and 5.\n\n\"MLP-PLR\" should be \"MLP-P-LR\" in the \"Models\" paragraph of Section 3.1\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FDTUdEiOmv", "reviewer": "Reviewer_9GeL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with tabular data, specifically examining in detail the question whether deep learning models can be trained to compete with traditional methods based on decision trees. In particular, the paper studies the role of pretraining, a training modality that is peculiar to deep learning and has been crucial in establishing its dominance in NLP and vision.\nThrough a extensive battery of experiments on several tabular datasets, the paper evaluates various pretraining methods and architectures, showing that specific types of pretraining allow deep learning models to be competitive against decision tree models on tabular dataset.\nThe most successful type of pretraining combines mask denoising and label noise, and the paper proposes an explanation of the success of label noise based on studying a synthetic dataset which allows to control the relevance of individual features to predict the target. This study reveals that masked and label noise denoising pretraining allows to better predict features that are less relevant to predict the target.\nFinally, the paper shows that the same pretrained deep learning model can be finetuned multiple times, and all the multiple finetuned models combined to increase performance even further.", "summary_of_the_review": "The paper presents a timely exhaustive study on the effect of pretraining for deep learning on tabular data, and how this can be used to beat traditional machine learning techniques like decision trees.", "strength_and_weaknesses": "Strengths:\n- Well-motivated and timely study that shows that deep learning models can consistently beat decision trees on tabular datasets\n- Extensive study on multiple datasets and invesigating multiple pretraining procedures \n- Interesting synthetic dataset construction to control feature relevance to study the effect of combine masked and label noise denoising pretraining\n- Careful comparison of different types of MLPs and transformer models\n- Useful study of computation and time cost due to pretraining\n\nThe paper has a few minor weaknesses, which can mostly addressed by clarifying some points.\n- In he synthetic experiment, is importance rank in Fig. 1 given by the influence of the vector p? Or is that derived from the feature importances?\n- During finetuning in the efficient ensembling case, is the whole architecture including the backbone trained end-to-end, or is just the heads that are being trained on a fixed backbone? This makes a great difference in terms of compute and memory: the difference between keeping around 15 different backbones and only one.\n- The paper is rather complete in terms of citations of previous works, but is however missing some notably very connected papers. It would be for instance beneficial to cite the paper Padhi et al. \"Tabular Transformers for Modeling Multivariate Time Series\", ICASSP 2021, which also uses masked denoising pretraining (although in tabular time series as opposed to simple tabular datasets).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject