Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper deals with tabular data, specifically examining in detail the question whether deep learning models can be trained to compete with traditional methods based on decision trees. In particular, the paper studies the role of pretraining, a training modality that is peculiar to deep learning and has been crucial in establishing its dominance in NLP and vision.
Through a extensive battery of experiments on several tabular datasets, the paper evaluates various pretraining methods and architectures, showing that specific types of pretraining allow deep learning models to be competitive against decision tree models on tabular dataset.
The most successful type of pretraining combines mask denoising and label noise, and the paper proposes an explanation of the success of label noise based on studying a synthetic dataset which allows to control the relevance of individual features to predict the target. This study reveals that masked and label noise denoising pretraining allows to better predict features that are less relevant to predict the target.
Finally, the paper shows that the same pretrained deep learning model can be finetuned multiple times, and all the multiple finetuned models combined to increase performance even further.
strength_and_weaknesses: Strengths:
- Well-motivated and timely study that shows that deep learning models can consistently beat decision trees on tabular datasets
- Extensive study on multiple datasets and invesigating multiple pretraining procedures 
- Interesting synthetic dataset construction to control feature relevance to study the effect of combine masked and label noise denoising pretraining
- Careful comparison of different types of MLPs and transformer models
- Useful study of computation and time cost due to pretraining

The paper has a few minor weaknesses, which can mostly addressed by clarifying some points.
- In he synthetic experiment, is importance rank in Fig. 1 given by the influence of the vector p? Or is that derived from the feature importances?
- During finetuning in the efficient ensembling case, is the whole architecture including the backbone trained end-to-end, or is just the heads that are being trained on a fixed backbone? This makes a great difference in terms of compute and memory: the difference between keeping around 15 different backbones and only one.
- The paper is rather complete in terms of citations of previous works, but is however missing some notably very connected papers. It would be for instance beneficial to cite the paper Padhi et al. "Tabular Transformers for Modeling Multivariate Time Series", ICASSP 2021, which also uses masked denoising pretraining (although in tabular time series as opposed to simple tabular datasets).
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very clearly motivated and presented
Quality: High-quality systematic study on the effect of different pretraining techniques on deep learning for tabular data
Novelty: The paper's strength is not innovation, but this type of systematic study comparing traditional ML and deep learning for tabular data is very timely
Reproducibility: The paper's results seem reasonably reproducible thanks to the detailed description of the experiments and hyperparameters used
summary_of_the_review: The paper presents a timely exhaustive study on the effect of pretraining for deep learning on tabular data, and how this can be used to beat traditional machine learning techniques like decision trees.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the pretraining practice on tabular data. It shows that using the object target labels during the pretraining stage is beneficial for the downstream performance. Several target-aware pretraining objectives are proposed. Experiments on various datasets validate the claim.

strength_and_weaknesses: The paper compares various supervised pretraining objectives on tabular data. Some helpful conclusions are drawn from the experimental results. 

Here are some suggestions:
1. The main assumption of the paper is that "Pretraining is always performed directly on the target dataset and does not exploit additional data", which is not always the case in the deep learning field. It seems the authors need to point out revisiting the "supervised" pretraining. The reviewer has two questions:
    1.1 There are some self-supervised learning strategies on tabular data, such as SubTab. Will they help get better performance?
    1.2 Could we pre-train on another dataset and then apply the pre-trained model to the target dataset? 
2. The authors need to make the formulation of the pretraining and fine-tuning stages clear. Which learnable parameters are shared between the two stages? Some readers may not be familiar with the tabular model.
3. The pretraining objectives introduce a lot of augmentation strategies. Will they also help GBDT methods?
4. If we use different deep models like ResNet or FT-Transformer, will we get similar results?
5. The authors validate that pretraining is a good trick to boost the ability of deep tabular models. Does it mean that pretraining becomes a basic step when we use deep models on tabular data? Then how to compare various deep models in a fair manner? Different methods may need different pretraining strategies.
clarity,_quality,_novelty_and_reproducibility: Although the novelty of the paper is a bit limited, the empirical results show some good practice for tabular data training. The experiments are convincing.

summary_of_the_review: The paper is clear and well-written. Several pretraining strategies are investigated in the paper. Experiments support the claim of the paper. There are some suggestions, and please see the weakness part.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Deep learning has become an important method for tabular data. Even though pretraining based DL has been demonstrated successful in computer vision and NLP problems, it is challenging to make reliable conclusion about pretraining efficacy in tabular DL. This work provides a fully labeled tabular datasets to understand if pretraining helps tabular DL in a fully supervised setting and compare pretraining methods to the strong supervised baselines. 

Using the target dataset, the method pretrain the model parameters with pretraining objectives, including generative positive pairs in contrastive-like objective or to corrupt the input for reconstruction in a self-prediction based objectives. In the second stage, the model is finetuned on the downstream classification or regression task. 
strength_and_weaknesses: Strengths:
- The paper generates some useful insights around how to pretrain a MLP model for tabular data and finetune the model to achieve stronger results on downstream tasks. There are also interesting discussions round pretraining objective. Self reconstruction and mask prediction are useful pretraining objectives, as compared to contrastive loss. 

- Combing supervised loss and unsupervised loss in the pretraining helps further increase the downstream performance. 

Weaknesses:
- The paper is not clear about its experimental setup: what is the pretraining dataset and what is the downstream datasets? Are they the same but with different losses? 

- As the paper pointed out that adding supervised loss and unsupervised loss during pretraining enhances downstream task performance. But the paper does not provide any evidence if adding target task prediction loss during pretraining would benefit generalization? Whether adding the prediction of some tasks would hurt the performance on some other tasks with different loss functions?

- If there is not universal solution between self-prediction objective, when applied in real, how should we pick the optimal objective, given some mixture of downstream tasks?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and is of decent quality. 
However, the paper is not novel. 
As the method is simpler, it should be pretty straightforward to reproduce. 
summary_of_the_review: The paper is insufficient for a publication in ICLR, as explained in the weakness section. 

Please provide more detailed explanation on the experimental setup, particularly:
- The pretraining dataset, dataset size.
- Downstream tasks. tasks losses, labeled dataset size, etc. 

Please also formulate the problem properly with the pretraining losses and downstream task prediction losses. 

If the downstream tasks have different prediction losses, how should we pick the pretraining recipe?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors explore different pretraining objectives for tabular datasets, apply them to vanilla and embedding-based MLP architectures, evaluate their relative downstream predictive performance, and compare their performance with gradient-boosted decision trees (GBDTs, namely CatBoost and XGBoost). Specifically, the authors explore unsupervised pretraining objectives, target-aware pretraining objectives, and combinations of both.

Experiments on 11 datasets (6 classification and 5 regression tasks) suggest several findings: 1) deep learning (DL) models generally benefit from pretraining in terms of downstream performance (this includes both simple MLP and transformer-based architectures, although there was no improvement of transformer-based models over simple MLP architectures); 2) target-aware pretraining objectives performed better than unsupervised objectives; 3) combining target-aware and unsupervised pretraining objectives tends to work best, and perform similarly or better than GBDTs on downstream tasks.
strength_and_weaknesses: Strengths
---
Pretraining is an important part of the significant success of DL models on downstream vision and NLP tasks, and thus exploring various approaches of pretraining for tabular data is an important and potentially significant endeavor.

The insight of self-prediction generally outperforming contrastive learning is interesting and significant.

Experiments are performed on a wide range of tabular classification and regression datasets.

Weaknesses
---
There should be a more detailed comparison to GBDTs. The performance of the best pretraining methods are often similar or marginally better than the GBDT methods. An empirical runtime comparison of the training and inference times between GBDTs and the MLPs would provide significant value. Runtime results in addition to model size comparisons would give readers and machine learning practitioners a better sense of the tradeoffs between these methods and ultimately decide if the marginal increase in predictive performance is worth the additional complexity of different pretraining methods.

No comparisons to AutoML methods such as AutoGluon, which often outperform simple GBDT methods for tabular tasks.

The writing could be more precise in places. For example, what percentage of the inputs are corrupted in the description of the contrastive approach described in Section 3.2? Also, in Tables 3 and 4, the caption describes the bolded results as statistically significant across all models; however, there are multiple bolded numbers in the same column, thus I'm not sure what the bolded results are significant to. There are also a number of grammatical errors; consider using a service like Grammarly to fix these issues.

I think a more detailed discussion of the differences between this work and the work by Gorishniy et al. (2021) would greatly benefit readers. Also a table summarizing all of the pretraining methods discussed in this paper would also help readers significantly.

Minor Weaknesses
---
The definitions of "standard ensemble" and "efficient ensemble" in Table 5 are not clearly stated. There are also no standard errors or significance tests for Table 5.

Consider using "\cite" when passively citing previous work.

When open quotes are used, they are facing the wrong direction.

Tables 1, 6, and 7 are shown in the middle of a paragraph, consider moving them to either before or after the paragraph, or to the top or bottom of the page.

Missing period in the captions of Tables 1, 2, and 5.

"MLP-PLR" should be "MLP-P-LR" in the "Models" paragraph of Section 3.1

clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved, but the paper provides a nice overview and evaluation adapting existing pretraining objectives from vision and NLP tasks to tabular tasks.
summary_of_the_review: The paper builds on work by Gorishniy et al. (2021) and expands the evaluation of pretraining objectives for tabular data. However, the paper lacks a more thorough comparison to GBDTs and AutoML methods, and the clarity of the paper could be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the effect of pretraining on the tabular benchmarks. Specifically, the authors focus on fully-supervised setups without any extra (unlabeled) dataset, and they examine various pretraining objectives on fully-labeled datasets by pretraining and fine-tuning on the same dataset. At the initial reviews, this paper received both strong positive and negative scores (8 vs 3), and the reviewers did not reach an agreement during the discussion period. The reviewers' concerns are the limited novelty, the marginal improvements, and the unclear presentation, although the paper provides some useful insights around how to pretrain a MLP model for tabular data.

justification_for_why_not_higher_score: AC also carefully read this paper, and generally agrees with the concerns of reviewers. AC found that the idea of pretraining on the same dataset is well-known and the improvements look marginal. Hence, AC thinks the observations of this paper are somewhat straightforward (even though the tabular domain is new). Moreover, AC feels this paper is not ready for publication at this moment due to the unsatisfactory presentation quality throughout this paper. For example, as many reviewers pointed out, there is no clear description and formulation about pretraining objectives. AC strongly suggests revising the manuscript with a more neat, clear, and comprehensive presentation. Considering all aspects, AC thinks this paper does not meet the high standard of the ICLR community, so AC tends to recommend rejection.
Decision: Reject