Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies transfer learning on deep tabular models. Authors create a transfer learning benchmark from the MetaMIMIC repository, and then conduct transfer learning experiments on it. Specifically, authors find that fine-tuning pre-trained FT-Transformer consistently outperforms classical models such as XGBoost and Catboost with different training data sizes. Authors also find self-supervised pre-training is inferior to supervised pre-training on tabular data. Additionally, to deal with applications with missed features, authors propose to use pseudo values for feature alignments. 
strength_and_weaknesses: Strength
1. Deep learning with tabular data is an important and unsolved problem. 
2. This paper is clearly written. 
3. The proposed approach of pseudo-value completion seems interesting and effective. 

Weaknesses
1. Previous studies (such as FT-Transformer) have already shown that DNNs can generally outperform XGBoost/CatBoost. Therefore, the main findings of this paper sound less surprising. Authors provide no insights about when DNNs transfer better than tree models and when not. 
2. All experiments use 11 splits as the upstream task and 1 as the downstream task. This may not be representative enough for various applications. I wonder whether using less splits in pre-training still results in the same conclusion. 
3. A minor typo: “with without”
clarity,_quality,_novelty_and_reproducibility: Good clarity, quality and reproducibility. Novelty is a bit limited.  
summary_of_the_review: Overall, this paper presents useful comparisons of transfer learning between deep Transformers and tree models on tabular data, though the findings are not so surprising. My major concerns are, 1) when (on which kinds of data) shall we use deep Transformers and when to use tree models, and 2) whether is the conclusion still right with fewer pre-training tasks?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ## Summary

* This paper investigates whether deep learning methods are effective for transfer learning on tabular data. In comparison to domains such as NLP and CV, tabular data transfer learning is not as well studied.

* The overall contributions of the paper are: (1) a thorough empirical evaluation of transfer learning methods on tabular data (considering gradient boosted decision trees, training deep models from random initialization, supervised pre-training, and self-supervised pre-training); (2) a strategy for transfer learning with tabular data when the pretraining and finetuning datasets differ in what features are measured; (3) a distilled set of recommendations for empirical work in this space.


## More details 
* The paper outlines an empirical testbed for understanding the benefit of transfer learning with tabular data. The authors adopt a medical intensive care unit dataset, MIMIC-IV, for the primary evaluation. This dataset has structured tabular data for a number of patients together with 12 downstream labels of interest. The evaluation setup for supervised pretraining involves: (1) conducting pre-training on 11 of the labels; (2) fine-tuning on the 1 remaining label, with different finetuning dataset sizes, to gauge performance. 

* There is also an evaluation on two non-medical datasets in the appendix. 

* In this experimental evaluation, the authors find that supervised pre-training outperforms training strong gradient-boosed decision trees (GDBT) baselines  with tuned hyperparameters (Xgboost and Catboost). This is an interesting result, since it demonstrates that neural models can be effective for transfer learning on tabular data. 

* The paper evaluates a variety of different neural network strategies in the experimental evaluation, including recently proposed strategies for tabular data such as FT-Transformer and TabTransformer, in addition to simpler methods such as MLPs and ResNets. A variety of different finetuning strategies are also considered for these models (e.g., linear evaluation, end2end training with a linear head/MLP head).

*  The second part of the experimental evaluation compares different pre-training strategies, namely supervised pre-training as compared to self-supervised strategies such as masked imputation. Overall, supervised pretraining strategies perform better than self-supervised pretraining methods.

* The paper then studies a setting where the pretraining and finetuning sets may contain different features, so straightforward transfer learning is not as readily appropriate. To tackle this, the paper suggests a process to impute the missing feature using an additional trained model, before re-running pretraining/finetuning. They find that this strategy improves over just excluding the missing feature (though the evaluation in this setting appears less complete). 

* The paper presents detailed information about experimental setups, including considerations of: how to handle the small data regimes, how to do hyperparameter search, ranges of considered hyperparameters, how early stopping applies in such extreme small data regimes. 
strength_and_weaknesses: ## Strengths

* The empirical evaluation in the paper is thorough and clear. The range of investigated methods (state of the art deep learning methods for tabular data, popular tree-based models, self-supervised methods) is very helpful in understanding trends and how these different strategies perform. Additionally, evaluation on multiple datasets of different types helps strengthen the empirical findings.

* The range of experiments and domains studied, together with distilled findings, is useful for practitioners to understand how best to apply these strategies in practice.

* The related work is fairly comprehensive as far as the ML space goes. There are other works that would be helpful to refer to, which are mentioned later in these comments.

## Neutral 

* One could argue that the main methodological novelty in this paper comes from the last section, which concerns how to handle situations when the pretraining and finetuning tasks differ in what features they consider. Personally I think the strength of the empirical study and evaluation is valuable as a standalone contribution, even if this is the only 'new' methodological aspect.

## Weaknesses

### Details on empirical study
Given the work's main contribution is the empirical study, there are some more details that would be good to include for full completeness.
1.  A clearer discussion of how many different seeds/splits were considered in experiments.
2. Why were these particular low data regimes considered, as opposed to other values? What happens if we scale up to larger dataset sizes (even a single run with a large data scale would help round off the discussion)? 
3. Was splitting into train/val/test on the upstream and downstream tasks on a per-patient level? What were upstream dataset sizes on average?
4. Dataset details are particularly crucial for the medical application domain -- could you provide a 1-2 sentence description of the tasks, and the label prevalences?
5. What do error bars on the ranks look like (even for a subset of the experiments)?

### Other related works
Although the related work surveys the ML field well, there are some specific relevant works in the medical ML literature that would be worth mentioning somewhere. They deal with structured data timeseries rather than prediction from a single vector, but they also investigate contrastive pretraining, masked imputation pretraining, and supervised pretraining: 
* https://proceedings.mlr.press/v139/yeche21a.html (ICML 2021)
* https://dl.acm.org/doi/pdf/10.1145/3450439.3451877 (CHIL 2021).

### Other things
* In the contrastive experiments, how were the corruption hyperparameters chosen? These might significantly impact performance, so should they also be incorporated in the hyperparameter search?
* This ICLR paper from this year: https://openreview.net/forum?id=CuV_qYkmKb3 studied contrastive learning on tabular data, and performed comprehensive experiments. This might be a better augmentation strategy to compare with, rather than the one studied.
* In the pseudo feature experiments, what happens if more than one feature needs to be imputed? How does this impact performance? Do the experiments consider removing different features in each run? What does the variance in performance/rank look like?
clarity,_quality,_novelty_and_reproducibility: The work is clear, and figures and writing are easy to understand. The work is of high quality, with clear contributions. The work's findings are mostly original, with some aspects being studied at least in part in prior works.
summary_of_the_review: Overall, I think the experimental evaluation in this paper is thorough and has useful insights for practitioners. I would therefore vote to accept it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the highly important problem of transfer learning with DNNs on tabular data. This avenue has huge potential in many practical applications. The problem and DNNs for tabular data are generally underexplored so this contribution could be important for the community. They propose a way to use DNNs for transfer tasks in several settings. The proposed schemes are compared to strong tree-based methods on many datasets, indicating the potential of transfer learning on tabular data.
strength_and_weaknesses: Strengths: The paper is well-written and easy to follow. The approach is simple but seems to work in many datasets and across different settings. Expireneltal evaluation is strong and demonstrates the potential of the approach.
Weaknesses: some related work on regularizations for tabular data is missing; for example

[1] Kadra et al. Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data 2021

[2] Yang et al. Locally Sparse Neural Networks for Tabular Biomedical Data 2022

[3] James Fiedler, Simple modifications to improve tabular neural networks

To improve the presentation, it would be worthwhile to remind the reader what are MLM and FS in the caption of figure 3.
Also, please explain the procedure for downstream pseudo features in the caption of figure 4.

Do you have intuition why this procedure is beneficial? Was this done (or analyzed) in any prior work? This seems related to the procedure commonly used to impute missing values.

clarity,_quality,_novelty_and_reproducibility: The problem and solution are novel. The quality of the writing and explanations are good. The authors detail all hyperparameters and use Optuna to optimize all methods. I feel that this approach is fair, and reproducibility considerations are satisfied. 
summary_of_the_review: Overall I enjoyed reading the paper. I think new solutions to challenges in tabular data are really important for the ML community. The authors did a good job of analyzing and presenting their work. The proposed solution is simple yet seems to be effective even when the number of labeled samples in the target distribution is not high. This can open the door to many follow-up works. I believe that the paper should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies transfer learning for tabular data. The authors take MetaMIMIC for experiments, where there are 12 medical targets. They train on 11 targets and test on the leaveout target. They find that transfer learning provides definitive advantages over gradient boosted decision trees. They further compare self-supervised learning and supervised pretraining on tabular and find that supervised pretraining is more helpful. Finally, they propose a new feature imputation method to learn tackle missing values.
strength_and_weaknesses: Strengths:

1. The paper studies transfer learning on 12 medical targets.

2. The paper demonstrates the benefits of using supervised pretraining for tabular data.

3. A method for feature imputation is proposed. 


Weaknesses:

1. All the experiments are performed in the medical domain. The 12 selected targets are closely related. Although the authors also validate on other datasets including Yeast functional genomics data and Emotions data, the targets in each dataset are too similar. Does the author test on other domains? For example, the authors can train on MetaMIMC and validate on Yeast functional genomics data.

2. Similarly, the authors should provide some analysis about the 12 targets of MetaMIMC. How correlated these targets are?

3. There are several works studying transfer learning for tabular data, e.g. TransTab (https://arxiv.org/abs/2205.09328). The novelty of this paper is ok but not significant. Although the paper cites the TransTab, it should not be regarded as concurrent work anymore because the paper is already on arxiv in May. The paper may need to carefully compare the differences with respect to these previous work. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 

The quality is sound. 

The novelty is ok but not significant given existing works about transfer learning on tables.  

Reproducibility is good.
summary_of_the_review: 1. The chosen targets may be too similar.
2. No analysis about the targets used. 
3. Missing comparison with previous work. Unclear about the novelty.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is an empirical study of whether recently introduced methods for tabular neural networks can work better than the more established methods methods based on ensembles of trees, once we take pre-training into account.

Strengths: 
* This paper adds interesting new evidence to an ongoing recent debate [1-4].
* The reviewers agreed that the empirical evaluation is thorough and clear.
* The method of "pseudo-features" shows that pre-training can be useful for data imputation as well, which is an interesting result.

Weaknesses:
* Some of the reviewers raise questions about whether the main findings of the paper. However, there is active debate in this area, so new evidence seems worthwhile.
* Reviewers also pointed out that these studies were limited to the medical domain. I agree that experiments in other domains would provide additional information that would be very useful, but there also seems to be broad agreement that the current result already make a worthwhile contribution.


[1] Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. "Why do tree-based models still outperform deep learning on tabular data?." arXiv preprint arXiv:2207.08815 (2022).

[2] Borisov, Vadim, et al. "Deep neural networks and tabular data: A survey." arXiv preprint arXiv:2110.01889 (2021).

[3] Shwartz-Ziv, Ravid, and Amitai Armon. "Tabular data: Deep learning is not all you need." Information Fusion 81 (2022): 84-90.

[4] Somepalli, Gowthami, et al. "Saint: Improved neural networks for tabular data via row attention and contrastive pre-training." arXiv preprint arXiv:2106.01342 (2021).

justification_for_why_not_higher_score: The limited set of domains might reduce the general interest of the paper somewhat.
justification_for_why_not_lower_score: Investigating the role of pre-training adds timely new evidence to an important debate.
Decision: Accept: poster