Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors introduce TabPFN. A transformer-based neural network for classification. Unlike traditional supervised approaches, the network is pre-trained to run classification on unseen datasets.
This transformer approach, runs inference on input data to produce a classification result. However, this input data contains the training and prediction datasets removing the need to do pre-training for a given dataset-classification task.

TabPFN is pre-trained on different synthetic datasets for the task of learning to obtain a hypothesis and use the hypothesis to make predictions. The limitations of the approach are clear, the dimensionality of the datasets is constrained by the transformer architecture, and therefore the scalability during training and inference grows quadratically. Nevertheless, this "ready out-of-the-box approach" offers state-of-the-art performance when compared to other classifiers. 
strength_and_weaknesses: The paper presents an approach based on PFNs, where the authors pre-train their network with synthetic data obtaining great results compared to other state-of-the-art approaches. They present a data generation approach that successfully transfers different hypothesis to the TabPFNs.

There are however claims such as "Predictions based on causal reasoning" that should be better substantiated. As the authors do not demonstrate this is indeed happening. Just because the synthetic dataset comes from an SCM, is no guarantee that TabPFN is doing causal reasoning.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and technically sound. It presents a new approach with state-of-the-art performance for limited datasets. Furthermore, the authors provide code which helps on reproducibility.
summary_of_the_review: The authors present a synthetic generator as well as a change in the transformer mask of PFNs to create a strong tabular classificator. They evaluate against state-of-the-art approaches and demonstrate increased accuracy as well as a significant reduction in time.

In general, I find the approach very interesting. However, I'm trying to understand the significance of the contribution compared to the PFNs paper. Indeed, the pre-training using SCMs is a significant portion of the contribution. However training on BNNs was already presented before and the speed characteristics are coming from PFNs.

As you mention in Fig. 1 that plots are based on the PFN paper, it would be good to repeat the same for Fig. 2, as there is an overlap.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission present tabPFN, an approach for fast learning on small tabular datasets. The approach uses a transformer to featurize the dataset and can learn with a signle pass forward through the network, without any backpropagation, which constitutes the biggest benefit of the approach. The model is an instance of "Prior-Data Fitter Network", fitted in a meta-learning style on rich simulated data. These data are generated via either Bayesian Neural Networks or Structural Causal Models, tuned to generate data similar to tabular data, with up to 2000 samples, 100 features and 10 imbalanced classes per dataset.

The methods is benchmarked on 30 small datasets from the OpenML-CC18 suite, comparing to established baselines, including ligthGBM and XGBoost. Benchmarks show better performance of the contributed model without fine-tuning or hyper-parameter selection, and in particular with very little time budget. *Edit* after rebuttal period, the authors added more datasets.
strength_and_weaknesses: In my eyes, the strength of this contribution is its originality. It contributes a novel incredient to tabular deep-learning research, and has promising benchmarks.

However, I worry about the benchmarks and claims of "much better performance" than XGBoost, LigthGBM in the introduction. Many practitionners have reported that the claims of tabular deep learning papers do not match their experience (which I found true with my own experimentation on many occasions). I honestly do not know why there is this disconnect, but it is not good press for our community. Along this line, Grinstztajn et al provided a benchmark which they claim is more realistic than many used in the literature. It would have been interesting to include this benchmark.

Tabular data have a form of rotational invariance (individual features are important). Is this enforced in the BNN sampling (I can see how it would appear in the SCM sampling).

Multi-class prediction: how is the noise added? A cursory description almost suggests that the link is deterministic: one specific configuration of X leading to a given class

Section 3.5: typo: unqiue

For lightGBM and XGBoost, the native handling of missing values should be used, rather than imputation

Which models do AutoML systems choose (to be able to give conclusion that situation various families of models)? This information would be interesting to add to the manuscript.


clarity,_quality,_novelty_and_reproducibility: The manuscript is clear and well written (though the profusion of acronyms impedes readibility). It is interesting research bringing new elements to the community. And the reproducibility is strong given the material shared.

summary_of_the_review: An interesting idea with very strong claims. I am not sure if the empirical evidence is quite as strong as the claims, and the claims should be moderated.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers classification problems with small tabular datasets (a few thousand samples, and up to ten classes and 100 features). It presents a pre-trained model that solves such problems within a second, outperforming tree-ensembles and AutoML methods that run for an hour. The pre-trained model is a 12-layer transformer-based prior-data fitted network (PFN, introduced in Muller 2022), which uses a prior based on structural causal models. The pre-training uses 18000 batches of 512 simulated datasets. The training process takes 20 hours on an 8-GPU machine. Then the network is applied to 30 OpenML datasets (which takes only 1 sec on a GPU) and provides better ROCAUC results compared to XGBoost, Catboost, Auto-SkLearn and Auto-Gluon, which have a full hour for hyperparameter search. Some of these datasets had known state-of-the-art results (OpenML-AutoML) that were compared for validation.

strength_and_weaknesses: The paper presents an interesting new approach for the important problem of tabular data classification. The presented technique is limited to small datasets, as mentioned above, due to compute constraints (quadratic in size), but these smaller sizes could still be of interest. It is generally well-written. 
A key weakness is that the pre-trained model requires providing the training and test data together. Even though the test data is not supposed to be used, it is difficult to verify this. Also, the generation of simulated data for the transformer training is not explained well, and it is important to understand how this was done, to make sure there was no leakage from the test datasets.
Another experiment that should have been done is using the default hyperparameters of Catboost, XGBoost and LightGBM, which may provide better results than an hour of search in a large hyperparameter space.


clarity,_quality,_novelty_and_reproducibility: The presented approach seems novel, the authors provided the code for reproducibility and the datasets are public. However, it is difficult to verify the quality, as mentioned above,  since the pre-trained model requires providing the training and test data together, and it is difficult to verify that the test data is not used. It is also unclear if comparing to other models with an hour of hyperparameter search in a large space is better than just taking their default hyperparameters. The paper is generally clear, but the generation of the simulated data for obtaining the pretrained transformer should be explained in more detail, to assure no leakage from the datasets later used for evaluating the pre-trained model.
summary_of_the_review: This paper presents interesting results for an important problem. To accept it, I would like to see a code that does not require providing the training and test data together, and still obtains the same results. Another experiment that should be done is using the default Catboost and XGBoost hyperparameters, which may provide better results than an hour of search in a large hyperparameter space. Also, the simulated data for the transformer pre-training should be explained in more detail, to assure there was no leakage from datasets later used for the evaluation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose TabPFN, a transformer-based prior-data fitted network trained to directly approximate the posterior predictive distribution (PPD), i.e., an infinitely large set of data-generating mechanisms. The main contribution of this work is the proposed prior, which is based on structural causal models (SCMs) and Bayesian neural networks (BNNs) in which distributions instead of point estimates are used for the hyperparameters of the prior. TabPFN is trained on 512 synthetic datasets generated from the prior and, once trained, can approximate the PPD for the proposed prior in a single forward pass for a set of target instances given a set of training examples. The result is a transfromer-based model trained to solve synthetically generated classification tasks from a tabular dataset prior in which predictions are very efficient.

Experiments on 30 small-scale classification datasets demonstrate TabPFN's ability to outperform gradient-boosted decision trees (GBDTs) and perform competitively with state-of-the-art AutoML approaches such as AutoGluon in terms of predictive performance while being significantly more efficient.
strength_and_weaknesses: Strengths
---
TabPFN takes a significantly different approach to tabular classification problems than many current deep-learning approaches, leveraging the transformer architecture and a novel data-generating prior to produce competitive predictions for new unseen tabular classification datasets in a fraction of a second.

Comparisons on toy datasets demonstrate TabPFN's ability to produce appropriate decision boundaries that sometimes better reflect the underlying dataset distribution than more traditional machine-learning approaches.

Experiments are performed on a large number (30) of tabular classification datasets.

Predictions for a new unseen test set require less than a second (using GPU).

Weaknesses
---
TabPFN is only suitable for small-scale tabular classification tasks (e.g., $\le$ 2000 examples), thus limiting TabPFN to larger problems.

The predictive performance of TabPFN and AutoGluon is similar (Table 1), and the combination of TabPFN (with ensembling) and AutoGluon tends to work best. However, TabPFN + AutoGluon seems only marginally better than AutoGluon on it own, questioning the necessity of adding a complex method such as TabPFN.

The empirical runtime comparisons between TabPFN and the competing methods is also not clearly described. Are the times in Table 1 showing the average total train+test time for the non-TabPFN methods? Also, are the non-PFN methods evaluated using a CPU or GPU? I think it would be useful to see training and test times as separate entities in Table 1, making it more clear where the tradeoffs exist between using TabPFN or a more traditional method like GBDTs or AutoGluon.

I think there could be a clearer discussion more precisely describing the differences between this work and the work by Muller et al. (2022) since this work is heavily inspired by that work. For example, the authors introduce their novel prior in Section 3 describing their use of SCMs and BNNs, but then also describe their use of BNNs follows the work by Muller et al. in Section 3.4. I think it also important to see a comparison to the work by Muller et al. in the experimental evaluation, or at least a discussion about why this comparison is not relevant.

Additional Comments
---
Are there any model size comparison results between TabPFN and competing methods? This information could further help ML practitioners when deciding whether to use TabPFN or a different method for their particular problem.

Minor Weaknesses
---
In Figure 1, the first description in the caption (Left (a)) is describing the right part of the figure, while the second description (Right (b)) is describing the left part of the figure.

Figures 3, 4, and 5 are not color-blind friendly.

Footnote 4 should go after the period.
clarity,_quality,_novelty_and_reproducibility: The work is relatively clear and well-written and appears novel.
summary_of_the_review: The work takes an interesting and relatively new approach towards small-scale tabular classification problems; with some refining, I think this paper can potentially make a significant contribution to the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper provides a neural network for classification that is pre-trained on synthetic data to be applied on unseen datasets, quickly and without hyperparameter tuning. The reviews agree that the idea is novel in that it is quite different than existing approaches for tabular data classification. Additionally, the empirical results seem to be thorough and convincingly show that the method provides high quality results on small datasets. One main drawback of the method is its limitation to small datasets, but I agree with the authors claim that (1) even with this limitation, the scope of influence is large as small datasets are quite common, and (2) given the originality of the paper, the idea could be extended to handle larger datasets in the future.

In the reviews, a few more concerns were raised regarding the representation of the test datasets of real-world problems, the paper's clarity, and some details of the algorithms. These were addressed in the rebuttal phase and seem to have been resolved. Overall, the paper is novel and shows promising results. I believe the scope of influence is great and that it would be a great addition to ICLR. I do urge the authors to carefully go over the reviews and discussions with the reviewers and incorporate the needed change to a camera-ready version.


justification_for_why_not_higher_score: The main reason is the scope. I personally find that tabular data is an under-researched area in deep learning, but the fact is that many of the people in ICLR will not find this paper relevant because of the topic.

justification_for_why_not_lower_score: The paper provides a novel approach to a highly explored area of tabular classification. This is a tough area to obtain high quality results given the great performance of tree-ensemble methods. Given the novelty of the approach, I think the ICLR audience would be interested in this paper, and think it deserves a spotlight.
Decision: Accept: notable-top-25%