TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
Authors: Noah Hollmann
Samuel Müller
Katharina Eggensperger
Frank Hutter
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Tabular Data
AutoML
Green AI
Bayesian prediction
Causal Reasoning
Real-time Machine Learning

Abstract: We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second, needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
TabPFN is fully entailed in the weights of our network, which accepts training and test samples as a set-valued input and yields predictions for the entire test set in a single forward pass.
TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior.
This prior incorporates ideas from causal reasoning: It entails a large space of structural causal models with a preference for simple structures.
On the $18$ datasets in the OpenML-CC18 suite that contain up to 1000 training data points, up to 100 purely numerical features without missing values, and up to 10 classes, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to $230\times$ speedup.
This increases to a $5\,700\times$ speedup when using a GPU. We also validate these results on an additional 67 small numerical datasets from OpenML.
We provide all our code, the trained TabPFN, an interactive browser demo and a Colab notebook at https://github.com/automl/TabPFN.

Published as a conference paper at ICLR 2023
TABPFN: A TRANSFORMER THAT SOLVES SMALL TABULAR CLASSIFICATION PROBLEMS IN A SECOND
Noah Hollmann Samuel Müller".! Katharina Eggensperger' Frank Hutter
1 University of Freiburg, Charité University Medicine Berlin
3 Bosch Center for Artificial Intelligence Equal contribution. Correspondence to noah. hol Imann@ charite rire.tte.e de & muellesa@cs. uni-freiburg
ABSTRACT
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than second, needs hyperparameter tuning and is competitive with state-of-the-art classification methods. TabPFN is fully entailed in the weights of our network, which accepts training and test samples a saluee input and yields predictions for the entire test set single forward pass. TabPFN is Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior. This prior incorporates ideas from causal reasoning: It entails large space of structural causal models with preference for simple structures. On the 18 datasets in the OpenML-CCI8 suite that contain up to 000 training data points, up 100 purely numerical features without missing values, and up to 10 classes, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-e AutoML systems with up to 230 speedup. This increases to a 700x speedup when using a GPU. We also validate these results additional 67 small numerical datasets from OpenML. We provide all our code, the trained TabPEN, an interactive browser demo and a Colab notebook at hnttps://githiie com aut.oml /
INTRODUCTION
Tabular data has long been overlooked by deep learning research, despite being the most common data type in real-world machine learning (ML) applications (Chui et al. 2018). While deep learning methods excel many ML applications, tabular data classification problems still dominated by Baadient-Boostedd Decision Trees (GBDT; Friedman 2001), largely due to their short training time and robustness (Shwartz-Ziv and Armon 2022).
We propose a radical change to how tabular classification is done. We do not fit a new model from scratch to the training portion of new dataset. Instead, we replace this step by performing single forward pass with large Transformer that has been pre-trained to solve artificially generated classification tasks from a tabular dataset prior.
Our method builds on Prior-Data Fitted Networks (PFNs; Müller et al. 2022 see Section 2). which learn the training and prediction algorithm itself. PFNs approximate Bayesian inference given any prior one can sample from and approximate the posterior predictive distribution (PPD) directly. While inductive biases NNs and GBDTs depend on them being efficient to implement (e.g., through L2 regularization, dropout (Srivastava et al. 2014) or limited tree-depth), in PFNs, one can simply design a dataset-generating algorithm that encodes the desired prior. This fundamentally changes the way can design learning algorithms.
We design a prior (see Section 4 based on Bayesian Neural Networks (BNNs; Neal I 1996 Gal 2016) and Structural Causal Models (SCMs; Pearl 2009 Peters et al. 2017) to model complex feature dependencies and potential causal mechanisms underlying tabular data. Our prior also takes ideas from Occam's razor: simpler SCMs and BNNs (with fewer parameters) have higher likelihood. Our prior is defined via parametric distributions, e.g., a log-scaled uniform distribution for the average number of nodes in data-generating SCMs. The resulting PPD implicitly models uncertainty

---

Published as a conference paper at ICLR 2023
all possible data-gen generaanng mechanisms, weighting them by their likelihood given the data and their prior probability. Thus, the PPD corresponds to an infinitely large ensemble of datt-rnnnr mechanisms, i.e,, instantiations of SCMs and BNNs. We learn to approximate this complex PPD in single forward-pass, requiring no cross-validation or model selection.
Our key contribution is introduce the TabPFN (see Section B single Transformer that has been pre-trained to approximate probabilistic inference for the novel prior above (described in more detail in Section 4 in single forward pass, and has thus learned solve novel small tabular classification tasks (5) 000 training examples, < 100 purely numerical features without missing values and < 10 classes) in less than second yielding state-of-the-an ahe performance.
To substantiate this claim, we qualitatively and quantitatively analyze the behavior and performance of our TabPFN on different tasks and compare to previous approaches for tabular classification on 18 small, numerical datasets (see Section 5). Quantitatively, the TabPFN yields much better performance than any individual "base-level" classification algorithm, such as gradient-boosting via XGBoost (Chen and Guestrin 2016). LightGBM (Ke et al.) (017) and CatBoost (Prokhorenkova ct al. 2018), and in less than second yields performance competitive to what the best available AutoML frameworks (Erickson et al.] 2020; Feurrr et al., (202T) achieve in one hour. Our in- depth qualitative analysis shows the TabPEN's predictions to be smooth and intuitive. Yet, its errors are quite uncorrelated to the errors of existing approaches, allowing additional performance improvements by ensembling. We also validate TabPFN's performance on an additional 67 datasets from OpenML. Vanschoren et al. (2014).
We expect the revolutionary character of our claims to be met with initial skepticism and thus open- source all our code and the pre-trained TabPFN for scrutinization by the community, along with scikit-learn-like interface, a Colab notebook and two online demos. Please see Section g for the links.
2 BACKGROUND ON PRIOR-DATA FITTED NETWORKS (PFNS)
First, we summarize how PFNs work; we refer to Müller et al. (2022) for the full details.
The Posterior Predictive Distribution for Supervised Learning In the Bayesian framework for supervised learning. the prior defines a space of hypotheses & on the relationship of a set of inputs the output labels y. Each hypothesis ø E 4 can be seen mechanism that generates data distribution from which we can draw samples forming a dataset. For example, given a prior based on structural causal models, & is the space of structural causal models, hypothesis o is one specific SCM, and a dataset comprises samples generated through this SCM. In practice, a dataset comprises training data with observed labels and test data where labels are missing or held out assess predictive performance. The PPD for a test sample Tiest specifies the distribution of its label p(- [test, Dtrain), which is conditioned on the set of training samples Derein == { 91), ..... (In, }n)]: The PPD can be obtained by integration over the space of hypotheses 4, where the weight of hypothesis determined by its prior probability p()) and the likelihood p(Dlo) of the data D given di
plyl,, D) x / p(u x, (oglx,0)po ())(((o)p(()dd
Synthetic Prior-fitting Prior-fitting is the training of a PFN to approximate the PPD and thus do Bayesian prediction. We implement it with prior which is specified by a prior sampling scheme of the form p(D)= = Ee [D))), which first samples hypotheses (generating mechanisms) with & ~ p(Φ) and then synthetic datasets with ~ p( D ()). We repeatedly sample such synthetic datasets D := (2;, (e)).e(......)) and optimize the PFN's parameters 0 to make predictions for the rest of the dataset Dergen
Drest D. conditioned training thus is the cross-entropy on held-out examples of synthetic datasets. For a single test point [Treett )leat)} Dreat, the training loss can be written as
D \ Drest- The loss of the PFN
LrrN
E ([(Hteat:Beeea)) IJUD )<pp() [- log qo (ytest Ftests Dtrain)]:
As shown by Müller et al. (2022), minimizing this loss approximates the true Bayesian posterior predictive distribution. We visualize this in Figure Figure[lajand aal detail the full training setup in Algorithm[] in the appendix. Crucially, this synthetic prior-fitting phase is performed only once for a given prior p( part of algorithm development.
2

---

Published as a conference paper at ICLR 2023
a(/a D) 4( (ooc D)
Done once, offline
Done per real world dataset, online
Sample synthetc dataset D. from phoor: D, p(D)
Real-world rmininng dotiset Dived and lest point
Triin TahPFN on synhetic dotasets (D...... Da)
Obeain (hees l'esst, Drcst? with single forwand
(01, yn (1z, y2) (ar,, ys)
(a) fitor-itting and inference
(b) Architecture and attention mechanism
Figure 1: Left (a): The PFN learns to approximate the PPD of a given prior in the offline stage to yield predictions on a new dataset in a single forward pass in the online stage. Right (b): Training samples {(11, 91 ,i.,..., (I3, ya)] are transformed to 3 tokens, which attend to each other; test samples 24 and JTs attend only to the training samples. Plots based on Müller et (2022).
Real-Worle Inference During inference, the trained model is applied to unseen real-world datasets. For novel dataset with training samples Dtrain and test features Ztest, feeding (Dirains Stest) as an input to the model trained above yields the PPD go [yllxtestt Dtrain) in single forward-pass. passd The PPD class probabilities are then used as predictions for our real-world task. Thus, PFNs perform training and prediction in one step (similar to prediction with Gaussian Processes) and do not use gradient-based learning on data seen at inference time.
Architecture PFNs rely on a Transformer (Vaswani et al. 2017) that encodes each feature vector and label as a token, allowing token representations to attend to each other, as depicted in Figure [b They accept variable length training set Dirain of feature and label vectors (treated as set-valued input to exploit permutation invariance) as well as a variable length query set of feature vectors ((test,m)) } and return estimates of the PPD for each query.
Ftest (z(test,
Tabular Data Amongst other experiments, Müller et al. (2022) demonstrated PPD approximation with PFNs on binary classification for tiny, balanced tabular datasets with 30 training examples. Here, we substantially improve performance and scale up to small datasets with up to 000 data points and up to 10 classes, including imbalance. We discuss detailed changes in Appendix CC2.6]
THE TABPFN: A PFN FITTED ON NEW PRIOR FOR TABULAR DATA
Our TabPFN is a Prior-data Fitted Network (PFN, see Section 2) that fitted on data sampled from a novel prior for tabular data we introduce in Section We modily the original PFN architecture (Mäler et al. 2022) in two ways. i) We make slight modifications to the attention masks, yielding shorter inference times. ii) Additionally, we enable model work datasets with different numbers of features by zero-padding. These architectural enhancements alongside our main contributions compared to Müller et al. (2022) are detailed in Appendix B.2
In the prior-fitting phase, we train the TabPFN once on samples from the prior described in Section To be more precise, we trained 12-layer Transformer for 18 000 batches of 512 synthetically generated datasets each, which required total of 20 hours on one machine with 8 GPUs (Nvidia RTX 2080 Ti). This yielded a single network that is used for all our evaluations. While this training step is moderately expensive, it is done offline, in advance, and only once for the TabPFN, as part of our algorithm development. The same TabPFN model is used for all experiments in this paper. Full details about our TabPFN training are given in Appendix
During inference, the TabPFN approximates the PPD for our dataset prior, i.e., approximates the marginal predictions across our spaces of SCMs and BNNs (see Section 4. including a bias towards simple and causal explanations for the data. In our experiments, we present predictions for a single forward pass of our TabPFN, well as predictions that ensemble 32 forward passes of datasets modified by a power transformation (applied with probability 0.5) and rotating the indices feature columns and class labels (see Appendix C.2.6 for details).
Since we already rotate the indices of feature columns and class labels while training the TabPFN, it can in principle learn representation that is invariant to both of these rotations. We expect that, using longer training time larger Transformer, TabPEN would learn these invariances and make these rotations obsolcte.
3

---

Published as a conference paper at ICLR 2023
(a) A BNN
(b) An SCM
(c) SCMs sampled from the prior
Figure 2: Overview of graphs generating data in prior. Inputs mapped to the output y through unobserved nodes 2. Plots based on Müller et al. (2022).
A PRIOR FOR TABULAR DATA
The performance of our method crucially depends the specification of suitable prior, as the PFN approximates the PPD for this prior. Section outlines outuinees a fundamental technique for our prior: we use distributions instead of point-estimates for almost all of our prior's hyperparameters, Section 4.2 motivates simplicity in our prior, while Sections 4.3 and 4.4 describe how we use Structural Causal Models (SCMs) and Bayesian Neural Networks (BNNs)
fundamental mechanisms to generate diverse data in our prior. Since our SCM and BNN priors only yield regression tasks, we show how convert them classification tasks in Section 4.5 We describe additional refinements our prior to reflect peculiarities of tabular data (correlated and categorical features, exponentially scaled data and missing values) better in Appendix C.2
4.1 FUNDAMENTALLY PROBABILISTIC MODELS
Fitting model typically requires finding suitable hyperparameters, e.g., the embedding size, number of layers and activation function for NNs. Commonly, resource-intensin searches need to be employed to find suitable hyperparameters (Zoph and Le 2017 Feurer and Hutter 2019). The result of these searches, though, is only a point estimate of the hyperparameter choice. Ensembling over multiple architectures and hyperparameter settings can yield a rough approximation over these hyperparameters and has been shown to improve performance (Zaidi et al.) [202 Wenzel al. 2020). This, however, scales linearly cost with the number of choices considered.
distribution
In contrast, PFNs allow us to be fully Bayesian about our prior's hyperparameters. By defining a probability distribution over the space of hyperparameters in the prior, such BNN architectures, the PPD approximated by our TabPEN jointly integrates over this space and the respective model weights. We extend this approach mixture not only over hyperparameters but distinct priors: we mix BNN and an SCM prior, each of which again entails mixture of architectures and hyperparameters.
4.2 SIMPLICITY
We base our priors on a notion of simplicity, such as stated by Occam's 's Razor or the Speed Prior (Schmidhuber 2002). When considering competing hypotheses, the simpler one is to be preferred. Work in cognitive science has also uncovered this preference for simple explanations in human thinking (Wojtowicz and Deleo 2020). Any notion of simplicity, however, depends choosing particular criterion that defines simplicity. In the following, we introduce priors based on SCMs and BNNs, in which we implement simplicity as graphs with few nodes and parameters.
4.3
SCM PRIOR
It has been demonstrated that causal knowledge can facilitate various ML tasks, including semi- supervised learning, transfer learning and out-of-distribution generalization (Schölkopf al. 2012) Janzing 2020 Rothenhäusler al. 2018). Tabular data often exhibits causal relationships between columns, and causal mechanisms have been shown to be a strong prior in human reasoning [Waldmann) and Hagmayer 2013 Wojtowicz and DeDeol 2020). Thus, we base our TabPFN prior SCMs that model causal relationships (Pearl) [209 Peters et al. 2017). An SCM consists of a collection 7 = ((21,...,33) ) of structural assignments (called mechanisms): 2; fe(2pAp(0) G2), where
4

---

Published as a conference paper at ICLR 2023
(a) Synthetic datasets
(b) Actual datasets
Figure 3: Each sub-plot shows the combination of two features, each dot represents a sample, color indicates the class label. (a) Two synthetic datasets generated by our causal tabular data prior. Numeric SCM outputs are mapped to classes as described in Section 4.5 (b) Two datasets from our validation datasets: Parkinsons (Left) and Wine (Right).
PAg(O (i) is the set of parents of the node (its direct causes) in an underlying DAG G (the causal graph), fi is (potentially non-linear) deterministic function and 6 a noise variable. Causal relationships in g are represented by directed edges pointing from causes to effects and each mechanism 2; is assigned to node in G, as visualized in Figure
Predictions based on ideas from causal reasoning Previous works have applied causal reasoning to predict observations on unseen data by using causal inference, a method which seeks to identify causal relations between the components of system by the use of interventions and observational data (Pearl 2010: Pearl and Mackenzie) 2018 Lin et al. 2021). The predicted causal representations then used to make observational predictions
novel samples to provide explainability. Most existing work focuses on determining a single causal graph to use for downstream prediction, which can be problematic since most kinds of SCMs are non-identifiable without interventional data, and the number of compatible DAGs explodes due to the combinatorial nature of the space of DAGs. In contrast, our TabPFN considers broad family of SCMs but without any causal guarantees. We skip any explicit graph representation in our inference step and approximate the PPD directly. Thus, we do not perform causal inference but solve the downstream prediction task directly. This implicit assumption of SCM-like processes generating our data can be explained in Pearl's "ladder of causation", an abstraction of inference categories, where each higher rung represents a more involved notion of inference (Pearl and Mackenzie 2018). At the lowest rung lies association, which includes most of ML. The second rung considers predicting the effect of interventions, i.c., what happens when we influence features directly. Our work can be considered as "rung 1.5", similar to Kyono et al. (2020) 202T): we do not perform causal reasoning, but make association based predictions on observational data assuming SCMs model common datasets well. In Figure (8 in Appendix B experimentally show that TabPFN's predictions indeed align with simple SCM hypotheses.
Defining a prior based on causal models To create PFN prior based SCMs, we have to define a sampling procedure that creates supervised learning tasks (i.e., datasets). Here, each dataset based on one randonly-sampled SCM (including the DAG structure and deterministic functions fo). Given an SCM, we sample a set zx of nodes in the causal graph G, one for each feature in our synthetic dataset, as well as one node zy from G. These nodes are observed nodes: values of 2x will be included in the set of features, while values from 2y will act as targets. For each such SCM and list of nodes 2x and Zy. n samples are generated by sampling all noise variables in the SCM n times, propagating these through the graph and retrieving the values at the nodes xx and zy for all n samples. Figure 2b depicts an SCM with observed feature- and target-nodes in grey. The resulting features and targets are correlated through the generating DAG structure. This leads to features conditionally dependent through forward and backward causation, i.c., targets might be a cause or effect of features. In Figure 3 compare samples generated by two distinet SCMs to actual datasets, demonstrating the diversity in the space of datasets our prior can model.
In this work, instantiate large subfamily of DAGs and deterministic functions f. to build SCMs described in Appendix[C.. Since efficient sampling is the only requirement we have, the instantiated subfamily is very general, including multiple activation functions and noise distributions.
5

---

Published as a conference paper at ICLR 2023
Ingest diata
Neighhors
Lagistic
Simple
Cathoodt
ASK1.2
TabPPN
Figure 4: Decision boundaries on toy datasets generated with scikit-learn (Pedregosa et al. 2011
4.4 BNN PRIOR
We also consider a BNN prior as introduced by Müller al. (2022) and mix with the SCM prior described above by randomly sampling datasets during PFN training from either one or the other prior with equal probability. To sample a dataset from the BNN prior, first sample an NN architecture and its weights. Then, for each data point in the to- be-be--geee be-generated dataset, we sample an input z, feed through the BNN with sampled noise variables and use the output y as a target (see Figure 2a), An experimental ablation of this BNN prior to our final mixture of both SCM-based and BNN-based priors in Appendix B.4 demonstrates the strength of our novel
prior.
4.5 MULTI-CLASS PREDICTION
So far, the described priors return scalar labels. In order generate synthetic classification labels for imbalanced multi-class datasets, we need to transform our scalar labels ý to discrete class labels We do so by splitting the values of into intervals that map to class labels:
We sample the number of classes N₂ ~ p( No), where p( N2)) is distribution over integers.
ii) We sample No class bounds B; randomly from the of continuous targets ý. iii) We map each scalar label ji to the index of the unique interval that contains it: y/ f- E,[B, < 2:J. where [:] is the indicator function.
For example, with N, {(-oc, -0.1] (-0.1, (0.5, oo)]. Any is would be mapped to the label 0 if it is smaller than -0.1, to if lies in (-0.1, (0.1,0.5) 0.5) and to 2 otherwise. Finally, we shuffle the labels of classes, i.e. we remove the ordering of class labels w.r.t. the ranges.
classes the bounds B. [-0.1,0.5) would define three intervals
5 EXPERIMENTS
5.1
EVALUATION ON TOY PROBLEMS
We first qualitatively compare TabPFN standard classifiers (without hyperparameter tuning) in Figure 4 The top row shows the moons dataset with noise. The TabPFN accurately models the decision boundary between samples; also, similar Gaussian processes, uncertainties are large for points far from observed samples. The second row shows the circles dataset with noise: the TabPFN accurately models the circle's shape with high confidence anywhere outside the region where samples mixed. The third row shows two classes and features from the iris dataset, while the fourth rows shows two classes and features from the wine dataset (both scikit-learn (Pedregosa et al., 201 in both of these cases, the TabPEN makes intuitive, well well-calibrated predictions.
5.2 EVALUATION ON TABULAR ML TASKS
Now, we turn to an empirical analysis of our method for real-world classification tasks. We compare method against tate-offttheart MI. and AutoMI. methods for tabular classification.

---

Published as a conference paper at ICLR 2023
0.94
14
Mean ROC AUC Rank
10
Mean ROC AUC
0.93
ROC AUC Wins
12
8
9
0.92
10
7
0.91
5
6
0.9
3
0.89
2
0.88
1s
5s
30s
5min
1h
18
58
30s
5min
1h
1s
58
30s
5min
Ih
Given Time Budget
Given Time Budget
Given Time Budget
TabPFN
Auto-sklearn 2.0
Autogluon Cocktail
L
XGBoost
LogReg
Catboost
KNN
SAINT
Figure 5: ROC AUC as function of the time allowed to train & tune methods, on 18 numerical datasets from the OpenML-CC18 Benchmark. We report the mean, mean wins and rank and 95% confidence interval across 5 splits for increasing budgets. The red star indicates performance of TabPFN with 32 permutations (which requires 0.62s on GPU). We report more results in Table [
Datasets test datasets, we used all datasets from the curated open-source OpenML-CC18
benchmark suite (Bischl et al.) 2021) that contain up to 2 000 samples (1 000 for the training split), 100 features and 10 classes. The resulting set comprises 30 datasets. We split these datasets into 18 datasets that contain only numerical features and no missing values, and 12 other datasets that contain categorical features and/or missing values. In the main paper, in Figure [ and Table we limit analysis to the case of numerical datasets without missing values, which development of our TabPFN prior on. Results on all of the 30 datasets are given in Appendix [.I and still show strong aggregate performance for TabPEN, albeit not as strong as for the purely numerical case, due to generally worse performance of TabPFN on datasets with categorical features and/or missing values. We focus on small datasets because (1) small datasets are often encountered in real-world applications (Dua and Graff 2017). (2) existing DL methods are most limited in this domain (Grinsztajn al., (022) and (3) the TabPFN would be significantly more expensive to train and evaluate for larger datasets, limitation detailed in Appendix A
focussed the
Baselines We compare against five standard ML methods and two state-of-the-art AutoML systems for tabular data. As ML models we considered two simple and fast baselines, K-nearest- (KNN) and Logistic Regression (LogReg). Additionally, we considered three popular tree-based boost- ing methods, XGBoost (Chen et al.) 2020), LighrGBM (Ke et al. 2017) and CatBoost (Prokhorenkoval cet al., 2018). For each ML model, we used 5-fold cross-vall validation to evaluate randomly drawn hyperparameter configurations until given budget was exhausted 100 configurations were eval- uated (for the search spaces, see Appendix (.2). We then chose the best-performing hyperparameter contiguration (maximum ROC AUC OVO) and refit on the whole training set. Where necessary, we imputed missing values with the mean, one-hot or ordinal encoded categorical inputs, normalized features and passed categorical feature indicators. As more complex but powerful baselines, we chose two state-of-ie -fre--ofthhearrr the-art AutoML systems: AutoGluon (Erickson et al.) 2020). which combines ML models including neural networks and tree-based models into a stacked ensemble, and skle-r 2.0 (Feurer et al. 2015) 2021) which uses Bayesian Optimization and combines the evaluated models into weighted ensemble We note that previous works have found that DI. baselines do not outperform match the performance of GBDT AutoML methods for small medium-sizer tabular data (< 10,000 samples; while matching GBDT performance on larger datasets) (Borisov) et al. 2021 Grinsztajn al. 2022 Shwarrzz-Ziv and Armon 2022). Furthermore, DL methods such as TabNet. SAINT, Regularization Cocktails, Non-parametric Transformers (Arik and Pfister 2021 Somepalli et al. 2021 Kadra et al. 2021 Kossen al. 202 are evaluated on much larger
Auto-skleam 2.0 optimizes ROC AUC for binary classification and cross-entropy for multi-class classifica- tion (as multi-class ROC AUC not implemented).
7

---

Published as a conference paper at ICLR 2023
LightGBM
CalBoust
XGBoost
ASKL2.0
AutoGilaon
TabPENn...
TabPEN
TabPEN AutoGluon
rank AUC
6,9722
4,9444
6.1944
4,4722
3.8056
2,9444
2.6667
Mean rank Ave.
6,8889
4,9722
6.0556
5.1667
3.8889
3,8889
2,8889
2.25
Mean rank CE
5,7778
5.4444
6,4167
3.1111
4.1389
3.0278
2.0833
Mean AUC OVO
0.9
0.924
0.924+.01
0.929 0096
0.93+
0091
0.932
0088
0.9344
0086
0,934 0084
Mcan Acc.
0.862+0012
0.864+.011
0.866+.011
0.87::014
0.881+001
0.873 0095
0.879+ 0089
0.886-t.0094
Mean CE
0.75:t.039
0.747:10029
0.759:1.04
0.813-.073
0.714:2.014
021
0.7161.019
0.71111.014
Mean time (5)
1.301 (CPU) 37.59 (CPU) 3109 (CPU)
(Tune + Train + Predict) 3280
3746
3364
0.0519 (GPU) 0.6172 (GPU) 3077 (GPU)
Table I: Results on the 18 numerical datasets in OpenML-CC18 for 60 minutes requested time per data-split. = values indicate a metric's s standard deviation. If available, all baselines optimize ROC AUC. TabPFN n.e. faster version of our method, without ensembling, while TabPFN + AutoGluon is an ensemble of TabPEN and AutoGluon. Separate inference times in Table
datasets and often use custom parameter tuning and preprocessing, We still evaluate two prominent (adra et al. 2021) and SAINT' (Somepalli et al. 2021). J.
DL methods: Regularization Cocktail |
Evaluation Protocol For each dataset and method, we evaluated 5 repetitions, each with a different random seed and train- and test split (50% train and 50% test samples; all methods used the same split given a seed). To aggregate results across datasets, we report the ROC AUC (one-vs-one (OVO) for multi-class classification) average, ranks and wins including the 95% confidence interval and compare to the performance of the baselines with a budget of {30, 60, 300, 900, 3600} seconds f Our TabPFN uses 32 data permutations for ensembling as described in Section 3 TabPFN without permutations, which we label "TabPFN.... in Table
we also evaluate
Results We now present results for the 18 purely numerical datasets without missing values, detailed in Figure Jnd Table Figure shows that TabPFN achieves a dramatically better tradeoff of accuracy and training speed than all the other methods: it makes predictions within less than a second one GPU that tie with the performance of the best competitors (the AutoML systems) after training one hour, and that dominate the performance of tuned GBDT methods. Unsurprisingly, the simple baselines (LogReg, KNN) already yield results with a small budget but perform worst overall. GBDT's (XGBoost, CatBoost, LightGBM) perform better but are still outperformed by TabPFN and the sfate-of-the-art AutoML. systems (Auto-sklearn 2.0, Autogluon).
TabPFN is much faster than methods with comparable performance. In the following, we compare the combined times for training and prediction (and tuning if applicable). TabPFNn... requires 1.30s on CPU and 0.05s on GPU on average to predict for one d performing comparably to the strongest baselines at five minutes; thus yields a 230x speedup on CPU and 5 700 speedup using a GPU. We ignore computational development costs of each method, see Appendix F.5 for why.
We would like to emphasize that the discussed results are aggregate results across datasets, and that no classification method, including TabPFN, performs best on all individual datasets. Indeed, there do exist datasets for which TabPFN is outperformed even by default baselines. Generally, TabPFN is less strong when categorical features or missing values present. Appendix [B. presents results for all 30 test datasets from the OpenML-CCC18 Benchmark, including 12 with categorical features and/or missing values. Appendix B.5 presents a more detailed overview of the results for different kinds of datasets, including on an additional 149 validation datasets from OpenML, confirming TabPFN's strong performance for purely numerical datasets. We also show per dataset results for each of the 179 datasets (our 30 test datasets and the 149 validation datasets); these demonstrate that, while TabPFN generally does better for datasets with numerical features, there also exist several purely numerical datasets for which TabPFN does not perform better than the baselines, as well as categorical datasets for which it does clearly perform best.
3 'The implementation of Regularization Cocktails does not currently support optimization for multi-class AUC ROC. We optimized for cross-entropy, which performed better than balanced accuracy (the default).
*When comparing methods to each other for given time budget Figure we drop methods that take more than 200% of the requested time budget; some methods also do not use their full budget.
"This includes overhead from conservative scikit-learn interface. We assume that our trained model is in
(GPU) memory already, which otherwise required additional 0.2s for us.
8

---

Published as a conference paper at ICLR 2023
OpenML-AAttoML Benchmark Additionally, we evaluated our TabPFN the small (< 1 000 training samples, 100 features, 10 classes) datasets of the OpenML-AutoML Benchmark, for which externally-validaated performance numbers are available. We use the setup provided by the OpenML- AutoMI. Benchmark, i.e.: metrics, official evaluation seripts and pre-released evaluations for an even wider range of AutoML. baselines. Using only an average of 4.4 seconds per dataset CPU, compared to 60 minutes for the baselines, TabPFN outperformed all baselines in terms of mean cross-entropy, accuracy and the OpenML Metric
single
O
We provide detailed results in Appendix B.2
In-depth analysis of TabPFN predictions We evaluated our model predictions in a plethora of ways to provide extra insights, such as the inductive biases of our method, see Appendix [B.3] We confirm that TabPFN learns to make predictions biased towards simple causal explanations, as detailed in B.3.1 while GBDT methods do not share this inductive bias. We also evaluate our method's invariance to feature rotations and robustness to uninformative features in Appendix [B.3.2 we observe that TabPFN is especially strong compared to baselines when datasets do not contain categorical features or missing values.
and B.3.3 In Figure 0
Ensembling We observe that TabPFN performs best on different datasets than our baselines, i.e., the per-dataset correlation of normalized ROC AUC scores between TabPFN and the strong baselines is lower than between the strong baselines, visualized in Figure [I] in our Appendix. This is likely due to the novel inductive biases of our approach, which lead to distinct predictions. Ensembling predictions is more effective when the considered methods make fewer correlated errors (Breimann) 2001), which encourages the use of more diverse strategies, making TabPFN an ideal candidate for ensembling with baseline methods (Wu et al. 2021). Also, TabPFN is evaluated so quickly, that predictions are almost free, compared the long runtimes of the baselines. To demonstrate its potential for ensembling, we include an entry "TabPFN + AutoGluon" in Table [ generated by averaging the predictions of TabPFN and AutoGluon; this strongly outperforms all other methods.
Model Generalization The PFN architecture accepts datasets of any length as input. However, when training our model the synthetic prior-fitting -ort-fitting phase, we limited synthetic data to a maximum size of 1 (24, as prior-fitting becomes more expensive with larger datasets. Nevertheless, we wondered: Would TabPFN generalize to larger training set sizes that were never seen during training? To test this, we used collection of 18 datasets from the OpenML-AutoML Benchmark (see Table [Tp from which we selected 000 samples each. We evaluated TabPFN on up to 000 training samples, using 5 random data splits and 5000 test samples. Surprisingly, our models generalize beyond sample sizes seen during training, as shown in Figure| 10 in Appendix I. B.I
6 CONCLUSIONS & FUTURE WORK
We have shown how a single Transformer, the TabPFN, can be trained to do the work of a full AutoML framework for tabular data and can yield predictions in 0.4 seconds that are competitive with the performance that the best available AutoMI. frameworks achieve in to 60 minutes. This slashes the computational expense of AutoML, enabling affordable, green, AutoMI..
The TabPFN still has important limitations: the underlying Transformer architecture only scales to small datasets as detailed in Appendix A our evaluations focused classification datasets with only up to 1 000 training samples, 100 purely numerical features without missing values and 10 classes, which motivates work on (1) scaling up large datasets. Our in-depth analysis of the inductive biases of TabPFN (see Appendix B.3). points towards extensions for (2) improved handling of categorical features. (3) missing values and (4) robustness to unimportant features. Also, our work motivates multitude of exciting follow-ups regarding (5) integration of TabPFN into existing AutoML frameworks; (6) ensembling to continue making improvements given more time; (7) dataset- dependent choices of the prior; (8) generalizations to non-tabular data and (9) regression tasks, as well as studies of the TabPFN regarding the dimensions of trustworthy AI (e.g.. (10) -uitt--disttitttion robustness; (11) algorithmic fairness, (12) robustness to adversarial examples, and (13) explainability). The almost instant satte-of-the-ate art predictions of TabPFN are also likely to give rise to (14) novel exploratory data analysis methods, (15) novel feature engineering methods and (16) novel active learning methods. Finally, our advances in causal reasoning warrant follow-ups on (17) approximating the effects of interventions and counterfactuals considering a distribution of SCMs.
'The OPenML. metric evaluates binary classification using ROC AUC and multiclass using Cross Entropy.

---

Published as a conference paper at ICLR 2023
7 ETHICS STATEMENTS
In terms of broader societal impact of this work, we do not see any foreseeable strongly negative impacts. However, this paper could positively impact the carbon footprint and accessibility of learning algorithms. The computations required for machine learning research have been doubling every few months, resulting in large carbon footprint (Schwartz et al.) 2020). Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers to apply these methods. The decreased computational time shown by TabPFN translates to reductions CO2 emissions and cost, making it available to audience that does not have access to larger scale computing.
As the TabPFN provides highly portable and convenient way of building new classifiers that work in real-time, it is likely to increase the pervasiveness of machine learning even further. While this can have many positive effects on society, such as better personalized healthcare, increased customer satisfaction, efficiency of processes, etc, it will also be crucial to study and improve the TabPFN under the lens the many dimensions of trustworthy Al other than computational sustainability, such algorithmic fairesss, robustness to adversarial examples, explainability, and auditability. We hope that its foundation in causal models and simplicity will allow possible avenues for work along these lines.
REPRODUCIBILITY
Code release In an effort to ensure reproducibility, we release code alongside our pre-trained TabPFN and notebooks to reproduce our experiments at https://github.. wtt../o///aa/maaaooooooooooooooc automl/ TabPFN.
Application to public benchmarks In our work, we evaluate TabPFN to publicly available bench- marks: the OpenML-CC18 Benchmark and the AptnML Benchmark. This ensures, that dataset choices not cherry-picked to method. Also, for the Open ApttML Benchmark, use official baseline results and evaluate our method using the evaluation scripts published for this benchmark
Availability of datasets All datasets used in our experiments are freely available at OpenML org (Vanschoren al.) 2014), with downloading procedures included in the submission. Further details on the datasets used can be found in Section F3
Online resources We created a Colab notebook, that lets you interact with our scikit-learn interface at htttps://colab....../ http://cccaabbbbrrr..a.eeaa coopearrch.goolee...o.oo com coom/drivee httdddiiiee///o/ / 1AtMV FLL Puspssssaattt-spss-s-aaaiiii
We created another Colab, where our evaluation and plots on 179 test and validation datasets can be reproduced easily, [attps://colab.. wwwoaaaa.rreearc... google. com/ drive/ (UUGaa::DDSsyybbcde>>xsvtaaaa
We also created two demos. One to experiment with the TabPFNs predictions (https) //huggingfaace. co/
and one to check cross- validation ROC AUC scores on new datasets (htt http:///hhggggcgggggeee co wtt.aacceccpppcceeeeeeeeeo
TabPFNEva vation). Both of them run weak CPU, thus it can require little bit of time.
Details of training procedures for TabPFN and baselines Details shared in the training procedure of all our Transformer models can be found in Appendix [F] An overview of the hyperparameters used for running the TabPFN and our baselines can be found in Tables 5 and respectively.
ACKNOWLEDGMENTS
Robert Bosch GmbH acknowledged for financial support. This research was supported by the Deutsche Forschungsgemminschatf (DFG, German Research Foundation) under grant number 417962828, the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and TAILOR. a project funded by EU Horizon 2020 research, and innovation programme under GA No 952215. We acknowledge funding through the
Available bttp://gsttti.ccommmmmmm/mummmem
10

---

Published as a conference paper at ICLR 2023
European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded the European Union
REFERENCES
S. Arik and T. Pfister. TabNet: Attentive interpretable tabular learning. In Q. Yang. K. Leyton- Brown, and Mausam, editors, Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI"2I), pages 6679-6666 6687. AAAI Press, 2021.
I. Beltagy, M. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05155 [cs.CL], 2020.
B. Bischl, G. Casalicchio, M. Feurer, P. Gijsbers, F. Hutter, M. Lang, R. Mantovani, J. van Rijn, and J. Vanschoren. OpenML benchmarking suites. In J. Vanschoren, S. Yeung, and M. Xenochris- tou, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
V. Borisov, T. Leemann, K. SeBler, J. Haug. M. Pawelezyk, and G. Kasneci. Deep neural networks and tabular data: A survey. arXiv:21 [cs.LGG). 2021.
L. Breimann. Random forests. Machine Learning Journal, 45:5-32, 2001.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, Proceedings the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' 16), pages 785-794. ACM Press, 2016.
Chen, S. Kornblith, Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. arXiv:2006.. 10029v2 [cs.LG], 2020.
M. Chui, J. Manyika, M. Miremadi, N. Henke, R. Chung, P. Nel, and S. Malhotra. Notes from the Al frontier: insights from hundreds of use cases, 2018.
Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://rcchie
N. Erickson, Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv:2003.06505 [stat.ML), 2020.
M. Feurer and Hutter. Hyperparameter Optimization. In F. Hutter, L. Kotthoff, and J. Vanschoren, editors, Automated Machine Learning: Methods, Systems, Challenges, chapter 1, pages 3-38. Springer, 2019. Available for free at http://automllorg/book.
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, Blum, and Hutter. Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett, editors, Proceedings of the 28th International Conference on Advances Neural Information Processing Systems (NeurIPS' 15), pages 2962-2970. Curran Associates, 2015.
M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-skleamn 2.0: Hands-free AutoML via meta-learning MxXiv:2007004074 [cs.LG), 2021.
J. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.

---

Published as a conference paper at ICLR 2023
L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on tabular data? arXiv:2207.0888 [cs.LG], 2022.
D. Janzing. Causal regularization, In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances Neural Information Processing Systems (NeurIPS 20). Curran Associates, 2020.
A. Kadra, M. Lindauer, F. Hutter, and Grabocka. Well-tuned simple nets excel on tabular datasets. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances Neural Information Processing Systems 211), pages -9924-23941. Curran Associates, 2021.
G. Ke, Q. Meng. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, Lightghm: A highly efficient gradient boosting decision tree. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings of the 30th International Confer- ence on Advances in Neural Information Processing Systems (NeurIPS' 17). Curran Associates, 2017.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR' 2015. Published online: iclrr. cc
J. Kossen, N. Band, C. Lyle, A. Gomez, T. Rainforth, and Y. Gal. Self-attention between datapoints: Going beyond individual input-output pairs deep learning. M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS pages 28742- 28756. Curran Associates, 2021.
T. Kyono, Y. Zhang, and M. van der Schaar. CASTLE: Regularization via auxiliary causal graph discovery. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS' 20), pages 1501-1512. Curran Associates, 2020.
T. Kyono, Y. Zhang, Bellot, and M. van der Schaar. Miracle: Causally-aware imputation via learning missing data mechanisms. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, Vaughan, and Y. Dauphin, editors, Advances in Neurul Information Processing Systems, pages 13806-23811 Curran Associates, 2021.
L. Lin, M. Sperrin, A Jenkins, G. Martin, and N. Peek. A scoping review of causal methods enabling predictions under hypothetical interventions. Diagnostic and prognostic research, 5(1): 1-16, 2021.
I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representations (ICLR' 2017. Published online: ielr. cc
S. Müller, N. Hollmann, S. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR 22), 2022. URL
Published online:
ic cc
V. Nair and G. Hinton. Rectified linear units improve restricted Boltzmann machines. In Fürnkranz and T. Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (/CML./0). Omnipress, 2010.
R. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Computer Science. Springer, 1996.
A. Ng. Feature selection, 11 vs. 12 regularization, and rotational invariance. In R. Greiner, editor, Proceedings of the 21st International Conference on Machine Learning (CMLL04) Omnipress. 2004.
J. Pearl. Causality. Cambridge University Press, 2 edition, 2009.
12

---

Published as a conference paper at ICLR 2023
J. Pearl. Causal inference. Causality: objectives and assessment, pages 39-58, 2010.
Pearl and D. Mackenzie. The Book of Why. Basic Books, 2018.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapau,, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
L. Prokhorenkova, Gusev, A. Vorobev, A. Dorogush, and Gulin. CatBoost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Proceedings of the 31 International Conference on Advances Neural Information Processing Systems (NeurlPS' 18). Curran Associates, 2018.
D. Rothenhäusler, N. Meinshausen, Bühlmann, and J. Peters. Anchor regression: heterogencous data meets causality. arXiv: 880.90229999 [stat.ME], 2018.
J. Schmidhuber. The speed prior: predictions. In J. Kivinen and R. Sloan, editors, Computational Learning Theory, pages 216-228. Springer, 2002.
simplicity measure yielding near-optimal computable
R. Schwartz, J. Dodge, N. Smith, and Etzioni. Green Al. Communications of the ACM, 63(12):54-63, 2020.
B. Schölkopf, D. Janzing, Peters, E. Sgouritsa, K. Zhang, and Mooij. On causal and anticausal learning. J. Langford and J. Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML' pages 459-466. Omnipress, 2012.
R. Shwartz-Ziv and A. Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
G. Somepalli, M. Goldblum, A. Schwarzschild, C. Bruss, and T. Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv:2106.01342 [cs. 2021.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 9929-1958. 2014.
Bojan Tunguz. Xgboost search space tweet. 1572642449302166110 2022. Accessed: 2022-09-28.
J. Vanschoren, J. van Rijn, B. Bischl, and L. Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations, 15(2)49--00, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U. von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings the 30th International Conference Advances in Neural Information Processing Systems (NeurIPS' 17). Curran Associates, Inc., 2017.
M. Waldmann and Y. Hagmayer. Causal reasoning. The Oxford handbook of cognitive psychology. pages 732-752, 2013.
F. Wenzel, J. Snoek, D., Tran, and R. Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems (NeurIPS'2O), pages 6514-6527. Curran Associates, 2020.
Z. Wojtowicz and S. DeDeo. From probability to consilience: How explanatory values implement bayesian reasoning. Trends in Cognitive Sciences, 24(12):981-99) 2020.
13

---

Published as a conference paper at ICLR 2023
Yanzhao Wu, Ling Liu, Zhongwei Xie, Ka-Ho Chow, and Wenqi Wei. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16464-16472, 2021. doi: 10.1109/VVPR4645722022 202.
I. Yeo and R. Johnson. A new family of power transformations improve normality or symmetry. Biometrika, 87(4):954-959, 2000.
M. Zaheer, G. Guruganesh, K. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang Qifan, L. Yang, and A. Amr. Big bird: Transformers for longer sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F, Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances Neural Information Processing Systems (NeurIPS pages 17283- 17297. Curran Associates, 2020.
S. Zaidi, A. Zela, T. Elsken, C. Holmes, F. Hutter, and Y. Teh. Neural ensemble search for uncertainty estimation and dataset shift. In M. Ranzato, A. Beygelzimer, K. Nguyen, P. Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances Neural Information Processing Systems (NeurlP'2II] pages 7898-79 19187711.11 Curran Associates, 2021.
B. Zoph and Q V. Neural Architecture Search with reinforcement learning. In Proceedings of the International Conference on Learning Representations (CCR'17), 2017. Published online:
i rc
14

---

Published as a conference paper at ICLR 2023
A LIMITATIONS
The runtime and memory usage of the Transformer-based PFN architecture used in this work scales quadratically with the number of inputs, i.e., training samples passed. Thus, inference on larger sequences (> 100 000) is hard on current consumer GPUs. A growing number of methods seek to tackle this issue and report similar performances while scaling linearly with the number of inputs (Zaheer et al. 2020; Beltagy et al. 2020). These methods can be integrated into the PFN architecture and thus into the TabPFN. Furthermore, and the number of classes to 10 as described in Section 5 While this choice is flexible, the precise TabPFN that we fitted cannot work with datasets that go beyond these limits. We also focused the development of TabPFN to purely numerical datasets without missing values, and while they be applied to datasets with categorical features and/or missing values, their performance is generally worse. We hope to tackle this problem in future versions of TabPFN by modified architecture and prior. Finally, we did not consider the existence of many uninformative features in our prior, leading performance degradation when such features are added; we hope address this issue in future versions of TabPFN. While baseline models (except Gaussian Processes) fit and predict in separate steps, TabPFN performs both at the same time. Thus, baseline models often faster than TabPFN in terms of pure inference time (see Table
our experiments we limit the number of features 100
B ADDITIONAL RESULTS
B.1 DETAILED TABULAR RESULTS
Has Categorical Features
Has Nans in Features
Multiclass
TabPFN
Autosklearn2
Autogluon
Catboost
XGB
LGBM
False
True
Reg. Cocktail
0.25
0.5
0.75
0
0.25 0.5
0.75
0 0.25 0.5
0.75
Normalized ROC AUC
Normalized ROC AUC
Normalized ROC AUC
Figure 6: Normalized ROC AUC performance on datasets from the OpenML-CC18 Benchmark, divided by dataset characteristics. For each plot, we split the datasets into two groups. Left: Orange bars indicate the performance on datasets that have categorical features. Middle: Orange bars indicate datasets that contain missing values. Right: Orange bars indicate multiclass datasets, while others binary.
In Figure 6 explore how the kind of dataset evaluated affects the performance of TabPFN, compared to our baselines. We find that TabPFN performs much better when no categorical features present. We also find that TabPFN performs better, when missing values are present in the data. This warrants an extension of our prior in future work, to make it more customized towards categorical and missing data. Our method seems work comparably well for binary and multi-class problems.
In addition the results in the main paper in Section 5.2 we report detailed results on all 30 datasets in OpenML-CC18, which are small enough, but might include categorical features and missing values. We show performance over time in Figure 7 and wide range of performance values and per dataset results with a hour time limit in Table
15

---

Published as a conference paper at ICLR 2023
0.9
25
10
0.89
Mean ROC AUCRRank
Mean ROC AUC
ROC AUC Wins
20
0.88
15
0.87
10
0.86
3
0.85
2
0.84
1s
5s
30s
5min
1h
1s
58
30s
5min
1h
18
58
30s
5min
1h
Given Time Budget
Given Time Budget
Given Time Budget
TabPFN
KNN SAINT
Reg. Cocktails LightGBM
Log XGBoost
Catboost
Autogluon
Auto-sklearn 2.0
Figure 7: ROC AUC performance over time on the 30 small OpenML-CC18 including datasets with categorical features and missing values. report the mean, mean wins and rank along with the 95% confidence interval across 5 splits for increasing training and tuning time budgets (Unlabelled ticks: Imin, 15min). The red star indicates performance of TabPFN with 32 data permutations (which requires 0.62s on GPU). We report detailed results with 60 min budget in Table T
B.2 RESULTS ON THE OPENMI-AUTOME BENCHMARK
We evaluate TabPFN using the official benchmarking scripts datasets, splits and baselines results of the AutoML-AutoML Benchmark. The full list of datasets can be found in Table I0 We note that these datasets are disjoint from our evaluation datasets; in fact 3 of these ("credit-g", "vehicle"" and tondd-rrrnuussosseevvieee the OpenML-CCC18 Benchmark as well, while one dataset ("Australian")) was included in our list of 150 meta-validation datasets. The evaluation using another set of benchmarking scripts with previously published train-test splits and baseline results, however, helps to confirm that: (1) Our baselines are well tuned and not outperformed by another method in the extensive OpenML-AutoNe Benchmark; (2) the runtimes of TabPFN are reproducible in a controlled environment provided by the OpenML-AutoMI Benchmark; and (3) TabPFN is not overfit to datasplits
were included in our evaluation datasets from
evaluation metric. While we used a 50-50 train-test split with 5 iterations for our experiments in Table 3 the OpenML-AutoMLL Benchmark uses 10-fold tross-validation which results in 90-10 splits with 10 iterations. Thus in the Open Benchmark, all methods use more training samples than in our experiments the OpenML-CC18 Benchmark, which leads to slightly stronger results.
B.3 IN-DEPTH ANALYSIS OF MODEL BIASES
Previous work by Grinsztajn et al. (2022) empirically investigates the inductive biases of tree-based and deep-learning models. They identify three challenges in developing tabular-specific models. Models must be (1) able fit irregular functions: (2) robust to uninformative features; and (3) preserve the orientation of the data. We perform and extend this analysis in the following section. We consider three model types in our analyses: TabPFN, GBDTs (LightGBM) and NNs (Standard Sklearn Multi Layer Perceptron with hidden dimensionality of 100). We do not seek to make absolute comparisons between these model types, which would warrant tuning of our baselines, but only seek to investigate their qualitative behavior in the following experiments.
B.3.1 FITTING IRREGULAR PATTERNS
We believe GBDT methods tend to learn non-smooth and irregular patterns the targets, while MLPs learn smooth, low-frequency functions, as can be seen in Figure 4 TabPFN learns target functions that are rather smooth, as suggested Figure 4 This is due to our model's prior, which prefers
Available http://gttta..cooommmmmmaaatlllt
16

---

Published as a conference paper at ICLR 2023
Table 2: ROC AUC OVO results on the 30 small OpenML-CCI8 (including datasets with categorical features and missing values) for 60 minutes requested time per dataset and per split. If available, all baselines are given ROC AUC optimization as an objective, others optimize CE. Overall each method got time budget of 150 hours, but not all methods used the full budget. Times for TabPFN refer to times on GPU. TabPEN runs training and prediction in a joint step and does not have any hyperparameter tuning, so only a single aggregate time is shown.
LightBMM
CalBoost
XGBoasi
ASKL20
AutoClace TabPIN,,.
TobPYN
TabPFN -AutoGluon
stale
0.9938
0.9245
0.9929
0.9919
0,9973
0.9958
mfeat-fourier
0.9786
0.9816
0.9803
0.8826
0.9843
0.9767
0.9811
0.9838
breast-w
0.991
0
0.9896
0.9939
0.9933
0.9931
0,9924
0.994
méeat- karhumen
0.9979
0.986
0.983
09975
0.9967
0.9919
0.9978
0.9985
0.6699
09671
0.9688
0.9657
0,9660
0.9722
mfcat zemike
0.9716
09759
0.7355
09812
0.9908
0.9812
0.9623
0.9901
eme
0.7288
0.7256
0.7299
0.7376
0.7331
0.7233
0.7276
0.7336
credit approval
0.9415
052399
0.9422
09406
0,9253
0.5294
credit g
0,7852
0.793
0.7894
0.7891
diabeses
0.8247
0,8383
0.8378
08343
0.8391
0.8412
0.841
0.8427
tie tac-tae vehicle
0.9988
0,9992
0.9543
0,9547
0.9739
0.9992
0.9232
0,9022
0,9282
0.904
0,9416
0.9568
0,9589
0.9538
eucalyplus
0,5979
0.9004
0.1322
0.9204
0,9245
0,9278
analcusdata author..
0.9599
0.9999
0.9997
1
0.9993
1
1
analcadataadmii
0.5461
0.3589
0.5743
0,5752
0.5657
0,5643
0,579
0,5756
ped
0.9301
0,9413
0.9291
0.9331
0,9428
0.9298
0,9383
0,944
pc3
0.8247
0.8265
0,8282
0.830S
0,8373
0.836
kc2
0.8141
0,8323
0.8227
0,8311
0.8212
0.8322
0.8366
0.8321
pel
0.8321
0.36
0.8489
0.8527
0.8578
0,877
0.5761
0,8739
bunknose authentic..
1
blood transfinsion.. ...
0.7403
0,7312
0.7501
0.7364
0.753
0,7549
lpd
0.0917
0.7279
0.7171
0.7212
0.723
0,7412
0.7379
0.75226
gear biodog
0.9126
0.9217
0.9191
0.9247
0.9276
0,9345
0.9336
0,9336
wabee
0.9904
0.9931
0.9904
0.9917
0.9956
0.995
0,9964
0.996
cylinder bands
037757
038782
038718
0
0.8344
0.8336
0.8751
dresses sales
0.5593
0,5696
0.5823
0.7055
0.5507
0,5333
0.5376
0,5509
MiceProtein
0.9997
0.9999
0.998
0.9999
0.99977
0.9999
1
0.9526
0.5992
steel plates fault..
0.9626
0.9655
0.9656
0.9694
0.9666
0,9619
0,9655
0,9687
clamate- model-simu.
0.9286
0.9344
0,9255
0.9291
0.9391
0.9426
0.9415
0.9421
Wins AUC OVO
0
Wins Aec.
2
Wins CE
M. rank AUC OVO
66167
4 469667
54167
4.05
3.7833
4.65
3.7
2.8167
Mene Apc.
6.5333
4.9833
$31833
48667
3.8167
4,5333
36166
2.4667
Mean rank CE
5,7333
5.6
3.4667
58
2.8667
4.6167
3,5333
2,3833
WiNTM AUC vs Tels.
514/21
9/4/17
655119
10K514
134413
48/188
15/7/8
Win/T/L Ace vs Tib.
60V24
9/1220
11XV19
11/2/17
12XV18
6/921
19/3/8
Win/UL CE TooP..
68924
SAV22
8X022
NSV22
200010
1/4/25
25007
Meus AUC OVO
0.884 012
0.89 +- 011
0.8114:0011
0.894: 011
0.895 001
0,891
0.894--001
0 +: 0097
Mena Ace.
0.815 014
08888
0821---013 013 0.821
016
0.33--0112 012
0.82
0.825-t
012
0.834
Menn
0.782 .014
0.767:5.06
0.758-2.047
0.815:
0.7244-015
0.742
0.732
0188
0.721
Predict (s)
Tune- Train
3241
0.0815
3718
0.0168
0.0685
3304
1.224
3127
21.18
0,0519
0.6172
3127
simple SCMs as explanations and thus less irregular decision planes. We note, however, that when many training samples are provided, TabPFN also fits more complex functions.
The tradeoff between complexity and number of training samples is explored in Figure X Here, we evaluate the training set cross-entropy loss on synthetic data generated from random SCMs. The number of training samples and the complexity of the generated data (number of hidden units the data generating graph) is varied.
B.3.2 ROBUSTNESS TO UNINFORMATIVE FEATURES
Tabular datasets contain a large fraction of uninformative features (Grinsztajn et al. 2022p. To evaluate robustness to uninformative features, we add an increasingly large fraction uninformative features to our data and show results in Figure Uninformative features are generated by copying existing features and shuffling their values randomly between samples. We find that TabPFN and MLPs are less robust to uninformative features than LightGBM. TabPFN could be adapted by including more uninformative features in the used prior. In a second experiment we drop an increasingly large fraction of features. We drop these features according to feature importance (ranked by Random Forest), first removing least informative features. We show results in Figure/](an (middle) and observe that the classification accuracy of TabPFN and GBDT is not much affected by removing up to
17

---

Published as a conference paper at ICLR 2023
Table 3: ROC AUC OVO results on the small datasets (5 1 111 examples, 100 features and 10 classes) for 60 minutes requested time per dataset and per split. If available, all baselines are given ROC AUC optimization as an objective, others optimize CE. Evaluation on this benchmark was performed with our previously released TabPFN in order to mitigate test-set overfitting. Mean O-enMM-Metrrc is not shown the table as averaging over mixture Cross Entropy and ROC AUC is problematic (however, TabPFN had the strongest average well).
AutoGluon
TunedRandomFoorest
vehicle
-038816
-0.3412
-0.1849
-042566
-0.4333
-0,2955
mucalpius
-0.6905
-0.7255
-0.6677
-0,7209
-0.7433
-0.7123
40,665
blood transfusion service center
0.7532
0.749
0,7557
0,6579
0,7332
0,7359
0,7593
Australion
0.941
0.9315
0,9111
0,9394
0.9356
0,9382
0.9385
cradit-g
0.7977
0.7891
0.7984
0.8017
0.7838
0.7989
DyenMI. Metric
Wins Act. Wins CF
1
3
2
M. mak OpenMil Metric
5.4
2.6
4.8
6.2
1.6
Moan malk Acc.
2,4
5.7
4.7
4,4
5.7
2.1
Moun Aze.
0.793 001
0.763.1 052
0.775
0.763:1.039
0.761
0.784
0.794.1003
0.537:
0.5024.050
0.545
0,499
(CPU)
Log. Regr.
LightGBM
Tabular PFN
Bamples
90
70-
0.025
110
130
150
-0.14
0.04
0.020
6 170
190
E 210
0.015
230
-0.12
0..2
250-
270
290
0.010
13
50
90
130
170
210
250
5
13
50
90
130-
170
210-
250
13
50
90
130
170
210-
2500
Complexity
Complexity
Complexity
Figure 8: Mean training set uncertainty (cross-entropy loss) on synthetic data generated from random SCMs. The number of training samples is varied on the y-axis and the complexity of the generated data (number of hidden units the data generating graph) on the x-axis. The cross-entropy mean averaged across the 100 samples and 000 SCMs for each point.
30% the features, but constantly diminishes. The MLP, which less not robust to uninformative features, performs even better when the 20% least informative features are removed.
We use our testing tasks from the OpenML CC-18 18 Benchmark, and, to simplify analyses, we drop multiclass datasets and datasets that contain more than 50 features (as adding 100% more features to a dataset with more than 50 features yields more than 100 features, which TabPFN cannot handle).
B.3.3 INVARIANCE FEATURE ROTATION
Each feature of tabular dataset typically carries meaning individually, as expressed by column names, such as age or sex. A learning algorithm rotationally invariant in the sense Ng (2004). it is left unchanged when rotation (unitary) matrix is applied to the features of both the training and testing set, i.e. when features are mixed. To remove uninformative features under feature rotation, an algorithm first has to restore the original orientation of the features, and then select informative ones. A rotationally invariant algorithm discards the data orientation and thus has to restore the original orientation internally. Thus, Ng (2004) shows that any rotationally invariant learning algorithm has a worst-case sample complexity that grows at least linearly in the number of irrelevant features.
18

---

Published as a conference paper at ICLR 2023
0.88
0.875
0.88
False
0.87
0.85
0.87
True
ROC AUC
0.86
0.825
0.86
0.8
0.85
0.85
0.775
0.75
0.84
0.84
0.725
0.83
0.83
0.7
0.82
L-GBM MLP TabPFN
20
40
60
80
100
20
40
60
80
Method
Percentage of features dropped
Percentage features added
Figure 9: Left: Performance of LightGBM, MLP and TabPFN when random rotations are applied the feature space. LightGBM loses most predictive accuracy when rotations are applied, MLP is unaffected and TabPFN performs only slightly worse. Center: Iteratively removing features according their reversed importance rank obtained from random forest leads MLPs initially performing better and overtaking LightGBM performance. TabPFN looses predictive accuracy and performs similar to baselines, when most features are removed. Right: Adding uninformative features leads to performance degradation of MLPs and TabPFN, while LightGBM remains relatively constant.
Figure 9 (left) shows the change in test ROC AUC when randomly rotating our datasets, and confirms that only MLPs are rotationally invariant. GBDT methods are highly sensitive to rotations, while TabPFN is less sensitive, but still performs better when no rotations are applied.
The theoretical results by Ng (2004) and empirical results by Grinsztajn et al. (2022) imply, that TabPFN's diminishing performance when uninformative features are added is linked to it's relative rotation invariance. Adjusting the prior to include more uninformative features could address these results.
We use datasets from our testing tasks from the OpenML. CC-18 Benchmark with numerical features only, since rotating categorical datasets problematic, as some GBDT classifiers treat categorical variables distinctly (e.g. generating embeddings per category).
B.4 ABLATION ON THE SELECTION OF PRIOR MODELS
We perform ablation experiments for a prior based solely on BNNs, SCMs and a mix of both using a hyperparameter to control the sampling likelihood during prior-fitting. The PFNs for each prior fitted using less compute than in our final experiments, thus their scores generally are slightly worse. The BNN prior, similar the one used in Müller al. (2022), provides diminished performance compared to the priors based on SCMs. Additionally, we can see that mixing BNN and SCM prior does not seem make big difference on our test set compared to a pure SCM prior.
BNN
SCM
SCM BNN
Mean CE
0.81 100009
0.771 0.881
0.776+000009 0.88340.003
Mean ROC AUC
0.865+0.007
Table An evaluation of the impact of the prior mixing on the final performance. Our final model was trained in the SCM BNN setting (see Section for details on the priors).
B.5 EXTENDED ANALYSIS ON A LARGER BENCHMARK OF DATASETS
We now provide an extended benchmark where we include an analysis on the additional 149 validation dataset (listed in Table & in order assess the generality of results and better understand
The dataset flags was removed enough splits could be generated by code.
19

---

Published as a conference paper at ICLR 2023
0.79
0.78
ROC AUC Mcan
0.77
0.76
0.75
0.74
0.73
1000
2000
3000
4000
5000
Training samples
Figure 10: Extrapolation performance of our TabPFN to dataset sizes never seen during training. Maximum number samples during training was 1024 (dashed red line). The shading indicates the 95% confidence interval over random data splits. A method that does not generalize, would be expected to flatten at 1024.
KNN
1.00
Log. Regr.
0.75
SAINT
Reg. Cocktail
0.50
LGBM
Catboost
0.25
XGB
LGBM
Autogluon
Autosklearn2
-0.25
Tabular PFN
K L S CXLAAX
Figure 11: Spearman correlation of per-dataset normalized ROC AUC performance (i.e. the ranking correlation of per-dataset normalized ROC AUC scores) between the considered methods. Ordering on the x-axis is the same as on the y-axis. The TabPFN performs well on different set of datasets than the baselines, i.e. the Spearman correlation with the GBDT methods is low, while GBDT and AutoML methods are highly correlated, and thus perform well on the same datasets. TabPFN correlates stronger with DL based methods (SAINT and Reg. Cocktail), which, however, do not perform as well as GBDT methods terms of absolute ROC AUC performance (see)
its strengths and weaknesses. We now also include comparisons with default random forests (RFs), support vector machines (SVMs), default XGBoost, etc. along with their tuned versions after one hour. This evaluation used 5 splits and followed the same experimental setup described in Appendix
shows results for our 30 test datasets (separated into the 18 purely numerical datasets without missing values we focus on in the main paper, and the 12 remaining ones), and Figure [3] shows results for the 149 validation datasets (again split into numerical datasets without missing values and the remainder). These figures demonstrate that on the purely numerical datasets without missing values, TabPFN Pareto-dominates all other methods in aggregate results across datasets, in terms of tradeoffs between ROC and time spent. For categorical datasets and datasets with missing values, it still performs well, but not nearly well for the numerical case without missing values we targeted.
Figure 12
We now also statistical tests. We note that when evaluating classifier, performance dataset is just one data point. In order to apply statistical tests, we need to perform analyses across datasets. The standard method for this critical difference diagrams introduced by Demšar
20

---

Published as a conference paper at ICLR 2023
Numerical
defacit Cathane:
AutoGluon
defact (827)
TabPFN GPU
TabPFM CPU
AutoSkleam2
liner CP IR58)
detsc God. Bcoet
lanze K3N
dclash LCBM
Moan ROC AUC +/-atd of spiits
Halva AutoML
Rog Cockoltle
luvce Logie
0.92
tcp. Begt
Lgg. Rege.
tmar
delach Fonat 120)
Random forea:
CarkBooot
0.91
LGEM
GB
KGB
Teoetk GPU
TeaPTk: GPU
TeaFFk: GRU
Taserh crU
detsut XGB
0.89
Busse XGB
0.87
0.015
0.15
1s
10s
1005
1000s
1h
2h
Time spent for optimize/fit predict
Categorical Missing
(12)
Math
0.85
AutoSkdeam
defuult
defaul CP (RS0)
AutoGluen
Nalve AutoMM
delact Grad. Sccet.
Bascc CWN
Mesnn ROC AUC +/- atd of spiits
TabPFN GPU
TabPFN
CuiBuvet Bag Cadisatis
hinsc LGBM
detsct LGBM
lunce Log. Way:
LGBM
Log. Regr.
CP
Faire Auc (mox hos, torations=100
detsik Rond Forest (ilest 190)
hine Randum forest
0.82
XGB
SVN
0.81
0.80
delautxXGB
lanze X25
MCHM
0.79
0.78
0.01s
0.1s
26
10s
100s
1000s
3.1
Time spent for optimize/fit predict
Figure 12: ROC AUC comparison on the OpenML-CC18 Benchmark. Baselines were tuned for one hour until 10000 configurations were exhausted (Log. Reg and KNN).
2006). Figure 14 performs these statistical tests to compare methods in two regimes: fast runs that require up to 30 seconds and tuned runs that require up an hour. We note again that TabPFN is much stronger on purely numerical datasets without missing features. In this case, for the short runs, statistically significantly outperforms all other baselines. For long runs, it also statistically significantly outperforms all other methods except the AutoML frameworks in 1-vs- comparisons; however, this not visible the figure because of the multiple testing correction applied due the multiple pairwise tests.
21

---

Published as a conference paper at ICLR 2023
Numerical and No Missing Values Validation Datasets (65)
Mcthid
Autordiunn
Reg. Cocktalla
0.82
AstoScicama
detscl Cathose
TabPFN GPU
TabPPN CPU
AutoSkdearn2
0.81
Moan RDC AUC 41- std of spiits
CalBeeat
Bigg Coctiolle
hatve Autcei teeation
AutoGiuaa
detach Fonet 132)
Random forest
XGB
detsil SVN
LGBM
Bisse SyM
OP
1oatth CPu th ars
Nalve Autooli
luteh CPu ehues
luteth CPU oru
boteek I ers 320
Log. Regi:
XGB
0.78
JNNM
0.77
0.14
1s
100s
1000s
1h
optimize/fit predict
Categorical or Missing Values Validation Da tadetts (84)
Hethad
AutoGiuon
Reg. Cockialls
Autrocieam2
AutoSideam2
dctash Calboos:
dctask ride
OP (RDE)
0.86
Nalve AutoMM
Mean BOC AVC 4/ std of splitt
Catsooat
0.85
XGB
Rag. Canidelle
RP
AutoGluon
defact SVN
TabPFN CPU
0.84
TobPFN GPU
Trr9h: CPU 0t are
Teeffh: CPU d are
4)
GB
Teoffh: CPU m are
ToffTh: CPU m ara 120
0.83
GP
Taftth CPU 24 emu
Taofth: CPU 74 ema
Log. Regr.
1aitth CPU 24, 2/oo
lutthh GPU PLon 521
27900
dctauk.c0 xce
LGBM
0.81
0.80
10s spent optimize/fit predict
1000s
Figure 13: ROC AUC comparison 149 validation datasets (see Table 8). Baselines were tuned for one hour or until 10 000 configurations were exhausted (Log. Reg and KNN).
22

---

Published as a conference paper at ICLR 2023
Numerical and Missing Values Validation Datasets (65)
Numerical and No Missing Values Test Datasets (18)
3
2
Fost Baselines
3.87
default SVM
TabPFN
default XGB
TabPFN
default XGB
default LGBM
default (RBF)
default
LGBM
Numerical and No Missing Values Validation Datasets (65)
Numerical and No Missing Values Test Datasets (18)
5
1
3
2
(1h) Bazelines
tuned
TabPFN
tuned LGBM
TabPFN
4.18
2.52
4:23
2.01
tuned LGBM
AutoSkleam2
tuned SVM
AutoSkleam2
4.20
tured GP (RBF)
tuned XGB
tuned XGB
tuned GP (RBF)
Categorical or Missing Values Validation Datasets (84)
Categorical Missing Values Test Datasets (12)
Baselines
356
2.36
3.32
1.75
default (RBF)
TobPFN
default GP (RBF)
TabPFN
328
2.70
3.33
2.42
default SVM
default XGB
default SVM
default XGB
3.00
default LGBM
default LGBM
Categorical Missing Values Validation Datasets (84)
Categorical Miccing Values Test Datasets (12)
Tuned Baseiines
4.25
2.46
4.25
2.25
tuned LGBM
AutoSkleam2
tuned GP (RBF)
AutoSkleam22
tuned SVM
tuned XGB
tuned LGBM
TabPFN
3.50
tured (RBF)
TabPFN
tuned XGB
tuned
Figure 14: Critical difference plots on average ranks with Wilcoxon significance analysis. We show plots for both our test set (OpenMI-CCC1)) and our validation set. We split each into subset of purely numerical datasets without missing values and the rest. We compare to two sets of baselines. Fast Baselines that finish tuning, training and prediction in less than 30 seconds average (even the TabPFN on CPU is within this bound). ii) Tuned Baselines with one hour budget for tuning, training and prediction (TabPEN still requires less than 30 seconds average with the same hardware). We note that in the fast run regime, on numerical datasets without missing values, TabPFN statistically significantly outperforms all other methods.
23

---

Published as a conference paper at ICLR 2023
While the analysis above aggregates across datasets, variation across datasets is large and TabPFN by no means works best on all datasets. All classifiers have worst and best cases, and we be- lieve that it is important to complement our reports of TabPFN's strong aggregate performance with the examples we encountered for which the current version does not do well. We do show results for all datasets one-by-one at https: http://ggttttt.u wwtp////aattmomaaa/eeeeeeaeeeeee /
blob/maingg mwwnaaiiiiiiaa http/mainn/nndiidiie wwt.iiidiiiiiaaaaaie plots.po..
The dataset with the worst relative performance for TabPFN is collins, with results shown in Figure T5| While most other methods obtain a ROC AUC close to, or identical to 1.0, TabPEN only achieves around 98%. We found this to be caused by uninformative features, which we have already shown to affect TabPFN's performance negatively in Section B.3.2] With just the 5 most important features (as judged by a random forest), TabPFN also achieves an accuracy of 1.0. (In the aggregate analysis above, we of course count the original, poor performance of TabPFN, to not cheat.)
collins
Mean ROC AUC +/- std of 5 splits
LOG0
Nahve Automtt
Cathoest
Biiss Airtcchoallmm Aartictelllieeeeagglaao
LOON
3000
Aviidlen
Hed
IOWM
Celece
4//aal di
0.53
0:09-
TappP GPu
TWAPPH CPU
APPS-
Time spent for optimize/fit + predict
Figure 15: Worst result for TabPFN: collins, due to uninformative features. We did not focus on uninformative features in creating TabPFN. With just the 5 most important features (as judged by random forest), TabPFN also achieves an accuracy of 1.0.
TabPFN also works poorly on purely categorical datasets (c.g., dataset sensory, Figure 16a or with many missing values (e.g., dataset meta, Figure l6b)
sensory
meta
Moan ROC AUC +/- std of 5 splits
5
Mean ROC AUC + 300 OF
Habre Astenss
Time spent for optimize/fit predict
spent for optimize/fit predict
(a) sensory, purely categorical
(b) meta, many missing values
Figure 16: Further poor results for TabPFN, on datasets with categorical features and missing values, which we did not focus on in creating TabPFN.
TabPFN works best for purely numerical datasets, with no missing values, e.g., datasets 2ouch2 and vehicle, see Figure I7
TabPFN still works well in several datasets where hyperparameter optimization does not help in the baselines (maybe due overfitting), e.g. Pizza-cuul cutteri and arsen i femallmlll be-bladder.
24

---

Published as a conference paper at ICLR 2023
Touch2
vehicle
Mean ROC AUC +/- std s splits
Mean ROC AUC +1 sta Silt
Lop App.
1000
2.2
Time spent for optimize/fit predict
Time spent for optimize/fit + predict
(a) Touch-2, -2, purely categorical
(b) venicie
Figure 17: Strong results for TabPFN for individual datasets with numerical features and no missing values.
see Figure 18 We attribute this to being Bayesian, having meta-learned not to overfit on small
datasets with simplicity prior using principles from causality.
PizzaCutter1
arsenic-female-blaader
splits
Mean ROC AUC +/- std of 5 spltts
Taltwol Adecliion
puthoot
Leg. Pege
Mean ROC
Time spent for optimlze/fit + predict
Time spent for optimize/fit + predict
(a) PizzaCutteri
(b) arrseni--femaee--baddeee
Figure 18: TabPFN can still yield strong results when hyperparameter optimization does not help (potentially due to overfitting)
It is not the case, however, that TabPFN works great on all numerical datasets without missing values and poorly on all categorical datasets. For example. TabPFN works poorly on the numerical dataset pm1 0 and great on the categorical dataset monks-po cocissspoblleeee em2 see Figure 19 It is therefore necessary to look beyond individual datasets to obtain the entire picture.
C DETAILS OF THE TABPFN PRIOR
C.I SCM PRIOR
The Sampling Algorithm We instantiate a subfamily of DAGs that can be efficiently sampled from by starting with a MLP architecture and dropping weights from it. That is. sample a dataset with features and n samples from our prior we perform the following steps for each dataset:
(1) We sample the number of MLP layers I p(1) and nodes p(h) and sample graph G(Z,
E) structured like an l-layered MLP with hidden size h.
(2) We sample weights for each Edge Esg as W., ~ (-).
(3) We drop random set of edges E É to yield a random DAG.
(4) We sample a set of & feature nodes N₂ and a label node N, from the nodes Z.
(5) We sample the noise distributions p(e) p(p(c)) from meta- distradistiibution. This yields an SCM,
with all f. 's instantiated as random affine mappings followed by an activation. Each 2; corresponds
to sparsely connected neuron in the MI.P.
25

---

Published as a conference paper at ICLR 2023
pm10
monks-problems-2
Sijjds
std 5
S of
Jarve Auturs
Mean ROC AUG #4 SS
Mean
Time spent for optimize/fit + predict
Time spent for optimize/fit predict
(a) pm10. purely numerical, but still poor performance (b) monks-probi pess-proble22. em2, categorical, hut still strong
performance
Figure 19: TabPFN can also yield poor performance on numerical datasets (pm) and strong perfor- mance on categorical datasets (monks -problem2.
With the above parameters fixed, we perform the following steps for each member of the dataset:
We sample noise variables G from their specific distributions.
(2 We compute the value of all 2 E Z with = a( a((()ePPaca Egjz;) + G,).
We retrieve the values the feature nodes N₂ and the output node N, and return them.
We sample one activation function a per dataset from {Tanh, Re ELU, Identity) (Nair and Hinton 2010). The sampling scheme for the number of layers p(l) and nodes p(h) is designed to follow a discretized noisy log-normal distribution, p(e) is noisy log-normal distribution and the dropout rate follows a beta distribution. The full information can be found in Table
C.2 TABULAR DATA REFINEMENTS
Tabular datasets comprise a range of peculiarities, e.g. feature types can numerical, ordinal, or categorical and feature values be missing, leading sparse features. We seek to reflect these peculiarities in the design of our prior as described in the following sections.
C.2.1 PREPROCESSING
During fitr-fitting, input data is normalized to step when evaluating on real data. Since tabular data frequently contains exponentially scaled data, which might not be present during prior-fitiing Johnson 2000). Thus, during inference on real tabular datasets the features more closely match those seen during prior-fitting. We use only training samples for caleulating z-statistics, power transforms and all other preprocessing. We take this preprocessing time into account when reporting the inference time of our method.
and unit variance, and apply the same
apply power scaling during inference (Yeo and
C.2.2 CORRELATED FEATURES
Feature correlation in tabular data varies between datasets and ranges from independent to highly correlated. This poses problems to classical deep learning methods (Borisov al.) (022)) When considering large space of SCMs, correlated features of varying degrees naturally arise in our priors. Furthermore, in real-world tabular data, the ordering of features often unstructured, however adjacent features are often more highly correlated than others. We use "Blockwise feature sampling" to reflect the correlation structure between ordered features. Our generation method of SCMs naturally provides way to do this. The first step in generating our SCMs is generating unidirectional layered network structure which nodes one layer can only receive inputs from the preceding layer. Thus, features in the same layer tend to be more highly correlated. We use this by sampling adjacent nodes in the layered network structure in blocks and using these ordered blocks in our set of features. In Figure 20 visualize the correlations of such generated dataset (right) and compare them
26

---

Published as a conference paper at ICLR 2023
30
20
10
10
15
15
20
30
35
35
40
Figure 20: Feature correlation matrices for real-world ("PC4 Software defect prediction", left) and a synthetic (right) dataset, where brighter colors indicate higher correlation.
real-world dataset (left), demonstrating that our prior yields correlation structures similar to those of real datasets.
C.2.3 GENERATING IRREGULAR FUNCTIONS
In real-world data, some features are consistently more important than others. While a random network weight initialization leads slightly different feature importances, the average effect of input features regresses to the mean when the hidden dimensionality increases. We amplify differences by sampling weight parameter for each input feature and multiplying all outgoing weights by this factor. In the prior, we randomly sparsify connections of the graph. Thus hidden variables and the output node are influenced by fewer parameters, yielding more irregular patterns, as a larger number of parameters once again regresses to the mean. We also extend sparsification to blocks of variables, leading to some groups of variables interacting more strongly. We also extend the way noise variables sampled. Instead of sampling Gaussian noise at each node from the same distribution, we first sample separate noise means and standard deviations for each node and then sample from this distribution. Also, generate non-uniformly distributed input data observed in data: We sample the input variables x (which are propagated through our network), from a mix of distributions, namely the Gaussian, Zipfian and Multivariate Distribution.
C.2.4 NAN HANDLING
We do not have special nan handling built into our model. We replace nan values with zero at test time.
C.2.5 CATEGORICAL FEATURES
Tabular data often includes not only numeric features but also discrete categorical ones. While categorical features should technically not be ordered, practice, they sometimes are, i.c., the categories represent binned degrees of some underlying variable. We introduce categorical features by picking random fraction Peat (a hyperparameter) of categorical features per dataset. Analogous to transforming numeric class labels to discrete multiclass labels, we convert dense features to discrete ones. Also analogous multiclass labels, we pick shuffling fraction of categorical features Psent probability where reshuffle categories. For details, Section 4.5 During prior-fitting. for categorical features of 20%.
C.2.6 DIFFERENCES TO PRIOR WORK ON PENS FOR TABULAR DATA
Prior work by Müller et al. (2022) has demonstrated tabular data classification using PFNs, but was limited to training samples, balanced binary-clasee classifcaisii and 60 features. Here we summarize the most important changes, this prior work:
27

---

Published as a conference paper at ICLR 2023
1) The PFNs for tabular data described by Müller et al. (0022) can only handle balanced binary datasets. In Section we show how to extend the prior to handle imbalanced classes and multi-class classification problems.
ii) We studied pre- proceessing techniques for the TabPFN, which not done all before. This includes outlier removal and power sealing during inference (Yeo and Johnson 2000). During training, also rotate feature indices (based on the assumptions that it should not be informative whether a feature is listed in the i-th or the j-th column, but that features may be listed in groups and that there may be more interactions between features with similar indice and class labels (based on the same assumptions).
iii) We construct ensembles over multiple pre-processings. Specifically, ensemble members differ in the feature column rotation, the class label rotation, and the use of power transform. We only include each combination of the l possible feature column rotations (for & features). possible class label rotations (for j-ary classification problems) and choices of transformation (power transform, none). Even if number of ensemble members is specified that is larger than 2kj, we only use 2kj ensemble members.
iv) We changed the transformer architecture to make faster. We shrank attention matrix sizes from (n m)² 2 to n²² + n m, for n. training points and m inference points.
v) We introduce a novel SCM prior. We compare the SCM, the improved BNN prior (which is more heavily based on the BNN setup of CC.2.6) and SCM + BNN Table 4 This improvesperformance by 2% in this smaller scale setup, which is bigger difference than between the performance of the final TabPFN and all basclines besides KNN and SAINT.
D DETAILS OF THE PRIOR-DATA FITTED NETWORK ALGORITHM
Algorithm
D describes the training method proposed by Müller et al. (0022 for PFNs.
Algorithm I: Prior-fitting of a PFN (Müller et al. 2022)
Input :A prior distribution over datasets p( DD), from which samples can be drawn and the number of samples K draw
Output: A model Qo that will approximate the PPD Initialize the neural network qo:
for j - 1 to K do
Sample D
v:)] ~ PDD:
Compute stochastic loss approximation &g >>>> (- log golys1o, D));
Update parameters with stochastic gradient descent on Vola:
end
E
SETUP OF OUR METHOD
E. TRANSFORMER HYPERPARAMETERS
We considered only PFN Transformers with 12 layers, embeddings size 512, hidden size 1024 in feed-forward layers, and 4-head attention. We used the Adam optimizer (Kingma and Ba| (215) with linear- and cosine annealing (Loshchilov and Hutter 2017). For each training we tested set of 3 learning rates, (.001. .0003,00....000 .0001 J. and used the one with the lowest final training loss. The resulting model contains 25.82 M parameters.
E.2 PFN ARCHITECTURE ADAPTATIONS
Attention Adaption The original PFN architecture Müller al. 2022 uses single multi-heas self- attention module (Vaswani et al. 2017) to compute the attention between all the training examples, well as, the attention from validation examples to training examples. We replaced this, with two
To avoid misunderstandings and give an example, rotation of column indices by 2 positions would change the columns of the X matrix from [11, 22, 73, s.a] [ 77, #1, 22). This is not a rotation of the X matrix just of its column indices.
28

---

Published as a conference paper at ICLR 2023
modules that share weights, one which computes self-attention among the training examples and the other that only compute tross-attention from validation examples to training examples. Conceptually, this is equivalent to the original architecture, except that we'r re using slightly different self-atte atfenttent mask than the original architecture, which allowed all examples to attend to itself (the diagonal 1), in this example:
0
1
1
1
1
1
0
(3)
1
1
1
1
1
For validation examples, we remove the attention to themselves. In terms of the example above:
1
1
1
1
0
1
0
(4)
1
1
1
1
1
0
Information about the state of the current position does still flow through the residual branch, though.
Flexible Encoder Datasets have unequal numbers of input dimensions (features), while PFNs use an encoder layer that accepts fixed dimensional inputs. Here we explain how datasets with different numbers of dimensions can be modelled with a single PFN: We draw the number of dimensions of a dataset during training uniformly at random up to 100. Our encoder changes to accomodate this training and inference with different numbers of features by zero-padding datasets where the number of features is smaller than the maximum number of features K and scaling these features by K. the magnitude stays the same.
k s.t.
E.3 TABPFN TRAINING
We trained our final model for 000 steps with a batch size 512 datasets. That is our TabPFN trained 9 216 000 synthetically generated datasets. This training takes 20 hours on 8 GPUs (Nvidia RTX 2080 Ti). Each dataset had a fixed size 1024 and we split into training and validation uniformly at random. We generally that learning curves tended to flatten after around 10 million datasets and were generally very noisy. Likely, this because our prior generates a wide variety of different datasets.
E.4 PRIOR HYPERPARAMETERS
The hyperparameters of our prior were chosen based on simplicity and our observations on the validation datasets (such as their class distributions or feature correlation strengths). Also, during algorithm development, we evaluated our models on this set of datasets to decide if our developed methods correct and working. Since prior hyperparameters specify distributions and not definite values, they can be chosen over a wide range and resemble the intervals chosen for a random hyperparameter search. The prior distributions we used given in Table 5
F DETAILS FOR TABULAR EXPERIMENTS
Here we provide additional details for the experiments conducted in Section 5 in the main paper.
F.I HARDWARE SETUP
All evaluations, including the baselines, compute cluster equipped with Intel(R) Xeon(R) Gold 6242 CPU 2. 2.80GHz using 1 CPU with up to 6GB RAM. For evaluation using our TabPFN, additionally use RTX 2080 Ti.
29

---

Published as a conference paper at ICLR 2023
Table 5: Overview of our prior hyperparameter distribution. For many features we use Log Uniform distribution with truncated normal noise, which we refer to TNLU(h]j, P. min, round). We sample from it by first sampling mean j and standard deviation o from p. a LogUniform(j.. P)) and then sampling from the resulting truncated normal distribution 0, b in.)). v is rounded to the closest integer, if round is set. The final sampled value then h + min.
TruncNormal(4, o²,
Sampling distribution p(v)
MLP weight dropout
0.9 Beta(a, b), where b ~ Uniform(0.1,5.0.
Choices
Sample SCM BNN Share Noise mean nodes Input feature scaling enabled
Uniform Choice
(True, False]
Uniform Choice
(True, False]
Uniform Choice
(True, False]
Sample y from last MLP layer
Uniform Choice
(True, False]
MLP Activation Functions
Uniform Choice
(Tanh, Leaky ReLU, ELU, Identity)
Blockwise Dropout
Uniform Choice
(Tree, False]
Keep SCM feature order
Uniform Choice
(True, False)
Sample feature nodes blockwise
Uniform Choice
(rree, False]
Max Mean já
Min Mean jı
round
min
MLP #layers
TNLU
True
MLP #hidden nodes per layer
TNLU
130
5
True
Gaussian Noise Std.
TNLU
0.3
0.0001
False
MLP Weights Std.
TNLU
10.0
0.01
False
0.0
SCM #nodes at layer
TNLU
12
1
True
F.2 BASELINES
We provide the search space used to tume our baselines in Table For CatBoost and XGBoost, we used the same ranges as Shwartz-Ziv and Armon (2022) with the following exception: For CatBoost we removed the hyperparameter max _ize since we could not find it in the official documentation. To be maximally fair XGBoost, also tried the search space of quadruple Kaggle grandmaster Bojan Tunguz (Tunguz 2022), which we adapted slightly by using softmax instead of logistic, as we the multi-class setting. XGBoost with this search space performed worse for all considered time budgets than the search space by (Shwartz-Ziv and Armon (022). The search spaces for the KNN, GP and Logistic Regression baselines were designed from scratch and we used the respective implementation from scikit-learn (Pedregosa et al. (011). For CatBoost and AutoSklearn, we pass the position categorical features to the classifier (AutoGluon automatically detects categorical feature columns). We normalize inputs for Logistic Regression, GP and KNN to the range [0. 1] using MinMax Scaling.
F.3
USED DATASETS
To construct and evaluate our method, we used the following four sets of datasets.
First, our meta-test (see Table comprises all datasets in the OpenML-CCI8 benchmark suite (Bis chl et al. 202T)avvalall at [OpeMML. org) with at most 2 000 samples, 100 features and 10 classes, which leaves us with 30 datasets that represent small, tabular datasets.
Second, our meta-validation set (see Tables and comprises 1 150 datasets from OpenML. org Van schoren et al., 2014). For this, we considered all datasets on (OpenML. org and applied the following filtering procedure: We dropped all datasets that are in the meta-test set and all datasets with more than 200 samples, 100 features or 10 classes. We also manually checked for overlaps and removed datasets where the number of features, classes and samples was identical to dataset in the meta-test set. Furthermore, we manually dropped FOREX (since it is a time series dataset) and artificially- created datasets, such as the Univ and Friedman datasets. The remaining neta-validation set then contains 150 datasets. This meta-validation set was used to guide the development of our prior hyperparameters as described in Appendix H..4
30

---

Published as a conference paper at ICLR 2023
Table Hyperparameter spaces for baselines. All, except LightGBM, adapted from Shwartz-Ziv and
Armon 2022
baseline
name
type
log
range
penalty
(I1, 12, none)
LogReg
max_ max_iter iter
int
[50, 500]
fit intercept
cat
(True, False) [e [e-
C
float
KNN
neighbors
int
[1, 16]
params_y. y_sccale
float
[0.05, 5.0]
yes
GP
params_lenge saans_length_scall
float
[0.1, 1.0]
yes
learning rate random_ random_strength 12 12_leafrreg
float
[e [e-5,
yes
int
[1,
float
[0]
yes
CatBoost
float
[0, 1.0]
yes
leaf_estimation iterations
int
[1, 20]
iterations
int
[100, 4000]
learning rate max_depth subsample
float
[e-
yes
int
[1, 10]
float
[0.2, 1]
colsample bytree
float
[0.2,
colsample_by bylevel
float
[0.2,
XGBoost
min_child___ child_weightt
float
[e -e-16 e51
yes
alpha
float
le-16 e²1
yes
lambda
float
[e le-16 16 e²1
yes
gamma
float
[e 16 e²]
yes
D_ o_cssimators
int
[100, 4000]
num_leaves
int
[5, 50]
yes
max_depth
int
[3, 20]
yes
learning_ran
float
[e
estimators
int
50, 2000
LightGBM
min_
float
[e ea1
yes
reg alpha
float
[0, le-1, 2,
100]
yes
reg lambda
float
[0, le-1, 1, 10, 20, 50, 100]
yes
subsample float
[0.2, 0.8]
31

---

Published as a conference paper at ICLR 2023
adebnood
ibsym_avc
0.8
passive aggressive
6.6
random, forest
Iblinesr_pe
5gd
eatre trees
gaussian
breaste
mitat tomuner
CMCC
ceeeeraane vibcc
orerzes Ticatmoni
climato modes demulation crasres
contrroe www.saa.............
intett mottt-maapp
Ensemble weights of classifiers used in AutoSklearn baseline for each dataset. Ensemble weights are averaged across 5 splits for each dataset in the OpenMI-CCC18 Benchmark after one hour of training.
Figure 21:
Third, subset of 5 datasets from the OpenML-AutoML Benchmark, comprises all datasets from the OpenMI-AuttOML Benchmark with at most 111 samples, 100 features and classes. This given in Table TO Due to the 10-fold cross-validation in the OAenML-AUtoMLL Benchmark this is identical the setup for our meta-test and meta-validation datasets above, where we used up 000 samples split 50-50 into training and test.
Fourth, our meta-generalizztion set, described Appendix comprises larger datasets from the OpenML AutoML Benchmark.
F.4 MODEL GENERALIZATION
For testing the TabPFN performance on longer sequences, as described in Section I0 of 18 datasets from the OpenML AutoML Benchmark that contain least 10 000 samples. The list of datasets used can be found in Table II For this evaluation, datasets with more than 100 features are limited to the first 100 features. When more than 10 classes are contained in the datasets, samples with any but the first 10 classes are discarded.
we used set
F.5 DETAILS ON TIME COMPARISONS
Time comparisons refer to combined fitting, tuning and prediction; see Table for the times split into tuning/fitting and prediction. The time taken for each baseline and TabPFN does include the one-time cost of development of each method. Thus, for AutoML baselines meta-learning cost was not included (e.g. Auto-Sklearn pipelines meta-learning. which involved running hyper-parameter search for 24 hours on 140 datasets (= 3360 CPU hours) (Feurer et al.) 2021)). For GBDT methods the manual time that went into defining suitable hyperparameter spaces and the manual crafting of algorithms that perform well on tabular datasets is hard to measure and was not included. For TabPFN the prior-fittie fitting phase (which is part of our algorithm development: i.e. developing ideas, writing code, trying ideas) is not included. This is fair to all methods, as these costs are not on the user side and are amortized over time.
32

---

Published as a conference paper at ICLR 2023
Table 7: Datasets used for the evaluation. These include all 30 datasets in the OpenML-CC18 benchmark suite with at most 2 000 samples, 100 features and 10 classes.
Name
#oett
ACot.
Allst.
aClosa
#Naas
Minor. Class Size
OpenMI ID
ballnee-scale
625
499
11
mffat-fourrir
77
200
onfar-karhmuen
65
2000
10
200
16
mfcat morphological
2000
10
200
18
mfeat zemike
18
10
200
22
eme
credit-approval
16
10
690
67
307
29
credit-g
21
14
1800
300
31
diahetes
268
vehicle
19
846
199
54
cucalyprus
20
730
448
105
188
anicuddatawwuh
71
analeunatti
pc4
38
1458
178
1049
38
1563
160
1030
Tanknote-untent.....
1872
610
1462
bloxd-tranctusioon....
748
178
1464
11
gear-hindeg
wdhe
31
569
212
1510
sylinder-burds
40
22
540
228
6332
MiceProtein
1728
40075
Raal-plates--ult
28
1941
40982
madaat--mdde-simml
33

---

Published as a conference paper at ICLR 2023
Table 8: Meta- Meta-Datasets used for developing the prior.
Name
#iferr
BCar.
Plust.
ICClass.
INaNs
Milor. Class Size
OpenMIl ID
breast cancer
10
10
256
2
13
bollc
27
20
368
2
1927
136
25
dermatology
366
20
35
glass
10
214
habemman
306
2
0
81
45
e
heart-h
14
294
782
106
51
heun stutlo
14
270
0
120
53
hepatilis
20
167
32
55
168
lonosphere
351
126
59
iris
150
50
tlas
heyes-ooth
160
329
monks problems-1
256
0
278
333
monks groblems
206
334
monks-grotlems-5
SPECT
267
2
55
336
SPECTE
45
349
95
337
grab-damage
synthetic
100
prin, onbe
200
100
446
atalcandata Jewsant Jowsatt
264
19
450
inish
222
malcaadata hoodwwammll
maacaadana_eeveeee reviewer
379
1418
360
backache
32
27
180
2
0
25
463
pron. synth
profh
10
672
1200
224
470
atalcandata germangas
400
100
455
binmed
sleepdatt
diggle. table_n2
310
9
604
raffsa, Indota
11
308
2
222
117
atoleandata vineyard
724
muchine epu
7
209
2
56
733
pharyox
11
10
2
74
738
price
serva
747
mellandaaa wildear
748
pmt0
8
500
266
750
wiscomsin
75
meta
22
$28
2
504
737
molcandata aprea3
450
794
analcandtta aprca2
amakaniata apecal
200
disclocare biss
662
2
317
774
bodyfie
15
252
2
124
778
cleveland
788
disclosure_xc x_ tampered
4
662
2
327
795
209
2
706
cholestarol
137
ciscuse funds
801
phesen
19
1945
2
1133
972
800
19
418
1239
188
B10
etvurivals
chscase vine2
2
212
814
chattield_
13
235
2
0
93
R20
boston, currected
223
625
semsory
disclosure noniee
662
2
329
R27
MuteMpy
8
4
398
189
831
kdd nino-small
782
466
274
B30
stock
10
462
148
Treas
10
286
120
844
analcanona passexurrey
10
fisherchh
63
854
vinnie
3
380
2
0
185
880
mu284
11
284
142
880
cbscuse geysrl
2
895
chocase consus6
400
2
165
900
195
census-
alsease census3
400
192
90S
chocase consus2
400
2
197
909
plana reiinol
3155
133
915
visunlizing galaxx
pollepes_ usnews
1302
7830
614
930
34

---

Published as a conference paper at ICLR 2023
Table 9: Meta-Datasets used for developing the prior (continued).
Name
#Pear.
OpenMLL
disclosure
662
931
socmob
1156
934
chocase whole
228
20
111
939
molte-trestment
16
527
542
80
940
arsemic female- bladder
80
makcaddan, hallofiame
17
2
1340
2
20
125
966
analaaddta, birthday
4
365
53
968
anolcaidatt draf
collins
500
0
80
987
pmn, Eglass
10
214
0
76
996
jEdii 4.2.4.3
9
369
0
165
1048
mul
403
0
31
Jidit 4.0.4.2
9
274
D
134
1073
PopularKids
1100
AoushingAssistantt
lungeancer, Mmgeaseer_GSt33111 GSE31210
226
D
35
1412
MegaWatil
38
253
2
0
27
1442
PizaCutter
661
0
52
1443
CostaMadrel
296
0
38
1406
CostMetlll
327
D
42
1447
KauggeoChasel
PieCharil
PieCharts
1077
D
134
1453
parkinsons
23
195
2
D
48
1488
planning
52
1490
qualitaive-hankrutt
s2-
10
2
402
D
160
1498
scods
8
210
D
70
1499
thoracie-surger surgcyy-uurery
user knowledge
mnolesule-customers
9
440
D
142
1511
heart long beach
14
200
D
10
1512
robot fuilures
21
1520
vertebra-olummn
Smarphone Bised...
2
180
30
4133
breast cuncer-..
10
10
277
2
D
81
23499
LEDDdisplay....
GAMETES Epistasus...
calendi-DDW
309
D
44
40663
corma
7
7
160
D
70
40669
moda-3-7-10
40680
thyoid-new
40682
solur-Marre
13
13
315
D
21
40686
three019
I0
512
D
238
40690
40693
tokyol
5arit5 plus
11
1124
D
557
40706
cleve
14
9
303
D
138
40710
dieveland-oominall
40711
40981
DiaheieMellitus
98
281
2
99
41430
conference, attendance
7
246
2
D
31
41538
CPMP-22055
41919
TuningSVMs
regime alimentaire
20
17
202
17
41
42172
tri-ctample
5
150
D
50
43261
Touch2
42544
penguins
68
42585
linnic
801
680
342
42638
Table Datasets used for the evaluation in the OP Benchmark. These include all datasets with at most 1 111 samples, 100 features and 10 classes.
Class
Clas
OpenML
credit-g
14
1000
wise
14
17%
71
973
Miroe-tranffiiioe-eevvii serereee-eeiiieeeeeee
74%
178
1464
Australian
35

---

Published as a conference paper at ICLR 2023
Table 11: Evaluation datasets for model generalization experiments.
Name
#Foon.
aCM
Alast.
#Clnss
ANaNs
Minor. Class Size
OpenMI. ID)
KDDCapoP _oppetoney
231
39
50000
8024152
890
1111
airlines
339383
0
240264
1109
bank marketing
17
10
45211
5269
146]
adult
13
48842
6465
11687
1590
covertype
55
45
581012
0
2757
1596
96320
jungle chess, chw.s__pessss 2pes.
44819
4335
41027
APSFailure
171
76000
10728095
1375
41138
79
53
2734000
212620
MiniRooNE
guillemmo
4297
20000
800S
41159
riccuddo
4297
20000
0
5000
4116]
volleen
jonnis
55
83733
1687
41168
helena
28
65196
100
111
41109
36


Reviews:
[{"review_id": "mGW7ZWGcO8", "reviewer": "Reviewer_gGgo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose TabPFN, a transformer-based prior-data fitted network trained to directly approximate the posterior predictive distribution (PPD), i.e., an infinitely large set of data-generating mechanisms. The main contribution of this work is the proposed prior, which is based on structural causal models (SCMs) and Bayesian neural networks (BNNs) in which distributions instead of point estimates are used for the hyperparameters of the prior. TabPFN is trained on 512 synthetic datasets generated from the prior and, once trained, can approximate the PPD for the proposed prior in a single forward pass for a set of target instances given a set of training examples. The result is a transfromer-based model trained to solve synthetically generated classification tasks from a tabular dataset prior in which predictions are very efficient.\n\nExperiments on 30 small-scale classification datasets demonstrate TabPFN's ability to outperform gradient-boosted decision trees (GBDTs) and perform competitively with state-of-the-art AutoML approaches such as AutoGluon in terms of predictive performance while being significantly more efficient.", "summary_of_the_review": "The work takes an interesting and relatively new approach towards small-scale tabular classification problems; with some refining, I think this paper can potentially make a significant contribution to the community.", "strength_and_weaknesses": "Strengths\n---\nTabPFN takes a significantly different approach to tabular classification problems than many current deep-learning approaches, leveraging the transformer architecture and a novel data-generating prior to produce competitive predictions for new unseen tabular classification datasets in a fraction of a second.\n\nComparisons on toy datasets demonstrate TabPFN's ability to produce appropriate decision boundaries that sometimes better reflect the underlying dataset distribution than more traditional machine-learning approaches.\n\nExperiments are performed on a large number (30) of tabular classification datasets.\n\nPredictions for a new unseen test set require less than a second (using GPU).\n\nWeaknesses\n---\nTabPFN is only suitable for small-scale tabular classification tasks (e.g., $\\le$ 2000 examples), thus limiting TabPFN to larger problems.\n\nThe predictive performance of TabPFN and AutoGluon is similar (Table 1), and the combination of TabPFN (with ensembling) and AutoGluon tends to work best. However, TabPFN + AutoGluon seems only marginally better than AutoGluon on it own, questioning the necessity of adding a complex method such as TabPFN.\n\nThe empirical runtime comparisons between TabPFN and the competing methods is also not clearly described. Are the times in Table 1 showing the average total train+test time for the non-TabPFN methods? Also, are the non-PFN methods evaluated using a CPU or GPU? I think it would be useful to see training and test times as separate entities in Table 1, making it more clear where the tradeoffs exist between using TabPFN or a more traditional method like GBDTs or AutoGluon.\n\nI think there could be a clearer discussion more precisely describing the differences between this work and the work by Muller et al. (2022) since this work is heavily inspired by that work. For example, the authors introduce their novel prior in Section 3 describing their use of SCMs and BNNs, but then also describe their use of BNNs follows the work by Muller et al. in Section 3.4. I think it also important to see a comparison to the work by Muller et al. in the experimental evaluation, or at least a discussion about why this comparison is not relevant.\n\nAdditional Comments\n---\nAre there any model size comparison results between TabPFN and competing methods? This information could further help ML practitioners when deciding whether to use TabPFN or a different method for their particular problem.\n\nMinor Weaknesses\n---\nIn Figure 1, the first description in the caption (Left (a)) is describing the right part of the figure, while the second description (Right (b)) is describing the left part of the figure.\n\nFigures 3, 4, and 5 are not color-blind friendly.\n\nFootnote 4 should go after the period.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ehUwBalg3Id", "reviewer": "Reviewer_pf2n", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers classification problems with small tabular datasets (a few thousand samples, and up to ten classes and 100 features). It presents a pre-trained model that solves such problems within a second, outperforming tree-ensembles and AutoML methods that run for an hour. The pre-trained model is a 12-layer transformer-based prior-data fitted network (PFN, introduced in Muller 2022), which uses a prior based on structural causal models. The pre-training uses 18000 batches of 512 simulated datasets. The training process takes 20 hours on an 8-GPU machine. Then the network is applied to 30 OpenML datasets (which takes only 1 sec on a GPU) and provides better ROCAUC results compared to XGBoost, Catboost, Auto-SkLearn and Auto-Gluon, which have a full hour for hyperparameter search. Some of these datasets had known state-of-the-art results (OpenML-AutoML) that were compared for validation.\n", "summary_of_the_review": "This paper presents interesting results for an important problem. To accept it, I would like to see a code that does not require providing the training and test data together, and still obtains the same results. Another experiment that should be done is using the default Catboost and XGBoost hyperparameters, which may provide better results than an hour of search in a large hyperparameter space. Also, the simulated data for the transformer pre-training should be explained in more detail, to assure there was no leakage from datasets later used for the evaluation.", "strength_and_weaknesses": "The paper presents an interesting new approach for the important problem of tabular data classification. The presented technique is limited to small datasets, as mentioned above, due to compute constraints (quadratic in size), but these smaller sizes could still be of interest. It is generally well-written. \nA key weakness is that the pre-trained model requires providing the training and test data together. Even though the test data is not supposed to be used, it is difficult to verify this. Also, the generation of simulated data for the transformer training is not explained well, and it is important to understand how this was done, to make sure there was no leakage from the test datasets.\nAnother experiment that should have been done is using the default hyperparameters of Catboost, XGBoost and LightGBM, which may provide better results than an hour of search in a large hyperparameter space.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Yxgc5PzRjxm", "reviewer": "Reviewer_zM87", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce TabPFN. A transformer-based neural network for classification. Unlike traditional supervised approaches, the network is pre-trained to run classification on unseen datasets.\nThis transformer approach, runs inference on input data to produce a classification result. However, this input data contains the training and prediction datasets removing the need to do pre-training for a given dataset-classification task.\n\nTabPFN is pre-trained on different synthetic datasets for the task of learning to obtain a hypothesis and use the hypothesis to make predictions. The limitations of the approach are clear, the dimensionality of the datasets is constrained by the transformer architecture, and therefore the scalability during training and inference grows quadratically. Nevertheless, this \"ready out-of-the-box approach\" offers state-of-the-art performance when compared to other classifiers. ", "summary_of_the_review": "The authors present a synthetic generator as well as a change in the transformer mask of PFNs to create a strong tabular classificator. They evaluate against state-of-the-art approaches and demonstrate increased accuracy as well as a significant reduction in time.\n\nIn general, I find the approach very interesting. However, I'm trying to understand the significance of the contribution compared to the PFNs paper. Indeed, the pre-training using SCMs is a significant portion of the contribution. However training on BNNs was already presented before and the speed characteristics are coming from PFNs.\n\nAs you mention in Fig. 1 that plots are based on the PFN paper, it would be good to repeat the same for Fig. 2, as there is an overlap.\n\n", "strength_and_weaknesses": "The paper presents an approach based on PFNs, where the authors pre-train their network with synthetic data obtaining great results compared to other state-of-the-art approaches. They present a data generation approach that successfully transfers different hypothesis to the TabPFNs.\n\nThere are however claims such as \"Predictions based on causal reasoning\" that should be better substantiated. As the authors do not demonstrate this is indeed happening. Just because the synthetic dataset comes from an SCM, is no guarantee that TabPFN is doing causal reasoning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I9_dSr1CRRR", "reviewer": "Reviewer_BJ77", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission present tabPFN, an approach for fast learning on small tabular datasets. The approach uses a transformer to featurize the dataset and can learn with a signle pass forward through the network, without any backpropagation, which constitutes the biggest benefit of the approach. The model is an instance of \"Prior-Data Fitter Network\", fitted in a meta-learning style on rich simulated data. These data are generated via either Bayesian Neural Networks or Structural Causal Models, tuned to generate data similar to tabular data, with up to 2000 samples, 100 features and 10 imbalanced classes per dataset.\n\nThe methods is benchmarked on 30 small datasets from the OpenML-CC18 suite, comparing to established baselines, including ligthGBM and XGBoost. Benchmarks show better performance of the contributed model without fine-tuning or hyper-parameter selection, and in particular with very little time budget. *Edit* after rebuttal period, the authors added more datasets.", "summary_of_the_review": "An interesting idea with very strong claims. I am not sure if the empirical evidence is quite as strong as the claims, and the claims should be moderated.\n", "strength_and_weaknesses": "In my eyes, the strength of this contribution is its originality. It contributes a novel incredient to tabular deep-learning research, and has promising benchmarks.\n\nHowever, I worry about the benchmarks and claims of \"much better performance\" than XGBoost, LigthGBM in the introduction. Many practitionners have reported that the claims of tabular deep learning papers do not match their experience (which I found true with my own experimentation on many occasions). I honestly do not know why there is this disconnect, but it is not good press for our community. Along this line, Grinstztajn et al provided a benchmark which they claim is more realistic than many used in the literature. It would have been interesting to include this benchmark.\n\nTabular data have a form of rotational invariance (individual features are important). Is this enforced in the BNN sampling (I can see how it would appear in the SCM sampling).\n\nMulti-class prediction: how is the noise added? A cursory description almost suggests that the link is deterministic: one specific configuration of X leading to a given class\n\nSection 3.5: typo: unqiue\n\nFor lightGBM and XGBoost, the native handling of missing values should be used, rather than imputation\n\nWhich models do AutoML systems choose (to be able to give conclusion that situation various families of models)? This information would be interesting to add to the manuscript.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%