Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers.

## Abstract

Abstract The recently introduced TabPFN pretrains an In-Context Learning (ICL) transformer on synthetic data to perform tabular data classification. As synthetic data does not share features or labels with real-world data, the underlying mechanism that contributes to the success of this method remains unclear. This study provides an explanation by demonstrating that ICL-transformers acquire the ability to create complex decision boundaries during pretraining. To validate our claim, we develop a novel forest dataset generator which creates datasets that are unrealistic, but have complex decision boundaries. Our experiments confirm the effectiveness of ICL-transformers pretrained on this data. Furthermore, we create TabForestPFN, the ICL-transformer pretrained on both the original TabPFN synthetic dataset generator and our forest dataset generator. By fine-tuning this model, we reach the current state-of-the-art on tabular data classification. Code is available at https://github.com/FelixdenBreejen/TabForestPFN.

## 1 Introduction

Tabular data classification is widespread across all industries, leading to an increased interest in the research field of deep learning for tabular data [, , , ]. This type of classification involves classifying a target variable based on a set of attributes, which is commonly stored in tabular format.
Examples of tabular classification include predicting the existence of chronic kidney disease based on blood test results [], estimating the click-through rate of advertisements [], and predicting the stability of pillars in hard rock mines [].
Despite the significance of tabular data, major breakthroughs in AI as shown on vision
[, , ] and language [, ] data have yet to reach the tabular domain. In fact, neural networks are currently outperformed by tree-based machine learning algorithms such as XGBoost [] and CatBoost [] in tabular classification tasks [, , ].

In an attempt to bridge this performance gap, a recent method called tabular prior-data fitted networks (TabPFN) [] introduces an in-context learning (ICL) [] scheme, demonstrating promising results [].
This tabular ICL-transformer can predict test observations with only one forward pass using training observations included in the context. Hollmann et al. [] generate their pretraining data synthetically, with a focus on creating realistic datasets that act as a “prior”. They make their datasets realistic by carefully crafting correlations between features, introducing variety in the feature importance, and by leveraging structural causal models, which can simulate causal relationships.

Even with realistic datasets, it is questionable why pretraining ICL-transformers would improve classification performance on real-world tabular benchmarks.
While pretraining in vision and language has clear purposes by transferring texture and grammar knowledge, explanations from a transfer learning point of view in the field of tabular data do not make sense, as datasets do not share features or labels.

In this work, we claim that the effectiveness of ICL-transformers stems from their ability to form complex decision boundaries. See Figure 1 for an example of such a decision boundary. Intuitively, the complexity is given by how far the decision boundary differs from a simple linear line. Neural networks that train from scratch on tabular data often have overly simple decision boundaries, a phenomenon known as simplicity bias [], while tree-based methods do not suffer from this []. We show that ICL-transformers, in contrast, learn how to create these complex decision boundaries during pretraining. To support our claim, we present three arguments:

- •
ICL-transformers trained on a highly complex but unrealistic dataset generator achieve performance similar to TabPFN. We create this novel forest dataset generator based on decision trees, and we call the ICL-transformer pretrained on this data TabForest. Although the generator outputs orthogonal decision boundaries and independent and identically distributed features, the performance of TabForest on individual benchmark datasets is strikingly similar to TabPFN.
- •
Pretraining dataset complexity is highly correlated with performance. To show this, we vary the parameters of our forest dataset generator that influence the complexity of the decision boundaries. We observe that higher complexity increases the performance.
- •
ICL-transformers can create decision boundaries with high complexity. This property emerges when fine-tuning on a specific dataset, as illustrated in Figure 1. Furthermore, we observe that complexity increases with the size of the model.

Figure: Figure 1: Comparison of decision boundaries for the Electricity dataset (OpenML ID 44156). Axis represent features, colors are predicted class probabilities, and dots are test observations. Fine-tuned variants show a higher complexity score (see section 5.4) than non-fine-tuned variants (Zero-shot).
Refer to caption: extracted/2405.13396v1/figures/boundary_plots.png

In addition to explaining why ICL-transformers work, we push their performance to new heights. TabForestPFN, the ICL-transformer trained on both the TabPFN and the forest dataset, reaches state of the art on two benchmarks [, ]. This ensemble boosts performance by seemingly selecting the best method between TabPFN and TabForest, leveraging the small unexplained differences in performance across various real-world datasets.

We achieve a second major push in performance by switching from zero-shot to fine-tuning. The additional fine-tuning phase improves performance on average, even when the number of observations is less than a thousand. Moreover, both fine-tuning and zero-shot performance highly depends on the context size, where larger contexts yield better results, even far beyond the context size pretrained on. These findings suggest that fine-tuning, using the largest possible context size, should be the default choice when working with ICL-transformers.

In conclusion, ICL-transformers are highly effective tabular data classifiers, capable of creating complex decision boundaries. This new insight advances our understanding of tabular ICL-transformers and opens up new avenues for further research to enhance their performance. With further developments, we anticipate a significant shift in the field of tabular data, moving from tree-based methods towards ICL-transformers.

## 2 Related Works

There are three main branches of tools for tabular data classification: classical statistical methods like linear regression, K-nearest neighbors, and support vector machines []; tree-based algorithms like XGBoost [], CatBoost [], and LightGBM []; and neural-network based methods like this paper. There are several papers benchmarking the different methods [, , , , ]. Overall, tree-based methods stand at the top, with neural networks ranging from inferior to at best competitive.

Nonetheless, there have been numerous approaches that tackle tabular data classification with neural networks. First, we have the class of neural networks trained from scratch: training starts from random initialized weights and is only trained on the data at hand. Research has focused on architectures [, , , , , ], embeddings [, , ], and regularization [, ].

In general, methods training from scratch can struggle because tabular datasets can be small. So, researchers have sought ways to use large volumes of tabular data or to change the training objective. Some employ self-supervised learning [, , , , , ], or closely related transfer learning techniques [, , ]. Others leverage pretrained LLMs [, ] to come to their prediction.

One of those related transfer learning methodologies is in-context learning, a new field sparked by TabPFN []. Currently, there is ongoing research on how to scale TabPFN to encompass more observations and features [, , ]. Amid these developments, we present our work.

## 3 Preliminaries

In tabular classification, we are interested in predicting targets $y\in{\mathbb{N}}$ given features ${\bm{x}}\in{\mathbb{R}}^{d}$, where $d$ is the number of features. We predict $y$ using an in-context learning (ICL) transformer pretrained on a synthetic dataset. The in-context learning allows the transformer to predict targets based on other observations included in the forward pass. In our work, zero-shot refers to one forward pass through the ICL-transformer without any fine-tuning, while fine-tune refers to one forward pass through the ICL-transformer after fine-tuning. Our work builds on TabPFN [], so below we explain their dataset generator and their transformer architecture.

### 3.1 TabPFN Dataset Generator

The TabPFN authors create their own synthetic dataset using Bayesian Neural Networks (BNN) and Structural Causal Models (SCM). To construct a dataset, they first create a BNN or a SCM with random characteristics and with randomly initialized weights. Then they randomly draw an input ${\bm{X}}$ and pass it through the model to generate output ${\bm{y}}$. Their final dataset is given by $({\bm{X}},{\bm{y}})$. See their paper [] for more details.

In their approach, they emphasize their ability to create realistic datasets, and even call their generator a "prior". They chose SCMs specifically because it can capture real-world causal mechanisms. One other aspect they focus on is simplicity, biasing the generator towards less complex input-output relationships. Additionally, they ensure the inputs ${\bm{X}}$ have natural correlation by correlating the features blockwise, and they vary their feature importance by tuning the magnitude of weights belonging to different features. These methods suggest the authors believe creating realistic datasets is important for achieving good performance.

### 3.2 Architecture

In our work, we use the architecture from TabPFN, and make no changes to isolate the effect of the dataset generator. This ICL-transformer has as input the features ${\bm{X}}_{support}\in{\mathbb{R}}^{|S|\times d_{f}}$ and targets ${\bm{y}}_{support}\in{\mathbb{N}}^{|S|}$ from support set $S$ and features ${\bm{X}}_{query}\in{\mathbb{R}}^{|Q|\times d_{f}}$ from query set $Q$. The output is a prediction for ${\bm{y}}_{query}\in{\mathbb{R}}^{|Q|}$. The query set $Q$ represents the observations we want to predict, while the support set $S$ are the observations we base our prediction on. This architecture accepts a fixed number of features $d_{f}$, see also the preprocessing discussed in Appendix A.3.

In this transformer, a token with dimension $d_{token}$ represents all features of a single observation. The creation of support tokens $H_{support}\in{\mathbb{R}}^{|S|\times d_{token}}$ and query tokens $H_{query}\in{\mathbb{R}}^{|Q|\times d_{token}}$ are given by equations 1 and 2.

$$ $\displaystyle{\bm{H}}_{support}$ $\displaystyle={\bm{X}}_{support}{\bm{W}}_{x}+{\bm{y}}_{support}{\bm{w}}_{y}^{T}$ (1) $\displaystyle{\bm{H}}_{query}$ $\displaystyle={\bm{X}}_{query}{\bm{W}}_{x}$ (2) $$

Here, we embed the features linearly using weights ${\bm{W}}_{x}\in{\mathbb{R}}^{d_{f}\times d_{token}}$. Input classes ${\bm{y}}_{support}$ are also embedded using a linear layer with weights ${\bm{w}}_{y}\in{\mathbb{R}}^{d_{token}}$, in which ${\bm{y}}_{support}$ is treated as a float. Biases are used but omitted in the equations for conciseness. In Figure 2, $E_{x}$ refers to the multiplication with ${\bm{W}}_{x}$ and $E_{y}$ represents the product with ${\bm{w}}_{y}$.

After the embedding, we push the tokens through a standard transformer architecture with a special attention mask. Support tokens are only able to see other support tokens, and query tokens can only see all support tokens and themselves, with no attention to other query tokens. This attention mask ensures the prediction of an observation does not depend on which other test observations are included in the forward pass. The complete architecture is given in Figure 2.

Figure: Figure 2: Base ICL-transformer architecture. On the left, dataset features and targets are separately encoded into tokens. On the right, the targets of the query dataset are used as label. In the middle is the ICL-transformer with the attention mask.

## 4 Methodology

The full tabular data classification pipeline is given by: synthetic data generation (3.1 and 4.1), data preprocessing (A.3), architectural design (3.2) and fine-tuning (4.2). In this section, we introduce our new dataset generator and our proposed fine-tuning procedure.

### 4.1 Forest Dataset Generation

Our goal is to create a simple dataset generator that generates datasets with complex patterns to train on, in contrast to the TabPFN [] generator that aims to create realistic datasets. We base our generator on decision trees, because of their ability to create highly complex decision boundaries with minimal computation cost. The idea is to "overfit" the decision tree to randomly generated features and targets. This fitted decision tree is then used as a data-generating process. See Algorithm 1 for the method and Figure 3 for examples of generated data.

Figure: Algorithm 1 Forest Dataset Generation

Our forest dataset generator allows datasets to vary in number of classes, observations, numerical features, and categorical features. There are two parameters that contribute to the decision boundary complexity. The base size is the number of observations used to fit the decision tree; more observations means more places for the decision tree to split on. The tree depth determines how deep the decision tree will go before exiting the fitting algorithm, with higher depth leading to greater complexity.

For every new synthetically generated dataset, we uniformly draw the hyperparameters from the bounds shown in Table 1. Because we have two hyperparameters that influence the complexity, we decided to keep the base size fixed. In the final step of the algorithm, the targets ${\bm{y}}_{2}$ are discretized by uniformly drawing bucket boundaries between 0.0 and 1.0, and assigning a class to each bucket, creating varying degrees of class imbalance.

**Table 1: Hyperparameters for the Forest Dataset Generator**
| Hyperparameter | min | max |
| --- | --- | --- |
| base size | 1024 | 1024 |
| dataset size | 128 | 1024 |
| tree depth | 1 | 25 |
| number of features | 3 | 100 |
| number of classes | 2 | 10 |
| ratio of categorical features | 0.0 | 1.0 |

Figure: Figure 3: Generated forest data. Every box is a generated dataset with its own classes (color) and features (axes). The data clouds look unrealistic: decision boundaries are always orthogonal, and there is no feature correlation. Generated with base size 1024, dataset size 1024, maximum tree depth between 1 and 25, two features, and between 2 and 10 number of classes.
Refer to caption: extracted/2405.13396v1/figures/synthetic_forest.png

### 4.2 Fine-tuning Procedure

In our work, we introduce fine-tuning to the tabular ICL-transformer. When fine-tuning, we like to draw support and query sets from our training data such that the performance generalizes to the test set. This requires careful consideration of dataset splitting. The benchmark datasets already provide us with a training-validation-test split, of which the validation set is used for hyperparameter search. We require an additional validation set for early stopping, so we split the training with ratio 80/20 in training-small and validation-early-stopping.

Every gradient decent step, we randomly draw a 80/20 support and query split from training-small. For validation, we draw the support set from training-small, and draw the query set from validation-early-stopping. Every sample of the validation-early-stopping is seen exactly once, while support samples are randomly drawn with replacement. We use early stopping based on the validation loss, which is calculated after every step.

The early stopping technique can decide that training zero epochs is optimal, which allows us to fall back on the zero-shot performance in case fine-tuning harms the performance. This is especially important when using very small datasets. At the same time, fine-tuning can make use of the full training dataset, while the number of samples zero-shot can use is limited by the maximum support size that fits on the GPU.

## 5 Experiments

In our results we consider four pre-trained architectures, each with a zero-shot and a fine-tuned version:

- •
TabPFN (original) is the original implementation by the TabPFN [] authors, fine-tuned by us. The weights are downloaded from their Github.
- •
TabPFN (retrained) is trained by us on the TabPFN-dataset.
- •
TabForest is trained by us on our forest dataset.
- •
TabForestPFN is trained by us on both the TabPFN-dataset and our forest dataset.

Comparing the difference in behavior and performance allows us to learn the effect of the different synthetic datasets. Training and hyperparameter settings are given in appendix A.4, benchmarks used and results obtained are given below.

### 5.1 Introduction of the Benchmark Datasets

**Table 2: WhyTrees Results. Normalized accuracy for mixed and numerical features as shown in Figure 4.**
|  | Mixed | Numerical |
| --- | --- | --- |
| TabPFN (original) Zero-shot | 0.530 | 0.587 |
| TabPFN (original) Fine-tune | 0.716 | 0.738 |
| TabPFN (retrained) Zero-shot | 0.207 | 0.351 |
| TabPFN (retrained) Fine-tune | 0.731 | 0.750 |
| TabForest Zero-shot | 0.208 | 0.350 |
| TabForest Fine-tune | 0.829 | 0.804 |
| TabForestPFN Zero-shot | 0.402 | 0.570 |
| TabForestPFN Fine-tune | 0.848 | 0.835 |

We show the results of our pre-trained architectures on two benchmarks, tested against publicly available results provided by the authors of the benchmarks. We include all their tested methods and datasets where possible. Appendix A.5 lists all used datasets.

The TabZilla [] benchmark tests 20 algorithms on 176 classification datasets with sizes ranging from 32 observations to over a million. We selected 94 out of 176 datasets, see appendix A.5. The medium size benchmark which we refer to as WhyTrees [] consists of 23 classification datasets with 2923 to a maximum of 10,000 observations. The benchmark is split into 7 datasets with only numerical features and 16 datasets with both numerical and categorical features.

Both benchmarks perform random hyperparameter search on their algorithms. TabZilla runs up to 30 times per algorithm and WhyTrees runs a few hundred times, up to 2500 runs. The ICL-transformers run only on default settings because we noticed little gains in performance when changing the fine-tuning hyperparameters.

### 5.2 Main Results of TabForestPFN

The results on the TabZilla benchmark are shown in Table 3, see Appendix A.6 for alternative presentations. For the WhyTrees benchmark, the comparison of fine-tuned TabForestPFN with the benchmark algorithm is shown in Figure 4, and the comparison with other ICL-transformer variants in Table 2.

**Table 3: Main Results on TabZilla. N. Accuracy stands for Normalized accuracy. Rank compares the relative rank of a method compared to all other methods on that dataset. Only the 5 best methods out of 26 are shown. Full table with all methods is presented in Appendix A.6**
| Models | Rank | N. Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| min | max | mean | median | mean | median |  |
| TabForestPFN - Fine-tune | 1 | 24 | 8.0 | 6.0 | 0.680 | 0.663 |
| TabPFN (retrained) - Fine-tune | 1 | 26 | 8.3 | 7.8 | 0.678 | 0.700 |
| CatBoost | 1 | 22 | 8.9 | 7.8 | 0.676 | 0.663 |
| XGBoost | 1 | 23 | 9.2 | 8.2 | 0.674 | 0.671 |
| TabPFN (original) - Fine-tune | 1 | 25 | 9.3 | 8.0 | 0.670 | 0.677 |
| .<br>.<br>. | .<br>.<br>. | .<br>.<br>. | .<br>.<br>. | .<br>.<br>. | .<br>.<br>. | .<br>.<br>. |

Figure: Figure 4: Main results on the WhyTrees Benchmark. TabForestPFN shows the mean over ten default runs for different fine-tuning seeds, all others use random search over the hyperparameters. See Table 2 for other ICL-transformers.

In these figures and tables, we see that the fine-tuned TabForestPFN outperforms all other methods on TabZilla and matches the performance of XGBoost with a thousand runs on mixed features and a hundred runs on numerical features on WhyTrees. For both benchmarks, we can also see that there is a significant gap between the zero-shot performance of our retrained TabPFN and the original TabPFN. We attribute this phenomenon to the training settings, see Appendix A.4

### 5.3 Similarity of TabPFN and TabForest

Now that we have established the strong performance of TabForestPFN, we take a closer look at the behavior of TabPFN versus TabForest. In Tables 4 and 6, we see that there is no clear favorite between the fine-tuned versions of TabPFN and TabForest: TabForest is the best on WhyTrees, while TabPFN is favored on TabZilla. Furthermore, TabForestPFN only has a marginal improvement over TabPFN on TabZilla, and similarly only a small performance increase over TabForest on WhyTrees.

In Figure 5 we show the performance of TabPFN, TabForest and TabForestPFN on individual datasets from the WhyTrees benchmark. Remarkably, all three methods often show similar performance across datasets, even though TabPFN and TabForest are pretrained on completely different synthetic data. These results suggest the TabPFN’s performance does not come from the fact that the synthetic dataset is a "prior". Instead the results suggest the performance is a property of an ICL-transformer’s learning process.

Furthermore, we see that there are some differences on some individual datasets, such as the eye movements and, to a lesser extent, the bank-marketing and MiniBooNE datasets. Although we cannot explain why these differences exist, it does prompt us to ensemble the synthetic datasets to create TabForestPFN, which seems to mimic the best method of the two.

Figure: Figure 5: Comparison of TabForest, TabPFN and TabForestPFN on 10 datasets of the WhyTrees benchmark. Performance of different ICL-transformers is often similar. All the benchmark algorithms are greyed out. See Appendix A.7 for all WhyTrees datasets and for colored benchmark lines.

### 5.4 Complexity of ICL-Transformers’ Decision Boundaries

In the previous section, we have seen TabForest and TabPFN show similar performance behavior. Now we take a look at their decision boundaries. Repeating the analysis of the WhyTrees’ authors [], we use Electricity dataset (OpenML ID: 44156) to predict a target on two features.

To capture the complexity of the decision boundary, we define the complexity score $V$. We split the feature space into a total of $n$ grid cells where each cell has a predicted probability of class one $p_{ij}$ for grid cell indices $(i,j)$. The complexity score $V$ is defined as the sum of absolute values between neighbor cells:

$$ $V=\frac{1}{n}\sum_{ij}|p_{i+1,j}-p_{ij}|+|p_{i-1,j}-p_{i,j}|+|p_{i,j+1}-p_{ij} |+|p_{i,j-1}-p_{ij}|$ $$

The complexity score represents how fast the prediction changes when moving along the grid.

We plot the results in Figure 1. We see that when fine-tuning, both TabPFN and TabForest can create decision boundaries that are more complex than regular MLPs. The complexity of the decision boundaries was one of the characteristics that explained why tree-based methods outperformed neural networks []. These results suggest ICL-transformers can also create complexity in the decision boundaries. In Appendix A.10 we also show that the complexity scales with the dimension of the ICL-transformer.

In our intuition, the ICL-transformer learns how to create these decision boundaries during pretraining. Different parameters of the network might be responsible for different parts of the decision boundary. During zero-shot, the ICL-transformer can then create the decision boundary out of these parts. During fine-tuning, the effect is stronger, as different parts come to the surface while irrelevant parts are tuned down.

### 5.5 Ablation of the Forest Dataset Generator

In Section 4.1 we discussed two ways to influence the complexity of the forest dataset generator: the tree depth and the base size, which is the number of observations to fit the tree algorithm. We expect the performance of TabPFN to increase when the complexity of the forest dataset generator increases.

In Figure 6 we show the results of pretraining different settings of base size and maximum tree depth on the WhyTrees benchmark. The tree depth is set to 1-25 as the base size changes, and the base size is fixed to 1024 as the tree depth changes. When scaling up the base size from 8 to 32 and the tree depth from 1 to 9, we observe that the performance increases, and stabilizes for higher complexities. We provide figures of the data generated with these lower complexity hyperparameters in Appendix A.9 to give an impression. The correlation between performance and complexity supports our claim that learning complex decision boundaries is the driving force behind performance.

Figure: Figure 6: Ablation of the base size and maximum tree depth parameters of the Forest Dataset Generator. Figure shows normalized test accuracy of TabForest on the WhyTrees benchmark. A smaller model is used (dim: 512 -> 256, layers: 12 -> 8).

### 5.6 Superiority of Fine-tuning over Zero-shot

In the main results, we have seen that fine-tuning performs better than zero-shot. We look at this comparison in more detail. Figure 7(a) presents the performance of TabForestPFN on individual datasets from TabZilla. We can see clearly that fine-tuning strongly outperforms zero-shot when there are more than 10,000 observations. However, fine-tuning struggles on the two smallest datasets, which have four and six validation samples. Overall, fine-tuning outperforms zero-shot on 66% of the datasets. This percentage decreases to 54% for datasets smaller than a 1000 observations, and increases to 84% to datasets larger than a 1000 observations.

Figure 7(b) illustrates the effect of context length on the performance of TabForestPFN on the zero-shot and the fine-tuning task. We see that a higher support size is always better, which is why we set the support size in our paper to 8192, even though we only pretrained on a maximum size of 1024 observations. In conclusion, fine-tuning on the largest possible support size appears to be the most effective approach for ICL-transformers.

Figure: (a) Differences in normalized accuracy of individual datasets from TabZilla. The color red means fine-tuning is the best. The darkest red represents at least 0.20 normalized score points improvement, and dark blue at least 0.20 normalized accuracy points degradation.

## 6 Conclusion

The introduction of TabPFN [] has opened up a new field of in-context learning (ICL)-transformers for tabular data classification. Our research has demonstrated that these ICL-transformers learn to create complex decision boundaries. Furthermore, by fine-tuning and ensembling the pretraining dataset, we achieved performance levels that surpass XGBoost [] and CatBoost [] on third-party benchmarks [, ]. Although tree based-methods can continue to gain performance through hyperparameter optimization, we expect ICL-transformers can match their gains by scaling up the pretraining process.

Despite these advancements, there are still obstacles to overcome for ICL-transformers to fully replace tree-based methods in the realm of tabular data. One major challenge is the performance limitation of ICL-transformers due to GPU memory constraints. Our work uses fine-tuning as a solution to this problem, but it would be valuable to compare this approach to other concurrent research such as prompt tuning [] and context distillation []. Moreover, our research solely focused on classification, although we expect a simple switch from cross-entropy loss to mean-squared-error loss would suffice to tackle regression tasks. Another area that requires exploration is the setting with an exceptionally high number of features [], where the performance of ICL-transformers is unknown []. Lastly, tree-based methods have an edge in explaining the feature importance, and research needs to be done if ICL-transformers can achieve similar interpretability []. Overcoming these challenges will cement the ICL-transformer as the clear successor of tree-based methods.

## Appendix A Appendix

### A.1 Ethics and Social Impact

Improving tabular data classification can provide major benefits to society. From medical to physics applications, better performance can save lives and money. There are, however, also more nefarious applications of tabular data classification, such as fraud risk detections based on ethnics or nationality; population analysis for micro-targeting political ad campaigns; and insurance premium discrimination based on underlying medical conditions.

Our models cannot detect the purpose for which the model is used. In contrast to large language models, our tabular data models take numerical data as input. Ethnicity, gender, or other sensitive information is represented by a class number. This means our models cannot recover the meaning behind the numbers.

A big benefit of this ’numerical anonymization’ is privacy. In our paper, we use synthetic data, which is completely privacy risk free. But even when pretrained on real data, recovering the original data can be extremely hard due to the lack of labels and contextual information.

In light of the above, we decide to publish the model and open access to anyone. We do not know an effective way to create any form of safeguards against misuse, and we would welcome any advice from the research community that addresses this issue.

### A.2 Code Availability

Code is available at https://github.com/FelixdenBreejen/TabForestPFN

The code includes everything: downloading datasets, preprocessing result benchmarks, training the ICL-transformers, pretrained weights on Google Drive, notebooks with analysis, and an easy example to get you started with applying the TabForestPFN to your dataset.

The code is built upon the works of , and . Although this resulted in a Frankenstein monster of a codebase, we have made great efforts to rewrite most of their code to integrate well. The code assumes a single server with access to 1 or more GPUs, with DistributedDataParallel used during pretraining and multiprocessing over different GPUs used during inference.

As currently there is no good tabular code base out there, we recommend anyone that is interested in doing research in tabular data to take a look at ours.

### A.3 Data Preprocessing

Before data is put into the neural network, the data is preprocessed. We use the exact same routine for both synthetic data and real-world data to ensure minimal differences in distribution and summary statistics of the input to the transformer. Algorithm 2 presents the procedure for preprocessing.

Figure: Algorithm 2 Data Preprocessing

Because we fixed the input size of the neural network to $d_{f}=100$ features, we first select a subset of a $d_{f}$ features using scikit-learn’s SelectKBest []. If there are less than $d_{f}$ features, we add zeros to ensure exactly $d_{f}$ features. Following TabPFN, we scale by multiplying by $100/d_{f*}$, where $d_{f*}$ is the number of features after selecting a subset. To be robust to skewness and outliers, we transform the data using scikit-learn’s QuantileTransformer to follow a normal distribution. We make no distinction between numerical and categorical values in all our preprocessing.

In comparison to TabPFN, our data preprocessing follows roughly the same scheme. One change is the use of a quantile transformer, while they use standard input or a power transformer. We consider this a preference, both seem to work fine.

Furthermore, the TabPFN authors like to ensemble outputs of the TabPFN architecture, varying the transformation function between standard input and power transformer. In our paper, we use no ensembling at all.

### A.4 Training Settings

All ICL-transformer architectures, including the original TabPFN, use the same model. The model consists of 12 layers, 4 attention heads, a hidden dimension of 512, and 10 classes as output dimension. Pretraining uses batch size 64, learning rate 1e-4, weight decay 0.00, AdamW optimizer with betas (0.9, 0.95), cosine scheduling, and maximum global gradient norm 1.0. Fine-tuning is performed under batch size 1, learning rate 1e-5, weight decay 0.00, no scheduling, with early stopping, and a maximum of 300 steps. To fit the model on an RTX 3090 with 24GB during fine-tuning, we set the maximum support size to 8192 samples and the maximum query size to 1024. Pre-training uses data generated with maximum support size 1024 and maximum query size 128. We choose these settings because the maximum support size affects performance, but the maximum query size only affects inference speed.

On these settings, we train TabPFN and TabForest for 300,000 steps, while TabForestPFN trains for 600,000 steps. Both TabPFN and TabForest can likely train longer; however, we experienced high instability in performance after 300,000 steps depending on the seed. Sometimes, the fine-tuning ability of the model would completely collapse, and sometimes the performance would slowly overfit. We also chose 300,000 steps because we did not see improvement in fine-tuning performance in the steps afterward. We chose to use 600,000 steps for the combined TabForestPFN, because we believed double the data would allow us to use double the steps, and we experienced no instability on this setting. Running 600,000 steps on one RTX 3090 takes 24 GPU-days.

Choosing 300,000 steps does have a downside. In the results in section 5.2, we saw the retrained TabPFN and the TabForest have poor zero-shot performance. The number of steps therefore is a trade-off: a small number of steps is stable and has high fine-tuning performance, while a large number of steps has good zero-shot performance. Because in this paper we are interested in maximum performance, we prioritize the fine-tuned version on 300,000 steps.

### A.5 Benchmark Metadata

Of both the TabZilla and the WhyTrees benchmark, we show the OpenML[] datasets we use as well as their characteristics. See Table LABEL:table:metadata_tabzilla and Table LABEL:table:metadata_whytrees. The TabZilla table presents the 94 datasets picked out of the 176 total datasets.

From the original 176 Tabzilla datasets, we excluded every dataset that does not have at least one completed run on default settings for every model, which brings the value to 99. Additionally, we exclude four datasets because they have more than 10 classes. The preprocessing code of one other dataset did not run without errors, and so is removed as well. The TabZilla authors did experiment with running TabPFN, but only on 62 datasets with a maximum support size of 3000 samples, so we redo their experiment.

**Table 4: Metadata of the TabZilla Benchmark. Splits refers to the number of cross validation splits.**
| OpenML | Observations | Features | Splits | Classes |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ID | Name | All | Train | Valid | Test |  |  |  |
| 3 | kr-vs-kp | 3196 | 2556 | 320 | 320 | 36 | 10 | 2 |
| 4 | labor | 57 | 45 | 6 | 6 | 16 | 10 | 2 |
| 9 | autos | 205 | 163 | 21 | 21 | 25 | 10 | 6 |
| 10 | lymph | 148 | 118 | 15 | 15 | 18 | 10 | 4 |
| 11 | balance-scale | 625 | 499 | 63 | 63 | 4 | 10 | 3 |
| 12 | mfeat-factors | 2000 | 1600 | 200 | 200 | 216 | 10 | 10 |
| 14 | mfeat-fourier | 2000 | 1600 | 200 | 200 | 76 | 10 | 10 |
| 15 | breast-w | 699 | 559 | 70 | 70 | 9 | 10 | 2 |
| 16 | mfeat-karhunen | 2000 | 1600 | 200 | 200 | 64 | 10 | 10 |
| 18 | mfeat-morpholog… | 2000 | 1600 | 200 | 200 | 6 | 10 | 10 |
| 23 | cmc | 1473 | 1177 | 148 | 148 | 9 | 10 | 3 |
| 25 | colic | 368 | 294 | 37 | 37 | 26 | 10 | 2 |
| 27 | colic | 368 | 294 | 37 | 37 | 22 | 10 | 2 |
| 29 | credit-approval | 690 | 552 | 69 | 69 | 15 | 10 | 2 |
| 30 | page-blocks | 5473 | 4377 | 548 | 548 | 10 | 10 | 5 |
| 35 | dermatology | 366 | 292 | 37 | 37 | 34 | 10 | 6 |
| 37 | diabetes | 768 | 614 | 77 | 77 | 8 | 10 | 2 |
| 39 | sonar | 208 | 166 | 21 | 21 | 60 | 10 | 2 |
| 40 | glass | 214 | 170 | 22 | 22 | 9 | 10 | 6 |
| 43 | spambase | 4601 | 3680 | 460 | 461 | 57 | 10 | 2 |
| 45 | splice | 3190 | 2552 | 319 | 319 | 60 | 10 | 3 |
| 47 | tae | 151 | 120 | 15 | 16 | 5 | 10 | 3 |
| 48 | heart-c | 303 | 241 | 31 | 31 | 13 | 10 | 2 |
| 49 | tic-tac-toe | 958 | 766 | 96 | 96 | 9 | 10 | 2 |
| 50 | heart-h | 294 | 234 | 30 | 30 | 13 | 10 | 2 |
| 53 | vehicle | 846 | 676 | 85 | 85 | 18 | 10 | 4 |
| 59 | iris | 150 | 120 | 15 | 15 | 4 | 10 | 3 |
| 2074 | satimage | 6430 | 5144 | 643 | 643 | 36 | 10 | 6 |
| 2079 | eucalyptus | 736 | 588 | 74 | 74 | 19 | 10 | 5 |
| 2867 | anneal | 898 | 718 | 90 | 90 | 38 | 10 | 5 |
| 3485 | scene | 2407 | 1925 | 241 | 241 | 299 | 10 | 2 |
| 3512 | synthetic_contr… | 600 | 480 | 60 | 60 | 60 | 10 | 6 |
| 3540 | analcatdata_box… | 120 | 96 | 12 | 12 | 3 | 10 | 2 |
| 3543 | irish | 500 | 400 | 50 | 50 | 5 | 10 | 2 |
| 3549 | analcatdata_aut… | 841 | 672 | 84 | 85 | 70 | 10 | 4 |
| 3560 | analcatdata_dmf… | 797 | 637 | 80 | 80 | 4 | 10 | 6 |
| 3561 | profb | 672 | 536 | 68 | 68 | 9 | 10 | 2 |
| 3602 | visualizing_env… | 111 | 88 | 11 | 12 | 3 | 10 | 2 |
| 3620 | fri_c0_100_5 | 100 | 80 | 10 | 10 | 5 | 10 | 2 |
| 3647 | rabe_266 | 120 | 96 | 12 | 12 | 2 | 10 | 2 |
| 3711 | elevators | 16599 | 13279 | 1660 | 1660 | 18 | 10 | 2 |
| 3731 | visualizing_liv… | 130 | 104 | 13 | 13 | 2 | 10 | 2 |
| 3739 | analcatdata_chl… | 100 | 80 | 10 | 10 | 3 | 10 | 2 |
| 3748 | transplant | 131 | 104 | 13 | 14 | 3 | 10 | 2 |
| 3779 | fri_c3_100_5 | 100 | 80 | 10 | 10 | 5 | 10 | 2 |
| 3797 | socmob | 1156 | 924 | 116 | 116 | 5 | 10 | 2 |
| 3896 | ada_agnostic | 4562 | 3648 | 457 | 457 | 48 | 10 | 2 |
| 3902 | pc4 | 1458 | 1166 | 146 | 146 | 37 | 10 | 2 |
| 3903 | pc3 | 1563 | 1249 | 157 | 157 | 37 | 10 | 2 |
| 3904 | jm1 | 10885 | 8707 | 1089 | 1089 | 21 | 10 | 2 |
| 3913 | kc2 | 522 | 416 | 53 | 53 | 21 | 10 | 2 |
| 3917 | kc1 | 2109 | 1687 | 211 | 211 | 21 | 10 | 2 |
| 3918 | pc1 | 1109 | 887 | 111 | 111 | 21 | 10 | 2 |
| 3953 | adult-census | 32561 | 26048 | 3256 | 3257 | 14 | 10 | 2 |
| 9946 | wdbc | 569 | 455 | 57 | 57 | 30 | 10 | 2 |
| 9952 | phoneme | 5404 | 4322 | 541 | 541 | 5 | 10 | 2 |
| 9957 | qsar-biodeg | 1055 | 843 | 106 | 106 | 41 | 10 | 2 |
| 9960 | wall-robot-navi… | 5456 | 4364 | 546 | 546 | 24 | 10 | 4 |
| 9964 | semeion | 1593 | 1273 | 160 | 160 | 256 | 10 | 10 |
| 9971 | ilpd | 583 | 465 | 59 | 59 | 10 | 10 | 2 |
| 9978 | ozone-level-8hr | 2534 | 2026 | 254 | 254 | 72 | 10 | 2 |
| 9984 | fertility | 100 | 80 | 10 | 10 | 9 | 10 | 2 |
| 10089 | acute-inflammat… | 120 | 96 | 12 | 12 | 6 | 10 | 2 |
| 10093 | banknote-authen… | 1372 | 1096 | 138 | 138 | 4 | 10 | 2 |
| 10101 | blood-transfusi… | 748 | 598 | 75 | 75 | 4 | 10 | 2 |
| 14952 | PhishingWebsite… | 11055 | 8843 | 1106 | 1106 | 30 | 10 | 2 |
| 14954 | cylinder-bands | 540 | 432 | 54 | 54 | 37 | 10 | 2 |
| 14965 | bank-marketing | 45211 | 36168 | 4521 | 4522 | 16 | 10 | 2 |
| 14967 | cjs | 2796 | 2236 | 280 | 280 | 33 | 10 | 6 |
| 125920 | dresses-sales | 500 | 400 | 50 | 50 | 12 | 10 | 2 |
| 125921 | LED-display-dom… | 500 | 400 | 50 | 50 | 7 | 10 | 10 |
| 145793 | yeast | 1269 | 1015 | 127 | 127 | 8 | 10 | 4 |
| 145799 | breast-cancer | 286 | 228 | 29 | 29 | 9 | 10 | 2 |
| 145836 | blood-transfusi… | 748 | 598 | 75 | 75 | 4 | 10 | 2 |
| 145847 | hill-valley | 1212 | 968 | 122 | 122 | 100 | 10 | 2 |
| 145977 | ecoli | 336 | 268 | 34 | 34 | 7 | 10 | 8 |
| 145984 | ionosphere | 351 | 280 | 35 | 36 | 34 | 10 | 2 |
| 146024 | lung-cancer | 32 | 24 | 4 | 4 | 56 | 10 | 3 |
| 146063 | hayes-roth | 160 | 128 | 16 | 16 | 4 | 10 | 3 |
| 146065 | monks-problems-… | 601 | 480 | 60 | 61 | 6 | 10 | 2 |
| 146192 | car-evaluation | 1728 | 1382 | 173 | 173 | 21 | 10 | 4 |
| 146210 | postoperative-p… | 88 | 70 | 9 | 9 | 8 | 10 | 2 |
| 146607 | SpeedDating | 8378 | 6702 | 838 | 838 | 120 | 10 | 2 |
| 146800 | MiceProtein | 1080 | 864 | 108 | 108 | 77 | 10 | 8 |
| 146817 | steel-plates-fa… | 1941 | 1552 | 194 | 195 | 27 | 10 | 7 |
| 146818 | Australian | 690 | 552 | 69 | 69 | 14 | 10 | 2 |
| 146820 | wilt | 4839 | 3871 | 484 | 484 | 5 | 10 | 2 |
| 146821 | car | 1728 | 1382 | 173 | 173 | 6 | 10 | 4 |
| 167140 | dna | 3186 | 2548 | 319 | 319 | 180 | 10 | 3 |
| 167141 | churn | 5000 | 4000 | 500 | 500 | 20 | 10 | 2 |
| 167211 | Satellite | 5100 | 4080 | 510 | 510 | 36 | 10 | 2 |
| 168911 | jasmine | 2984 | 2386 | 299 | 299 | 144 | 10 | 2 |
| 190408 | Click_predictio… | 39948 | 31958 | 3995 | 3995 | 11 | 10 | 2 |
| 360948 | libras | 360 | 288 | 36 | 36 | 104 | 10 | 10 |

**Table 5: Metadata of the WhyTrees Benchmark. Splits refers to the number of cross validation splits.**
| OpenML | Observations | Features | Splits | Classes |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ID | Name | All | Train | Valid | Test |  |  |  |
| 44089 | credit | 16714 | 10000 | 2014 | 4700 | 10 | 2 | 2 |
| 44120 | electricity | 38474 | 10000 | 8542 | 19932 | 7 | 1 | 2 |
| 44121 | covertype | 566602 | 10000 | 50000 | 50000 | 10 | 1 | 2 |
| 44122 | pol | 10082 | 7057 | 907 | 2118 | 26 | 3 | 2 |
| 44123 | house_16H | 13488 | 9441 | 1214 | 2833 | 16 | 3 | 2 |
| 44125 | MagicTelescope | 13376 | 9363 | 1203 | 2810 | 10 | 3 | 2 |
| 44126 | bank-marketing | 10578 | 7404 | 952 | 2222 | 7 | 3 | 2 |
| 44128 | MiniBooNE | 72998 | 10000 | 18899 | 44099 | 50 | 1 | 2 |
| 44129 | Higgs | 940160 | 10000 | 50000 | 50000 | 24 | 1 | 2 |
| 44130 | eye_movements | 7608 | 5325 | 684 | 1599 | 20 | 3 | 2 |
| 44156 | electricity | 38474 | 10000 | 8542 | 19932 | 8 | 1 | 2 |
| 44157 | eye_movements | 7608 | 5325 | 684 | 1599 | 23 | 3 | 2 |
| 44159 | covertype | 423680 | 10000 | 50000 | 50000 | 54 | 1 | 2 |
| 45019 | Bioresponse | 3434 | 2403 | 309 | 722 | 419 | 5 | 2 |
| 45020 | default-of-cred… | 13272 | 9290 | 1194 | 2788 | 20 | 3 | 2 |
| 45021 | jannis | 57580 | 10000 | 14274 | 33306 | 54 | 1 | 2 |
| 45022 | Diabetes130US | 71090 | 10000 | 18327 | 42763 | 7 | 1 | 2 |
| 45026 | heloc | 10000 | 7000 | 900 | 2100 | 22 | 3 | 2 |
| 45028 | california | 20634 | 10000 | 3190 | 7444 | 8 | 1 | 2 |
| 45035 | albert | 58252 | 10000 | 14475 | 33777 | 31 | 1 | 2 |
| 45036 | default-of-cred… | 13272 | 9290 | 1194 | 2788 | 21 | 3 | 2 |
| 45038 | road-safety | 111762 | 10000 | 30528 | 50000 | 32 | 1 | 2 |
| 45039 | compas-two-year… | 4966 | 3476 | 447 | 1043 | 11 | 3 | 2 |

### A.6 TabZilla Further Results

**Table 6: Main Results on TabZilla. N. Accuracy stands for Normalized accuracy. Rank compares the relative rank of a method compared to all other methods on that dataset.**
| Models | Rank | N. Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| min | max | mean | median | mean | median |  |
| TabForestPFN - Fine-tune | 1 | 24 | 8.0 | 6.0 | 0.680 | 0.663 |
| TabPFN (retrained) - Fine-tune | 1 | 26 | 8.3 | 7.8 | 0.678 | 0.700 |
| CatBoost | 1 | 22 | 8.9 | 7.8 | 0.676 | 0.663 |
| XGBoost | 1 | 23 | 9.2 | 8.2 | 0.674 | 0.671 |
| TabPFN (original) - Fine-tune | 1 | 25 | 9.3 | 8.0 | 0.670 | 0.677 |
| TabForestPFN - Zero-shot | 1 | 26 | 10.6 | 10.0 | 0.633 | 0.650 |
| TabForest - Fine-tune | 1 | 24 | 10.6 | 9.5 | 0.650 | 0.667 |
| TabPFN (original) - Zero-shot | 1 | 25 | 10.9 | 10.2 | 0.632 | 0.606 |
| LightGBM | 1 | 26 | 11.1 | 11.0 | 0.646 | 0.653 |
| RandomForest | 1 | 25 | 11.3 | 11.0 | 0.636 | 0.637 |
| Resnet | 1 | 26 | 11.9 | 10.0 | 0.602 | 0.613 |
| NODE | 1 | 26 | 12.1 | 12.0 | 0.611 | 0.596 |
| SAINT | 1 | 26 | 12.2 | 12.8 | 0.600 | 0.614 |
| SVM | 1 | 25 | 12.5 | 13.2 | 0.589 | 0.573 |
| FT-Transformer | 1 | 23 | 12.6 | 12.5 | 0.599 | 0.601 |
| DANet | 3 | 25 | 14.5 | 15.0 | 0.596 | 0.608 |
| MLP-rtdl | 1 | 26 | 15.8 | 17.8 | 0.514 | 0.527 |
| TabForest - Zero-shot | 3 | 25 | 15.9 | 17.5 | 0.571 | 0.583 |
| TabPFN (retrained) - Zero-shot | 3 | 25 | 15.9 | 17.5 | 0.571 | 0.583 |
| STG | 1 | 26 | 16.1 | 17.2 | 0.495 | 0.472 |
| LinearRegression | 1 | 26 | 17.5 | 19.5 | 0.459 | 0.431 |
| MLP | 1 | 26 | 17.7 | 20.0 | 0.475 | 0.417 |
| TabNet | 2 | 26 | 18.0 | 19.0 | 0.503 | 0.490 |
| DecisionTree | 1 | 26 | 18.7 | 21.0 | 0.413 | 0.372 |
| KNN | 2 | 26 | 19.4 | 21.5 | 0.399 | 0.372 |
| VIME | 1 | 26 | 21.8 | 23.8 | 0.295 | 0.221 |

In the TabZilla main results Table 3, we have shown the five highest performing methods. The full results are shown in Table 6, which includes all ICL-transformer variants and methods implemented by the TabZilla authors. Because the rank is calculated over all included methods, which ICL-transformer variants we include might change the results. Therefore, we check if the results are the same if we use calculate the rankings one ICL-transformer at the time.

Table 7 shows the results of only the fine-tuned TabForestPFN versus the rest of the benchmark. We do this for every ICL-transformer and aggregregate the results in Table 8. All results are qualitatively the same as in Table 3.

**Table 7: Main Results on TabZilla. N. Accuracy stands for Normalized accuracy. Rank compares the relative rank of a method compared to all other methods on that dataset.**
| Models | Rank | N. Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| min | max | mean | median | mean | median |  |
| TabForestPFN - Fine-tune | 1 | 18 | 5.8 | 4.2 | 0.680 | 0.663 |
| CatBoost | 1 | 15 | 6.2 | 5.2 | 0.676 | 0.663 |
| XGBoost | 1 | 16 | 6.3 | 5.0 | 0.674 | 0.671 |
| LightGBM | 1 | 19 | 7.6 | 6.0 | 0.646 | 0.653 |
| RandomForest | 1 | 18 | 7.9 | 8.0 | 0.636 | 0.637 |
| NODE | 1 | 19 | 8.4 | 8.0 | 0.611 | 0.596 |
| Resnet | 1 | 19 | 8.4 | 8.0 | 0.602 | 0.613 |
| SAINT | 1 | 19 | 8.5 | 7.2 | 0.600 | 0.614 |
| SVM | 1 | 18 | 8.6 | 8.0 | 0.589 | 0.573 |
| FT-Transformer | 1 | 16 | 8.8 | 8.5 | 0.599 | 0.601 |
| DANet | 2 | 19 | 10.0 | 10.0 | 0.596 | 0.608 |
| MLP-rtdl | 1 | 19 | 11.4 | 12.0 | 0.514 | 0.527 |
| STG | 1 | 19 | 11.5 | 12.0 | 0.495 | 0.472 |
| LinearRegression | 1 | 19 | 12.3 | 13.8 | 0.459 | 0.431 |
| MLP | 1 | 19 | 12.6 | 14.0 | 0.475 | 0.417 |
| TabNet | 1 | 19 | 12.8 | 13.8 | 0.503 | 0.490 |
| DecisionTree | 1 | 19 | 13.3 | 14.0 | 0.413 | 0.372 |
| KNN | 2 | 19 | 13.9 | 15.0 | 0.399 | 0.372 |
| VIME | 1 | 19 | 15.7 | 17.0 | 0.295 | 0.221 |

**Table 8: Main Results on TabZilla. N. Accuracy stands for Normalized accuracy. Rank compares the relative rank of a method compared to all other methods on that dataset. This table displays individual results: Table 7 is run individually for all ICL-transformer variants, and the row of the ICL-transformer is copy-pasted here.**
| Models | Rank | N. Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| min | max | mean | median | mean | median |  |
| TabPFN (original) - Zero-shot | 1 | 19 | 7.4 | 6.0 | 0.632 | 0.606 |
| TabPFN (original) - Fine-tune | 1 | 19 | 6.4 | 5.0 | 0.670 | 0.677 |
| TabPFN (retrained) - Zero-shot | 2 | 19 | 10.7 | 11.8 | 0.571 | 0.583 |
| TabPFN (retrained) - Fine-tune | 1 | 19 | 5.9 | 5.0 | 0.678 | 0.700 |
| TabForest - Zero-shot | 2 | 19 | 10.7 | 11.8 | 0.571 | 0.583 |
| TabForest - Fine-tune | 1 | 19 | 7.4 | 7.0 | 0.650 | 0.667 |
| TabForestPFN - Zero-shot | 1 | 19 | 7.2 | 6.0 | 0.633 | 0.650 |
| TabForestPFN - Fine-tune | 1 | 18 | 5.8 | 4.2 | 0.680 | 0.663 |

### A.7 WhyTrees Further Results

The main results in Figure 4 report the normalized accuracy aggregated over all datasets. In Figure 8 we show the comparison between ICL-transformers on all 23 datasets. In Figure 9 we show the colored results on each individual dataset where we compare fine-tuned TabForestPFN with all the benchmark methods.

Figure: Figure 8: Comparison of TabForest, TabPFN and TabForestPFN on the WhyTrees benchmark. Performance of different ICL-transformers is often similar. All the benchmark algorithms are greyed out. See Appendix A.7 for colored benchmark lines.

Figure: Figure 9: Individual dataset results on the WhyTrees Benchmark. TabForestPFN shows the mean of ten default runs, all others use random search over the hyperparameters. See Table 2 for other ICL-transformers.

### A.8 One-by-One Comparisons

In Figure 10 we plot one-to-one comparisons of fine-tuned TabForestPFN versus CatBoost, TabForest and TabPFN. We see no clear correlations in the other comparisons between performance difference and model.

Figure: Figure 10: Differences in normalized accuracy of individual datasets from TabZilla. The color red means the left-mentioned method is the best, blue for the right-mentioned method. The darkest red represents at least 0.20 normalized score points improvement, and dark blue at least 0.20 normalized accuracy points degradation.

### A.9 Synthetic Data with Lower Complexity

In the ablation we have seen that even with a forest dataset generator with lower complexity parameters, we still have similar performance. To give an idea of how complex the data is, here we showcase the generated data. Figure 11 displays generated data with base size 32, and Figure 12 displays generated data with maximum tree depth 9.

Figure: Figure 11: Generated forest data. Every box is a generated dataset with its own classes (color) and features (axes). Generated with base size 32, dataset size 1024, tree depth between 1 and 25, two features, and between 2 and 10 number of classes. See also Figures 3 and 12.
Refer to caption: extracted/2405.13396v1/figures/synthetic_forest_base_size_32.png

Figure: Figure 12: Generated forest data. Every box is a generated dataset with its own classes (color) and features (axes). Generated with base size 1024, dataset size 1024, tree depth between 1 and 9, two features, and between 2 and 10 number of classes. See also Figures 3 and 11.
Refer to caption: extracted/2405.13396v1/figures/synthetic_forest_depth_9.png

### A.10 Decision Boundaries for Scaled TabForest

In the complexity boundary results, we saw that TabForest has more complex decision boundaries when fine-tuned. Here we show that the complexity of the decision boundary is correlated with the dimension size of the transformer.
See Figure 13 for the results. We see that the decision boundaries become more complex when the dimension of the ICL-transformer increases.

Figure: Figure 13: Comparison of decision boundaries for the electricity dataset. Axis represent features, color is predicted class probability, dots are test observations. When increasing the dimension size of the transformer, complexity increases.
Refer to caption: extracted/2405.13396v1/figures/boundary_plots_dim.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: I do not have further questions besides the weaknesses listed above. It would be great if the authors could address the weaknesses during the rebuttal. I'm open to adjusting my ratings based on the authors' responses.
soundness: 2
strengths: S1. The paper identifies that fine-tuning is still needed to achieve the best performance on each downstream task.

S2. It proposes a novel, seemingly simple-to-implement way to create more complex pre-training data. The idea to incorporate decision trees in data generation is neat, as it encourages the neural net model to mimic Tree-based models' predictions.

S3. Expensive experiments are conducted.
summary: This paper studies in-context learning (ICL) for tabular data, building upon TabPFN. The authors showed that fine-tuning leads to better downstream performance, and better pre-training with more complex data (forest-based) helps the fine-tuning performance. The authors proposed a novel way to create less realistic but more complex pre-training data. The resulting pre-trained model (including both original and new pre-train data), TabForestPFN, when fine-tuned, achieves the best performance across a wide range of tabular datasets and tasks.
weaknesses: W1. The motivations of this paper are not very clear and convincing. First, it is intuitive that fine-tuning could create more complex boundaries and improve the downstream performance --- the fine-tuning process needs to fit the downstream data. Second, it is intuitive that when the downstream data are sufficient (large context), then fine-tuning would improve more. (Otherwise, it may overfit.) Third, I'm not entirely convinced that pre-training from scratch cannot generate complex boundaries. Basically, if one over-fits the model to the training data, it should create a complex enough boundary. Fourth, I didn't fully capture "We find that fine-tuned ICL-transformers, in contrast, are able to create these complex decision boundaries similar to tree-based methods." Again, it is intuitive that fine-tuning would create more complex boundaries; do ICL-transformers play an important role in this statement? 

W2. It is a bit hard to digest what each step in Algorithm 1 does. Can the authors provide a visualization, for example, a series of sub-figures where each of them corresponds to the outcome of each step in Algorithm 1?

W3. Experimental results on WhyTrees show good signs of TabForestPFN (or just TabForest) but Table 3 (on TabZilla) seems to suggest no benefit of TabForestPFN over TabPFN if both are fine-tuned. Can the authors provide more details about these? 

W4. I'm not sure if I completely buy the argument that TabPFN achieves better ZSL performance because its training data are more realistic. If the downstream tasks (realistic data) do require more complex boundaries, isn't it possible to create realistic but complex data?

W5. The analyses lack sufficient insights. For example, the way section 5.6 is written seems to be too intuitive: fine-tuning excels when there are sufficient samples; a larger context (i.e., more training data for the downstream tasks) improves the performance. I am curious why at larger support sizes (Figure 7 (b)), fine-tuning shows more benefits than zero-shot. It would be great if the authors could provide more insights rather than superficially summarizing the observations.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Is there any result of training TabPFN, including the data used when fine-tuning TabForestPFN? Is it true that this data can only be used in the fine-tuning process?
2. I understand that the TabForestPFN method improves performance by affecting the decision boundary, but then I do not know if the method affects anything other than the decision boundary. It's not convincing that it only complicated the decision boundary. Authors need to explain how one can be sure that the performance improvement is due to the decision boundary.
soundness: 2
strengths: 1. It is interesting to see that they constructed their own dataset to complicate the decision boundary and fine-tuned with that dataset to increase performance.
2. It seems that the transformer-based model can show higher performance than the machine learning-based model. It is interesting to see a fine-tuning approach that can be used in the future to train deep neural networks on tabular data.
summary: This paper presents a TabPFN method for tabular data classification. They show that their TabPFN achieves better performance than traditional machine learning models, such as random forests. The authors propose TabFPN based on the complexity of the decision boundary and its relevance to the classification of tabular data.
Based on their insights on the decision boundary, the authors propose a fine-tuning method for TabPFN models to improve performance by complicating the decision boundaries. To complicate the learning of decision boundaries, they generate unrealistic data that is different from existing tabular data and use it as fine-tuning data.
weaknesses: 1. Although the authors' TabPFN method achieves better results than the classical tree-based machine learning models, the performance improvement is marginal. For example, in Table 3, TabPFN outperforms XGBoost by only about 0.03 to 0.04.
2. Figure1 shows that the fine-tuned model has a more complex decision boundary than the untuned model. However, when comparing the fine-tuned model and random forest, it is difficult to say that the decision boundary looks more complicated. The fine-tuned model seems to have a more complex decision boundary than the non-fine-tuned model, but when comparing it to a random forest, it is hard to say that the decision boundary is complex.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weakness. My major concern is the contribution (novelty) of this paper and the evaluation.
soundness: 3
strengths: 1. Experiments in this paper show that they achieve Sota performance on some of the datasets.
2. Authors give a visualization on the decision boundaries for the finetuned model and the zero-shot model, which helps to explain the effectiveness of their method.
summary: This paper extends a previous tabular data classification method TaabPFN to the finetuning setting, which tends to have a complex decision boundary. The authors have made a dataset by synthesizing data via a decision tree, which helps to show the fact that tabForest achieves better finetuning performance with more complex generated datasets. Authors mainly evaluate their method on WhyTrees and TabZilla and achieves good performance.
weaknesses: 1. Lack of novelty. This paper extends a previous model from ICL to finetuning and obtains a better performance, which is a very naive result since training (finetuning) a model always leads to better performance than using prompts (ICL), this makes the findings in the paper not meaningful. 
2. Similarly, the model after training can have a more complex decision boundary than the zero-shot model, this is also a very straightforward conclusion that does not bring anything new for me. 
3. Authors compare their methods with some very traditional models such as SVM, MLP, Decision Tree, XGBoost, and so on. Most of these methods have been proposed for more than ten years. A comparison of these methods can not prove that the proposed methods are good. Please compare  with more SOTA methods, and also report some other information such as the parameters, inference time for each model.
4.  The writing should be improved, especially for the figures in the paper.
5. Authors mentioned that the proposed finetuned method is an approach to reduce the requirements for GPU memory by two times, but no quantitative numbers for GPU memory required are provided in this paper.