TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Authors: Siyang Liu
Han-Jia Ye
Venue: ICML 2025 poster (2025)
Keywords: Tabular data
Machine learning
Tabular foundation model
TabPFN
Primary area: deep_learning

Abstract: TabPFN has emerged as a promising in-context learning model for tabular data, capable of directly predicting the labels of test samples given labeled training examples. It has demonstrated competitive performance, particularly on small-scale classification tasks. However, despite its effectiveness, TabPFN still requires further refinement in several areas, including handling high-dimensional features, aligning with downstream datasets, and scaling to larger datasets.
In this paper, we revisit existing variants of TabPFN and observe that most approaches focus either on reducing bias or variance, often neglecting the need to address the other side, while also increasing inference overhead. To fill this gap, we propose  Beta (**B**agging and **E**ncoder-based Fine-tuning for **T**abPFN **A**daptation), a novel and effective method designed to *minimize both bias and variance*. To reduce bias, we introduce a lightweight encoder to better align downstream tasks with the pre-trained TabPFN. By increasing the number of encoders in a lightweight manner, Beta mitigates variance, thereby further improving the model’s performance. Additionally, bootstrapped sampling is employed to further  reduce the impact of data perturbations on the model, all while maintaining computational efficiency during inference. Our approach enhances TabPFN’s ability to handle high-dimensional data and scale to larger datasets. Experimental results on over 200 benchmark classification datasets demonstrate that Beta either outperforms or matches state-of-the-art methods.

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Si- SinYang Liu
Han-Jia Ye
Abstract
1. Introduction
Tabular data is one of the most widely used data formats across various domains, including finance (Cao & Tay, 2001). healthcare (Hassan et al., 2020), e-commerce (Ned- erstigt et al., 2014), and medical analysis (Schwartz et al., 2007; Subasi, 2012). Despite its ubiquity. modeling tabular data with deep learning methods remains a challenge due to its heterogeneous nature (Borisov et al., 2024b). Yet re- cent advancements have led the development of tabular foundation models (van Breugel & van der Schaar, 2024). such TabPFN (Tabular Prior-Fitted Networks) (Hollmann et al., 2023; 2025). TabPFN operates in two stages: pre- training and inference. During the pre- pre-training stage, the model is pre-trained on a diverse set of synthetic datasets. In the inference stage, given a new task and a set of labeled examples as "prompt," TabPFN directly predicts the labels of test samples using in-context learning, without requiring further parameter updates. This approach enables TabPFN to achieve performance comparable to surpass tree- based methods, particularly on small tabular datasets (McEl- fresh et al., 2023).
TabPFN has emerged promising in-context learning model for tabular data, capable of di- rectly predicting the labels of test samples given labeled training examples. has demonstrated competitive performance, particularly on small- scale classification tasks. However, despite its effectiveness, TabPFN still requires further re- finement in several areas, including aligning with downstream datasets, handling nigh-dimensional features, and scaling to larger datasets. In this paper, we revisit existing variants of TabPFN and observe that most approaches focus either on reducing bias or variance, often neglecting the need to address the other side, while also increasing inference overhead. To fill this gap, we propose BETA (Bagging and Encoder-bass based Fine-tuning for TabPFN Adaptation), novel and effective method designed to minimize both bias and variance. To reduce bias, we intro- duce lightweight encoder to better align down- stream tasks with the pre-trained TabPFN. By in- creasing the number of encoders in lightweight manner, BETA mitigate variance, thereby fur- ther improving the performance of the pretrained TabPEN. Additionally, bootstrapped sampling employed to further reduce the impact of data perturbations on the model, all while maintaining computational efficiency during inference. Our approach enhances TabPFN's ability handle high-dimensional data and scale to larger datasets. Experimental results on over 200 benchmark clas- sification datasets demonstrate that BETA either outperforms or matches state-of-the-an methods.
TabPEN has shown potential across a wide range of applica- tions, including tabular data generation (Ma et al., 2024a; Margeloiu et al., 2024b), data augmentation (Margeloiu et al., 2024a), and time series forecasting (Hoo et al., 2025). These cases highlight the versatility of TabPFN, posi- tioning it as a model worthy of further exploration. However, alongside application-driven studies, there is a growing in- terest in improving TabPFN's performance from various perspectives (Feuer al., 2024; Thomas et al., 2024; Xu et al., 2025; Ma et al., 2024b). While previous research has reported performance improvements, the underlying reasons driving these gains remain unclear. These advance- ments are often fragmented, with each approach focusing on a specific aspect, and some methods even sacrifice effi- ciency for enhanced performance, without a comprehensive understanding of how to improve TabPFN systematically.
'School of Artificial Intelligence, Nanjing University, China "National Key Laboratory for Novel Software Technology, Nanjing University, China. Correspondence Han-Jia Ye <ychj@ wawdaaamaaaaauu maanda.nu..euu.cn.
In this paper, we adopt the bias-variance decomposition framework introduced by Nagler (2023) to analyze the gen- eralization error of TabPFN and its variants. This frame- work allows us to revisit and categorize existing methods, revealing that performance improvements typically arise
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
from addressing either bias variance. However, these approaches often neglect the other aspect, leading to subop- timal performance. Therefore, there is need for a method that simultaneously addresses both bias and variance to im-
Remark. We have noticed that the latest release of TabPFN- v2 (Hollmann et al., 2025) has partially alleviated some of the limitations previously mentioned. Specifically, TabPFN- v2 incorporates design improvements that enable it to handle larger datasets and more features. However, it is important to highlight that TabPFN-v2 is a concurrent work, and while it partially mitigates these limitations, it does not fully resolve the challenges posed by dataset size and feature count. Thus, many of the improvements proposed in this paper are gen- eral enhancements that can complement TabPPN-v2 and po- tentially further its applicability to broader range of tasks.
prove performance.
To this end, we propose a novel, effective, and scalable approach, BETA Our method enhances TabPFN's per- formance by introducing a fine-tuning stage, enabling parameter-efficient adaptation that aligns the downstream dataset distribution with the pre-trained TabPFN to mitigate bias. To further reduce variance, BETA maps raw data into multiple latent spaces and employs computationally efficient bootstrapped sampling during inference. Beyond perfor- mance improvements, BETA offers a lightweight and seal- able solution that effectively handles high-dimensional data and large datasets while maintaining inference efficiency.
2. Preliminary
In this section, we provide a brief overview of TabbPFN and analyze its properties, while also revisiting existing variants.
BETA improves performance by refining both the fine- tuning and inference stages to reduce bias and variance. In the fine-tuning phase, we employ a lightweight encoder module as an input feature adapter, which transforms datasets with arbitrary dimensionality into multiple fixed- dimensional representations, thereby naturally enabling dimensionality reduction. To further enhance generalization, we integrate Batch Ensemble (Wen et al., 2020; Gorishniy et al., 2025) to increase the number of encoders in a lightweight manner, which introduces diversity in learned representations and reduces variance. These enhancements enable BETA to improve robustness and scalability, ensuring better adaptation to downstream datasets. In the inference phase, we introduce bootstrapped sampling, technique that has been largely overlooked in previous TabPFN variants. By generating multiple subsets of the dataset as support sets for the context composition, BETA further reduces variance. Furthermore, our method seamlessly integrates with Error-Correcting Output Codes (Dietterich & Bakiri, 1995) (ECOC) to effectively handle multiclass classification tasks with more than 10 classes.
2.1. TabPFN
We consider a tabular dataset consisting of N examples and features. Each instance 2; E R2 is represented by d feature values, where Σi,j denotes the j-th feature of instance æ;, These features can be numerical (ai E IR) or categorical (x with categorical values often encoded as integers. Each instance is associated with a label yts where y E [C] = {l.....C) for classification task, and yi E IR for regression task. Given a training dataset, Derain = {( + Verain) ) and test samples, X test loststl Noan, the goal is to predict the correspond- ing labels, Yest [yrest
Nustt as accurately as possible.
TabPFN follows a two-stage process: a pre-training stage, where the model is trained on synthetic datasets by minimiz- ing the discrepancy between the predicted label of the test instance and its true label, and an inference stage, where it directly predicts the labels of test samples given a set of labeled training examples.
Following pre-training, TabPFN takes the entire training dataset Dreain = (Xerain Btrsin) and the features of query points X, as context to make predictions. This is done using PFN-Style Batching (Hollmann et al., 2023), where Niest test samples are batched into a single "prompt," as illustrated in Figure 1. Since TabPFN is trained with fixed input dimensionality, typically set to predefined value daax (e.g., 100), datasets with fewer features (d < dmmx)) need to be extended via zero-padding before being processed. Formally, given an input instance I; E Rd, the padded input at E Rddma is obtained as:
Experimental results multiple benchmark datasets, in- cluding over 200 classification tasks, demonstrate that our method significantly improves the performance of TabPFN. We describe our main contributions below.
1. We introduce an adaptation method for TabPFN that addresses key limitations related to dataset size, high- dimensional features, and multiclass classification tasks. 2. By analyzing the generalization error of existing TabPFN variants through bias-variance decomposition and exper- iments on real-world datasets, we introduce BETA, method that effectively mitigates both bias and variance.
a = [z,, 0] E Rddax
3. We achieve state-of-the-art performance on the largest benchmark datasets to date, demonstrating the robustness and scalability of our method for real-world tabular tasks. The code is available at https: / / thub. com/ LAMDA Tabularr
where 0 E [R represents the sero-padding applied to match the required input dimensionality. However, if a dataset exceeds this limit (d > dmax). TabPFN cannot directly process such datasets, as its architecture is not de-
2

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
sketching techniques, such as CoreSet, K-Means, and Data Distillation (Ma et al., 2024c), which aim compress large datasets into representative subsets while preserving essen- tial information (Feuer et al., 2023). TuneTables (Feuer et al., 2024), on the other hand, attempts to encode the dataset into a compact learned representation, reducing memory overhead. Another class of methods selects sample- specific contexts instead of fixed support set. For instance, MixturePFN (Xu et al., 2025) partitions the training set into multiple subsets and assigns the most relevant one to each test sample. Similarity. LocalPEN (Thomas et al., 2024) and TabDPT (Ma et al., 2024b), instead use nearest neigh- bors to dynamically construct the support set Derain for each query (KNN-based selection). While these approaches improve TabPFN's performance by refining Dirain. they disrupt PFN-style batching, significantly reducing inference efficiency.
Training Set Text Set
N Nirai
N
AI
N
d
X
TabPFN
Figure The inference process of TabPFN. The training set and test set are concatenated, with the training labels added to the input sequence. TabPFN then performs a single forward pass to generate predictions for all test samples.
signed to accommodate nigher-dimensionn inputs.
Fine-tuning Strategies for Enhancing TabPFN Perfor- mance. The second approach to improving TabPFN's performance focuses on fine-tuning the pre-trained model to better adapt to downstream datasets. As shown in Equation 1, the predictive distribution po(y4 Xx Dirain) depends on both the training set Dtrain and the model parameters O. While optimizing Dirain improves the quality of support examples, fine-tuning enhances qa, enabling better alignment with downstream tasks. A straightforward approach is to fine-tune all model parameters, as seen in TabForestPFN (den Breejen et al., 2024) and Lo- calPFN (Thomas et al., 2024), which improves performance by adapting TabPFN's learned prior to specific datasets. However, full fine-tuning incurs substantial computational costs due to the large number of model parameters. To mitigate this, Tune Tables (Feuer et al.. 2024) offers more efficient alternative by applying prompt-tuning (Lester et al., 2021) for classification tasks, which adjusts only a small set of parameters, and fine-tuning the entire model for regression tasks. Another efficient approach adapter- based fine-tuning, employed in MixturePFN (Xu et al., 2025), which fine-tunes additional adapter layers (Houlsby et al., 2019) rather than modifying the entire model. A detailed related work is presented in Appendix A.
TabPFN outputs a probability distribution over possible labels yn E {1,.... C). Specifically, let qe denote the logits produced by TabPFN, where represents the parameters of the pre-trained model. The posterior predictive distribution given by:
exp(go(X (g.. D (rrain)(u))
(y Xq. Dtrain)
Ee=1 exp(ge (Xq. (4)) (1) where Dirain denotes the zero- zero-paddes paddeaddd training dataset, act- ing as the support set for the context composition. Since TabPFN performs inference directly, the model typically applies multiple rounds of feature shuffling on the original features, followed by prediction. The final prediction is obtained by averaging the results from these multiple in- ferences. For more details about how the model processes inputs, please refer to Appendix B.4.
2.2. TabPFN Variants
Existing research to improve the performance of TabPFN has primarily focused on two different strategies, stemming from its dependency on go and Derain in Equation 1. One approach optimizes context selection, refining Dirain pro- vide more informative support examples. The other concen- trates on fine-tuning TabPFN, improving qu to better adapt to downstream tasks. These two strategies offer complemen- tary solutions for enhancing TabPFN's 's overall performance. Context Selection for Scaling Ta The transformer architecture in TabPFN inherently leads to quadratic growth in memory usage as the context length increases. As result, the size of the support set used context in each prediction is limited. Meanwhile, a well-chosen Derain provides more relevant support examples, improving generalization. A simple approach is random subsampling (McElfresh et al., 2023), but this can degrade performance, especially on large datasets (Ma et al., 2024c). More structured methods include
2.3. Generalization Analysis of TabPFN Variants
While various TabPFN variants have been proposed to en- hance performance, their underlying mechanisms remain in- sufficiently understood. In this section, we leverage the bias- variance decomposition framework, as described Equa- tion 4, which is introduced by Nagler (2023), to analyze the generalization error of TabPFN and its variants on two real- world datasets: Adult (Barry & Ronny, 1996) and Bank (S. et al., 2014). To assess the impact of different strategies, and based on our previous analysis, we evaluate the original TabPFN (without context length restrictions) alongside sev-
3

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
TaаPFN
0.005
0.22
TabPFN-1000
TabPFN- Baggging
1055
0.15
Square of Bias
0.004
9.11
IabPrN-en-Bl
KNN PFN
TabPFN finetune
0.21
2,003
0.13
Ours
2.005
3,99
0.05
0.20
0.891
0.244
0.001
Size
Size
Sizee
(a) Gemeraiization ermor on the Adult Dataset.
(b) Bias on the Adullt Dataset.
(c) Variance on the Adult Dataset.
3.075
2.005
1.60
Square Bies
2,0725
3.043
2,006
2.005
1011
3.04
3.060
20000
452
000
Size 3203
12000
3202
3400
280
1.000
6400
12000
Size
(d) Generalization Erzor on the Bank Dataset.
(c) Rios on the Bank Datnset.
(1) Varionee on the Bank Dataset.
Figure 2: Generalization Error, Bias, and Variance for different methods on the Adult and Bank datasets. The methods shown include TabPFN-1000 (subsample size = 1000). TabPEN, TahPFN-en-16, TabPFN-Bagging, KNN-PFN, TabPFN-finetun and Ours (BETA). The legend is located in the top-right plot for clarity.
eral representative variants. These include TabPFN-1000 (random subsampling of 1000 samples), TabPFN-en-Ik 16 (ran-
tional findings.
1) Sine-Tuning and its Impact: Figure 2 (b,e) shows that fine-tuning TabPPNN-inntunne effectively reduces bias by aligning the model's prior with the characteristies of the dataset. However, it also increases variance, especially when the training set is small, where overfitting amplifies predic- tion variability, shown in Figure 2 (c,f). These results suggest that fine-tuning requires careful regularization to balance bias and variance.
domly sample once and perform feature shuffling 16 times to form an ensemble of 16 predictions). KNN-PFN (KNN- based context selection), TabPFN-finetune (full model fine- tuning), and TabPFN-Bagging Bagging (bootstrapped sampling with 16 varying contexts). TabPFN-Bagging unlike ensemble methods that require training multiple independent models, employs bootstrapped sampling to construct diverse support sets within a single inference process. A detailed description of its implementation is provided in Appendix B.5.
2) Ensemble Strategy and Bias-Variance Tradeoff: Fig- ure (b,c) indicates that the ensemble-based method (TabPFN-en-16) reduces variance but may increase bias due to unsupervised feature transformations. This observa- tion highlights that ensemble-stye TabPFN, while helpful in reducing variance, may require additional bias-reducing strategies to optimize overall model performance.
To further contextualize these findings, we compare all eval- uated methods against our proposed approach, BETA. The experimental results, presented in Figure 2, reveal several key trends in the bias-variance tradeoff across different TabPFN variants. To complement the visual analysis, Ta- ble reports the relative percentage changes bias and variance (negative values indicate reduction) for several rep- resentative variants, averaged across dataset sizes from Fig- ure 2, for both the Adult and Bank datasets. It can be ob- served that, except for our method, none of the representa- tive TabPFN variants manage to simultaneously reduce both bias and variance across the two datasets.
3) Effectiveness of Bagging: As shown in Figure 2 (c.f), Bagging significantly reduces variance while maintaining stable bias. By introducing diversity through bootstrapped sampling, it achieves variance reduction comparable to en- semble methods but at a lower computational cost. These results highlight Bagging as a simple yet effective approach for improving the performance of TabPFN.
As shown in Figure (b,c,e.f), increasing context length reduces variance while bias plateaus, consistent with prior findings (Nagler, 2023). Similarly, KNN-based context se- lection reduces bias but increases variance compared to using the full dataset, as it relies on localized subsets. Since these trends align with previous work, we focus on addi-
The results above, based on additional findings not covered in (Nagler, 2023), indicate that TabPFN variants typically impact either bias or variance, but rarely both simultane- ously. For instance, KNN-based context selection reduces bias but increases variance, while the original ensemble
4

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Table 1: Comparison of TabPEN and related methods in terms of their impact on bias and variance, as well as their effective- ness in handling large datasets, sigh--imensional features, adaptability to multiclass classification, and inference efficiency. The compared methods include TabPFN (Hollmann al., 2023), TuneTables (Feuer et al., 2024), TabForestPFN (den Breejen al., 2024), LocalPFN (Thomas al., 2024), and MixturePFN (Xu al., 2025). Our proposed method (BETA) distinguished for its ability to balance bias and variance while maintaining efficient scaling, adaptability to multiclass classification, and lightweight fine-tuning. Different row colors indicate the strategies used for improving performance ) scalability (), and efficiency The "No Additional Inference Cost" entrie refers to computational efficiency relative to the ensemble-style TabPFN setting commonly used in practice, as discussed in subsection 2.1.
BETA (Ours)
TabPFN
Tablls
TaFForestPPN
LocaLPFN
MixturePFN
Scales to Large Dutasets
/
Handles Ditenniinnnninll Data
X
Adapts to More Than Classes
X
Reduces Bias
X
V
Reduces Variance
X
X
X
No Additional Inference Cost Fine-Tuning
X
lightweight encoder
X
prompt & backbone
backbone
backbone
adapter
strategy in Hollmann et al. (2023) (TabPFN-en-16) lowers variance but may increase bias. These observations highlight the need for a method that jointly optimizes both aspects, motivating the development BETA.
represent the encoder with parameters Φ. To enhance ex- pressiveness and incorporate nonlinearity, these layers are constructed using a sequence of operations as described by Gorishniy et al. (2021):
Existing approaches to improving TabPFN focus on context selection fine-tuning, but often introduce trade-offs in computational efficiency, scalability, and adaptability. A comparison in Table 1 highlights how different strategies affect bias, variance, and their ability to handle large datasets and high-dimensione features efficiently.
Es. (a) Linear (Dropout (ReLU (Linear (x)))) (2)
This transformation can extended by stacking multiple such blocks, allowing the encoder learn hierarchical rep- resentations suited for complex downstream tasks.
Given a query set X, and support set (X,, ya), we define their respective latent representations as:
3. BETA
Zg Ex Z, Egg (X.).
The posterior predictive distribution over the query labels is then expressed as:
To address both bias and variance issues of TabPFN observed during the previous experiments, we propose unified strategy for improving the performance of TabPFN. In addition to the inference stage, we introduce a fine-tuning stage. Our approach improves performance in both fine-tuning and inference, as shown in Figure 3. During fine-tuning, we refine input representations to better align the downstream data distribution with the pre-trained TabPFN, reducing both bias and variance. In the inference stage, we incorporate bootstrapped sampling to further reduce variance without additional computational overhead, compared to the ensemble-style TabPFN setting, as discussed in subsection 2.1.
go ygg Zg: (Z.: (..)....
This formulation ensures that the encoder effectively maps raw features into a structured space, enabling TabPFN to better capture relevant patterns while mitigating bias.
This fine-tuning method provides lightweight and effec- tive approach to reduce bias by better aligning the model's prior with the downstream tasks. It enables end-to end-to-end fine-tuning, facilitating a more efficient adaptation of the pre-trained TabPEN to the downstream tasks.
Mitigating Variance with Multiple Encoders. To further mitigate variance, we introduce multiple encoders, each learning distinct transformation of the input data. By jointly training these encoders while keeping the pre-trained TabPFN parameters 0 frozen, the model captures diverse feature representations, reducing variance.
Minimizing Bias with Encoder-Based Fine-Tuning. Our analysis in subsection 2.3 highlights the importance of min- imizing bias for improved generalization. To achieve this, we introduce a lightweight encoder while keeping the pre- trained TabPFN parameters frozen. This encoder transforms raw input features into a latent space that better aligns the pretrained TabPFN with the downstream data distribution.
For the k-th encoder, the support and query set representa- tions are encoded follows:
Let 3; E R denote the raw feature vector, and let Eq.
(k) = (k) (z.
N.
i=1 2 = [k)
(zx
N,
i=1
5

---

TabPEN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Trainable
Frozen
Concatenate
Element-wise add
Train Set
Query Set
Repeat
X
AI
E,(0)
Support Set
Repeat
Encoder E
Loss
LOSSS
Fine-tuning Stage
V.
TabPFN
Train Set
Inference Stage
Support Set (Bootstrap Sampling)
Mean
E,
X
X
Test Set
Repeat
Encoder
TabPFN
Figure 3: Overview of the proposed method, BETA, which consists of the fine-tuning stage and the inference stage. During the fine-tuning stage, only the encoder parameters equipped with Batchen are updated, while the parameters of the original pretrained model remain frozen.
where N, and N₂ denote the number of samples in the support set and the query set, respectively. The multiple encoders are trained jointly by minimizing the sum of the individual losses across all encoders. The optimization objective for the fine-tuning phase is:
where W is shared across all base models, and Tke 8k. by are specific to each base model (Wen al., 2020).
Bootstrapped Sampling for Variance Reduction in Infer- ence Stage. To further reduce variance during inference, we apply bootstrapped sampling, generating random subsets of the training
support sets. These used to compute predictions across multiple encoders, and their aggregation stabilizes the final output without additional computational overhead. For each encoder k. the bootstrapped support set [ bootstrap'
min Leotal log 40
Ug
(k)
(k)
Uz
Minimizing Lroral in Equation 3 jointly trains all encoders to generate TabPFN-cone compaaccoeetible latent representations, with the different encoder initializations encouraging diverse rep- resentations and thereby reducing variance and providing more stable predictions.
is D (k) bootstrap
Y bootstrap The final pre- diction is obtained by averaging over all encoders:
go E
(k) (Itest),
(E(1)
(k) bootstrap)
Y. (K)
bootssrapp)
Enhancing Performance with Batch Ensemble for Com- putational Efficiency. To further enhance performance without introducing additional computational cost, we inte- grate the Batch Ensemble technique into the encoder. Specif- ically, we replace the linear layers in Es with Batch En- semble versions, allowing the model to maintain diversity while avoiding the overhead of training multiple indepen- dent encoders. This technique introduces shared weight matrices and member-specine scaling factors, redueing the number of trainable parameters while preserving the benefits of ensembling.
This approach reduces variance while maintaining computa- tional efficiency, ensuring scalability and robustness.
Expanding to MultiClass Tasks Beyond 10 Classes. To address TabPFN's limitation in handling tasks with more than 10 classes, we integrate the Error-Coreettig Output Code (ECOC) (Dietterich Bakiri, 1995) strategy. Instead of training separate classifiers, distinct encoders within sin- gle model handle individual binary classification tasks, abling efficient multiclass classification without additional computational overhead.
The output the k-th base model for encoder layer is given by:
Summary of Our Approach. Our method reduces bias using a lightweight encoder to align input representations with the pre-trained TabPFN, which is effective for high-
(x=) = sk @ (W(rk 2 2:) + bb
6

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
dimensional data. Variance is reduced through inference with bootstrapped sampling, enabling better generalization on large datasets. For multiclass tasks, we integrate Error-Correcinng Output Codes (ECOC) to efficiently handle more than ten classes. Additionally, we preserve PFN-style batching, ensuring inference efficiency and scalability for large-scale and high-dimensional datasets.
(TabT) (Huang et al., 2020), and TabNet (Arik & Pfister,
2021). (3) Neighborhood- Methods: To explore neighbor-based strategies, we evaluate TabR (Gorishniy et al., 2024) and ModemnNC (MNCA) (Ye et al., 2025b), (4) Ensemble- Based Methods: We include methods that leverage ensemble strategies, such as NODE (Popov et al., 2020), GrowNet (Badirli et al., 2020), and TabM (Gorishniy et al., 2025). For TabM, we evaluate two configurations: TabM-32, which uses the setting of 32 base models in the original paper, and TabM-16, variant with 16 base models to facilitate fairer comparison with BETA, which also uses 16 base models by default. (5) TabPFN and Its Variants: Lastly, we compare with TabPFN and its recent variants, including KnnPFN (Nagler, 2023; Thomas et al., 2024), LocalPFN (Thomas et al., 2024), MixturePNN (Xu et al., 2025), and TuneTables (Feuer et al., 2024).
4. Experiments
4.1. Experiment Setup
Datasets. In our experiments, we evaluate BETA on one of the largest publicly available tabular benchmark TALENT (Ye et al., 2024b), which includes 120 binary classification datasets and 80 multi-class classification datasets. These datasets are collected from various sources such as UCL, OpenML, Kaggle, and others. To ensure fairness, we remove two datasets, "PizzaCutter3" and "PieChart3." as they overlap with TabPFN's validation set (Hollmann et al.. 2023). In addition, to validate the ability of BETA to handle high-dimensional feature datasets, we also include 20 high-dimentn -igh-dimensional datasets sourced from the scikit-feature repository.
Implementation Details. All datasets are pre-processed following the methodology outlined in Gorishniy et al. (2021). For deep learning- methods, we set the batch size to 1024. Hyper-paramede parameters for the compared methods are tuned using Optuna (Akiba et al., 2019), performing over 100 trials. The search ranges for hyper-parameters are determined based Liu et al. (2024) and the official implementations of cach method. Once the optimal pepee-parameters are identified, they are fixed for the final evaluation using 15 random seeds. To ensure a fair compar- ison, all TabPFN variants, including BETA, are evaluated using their default hyper-parameters without additional tuning. More details can be found in Appendix B.
Evaluation. For the TALENT datasets, we follow the eval- uation protocol from Gorishniy et al. (2021) and Ye et al. (2024b). Each dataset is randomly split into training, val- idation, and test sets with proportions of 64%, 16%, and 20%, respectively. For each dataset, we train each model using 15 different random seeds and calculate the average performance on the test set. We report accuracy as the evaluation metrie, where higher accuracy indicates better performance. For more detailed comparison with other TabPFN variants, we separately evaluate datasets with fewer than 10 classes, those with more than 10 classes, and high- dimensional datasets. Further details on the experimental setup, are provided in Appendix B.
4.2. BETA: State-of-the-Art Performance
We conducted pairwise significance testing using the Wilcoxon- Holm test (Demsar, 2006) among BETA and all the compared methods. To ensure a fair comparison, we selected 186 datasets from TALENT with fewer than 10 classes, as TabPFN and its variants are not capable of han- dling datasets with more than 10 classes. From Figure 4, it is evident that BETA outperforms other methods, even without
Methods Compared. We compare BETA against five categories of methods to evaluate its effectiveness compre- hensively: (1) Classical Machine Learning Algorithms: This category includes widely used classical approaches such as Support Vector Machines (SVM), K-Nearest Neighbors (KNN), and tree-based methods like Random Forest (RForest) (Breiman, 2001), XGBoost (XGB) (Chen & Guestrin, 2016), CatBoost (CatB) (Prokhorenkova et al., 2018), and LightGBM (LightG) (Ke et al., 2017). (2) Tabular Deep Learning Models: We consider state-of- the-art deep learning models for tabular data, including MLP, ResNet, FT-Transformer (FT-T) (Gorishniy et al., 2021), MLP-PLR (Gorishniy et al., 2022), DCNv2 (Wang et al., 2021), AutoInt (Song et al., 2019), SNN (Klam- bauer et al., 2017), ExcelFormer (ExcelF) (Chen et al., 2024), DANets (Chen et al., 2022), TabTransformer
tuning. This includes methods based on nearest neighbors such as TabR and ModernNCA. ensemble-based approaches like TabM, traditional tree mod- els, and other TabPFN variants. These results underscore the significant potential of pre-trained models for tabular data and demonstrate the effectiveness of our proposed method in adapting to downstream datasets.
Scale to Large Datasets. We evaluated BETA and the variants of TabPFN on 173 tabular datasets, excluding those with more than 100 features, which TabPFN cannot handle. These datasets were sorted by the number of rows in ascending order, and the relative improvement of these methods over TabPFN was shown in Figure 5. The results clearly show that the performance improvement
7

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
34
31
28
22
19
18
13
Beta
Relative Improvement
LocalPFN
NCM
BREA
1.08
TabPFN
Dammy
TabM-32
NB
Tablt
1.00
TabNet
MNCA
GrowNet
RealMLP
1:24
SVM
CatB
KNN
LocalPFN
LR
LightG
1.02
TabT
MixturePNN
NODR
TabM-16
DANets
XGB
ExcelF
KmaPFN
a.b
s/a
SNN
FT-T
logao(Dataset Size)
TabPFN
MLP--LR
Tune Tables
ResNot:
MLP
DCN22
Figure 5: The relative improvement of LocalPFN (Thomas et al., 2024) and BETA (Ours) over TabPEN across 173 tabular datasets. The x-axis represents the logarithm (base 10) of dataset size (number of rows). The curves represent smoothed results to better illustrate trends in performance improvement as dataset size increases. For clarity of presen- tation, the curves have been smoothed.
Autolnt
RForest
Figure 4: The critical difference diagrams based on the Wilcoxon- Hooom test with significance level of 0.05 to de- tect pairwise significance for TALENT datasets with fewer than 10 classes. The correspondence between method names and their abbreviations can be found in (Ye et al., 2024b).
fication tasks with larger number of categories.
by BETA becomes more pronounced as the dataset size increases, highlighting the scalability and effectiveness of our approach in handling larger datasets. Notably, on the largest datasets in the benchmark, LocalPFN slightly outperforms BETA in terms of accuracy. However, on more other datasets, BETA demonstrates superior performance.
4.3. Ablation Study
We perform an ablation study on the classification datasets of TinyBench2 (Ye et al., 2024b) to examine the contribu- tions of key components, including the number of encoder paths, periodic activation, Batch Ensemble, and partial parameter tuning. Our results indicate that each of these design elements plays a crucial role in enhancing BETA'S 'ss performance.
Handling High-Dimen Dimensionnl Datasets. To assess the ef- fectiveness of BETA on high high-dimensional datasets, we con- ducted experiments on 20 datasets with extremely high fea-
ture dimensions, as detailed in Table These datasets were selected evaluate the scalability and adaptability of differ- ent methods in complex feature spaces. The average ranks of the compared methods are summarized in Figure 7. The results show that BETA achieves the best performance, at- taining the lowest average rank and outperforming all other methods. TabM and MLP also demonstrate competitive results, ranking second and third, respectively. In contrast, traditional models such RandomForest and XGBoost, as well deep learning-basee -base ModernNCA, exhibit lower ranks, highlighting their limitations in high-dimensional set- tings. Due to memory constraints, we were unable to com- pare with methods such as FT-T and TabR in this setting.
Influence of Encoder Path Count on Performance Im- provement over TabPFN. Figure 6 (a) presents the results of an ablation experiment designed to investigate the im- pact of different encoder path counts on the performance of BETA. The primary goal is to explore how the number of encoder paths influences the model's relative improve- ment over TabPFN. We display the relative improvement for BETA with different encoder path configurations. From the plot, we observe that as the number of encoder paths (denoted by K) increases, the relative performance improve- ment of BETA over TabPFN becomes more pronounced. This trend indicates that a higher number of encoder paths contributes positively to the model's s ability to outperform TabPFN, suggesting that more encoder paths enhance the model's expressive power and its ability to capture more complex patterns in the task.
Performance on MultiClass Classification Tasks with More Than 10 Classes. In addition, we further inves- tigated the performance of BETA on classification tasks with more than 10 categories. We selected 12 classification datasets with more than 10 classes from TALENT (Ye et al., 2024b) and compared BETA with other methods. We report the mean accuracy of each method across the 12 datasets in Figure& 8 and find that BETA outperforms the compared methods, demonstrating its superiority on multiclass classi-
Impact of Some Key Components. Figure 6 (b) visualizes the effect of three key components-perionee activation, Batch Ensemble, and partial parameter tuning- on the rel- ative performance improvement of BETA over TabPFN. In this experiment, we compare the original BETA with three
8

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Beta
w/o
w/o PA
TabPFN
TabP FN
-0.01
0,00
0.01
0.02
0.03
0.04
0.05
-0.01
0.00
0.01
0.02
0.03
0.04
0.05
0.06
Relative improvement over TabPFN
Relative improvement over TabPFN
(a) Infulence
(b) Influence of some important components
Figure 6: (a) The box plot demonstrates the effect of varying the number of encoder paths, K, on the relative improvement over TabPEN. The horizontal axis represents the relative improvement in performance, while the vertical axis lists numbers of different encoder path settings; (b) This box plot illustrates the effect of different components periodic -prriooi activation (PA), Batch Ensemble (BE), and partial parameter tuning (PPT) on the relative improvement of BETA over TabPFN. The plot compares the original BETA with three variations: (1) BETA with partial parameter tuning (w/o PPT), (2) BETA without periodic activation (w/o PA), and (3) BETA without Batch Ensemble (w/o BE).
the model's ability to outperform TabPFN, and the absence of any of these components compromises the model's 's ability to leverage its full potential while also increasing computa- tional resource usage. For more comprehensive and detailed ablation experiments, please refer Appendix D.5.
Beta
MLP
3.50
Methods
RForest
TabM
4.03
Pgate
5. Conclusion
XGB
4.50
MNCA
In this paper, we propose BETA, a novel approach that enhances the performance of TabPFN by simultaneously addressing both bias and variance. Through a combination of lightweight encoder-based fine-tuning and bootstrapped sampling, BETA significantly improves TabPFN's adaptabil- ity to high-dimensional, large-scale, and multiclass clas- sification tasks. Our method efficiently reduces bias by aligning downstream data distributions with the pre-trained TabPFN and reduces variance through diverse latent rep- resentations and robust inference techniques. Experimen- results on over 200 benchmark datasets demonstrate that BETA consistently outperforms or matches offte-of-the- methods, highlighting its potential handle complex tabular data tasks with enhanced scalability and robustness. These contributions provide scalable and effective solution for leveraging TabPFN in real-world applications, ensuring its success across broad range of tabular data challenges.
KNN
6.05
3
Average Rank
Figure 7: Average ranks of methods on 20 datasets. We compare BETA with TabM, KNN, MLP, XGBoost (XGB), RandomForest (RForest), ProtoGate (Pgate) (Jiang et al., 2024b), and ModernNCA (MNCA). Lower ranks indicate better performance.
variations: without periodic activation, without Batch Ensemble, and one with full parameter tuning. The plot reveals that all of these components contribute posi- tively to the model's performance. Specifically, removing any of these elements leads to a noticeable decline in relative improvement over TabPFN, highlighting that all parts are essential for achieving the best performance. Notably, the variation without partial parameter tuning (PPT) shows the highest upper-bound improvement, but it also incurs signif- icantly higher computational costs. Moreover, the median relative improvement decreases compared to BETA, further emphasizing the superior efficiency of our approach. This suggests that periodic activation, Batch Ensemble, and par- tial parameter tuning each play a crucial role enhancing
Acknowledgements
This work partially supported by National Key R&D Pro- gram of China (2024YFE0002800), NSFC (62376118), Key Program of Jiangsu Science Foundation (BK20243012) Collaborative Innovation Center of Novel Software Tech- nology and Industrialization.

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Impact Statement
Borisov, V., Leemann, T., SeBler, K., Haug. J., Pawelczyk, M., and Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Networks and Learning Systems, 35(6),7499-7515 2024b.
Our work advances the field of tabular learning and in- context learning (ICL) by introducing a method that sig- nificantly improves the scalability, adaptability, and per- formance of TabPFN. Given that tabular data is widely used across industries such as finance, healthcare, and e- commerce, achieving state-of-the-art classification accuracy has far-reaching benefits. Our approach enhances the usabil- ity of tabular foundation models (TFMs), providing a more effective and scalable alternative to conventional methods while preserving inference efficiency. We hope that our results inspire further research into ICL-driven approaches for tabular data and their applications in broader realdworld tasks. As our method is built on pre-trained TabPFN, its deployment in high-stakes applications should consider po- tential limitations, such as reliance on the model's prior and sensitivity to dataset characteristics. We do not fore- see significant negative societal impacts arising from our work.
Breiman, L. Random forests. Machine Learning, 45(1):
5-32, 2001.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J.. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I,, and Amodei, D. Language models are few-shot learners. In NeurlPS, pp. 1877-1901, 2020.
Cai, H.-R. and Ye, H.-J Understanding the limits of deep tabular methods with temporal shift. CoRR, abs/2502.20260, 2025.
References
Cao, L. and Tay, F. E. H. Financial forecasting using support vector machines. Neural Computing and Applications, 10(2):144-12, 2001.
Aghajanyan, A., Gupta, S., and Zettlemoyer L. Intrin- sic dimensionality explains the effectiveness of language model fine-tuning. In ACL-I/CNLP, pp. 7319-7328. As-
sociation for Computational Linguistics, 2021.
Chen, J., Liao, K., Wan, Y., Chen, D. and Wu, J. Danets: Deep abstract networks for tabular data classification and regression. In AAAI, pp. 3930-3938, 2022.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In KDD, pp. 2623-222222222 2203-2631,22 2019.
Chen, J., Liao, K., Fang, Y., Chen, D., and Wu, J. Tabcaps: A capsule neural network for tabular data classification with bow routing. In ICLR, 2023a.
Alan, J., Tennison, L.,, Jonathan, C., Fergus, I., and van der Schaar Mihacla. Tangos: Regularizing tabular neural networks through gradient orthogonalization and special- ization. In ICLR, 2023.
Chen, J., Zhang, A., Shi, X., Li, M., Smola, A., and Yang. D. Parameter-efffcient fine-tuning design spaces. In ICLR, 2023b.
Arik, Ô. and Pfister, T. Tabnet: Attentive interpretable tabular learning. In AAAI, pp. 6679-6687, 2021.
Chen, J., Yan, J., Chen, Q. Chen, D. Z., Wu, J., and Sun, J. Can a deep learning model be a sure bet for tabular prediction? In KDD, pp. 288-2986669999 188-296,70 2024.
Badirli, S., Liu, X., Xing, Z., Bhowmik, A., and Keerthi, S. Gradient boosting neural networks: Grownet. CoRR, abs/2002.07971, 2020.
Chen, K.-Y., Chiang, P.-.., Chou, H.-R., Chen, T.-W., and Chang, T.-H. Trompt: Towards a better deep neural network for tabular data. In ICML, pp. 4392-4434, 2023c.
Barry, B. and Ronny, K. Adult. UCI Machine Learning Repository, 1996.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In KDD, pp. 785-794, 2016.
Basri, R., Galun, M., Geifman, A., Jacobs, D. W., Kasten, Y., and Kritchman, S. Frequency bias in neural networks for input of non-uniform density. In ICML, pp. 685-694, 2020.
Demsar, J. Statistical comparisons of classifiers over multi- ple data sets. Journal of Machine Learning Research, 7: 1-30, 2006.
Borisov, V., Leemann, T., Seßler, K., Haug. J., Pawelezyk, M., and Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Nenworks and Learning Systems, 35(6):749--7519 2024a.
den Breejen, F., Bae, S., Cha, S., and Yun, S. Why in- context learning transformers are tabular data classifiers. CoRR, abs/2405.1 13396, 2024.
10

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Dietterich, T. G. and Bakiri, G. Solving multiclass learning problems via error-correctngg output codes. Journal of Artificial Intelligence Research, 2:263-286, 1995.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A.,
Körfer, M., Hoo, S. B., Schirmeisterr, R. T., and Hutter, F. Accurate predictions small data with tabular foundation model. Nature, 01 2025.
Feuer, B., Hegde, C., and Cohen, N. Scaling tabpfn: Sketch- ing and feature selection for tabular prior-data fitted net- works. CoRR, abs/2311 10609, 2023.
Holzmüller, D., Grinsztajn, L., and Steinwart, I. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. In NeurlPS, pp. 26577-26655 2024.
Feuer, B., Schirrmeister, R. T., Cherepanova, V., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C. Tunetables: Context optimization for scalable prior-data fitted networks. In NeurlPS, pp. (3430-83466) 2024.
Hoo, S. B., Müller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. CoRR, abs/2501 1.02945, 2025.
Fu, Z., Yang, H., So, A. M., Lam, W., Bing, L.,, and Collier, N. On the effectiveness of parameter-efficient fine-tuning. In AAAI, pp. 12799-12880 2023.
Houlsby, N., Giurgiu, A., Jastrzebski, S.. Morrone, B., de Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for NLP. ICML., pp. 2790-2799, 2019.
Gardner, J., Perdomo, J. C., and Schmidt, L. Large scale transfer learning for tabular data via language modeling. CoRR, abs/2406. 12031, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. In NeurlPS, pp. 18932-18943, 2021.
Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings. CoRR, abs/201 2.06678. 2020.
Gorishniy, Y., Rubachev, I., and Babenko, A. On embed- dings for numerical features in tabular deep learning. In NeurlPS, pp. 14991-250044 2022.
Jiang, J.-P., Ye, H.-J,, Wang, L., Yang. Y., Jiang, Y., and Zhan, D.-C. Tabular insights, visual impacts: Trans- ferring expertise from tables images. In ICML, pp. 21988-222 22009, 2024a.
Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR, 2024.
Jiang, J.-P., Liu, S.-Y., Cai, H.-R., Zhou, Q. and Ye, H.-J. Representation learning for tabular data: A comprehen- sive survey. CoRR, abs/2504.16109 2025.
Gorishniy, Y., Kotelnikov, A., and Babenko, A. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. In ICLRR 2025.
Jiang, X., Margeloiu, A., Simidjievski, N., and Jamnik, M. Protogate: Prototype- bbstlyppeeeeddddddeee neural networks with global- to-local feature selection for tabular biomedical data. In ICML, pp. 21844-21878, 2024b.
Grinsztajn, I.., Oyallon, E., and Varoquaux, G. Why do tree- based models still outperform deep learning typical tabular data? In NeurIPS, pp. 507-520, 2022.
Kasneci, G. and Kasneci, E. Enriching tabular data with contextual LLM embeddings: A comprehensive ablation study for ensemble classifiers. CoRR. abs/2411.01644 2024.
Han. Z., Gao, C., Liu. J., Zhang, J., and Zhang, S. Q. Parameter-efficien fine-tuning for large models: A com- prehensive survey. CoRR, abs/2403. 146088 2024.
Hassan, M. R., Al-Insaif, S., Hossain, M. I., and Kamruz- zaman, J. A machine learning approach for prediction of pregnancy outcome following IVF treatment. Neural Computing and Applications, (2(7):2283-2297) 2020.
Ke, G.. Meng, Q., Finley, T. Wang, T., Chen, W., Ma. W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, pp. 3156-3154, 2017.
He. Н., Cai, J.. Zhang, J., Tao, D., and Zhuang, B. Sensitivity-awarc visual parameter-efficicn fine-tuning. In ICCV, pp. 11791-11801. IEEE, 2023.
Kim, M. J., Grinsztajn, I.,, and Varoquaux, G. CARTE: pretraining and transfer for tabular learning, In ICML, pp. 23843-23866, 2024.
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. Tabpfn: A transformer that solves small tabular classifi- cation problems second. In ICI., 2023.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, Self normaliinn neural networks. In NIPS, pp. 971-980, 2017.

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Koloski, B., Margeloiu, A., Jiang, X., X., Skrlj, B., Simidjievski, N., and Jamnik, M. LLM emheddings for deep learning on tabular data. CoRR, abs/2502. 1156, 2025.
Margeloiu, A., Jiang, X., Simidjievski, N., and Jamnik, M. Tahebm: A tabular data augmentation method with distinct class-specific energy-based models. In NeurIPS, pp. 72094-7022 12122,1144, 2024b.
Koshil, M., Nagler, T., Feurer, M.. and Eggensperger, K. Towards localization via data embedding for TabPFN, In NeurlPS Workshop, 2024.
McElfresh, D. C., Khandagale, S., Valverde, J., C., V. P., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? In NeurlPS, pp. 76936-76369, 2023.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In EMNLP, pp. 3045-3059. Association for Computational Linguisties, 2021.
Nagler, T. Statistical foundations of prior-data fitted net- works. In ICML, pp. 25660-25676, 2023.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pp. 4582-4597. Association for Computational Linguistics, 2021.
Nederstigt, L. J., Aanen, S. S., Vandic, D., and Frasincar, F. Floppies: a framework for large-scale ontology pop- ulation of product information from tabular data in e- commerce stores. Decision Support Systems, 59:296-311, 2014.
Lialin, V., Deshpande, V., and Rumshisky, A. Scaling down to scale up: guide to parameter-efficien fine-tuning. CoRR, abs/2303.15647. 2023.
Popov, S., Morozov, S., and Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020.
Liu, L., Fard, M. M., and Zhao, S. Distribution embed- ding networks for generalization from diverse set of classification tasks. Transactions on Machine Learning Research, 2022, 2022.
Prokhorenkova, L. O., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features. NeurlPS, pp. 6639-6649, 2018
Liu, S.-Y., Cai, H.-R., Zhou, Q.-I.,, and Ye, H.-J. TAL- ENT: A tabular analytics and learning toolbox. CoRR, abs/2407.04057, 2024.
Qu, J., Holzmüller, D., Varoquaux, G., and Morvan, M. L. TabICI: A tabular foundation model for in-context learn- ing on large data. In JCML, 2025.
Liu, S.-Y., Zhou, Q. and Ye, H.-J Make still further progress: Chain of thoughts for tabular data leaderboard. CoRR, abs/2505. 13421, 2025.
Rubachev, I., Kartashev, N., Gorishniy, Y., and Babenko, A. Tabred: A benchmark of tabular machine learning in-the-will CoRR, abs/2406. 19380, 2024.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In ICLR, 2019.
S., M., P., R., and P., C. Bank Marketing. UCI Machine Learning Repository. 2014.
Ma, J., Dankar, A., Stein, G.. Yu, G., and Caterini, A. L. Tabpfgen tabular data generation with tabpfn. CoRR, 0ls/2406.05211..1 2024a.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Den- ton, L., Ghasemipour, S. K. S., Lopes, R. G., Ayan, B. K., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, pp. 36479-36494 2022.
Ma, J., Thomas, V., Hosseinzadeh, R., Kamkari, H., Labach, A. Cresswell, J. C., Golestan, K., Yu, G., Volkovs, M., and Caterini, A. L. Tabdpt: Scaling tabular foundation models. CoRR, abs/2410.18164, 2024b.
Schwartz, L. M., Woloshin, S., and Welch, H. G. The drug facts box: providing consumers with simple tabular data on drug benefit and harm. Medical Decision Making, 27 (5):6556622, 2007.
Ma, J., Thomas, V., Yu, G., and Caterini, A. L. In-context data distillation with tabpfn. CoRR, abs/2402.006711, 2024c.
Mao, Y., Mathias, L., Hou, R., Almahairi, A., Ma, H., Han, J. Yih, S., and Khabsa, M. Unipelt: A unified framework for parameter-efficient language model tuning. ACL, pp. 6253-6264, 2022.
Song, Shi, C., Xiao, Z., Duan, Z. Xu, Y., Zhang, M., and Tang, J. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, pp. 1161 1170, 2019.
Margeloiu, A., Bazaga, A., Simidjievski, N., Liò, P., and Jamnik, M. Tabmda: Tabular manifold data augmenta- tion for any classifier using transformers with in-context subsetting. CoRR, abs/2406.01 2024a.
Subasi, A. Medical decision support system for diagnosis neuromuseular disorders using dwt and fuzzy support vector machines. Computers in Biology and Medicine, 42(8))806-815 2012.
12

---

TabPEN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Thomas, V., Ma, J., Hosseinzadeh, R., Golestan, K., Yu, G., Volkovs, M., and Caterini, A. L. Retrieval & fine-tuning for in-context tabular models. CoRR, abs/2406.05207, 2024.
Ye, H.-J., Zhou, Q-L.,, and Zhan, D.-C. Training-free generalization heterogeneous tabular data via meta- representation. CoRR, (bss/2311.00555 2023.
Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q.-L,, and Zhan, D.-C. closer look deep learning methods on tabular datasets. CoRR, 00s/2407.00956 2024b.
Tschalzev, A., Purucker, L., Lüdtke, S., Hutter, F., Bartelt, C., and Stuckenschmidt, H. Unreflected use of tabular data repositories can undermine research quality, In ICLR Workshop, 2025.
Ye, H.-J., Liu, S.-Y., and Chao, W.-L. A closer look at tabpfn v2: Strength, limitation, and extension. CoRR, abs/2502.17361, 2025a.
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter-efficient tuning of pre-trained mod- els using dynamic search-free low-rank adaptation. EACL, pp. 1266-3279 2023.
Ye, H.-J., Yin, H.-H., and Zhan, D.-C. Revisiting nearest neighbor for tabular data: deep tabular baseline two decades later. In ICLR, 2025b.
van Breugel, B. and van der Schaar, M. Position: Why tabular foundation models should be a research priority. 2024.
Zhang. J. O., Sax, A., Zamir, A., Guibas, L. J., and Malik, J. Side-tuning: A baseline for network adaptation via additive side networks. In ECCV, pp. 698-714, 2020.
In ICMI., pp.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Zhou, Q.-L., Ye, H.-J., Wang, L., and Zhan, D.-C. Unlock- ing the transferability of tokens in deep models for tabular data. CoRR, abs/2310.15149, 2023.
Wang, R., Fu, B., Fu, G., and Wang, M. Deep & cross network for ad click predictions. In ADKDD, pp. 12:1- 12:7, 2017.
Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. Xtab: Cross-table pretraining for tabular transformers. In ICML, pp. 43181 11884444444201100111111111104444 2023.
Wang, R., Shivanna, R., Cheng, D. Z.,, Jain, S., Lin, D., Hong, L., and Chi, E. H. DCN V2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW, pp. 1785-17977 2021.
Wang, and Sun, J. Transtab: Learning transferable tabular transformers across tables. In NeurlPS, pp. 2902-2915, 2022.
Wen, Y., Tran, D., and Ba, J. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2020.
Wu, J., Chen, S., Zhao, Q., Sergazinov, R., Li, C., Liu, S., Zhao, C., Xie, T., Guo, H., Ji, C., Cociorva, D., and Brun- zell, H. Switchtab: Switched autoencoders are effective tabular learners. In AAAI, 15924-15933, 2024.
Xu, D., Cirit, O., Asadi, R., Sun, Y., and Wang, W. Mixture of in-context prompters for tabular pfns. In ICLR, 2025.
Yan, J., Zheng, B., Xu, H., Zhu, Y., Chen, D. Z., Sun, J., Wu, J., and Chen, J. Making pre-trained language models great on tabular prediction. In ICIR, 2024.
Yang, Y., Wang. Y., Liu, G., Wu, L.. and Liu, Q. Unitabe: A universal pretraining protocol for tabular foundation model in data science. In ICLR, 2024.
Ye, H., Fan, W., Song. X., Zheng. S., Zhao, H., dan Guo, D., and Chang, Y. Ptarl: Prototype-basee based tabular representa- tion learning via space calibration. ICLR, 2024a.
13

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
The Appendix consists of four sections:
1. Appendix A: Related work.
2. Appendix B: Datasets and implementation details.
3. Appendix C: Hardware and limitations.
4. Appendix D: Additional experimental results
A. Related Work
A.1. Tabular Data Learning
Tabular data is one of the most widely used dataset types in machine learning. Gradient-hoosted decision trees (GBDTs) (Chen & Guestrin, 2016; Prokhorenkova et al., 2018; Ke et al., 2017), remain a strong baseline for tabular tasks due their efficiency and high performance. As an casemble-based method, GBDTs construct multiple decision trees to iteratively minimize the residual loss. With the advancement of deep learning, an increasing number of studies have explored using deep learning methods for tabular data prediction (Borisov et al., 2024a; Jiang al., 2025). These approaches include MLP variants (Klambauer et al., 2017; Gorishniy et al., 2021; 2022; Holzmüller et al., 2024), neural networks specifically designed for tabular structures (Wang et al., 2017; 2021; Chen et al., 2023a), attention based models (Song et al., 2019; Huang et al., 2020; Gorishniy et al., 2021; Chen et al., 2024; Jiang et al., 2024a; Zhou et al., 2023), methods incorporating regularization (Ye et al., 2024a; Alan et al., 2023; Wu et al., 2024), tree-mimic methods (Arik & Pfister, 2021; Popov et al., 2020; Badirli et al., 2020), and context-based methods (Gorishniy et al., 2024; Ye et al., 2025b). Despite these advancements, recent benchmarks (Grinsztajn et al., 2022: McElffreh et al.. 2023; Ye et al., 2024b) have consistently demonstrated that gradient- boosted decision trees (GBDTs) outperform deep learning in tabular prediction tasks. The superior performance of GBDTs can be attributed to two main factors: (1) their ability to handle heterogeneous tabular datasets, which often describe high-frequency target functions (Basri et al., 2020; Grinsztajn et al., 2022). and (2) the ensemble nature of GBDTs. Prior attempts to introduce ensemble-like mechanisms into tabular deep learning (Badirli et al., 2020; Popov et al., 2020; Chen et al., 2023c), have not been widely successful (Grinsztajn et al., 2022; Ye et al., 2024b). However, recent works like TabM (Gorishniy et al., 2025) integrate Batch Ensemble (Wen et al., 2020) techniques into the tabular domain, showing how efficient ensembling can be achieved with deep learning models.
A.2. Tabular Foundation Models
While tabular foundation models (TFMs) are not as developed as foundation models other domains, such as computer vision (Saharia et al., 2022) and natural language processing (Brown et al., 2020), recent efforts have introduced various architectures to bridge this gap (van Breugel & van der Schaar, 2024). Some approaches aim to explore model components that can be shared across datasets (Liu et al., 2022; Zhu et al., 2023), while others focus utilizing the semantic information inherent in tabular datasets (Wang& & Sun, 2022; Yan et al., 2024; Gardner et al., 2024; Yang et al., 2024; Kim et al., 2024; Kas- models, such as TabPFN (Hollmann et al., 2023; 2025; Ye et al., 2025a) and TabICL (Qu et al., 2025) stand out for their exceptional performance and efficiency on small datasets. By leveraging the in-context learning capability of transformer (Brown et al., 2020), they can make predictions for unseen instances without parameter updates. However, TabPFN faces limitations related dataset size and feature dimension- ality. To address these challenges, some studies have explored improvements in its architecture and training paradigm (den Breejen et al., 2024; Ma et al., 2024b), while others focus on adaptation techniques that expand TabPFN's applicability to broader range of downstream tabular datasets (Feuer et al., 2024; Thomas et al., 2024; Koshil et al., 2024). Our work falls into the latter category, offering a straightforward, efficient, and effective approach to align TabPFN with downstream datasets.
neci & Kasneci, 2024; Liu et al., 2025; Koloski et al., 2025). Transformer-base
A.3. Parameter-efficient Fine-Tuning
Some approaches for adapting Tabular Foundation Models (TFMs) to downstream tasks simply fine-tune the entire model (Thomas et al., 2024; den Breejen et al., 2024), which leads to performance improvements but incurs high computa- tional and storage costs. Parameter-efficient fine-tuning (PEFT) offers a solution to the challenge by enabling adaptation with a minimal number of trainable parameters (Lialin et al., 2023). Based on their operational mechanisms, Han et al. (2024) broadly categorized PEFT methods into four paradigms:
14

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Additive Methods. These methods incorporate additional lightweight modules into the model architecture, such as adapters (Houlsby al., 2019; Zhang et al., 2020; Xu et al., 2025) or soft prompts (Li & Liang, 2021; Feuer et al., 2024). Selective Methods. Instead of introducing new parameters, selective methods strategically identify and update the most relevant parameters while freezing the rest (Fu et al., 2023; He et al.. 2023).
Reparameterized Methods. These methods employ low-rank decomposition or equivalent transformations to reduce the parameter space during fine-tuning (Aghajanyan et al., 2021; Hu et al., 2022; Valipour et al., 2023).
Hybrid Methods. By combining the strengths of multiple PEFT strategies, hybrid methods create a unified framework that enhances fine-tuning performance while maintaining efficiency (Mao et al., 2022; Chen al., 2023b).
Our proposed method adopts parameter-efficient tuning focused input feature adaptation. This design is motivated by the unique characteristics of tabular data: Tabular datasets are inherently heterogeneous, with varying structures and feature distributions across different datasets (Zhou et al., 2023; Ye et al., 2023; Borisov et al., 2024b). Adapting TabPFN through input feature alignment effectively mitigates the constraints on input dimensionality, enhancing its applicability across a wider range of tasks. By employing parameter-efficient fine-tuning, we align TabPFN with downstream datasets, addressing its existing limitations (Hollmann et al., 2023).
B. Datasets and implementation details
In this section, we outline the descriptions of the datasets used in the experiments and the preprocessing steps applied to them before training. Additionally, we will describe the implementation details of BETA and the comparison methods.
B.1. Datasets information
For both the main experiments and the classification tasks with more than 10 categories, we use the current largest tabular benchmark (Ye et al., 2024b), which includes 120 binary classification datasets and 80 multiclass classification datasets, spanning diverse domains such as healthcare, biology, finance, education, and physics. For more detailed information about these datasets, please refer to Ye et al. (2024b).
For each dataset, we randomly select 20% of the instances to form the test set. The remaining 80% is further split, with 20% reserved as validation set. This validation set is used for hyperparameter tuning (for the comparison methods) and early stop- ping. The hyperparameters that yield the best performance on the validation set are selected for final evaluation on the test set.
Remark: BETA are based on pre-trained models TabPFN (Hollmann et al., 2023). To ensure a fair comparison, we do not perform hyperparameter search, and instead use the default hyperparameters for all TabPFN-based methods
For high-dimensional datasets, we obtained the data from scikit-feature repository, and excluded those with more than 10 classes. This resulted in a final set of 20 datasets, as shown Table 2. The dataset splitting follows the same approach as in the main experiments, but due the smaller number of instances in these datasets, we the default hyperparameters for the experiments and do not perform hyperparameter search. The validation set is only used for early stopping. For each dataset, we perform five random splits and run all methods three times on each split using three different seeds (0, 1, 2), resulting in a total of 15 runs per dataset. The final results are reported as the mean of these 15 runs.
B.2. Dataset Pre-processing
Unless otherwise specified, we adopt the data preprocessing pipeline outlined by Gorishniy et al. (2021). For numerical features, we standardize by subtracting the mean and scaling the values to unit variance. Categorical features are converted into a model-compatible format using one-hot encoding.
B.3. Implementations
BETA. For all experiments except for the ablation study, we set the context size to 1000 and fixed the number of bootstrap sampling iterations to 16. For the feature transformation encoder, the main structure is a two-layer MLP, with both the hidden and output dimensions set to 100, using the ReLU activation function. In addition, to enhance the expressive power of the encoder, we also apply periodic activation (Gorishniy et al., 2022). The initialization of Batch Ensemble is inspired by Gorishniy et al. (2025). During fine-tuning, we use the pre-trained model checkpoint', and fine-tuning is performed
ciciissdddffffo_edec diff real Lettrreeeeeeeeeeeee checkpoint n 0. 0_epoch_42..clk
15

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Table 2: Dataset Information for High--Dimensionl Data Experiments: A collection of 20 datasets with varying numbers of instances, features, and classes used in our high-diman nigh-dimensiooal experiments.
Dataset
#Instances
#Features
#Classes
Dataset
#Instances
#Features
#Classes
BASEHOCK
1993
4862
2
lung discrete
73
325
7
PCMAC
1943
3289
2
WarpPIE11P
210
2420
10
RELATHE
1427
4322
2
oriraws10P
100
10304
10
ALLAMI
72
7129
2
Prostate_ct __
102
5966
2
CLL_SUB_III
111
11340
3
SMK CI___ANN177
187
19993
2
colon
62
2000
2
warpARIOP
130
2400
10
GLI_85
85
22283
2
arcene
200
10000
2
GLIOMA
50
4434
4
gisette
7000
5000
2
leukemia
72
7070
2
madelon
2600
500
2
lung
203
3312
5
TOX 171
171
5748
4
using the AdamW (Loshchilov Hutter, 2019) optimizer with a learning rate of 0.003, weight decay of le-5, and a batch size of 1024. In the fine-tuning phase, only the encoder parameters are updated, while the pre-trained TabPFN parameters nigh-dimensiooal datasets, due the large number of features and the limitations of device
frozen. For experiments memory, we do not use periodic activation. In the classification experiments with more than 10 classes, length of 32, which means there are 32 paths in the encoder. Additionally, we observed that for certain datasets, simply adjusting the output dimension of the pre-trained model and fine-tuning it can yield strong results (Feuer et al., 2024). Based on validation set performance, we choose between ECOC and fine-tuning the output layer ML.P approaches.
code
TabPFN and its variants. For TabPFN, we use the implementation from TALENT (Liu et al., 2024); For Tanelables, we adopt the official code² and use the prompt tuning mode without performing full model fine-tuning, For LocalPFN, since the code has not been released, replicate the hyperparameters provided in the original paper. Specifically, of neighbors k 1000. During experiments, we found that fine-tuning LocalPFN with smaller learning rate yields better performance, so we set the learning rate to le-5; For MixturePFN, we reproduce the model using the hyperparameters recommended in the original paper. The number of training samples in the prompt is set to B = 3000, and the number of experts set to the training set size divided by B, rounded up to the nearest integer.
set the number
Other Methods from TALENT. For the other methods TALENT, we either use the results provided (Ye al., 2024b) or perform a hyperparameter search using the hyperparameters provided by TALENT. In the case of hyperparameter tuning, we conduct 100 trials of hyperparameter search for each m For each search, the experiments using 15 different seeds and report the average results across these runs.
B.4. Details of TabPFN
In the original TabPFN approach, the model processes the input data in two phases: pre-training and inference. The pre-training phase involves training the model synthetic data, while the inference phase is used to make predictions on new, unseen data.
For the input, we consider a training dataset Dtrain = train Itraiu), where Xtrain E IR Nerain xd represents the feature matrix, and 1/rain E R represents the corresponding labels. Here, Nerain denotes the number of training samples, and the number of features. For datasets with fewer features than the fixed input dimensionality dmaa. zero-padding applied extend the feature vectors the required size dmax. Specifically, each feature vector X Bd is padded with zeros form X, E Ruass as follows:
X; = [X 0] E Rdaas
The zero-padded feature matrix Xusin is then used during inference for making predictions. "htpss//githuh.com///// hatps///ghhha..mmm//l///ooooooo http:///w.......AAAAAAAAAAAAAena CAAAAAAATAAAAAAAAAAAAAAATTCCIEE
16

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
For inference, consider support set S containing the training examples and a query containing the new, unseen instances for which wish to make predictions. The query consists of Ntest test samples, and the goal is predict the corresponding labels yhest.
TabPFN uses PFN-style batching for inference, where both the support set S and query set Q are batched together into a single prompt, which is then fed into the pre-trained model for prediction. This batching process allows the model to utilize the support set as context for making predictions on the query set.
During the processing, the features of both the support set and the query set are transformed into token representations, The support tokens Lupport are obtained as:
Luppport = Xsupport W₂ + Vaupport W
where X Ssupport E dd is the zero-padded feature matrix of the support set, Wa, E IRdmmax Xdeaken is the embedding matrix for the features, and Wy Riokken is the embedding matrix for the labels. The query tokens Lquery are similarly transformed as:
Lquery = X querr W
where X Xquery E X represents the query set of test samples.
These token representations are then passed through a standard transformer model, which includes a special attention mechanism. The attention mask ensures that the support tokens only attend to other support tokens, and query tokens can attend to both the support tokens and themselves. Query tokens do not attend to other query tokens, preventing any information leakage between query samples during inference. Finally, the output tokens corresponding to the test instances are extracted and mapped to 10-class logits for classification.
B.5. Implementation of BabPFN-Bagging
TabPFN-Bagging is a variance reduction technique that leverages bootstrapped sampling to create diverse support sets for model inference. Unlike standard ensemble approaches, which require training multiple independent models, TabPFN- Bagging generates multiple resampled versions of Dirain within a single inference process, reducing variance without additional computational overhead.
Bootstrapped Sampling in TabPFN. Given training dataset Derain {(x;; y:) N we generate K bootstrapped support sets (k) where each set is constructed by randomly sampling Neub instances with replacement:
((k)
train = {(( yi
(k)
(1)
Noab
k k=1...,KK K.
Each (R)
train serves as an alternative context set for model inference, introducing diversity into the prediction process.
Prediction Aggregation. For each test instance æq. we compute the posterior predictive distribution using the bootstrapped support sets. Given the pre-trained TabPFN model parameterized by 6, the predictive probability for label yy computed as:
exp(go(xq. (A)
(k) Pg (y/ Tq. ((h))
ΣC exp(ge (zq. D (rsin) 14)
where qe denotes the logits produced by TabPEN for the given support set (A)
To obtain the final prediction, we aggregate the results across bootstrapped support sets using either uniform averaging:
po(yg| Eq. Dtrain) =
1
Pg
(yg
Zg.
D (A) train
K
1=1
or weighted aggregation method, where weights wa are assigned based the confidence of each individual model:
(e Ee.
D Derain) =
WkPe (By Eq. D
train )
Σ
wk = 1.
Rm
km
17

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Computational Efficiency. Since bootstrapped sampling only modifies the selection of Dirain without altering the model architecture, TabPFN-Bage Bagging incurs additional computational cost beyond standard inference with ensemble. Unlike ensemble-based methods that require multiple independent model evaluations, all computations occur within single forward pass of the pre-trained TabPEN. This makes TabPFN- TabPFN-Bagging Bagginn an efficient and scalable variance reduction strategy.
C. Hardware and limitations
C.1. Hardware
Most of the experiments were performed with four NVIDIA GeForce RTX 4090 GPUs, four NVIDIA RTX 6000 Ada GPUs, and eight NVIDIA GeForce RTX 3090 GPUs.
C.2. Limitations
While our approach significantly improves the scalability and adaptability of TabPFN, it has certain limitations that present opportunities for future research. Due the current design of TabPFN, our method has been evaluated solely on classification tasks, and extending it to regression tasks remains open challenge. As tabular foundation models continue to evolve, anticipate that our approach can be adapted to support regression, broadening its applicability to wider range of real-world problems.
D. Additional Experiments
D.1. Performance on MultiClass Classification Tasks with More Than 10 Classes
To evaluate the effectiveness of BETA on multiclass classification tasks, we conducted experiments on 12 datasets containing more than 10 classes, sourced from TALENT(Ye et al., 2024b). These datasets selected to assess the scalability and adaptability of different methods in handling more complex classification problems beyond the 10-class limitation of TabPFN. Figure& 8 presents the mean accuracy of each method across these datasets. We compare BETA against several strong baselines, including KNN, XGBoost (Chen & Guestrin, 2016), RandomForest (Breiman, 2001). MLP (Gorishniy et al., 2021), ), ModernNCA (Ye et al., 2025b), and TabM (Gorishniy et al., 2025). All methods, including BETA, were evaluated using their default hyperparameters without any tuning, ensuring fair comparison of out-of-the-boo performance. From the results, BETA achieves the highest mean accuracy across all datasets, demonstrating superior generalization ability
0.70
Mean Accuracy
0,6800
0.68
0.66
8.6181
0.64
8.210
0.62
0.60
0.58
0.56
KNN
MLP
RForest XGB TabM MNCA
Bela
Method
Figure 8: Comparison of the mean accuracy across 12 classification datasets with more than 10 classes. Since the original TabPFN cannot process classification tasks with more than ten categories, we compare BETA against alternative methods, including TabM, KNN, MLP, XGBoost (XGB), RandomForest (RForest), and ModernNCA (MNCA). in hagh-cat classification settings. Notably, tree-based methods like XGBoost and RandomForest exhibit a decline in performance as the number of categories increases (Gorishniy et al., 2021; Ye al., 2024b).
D.2. Performance on Real Datasets: Bias and Variance
The generalization error of TabPEN can attributed to two primary components: bias and variance. These components are influenced by the model's architecture, pretraining assumptions, and dataset characteristics. Mathematically, the
18

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
generalization error can be decomposed by analyzing the discrepancy between the model's predictions, go (y | x. Dn). and the true conditional distribution, Po It). Specifically, it can be expressed as:
go (y | x, D.)) - po(y x) = go (y x, D.)) - -ED,<p (g0 (u | x, D.)) + PD, (go
(v | x, Dn)l - po( x).
(4)
Variance
Bins
Here, E|| denotes the expectation over training datasets D, sampled from the true distribution po- Understanding these error components is crucial for addressing the limitations of TabPFN and improving its performance across diverse tabular datasets.
Revisiting the results shown in Figure 2, we observe several key phenomena that highlight the effectiveness of BETA in reducing generalization error across varying dataset sizes:
Lowest Generalization Error Across All Dataset Sizes: Our method consistently achieves the smallest generalization error, demonstrating robust performance regardless dataset size.
Impact of Fine-Tuning on Small Datasets: For small datasets (c.g., with only a few hundred samples), fine-tuning the encoder may slightly increase bias. This occurs because the limited data volume can lead to overfitting during fine-tuning.
The bias reduction is primarily attributed to the fine-tuned encoder, which aligns TabPFN with the downstream dataset distribution, improving compatibility between the pre-trained model and the target task. Variance reduction, on the other hand, is not solely due to Bagging. The learning of multiple encoders also contributes significantly. When dataset sizes are small (c.g.. fewer than 1000 samples), where each bootstrap sample covers the entire training set, training multiple encoders still introduces diversity in representations, further reducing variance.
Mechanisms of Bias and Variance Reduction:
These findings highlight the effectiveness of BETA in addressing the bias-variance tradeoff, making it robust and scalable solution for real-world tabular learning.
To further support our findings, Table 3 presents the relative percentage changes in bias and variance (negative values indicate reduction) of our method Beta and some variants compared to TabPFN. The values are averaged over the dataset sizes 1200, 400, 800, 1600, 3200, 6400, 12800) used in Figure on the Adult and Bank datasets. For completeness, we also include results for LocalPFN (Thomas et al., 2024) and a simplified version of BETA, denoted as BETA-S, which uses only a single encoder (i.e., without Batch Ensemble and bagging)., which are not shown in Figure 2. Since TabPFN's s absolute variance these datasets is smaller than its bias, the percentage changes in bias are accordingly smaller.
Table 3: Relative percentage changes (%) in bias and variance compared to TabPFN. Negative values denote reductions.
Metric
Dataset
BETA
BETA-S
KnnPFN
LocalPFN
TabPFN-Finetune
TabPFN-Baggi
Adult
-4.40
-3.60
-0.77
-1.68
-2.69
+1.73
Bias
Bank
-1.65
-1.19
-1.33
-1.47
-1.05
+0.53
Adult
-16.23
+46.99
44.99
+18.52 +2.06
+39.27
-9.91
Variance
Bank
-36.57
+27.12
+27.49
+10.35
-9.32
D.3. Comparison with Bagging Versions of Standard Baselines and Tabular Foundation Models
To further contextualize the performance of BETA, we conduct comparative study between recent tabular foundation models and classical machine learning baselines with and without Bagging, Specifically, we include TabPFN v2 (Hollmann et al., 2025) and TabICL (Qu et al., 2025) as representative foundation models. For each, we report two variants: the original version using default hyperparameters as described in their respective papers, and a standardized version using the same context size (1000) and ensemble size (16) as BETA, denoted as TabPFNN-2-S and TabICL-S, respectively.
While we only applied inference-stage bagging for TabPFN in the main experiments, this analysis systematically evaluates wider set of models. For CatBoost (Prokhorenkova et al., 2018), LightGBM (Ke et al., 2017), and
the effect of Bagging P (Holzmüller et al., 2024), we trained 16 independent models per dataset using different random seeds. Fach model was trained on a subsampled training set of size max(0.8 x train_size, 1000) to ensure the subset size is comparable to that used by BETA during inference. Notably, TabPEN allows for changing the context size at inference time without retraining. For other baselines, bagging would significantly increase the cost-scaling nearly linearly with the number of base models. We report the average ranks of above methods in Table 4. The results highlight that Bagging substantially improves the
19

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
performance of traditional baselines. Results show that the full versions of TabPFN-v2 and TabICL outperform BETA, while their subsampled variants (TabPFN-v2-S and TabICL-S) perform significantly worse. This highlights the sensitivity of these foundation models to their configuration settings, particularly context size and ensemble strategy. Currently, BETAiS constrained by memory limitations inherent to our implementation. However, we believe that well-optimized version of TabPFN-vl could support larger context sizes with lower memory overhead. This would allow our method to scale more effectively and potentially yield even stronger performance.
Compared Methods Average Rank
BETA 4.63
TabPFN v2 4.07
TabICL 4.40
TabPFN v2-S 5.83
TabICL-S
5.84
Catboost 5.13
Catboost-B 5.02
XGBoost 7.27
XGBoost-B 7.27
RealMLP 7.06
RealMLP-B
5.62
Table 4: Average ranks of tabular foundation models and classical baselines with and without Bagging. For TabPFN v2 and TabICL, report both their default configurations and standardized variants (with context size 1000 and ensemble size 16, denoted by "-S"). For traditional methods, we compare vanilla versions against their Bagging counterparts(eenoted by "-B"). Lower rank indicates better average performance across datasets.
D.4. Impact of Cross- Validation on Model Performance
Recent work (Tschalzev et al., 2025) has highlighted methodological concerns in tabular model evaluation protocols, particularly regarding hyperparameter optimization (HPO). One key recommendation to replace single train/validation splits with cross-validation (CV), reduce the risk of overfitting and improve the reliability of performance comparisons.
Motivated by these findings, conducted an experiment compare standard holdout-based tuning with 5-fold CV on classification datasets with fewer than 3,000 samples. We evaluated three representative methods: XGBoost, RealMLP, and BETA For XGBoost and ReaIMLP, we performed HPO using 5-fold CV and ensembled the predictions the five resulting models. For BETA, which use default hyperparameters, we trained five models using different training folds and averaged their predictions during inference.
Table 5: Comparison of model performance (Average AUC) with and without cross-validation All results are averaged over the same set of classification datasets with fewer than 3,000 samples.
Method
Holdout (Avg AUC)
5-Fold CV (Avg AUC)
XGBoost 0.8452
0.8470
ReaLMLP
0.8610
0.8739
BETA
0.8759
0.8785
As shown in Table 5, cross-validation consistently improves performance across all three methods. The gain is especially notable for ReaLMLP, which benefits from more robust tuning. Even for BETA, which uses fixed hyperparameters, averaging across folds yields a modest but measurable improvement.
D.5. Additional Ablation Study
To show the sensitivity of the performance of BETA to the number bootstrapped samples and encoder paths, we conduct additional experiments. Table and Table 7 summarize the results on the TinyBench2 classification datasets.
Table Average performance rank of BETA with different numbers of bootstrapped samples across TinyBench2 (Ye et al., 2024b) datasets. Lower is better.
Bootstrap Size
500
1000
2000
3000
TabPFN
Average Rank
3.39
2.87
2.57
1.91
4.24
20

---

TabPEN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Table 7: Relative improvement (%) of BETA with different numbers of encoders over TabPFN.
Encoders
2
4
8
Relative Improvement
1.70%
2.60%
3.19%
3.23%
3.76%
These results demonstrate that increasing the number of bootstrapped samples and encoders leads to improved performance, albeit with higher computational costs.
D.6. Comparison of inference time and the number of learnable parameters.
We compared the inference time (in seconds) of BETA with other methods, as well as the average rank in the main experiment. Additionally, the number of learnable parameters was roughly estimated by the size of the stored checkpoint. The results confirm that BETA not only delivers state-of-the-art classification accuracy but also maintains low inference time and reduced memory overhead, making it highly scalable and efficient for various tabular classification tasks.
Table 8: Comparison of Inference Time, Training Time, and Number of Learnable Parameters. All metrics computed as the mean values across multiple datasets. For BETA, results are reported with the number of encoders set to 4.
Metric
BETA
TabPFN
LocalPFN
FT-T
TabR
Inference Time (s)
0.65
0.48
12.51
0.36
0.38
Checkpoint Size (MB)
0.60
98.65
7.15
14.07
Finetuning/Training time
62.29
186.73
58.9
121
D.7. Handling Distribution Shifts
To evaluate the robustness of BETA under distribution shift scenarios (c.g., covariate and concept drift), we follow prior work and incorporate plugin techniques into our method.
Specifically, we evaluate BETA on three classification datasets from the TabRed benchmark (Rubachev et al., 2024), which are designed exhibit temporal shifts. We integrate two techniques discussed in Cai & Ye (2025) into our method:
BETA-Split: A temporal-aware splitting method that assigns samples closer to the test set in time to the training set, and those further away to the validation set.
BETA-temporal: A variant of BETA that includes temporal embedding to enhance generalization under time-evolving data.
Table 9 reports the AUC scores of BETA and its variants, along with a standard MLP baseline. The results demonstrate that both plugin techniques improve performance, particularly in settings with stronger temporal distribution shifts. These findings underscore the flexibility of ICL-based models like BETAiN handling real-world data drift.
Table 9: AUC results of BETAand variants under temporal distribution shifts TabRed datasets.
Dataset
BETA
BETA-split
BETA-temporal
MLP
ecom-offers
0.5692
0.6264
0.6348
0.5866
homesite-insuree
0.9600
0.9602
0.9521
0.9404
homecredit-default
0.6114
0.6710
0.6637
0.4730
D.8. Visialzation results
To gain deeper insights into the properties of BETA, we visualize the learned embeddings Eq (x) for BETA, MLP, Mod- emNCA, and TabR using T-SNE (Van der Maaten Hinton, 2008). As shown in Figure 9, we present the embeddings of
21

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
three datasets (Bank, KDD, and Spambase), comparing different methods as well the representations learned by different encoder paths within BETA.
All deep tabular methods effectively transform the feature space, making it more conducive for classification compared to raw input features. MLP produces well-separated clusters, grouping same-class samples into distinct, compact regions. In contrast, TabR and ModernNCA form multiple clusters for the same class, positioning similar samples closer to each other while maintaining local class separability, Unlike these methods, BETA does not enforce strict clustering of same-class samples into single compact groups. This behavior aligns with TabPFN's pretraining paradigm, where the model does not require fully separable embeddings to make accurate predictions.
Moreover, we observe that different encoder paths in BETA map raw features into diverse representation spaces, contributing to variance reduction and improved robustness. The variation among encoder paths enhances the diversity of the learned representations, which is a key factor in BETA'S 's superior generalization. The visualization further confirms that BETA preserves the flexibility of the feature space while effectively adapting to downstream classification tasks.
D.9. Detail Results
Table 10: Average accuracy of different methods on 186 datasets. Each value represents the mean accuracy over 15 random seeds. The dataset ID corresponds to the numbering in Ye et al. (2024b). "XGB," t "CatB," "RF"" "LightG," "TNet," "FTT." "DCN2," " "TabT," "GNet," " "MNCA," " "M-PLR," "Exee,"" and "PEN" denote "XGBoost," "Catboost," "RandomForest," "LightGBM," "TabNett" "FF--ransformerr" "DCNv2,"" "Tabbransformer""" "GrowNet," "ModernNCA."" "MLP-PLR," "RealMLP." "FrceeFoormrr" and "TabPFN," respectively. In each cell, if the experiment fails to perform due to being out of memory, we denote it by "_." The original TabPFN cannot handle datasets with more than 10 classes or more than 100 features.
ID
KNN SVM
CatB
LightG MLP
ResNet NODE
TNet DaNet
AutoInt DCN2
SNN
TahT GNet TabR MNCA
M.PLR RMLP
ExcelF
PFN
TabM- 16 BETA
8
94.25
98
94.48
94,67
94.22
94.56
94.27
94.20
94,22
94.18
94.23
94.20
94.31
94,21
94.31
94,50
93.78 94,39
94.29
94.29
94.36
94.37
94,22
94,29
94.22
10 75.96
73.66
74.55
74.45
74
73.99
73.44
73.55
72.97
73.41
73.11 73.39
73.32
73.43
73,56
63. 16 73.51 75.69
75.64
73.37
73.70
73.27
73.22
73.94
73.58
11 98.
97.81 8 98. 76
78
98
52
98.73
98
98.46
98.56
98 34
98
40
98
98.48
98
98
95.94
94 98 39
98.68
98.59
98.52
98.62
98.56
98.27
98.62
98.80
12 55.74
58.60
58.93
57 30
58
57.90
57.91
58.81
57.72
57.95
58.29
58.38
57
52.73
52
58.23
58.56
58.01
55.58
58.70
58.56
17 78.31
72 80
81
81,33
80 14
81
81.53
88
87
79.74
81
81.31
81.00
81.53
81.50
81 39 81
12
81.14
81 08
81.42
81.36
80,80
75.99
81.45
81.37
18 83.85
61 86.97
87,59
86
86
86.56
86.56
86.54
86.31
86.52
87.09
86.74
86.74
86.82
83 86
87
85.42
87.45
87.35
87.06
85.79
86.75
87.94
19 66.79
70.82 68.58 67.61 68.91
68.53
69.75
70.35
65.42
65.77
70.75
70.15
70,02
70.12
69.58
69.28 66.94
70.50
68.11
69.30
70.50
69.70
70.42
68.03
70.27
25 85.22 84. 13 90.00 90.82
87.65
90.70
88.47
88.51
87.30
87.87
88.30
89.23
88.47
88.58
88.43 85.01 87.43 91.35
90.53
88.78
89.36
89.15
87.31
89.53
89.90
26 67.37
68. 78 73.43
73
40
23.23
73.37
73.35
73.07
73.20
71.47
75.15
73.22
73.26
73.21
73.28
67.99
99
66 73.08
73.20
73.25
73.29
73.17
72.54
73.15
73.24
27
83. 25.255777 79 21 83 25 83 26
83 25
83.32
83.34
83.28
83.31
83
83.21
83.33
83,
83.
83.
83 20
83 26 83.22
83.31
83.33
83.33
83.36
83.15
83.31
83.32
28 87.92 17
90.57
87.97
92.67
94.07
93.28
83.24
99
91.51
93.72
93.71
92.47
94
91. 60 92.72
94.94
92.08
93.33
90.03
94.44
90.97
93.60
29 89.58 87 29 91 60
91.33 88 51
91
92.88
93.72
93.06
85.99
93.82
90.82
93.43
93.50
93, 15 92
82 91 39 94.36
94.35
91.93
94,29
90,31
94,64
90.83
94.15
30 86.46 87 29 9
86
88. 78
89.99
93.04
93.22
92.61
84.94
92.15 90.06
91.53
92.57
92.
74 98.68 93.93
93-42
92.17
92.62
89.21
93.51
90.67
92.64
31 86.46 65.62 94.14 90.21
98.92
90.39
91.85
92.15
91.81
85.58
92.15 89.00
91.31
17.17
91, 94
91.29
11 92.96
93.22
89,35
92.78
88.79
92.86
89.24
92,86
32 86.04 87 50 8R. 19 87.51 87.15
88.10
91.29
92.64
91.54
82.06
91.72 88.96
91.25
92.85
91 60
91.61
8R 99
89
93.81
89 89.47
92.22
86.68
92.75
86.68
92.64
34 77.8 63.11 78.01 79.72
58 54
77.12
77.56
76.98
71.94
70.34
34 77.08
77.70
77.12
77.71
76
98
77
64
82.01
79
78.91
78.50
78.41
66.76
79.06
47
35 86.61 88. 90 88.4 89.05
87. 1
89.29
88.74
89.06
85.80
85. 30 88.07
89.27
89.08
89. 17
88 26
88
24
86
89.51
89.14
88.75
89.40
89.
89.32
89.43
89.51
37 62. 72.27 72.67 74,
68
72.01
74.81
57.72
60.87
75.61 73.19
71.60
76 73
71.77
75.04
74.69
73.74
73.80
72.61
75.01
75.82
39 62.55 63.66 64 67 45 66.
66
60
67.54
66.32
66.46
66.34
66.28
65.33
67.28
66.50
66
65,
84
62.92
64.76
67.97
67.41
66.80
66.94
65.02
66.88
67.28
40 79.81 72.50 84.77 85.00
83
84.93
85.03
84.81
84.38
81.57
85.08
84.77
84.86
84.97
84.65
77.03 84.60
83.82
84.59
85.07
84.35
85.06
85.01
84.53
84.68
41 71.90 72.71 73.96
74,27 73.65
74.72
73,25
73.40
74.51
10
73.43
74.10
73.93
73.74
73.42
53.53
72.75 73.63
74.28
73.99
74.21
74.11
73.82
73.85
74.04
42. 62. 13 71 67. 14
68. 16
67.
50
62.12
73.84
74.80
55.77
67.87
72.77
73.46
70.90
73
71 93
86
70.
10
74.28
70.57
69 69.45
74.22
68.07
75.00
64.80
75.26
43 57.65 34
63.81
66.30
58
49
66.42
71.21
71.21
59.00
71.06
71.12
69.01
70.89
71.00 08
70
70.
10
71.16
71.15
70.08
71.42
70.52
69.78
71.13
44 59. 00 75.7552222 20
66.27
63.98
65.
54 66.18
69.01
74.64
65.23
51.35
53.70
51.79
65.67
74.
72.62
74,
24
67.
75.20
75.44
65.00
75.19
52.97
75.59
62.05
75.55
45 59. 76.64 64.87
65.58
64
16 64.60
73.04
76.02
55.06
56.71
16 67.03
68.83
70.50
74. 02.02.724
13
69
76.55
75.26
67.74
77.09
64.21
75.89
63.05
76.17
46
34,80 71.42 64.07
64.95
18
64,40
71.73
71.78
54.36
69
70.58
69.13
70.06
71.32
71. 38
71.32
65.47
72.02
72.09
67.77
72.08
69.28
67.25
70.61
71.36
47 52.54 70.41 60.23
61.58 56.53
62.28
69.85
70.92
$2.25
64.13
65.62
61.84
68.84
69,09
68.
66
99
63.36
70,80
71.09
64.53
70.56
61.46
58.13
65.59
66.28
48 53.13 69.16333133 60.80
61.65
56.57
62.25
68.31
69.32
53.47
67.62
64.94
67.17
65.59
69.76
68
82 69.65
65
62
71 .09
70.98
61.62
70.84
56.90
54.51
67.46
59.37
49
$2.04 7 39.3999000 65.25
89
57. 13
59
13 72.04
70.05
70.63
56.71
70.
66.65
67.67
52 59
26
71
50
65 45
71 22
70.41
61.36
73.32
62.65
73.08
63.65
72.84
50 54.61 70.99
62.59
57 18
62.72
70.61
53.38
62.43
68.71
68.80
69.71
70.45
69.90
34
65
86
64.32
71.18
63.79
71.31
66.44
70.26
69.15
52 80.37 77.62 85. 85.97
80.72 85.76
86.74
87.11
81.42
K3.01
85.79
87.11
86.48
86.
39 86
86 36 86
20 87,34
87.19
86.60
87.20
K5,30
K3.86
87.15
87.08
53
74.00 76.67 75.38
11 77.93
64
76.29
77.56
78.62
75.11
76.18
77.02
76.24
76.93
77
74,58
75 78.04
78.40
77.24
77.69
74.04
79.93
78.98
78.56
55
55,94
48.77 60.35 68.08
58.96
61.02
39.40
60.23
56.96
49.38
55.85
49.31
54.12
58.90
63 65
64.33
57.75 63.77
59,90
55:40
65.31
50.98
58.94
48.85
65.69
56
64.06 50. 19 12 67
96
64
68.23
65.12
65.71
$3.29
49. 54
63.88
53.25
59.42
66.
66 65.
64.83 67.79
65.71
63.19
67.33
$2.06
65.56
58.00
68.35
57
57.26
88 59.87
59.26
59
58.82
60.09
59:61
59
58.74
59.00
59.86
59.51
58.95
59.57
58
09
5R
84 59.14
59.25
59.48
59,81
59.41
59.00
59.77
59.85
58
65.67
45. 58 68. 94 69. 28 62.
14
68.33
63.42
61.76
52. 30
49 26
61.34
34
65.65
63,
62.
63
57.67
67
54
99 73.92
69:51
66.79
69.43
68.44
$5.93
69.80
66.06
60
79.10
05 79.37
79.51
77.96
79.34
79.53
79.29
79.10
78.32
79.38
79.80
79.60
79.72
79.22
68
72.32
79.84
79.58
79.56
77.35
79.82
79.74
61 93.70 85.77 77...777000 90.7 78 93.47 92 18
91.29
89.10
88.32
91.62
83.78
88.01
88.94
87.11
W8 21
88. 15
87 65
90.50
87.98
91.34
89.08
91.15
85.46
90.78
87.42
92,30
63
84.16 80.64 89.71 89.21
21
87
30
45
87.37
87.21
84.43
86.20
86.37
88.50
87.19
87.46
87.
30
48
89.47
89.70
88.53
88.88
88.25
83.37
89.10
87.73
64
89.50 89.87 92.32
92.07 89.05
05
84.27
93.01
94.63
72.64
93.38
91.54
93.35
92.83
93.43
93.97
91.21
36
89
96.50
96.85
93.29
95.45
93.39
94.90
95.45
65
75.75 75. 75 75.75 75.75
75.75
75.75
75.73
75.74
75.76
75.70
75.74
75.70
75.74
75.66
75.69
73
7
75.73
75.68
75.71
75.71
75.70
75.75
75.72
75.76
66 92.59
55 90
94, 72
94
03
94.18
94.41
94.28
94.59
93.38
45
94
94,
94.52
94.
02
98
94.45
94,18
94.27
94.29
94,
93.70
94.35
94.24
67 88.70 88. 08 88.
27
41 88
41
46
88.75
88.31
88.02
87.94
88. 25 88.60
88, 39
87.8
85
38.37
88.
17
88
25
87.67
88.62
88.43
88.58
87
88.41
87.85
88.35
98.66
98.03
98.30
98.21
98.98
94.81
97.
98.69
98.79
98.79
98.89
SW
98.
99.99
99.56
98.70
99.40
98.31
97.27
98.99
98.56
22

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
70 71.70
74.85
80.77
80.
79
80.01
80.21
77.06
77.17
80.40
76.76
76.35
80.28
76.50
77.10
76.72
73.55
74.63
79.72
79.58
78.61
79.25
80.25
77.58
80.33
81.34
71
69.
79
73.39
19.398
81
31.38
81
03 80.48
80.34
75.52
75.02
77.41
76.84
84
75.00
80.38
76.51
75.65
76.40
74.75
71.66.668 79.41
79.63
79.81
79.43
80.70
76.82
80.27
81.14
77
89.51
99
08 99.19911
12
98
82 96
95
97.79
98.60
99.43
98.44
98.40
98.87
98.
99.01
98.
95.
31
95
98
98
99.11
99.45
99.00
98.53
99.65
99.76
78
84.8
85
87
88
89
88.71 87
91
88.06
88.71
84.91
84,06
86.87
89.68
87.78
89.09
88.85
89.00
09 86
88
87.82
88.81
87.96
88.95
90.14
89.70
80 84.65
79 34
67.63
87.87 86 90
87.88
87.24
87.61
87.01
86.9%
86.91
87.59
87.03
87. 17
86.88
K3.09
86
70
88.13
85.96
87.05
87.86
87.58
86
87,83
88.24
87.95
87
88.
88.05
27.22
88.05
87.71
87.65
85.15
84,94
88.12
86.88
88.26
86.43
88.
87.
86
87.95
90.19
88.18
88.04
88.04
88,88
88.50
89.08
84 67.25
92
90.80
93.57
87 02
91.08
91.57
92.65
89.53
88.07
91.75
93.67
93.73
95.50
95.42
57.23
84
93
98.12
96.97
96.27
95.27
96.77
96.88
97.17
88
82.89
19 84.23 84,27 07
83.64
83.05
82.56
84,25
83.77
83.30
83.63
83.83
83:00
83.04
83.08
08 83. 83.44
83.26
83,49
83.85
83.39
83,38
83.41
83,70
92. 72.39 69 23 71.17 72.91
72.04
70.90
72.11 73.38
72.19
66.47
68.86
72.14
72.09
73.06
71.59
86.86 70. 10 70.95
72.89
69.53
71.59
70.75
75.52
72.59
75.45
93 96.61 93
98
97 22
96.
96 39
97.19
97.27 97.40
94,86
94.95
97.33
96.76
96,
96.90
97 09
97 30 96 20 97,7
97.67
97.33
97.25
96.98
94.
97.47
96.47
96 71.43 68
45 77.75
77
78.14
76.06
73.03
72.51
75.58
76.62
73.64
76.
61 76.02 75.97
75.24
7.5766
74,03
76.58
75.58
77.01
77.40
98 86. 40 85 87
87.
87,
25 87
40
87.47
86.91
86.77
86.07
16.13
85.08
86.72
87.07
W7.
86.
86 39
86. 33 86.87
866
87.01
86.96
86.85
67.45
67.25
87.27
99 89.
85.
05.0668888
87.30
87
84.08
86.67
86.67
69.76
79.87
86,95
85.12
87.08
87.7
26
85.69 86.03
84.55
87.27
85.88
86.89
87.52
84.93
87.84
100 83, 31 82
21 85, 28
89 76 55
88
85.85
85.73
84,74
83.74
84.33
85.66
85,50
85.91
85 40
85 02
81 78 86.09
86.27
86,96
85.05
83.12
86.78
86.35
102 72. 93.
67
97.
81.77
97.60
94.58
95.27
97.
97.25
97.10
97.03
95.02
96.83
92.62 91.96
76.52 97,
97.38
97.22
97.21
97.40
96.53
97.21
97.16
103 62. 18 64
22
66
83 67. 00
66
30
66.75
65.9%
67.43
66
68
66.
38 65.39
68.48
66.60
67.
67. 10 64 21 64. 80 67 798
67
67.17
67.50
67.98
65.68
67.74
68.49
10G 85 46 86. 10
95 50
95, 17 94 22
95
80
87,94
89.09
91.29
89
32
87
36
95.19
92,58
91. 96 91 ,9 85 74 90. 85 94.09
93.73
93.58
94.16
93.20
93,30
94.78
95.32
66
98, 98.95
95
98.95
98.92
98.82
98.95
98.81
98.93
98.93
98.95
98.
988
92
82
98.
98.88
98.79
98,90
98.89
98.93
98.95
98.94
98.95
109 67 60. 98 61 44 62.
50 62.93
62.86
63.01
62.80
65.05
60.66
62.58
63.10
62.13
62.58
61. 15 62. 35 61
32 62.63
62.80
62.13
61.64
62.23
61.54
61.68
62.69
110 67.
96
65,
27
67.93
70.56
68.60
65.35
66.25
68
68
69
65.97
70.03
66.30
69,80
60,06
74.45
62.16
74.51
111 96, 58 96
65 98.
83 98, 79
98 36
98
69
98
98:42
94.61
97.11
98.46
98.49
98, 94
51
98, 35
97
61
98
04 98.72
99.07
98.65 63
98,80
98,57
99.14
98.22
99.12
114 59.57 61. 87 61 02 61.84
61 90
60.97
63.15
63.57
59.62
61.18
63.29
63.01
62.73
62.62
62.
60.68
68
62. 11 62.71
63.04
62.54
63.25
62.93
61.98
62.95
63.51
116 82.8 89 83 85
06
84 76
85
84.34
84.45
82.75
83.02
84,07
84,80
83.39
84.05
83
08
46
82,
86 84.83
84.97
84.73
84.35
84.72
84,42
85.73
85.61
117 79. 70774888 83.07 83 62
84
84.54
83.82
83.48
82.92
81.34
80.25
82.63
83.69
83.64
83.
83
39
81
80
82
38 83.57
83.21
83.96
83.37
83.87
84.05
84.05
84.25
80.94 84 85.03
83.86
84.49
H2.9%
81.55
82.21 84.83
84.45
84.76
48
81
84.62
83
83.72
84.15
83.69
84,00
84.05
84.57
119 84.07 82.
86.93
87.31 86.23
87.41
65.19
85.16
86.06
65.25
85.35
85.8%
85.61
65.69
85.
84
65
85
68
86 90
87.28
86.39
86,
86.02
84.44
87.19
86.46
63. (3.50 60.660000 50 61 67 62.88 63. 17
61.27
57.20
58.05
58.65
59.02
60.32
60.05
60.97
63
63
59
88
62.42
60.08
62.17
60.93
59. 13 58,40
62.37
60.80
122 96. 16 95 81 98.
03 78 96 83
97.60
96.49
96.50
95.87
96.58
96.44
97.09
97.26
96.30
96,
96
20 96,
10 97,07.00 004
97.13
97.16
97.13
97,46
97.29
97.28
97.65
123 97. .9 96.77 98. 22 97.99
97.73
98.53
96.67
97.02
96.69
96.59
97.01 97.73
97.43
97.03
97. 62
96,
62 96.27
97.93
97.79
97.81
97.67
97.60
96.99
988.5
98.13
124 988 82 99 41 98. 78 98.
58 98.66
99.09
98,70
97.71
97.99
90.53
98.62
97.24
98..42
988
93
98, 74
98.
38 98.
19 98.86
99:41
98.78
99.33
97.95
99.37
99.13
99.37
128 83. 523322222
32. 13111118000 87.72
98.96
77
88.51
78.10
77.14
60.77
63.88
70.36
80.80
76.12
77.91
77.27
61.29
71 90,38
91.67
91.32
83.27
77.76
63.04
92.65
65.56
130 54. 40
60.53
65.72
66.
61
64.33
58.39
$1.04
57,85
62.77
59.17
57.65
$2.81
01 46.51 57.91
50.77
55.87
62.71
57.64
50.15
61.11
65.39
1 35.00
31 21
35
45
75.79
35
48
39.24
34.00
35.58
34.73
23.36
33.88
35.55
12.36
36.
36
32.67
34.
15 24
35.64
34.45
32.27
33.61
30.09
34.21
34.58
36.76
333 89 42
19
90.00000
60
90.
89
98
90.52
90.60
90.35
89.9
99
90.45
90.95
90.06
90.50
90.70
89
99 25 90.87
90.74
90.71
90.95
90.92
89.26
90.89
90.97
136 55, 6- 53.
48. 12
51 85 47.32
56.41
53.33
52.87
53,31
51.71
55.13
54.76
55,22
$4.35
12.12 54.55
52.75 54,04
52.75
53.21
53.24
53.62
55.64
51.81
55.64
137 92. 54 95 52
93. 98
93, 88 60
94.20
92.54
93.13
91.64
92.31
92.89
93.76
92.76
93.58
93.86
17 94.15 94,73
92.74
94.10
93.68
93.33
93.33
94.13
94.45
139 88.
89
50
98.
.0 99.
95
92
98.90
98.15
98.48
75.57
89.85
98.94
98.21
98.65
97.98
98.
99
50
67 99.34
99:46
98.75
99.54
97.32
97.51
98.02
99.17
141 89.
30
86. 25
94
88 75.75
94 39
94.65
93.25
93.84
93.32
89.97
93.05
96.34
95.51
99
94. 89
29
92 15 95.75
95.49
95.15
96.30
95.
93.53
96.57
96.47
49. 54.58 56.
57.02
82
57.51
55.89
56.29
55.03
54,46
56.72
$6.61
$7.
56.47 47.71
47.25 56.27 55.66
54.78
56.88
57.27 57.31
55.21
57.47
146 96 48 96 74
96
98
89
96 99
97.01
96.79
96.64
97.09
96.74
96.85
96.65
96.87
96
96.
87
96
71 96.83
97.89
96.63
96.81
96.88
97.00
96.71
96.92
147 75.88 68. 13 78
45 76.37 74.69
75.08
75.95
76.60
69.81
69.07
74.69
76.64
75.92
73.59 72.66 75.01
65 59 86.64
85.17
77.09
79.04
71.45
69.36
78.79
73.73
149 77. 14 65.95
85.
87.17
73.82
83.97
82,40
83.40
83.81
81.51
84.90
85.55
85,48
18.18
86.02
87.33 86.67
86.53
82.77
86.49
84.89
66,04
87.33
82.58
150 49.8
5 30
80
60.
16 56.63
60.52
57.42
56.66
61.06
50.98
55.25
47.50
58.24
59.53
56.81 50.80 $2.09 59.39
57.24
59.30
58.31
56.81
59.16
61.31
61.33
153 67.8
70.9
98 78.17 77.90 .6.96777
77.95
77.82
75.80
75.40
62.25
74.69
75.54
75.88
75.61
75.56
75.28 74.67 74.43
77.55
72.98
77.23
77.82
75.78
76.93
77.43
77.65
154 72.50 75 SO 76.8
77.67 76. 70
74.50
73.30
75.77
78.07
68.37
75.33
79.13
74.33
7%, 97 77 40 75 83
75.40
78.50
74.23
74.43
78.43
74,80
78.13
78.63
93.4
96.07 96 93.36
96.06
96.00
95.99
95.57
95.73
96.28
10
95
95.05
96.34
96.17
96.06
96.08
95.95
91.32
96.44
95.71
156 60 63.01 64. 20 02 62. 18
67.17
62,97
63.14
63.91
62.53
63.13
63.87
64.06
64. 10 64, 22 62.47
63 76
63.96
63.95
63.93
63.79
61.45
63.84
63:60
160 67 94.3 31 05.02550020 94,82
94.89
94.85
94.49
94.73
94.44
94.35 94.60
94.71
94.73
94.67
94, 90.68
94.
94.82
94.93
94.69
94.52
94.60
94.72
94.95
94.98
162 98.4 41 98.41 98.76 98. 66 98.41 98.70
98.52
98.41
98.41
98.37
98.35
98.76
98%
98 39 98
98.4 98
98.76
98.68
98.42
98.87
98.58
98.44
98.71
99.00
163 89.6 66 96 08 96.30 96.52 94.97 96.99
92.24
96.33
90.79
81.72 93.39
93.50
95.57
95.02
95,
95.61
94.85
95.65
93.73
96.54
96.51
96.70
96.76
164 40.58 26 38
38,
44
39 35
39.49
38.51
37.15
40,27
37.97
35,90
39,86
39.06
36.92
39
38.94
36 82 39.26
39.89
37,79
38.13
39.45
40.32
39,58
40.32
165 91 48.48999 91. 14 92. 10
92. 18
91 29
92.29
92.46
92.54
91.07
91.67
92.24
92.53
92.47
92.49
38 92. 18 35.09
92.74
92.58
92.42
92.64
92.41
92.19
92.45
92.37
166 92. 22.22.262
94
88
95.69
91.61
81,95
63.
94.56
95.35
95.26
95
95.73 91, 70 81 96.8.19
99.11
95.42
98,95
93.67
92.84
96.90
95.49
167 84,27 76.01 9
91
69
85 26
92.31
85,37
85.52
83
81.92
84,95
87,51
86.05
85.55
85,
79.49
84
95.62
86,82
87.70
87.44
79.45
88. 66
84.99
169 84.63 73.74 90.06
94,
69
18 96.08
93.24
93.74
75.43
92.10 94.29
97.13
95.24
95.19
96.75
86. 70 80.80 98.03
98.28
97.442
95.92
96.83
96.78
97.49
97.75
170 59. 46 64. 19 67.6
67.70 67.66
65.72
65.81
68.83
70.90
49
61.85
67.12
67.43
66.71
66.13
61.44
68.24
70.77
68.87
67.97
69.73
69.77
68.06
72.39
171 61. 24 49. 56
72.64
71.88 64.62
72.85
62.77
63.71
60.82
54,80 59.63
71.59
61.15
62.87
63.33
54.
58 43
96.98
98.55
80.07
76.66
59.58
$5.06
92.71
75.26
172 58.0
55
58
63
25
61.52
50.57
63.93
57.08
57.54
56.
56.45
57
59.45
58.31
57.41
$7, 56.
73
55,
65.95
90.88
63,30
59.63
60.28
57.84
59.87
63,16
174 $9
48.4 45
61 23
61.
11
66.66
60.11
57.63
58.42
43.57
50.95
50 57.51
$7.03
$7.
57. 76
51.26
55
80 58.91
59.95
56.48
57.49
55.24
57.75
56.08
58. 14
177 99.
96
99 42
49.999999
99
40
99.47
99.41
99.50
80
98. 39 99.35
99.20
99.45
99:40
99. 34
26
48
31 99.48
99.49
99.57
99. 66
99.24
99.63
99
179 84, 02
90.87
92.99 91
91 05.05500
91-45
91.99
88.31
89.92
91.84
89.04
88.77
92.88
94,40
89.92
94. 52
84,
35
88
93.55
91.29
90.96
90.68
91.29
90.81
93.00
93.06
1827 71.45
25 72.33
7
72.30 71.98
72.21
71.81
71.75
72.05
71.21 71.94
72.17
71.95
70.87
71.63
71.7
71
72.17
72.13
72.40
72.17
72.45 71.75
72.21
72.29
183 55. $5.56
78.66
68
57.78
5 66
54.98
98
60.27
65.49
77.64
49.19
49.82
61.26
50.81
54.87
59.53
62. 69
89,
44
58.41 70.04
78.11
$3.44
79.64
54.54
51.63
52.67
63.43
18488 86. 32 81 46
88.
26
88
47
87 88
88.08
87.17
86.97
87.23
85.03
86.86
87.47
87.21
86.85
62
86
26
86
32 87.02
87.04
86.83
87.44
87.22
87.
87.41
87.75
97.9 19.999999 97,99
98 97.
98
97.81
97.83
97.86
97,54
97.83
97.80
97.89
97.83
97.
86
85
97.91 97.85
98.00
97.94
97,85
97.93
97,80
97.97
97.95
192 w.. 99 95 100.0
100.0
100.0
100.0
95.06
96.
84.49
92.79
93.67
100.0
93.65
97. 80
97 23
22 86. 12 99.98
100.0
99.93
99.84
100.0
99.86
100.0
100.0
193 69.5 00..3300 67. 0000009990 70.10
1011000000 26
74.42
76.85
76.29
76.05
72.25
67.77
75.52
75.70
74.31
74 75.
76.02
73.99 76.83
75.90
75.57
76.65
74.60
64.39
77.18
75.57
194 93. 101100700 79 545555995
92.
98
93.08
93.11
93.09
80.12 76.80
79.76
73.56
79.84
79.73
79.94
80.02
79.87
74. 12 76. 93.09
92.92
80.45
93.10
79.94
81.59
80.26
80,18
196 78. 09.399977 77 05 80.29
79. 72 81 42
80.89
75.02
76.64
78.22
76.68
75.31
78.92
77.98
76.87
77.54
75.63
80.04
78.75
78.05
76.91
78.88
79.65
79.30
197 81 67 81 29
44
80
81
80.89
81,25
80.95
81.31
80.94
80.78
81.41
81,35
81.09
00
81.31
81
24 81.65
80:90
81,28
81,04
80.86
81.32
81.53
81.27
198 88. 14 68. 01 86. 10 86
47 82.62 62
86.59
95.27
86.96
85.51
87
84
92.
97.61
95.52
96.
90.57
71 44
64
80 98.98
99:50
99.07
99.50
97.34
82,
94.41
96.99
86.00 025.002880
07 86.2
86
06.22480
86 98
87.44
85.81
85,94
85.02
84.11
85.40
85.34
86.26
85.53
84.8
86.07
85
85.96 84.74
84.53
85.26
86.10
84.71
86.62
86.03
86.73
200 83. 727772822 85.8 81 88.36 87.74 88 27
88.70
84.96
85.16
83.28
84.69
84,89
84.87
84.5K
K5.07
85.07
84, 96 84.23 87.60
88.14
87.87
87.73
84.82
85.99
87.41
$6.03
203 93. 97.9 90199999999 99.9999. 99.84 99.27
99.84
99:40
99.52
99.77
96.02
99.43
99.69
99.68
99.67
99.54
43 99.38
99.78
99.76
99.60
99.69
99.51
99.24
99.44
99.80
205 93.82 100.0 100.0 100.0 100.0
100.0
99.99
100.0
100.0
99.93
100.0
98.02
100.0
100.0
100.0 94.
100.0
100.0
100.0
100.0
94.39
100.0
100.0
100.0
206 72. 34..47777 73 44 73.01 73.49
73.
72.41
73.78
73.74
73.49
70.73
72.78
73.31
73.54
72.59
73.26 73.62 73.39
73,83
73.18
72.78
73.41
73.70
73.55
73.41
73.47
207 72.97 72.03 74.21 74.31
74.50
50
74.52
73.97
76
73.93
72.33
73.50
73.61
73.89
73.48
73.60 73.58 73.97
73.66
73.88
73.78
73.33
73.45
73.41
73.94
73.99
209 67.667 83 60. 35 86.32 86.4377760
74.12 12
86.38
61.52
59.38
76.45
55. 30
58.56
56
60.91
57,31
59.
17
61
41 59
52 50.91
72.40
83.32
62,48
84.67
58.27
64.13
82.60
210 98.668 19 9%.
98.55 02
98.76
98.67
98.71
98.21
98.31
98.78
98.71
98.70
98.65
98.
17 98
98.72
98.56
98.71
98.1
98.64
98.65
98.69
98.75
211 82.76 62.56 13 81.28
82.04
83.32
72.41
76.29
73.92
66.40
72.91
71.23
68.01
67
71 66
66.83
81 01.05 83,25
82.99
81.25
82.46
73.60
77.50
80.43
78.89
213 96.25 97.75 96.60
96.80
96.55
96.88
96.15
97.25
93.83
91.23
96.02
96.42
95.93
95.77
96.32
96.63
95.80 96.93
97.22
96.37
97.10
96.53
96.57
96.55
23

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
214 86. 83.50 50 85.72
86.02
86.23
84.45
85.97
83.30
77.42
84.07
85.23
83.90
84.67
83.88
81.88
84.25 86.53
86.48
85.23
88.08
85.55
83.63
86.72
87.73
215 96.00 94 25 94,00
95.95
95.02
95.63
96.50
93.77
84.10
95.42
95.55
95.27
95.35
94.83
94,72
94,48
96.53
95.75
96.42
94.95
94,62
96.07
96.58
216 76.00
92 74.88 24 74.90
75.33
76.43
76.42
69.00
74.53
76.92
76.68
76.37
75.92
76.
76.25
73.92 76.33
75.53
76.15
75.43
74.45
76.10
76.00
76.67
217 92. 25 98.75 75 93 20 95.
95.92
95.47
95.65
94.55
86.85
94.52
94,90
95.85
96.00
95.60
96.096.37
96.50
95.08
95.88
95.00
96.00
94,08
218 80,50 82 78.
78.55
76.03
79.45
84.77
83.62
67.93
12.63
83.47
83.35
83.17
82.98
82.50 81.72 81.03 87.27
79.02
82.18
K3.03
79.77
81.58
83.90
81.97
219 100.0 61 90 99.07 97.96
98.61
99.23
99,94
98.80
85.31
99.14
99.32 99.35 99. 99.88
100.0
97.96
99.60
98.33
100.0
220 58.63 56 S0 62.03
81 61
61.92
63.00
62.93
61
42 62.20
62.76
62.18
62.90
62.22 59.67
67 46 79 63.28
63.08
62.77
63.79
62.59
60.60
62.39
63.70
221 96
69 98.05 98.
01
97.95
98.32
98.04
93.13
97. 19 98.07 98.17
98.21
98.
98. 88, 56 83 98.93
98.47
98.33
98.47
98.
93,40
98.53
98.27
222.9 91. 12 77.85 85 94.59 94.66 94 56 94.58
91.94
91.35
92.73
92.61
91.48 92.76
90.90
92.50
82.88 85.09 53 94.26
94.28
93.86
93.24
92.79
91.19
93.47
93.10
224 96. 56 95 8 96.80 96 96 34
96.97
96.87
96.94
96, 19 96.55
96.93 96.91
96.92
96.81
94
97.00 00 96 96 97.00 02 96.90
96.98
96.93
96.40
96,31
97.03
96.86
78.9
68
100.0
100.0
94.53
90
99.37
98.45
98.
91,93
99.40 40 99.89
99.53
97.73
99,50
100.0
100.0
226 72.68 74 25 15 12 75.08 72.27
74.75
74,50
74.28
73.09
73.36
74.5%
75.00
74.55
74,81
84.83 73.62 71 K5 74,81
74.75
74.71
74.79
74.07
73.12
74.78
74.98
79 90.51 90.63
33
90.19
88.71
88.75
89.08
90.19
89.76
90.32
89 69.69
88
89.65
90.53
90.21
90.11
89,98
89.69
89.89
231 97.9 95 95 82 98. 24 98, 24
98 01
98.31
98.25
98.64
95.71
94.33
98.05
97.79
97.98
98, 35 98, 28 98 07 98 19 98.71
99.13
97.85
98.70
96.98
97.89
98.62
98.35
232 95.
08 99.79 94, 40
95. 00
94.81
93.74
93.74
93.69
93.46
93.03
94.83
94.53
94.31 94.00 93.52 94.46
94,42
94.66
93.37
94,02
94.52
95.13
94.94
95.12
233 98. 23 65 98. 14 97.8 81 98 23
97.87
97.43
97.68
98.
96.06
97.26 97.53
97.55
97.47
97.5 98.08
97.91
97.76
96.73
97.30
97.83
98.07
98.04
234 96. 16 95 65 97 45 96.67
97
97.12
96.69
96.68
92.95
95.27
96.94
96.71
97.11
96.63
96.61
96 40 96
96.77
96.68
96.85
97,11
96.32
97.27
96.74
97.23
93.
81 93.51
93.57
93.75
92.73
10
93.24
92.19
93.36
9:3.24
92.73
92.43
933.39
92.70
91.83
93.72
93.70
236 89. 14 88. 7 88, 56 88,80 89
87.73
28,20
88.67
89.78
88.84
89.07
89.78 79
89,52
68.67
89 89. 76 89.63 88.46
88.75
B8.24
87.92
89.69
89.
66)
89 89.48
237
27
90.
89. 13
91
89.89
88.52
87.79
87.47
90.05 89.59
90.34
89.75
90.
89.75 89.89
89.43
89.86
89,91
90.78
90.21
90.64
238 99. 09 92.8 81 94,1 14
99.
13 98. 79
95.09
99.50
99.48
98.
98, 74 99.29
99.35
99.35
99.42
99. 16
99. 24 22 76
99.43
99:40
99.29
99.60
99.37 98.88 88
99:41
99 /4
239 73. 86 70.47
76.73
75.
75.79
76.03
71.87
71.16
75.27
69.87
71.01 69.81
72.48
70.27 71.59 71.18 91 74.11
76.81
72.31
73.82
73.03
73.88
74.09
240 86
94
87 66
97
88 52
88.07
86,84
87. 14
85
84, 66
10 88.09
86.22
87.
10
85 93
70
86.43 88.92
89:71
86.98
87.84
86 54
86,24 24
87.06
87.10
241 95. 3-4334880
88. 15
98. 24
98.
60
97 32
98.18
98.54
08 85.85
98.08
97.70
98.53
99.14
99.05
98.
82 98.
97,58
71-42 98.87
98.85
99.04
98.59
98.63
97.48
98.96
99.21
67.91 73.50
65
74.79
76.58
74.60
74.78
74,92
71.08
74.61 75.47
74.57
75.85 75
73.24 74.93
73.45
74.90
76.55
75.32 74.93
74.87
77.04
249 99.39 92. 13
91
22.23
92 18
92.34
91.79
92.10
91.91
91.66
91.63
92.04
92.05
92
92.
44
91 36 92.16
92.98
91.75
92.20
91,92
92.98
91.74
92.07
250 77.57 05
97.1
55
97.
97
91
97.41
97.66
96.88
94.85
81.80
97.65
96.95
96.96
97.
79
97
97.95
98.02
97.79
97.61
97.74
98.04
06
97.
251 65. 49 63
08
79,99
93
78. 58 72.95
78.01
65.62
65.92
64.75
62.81
66.65
71.96
68,
64.
88
64. 90
62.
82
62
58 87.79
83.75
75.03
74.72
71.68
67.38
76.52
80.44
252 90.75 83.67 67 91
42
91 35 90.86
91.62
89.93
90.44
86.12
87.90
89.79
89.59
89.70
90.45
90.41 86 89.81 91.31
91.04
90.40
91.56
89.47
90.77
89.63
91.27
254 88,
82.99 97
49
92.02
91. 96
91.77
91.99
91.86
79.57
87.92
91.85
91.37
90..99
91.88
89.06 91.39
24.04 91.90
92.77
89,86
91.33
90.59
92.34
91.18
92.62
255 93.04999 93 23 92. 73 92.93
1 93.28
93.05
92.71
90
92.87
92.51
92.80
92.73
92.50
92.82 92.6
92. 92.96
93:42
92.96
92.66
92.82 93.42
92.74
93.42
90.28 21 91.1
92.04
91.68
87.29
90.70
92.50
90.89
92 91.68
92.20
89.78
93.88
90.80
92.29
90.87
258 90.59 86 96 90.00 90.21
90.26 26
89.88
90.92
90.93
90.78
90.89 90.66
90.69
90.82
90.95
89 90.55 91 91.07
90.78
90.73
90.79
90.29 89.82
90.74
90.86
259 85
90 86 68
86
40
80
85,72
86 18
86.73
85,27
85.94
86.55
86.37
86.09
86
78 85 52 86.74
86.63
86.15
86.28
86,27 86.12
86,75
87.02
260 99.9 91 92. 68 99, 98 99.77
99.97
85.39
99.91
99.91
99.97
99.73
99.93
99.96
99.96
99.89
99.92
99. 62 94.77 99.94
99,96
99.95
99.96
99.95
99.96
99.95
263 91.75 91.9999 91 95 23 94.67
93. 30
94.61
93.86
93.91
93.99
91.34 93.74 93.93
93.56
93.83
92, 96
93.81 93.23 93.88
93.20
93.04
94.35
93.46 94.73
93.40
94.77
264 86.8 94 04 96, 95. 29 95
96.31
93.41
94.12
77.30
86.73
68.06
94.93
9:3.55
94.56
94
94
51 46.46 93.34
94.37
92.22
95.78
94.87
85,50
95.60
95.62
265 833 UO 83.1 13 82 40 40
82. 37
83.77
81.27
82.40
70.17
17.10
K3.43
K3.30
K2.57
H2.23
87.17
82 M3
83
82.53
88
83.73
K3.53
80,53
84.00
83.30
84.13
74.30
72.97
75.13
72.20
70.30
72.27
68.73
69.73
71.23
72.50
71.57
77.77
68
72.40
72.50
70.73
71.57
73.03
73.10
72.40
72.27
268 77. 12 72.4 4 7 1 77 77
76.90
75.17
76.35
57.48
72.44
74.96
76.32
63.64
67.
69 75.65
74,
76
28
76
78.63
78.11
74.67
77.53
75.39
76.28
77.99
78.61
272 80.3
59 83 99 82.92 83 08
68
81.39
82.68
85.21
78.77
81.50
81.04
79.54
83.
80.
81 98 80.
90
82.81
83.64
82,33
80.59
82,52
83.78
83.16
273 88. 20 91.6 61 94.54
95.24
93.16
94.24
93.09
93.29
91.54
92.31
93.29
94.71
92.70
93.57
89.94
50 91.61 96.38
95.88
94.63
95.04
93.30
93.20
94.87
95.62
274 97. 14 96. 77 97.02 18 97.17
97.12
96.85
96.67
96.78
96.81
96.86
96.74
96.74
96.62 96.6 96.
86 96.60 96.78
96.74
96.62
96.67
96.69
96.92
96.90
96.98
275 77. 15 59 60
80.57
79.65
80.60
79.36
79.56
79.88
78.54
78,90
80.05
79.82
80.02
79.57
20
79.02
80.13
79.69
80.31
79.95
80,05
79.70
80.21
80.25
277 95. 0O0
93 82 99
52
99 49 99.64
99.53
98.67
98.50
97.56
98.44
98.42
98.94
98
9% 40 9% 28 47
97.13
13 98. 96
99.21
99.10
99.48
99.15
98.56
99.18
99.30
278 95.6 71
15 99.52
99 29 99.74
98.42
98.08
93.53
97
98.14
98.58
98.47
98.
98,
25
96
12
98.61
98.73
98.71
99.19
98.58
98,23
99.43
99.34
279 67, 14 69 95 68.
93 24 69 29
69.69
68.66
68.76
63,50
66.35
68.93
69.32
67.00
69.
24
69
60
68
98
68
75 67.39
66.92
68.83
69.06
69.38
69.06
69.06
69.10
282 49. 14444444 46. 96 50 60.6665556666666 70
51 46
51.47
50.70
$0.90
47.51
48.51
50.43 $1.53
49.80
50.70
51
46.92
47. 50 50.85
$2.43
$1.00
51.63
50.33
49.94
50.96
52.38
283 97.9 97.91
91
97. 63 97.74
45
97.57
97.73
97.45
97.91
96 29
97.83
97.82
97.62
96.76
97.8 81 50.40
64.04 97.62
97.78
97.82
97.85
47.40
97.91
47.77
97.78
285 72. 94 75 96 06.04 29 72.78
78.12
82.35
84.94
78.20
65.84
80.43
78.24
80.63
84.39
81.
81.
76
75. 76 85.76
80.24
81.33
82.31
77.84
84.35
80.90
84.47
288 92.44 68 68
99 58 65
99
30
82
93.41
93.94
90.96
92.02
92.74 97.97
95,
92.82
92.72 92.23 23 90.
97.73
98.77
98.89
99,30
98.39
94,94
99.22
99.12
289 90.5 89 50 90. 89. 98 90.01
90.41
90.78
91.05
90.06
89.21
90.72 90.13
90.40
90.56
90.25 90.52 90.33 90.50
89.
90.72
90.92
89.78
89.56
89.60
90.45
290 85. 10
85
85.96
10
84.95
85.57
86.23
86.25
82.67
86.02 86.11
85.45
85.88
85.32
54 40.77 86.04
86.24
86.24
86.70
86.09
85.91
86.29
86.50
291 84, 10 86 20 83 69
84,38 82.9
81.73
85.87
86.38
85.69
80.36
85.69
86,42
85,57
85.61
85.67 85.57
85
86.26
87.02
85.51
86.27
85.04
86.53
85,59
86.09
293 86.72 72
86 90.43
89
88.17
87.31
88.73
89.62
83.91
88.95
90.14
86.77
90.87
86.81
89.72 87 89.50
87.53
B8.14
89.64
87.23
89.94
86.96
90.99
295 77. 10 70.45 75
75.34 73.86
73.39
72.68
72.59
75.17
72.04
71.90
72.34
71.62
72.11
72.05
28 74.48 73.93
74.82
72.98
72.90
72.20
73.41
71.81
73.96
297 65.6
8 63.62 73
29
63.60
58.67
58.08
58.23
54.69
57.17
58.19
55.67
56.17
58.52
51.92 75.
63.62
60.65
58.75
55.15
58.13
58.29
60.52
298 63.5 37 53.06 42
02
35
62.95
57.79
$7.90
54,30
49.56
54.98
57.17
60.93
55,86
58.84 53.08
31.65 62.65
63.27
59.42
60.81
53.27
57.95
53.49
60.56
299 59.60
58.59 59.84
60.83
62.47
58.25
60.31
60.34
59.57
52.55
59.12
59.82
$9.06
59.28
57.53
59.
16.66
59,86
60.34
59.87
59.26
56.72
59.57
60.88
61.12
24

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
Table 11: Classification accuracy (mean + standard deviation) across 20 high-dimensional datasets. Each method is evaluated over 5 splits and random seeds, and the best-performing method for each dataset is highlighted. "MNCA", "XGB", "RF", "PG" denote "ModemNCA",, "XGBoost", "RandomForest", "ProtoGate", respectively.
Dataset
BETA
MNCA
TabM
MLP
KNN
XGB
RF
PG
CLL SUB
69.86
02.90
72.17 13.82
62.46.6.22
57.39 16.96
73.04
70.14
67.61
BASEHOCK
96.5640.77
96.31
96.89+a.81
97.01
71.88
95.2944.78
96.73_1.14
96.32_2.70
Prostate GE
88.89
81.27=8.60
90.1645.95
86.98_6661
80.00
89. 89.52.6607
87.94_7.75
84.13
PCMAC
8..77
88.21 1.44
88 88.02
18.53
66.4815.
91.6441.45
92.20 1..09
88.21 23.79
GLI 85
86.67
81.57
79.22++6.02
85,49
76.47
82.35
83.92
81 81.96
RELATHE
1.445
88. 1.212
89.91 1.29
90.54 1:770
75.03 4..03 4.775
87.1 1:119
87.30
89 89.92
SMK CAN 187
7.06
63.51 66.36
67. 27.02.+sh 66.00
26.84
69.47
66.49 66.49.66.8 46.62
70.70=4.17
70.71
WarpPIE10P
100.00.0.00
98.41 2:1.0
100.00.1.00
19.05
92.38
94.9213.46
98.57
97.79 6.68
leukemia
95.11433.90
=5.22=6.00
91.1 91.11 14.66
95.1 22.95
86.67
97.78
92.00=22.7
94.00
orlraws
96.67
97.67
99.00
98.33
92.0
84.33
99.00
92.67 25.73
GLIOMA
72.67.40.29
58.00
62.00
60.67
68.00
66.67 18.69
64.00
69.91
lung_discrett
12.00
28.84
76.00
71.56
15.93
77.33
71.56
70.01 27.88
WarpARTOP
91.54
83.08
96.15 96.15+.4.21
85.64_26.95
53.08
81 81.28490.0
87. 87.18-6.77
90.04
TOX 171
91.24
76.00 26.33
87.81 1:3.04
$8.19
70.86
78.10
78.67 15.10
85.52=7.41
lung
95.77+1.06
91.54-23.4
93.01 4:2.8
95.45
93.6641.55
93.66422.4
92.68
95.43
ALLAML
96.89 1.4.12
87.56
92.00122.07
95.56
81.33
96.00 14.07
96.44_3.33
91.14
colon
18.46
82.5647.14
78.97 -9000
12.87
18.17
82.56-6.67
78.46_8.04
gisette
97.32.0.21
97.15
97.57 10.32
95.04
97.55
96.82
97.18
madelon
58.29 42.11
64.97
56.97
56.44
53.58
75.77 +2.53
70.42_2..0
72.13
arcene
86.67
81.67 23.84
78.83 16.38
85,50
84.50
75.00
86.83 25.44
85.33 24.82
25

---

TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems
(a) Raw Feature
(b) Bank MLP
(c) Bank ModernNCA
(d) Bank TabR
(c) Bank BETA
Bank BETA
(g) Bank BETA
(h) Bank BETA
(a) KDD Raw Feature
(b) KDD
(c) KDD ModernNCA
(d) KDD TabR
(e) KDD BETA
(I) KDD BETA
(g) BETA
(b) KDD BETA 4
(a) Spambase Raw Feature
(b) Spambase MLP
(c) Spambase ModernNCA
(d) Spambase TabR
(e) Spambase BETA
(1) Spambase BETA
(g) Spambase BETA
(h) Spambase BETA
Figure 9: Visualization of the embedding space of different methods.
26


Reviews:
[{"review_id": "az5qGxFPt1", "reviewer": "Reviewer_GHpF", "summary": "The manuscript first analyzes the generalization error with a bias-variance decomposition, finding both bias and variance have a non-negligible contribution to the overall error. They then propose an extension tackling both error sources named BETA, which trains multiple dataset-specific encoders (to tackle bias) while averaging the predictions resulting from the different encodings (to minimize variance). Additionally, to further reduce variance during inference, they also use bootstrapped sampling, meaning several random subsets of the training subset as support sets and averaging of the resulting prediction. To scale to more classes, they use Error-Correcting Output Codes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:\n\n---\n**Q1: Finetuning time plus inference time compared to other methods.**\n\n---\n**A1**: In **Table 3**, we have provided a comparison of **inference time, average rank, and the number of learnable parameters**.\n\nTo address this point more comprehensively, we include the **fine-tuning/training** time for **BETA** as well as other methods listed in Table 3. Notably, since **Beta only fine-tunes the encoder parameters**, it has a clear advantage in both **fine-tuning and inference efficiency** compared to other variants.\n\n| Methods                      | Beta  | LocalPFN | MixturePFN | FT-T | TabR |\n| ---------------------------- | ----- | -------- | ---------- | ---- | ---- |\n| finetuning/training time (s) | 62.29 | 186.73   | 83.14      | 58.9 | 121  |\n| inference time (s)           | 0.91  | 12.51    | 3.24       | 0.36 | 0.38 |\n\nAdditionally, it is important to highlight that methods like **FT-T and TabR** underwent an extensive hyperparameter tuning process in [2], spanning **100 trials**. As a result, their actual tuning time is approximately **100 times** their reported training duration.\n\n---\n**Q2: Claim: Response to \"Scales to Large Datasets\"**\n\n---\n**A2**: We refer to datasets with a **large number of samples** as **large datasets**. \n\nTabPFN is inherently limited by its **attention mechanism over instances**, leading to high memory overhead. As a result, it typically requires subsampling the training set to remain computationally feasible. To mitigate the impact of context size on model performance during inference, we introduce bagging for TabPFN.\n\nAdditionally, since TabPFN is pretrained only on synthetic datasets with fewer than 1000 rows, we incorporate multiple lightweight encoders to reduce bias, enabling the model to better scale to large datasets.\n\n---\n**Q3: About TuneTables: Formulation seems strange to me, TuneTables only optimizes the prompt, so what does this \"either finetuning the entire model\" what do you mean here?**\n\n---\n**A3**: In **classification tasks**, TuneTables employs **prompt tuning**, where *“which adjusts only a small set of parameters, reducing resource consumption”* specifically refers to prompt tuning. However, in their **extension to regression tasks**, they adopt **end-to-end fine-tuning**. This is why we mentioned *\"fine-tuning the entire model.\"* We will clarify this distinction in the final version.\n\n---\n**Q4: I don't see how it \"ensures\" \"diverse representations\", there is nothing enforcing that as far as I see, so ensure seems a misleading formulation here (as one could imagine losses that would ensure it).**\n\n---\n**A4**: We appreciate the reviewer’s observation. The goal is to ensure that each encoder generates **TabPFN-compatible latent representations**, while diversity is encouraged through different initializations.\n\nAdditionally, **TabM** [1] has discussed diversity in base models, and we further analyze BETA’s encoder diversity through **embedding visualizations in Appendix Figure 9**, showing that the learned representations exhibit diversity. We will provide a detailed explanation of **diversity** and **TabPFN-compatible latent representations** in the final version.\n\n---\n**Q5: Response to Figure Presentation.**\n\n---\n**A5**: We appreciate the reviewer’s helpful suggestions.\n\n- **Figure 3**: We will adjust the alignment of **\"Inference Stage\"**, **\"Mean\"**, and **\"Loss\"** to ensure proper centering within their respective boxes.\n- **Figure 5**: We will modify the **x-axis** to directly represent **actual dataset sizes** and consider using a **scatter plot of dataset size vs. relative improvement** for better clarity. Additionally, we will correct the **typo in the y-axis**.\n\nThese improvements will be incorporated into the final version.\n\nWe sincerely appreciate the reviewer’s **careful and constructive feedback**. We believe these valuable suggestions will **further enhance the quality** of our paper, and we will incorporate the necessary revisions accordingly.\n\n[1] Tabm: Advancing tabular deep learning with parameter-efficient ensembling. 2025.\n\n[2] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024.", "claims_and_evidence": "The claim \"Scales to Large Datasets\" I am not sure if it is supported, do you mean through the error-correcting output codes? do they affect scaling behavior if there are only two classes? It seems unclear to me how \"scales to large datasets\" is meant, given \"handles high-dimensional data\" and \"adapts to more than 10 classes\" is separately mentioned.\n\n[Update: explained by authors to mean the bagging part]", "methods_and_evaluation_criteria": "Benchmark itself seems suitable as far as I can judge it.\n\nI am missing an evaluation of the overall computation time, so finetuning time plus inference time compared to other methods, Seems very relevant here to me. [Update: promised by authors to add]", "theoretical_claims": "None", "experimental_designs_or_analyses": "See methods", "relation_to_broader_scientific_literature": "Results on one method to further improve tabular in context prediction and marry it with partial finetuning.", "essential_references_not_discussed": "None I am aware of", "other_strengths_and_weaknesses": "See below", "other_comments_or_suggestions": "\"To mitigate this, TuneTables (Feuer et al., 2024) offers a more efficient alternative by either fine-tuning the entire model or applying prompt-tuning (Lester et al., 2021), which adjusts only a small set of parameters, reducing resource consumption\" -> Formulation seems strange to me, TuneTables only optimizes the prompt, so what does this \"either finetuning the entire model\" what do you mean here?\n\n[Update]: explained by authors. thanks\n\n\"Minimizing L_total in Equation 6 ensures that the model jointly trains all encoders to generate diverse yet TabPFN compatible latent representations, thus reducing variance and providing more stable predictions.\"\nI don't see how it \"ensures\" \"diverse representations\", there is nothing enforcing that as far as I see, so ensure seems a misleading formulation here (as one could imagine losses that would ensure it).\n\nIn Fig.3 , \"Inference Stage\" is not centered in the box, also \"Mean\"/\"Loss\" isn't? [Update: change promised, thanks]\n\nFigure 5,  I think the x axis is just confusing in this way, it should represent actual dataset sizes, and then maybe one could just make a scatter plot of dataset size vs relative improvement  (notice also typo in y axis).  [Update: change promised, thanks]", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvzkc7seWk", "reviewer": "Reviewer_3Q5h", "summary": "This paper narrows its study to an adaption method for TabPFN, which incorporates a fine tuning encoder, boostrapped sampling in the inference stage into the whole pipeline. The BETA is able to solve the bias and variance and achieve comparable performance.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:\n\n---\n**Q1: Limitation of Novelty**\n\n---\n**A1**: BETA builds on existing techniques, with its novelty lying in breaking key limitations of TabPFN while maintaining inference efficiency. Simplicity and effectiveness are highly valuable, especially for tabular data:\n\n1. Overcoming TabPFN’s Limitations:\n   - (a) **Handling Arbitrary Feature Dimensions**: We introduce multiple lightweight encoders, incorporating *batch ensemble* techniques, to map raw features into a unified dimension while keeping TabPFN’s parameters frozen. \n   - (b) **Scaling to Large Dataset**: Bagging extends TabPFN to datasets with more samples.  \n   - (c) **Handling Large-Class Problems**: Assigning ECOC codes to different encoders overcomes TabPFN’s class limitations. \n2. BETA remains compatible with PFN-style batching, ensuring minimal extra model inference computational cost. \n3. An effective combination of techniques for TabPFN that addresses key weaknesses and significantly outperforms widely used, finely tuned baselines is a meaningful contribution worthy of further study.\n\n---\n**Q2: But the authors claims that on more other datasets, BETA demonstrates superior performance desipite that this experiment verify the BETA should perform better on large scale datasets.**\n\n---\n**A2**: This experiment aims to demonstrate BETA’s improved generalization on large datasets. As shown in Figure 5, BETA’s performance gain over TabPFN grows with dataset size.  \n\nHowever, on the largest datasets, LocalPFN performs better because it assigns a unique context to each sample, preventing context sharing and making it **incompatible with the highly parallelizable batching strategy used in PFN**. While this boosts performance, it comes at the cost of much lower efficiency—as shown in Table 3, inference time increases significantly. \n\nLocalPFN is a valuable contribution, but this highlights a trade-off on large datasets: BETA offers better efficiency, while LocalPFN prioritizes local adaptation at a high computational cost.\n\n---\n**Q3: Figure 2 introduce some settings based on TabPFN, missing the variants of table 1.**\n\n---\n**A3**: In Section 2.2, we categorize major TabPFN variants into Context Selection and Fine-tuning. As shown in Table 1, all listed variants fall into these two categories.\n\nAs stated in Section 2.3, we selected TabPFN-KNN and TabPFN-Finetune as **the most representative methods** for these categories, with other approaches being minor modifications or combinations of these two. To ensure clarity and avoid redundancy, Figure 2 presents results for these key methods. \nThe following table shows the relative percentage changes in bias and variance (**negative values indicate reduction**) of Beta and other variants compared to TabPFN, averaged over the dataset sizes used in Figure 2, on the *Adult* and *Bank* datasets. Since TabPFN's absolute variance on these datasets is smaller than its bias, the percentage changes in bias are much smaller than those in variance.\n\n|Adult|Beta|TuneTables|TabForestPFN|LocalPFN|MixturePFN|TabPFN-Bagging|\n|-|-|-|-|-|-|-|\n|Bias|**-4.29** | -1.58 | -2.96 | -3.67 | -3.47 |+1.75|\n|Variance|**-16.52** | +17.93 | +7.76 | +4.38 |+5.26|-10.16|\n|Bank|||||||\n|Bias|**-1.48** |-0.25|-1.12|-1.39|-1.09 |+0.39|\n|Variance |**-39.07**|+25.13|+18.47|+23.02 |+10.35|-9.37|\n\n---\n**Q4: No evidence to show the claims in Table 1.**\n\n---\n**A4**: These claims are derived from the specific improvement strategies each method applies to TabPFN. For example, none of the other methods adjust TabPFN’s input representation, meaning they cannot handle high-dimensional datasets. Additionally, evidence supporting the claims about **bias and variance** can be found in *Figure 2* and the table in *A3* above.\n\nWe greatly appreciate your suggestion and will further clarify Table 1 in the final version to ensure the claims are well-supported.\n\n---\n**Q5: Why are the encoders in fine-tuning stage placed before TabPFN instead of after it?**\n\n---\n**A5**: \n\n1. Placing encoders before TabPFN allows it to handle **arbitrary feature dimensions** by mapping raw features to a fixed dimension. \n\n2. Our preliminary explorations found that adding encoders after TabPFN provided almost no improvement. \n\n3. Moreover, placing them before TabPFN integrates effectively with batch ensemble techniques, enhancing performance.\n\n---\n**Q6: Some foundation tabular model is missing**\n\n---\n**A6**: CARTE models tabular data as a graph, embedding column names and entries with a graph-attentional network. UniTabE encodes column names, data types, and cell values as tokens using an encoder-decoder architecture with contrastive learning.\nUnlike TabPFN, which relies on a pre-trained Transformer without semantic modeling, these models incorporate semantic information. We will enhance the discussion on tabular foundation models in the final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "A narrowed contribution to certain model", "essential_references_not_discussed": "Some foundation tabular model is missing, such as [1] and [2]\n\n[1] Kim, M.J., Grinsztajn, L. &amp; Varoquaux, G.. (2024). CARTE: Pretraining and Transfer for Tabular Learning. Proceedings of the 41st International Conference on Machine Learning\n[2]Yang, Yazheng et al. “UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science.” International Conference on Learning Representations (2023).", "other_strengths_and_weaknesses": "Pros:\n\n1.\tThis work is a new variant of TabPFN, which aims to solve the bias and variance problem in the original TabPFN model. The problem is well motivated by several explorative experiments.\n2.\tThe method incorporates previous techniques to mitigate both bias and variance issues, which is easy to understand.\n3.\tExperiments demonstrate the effectiveness of the proposed method.\n\nCons:\n\n1.\tThe proposed BETA considers encoder to mitigate bias and bootstrapped sampling for variance, which are the techniques already used. Only incorporating both techniques into one pipeline makes this work limited in novelty.\n2.\tThe table 1 gives the comparions between BETA and other variants of TabPFN. However, there is no evidence to show whether the claims are justified.\n3.\tThe experimental design is not so convicing. For example, scale to large datasets, BETA performs worse than LocalPFN on the large scale datasets. But the authors claims that on more other datasets, BETA demonstrates superior performance desipite that this experiment verify the BETA should perform better on large scale datasets. On investigating that BETA is able to reduce bias and variance, the Figure 2 introduce some settings based on TabPFN, not the variants of table 1, which makes the conclusion is not so convicing.", "other_comments_or_suggestions": "1.\tThe experiments can be carefully designed according to the claims of the proposed method. \n2.\tMore recent studies on foundation model or transfer model should be discussed.", "questions_for_authors": "Why are the encoders in fine-tuning stage placed before TabPFN instead of after it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pIGrM3Q2V", "reviewer": "Reviewer_RCWy", "summary": "The authors introduce BETA, a TabPFN variant featuring multiple improvements to the original TabPFNv1 model. BETA introduces encoder-based fine-tuning, multiple encoder fine-tuning, batch-ensemble encoding for inference optimization, inference-time bootstrapped sampling, and an error correcting output code strategy to extend model support beyond 10 target classes. Across over 200 datasets and 15 seeds, BETA outperforms a suite of 28 baseline methods with 0.05 CD significance. BETA also performs well on high-dimensional datasets and maintains TabPFNv1-level inference throughput and a small model artifact of under 1 MB. The authors additional show ablation studies covering variants of TabPFN in Figure 2 with discussion of bias-variance tradeoff impact.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:\n\n---\n**Q1: What is the validation set used for?  Were the models early stopped to maximize accuracy?**\n\n---\n**A1**: We follow [1, 2, 5, 6], where the validation set is used only for hyperparameter tuning and early stopping, not for training. For example, for baselines like LightGBM, XGBoost, and FT-Transformer, we use the validation set for early stopping. For BETA and TabPFN variants, we use it for early stopping but do not refit on the combined training and validation set.\n\n---\n**Q2: Why accuracy?**\n\n---\n**A2**: We followed the evaluation metrics used in [1, 2, 5, 6]. We also computed AUC-based average ranks (see below). RealMLP's lower AUC ranking may be linked to label smoothing, aligning with findings in [3].\n|Method|avg rank|\n|-|-|\n|Beta|5.1|\n|catboost|7.3|\n|ModernNCA|7.64|\n|LightGBM|8.92|\n|RealMLP|10.04|\n|TabPFN|10.38|\n|FT-T|10.92|\n|MLP|12.19|\n\n---\n**Q3: Bagging**\n\n---\n**A3**: We agree that other baselines may benefit from bagging. However, since TabPFN can **change context without retraining**, we only applied inference-stage bagging for BETA. As a result, for other baselines, bagging would **significantly increase the cost**—scaling nearly linearly with the number of base models.\n\nTo evaluate bagging’s impact on other models, we trained 16 classifiers per dataset with different random seeds on subsets of size max(0.8 × train_size, 1000) for CatBoost, LightGBM, and RealMLP, ensuring that each subset was at least as large as the ones used by BETA during inference. \n|Method|avg rank|\n|-|-|\n|Beta|6.1|\n|realmlp-bagging|6.97|\n|catboost-bagging|7.42|\n|catboost|8.06|\n|realmlp|8.91|\n|lightgbm-bagging|9.37|\n|lightgbm|10.01|\n\nWe appreciate your suggestion and will further enhance the discussion on Bagging in the final version.\n\n---\n**Q4: HPO**\n\n---\n**A4**: We followed the hyperparameter tuning approach of [1, 2, 5, 6]. We carefully read the article you referenced [4] and implemented the 5-fold CV evaluation protocol from [4] (Section 2). However, we found this protocol to be significantly **more computationally intensive**.\n\nDue to computational constraints, this approach was applied only to XGBoost, RealMLP, and Beta on classification datasets with fewer than 3,000 rows (from [1]).\n\nFor XGBoost and RealMLP, we performed HPO using 5-fold CV, then averaged the predictions from the 5 trained models.\nFor BETA, we maintained its default hyperparameters but trained five models on different training folds and averaged their predictions.\n\nThe results, shown in the table below, indicate that cross-validation does improve model performance, even for methods like BETA that do not require HPO, though the gain for BETA is relatively modest.\n\nWe sincerely appreciate your valuable feedback on HPO strategies, and in the final version of the paper, we will conduct a more detailed evaluation of these effects.\n\n|Method|Avg AUC|\n|-|-|\n|XGBoost|0.8452|\n|XGBoost-cv|*0.8470*|\n|RealMLP|0.8610|\n|RealMLP-cv|*0.8739*|\n|Beta|0.8759|\n|Beta-cv|*0.8785*|\n\n---\n**Q5: I agree with the authors that TabPFNv2 is concurrent work, however I think it would elevate the paper if it is possible to include a version of Figure 4 with TabPFNv2 results in the appendix.**\n\n---\n**A5**: In the final version of the paper, we plan to include additional baselines. Due to time constraints, we evaluated TabPFN-v2 and TabICL, both using default settings.\n\nTo ensure a fairer comparison, we also tested subsampled variants (TabPFN-sub, TabICL-sub) with 16 training subsets and different preprocessing strategies.\n\nResults show that full versions of TabPFN-v2 and TabICL approach BETA’s performance, while **subsampled versions perform significantly worse**. Our method is currently limited by memory constraints, but a well-optimized TabPFN-v1 could allow larger context sizes, potentially further improving results.\nThis suggests that our method is currently constrained by engineering-related memory limitations. If a highly optimized version of TabPFN-v1 were available, we believe our approach could scale to even larger contexts, potentially further improving performance.\n\n|Method|avg rank|\n|-|-|\n|Beta|8.65|\n|PFN-v2|8.76|\n|TabICL|9.16|\n|PFN-v2-sub|12.56|\n|TabICL-sub|12.76|\n\n---\n**Q6: Sensitivity Analysis**\n\n---\n**A6**: Please refer to A1 for reviewer 2vtp.\n\n[1] Revisiting deep learning models for tabular data. 2021\n\n[2] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024\n\n[3] Better by default: Strong pre-tuned mlps and boosted trees on tabular data. 2024\n\n[4] Unreflected Use of Tabular Data Repositories Can Undermine Research Quality. 2025\n\n[5] On Embeddings for Numerical Features in Tabular Deep Learning. 2022\n\n[6] TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023. 2023", "claims_and_evidence": "The author's put substantial effort in providing convincing evidence for the claims they make in the paper.", "methods_and_evaluation_criteria": "Yes. The evaluation criteria, proposed methods, and comparison baselines all make sense for the target application.", "theoretical_claims": "While I have not dived deep into the specifics of the theoretical claims, the approaches described make sense and appear reasonable.", "experimental_designs_or_analyses": "The experimental design appears solid and sufficiently large-scale to warrant statistically significant take-aways. I do however have some major reservations based on the specifics of how the experiments were conducted to ensure a fair comparison for all methods, which I detail below and look forward to the authors providing clarifications on:\n\n### \"Each dataset is randomly split into training, validation, and test sets with proportions of 64%, 16%, and 20%, respectively.\" & \"To ensure a fair comparison, all TabPFN variants, including BETA, are evaluated using their default hyper-parameters without additional tuning\"\n- What is the validation set used for? TabPFN generally doesn't need to early stop or hyperparameter tune, so are we just throwing away useful data as unused validation data instead of incorporating it into the train_data to fit a better model?\n- Is the validation dataset combined with the training data for the final model fit after HPO?\n\n### We report accuracy as the evaluation metric, where higher accuracy indicates better performance. \n- Why accuracy? Generally ROC AUC or logloss are more informative. Accuracy can be highly sensitive to the decision threshold, which is not something every model includes as a tunable parameter. Accuracy also doesn't do well in measuring how well calibrated the model is.\n- Were the models early stopped to maximize accuracy? For example, LightGBM, XGBoost, RealMLP, etc. can all be early stopped using validation data to optimize a target metric. This would greatly improve their results. Were the fine-tuned TabPFN methods (including BETA) early stopped to optimize accuracy? If so, were these methods ever refit on all of the data afterwards using the found optimal epoch/iteration?\n\n### SOTA Claim & Bagging\n- BETA essentially implements internal bagging. Bagging is well known to be virtually universally helpful in improving the strength of models. This also applies to models such as RealMLP, LightGBM, CatBoost, etc. Have the authors considered also fitting bagged versions of the baselines to more faithfully compare top-end performance potential of the methods? I would suspect BETA to at best only marginally improve with explicit bagging (since it already incorporates internal bagging), whereas methods such as RealMLP, CatBoost, etc. will likely benefit greatly from it. This may somewhat change the take-aways in the paper. Note that even if BETA is no longer #1, that does not become a weakness of the paper.\n\n### SOTA Claim & HPO\n- BETA and the TabPFN methods are not hyperparameter tuned, however the other methods are. Tuning via a single train/val split is suboptimal and can lead to major overfitting. Tuning via cross-validation is much more preferable and would lead to better results, especially on smaller datasets. Have the authors considered doing this to compare with stronger baselines?\n\n### General comment on experimental design\nI suggest the authors read a recent paper detailing shortcomings of many recent tabular benchmark studies, including those that this paper builds its experimental design off of: https://arxiv.org/pdf/2503.09159 . The paper shares similar concerns that I have mentioned above, such as the usage of a holdout set instead of cross-validation.", "relation_to_broader_scientific_literature": "The contributions are highly related to the TabPFN and tabular foundation model literature, many of which are recent concurrent works (TabPFNv2, Attic, TabICL, etc.).", "essential_references_not_discussed": "The authors do a good job of highlighting relevant works.", "other_strengths_and_weaknesses": "I like the usage of Batch Ensemble for compute efficiency. In general the authors apply many clever tricks which are very sensible.\n\nIt would have been nice to have an ablation study of iteratively adding the BETA improvements to TabPFN, showing the impact of each component that make up BETA. This would be a very strong addition and would help justify each component (whereas currently it is unknown if certain components are largely superficial / non-impactful).", "other_comments_or_suggestions": "I agree with the authors that TabPFNv2 is concurrent work, however I think it would elevate the paper if it is possible to include a version of Figure 4 with TabPFNv2 results in the appendix.\n\nMethods that are missing from Figure 4 that could be useful to include:\n- TabPFNv2\n- TabForest\n- TabForestPFN\n- TabPFNMix\n- Attic\n- TabICL\n- An AutoML system, as done in the TabPFNv2 paper\n- Bagged baseline models", "questions_for_authors": "Refer to the other sections. My score is contingent on the notable questions and concerns being addressed. The paper is generally of very high quality, but with some major potential pitfalls that might lead to incorrect conclusions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "opzu6pe7Gz", "reviewer": "Reviewer_2vtp", "summary": "This paper introduces BETA, a method to improve the scalability and performance of TabPFN (a transformer-based technique for tabular classification). BETA combines (1) a lightweight encoder, fine‐tuned (with the pre-trained TabPFN frozen) to re-map raw features into a latent space and better align with downstream data distributions (to reduce bias), (2) multiple encoder paths using a Batch Ensemble strategy to introduce diversity and reduce variance, (3) bootstrapped sampling during inference to generate diverse support and prediction sets, which are aggregated (uniformly or with weights), (4) Error-Correcting Output Codes to extend the model to multi-class classification tasks (beyond 10 classes).\n\nExtensive experiments on benchmark datasets (e.g., TALENT, high-dimensional datasets) and ablation studies, as well as bias-variance analyses, support the claim that BETA improves accuracy and efficiency relative to prior TabPFN variants and other state-of-the-art tabular models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:\n\n---\n**Q1: Sensitivity Analysis**\n\n---\n**A1**: We sincerely appreciate the reviewer’s valuable suggestion. To address the sensitivity of BETA’s performance to the **number of bootstrapped samples** and **encoder paths**, we have conducted additional experiments.\n\n- In **Appendix D.3**, we provide an analysis of the **number of encoders** and other key components in BETA.\n\n- Additionally, we include performance rankings for BETA across various datasets from the **Tiny Benchmark2** classification dataset [1], where we vary the **number of bootstrapped samples**. A table summarizing the results is presented below:\n  | size         | 500  | 1000 | 2000 | 3000     | tabpfn |\n  | ------------ | ---- | ---- | ---- | -------- | ------ |\n  | average rank | 3.39 | 2.87 | 2.57 | **1.91** | 4.24   |\n- We also summarize the performance of Beta with different **number of encoders** and their corresponding **average training time** across the datasets of **Tiny Benchmark2**. Specifically, we show the **average percentage change in performance relative to the original TabPFN**.\n  | number of encoders   | 1    | 2    | 4     | 8     | 16    |\n  | -------------------- | ---- | ---- | ----- | ----- | ----- |\n  | relative improvement | 1.7% | 2.6% | 3.19% | 3.23% | 3.76% |\n  | finetuning time (s)  | 31   | 48   | 73    | 126   | 203   |\n  \n\nThe result demonstrates that both **the number of bootstrapped samples** and **the number of encoders** lead to **better performance**, but they also **increase computational costs**.\n\nWe hope these additional details address the reviewer’s concerns. Additionally, we will provide a brief summary of the **experimental details and sensitivity analysis** in the main text of the final version of the paper.\n\n---\n**Q2: Recent Work on Contextual Embeddings**\n\n---\n**A2**: We sincerely thank the reviewer for the suggestion. We have noted that the paper referenced by the reviewer, [2], provides a detailed analysis of the **combination of ensemble methods with advanced embedding techniques**. We will incorporate a discussion of this work and related research in the final version of the paper. Additionally, we plan to explore how these techniques could be integrated with BETA in our future work.\n\nWe appreciate the reviewer’s insight and will ensure this topic is adequately addressed in the final version of the manuscript.\n\n---\n**Q3: Extending BETA to Regression Problems**\n\n---\n**A3**: Thank you for your insightful comment. We applied BETA to regression datasets by using a pre-trained regression version of PFN, where the final classification head has an output dimension of 1, while keeping the other modules unchanged. The only modification was replacing the loss function with MSE loss and fine-tuning it on downstream regression datasets.\n\nWe evaluated the regression version of BETA on 100 regression datasets from **[1]** and compared the **average rank** to other tuned methods (we only display a subset of the methods for clarity) presented in **[1]**. Our results show that the regression version performs well, although its performance is slightly lower compared to its classification counterpart.\n\n|Method|catboost|tabr|Beta|FT-T|XGBoost|MLP|KNN|\n|-|-|-|-|-|-|-|-|\n|avg rank|7.3|9.2|9.3|9.8|10.0|12.7|17.8|\n\nWe will clarify these findings and potential directions for future work in the final version of the paper. Thank you again for your valuable suggestion!\n\n---\n**Q4: Response to Testing BETA under Distribution Shifts**\n\n---\n**A4**: Thank you for your thoughtful question. We have tested BETA on three classification datasets from **TabRed** [3] and integrated the **split method** (Beta-split) and **temporal embedding techniques** (Beta-temporal) discussed in [4] into BETA. The table below presents the **AUC results** of **BETA** on these datasets, showing that the plugin techniques are effective for improving BETA’s performance. Additionally, these results highlight the potential of **ICL methods** for addressing **temporal shift** scenarios.\n\n|dataset|Beta|Beta-split|Beta-temporal|MLP|\n|-|-| -|-|-|\n| ecom-offers|0.5692|0.6264| **0.6348**|0.5866|\n| homesite-insurance|0.9600| **0.9602** |0.9521| 0.9404 |\n| homecredit-default|0.6114| **0.6710** |0.6637| 0.4730 |\n\n\nWe will include these findings and further clarify this in the final version of the paper. Thank you again for the valuable suggestion!\n\n[1] A Closer Look at Deep Learning Methods on Tabular Datasets. 2024\n\n[2] Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers. 2024\n\n[3] Tabred: A benchmark of tabular machine learning in-the-wild. 2025\n\n[4] Understanding the Limits of Deep Tabular Methods with Temporal Shift. 2025", "claims_and_evidence": "The claims are supported by comprehensive experiments over 200+ datasets, detailed ablation studies, and analyses of bias-variance decomposition. The experimental results, significance tests (e.g., Wilcoxon-Holm tests), and efficiency metrics (inference time, parameter count) provide empirical support.\n\nWhile the empirical evidence is good, some claims regarding theoretical guarantees for bias and variance reduction rely on standard decompositions (i.e., the bias-variance decomposition, Eq. (13), is well established in the literature). Additional theoretical justification or analysis could further strengthen the bias and variance reduction claims.", "methods_and_evaluation_criteria": "The introduced methods are sound. The use of diverse benchmark datasets (e.g., the TALENT benchmark and high-dimensional datasets) along with multi-seed experiments and statistical significance tests is appropriate for evaluating the proposed methods.\nA more in-depth sensitivity analysis regarding hyperparameter settings (e.g., number of bootstrapped samples or encoder paths) would enhance the evaluation.", "theoretical_claims": "The theoretical aspects used in this paper are well-established in the literature.", "experimental_designs_or_analyses": "The experimental setup is robust. The authors use multiple random splits and seeds and compare against a range of competitive baselines (classical and deep learning models). The bias-variance analysis is clearly presented; efficiency metrics (inference time, checkpoint size) are well reported.\n\nSome important experimental details (e.g., hyperparameter ranges and exact training protocols) are in the supplementary material. Including a brief summary in the main text would improve the clarity.", "relation_to_broader_scientific_literature": "The paper builds directly on prior work in tabular deep learning, notably TabPFN, TuneTables, and MixturePFN, while incorporating ensemble methods and parameter-efficient fine-tuning techniques known from the broader deep learning literature. There is a strong focus on TabPFN techniques; the combination of ensemble methods with advanced embedding techniques (e.g., https://arxiv.org/abs/2411.01645) could be further analyzed and discussed.", "essential_references_not_discussed": "There are several recent papers exploring how the rich contextual and semantic representations generated by large language models can be integrated into prediction models for tabular data. Mentioning this stream of work in the introduction would inform and help readers understand the broader context.", "other_strengths_and_weaknesses": "Strengths:\n1. The approach builds on well-established techniques.\n2. The experimental evaluation is extensive and covers a wide range of datasets and settings.\n3. The integration of ECOC to overcome multi-class limitations is well-motivated.\n\nWeaknesses:\n1. The theoretical foundation of the method could be stronger.\n2. Some critical implementation details and hyperparameter sensitivity analyses should be summarized in the main part.\n3. The limitations section is brief, and further discussion on potential challenges (e.g., distribution shifts, application to regression) would be welcome.\n4. More recent work on improving predictions on tabular data through contextual embeddings could be included and discussed.", "other_comments_or_suggestions": "1. A brief discussion on the computational trade-offs of adding more encoder paths versus performance gains.\n2. See weaknesses above.\n\n## After Rebuttal \nThe authors do address significant and well-known shortcomings of TabPFN and demonstrate consistent improvements on a wide benchmark. However, I do strongly agree with Reviewer 3Q5h that the code should be public. I strongly assume the link to the open-source code will be made available upon publication. Otherwise, there would be a serious replicability limitation. In good faith that the authors will open-source their code and adjust the paper according to the reviews, I would maintain my current score.", "questions_for_authors": "1. Can you elaborate on how the combination of multiple encoder paths specifically contributes to variance reduction? Are there any theoretical bounds or insights that could be provided?\n2. How sensitive is the performance of BETA to the choice of the number of bootstrapped samples and encoder paths? Would you consider adding a sensitivity analysis section?\n3. You briefly mention the limitation regarding regression tasks. What modifications would be needed to extend BETA to regression problems, and do you have preliminary results in that direction?\n4. Have you tested BETA under scenarios with distribution shifts (covariate or concept drift)? If not, could you discuss potential approaches to handle such scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission introduces a generalization of the TabPFN model, a pretrained model for tabular learning. The work analyze that the predictor error comes from both bias and variance, and the TabPFN generalization combines multiple ideas to address them, such as bagging, a better fine-tuning strategy and better multiclass support. The submission generated much interest from the reviewers and discussion. The reviewers appreciated the performance improvements and the evaluation, including the work during the rebuttal to add a comparison to work released after the ICML deadline.

There was a discussion amongst reviewers and AC on the importance of open sourcing the code underlying the present work. The AC would like to stress the need for open sourcing once the work is accepted.
Decision: Accept (poster)