ICML 2025

TabPFN Unleashed: A Scalable and Effective Solution to
Tabular Classification Problems
Si-Yang Liu 1 2 Han-Jia Ye 1 2
Abstract
TabPFN has emerged as a promising in-context
learning model for tabular data, capable of di-
rectly predicting the labels of test samples given
labeled training examples. It has demonstrated
competitive performance, particularly on small-
scale classification tasks. However, despite its
effectiveness, TabPFN still requires further re-
finement in several areas, including aligning with
downstream datasets, handling high-dimensional
features, and scaling to larger datasets. In this
paper, we revisit existing variants of TabPFN
and observe that most approaches focus either
on reducing bias or variance, often neglecting
the need to address the other side, while also
increasing inference overhead. To fill this gap,
we propose BETA (Bagging and Encoder-based
Fine-tuning for TabPFN Adaptation), a novel
and effective method designed to minimize both
bias and variance.
To reduce bias, we intro-
duce a lightweight encoder to better align down-
stream tasks with the pre-trained TabPFN. By in-
creasing the number of encoders in a lightweight
manner, BETA mitigate variance, thereby fur-
ther improving the performance of the pretrained
TabPFN. Additionally, bootstrapped sampling is
employed to further reduce the impact of data
perturbations on the model, all while maintaining
computational efficiency during inference. Our
approach enhances TabPFN’s ability to handle
high-dimensional data and scale to larger datasets.
Experimental results on over 200 benchmark clas-
sification datasets demonstrate that BETA either
outperforms or matches state-of-the-art methods.
1School of Artificial Intelligence, Nanjing University, China
2National Key Laboratory for Novel Software Technology,
Nanjing University, China.
Correspondence to: Han-Jia Ye
<yehj@lamda.nju.edu.cn>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267,...