A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities
Authors: Han-Jia Ye
Si-Yang Liu
Wei-Lun Chao
Venue: NeurIPS 2025 poster ()
Keywords: Tabular Data
Tabular Foundation model
Primary area: deep_learning

Abstract: Tabular datasets are inherently heterogeneous, presenting significant challenges for developing pre-trained foundation models. The recently introduced transformer-based Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented *in-context learning* performance across diverse downstream datasets, marking a pivotal advancement in tabular foundation models. In this paper, we take a closer look at TabPFN v2 to examine how it effectively handles heterogeneity and achieves high predictive accuracy, and to explore how its limitations in high-dimensional, many-category, and large-scale tasks can be mitigated. We find that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs, eliminating the need to explicitly learn dataset-specific attribute embeddings to address heterogeneity. We further show that TabPFN v2 can be transformed into a feature extractor, revealing its ability to construct a highly separable feature space for accurate predictions. Lastly, we demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-conquer strategy, enabling scalable inference without requiring re-training. By uncovering the mechanisms behind TabPFN v2's success and introducing strategies to extend its applicability, this study offers key insights into the design of future tabular foundation models.

A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities
Han-Jia Ye & Si-Yang Liu
School of Artificial Intelligence, Nanjing University National Key Laboratory for Novel Software Technology, Nanjing University {yehj, liusy)$llan @laddaammaa...mm nju. nju.edu.c. edu. cn
Wei-Lun Chao
The Ohio State University chao 20000s0..uu.. edu
Abstract
Tabular datasets are inherently heterogeneous, presenting significant challenges for developing pre-trained foundation models. The recently introduced transformer- based Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning performance across diverse downstream datasets, marking pivotal advancement in tabular foundation models. In this paper, we take a closer look TabPFN v2 to examine how it effectively handles heterogeneity and achieves high predictive accuracy, and to explore how its limitations in nigh-dimensional, many-category, and large-scale tasks can be mitigated. We find that TabPFN v2 infer attribute relationships even when provided with randomized attribute token inputs, eliminating the need to explicitly learn dataset-specifie attribute embeddings to address heterogeneity. We further show that TabPFN v2 can be transformed into feature extractor, revealing its ability construct highly separable feature space for accurate predictions. Lastly, we demonstrate that TabPFN v2's limitations can be addressed through test-time divide-anddconquer strategy, enabling scalable inference without requiring re- training. By uncovering the mechanisms behind TabPFN v2's success and introducing strategies extend its applicability, this study offers key insights into the design of future tabular foundation models.
Introduction
Tabular data is ubiquitous across a wide range of applications, including healthcare 32] finance 4
and scientific research [33 (2). In this format, each instance (e.g., patient's record) represented as a vector of attributes, and the goal of a machine learning model is to map these vectors to their corresponding labels [7]. Traditionally, tree-based models [56] 12 have dominated this domain, but
recent advances in deep tabular models are increasingly closing the performance gap [22 29 (6)..
However, unlike vision and language domains, where pre-trained foundation models have driven significant progress [40] 8I], tabular data is still desperately awaiting similar breakthrough [63] 77 (66). A primary challenge arises from the inherent heterogeneity of tabular datasets, which often vary in dimensionality and attribute meanings, making the development of effective and versatile foundation models difficult. Additionally, there is urgent need for such models, as many tabular datasets are small-scale-such as medical data with limited patient numbers. Training individual models from scratch for these datasets is highly sensitive to hyperparameter choices and often fails to generalize due to limited data 18 24 25
27 54
82
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recently, the Tabular Prior-Fittee Network v2 (TabPFN v2) 28 has emerged as a significant step forward. Built on transformer architectures [ and pre-trained on gigantic synthetic datasets [27 TabPFN v2 can be directly applied to diverse downstream tasks without additional tuning. Specifically, TabPFN v2 takes both a labeled training set and an unlabeled test instance as input, predicting the test label in an "in-context learning" manner. When evaluated across both classification and regression tasks, TabPFN v2 consistently outperforms prior tabular methods, achieving state-of-the-ar accuracy.
Motivated by the remarkable performance of TabPFN v2, we aim to take a step further to understand the mechanisms behind its succes specifically, how it effectively handles dataset heterogeneity and achieves high predictive accuracy. In addition, we investigate how to overcome its current limitations namely, its suggested data regime of no more than 10,000 samples, 500 dimensions, and 10 classes 28 ideally without requiring model re- re-training. We outline major insights as follows.
1. TabPFN v2 internalizes attribute token learning to handle data heterogeneity. Given an instance with d attributes, TabPFN v2 transforms it into a set of fixedddieensional tokens and uses a transformer architecture handle variability in d, following [64 22 774.. In sharp contrast to prior methods that rely on known attribute semantics (e.g., word vectors) or learn dataset-specific attribute tokens, TabPFN v2 instead employs randomized attribute rokens-resampled at each inference. This design "syntactically" allows TabPFN v2 to be directly applied to new downstream datasets with varying dimensionalities and attribute meanings without additional tuning, but raises a fundamental question: how does it still make accurate predictions? Our analysis shows that, regardless of the randomness, TabPFN v2 can consistently infer attribute relationships through in-context learning, essentially integrating attribute token learning into the inference itself. In short, TabPFN v2 unifies representation learning and prediction within a single forward pass.
2. TabPFN v2 can be repurposed as a feature extractor for downstream tasks. The exceptional predictive performance of TabPFN v2 suggests that it produces instance-level feature represen- tations that are highly discriminative. However, verifying this is non-trivial, as TabPFN v2's in-context learning mechanism assigns distinct roles to labeled training and unlabeled test in- stances, resulting in embeddings that are not directly comparable. To overcome this, we propose a folle-onefooldoutt strategy that enables the extraction of instance features more elosely aligned across training and test data. Our findings reveal that TabPFN v2 effectively maps tabular instances into nearly linearly separable embedding space. Remarkably, training a linear model on these features yields accuracy comparable to that of TabPFN v2's in-context Iearner, highlighting its potential as powerful feature encoder. This not only offers insights into TabPFN v2's inner workings but also opens the door to broader applications (e.g., visualization and error analysis).
3. Test-time divide-and
effectively mitigates TabPFN v2's limitations. As noted in 28), TabPFN v2 faces challenges when applied to high-dimensional, many-category, or large-scale datasets. Rather than resorting to model re-training, we show that these limitations can be effec- tively addressed through carefully designed post-hoc divide-and-conquer strategies, reminiscent of test-time scaling techniques developed for large language models [1). Empirical results show significant accuracy gains across these challenging data regimes, highlighting the potential of advanced post-hoc methods to further extend the capabilities of tabular foundation models.
Remark. This paper presents a timely and in-depth investigation into TabPFN v2, offering valuable insights for advancing tabular foundation models. While we do not propose new architecture or training scheme, our contribution lies in the novel analysis and principled extension of TabPFN v2. This reflects a growing trend in foundation model research, where understanding, evaluating, and adapting powerful models is increasingly seen as being as impactful as designing new ones.
2 Related Work
Tabular foundation models. Pre-trained models have revolutionized the vision and language domains 40 81], but their adoption in tabular data remains limited due the substantial heterogeneity across datasets. Variations in attribute spaces, dimensionalities, and label distributions present significant challenges for joint training and transferability. One solution leverage the semantic meanings of attributes, as demonstrated by methods that convert tabular instances into textual descriptions and
We note that TabPFN v2 (28), like many recent large language models (LLMs) and foundation models, does not release its training data or training recipe. Accordingly, our focus is on understanding the properties of the released pre- pre-trained model and exploring ways extend its applicability.

---

Randion sckens
Leaomable rofoens
16 logir
Scll-Atention
Gonchet
MMMIA
Sel- Altenbine
MLP
Cutll
Talil
5010
171-2
Frantve Tokenizur
Prodictine
MLPP PIR
230
Rirory Clossd Reation Task
Tobular Contect
Claseifization
Figure/ 1: Left: Illustraiion of TabPFN v2's mechanism for binary classification 28
{Ai,... Aa) denote d
attributes of the task. Training examples and test instance are combined into a tabular context and transformed into (N 1) x (d + 1) & tensor using combination of learnable and randomized tokens. Two types of self-attention are applied alternately across rows (inter-sample) and columns featuree The output token corresponding the (dummyy lahel of the test instance is processed through an MLP to generate 10-class logit. Right: Wilcoxon-Holm test at a significance level of 0.05 over 273 small- to medium- scale datasets. We omit the 27 datasets used to select TabPFN v2's checkpoint from the 300 datasets in 75
Alternatively, some approaches aim to improve transferabilityy by pre-computing attribute tokens based semantic embeddings 774 39] In practical domains such as healthcare or scientific measurement, the semantic meanings of attributes often inaccessible due to privacy constraints, annotation costs, lack of describability. To address this, [77 proposed representing each instance by its similarity profile to a fixed number of nearest neighbor examples, thereby mapping it into a consistent latent space with shared dimensional semantics. The TabPFN family [27 [28] leverages the in-context learning capabilities of transformers to directly predict labels by contextualizing test instances among training examples. This strategy
apply large language models for prediction 226 79 69 7
inspired subsequent pre- pre-trained tabular models such as 48 15 57]. While TabPFN v1 pads attribute vectors fixed dimension, TabPFN v2 introduces a specialized attribute tokenizer to handle heterogeneous input spaces. Meta-learning has also been explored to generate model weights Other pre-trained models rely on
tailored for downstream tabular tasks with limited data [34 6 50
lightweight
to adapt to variations in attribute and label spaces [44 80 62, 82).
Background
Learning with a single tabular dataset. A tabular dataset D = {(x,,, H contains N training examples, corresponding
the rows in table. Each instance æ; characterized by d features or attributes (i.e., columns in the table), where d typically varies across datasets. Its label yi belongs to [C] = (I...., C}} for classification task is numerical value for a regression task. We assume that all attributes of an instance are numerical (continuous). Categorical (discrete) attributes, if present, are transformed using ordinal or one-hot encoding beforehand. The goal of tabular machine learning is to learn mapping f from instances to their labels. Specifically, given an unseen instance x" E R sampled from the same distribution as D, the learned mapping f predicts its label as ÿ = f(x" DD) A smaller discrepancy between i and the true label y" indicates stronger generalizability of
TabPFN. The original TabPFN implements f for classification using transformer-like architec- ture (27). Both training and test instances are first zero-padded to a fixed dimension k' (e.g.. 100). Then, 3; and yi are linearly projected ã; g and ÿ; R respectively. TabPFN processes both a labeled training set and an unlabeled test instance jointly, predicting the test label in an in-context
learning manner. The task context is defined as C ( + ),..., (ā...,,@ +#N) (#*)) }} (N+1)xx, consisting of N + tokens, each of dimension k. These tokens are processed by multiple transformer layers, which accommodate variable-length inputs (i.e., variable N). The output token corresponding the test instance is passed through a multi-layer perceptron (MLP) to produce a 10-class logit.
TabPFN v2. The recently proposed variant (28 introduces several key modifications. First, each of the d attributes in D is embedded into a k-dimensional space, with random perturbations added to differentiate attributes. Together with the label embedding yi E R each training instance x; is represented by (d + 1) tokens with dimension k. For test instance x", where the label is unknown, a dummy label (e.g., e., the average label of the training set) is used to generate the label embedding ğ"
The full input to TabPFN v2 co-ooprisi the training set and the test instance-is is thus represented as a tensor of shape (N + 1) x 1) Two types of self-attention are applied in alternation: one over samples (among the N +1 instances) and the other over attributes (among the d+1 dimensions),
3

---

1.40%
16k
TabPFN v2 CatB
MNCA
R-MLP LR
Method
High-Ding 3.36
2.82
4.41
2.14
2.27
Phond
300
Large-Seale >10 classes
3.97
1.89
2.27
1.94
4.47
3.33
2.75
3.17
1,42
4.33
Cumulative Percentage Optimal Achievements (16)
Figure 2: Probability of Achieving the Maximum Accuracy or Minimum RMSE across 273 datasets. Values inside rectangles show the percentage of datasets on which method achieves the best result.
Table 1: Average rank (lower is better) of TabPFN v2
representative baselines on 18 high-dimensional, 18 large-scale, and 12 datasets with more than 10 classes. Full results with our extensions are in Figure 5
enabling in-context learning along both axes. Finally, the output token corresponding to the test instance's dummy label ý is extracted and mapped value logit for regression. An overview of this process is illustrated in Figure (left).
10-class logit for classification or single-
The weights in TabPFN v2 pre-trained on diverse synthetic datasets generated using structural
causal models (SCMs), with the checkpoint selected based on real-world datasets. For additional details, including feature pre-processing, acceleration, and post-hoc ensembling, please refer to (28)
Remark. the tabular domain, years of research into deep and foundation models have culminated in TabPFN v2 (28) breakthrough that, for the first time, enables deep models consistently outperform traditional methods without tine-tuning. However, due to venue constraints, many technical details were omitted from the main paper. For example, the use of randomized tokens was documented in the supplementary material and code. In light of this, we aim to systematically analyze TabPFN v2, as we believe such study is more impactful than proposing yet another architecture.
Comprehensive Evaluation of TabPFN v2
Before presenting our studies, first extend TabPFN v2's evaluation beyond the original set of datasets to over 300, covering a much broader range domains, attributes, scales, dimensionalities,
and tasks 23 49
75 60), aiming to more thoroughly assess its generalizability and limitations.
4.1 Setups
We first adopt the benchmark from 75 comprising 120 binary classification, 80 multi-class classifi- cation, and 100 regression tasks. It resol ves common issues such as mislabeled data and redundancies from overlapping dataset versions
enabling more reliable evaluations. Out of the 300 datasets, 27 belong to the validation set used for checkpoint selection in TabPFN v2 (28). To avoid evaluation bias, we exclude these datasets and report results on the remaining 273 datasets.
Following the protocol 21 22 each dataset is randomly split into training, validation, and test partitions in 64%/16%%/20% ratio. TabPFN v2 predicts test labels directly using in-context learning, without any additional parameter or hyperparameter tuning. Baseline tabular methods- both deep and prarioionall-eeffrn hyperparameter tuning using Optuna [I. with 100 trials on the training set and early stopping based on validation performance.
All methods evaluated using 15 random seeds, and report the average performance across seeds. For classification tasks, we report accuracy (higher is better), while regression tasks are evaluated using Root Mean Square Error (RMSE; lower is better). For tasks with more than 10 classes, we adopt the built-in Error-Correcting Output Codes (ECOC) strategy for TabPFN v2.
4.2 Empirical Results: Strengths of TabPEN v2
We compare TabPEN v2 against 26 representative tabular methods (see Appendix for full references). To assess statistical significance, we apply the Wilcoxon-HHolm test with significance level of 0.05 [4]. As shown in the critical difference diagram in Figure [ (right), TabPFN v2 consistently outperforms both tree-based methods, such as CatBoost [56]. and deep tabular models, including
RealMLP (29), ModernNCA (76). TabM [9] TabR (11. and FT-Transformer [22

---

To further assess performance, we report the Probability of Achieving the Maximum Accuracy or Minimum RMSE (PAMA) [3]. which measures the proportion of datasets on which a method achieves the best performance. As shown in Figure TabPFN v2 attains the highest score, delivering the top results on 26.02% of the datasets datasets- outerforming other methods such as ModernNCA (11.79%) and TabM (9.78%). These results underscore TabPFN v2's strong generalizability.
4.3 Empirical Results: Limitations of TabPFN v2
The above evaluation focuses on small- to medium-scal datasets, specifically those with fewer than 10,000 examples. However, as noted in (28). the computational complexity of transformers constrains TabPEN v2's ability to scale effectively datasets with larger sample sizes or higher dimensionality.
To verify this, we conduct additional evaluations on 18 high-dimensional datasets with d 2 2,000 [36] and 18 large-scale datasets where N d > 1,000,000. For high-dimensional datasets, the same protocol as before. For large-scale datasets, due to the prohibitive cost of hyperparameter tuning, default hyperparameterss are used for all methods. The average ranks of several representative methods are summarized in Table The full souults-along with our extensions are in Section 7
follow
As shown, TabPFN v2's performance degrades both large-scale and nigh-dimensional datasets. datasets,
On large-scale datasets, it ranks below both CatBoost and ReaLMLP; on even falls behind the simple Logistic Regression (LR) model. Beyond these two limitations, Table [ also reports results on the 12 datasets in Section 4.2 that contain more than 10 categories, where the ECOC strategy currently used by TabPFN v2 appears ineffective in achieving high accuracy. While increased computational complexity may contribute to this reduced effectiveness, we hypothesize that TabPFN v2 was pre-trained exclusively small- to medium-scale synthetic datasets with fewer than 10 categories, leading to a mismatch when applied to larger or more complex real-world data.
These results underscore the limitations of TabPEN v2, suggesting areas for further improvement.
5 How Does TabPFN v2 Effectively Handle Data Heterogeneity?
Section 4 demonstrates TabPFN v2's excellent generalizability to heterogeneous downstream tasks while also highlighting its current limitations. the rest of the paper, we first examine the mechanisms behind its strengths, followed by methods to overcome its limitations.
5.1 Diving into TabPEN v2's Mechanisms for Heterogeneous Input
Revisiting the problem. As noted in Sections and tabular datasets often differ in both the number of attributes (i.e., d) and the semantics of those attributes. Even when dimensionalities match, the dimensional semantics from different datasets are typically not directly comparable. A robust tabular foundation model must therefore handle such heterogeneity effectively, enabling it to learn from diverse pre-training datasets and transfer its capabilities to new downstream tasks.
Tokenization as a feasible solution. Among prior approaches, the most relevant to TabPFN v2 are baken-based methods (64 22 74]. The core idea convert d-dimensional instance E IRd into a set of d. fixed -imensooal tokens (cach of dimension k), with one token per attribute. This enables the use of transformer architectures, which naturally accommodate variability d across datasets.
To embed each attribute into a shared k-diimnnsionl space, prior work either uses pre-defined semantic embeddings 774] (e.g., word vectors of attribute names) or learns dataset-specifi embeddings [64] 22
Given cetrribute-speccfi tokens (ri...., ra] E dx each instance x; E IR can then be transformed into [x] T1, , x. ra] E gdxk where x! denotes the j-th element of æ;
By embedding all attributes into shared, fixed-dimensional feature space, this approach allows the transformer to learn transferable patterns and knowledge from heterogeneous datasets.
Difficulty in direct generalization. While appealing, the aforementioned methods face notable challenge when applied to downstream tasks: attribute names or semantics are not always acces- sible, as discussed in Section 2 Although it is possible to learn dataset-specific attribute tokens, doing so incurs additional computational cost and prohibits the reuse of previously learned tokens. Consequently, this limits the direct generalization of the foundation model to new tasks.
5

---

1912
Pi aaniinnnamn
(a)
(b) Layer
(c) Layer 6 (d) Layer 9
(c) Layer 12
(f) Legend
show PCA projections of the d attribute tokens from all N training instances various layers for the churn and bank datasets. Colors indicate different attributes (see legend on the right). The second and fourth rows display attention maps. Each matrix cell represents the average attention weight between attributes; the last element along each axis (e.g. the last column and row) corresponds to the label. The first plots in the second and fourth summarize the cosine similarity of attention maps
Figure 3: Attribute relationships inferred by TabPEN v2. The first and third
strrbute-wisse
random seeds. See text for details.
TabPFN v2's mechanisms. TabPFN v2 builds prior liken-based methods by representing each instance æ; as sequence of tokens. However, rather than assigning a deterministic token to each attribute, TabPFN v2 samples random tokens at inference time. Specifically, it learns a shared vector u E IR that lifts each element of æ; into a k-dimensiional space. To distinguish attributes, TabPFN v2 adds a random perturbation to each one For the j-th attribute (i.e., x.)). ). the representation becomes .r. u + r. where r; = Wpy. Here, Pj E IRA' is randomly generated vector, and W E & is learned projection matrix that conditions the perturbation. The full instance æ, is then represented as:
[ u + TI....,
o u + ra. 9.] E (x(++))
where the last token ys encodes the label information (see Figure T for illustration).
5.2 TabPEN v2 Internalizes Attribute Token Learning
TabPFN v2's randomized tokenization scheme eliminates the need to define attribute- or dataset- specific tokens across tasks, thereby syntactically enabling direct application of the pre-trained model. At first glance, this may appear to disregard the valuable semantic meaning of attributes. However, we show that through in-context learning, TabPFN v2 can consistently infer relationships among attributes within a titaaset-dsspii the randomness introduced during tokenization. Specifically, we analyze the behavior of attribute tokens from three perspectives, illustrated in Figure 3 using two representative downstream datasets: churn and bank.
First, we visualize the attribute token embeddings (i.e., the first d tokens in Equation (T) across all N training instances. The first and third rows of Figure present PCA projections of these tokens at the input stage and after transformer layers {3. 6, 9. 12}. with colors indicating different attributes. Initially, tokens from different attributes appear randomly scattered. However, as the input progresses through the transformer layers, these tokens become increasingly structured. For example,
This detail was identified from the supplementary material and code of (28

---

Amrratt yol dod0
Araaaryy-56 3630
Arcanary 25
Accuracydh we..yy......... 50.0
Actunard 9330
Accurscyd 9593
Aocoracycd..t6/5
Accurasy stral
Accuracyt
Acturaoyd
Accuracyd
Accuracy 0.589
Accuracy 0.745
Ancurasy 4019
Accurscyd. 7951
Actureyg 1825
Accurseyd 8123
(a) Raw Feature
(b) Vanilla
(c) Layer
(d) Layer 6
(c) Layer 9
(f) Layer 12
Figure 4: Visualization of the extracted instance features from four datasets: churn (first row, binary), bank (second row, binary), website phishing (third row, three classes), and KDD (fourth row, binary). Blue and indicate classes; darker crosses and lighter circles denote training and test samples. (a) shows the raw input features (e.g., 22), while (b) presents embeddings from the vanilla strategy. (c)-(f) display embeddings produced by method at different layers. Classification accuracy is reported by training linear logistic regression model on the training embeddings and evaluating on the set.
in the bank dataset (which predicts term deposit subscriptions), attributes such "job," "education," and "balance" eventually cluster into semantically coherent groups
Second, we examine attribute-wise attention patterns across layers, including attention to the label token. The second and fourth rows Figure 3 show heatmaps of attention weights averaged over heads and training instances. Each row in the heatmap represents the attention distribution from attribute to all others; the last element along each axis (c.g.. the last column and row) corresponds to the label. Darker shades indicate stronger attention. We observe consistent pattern across datasets: in early layers, attributes predominantly attend to the label token, likely to absorb task-specific uniformly distributed, facilitating information
signals. In intermediate layers, attention becomes exchange across attributes. deeper layers, attention concentrates on semantically relevant attributes, suggesting the model has inferred inter-attribute relationships useful for prediction.
Lastly, to assess robustness against random token initialization, we compute attribute- wise attention weights across 10 runs. The cosine similarities and variances of these attention patterns are summa- rized in the first plots the second and fourth rows in Figure The results confirm that attention patterns remain stable across except for the first layer.
Remark. The above results suggest that TabPFN v2 can reliably infer meaningful attribute relation- ships through in-context learning. Although input embeddings are randomized, they consistently differentiate attributes across instances so-fffonnctnllly akin one-hot encodings. Pre-training on di- verse tasks thus enables the model to extract predictive patterns (e.g., co-occurrence across attributes, value distributions, and relative magnitudes) directly from the statistical structure of each dataset, without relying on pre-defined attribute semanties. As result, the model effectively internalizes attribute token learning within the inference process. the Appendix for further discussion.
TabPFN v2 Can Be Transformed into an Effective Feature Encoder
In Section 5 we show that TabPFN v2's in-context learning process infers meaningful attribute relationships. Here, we examine whether TabPFN v2 also produces separable instance representations.
6.1 Naive Feature Extraction Fails
As shown in Figure | (left), TabPFN makes predictions based on the output token corresponding to the (dummy) label embedding ÿ" of the test instance. This output token can thus be interpreted the
7

---

instance embedding for the test example. A natural extension to obtain embeddings for the training instances is to extract the output tokens corresponding to the training label embeddings {
However, as shown in Figure (b), this naive approach leads to surprisingly discrepant feature distributions between training (darker cross) and test (lighter circle) examples. As result, linear classifier trained on these embeddings performs poorly on the test set. We attribute this discrepancy the distinct roles of labeled training data and unlabeled test data in TabPEN v2's in-context learning process. Specifically, the label embeddings for the training instances are derived from true labels, whereas those for the test instances rely on dummy labels. This mismatch renders the resulting output embeddings comparale between training and test instances.
6.2 Leave-one-fold-ont Feature Extraction
To address this challenge, we propose a seeve-ooe-fold---e strategy that enables the extraction of comparable embeddings for training and test data. In the TabPFN v2 framework, we treat examples with true labels as the support set S, and those with dummy labels as the query set Q. Under the standard configuration, corresponds to the labeled training set and Q to the unlabeled test instances. To extract comparable embeddings for the training examples, they must also be included in Q with dummy label embeddings. This, however, creates a dilemma: effective in-context learning relies on maximizing the size of S to ensure sufficient knowledge transfer to Q. Including training examples in @ thus competes with the need to keep S as large as possible.
To overcome this dilemma, we partition the training set into multiple folds (e.g., 10). In each round, one fold serves as 2-with dummy labels used for embedding extraction- while the remaining folds form S with true labels. This setup preserves sufficient label supervision in S while enabling the extraction of embeddings for training instances in Q. Results in (igure[(c)-(1) show that embeddings extracted by this strategy (with 10 folds) more faithfully capture dataset structure. We observe that TabPFN v2 simplifies the original tabular data distributions, transforming datasets into nearly linearly separable embedding sances-especiallyy after intermediate transformer layers.
We also experimented with a variant that uses the same context and query without partitioning, where the context contains all training samples and the query set includes both training and test points. This "non- nonn-pattitioned" strategy improves upon the vanilla feature extraction baseline but still underperforms compared to our proposed sanve-one-fol--oee that appear in the support set (either with dummy or ground truth labels) are treated inconsistently. preventing the network from fully distinguishing between training and test roles and thereby degrading feature consistency. Detailed results for this variant are provided in the Appendix.
method. We attribute this to role ambiguity: query points
6.3 Validation of Embedding Quality
Table 2: Average rank (lower is better) of TabPFN v2 and linear classifiers trained on the extracted embeddings across 29 classification datasets. Combined: embeddings from up to three layers (from the 12 available layers) are selected and concatenated, based on the validation set performance.
To validate the quality of the extracted em- beddings, we train a logistic regression on embeddings derived from the training set and evaluate it on test set embeddings. The average rank across 29 classification datasets from the tiny benchmark2 in (75
4 TabPFN v2 Vanilla Layer Layer 9 Layer 12 Combined
Rank 2.69
5.97
4.28
4.00
2.12
1.94
is reported in Table 2
Remarkably, training simple linear classifier on the extracted embeddings achieves performance comparable to that of TabPFN v2's in-context learner. Furthermore, concatenating embeddings from multiple layers (e.g., both output and intermediate representations) can sometimes lead to even better results. These findings underscore TabPFN v2's potential as a strong and versatile feature encoder, suggesting broader applicability in downstream tasks such tabular data analysis.
Improving TabPFN v2 via Test-Time Divide-and-Conquer
This section addresses the limitations discussed in Section 4.3
aiming to extend TabPFN v2's applicability beyond the boundaries. Specifically, we propose post-hoc divide-and-conquer strategies (CoT) prompting [70], which decompose challenging tasks into simpler subtasks that TabPFN v2 can effectively handle.
inspired by
8

---

7.1 High Dimension Datasets
High-dimensiooal datasets [36] present a unique challenge due to the quadratic complexity of TabPFN v2 with respect to the number of dimensions. To mitigate this, we propose subsampling the feature space into smaller subsets, processing each subset independently, and combining the predictions in ensemble (bagging) fashion, similar to random forests [8]
In detail, we iteratively sample m subsets, each containing d' < d randomly selected attributes. For each subset, we leverage TabPFN v2's ability to handle lower-dimensional data to obtain predictions. We denote this divide-and-conquer and then ensemble strategy as TabPEN v2", which aggregates outputs using averaging (for regression) majority voting (for classification). To address high-dimensions tasks, we introduce a baseline variant, TabPEN v2-PCA, which incorporates dimen- sionality reduction. Specifically, TabPFN v2-PCA reduces the feature dimension to 500 using PCA to satisfy the input constraints of TabPFN v2. This process is repeated multiple times with different PCA projections, and the resulting predictions are aggregated via bagging improve robustness.
H summarizes the results on 18 high-dimensional classification datasets. A variant that utilizes PCA to reduce the dimensionality, together with bagging, resolves the dimensionality issue to some extent. TabPFN v2' with d' = 500 and m = [d/d'] significantly increases the mean accuracy (to the highest), effectively extending TabPFN v2's scalability to datasets with d 2 2000.
7.2 Multi-Class Problems with More Than 10 Classes
To extend TabPFN v2 to tasks with more than 10 categories, we propose decimal encoding approach that decomposes multi-class problems into multiple 10-class subproblems, ensuring compatibility with TabPFN v2's constraints.
For a task with C' >> 10 classes, we encode each label y E [C] as a t-digit decimal representation. where 1 flogıo C1. For each digit position j E {1,-.. f), train separate TabPFN v2 model fy to predict the j-th digit. During inference, the predicted digits are reconstructed to obtain the final class label. This strategy is also developed in [48). and we denote it as TabPFN v2-DPT. As the decimal encoding inherently introduces artificial correlations among classes - classes that share the same digit at given position are grouped together, even if they are semantically unrelated. To mitigate this effect, our TabPFN v2" randomly permutes the class-to-digit mapping vC times, leading to different groupings in each run, and the prediction results are ensembles to improve robustness.
We consider the following variants of TabPFN v2 to address the 10-class limit in classification tasks:
TabPFN v2-ECOC: We use the implementation provided in the official TabPFN extensions reposi- tory, which applies Eiror-Correctio
Output Codes (ECOC).
TabPFN v2-DPT: We encode each class label as f-digit decimal string and train a separate TabPFN v2 to predict each digit. For instance, a 15-class problem is decomposed into two subproblems: one for the tens digit (classes {0, 13) and one for the ones digit (classes (0.....9)). The predicted digits are then decoded to recover the final class label.
We implement TabPFN v2 based on TabPFN v2-DPT for efficiency. Specifically, TabPFN v2 permutes the class-to-digit mapping vC times. For fair comparison, we also increase the number of ensembles in TabPFN v2-DPT to VC per digit to match the total number of predictions. As shown in Figure 5 (middle), this approach achieves the second-best mean accuracy on 12 datasets with more than 10 classes while preserving computational efficiency.
7.3 Large-Scale Datasets
For large-scale datasets, we randomly sample 10,000 training examples from the full training set as the support set and treat the remaining training examples and test instances the query set. We extract their embeddings to form a new tabular dataset, on which a logistic regression classifier is trained to make predictions on the test set embeddings. This process is repeated four times, and the final predictions are aggregated. We denote this version as TabPFN -SQ
We also investigate integrating TabPFN v2 with decision trees to handle large-scale tasks. We note that a similar strategy was mentioned in (2 to handle within-dataser heterogeneity for drastically different purpose. Specifically, we sample 32 subsets from the original training set, each containing

---

Cafk
Mathod
Mathod
614
Mean Accurascy
Moinn Accuracy
Average Rank
Mean accuracy on 18 high- seh-diimensioall datasets. "-PCA" another variant using PCA to reduce dimensions. Middle: Mean accuracy on 12 datasets with more than 10 classes. "-ECOC" denotes the multi-class ECOC strategy implemented by (28). Right: Average rank on 18 large-scale datasets. "-B" refers to the variant that randomly subsamples 10,000 training examples four times and aggregates their predictions. "-K" denotes the variant that selects representative subset of 10,000 training examples based on proximity prototypes obtained KMeans. variants improve TabPFN v2.
Figure 5: "97 indicates our extension. Left:
60% of the original data (sampled without replacement). For each subset, we first train a shallow decision tree by setting the minimum number of samples required split internal node 10,000. The decision tree partitions the training set into smaller, more manageable subsets. During inference, a test instance first passed through each of the shallow decision tree to leaf node and then predicted by the corresponding TabPFN v2 model. The predictions from all 32 models are aggregated. We denote this extension TabPFN -D*
We consider the following variants of TabPFN v2 to scale to larger datasets:
TabPFN v2-DT: A shallow decision tree is trained with minimum split size of 10,000. The tree partitions the dataset into smaller subsets, and separate TabPFN v2 model applied to each leaf node. At inference, a test instance is routed through the tree to a corresponding leaf, where it is predicted by the respective TabPFN model.
TabPFN v2-B: A bagging-bat based variant that randomly samples 10,000 training examples four times and aggregates their predictions.
TabPFN v2-K: Selects representative subset of 10,000 training examples based on proximity to KMeans-derived prototypes.
Our TabPFN v2 DDD is an extension of TabPFN v2-DT to forest-based ensemble. Specifically, we sample 32 subsets from the original training data, each containing 60% of the samples (without re- placement), and train separate TabPFN -DT model on each subset. During inference, predictions from all 32 models are aggregated-edd e.g. by majority voting or averaging-deppnddngg on the task type.
Figure 5 (right) shows the average rank results, including TabPFN v2* and TabPFN v2*-DF alongside variants using bagging and KMeans-based sampling. We observe that all variants improve upon the vanilla TabPFN v2 on large-scale datasets, with TabPFN -D= and TabPFN v2* achieving the most significant improvement.
8 Conclusion
We present timely investigation into TabPFN v2, a groundbreaking foundation model for tabular tasks. Our analysis uncovers the core mechanism behind TabPFN v2's strong performance across heterogeneous tabular datasets: it can infer attribute relationships po-the-fly-ente from randomly initialized token inputs-withe without relying on dre-ned semantics learning dataset-specific repre- sentations. We also demonstrate that TabPFN v2 can be repurposed as powerful feature encoder, enabling broader applications such as data visualization and diagnostic analysis. To address its limitations in more complex data regimes, we introduce post-hoc divide-anddconquee strategies that extend TabPFN v2's utility without requiring model re-training. Together, these contributions offer fresh insights into ad advancing the development and application of foundation models for tabular data.
10

---

Acknowledgment
This work is partially supported by National Key R&D Program of China (2024YFE0202800) Fundamental Research Funds for the Central Universities (2024300373), Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In KDD, pages 2623-2631, 2019.
[2] Xavier Amatriain, Alejandro Jaimes, Nuria Oliver, and Josep M Pujol. Data mining methods for recommender systems, In Recommender systems handbook, pages 39-71. Springer, 2010.
[3] Michael Arbel, David Salinas, and Frank Hutter. Equitabpfn: A target-permutation equivariant prior fitted networks. CoRR, abs/2502.06684, 2025.
[4] Sercan Ó. Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, pages 6679-6687, 2021.
[5] Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, and Sathiya Keerthi. Gradient boosting neural networks: Grownet. CoRR, abs/2002.07971, 2020.
[6] David Bonet, Daniel Mas Montserrat, Xavier Giró Nieto, and Alexander G. Ioannidis. Hyper- fast: Instant classification for tabular data. In AAAI, pages 11114-11123 2024.
[7] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Networks and Learning Systems, 35(6)7799--519) 2024.
[8] Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
[9] Jintai Chen, KuanLun Liao, Yanwen Fang, Danny Chen, and Jian Wu. Tabcaps: capsule neural network for tabular data classification with bow routing. In ICLR, 2023.
[10] Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z. Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In AAAI, pages 3930-3938, 2022.
[11] Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Ziyi Chen, Jian Wu, and Jimeng Sun. Can deep learning model be a sure bet for tabular prediction? In KDD, pages 288-296, 2024.
[12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD, pages 785-794 794,, 2016.
[13] Manuel Fernández Delgado, Eva Cernadas, Senén Barro, and Dinani Gomes Amorim. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research, 15(1):3133-31) 1181,,1112111 2014.
[14] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006.
[15] Felix den Breejen, Sangmin Bae, Stephen Cha, and Se- Youug Yun. Fine-tuned in-context learning transformers excellent tabular data classifiers. CoRR, abs/2405. 133966 2025.
[16] Benjamin Feuer, Chinmay Hegde, and Niv Cohen. Scaling tabpfn: Sketching and feature selection for tabular prior-data fitted networks. CoRR, abs/2311. 2023.
[17] Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. NeurIPS, pages 83430-83464, 2024.
[18] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In NIPS, 2015.
11

---

[19] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. In ICLR, 2025.
[20] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. In NewrlPS, pages
2022.
[21] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR, 2024.
[22] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurlPS, pages 18932-18943, 2021.
[23] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree- rree-bsed models still outperform deep learning on typical tabular data? In NeurIPS, pages 507-520, 2022.
[24] Isabelle Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera. Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sehag, et al. Analysis of the automl challenge series. Automated Machine Learning, 177:177-219 2019.
[25] Sungwon Han, Jinsung Yoon, Serean Ö. Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. In ICMI., pages 17454-17479, 2024.
[26] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tablim: few-shot classification of tabular data with large language models. In AISTATS, pages 5549-5581, 2023.
[27] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: transformer that solves small tabular classification problems in second. In ICLR, 2023.
(28) Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-326 2025.
[29] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. In NeurlPS, pages 26577-26658, 2024.
[30] Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. CoRR, abs/2501. .02945 2025.
[31] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. CoRR, abs/2012.06672 2020.
[32] Stephanie I. Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Esteban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, et Early prediction of circulatory failure in the intensive care unit using machine learning. Nature medicine, 26(3):364-373, 2020.
[33] Ovidiu Ivanciuc et al. Applications of support vector machines in chemistry. Reviews in computational chemistry, 23:291, 2007.
[34] Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from tasks with heterogeneous attribute spaces. In NeurlPS, pages 6053-6063, 2020.
[35] Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. Tangos: Regularizing tabular neural networks through gradient orthogonalization and specialization. In ICLR, 2023.
[36] Xiangjian Jiang, Andrei Margeloiu, Nikola Simidjievski, and Mateja Jamnik. Protogate: Prototype-based neural networks with global-to-local feature selection for tabular biomedical data. In ICML, pages 21844-21878, 2024.
[37] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In NewrIPS, pages
2021.
12

---

[38] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, pages 3146-3154, 2017.
[39] Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. CARTE: pretraining and transfer for tabular learning. In ICML, pages 228666, 2024.
[40] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yes Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, pages 3992-4003, 2023.
[41] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NIPS, pages 971-980, 2017.
[42] Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards quantifying the effect of datasets for benchmarking: look at tabular machine learning. In ICLR Workshop, 2024.
[43] Boris Kovalerchuk and Evgenii Vityaev. Data mining in finance: advances in relational and hybrid methods. Springer Science & Business Media, 2005.
[44] Lang Liu, Mahdi Milani Fard, and Sen Zhao. Distribution embedding networks for general- ization from a diverse set of classification tasks. Transactions on Machine Learning Research, 2022.
[45] Si- YanY Liu and Han-Jia Ye. Tabpfn unleashed: A scalable and effective solution to tabular classification problems. CoRR, abs/2502.02527, 2025.
[46] Si- Yang Liu and Han-Jia Ye. Tabpfn unleashed: A scalable and effective solution to tabular classification problems. In ICML, 2025.
[47] Junwei Ma, Apoorv Dankar, George Stein, Guangwei Yu, and Anthony L. Caterini. Tabpfgen tabular data generation with tabpfn. CoRR, 0es/22406052266 2024.
[48] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L.. Caterini. Tabdpt: Scaling tabular foundation models. CoRR, abs/2410.18111 2024.
[49] Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ra- makrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In NeurlPS, pages 76336-76369, 2023.
[50] Andreas C. Mueller, Carlo Curino, and Raghu Ramakrishnan. Mothernet: Fast training and inference via hyper-network transformers. In ICLR, 2025.
[51] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. CoRR, abs/2501.19393, 2025.
[52] Youssef Nader, Leon Sixt, and Tim Landgraf. DNNR: differential nearest neighbors regression. In ICMI, pages 16296-16317 2022.
[53] Thomas Nagler. Statistical foundations of prior-data fitted networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, ICML., pages 25660-25676, 2023.
[54] Soma Onishi, Kenta Oono, and Kohei Hayashi. Tabret: Pre-training transformer-based tabular models for unseen columns. CoRR, abs/2303.15747, 2023.
[55] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2020.
[56] Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Doro- gush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. NeurIPS, pages 6639-6649, 2018.
13

---

[57] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabicI: A tabular foundation model for in-context learning on large data. ICML, 2025.
[58] Cristóbal Romero and Sebastián Ventura. Educational data mining: a review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, (0(6))601-618 2010.
[59] Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. CoRR, abs/2207. .002008, 2022.
[60] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: benchmark of tabular machine learning in-the-wild. In ICLR, 2025.
[61] Sergio Ruiz- Villafranca, José Roldán Gómez, Juan Manuel Castelo Gómez, Javier Carrillo Mondéjar, and José Luis Martínez. A tabpfn-based intrusion detection system for the industrial internet of things. The Journal of Supercomputing. (0(14))20080-2011 2024.
[62] Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. In ICML, pages 31030-31056 2023.
[63] Gowthami Somepalli, Avi Schwarzschild, Micah Goldblum, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. In NewrlPS Workshop, 2022.
[64] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu. Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, pages 1661-1170 2019.
[65] Valentin Thomas, Junwei Ma, Rasa Hosseinzadeh, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony L. Caterini. Retrieval & fine-tuning for in-context tabular models. In NeurlPS, pages 108439- 2024.
[66] Boris van Breugel and Mihaela van der Schaar. Position: Why tabular foundation models should be a research priority. In ICML, pages 48976-48993, 2024.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[68] Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. DCN V2: improved deep cross network and practical lessons for web-scale learning to rank systems. In WWW, pages 1785-1797, 2021.
[69] Zifeng Wang, Chufan Gao, Cao Xiao, and Jimeng Sun. Anypredict: Foundation model for tabular prediction. CoRR, 1208,, 2023.
[70] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Cfain-of-thought prompting elicits reasoning in large language models. In NeurlPS, pages 24824-24837, 2022.
[71] Xumeng Wen, Han Zhang, Shun Zheng, Wei Xu, and Jiang Bian. From supervised to generative: A novel paradigm for tabular deep learning with large language models. In KDD, pages 2323-3333, 2024.
[72] Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongehao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, Daniel Cociorva, and Hakan Brunzell. Switchtab: Switched autoencoders are effective tabular learners. In AAAI, pages 15924-15933, 2024.
[73] Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. Mixture of in-context prompters for tabular pfns. CoRR, abs/2405.165 16156, 2024.
[74] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z. Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In ICLR, 2024.
[75] Han-Jia Ye, Si- Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan. closer look at deep learning on tabular data. CoRR, 0iw/2407.00956. 2024.
14

---

[76] Han-Jia Ye, Huai-Hong Yin, De-Chuan Zhan, and Wei-Lun Chao. Revisiting nearest neighbor for tabular data: A deep tabular baseline decades later. In ICLR, 2025.
[77] Han-Jia Ye, Qi-Le Zhou, Huai- Mung- Yin, De-Chuan Zhan, and Wei-Lun Chao. Rethinking pre-training in tabular data: A neighborhood embedding perspective. CoRR, abs/231 1.00055, 2025.
[78] Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dan dan Guo, and Yi Chang. Ptarl: Prototype-basee based tabular representation learning via space calibration. In ICLR, 2024.
[79] Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pre-training empowers models for tabular prediction. In EMNLP, pages 14836-14854, 2023.
[80] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. CoRR, abs/2307.10805 2023.
[81] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1-65, 2024.
[82] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. In ICMI,, pages 43181-43204 2023.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
In Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
15

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We have disclosed all key information necessary to reproduce the experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, provide access to the model. In general. releasing code and data is often
16

---

one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: We provide sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guidess/odeSuuuee wow.edddddbmissssoooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctaips.cc/uublcc/eee http//uubbc/gguudeeee httpi///guideee/oeeuuubbiiiii wwwbbbmmssiiooooooooioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: In Appendix.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics httpps:/nnerrppp http://hhh/////orppo.ooooooo//p/ cttte//ebb.i.iiiiiccceceeeellll
didn't 'ttntt make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We make sure that preserve anonymity.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We are confident that the creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/coddemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: this work, the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ptttps:/nnuurjin wwwparipsscccCooofeeee / /LLM) for what should or should be described.
21

---

In the appendix, we provide additional details, discussions, and experimental results to complement the main paper:
Appendix A Additional related work and discussions on methods closely related to our study (cf.
Section 2 of the main paper).
Appendix Detailed descriptions of the comparison methods used our evaluation (ef. Section of the main paper).
Appendix C Additional qualitative and quantitative results for feature extraction using TabPFN v2,
supplementing Section 6 of the main paper.
Analysis of the impact of feature engineering and ensemble strategies in TabPFN v2, as well as preta-feature-baseer analysis to identify the conditions under which TabPFN v2 performs well or poorly.
Appendix D
Appendix Complete results corresponding to the tables and figures referenced in the main paper.
Appendix Limitation and social impact of the paper.
A
Additional Related Work
Learning with Tabular Data. Tabular data is prevalent across diverse fields, including healtheare, finance, and education 43
32
58 []. Tree-based models, such as XGBoost [12, LightGBM (38),
and CatBoost (56). have long dominated this domain. However, recent advances deep neural networks (DNNs) have demonstrated strong potential for tabular data [7]. Popular architectures like
multi-layer perceptrons [22 37 and Transformers [31 have been adapted tabular tasks, alongside
custom architectures designed specifically for tabular data [1 68
Deep tabular methods can be broadly categorized into types. The first type directly processes
features (29 21 766, sometimes incorporating feature-specificc encoding strategies [20]. The second
type tokenizes features, transforming an example into set of tokens [64 31 (9). Comprehensive
benchmarks have been developed to evaluate these methods across diverse datasets [23 49 75 60).
highlighting the strengths and weaknesses of deep tabular models in various scenarios.
Variants of TabPFN. TabPEN's success stems from its pre-training on massive synthetic datasets, mall--ccale classification tasks [27]. Motivated by its capabilities, researchers have explored a variety of applications, including tabular data gener- ation [47. anomaly detection [1], and time series forecasting (30] (53) provided bias-variance analysis of TabPFN, offering insight into its generalization behavior. Another line of research fo- cuses on improving scalability by addressing TabPFN's sensitivity context size [16] 73). Further strategies to enhance downstream performance include context adaptation with nearest neighbor (65), partial fine-tuning [17 45). pre-training on real-world datasets [48), scalable ensemble (46), and more powerful and efficient pre-training on synthetic data [57]. Most of these variants remain restricted to classification tasks due to limitations in TabPFN vl.
enabling strong in-context learning performance
The recently introduced TabPFN v2 (28] extends TabPFN to support regression tasks and accom- modate larger context sizes. In this paper, conduct a comprehensive evaluation of TabPFN v2, analyze its strengths, and introduce methods to overcome its scalability and applicability challenges.
B Evaluation Details
Experimental Compute Resources. All experiments were conducted using 4 NVIDIA RTX 6000 Ada GPUs and 2 Intel(R) Xeon(R) Platinum 8352V CPUs.
Please refer 75 for details of the 300 small to medium datasets. For high- high-dimensional datasets, selected 18 datasets with more than 2000 features from the feikit- repository Detailed statistics of high- nigh-dimensiooal datasets and large-scale datasets are reported in Table 3 and Table 4
We follow 75 and use different colors to represent various categories of methods in the result figures, ensuring clarity and easy comparison. Figure (right) and Figure 2 of the main paper, we compare the following methods:
22

---

Table Dataset Information for ligh-Dimensional Data Experiments: A collection of 18 datasets with varying numbers of instances, features, and classes used in our high-dimensional experiments.
Dataset
#Instances
#Features
#Classes
Dataset
#Instances
#Features
#Classes
BASEHOCK
1993
4862
lung
203
3312
PCMAC
1943
3289
WarpPIE1OP
210
2420
RELATHE
1427
4322
oriraws100
100
10304
10
ALLAML
72
7129
Prostate GE
102
5966
2
CLL_SUB_IL
111
11340
SMK LANK_CAN_LTT 187
187
19993
colon
62
2000
warpARIOP
130
2400
10
GLI 85
85
22283
2
arcene
200
10000
2
GLIOMA
50
4434
gisette
7000
5000
2
leukemia
72
7070
2
TOX_171
171
5748
Table 4: Dataset Information for Large-scale Data Experiments.
Dataset
#Instances
#Features
#Classes
Dataset
#instances
#Features
#Classes
BNGícredit-a)
1,000,000
15
2
CDC Indicators
253,680
21
2
Higgs
1,000,000
Smoking signal
991,346
34,465
118
sf-police-incidents
2.215,023
8
Data Crowdfunding
671,025
11
4
Fashion-MNISS
70,000
784
10
covertype
581,012
54
7
jannis
83,733
54
poker-hand
1.05,0099
volkert
58.310
180
Airlines DepDelay
10,000,000
Wave Energy_ Farm
36,043
99
UlndoorLoc
21,048
520
blogfeedback
60,021
276
microsoft
1,200,192
136
yahoo
709,877
699
Classical Methods ): The classical methods include Dummy, Logistic Regression (LR), K-
Nearest Neighbors (KNN), Support Vector Machines (SVM), Naive Bayes, Linear Regression, and
DNNR 52. which serve basic baselines for classification and regression tasks.
Tree-based Methods Tree-based methods such as Random Forest [ XGBoost 12 Light-
GBM [3 and CatBoost (56) are known for their high performance on tabular data.
MLP variants MLP variants, including vanilla MLP, MLP-PIR. Self-Normalizing Neural Networks [41, Residual Network [22), and TabM [19 enhance the flexibility and generalization of traditional MLP architectures through advanced regularization and residual connections.
Special A Architectures ); Methods with specially designed architectures, such as DCNv2 (68],
DANets [TO], and TabCaps [9], focus improving feature interaction and abstraction to capture complex relationships tabular data.
Token-based Methods
Token- based methods like AutoInt (64], TabTransformer 31 FT-
Transformer [22], and ExcelFormer [IT] represent features as tokens, enabling models to capture higher-order interactions through attention mechanisms.
Regularization- -assed Methods ): Regularization-basse methods, including TANGOS [35], Tab [72). and PTaRL (78). aim to improve model generalization by incorporating regulariza-
tion techniques during training to enhance the robustness of predictions.
Tree-mimic Methods
Tree-mimic methods, such NODE (55), GrowNet [5], and TabNet (4.
combine the interpretability of decision trees with the power of deep learning, employing attention mechanisms to select important features.
Context- boased Methods ): context-based methods like TabR 21 and ModernNCA 76 lever- age contextual information from the training data to improve predictions by utilizing neighborhood- based and in-context learning strategies.
In addition to the aforementioned methods, for other experimental results, we will demonstrate the performance of TabPFN v2 and its variants, which are represented by emerald teal ensuring that their experimental effects clearly distinguished from the other methods.
The standard checkpoint released by TabPFN employs feature grouping size of which complicates the analysis of individual feature embeddings and inter-feature relationships. To facilitate such analysis, we use a modified checkpoint with group size= for the experiments in Figure the main paper, which is available at HuggingFace
Remark.
23

---

C Additional Feature Extraction Results
In this section, provide further results regression tasks and additional variants of our feature extraction strategy to validate the effectivenesss and robustness of the proposed leave-one- fotd-out method with TabPFN v2. We also compare this supervised approach to unsupervised embedding extraction methods, as well as a non-partitioned variant that uses the same context and query without data folding.
C.1 Additional results for leave-one-fold-out feature extraction strategy with TabPFN v2 on regression tasks
To further demonstrate the versatility of our approach, we evaluate TabPFN v2 on regression tasks by extracting embeddings using the leave-one-fold-our strategy and training simple linear regressors on top. As shown in Table 3 our embeddings consistently outperform both vanilla and raw features across different regressors, achieving the best average rank. This result indicates that TabPFN v2 can also serve an effective feature extractor for regression problems, further supporting its general applicability beyond classification tasks.
Table 5: Average rank comparison of embeddings regression tasks using different regressors.
Lower is better. LR denotes Linear Regression.
Extraction Strategy
Vanilla LR
Vanilla
Raw
Raw RidgeCV
Ours LR
Ours RidgeCV
Regressor
RidgeCV
LR
Avg. Rank
5.58
5.08
3.83
3.17
2.08
1.25
C.2
Comparison Between Supervised and Unsupervised Feature Extraction
Our cave-one-fold-our strategy explicitly incorporates label information and accounts for the distinct roles of training and test instances. To further understand the nature of the extracted embeddings, compare this supervised strategy to two unsupervised alternatives provided by the TabPFN extensions repository
Let X ggnd denote a dataset with n samples and d features:
Insupervised-Dummpn Each sample is paired with constant sgeudo-target y = O E R", forming a regression task. The embedding ED E R obtained by extracting the output tokens of the TabPFN regressor.
For each feature E .......dd, we treat x(i) = X. as pscudo-target and use the remaining features (-)) as input. Depending on the type of x()), TabPFN is applied in classification or regression mode to obtain E(J E Rxb, These embeddings are concatenated to a high-dimensional form:
Unsupervised-Permett Permute:
Ep concat(E())) (2)) ..... E(d) (dx(dd))
We compare these unsupervised methods with our supervised fove--on-fold-ou strategy in Table Overall, unsupervised approaches underperform compared to the supervised ones and fail to recover the classification ability of TabPFN v2. This performance gap arises because label information is introduced only post hoc via linear classifier rather than during embedding extraction. Among the unsupervised methods, the permutation-ban approach performs better, likely due to its ability to encode attribute-specife structure.
Figure 6 presents a visual comparison of embeddings produced by the three methods using the same color and marker scheme as in Figure 4 Since unsupervised methods lack label supervision during embedding generation, their embeddings tend scatter broadly without forming well-separated clusters by class. These results further highlight the contrasting goals of supervised and unsupervised strategies class separation versus feature distribution coverage, respectively.
C.3 Results for the Non-Partitioned Feature Extraction Variant
Using the same context and query without partitioning, we experimented with a variant where the context contains all training data and the query set includes both training and test points. This

---

Table Average rank (lower is better) TabPFN v2 and linear classifier trained on the extracted embeddings across 29 classification datasets. In addition to the supervised feature extraction strategy considered in the main paper (including the vanilla one, our cave-one-fold-on and the version based on the combined features), compare with two unsupervised embedding extraction approaches by appending a column of dummy labels with zero values and permuting each column as labels, respectively.
TabPFN v2
Vanilla
Dummy
Permute
Ours
Combined
Rank
2.66
5.72
4.90
3.69
2.16
1.88
Accuracy: 8600
AccuracyrO. 5650
Accuracy 0.8660
Accuracy:O 8740
Accuracy-0,9500
Accuracy: 0 8876
Accuracy: 6105
Accuracy 8843
Accuracy: 9012
Accuracy Accuracy.0.30pp 0.9085
Accuracy:0.5am
Accuracyo.
Accuracy
Accuracy:
Accuracy 0.8123
Ac Accuracy
A 3690
Accuracy
Accuracy:O 9004
Accuracy 0.9 9188
(a) Raw Feature
(b) Vanilla
(c) Dummy
(d) Permute
(c) Ours
Figure 6: Comparison between unsupervised and supervised (ours) feature extraction. Visualization of extracted embeddings for four datasets: churn (first row, two classes), bank (second row, two classes), KDD (third row, two classes), and website_phh website_phishing (fourth row, three classes). We use crosses to denote training examples and circles denote test examples. (a) shows the raw features, while (b) presents the embeddings extracted using the vanilla strategy. (c) and (d) refer to unsupervised embedding extraction approaches by appending column of dummy labels with zero values and permuting each column as labels, respectively.
(e) depicts the embeddings obtained using our proposed methods. The accuracy value is calculated by training linear model (logistic regression) over the extracted embeddings on the training set and predicting on the test set.
non-partitioned strategy improves upon the vanilla feature extraction baseline but still underperforms compared to our proposed leave-one-fold-out method. We attribute this to role ambiguity: query points that also appear the support set (with dummy or ground-truth labels) are treated inconsistently, preventing the network from fully distinguishing between training and test roles and thereby degrading feature consistency.
D Influence of Key Modules and Meta-Feature Analysis
investigate the influence two key components in TabPFN v2, i.e., the feature engineering that pre-processes the raw features of a given tabular dataset and the post-hoc ensembleing. In addition, we analyze the conditions under which TabPFN performs well or poorly through reta--eeture-hiee classification analysis.
25

---

Table Performance ranking (lower better) of different feature extraction strategies on classifica- tion tasks. The non-partitioned variant uses the same context and query without data folding.
Method
Avg. Rank
Leave-one-fold-out (Ours)
2.10
TabPFN v2
2.72
Non-partitioned (12 layers)
3.07
Non-partitioned layers)
4.29
Non-partitionel (6 layers)
4.43
Vanilla (12 layers)
5.86
Vanilla (9 layers)
6.62
Vanilla (6 layers)
6.90
W/ Feature Engineering
0.8
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
W/O Feature Engineering
Figure 7: Scatter plot comparing the normalized Accuracy/R² scores. The x-axis represents the nor- malized Accuracy/R² scores without Feature Engineering, while the y-axis represents the normalized Accuracy/R² scores with Feature Engineering. The red dashed line (y x)) serves reference, indicating equal performance.
Feature engineering. TabPFN v2 pre-processes the features of given tabular dataset with various strategies, such as quantile, category shuffling, SVD, and power transform. Specifically, we examine the effects of adding fingerprint features (add_f bnngerprint_fettuue (polynomial,
and polynomial features Sallyyomiallffftureeeeeee features) the raw tabular data. The results indicate that TabPFN performs well even without the use of these engineered features, suggesting that, for the benchmark datasets of 775, these specific feature engineering techniques do not provide significant improvement. This finding highlights the robustness of TabPFN v2 and its ability to handle raw features effectively, without the need for extensive pre-processing or feature construction. We show the influence of this step in Figure 7
Model ensemble. Post hoc ensembling (PHE) involves applying TabPFN to the datasets multiple times with different perturbations and aggregating the predictions of these base models at different temperatures. We show the change of performance numbers (i.e., the number of base models) in Figure & On the benchmark of (75), observe that, overall, ensemble methods improve performance, with larger ensemble sizes yielding better results. However, we also note that even without ensembling. TabPFN v2 performs exceptionally well, and the relative performance gain from ensembling is limited. This suggests that while ensembling can provide further improvements, the base TabPFN v2 model is already highly effective its own. The equivariant property described in 3 provides insight into this phenomenon. Since TabPFN v2 introduces random tokens to handle heterogeneous features, the model becomes less sensitive to the arbitrary ordering of features, effectively enforcing equivariance this aspect. As a result, the benefits of ensembling through feature order permutations are less pronounced compared to TabPFN vl.
TabPEN v2 w.r.t. the number of ensemble
better understand the conditions under which TabPFN v2 performs well or poorly, we conducted meta-learning-basee classification analysis using 300 datasets. Specifically, we used the average rank of TabPFN v2 across datasets as performance indicator. The threshold for classification set at the mean rank, 6.3 Datasets where
Peta-Feature Analysis of TabPEN v2 Performance.
26

---

20130
Relative Improvement
2.0.00
230200
230000
-3,002
-200000
Ensemble Size
Figure 8: Box plot of relative performance improvements of TabPFN v2 with post hoc ensembling (PHE) across different ensemble sizes (2, 4, 8, and 16 base models). The relative improvement is calculated as the performance gain over the non non-ensemble model, where higher values indicate stronger performance. The box plots show the median, interquartile range (IQR), and outliers for each ensemble size.
TabPFN v2 achieved rank lower than equal to 6.31 were labeled "Good", while those with a higher rank were labeled as "Bad". We extracted meta-features from each dataset and used them train decision tree classifier, aiming to distinguish between the "Good" and "Bad" categories. All the meta-features utilized in this task are detailed in Table g accompanied by their respective explanations. A visual depiction of a simple depth-3 decision tree is shown in Figure | and the tree reveals key factors that influence the effectiveness of TabPFN v2. The decision tree visualizes how to predict whether TabPFN v2 performs well ("Good")) or poorly ("Bad"") on a given dataset, based on dataset meta-features. The root node splits on the number of instances (nr (nr_ing) inst), indicating that TabPEN v2 tends to perform better on datasets with fewer than 24,350 samples. For these smaller datasets, the left subtree further splits on the mean joint entropy (joint_ent mean), where higher values (greater than 3. 3.028) are associated with improved performance. For datasets with lower mean joint entropy (< 3.028). TabPFN v2 also tends to perform well when the number of rows is relatively small (I 4,862). contrast, the right subtree, which represents larger datasets, reveals that a low standard deviation of the interquartile range (iq_range std) across features (< 0.657) is linked to poorer model performance.
E Detailed Results
We list the detailed results of TabPFN v2 and our extensions on various benchmarks.
We present the main results of TabPFN v2 on 300 datasets in Table The table includes accuracy for classification tasks and RMSE (Root Mean Squared Error) for regression tasks, along with the corresponding mean and standard deviation for each dataset. Notably, from these results in Table g as they were used by TabPFN v2 to select the best checkpoint. These excluded datasets, which are not shown in Figure (right) and Figure 2 of the main paper, include: (1) ada prior, allbp, baseball, delta ailerons. eye movements, eye movements bin, GAMETES Epistasis 2-Way _0atts2 O. EDM-i hill-valley, Japanese Vowels, Vowels, jungle_chher chess 2pes 2pes_raw www_endgame___ _ompletee led24, Longtudinal-surver page-blocks. ringnorm, rl, thyroid-ann, waveform-5000,
excluded 27 datasets
raw
(2) debutanizer, delta elevators, mauna- loaa--loaaaameteiicccoo puma32H, stock_ fardamento02, trea- sury, pouther_izmic wind.
In Table 10 we showcase the performance of various models on 18 high-dimensional datasets. The results display the mean accuracy of different models, including ModernNCA (MNCA), MLP, KNN, RealMLP, XGBoost (XGB), Random Forest (RForest), Logistic Regression (LogReg), and TabPFN v2 (PFN-v2), along with variants like TabPFN v2-pca and TabPFN v2*, This highlights the ability of these models to handle high-dimensional data with many features.
We demonstrate the performance of various models on 12 multi-class classification tasks with more than 10 classes Table [T] The table provides the mean accuracy models like KNN, TabPFN-v2* XGBoost (XGB). CatBoost (CatB), Random Forest (RForest). ModernNCA (MNCA), MIP, Logistic Regression (eggReg), and RealMI.P, showcasing how they perform on multi-class tasks with larger number of classes. Additionally, we compare PEN-v2-ECOCK
27

---

gini 24350.5
samples 103]
False
joint mean <= 3.028
gravity <= 1.2 1.277
gini = 0.405
gini 0.361
samples - 262
samples 38
value - [186. 74]
value - 19, 291
class Good
class = Bad
rarrgeeeee gini class
samples value [24,
gini
gini samples value [164,
samples value [0,
gini = 0:47
gini 0.142
gini = 0.338 amples - 209 value [164, 45]
gini = 0.0
gini = 0.43
gini 0.0 sa maples 4 value [4. 0 class Good
samples 37 value = [23, 14] class Good
samples 13 value [1, 12]
samples -
samples - 16
value = [0, 3]
value = [5, 11]
class Bad
class = Good
dlass Bad
class = Bad
Figure 9: Decision tree for predicting TabPEN v2 performance (Good vs. Bad) based on dataset characteristics, constructed from experiments 300 datasets. The splits meta-features such as number of instances far_inst), joint entropy (joint_ern point__nnt.. ent.mean), number of numerical features (tr_,,m), distance between minority and majority classes' center of mass ggravity) and interquartile range (iq_range) statistics. Each split is chosen to maximize information gain. The leaf nodes indicate the predicted performance class, the Gini impurity, and class distribution. This tree provides insights into the types of datasets where TabPFN v2 is expected to perform well.
multi-class classification solution provided by (28). This method extends TabPFN-v2 by leveraging Iiror-Correcting Output Codes (ECOC) to enhance multi-class classification performance
we compare the performance of various models on 18 large-scale datasets. The results show the mean accuracy or RMSE for MLP, Logistic/Linearr Regression (LR), KNN, XGBoost (XGB), Random Forest (RForest), CatBoost (CatB), ModernNCA (MNCA), ReaLMLP, and different versions of TabPFN (PFNv2, PFNv2 with K-means, PFNv2 with Bagging, and PFNv2*). This illustrates the models' performance on large-scale datasets.
In Table 12
In Table [3] we show the performance of TabPEN v2 and the extracted feature embeddings across classification datasets. The table includes average classification accuracy for each dataset when using feature embeddings from different transformer layers (Layer 6, Layer 9, Layer 12), well combined approach where embeddings from multiple layers are concatenated. The "selected layers" column indicates the layers chosen based on validation set performance, offering insights into how different layers contribute to overall model performance. In addition evaluating the performance of TabPFN v2 and the extracted feature embeddings, we also compared the results with embeddings obtained using the vanilla strategy (Vanilla).
Table 9: Main results of TabPFN v2 on 300 datasets, including accuracy (for classification tasks) and RMSE (for regression tasks), along with the corresponding mean and standard deviation for each dataset. Among the 300 datasets, 200 classification datasets, and 100 regression datasets. The results demonstrate the effectiveness of TabPFN v2 across both classification and regression tasks.
Dataset
Mean Sid
Dalaset
Mean Std
ASP- WWw.PPTASSCU-Caastiiico
43.50 1: 1,27
Amazon, Amazon__employe wwwplyeeeppeeeeeeeeeeee access
94,22 : U.D4
BLE_RSSI__A Roslllocaliootion
73.37 0.15
BNGrbreast-w)
98.56 +
htwpyy/www..ot.ii.oooooooooooooe class/many class classcclassifier.py
28

---

BNGleme)
57.89 0.17
BNG(li-tae-to))
79,42 + 0.26
Bank Customer Churn Dataset
87.53 1.152
Baskethall_
70.65 1 0.47
Califomia- classing-Cassiiicatiinn
9.17
Cardiovascular- Disease-daatsse
72.92 0.13
Click, prediction small
83.29 1: 0.03
Comtmmnant 10.0GHI2
94.42 1 0.36
Contaminant- H0.
95.17
Contaminant 1100Hz
Contaminant-9
93.01 0.47
Contaminant Sentiinantt-9.CCG 9.5HH2
93.21 + 0.50
Credi_e
69.98 2: 0.15
Customer Personality_ Analysis
90.03 + 0.21
Diabetic Retinopatyy Debrecen
72.81 1,07
E-ComnereShipppingaaa
67.54 1 0.21
Employee
84.80 0.30 0.30
FICO BEOO-HLLO---leee
OC Oc-clemmed
75.35 1 0.21
FOREX DOREX_mndedd-e2 mudead-dny-HHtt High
74.51
FOREX, audcad-hour- High
71.01 0.20
FOREX audchf day- High
76,56 0.45
0.45
FOREX audipy modipy--dy--aa High
78.00 1: 0.28
FOREX audipy-houur-eeeee hour--Highhh
7
FOREX audsgd-hour- High
+
FOREX_sudusd-one www.uuddd-uuu---ee High
69.57 0.48
FOREX_ cadjpy--dyy-H-eeee High
71.68 + 0.53
FOREX FOREX_caljpy-hne Seii__aallyy--h--Hgiee
70.55 0.40
Firm Teacher-Directioon
84.42 + 0.47
Fitness Club_e
79.67 0.24
GAMETES Epistasis
68.75 0.82
GAMETES Heterogeneity
65.90 1.84
Gender Gap in_Spanish WP
60.58 + 0.24
71.36
HR ARaAnalytics
80.02
Heart Seant-Dieass-DDaae Datass-
91.23 11.23:00 0.54
INNHotelGGmup
87.98 + 0.23
Indian_ pincs
9641 0.23
Insurance
75.75 +
Intersectionl-Bee Baaecooonn--i---sssssseeee
94.73 0.13
b-this-a-good-cusinne
88.41 + 0.00
Japanese Vowelss
99.68 0.08
KDD
80.14 1 0.46
KDDDCup09 Supeellngg
81.06 0.26
Long
99.88 1 0.00
MIC
90.20 0.56
MagicTelescope
88.13 + 0.21
Marketing Campaign
0.41
Mobile Pttco_elraseceeeeeeeeeeeeeeeeee Classification
Nutrition Health Health_Survey
83.45 0.22
Performance Prediction
73.23 + 0.61
Phishing Websites
96.74 0.13
PisChart3
87.31 +
Pina Indians Diabetes, Database
75.93 1: 0.66
PizzaCutter3
88.20 + 0.45
Pumpkin_Seer Seeds
87.93 0.21
QSAR biodegradation
88.50 1 0.50
Rain in_Australia
83.88 0.11
SDSSI7
97.33 + 0.06
Shipping
68.73 0.40
Telozom Churn_ Dataset
95.18 + 0.50
WJL Charaacres
VulNone
Water, Quality and Potabilit
65.49 1: 0.50
Waterstress
71.37 1 0.96
Wilt
99.28 0.06
abalome
63.58 + 0.38
acceleromettr
73.96 1.32
ada
85,40 1 0.25
ada agnostic
83.99 1: 0.34
ada_prion
85.32 1 0.19
adult
85,93 0.12
airlincs_2000
62.28 0.48
allbp
97.85 - 0.19
allrep
98.65 + 0.12
analcatdata authorship
artificialchharatter
antoUniv au4 2500
69.81 1.00
autoUniv au7 100
41.18 1: 1.58
bank
90.86 0.19
banknote autheeliccaiinn
55.64 + 0.18
baseball
93,81 0.40
evaluaaiion
98.29 1 0.22
churn
96.33 :: 0.28
cmc compass
59.59 + 0.49
company cankkruptey_prodiittie connect-4
97.33 0.07
71.05
76.78 1 0.35
contraceptive weteeeeeeeteeeeeeeeeeeeee choice credit-g g dabetes 130-us hhoqpitlls
62.10 + 0.37
customer customer-sstisfaetinn satisfaction in airline
94.79 + 0.11
63.08 + 0.07
default oeeerriii___aaaaaeeeeee ctients dis
82.63 0.08
delia sileross dna
95.47 1 0.09
99.07 0.14
97.25 1 0.20
drug_ consumption
40.32 4: 0.00
dry wry_bean_ddar dataset clectricity
92.76 + 0.10
oog-eyo-state
98.34 0.12
86.37
estimation.e jf__obsity__evee Jevels
98.66 1: 0.24
eucalyptus
72.88 1 1.17
eye movementts
eye mo._moeemet,,, gas-drift
firt-order-hhoorrmmpoiii
61.12 0.70
99.47 + 0.04
playp_yy_ddlssttttteep extended
92.60 0.44
helena
33.32 1 0.21
heloe
72.75 0.20
hill-valley htru
98.33 1 0.52
house_16k IGH
88.55 0.18
97.95 + 0.06
bbn-emppoye--perfoomee
velicie,
so_yciiceeccuuou coupon
internl Jirewall
92.85 0.30
internet_ Lusage
54.34 1: 2.64
jasmine
jml
1
jungle_chess_ wwwssshhpssssseeeeee__ 2et.222pp____________________e_e naw_endgame
85.97 + 1.82
kel
86.65 + 0.34
kdd ipums, Ia_ 97_997.mm 97-small
83.50 0.12
kr-vs-k
78.46 1
kr-vs-kp
99.64 0.15
kropt
77.96 + 0.63
an--schoo-addmisioo--bnny
100.0 0.00
lod24
73.29 + 0.62
led7
madeline
90.72 1: 0.48
mammography
98.71 1 0.05
maternal health_ risk
mfeut-factors
+
mffattfourrier
89.85 0.86
mnfet-karbunen
96.42 + 0.24
morphllnopplllliiaal
76.63 1: 0.50
mffeut pixel
96.10 1 0.32
mffat-zernkke
84.10 0.87
protepprtinn cecrresccpsssioon
100.0
micronggregation2
62.80 1: 0.14
mobile Bebele__66_.._ oversampling
98.11 + 0.08
movilla4
naticusdrvid--am pprrmssisssssssssssoos
national-longe longitudinal survey-uveeeeee hinary
100.0 0.00
okcupid stem
74.47 1: 0.12
ooe-hundred-plantse plant----tttnngnnnn
88.56 0.74
one-bundred- saandredplannt-shaqe
79,52
noo-hundree--laant--ee
90.94 0:944000 0.75
online shoppers
90.65 + 0.10
opidigits
98.59 0.12
ozone-level-Bit
94,92 1 0.25
ozone_level
0.11
pagc-blocks
pel
93.51 0.46
pe3
88.78 1 0.27
pendigits
philippine
$4.20 1.24
phoneme
88.47 4 0.35
pol
0.09
prodict_ students, students_dropour
78.11 +
rice.
cwwmoo.mmaeoo and osmancik
92.74 0.23
ringnorm
98.00 + 0.13
29

---

86.04 +: 0.44
salimage
92.30 1: 0.29
segment
93.91 0.19
seismicehurmps
93.40 1: 0.08
semeion
92.41 0.94
shill-bidding
90.31 0.18
shrutime
86.97 1:97::000 0.11
shuttle
99.86 1: 0.04
spambase
94.85
splice
sports articles
84,93 0,40
statlog
72.13 + 0.97
stoeLplates faults
84.68 0.55
svmguidee3
85.54 + 0,54
sylvine
97.30 0.27
laiwanese conkruppey_presdittonn
97.20 1: 0.07
telco-custone seao-customer-chmm
80.29 1: 0.28
texture
100.0 1 0.00
W thyroid
99,48 0.06
thyroid-ann
99.
thyroid-de dis
68.75 1: 0.34
turiye -evaluttenantiionn
51.74 1: 0.18
twonorn
97.94
vehicle
84.31 +
walking-activityy
61,22 0.22
wall-robat-nviiaatoon
99.44 + 0.10
water water_quality quality
90.12 :: 0.12
waveform-
86.29 + 0,26
waveform_yl
86.59 0.25
website phishing
90.48 1 0.48
wine
75.12 : 0.72
wine-quality-red
58.35 + 0.76
wine-qualitywwhite
64.15 0.69
yeast
60.18 0.65
110-Camera--Datise
2dplanes
RSSI Estimation
0,00068 1: 0.00
RSSI Estimation!
0.00992 1: 0.00
Ahalome reg
2.08 0.00
Ailerons
0.00015 + 0.00
Fiat
716.20 + 4.05
BNGicchoMonths)
11.11 0.03
BNG(lowbwt)
455.27 + 0.78
BNG(mv)
4,63 + 0.01
BNG(stuck)
correction
Bias correction r.2
0.52 4: 0.01
Bruzilian monses_reppodueee 1988
0.01 +: 0.00
PPMP-2015-regresiig
478.02 5.40
364 = 0.24
CookbookReviews
1.52 1 0.02
Data Science Science_Salaniie Salaries Facebook Comment, Volume
60237.28 102.97
Diamonds
533. - 6.30
23.16 0.20
Food Delivery_ Time
+
Gcodreads-Comptter Job Prolitabiiiy Laptop Prices, Dataset
IBEE8021 Iaa-GATS bike sharing demand
0.02 + 0.00
13.14 + 0.02
Wave Energy Perth 100
15507, 90 + 104.31
Wave Fnergy Fnergy_Sydnrcy 100
14737.67 150.43
Encrgy_Sydnncy... MumiHoussnn20066
4567.97 64.02
MIP-2016-roressionn
20966.10 454,90
83101.09 507.30
Mobile Phone Market
714,87 + 11.15
Moncyball
19.42 + 0.08
NASA NASA_PHM200S PHM2008
40.24 + 0.06
NHANES_ _ag.____pp prediction
OuliseNewaPPpularity
8606.54 =
Parkinson Sound Record
14.58 1: 0.09
Parkinsons, Telennomioorigg
0.60 +: 0.04
Physicochemical.
HANHHHADD-- HANND-rumimee
Shop_Customer Customer Data
28.56 0.01
Superconductivty
10.17 + 0.07
Wine Quality_rod
+
Wine. Wine_Quality Quality_ulllaa _hite
0.68 0.00
sirfoil Sei__slf__oois
1.16 +: 0.02
analcatdata supreme
0.09 + 0.00
archive2
342.64 3:20
archive_r$t Portuguese
2,86 + 0.02
suction_verrflicttonn verification
suction
avocado
bank32nh
0.08 1 0.00
bank8FM
0.03 1 0.00
chscase
colleges
0.14 + 0.00
combined cycle__eeeeee sycle_powee power_plant
3.22 - 0.05
communitioss_mo aaciiiss__nnd_tiieeeeeeee crime
0.13 +
concrete wwwproooioeeeeeeeeeeeoooo
4.63 0.07
epe_act
2.55 1 0.03
sma_s
3.06 1 0.02
dataset_sale sales
4.04 + 0.02
debutanizer
0.04 + 0.00
della_elevatoos
0.00
lifa
0.78 0.00
fried
1.01 1: 0.00
garnents_ worker_ productprddyttiiit
gas turbine _em.ssionnnnnoo
0.00
healtheare_ healtbeerr_insuuannee_ lte__jnssso__pceeeeee expensse
4716.87 + 36.52
bouse_ 16H_reg
2963 1,75 + 251.56
house SL
28617.41 202.41
house_prices, Brccc__pcee___aeee nomminl
30676.02 + 2455.48
house sales_aeee rellucd
132655,03 1847.33
houses
42559.96 98 928.78
housing_ptss pricee________eeeeee_ ptlppppccoeeeeeeeee prediction
1009361.62 + 8758.05
kinkaam my pole
0.08 + 0.00
loa atmospheric co2
pol_reg
3.84 + 0.10
3.21 + 0.14
puma32H
+
pumaSNH
3.24 0.00
quar_ aaar_aauatic___ toxicity
1.0 + 0.01
lish_toxiiiiy
0.86 + 0.01
satellite iinage
0.65 1
sensory
0.77 0.01
socmoh
19.53 : 0.64
0.09 0.00
steel industttycccoooom energy
0.37 1 0.03
stock
0.65 + 0.01
funlamento02
0.08
sulfur
topo_2_
0.03 + 0.00
treasury
0.23 0.00
us crime
0.14 -
volume
$2.09 + 0.34
weather_izmn
1.09 + 0.01
wind
2,83 1 0.00
winetguality
0.72 1 0.00
ypop_4. 4_1
0.03 1 0.00
F
Limitations
While this paper does not introduce new model architecture or training paradigm, it offers timely and principled analysis of TabPFN v2, a powerful tabular foundation model. Our contributions in empirically evaluating its strengths, identifying its limitations, and proposing practical extensions that enhance its particllllll to large-scale, high-dimensional, and multi-class settings. A key limitation of our work is that the proposed extensions primarily post-hoc and do not fully
30

---

Table 8: Meta-features used in the meta-feature analysis of TabPFN performance.
Meta-Feature
Explanation
attr _or_ class_cone cone
The concentration coef. of each pair of distinct attributes.
The concentration coefficient between each attribute and class.
class ent
The target attribute Shannon's entropy.
inst to attr
ratio between number of instances and attributes.
mean
The mean value of each attribute.
sd
The standard deviation of cach attribute.
The variance of cach attribute.
range
The range (max min) of each attribute.
q range
interquartile range (IQR) each attribute.
ur attr
The total number of attributes.
sparsity
The (possibly normalized) sparsity metric for each attribute. The trimmed mean of cach attribute.
mmaan
nr_bin
number binary attributes.
ca_
The number categorical attributes.
nr_num
The number of numeric features.
nr_norm
The number of attributes normally distributed based in given method.
nr_cor_attr
The number of distinct highly correlated pair of attributes.
gravity
distance hetween minority and majority classes' center of mass. The number distinct classes.
ur class
joint ent
The joint entropy between each attribute and class. Shannon's entropy for each predictive attribute.
attr ent
cov
The absolute value of the covariance of distinct dataset attribute pairs.
eigenvalues
The cigenvalues of covariance matrix from dataset.
eq_num_attt
The number of attributes equivalent for a predictive task.
max
The maximum value from each attribute.
min
The minimum value from each attribute.
median
median value from each attribute.
fleq_elass
The relative frequency of each distinct class.
mad
The Median Absolute Deviation (MAD) adjusted by factor.
mad
The Median Absolute Deviation (MAD) adjusted factor.
mut_inf inf
The mutual information between each attribute and target.
nr_inst
number instances (rows) in dataset.
nr_outliers
The number of attributes with at least one outlier value.
ratio
The noisiness of attributes.
imblance ratio
The ratio of the number of instances in the minority the majority class.
altr inst
ratio between the number of attributes.
address the scalability constraints inherent to the original architecture. Nevertheless, by improving model usability without retraining, reduce the computational and environmental costs typically associated with large model development.
To the best of our knowledge, this work poses explicit ethical concerns. It provides practical guidance for applying pre-trained tabular models in real- world domains such as healthcare and finance, where transparency and efficiency essential. Our study highlights the value of understanding and extending foundation models alongside architectural innovation.
31

---

Table 10: Performance of various models on 18 high-dimensional datasets. The results show the mean accuracy of different models, including ModemNCA (MNCA), MLP, KNN, RealMLP, XGBoost (XGB), Random Forest (RForest), Logistic Regression (LogReg), TabPEN v2 (PFN-v2), TabPFN v2 with PCA (v2-pea), TabPFN v2 with subsampling (v2*), ProtoGate (ProtoG), and CatBoost (CatB). The performance is evaluated on digh-dimensional datasets, with the values representing
mean accuracy for each model.
Dataset
MNCA
MLP
KNN
RealMLP
XGB
Rifersst
LogReg
PFN-v2
x2-pca
V2"
ProtoG
CatB
SUR. 11I
62.90
72.46
73.04
70.14
73.91
70.14
71.59
BASEHOCK
96.31
97.01
71.8%
97.46
95.29
96.73
96.99
69.09
97.41
97.36
96.32
95,87
Prostate
81.27
86.98
87.94
95.24
94,92
PCMAC
88.21
88.53
66.48
90.15
91.64 91.64
92.20
87.15
92.70
90.76
90.14
88.21
92.01
GLL 85
81.57
83,92
80.39
80.78
RELATHE
88,18
90.54
75.03
90..23
87.11
87.30
90.49
86.36
87.65
89.95
89.92
90.35
SMK CAN_
63.51
70.70
71.05
71.75
70.71
71.40
WarpPIE10P
98.41
99.05
92.38
100.0
94.92
98..57
100.0
100.0
100.0
100.0
97.79
98.89
leukemia
90.22
97.78
92.00
urlraws 10P
97.67
98.33
92.00
99,00
84.33
99.00
99.00
92.00
99.33
99.67
92.67
99.00
GLIOMA
58.00
67.33
66.67
62.67
68.67
warp-RROP
83,08
85.64
53.08
97.44
81.28
87.18
97.69
90.77
95.38
96.67
90.04
87.44
V01_
76.00
78.10
lung
91.54
95.45
93.66
95.28
93.66
92.68
95.12
95.28
93.50
95.61
95.43
93.01
ALLAML
87.56
95.56
81.33
96.00
96.44
92.89
93.78
94.67
colom
78.46
78.97
76.92
83.08
74.87
82.56
86.15
81.54
78.46
79.49
78.46
77.95
gisette
97.21
97.57
95.04
97.55
96.82
97.51
97.35
97.26
97.18
97.78
arcene
81.67
85.50
84,50
81.00
75.00
86,83
88.00
83.67
88.33
92.00
85.33
85.00
Mean
82.86
87.11
77.29
88.83
84.76
86.87
89.36
85.25
87.29
89.73
86.37
87.48
Table 11: Performance of various models on 12 multi-class classification tasks with more than 10 classes. The results show the mean accuracy of different models, including KNN, PFN-v2*, PFN- v2-ECOC, XGB (XGBoost), CatBoost (CatB), Random Forest (RForest), ModemnNA (MNCA), Multi-layer Perceptron (MLP), Logistic Regression (LogReg), and ReaLMLP. The performance is evaluated on 12 multi-class datasets with more than 10 classes, with accuracy values presented for each model on the respective datasets.
PFN-22
PFN-x2-DPT
PIN-v2-ECOC
RForest
MNCA
MLP
LogReg
RealMLP
10l-plantt-lexture
79.69
91.94
82.67
84, 92
77.06
89.73
82.65
80.52
83.92
86.88
88.35
no0-plants-margin
77.50
88.56
$1.91
79.40
74.25
84.06
82.79
77.60
80.44
79.69
83.58
shnppe
79.52
72.15
47.33
1:31 Pen, Characters
36.26
45.71
33.38
44,20
30.35
38.88
34.24
44.03
37.75
19.41
46.37
texture
98.45
100.0
99.98
100.0
98.55
99.13
96.76
99.68
99.
99.64
99.95
96.69
91.55 36.56
walking-activit
60.29
61.22
57.28
61.92
65.06
64.92
61.74
64.85
60.64
27.02
65.13
belena
28.94
33.31
28.54
19.20
32.42
37.90
33:91
36.58
37.91
33.40
38.55
Internet_ usage
33.91
kropt
71.22
77.96
71.44
77.11
86.95
79.26
71.77
78.27
64.45
28.08
92.03
kr-vs-k
70.78
78.46
71.54
76.29
87.26
74.81
71.60
76.83
65.03
28.03
91.85
ASP-POTASSCD
34.75
43.50
45,27
42.24
41.06 08
42.0
37.45
29.63
41.70
Moan
61.94
70.93
65.67
66.69
66.47
67.47
64.01
68.00
62.16
51.40
72.51
Table 12: Performance of various models on 18 large-scale datasets. The results show the mean accu- racy/RMSE of different models, including MLP, Logistic Regression/Lineat Regression (LR), KNN, XGBoost (XGB). Random Forest (RForest), CatBoost (CatB), ModernNCA (MNCA), ReaLMLP, and various versions of TabPFN v2: original TabPEN v2 (PFNv2), TabPFN v2 with K-means (PFNv2-K), TabPFN v2 with Bagging (PFNv2-B), PFNv2* (TabPFNv2*). PFNv2-DT (TabPFN-DT), and PFNv2-
DE "TaPPEN-DF
Dutesct
MLP
LB
KNNN
XGB
BFures
Ca:B
MNCA
ResIMLP
PEN-2
PFN:2-K
PFN-V-B
PEN-2
PRN-2-DT
PFN-2.DE
BNGinndlis)
00.07
$5.95
87.41
90.21
89.25
01.13
$0.98
90:91
50.55
89.01
89.56
80.80
90:45
90.43
CDC Indicaties
36.79
35.55
30.39
90.70
95.00
90.74
36.76
56.76
30.65
00.08
50,
80.74
55.75
66.70
Higgs
71.87
74.81
75.36
72.01
73.51
Sumning_sppall
73.08
73.09
73.55
73.74
73.84
sfpollic sacideuts
85.87
87.84
87.57
$7.54
$3.54
$7.54
$7.84
Tate Crowditrdling
06.45
67.04
05.70
95.80
95.20
95.81
06.53
96.71
94.55
91.51
94,
95.07
98.
96.83
Fachion MNIST
$9.54
55.69
86.00
92.11
85.57
90.24
$0.35
90.25
68. 40
82.82
$3.59
86.25
91
78.92
coventye
96.01
72.54
92.76
95.30
75.30
90.77
97.31
$7.35
54.54
$3.95
$4.16
95.35
97.35
97.44
junni
11.90
64.60
65.67
71.83
69.19
72.26
72.57
73.00
70.34
70.26
70.59
71.31
72.57
72.50
hane--had
59.90
50.12
$4.01
99.51
64.63
07.69
76.31
99.58
41.97
38.56
36,50
54.12
91.13
02.35
volker
69.55
35.25
67.41
69
62:71
2008
77.18
73.76
60.81
61.15
62.81
64.84
08.05
67.75
Atrines Depieday (ctol)
1,005
2.953
1.170
3.907
2.881
2.92
3.903
3.807
Wase Erergy 101)
5.190
15.19
92.20
7.294
7.173
6.188
7.314
706
6.515
6.785
Ullnatent (x10))
0.004
23,19
9.130
7.455
7.422
Magleeehaack (c.00")
3,026
2044
1.963
2,105
2.000
1.314
1.944
nicrosoft Ex10
7.577
7.514
7.366
7,43
5.077
7.730
7,344
7.728
Yuhoo
7.514
8132
7.611
8,099
32

---

Table 13: Performance of TabPFN v2 and the extracted feature embeddings across 29 classification datasets. The table shows the average classification accuracy for each dataset when using different layers (Layer 6, Layer 9, Layer 12) of the transformer as feature embeddings, as well as the "combined" approach, where embeddings from up three selected layers are concatenated. The "selected layers" column indicates the specific layers chosen for each dataset based on validation set performance. "Vanilla" refers the embeddings extracted using the vanilla strategy, which utilizes only the 12th layer of the transformer. "S" and "p" refer to unsupervised embedding extraction approaches by appending a column of dummy labels with zero values and permuting each column as labels.
respectively, as described in Appendix C
PFN-v2
Vanilla
layer-6 6
layer-9
layer-12 12
cumbined
selected layers
FOREX audchf-daay-Higg High taiwanese Bactuuupiicpppeddcttie
77.38
74.11
(5,9,
96.99
56.45
96.77
95.75
97.14
96.77
97.07
97.14
(6)
d
85.51
50.00
60.56
70.82
66.90
69.52
86.72
87.53
(11, 12)
pc3 eye movements hin
89.46
10.22
89.78
86.90
90.10
88.82
88.82
88.82
61.83
50.00
$5.12
57.95
59.72
59.40
62.16
62.16
(6, 12)
BNG(breast-w)
98.43
69.51
98.51
98.46
98
98.51
FOREX FORFX_cadjpy-hnne cadipy-hour--igh
69.53
51.79
66.55
71.12
62. 12
64.87
70.66
70.88
(4,5, 6)
99.34
85.43
98.41
98.54
98.41
98.28
99.34
99.47
(4.5,
sylvine
97.46
85.66
72.78
95.71
92.49
93.95
97.27
96.49
(1.
BNG(tic-La Laettc--aeetee)
(5. 12) (8)
online shoppers
90.59
84.51
85.93
89.46
90.12
90.11
90.63
90.02
Curdiovascular Disease datasel
credit
78.04
62.31
75.86
78.31
77.62
77.80
77.95
77.59
(4,6, 9)
FOREX audsgd-hour-Hipg
67.26
57.24
61.06
69
70.41
(7. 12) (1, 11) (10, 11, 12)
waveforn-5000
86.00
80.60
$5.70
87.10
85,60
85.60
86.40
86.90
jungle_che cchess
85.65
39.60
64.12
72.14
78.55
80.44
86.66
86.85
BNG((mm)
57.40
42,62
$2.48
$5.16
56.19
56.72
57.72
$7.88
(9, 12)
page-blocks
97.35
94.25
95.43
96.35
96.07
96.71
97.17
97.35
(6. 12)
93.07
72.29
69.26
87.23
91.99
88.10
93.51
92.64
(2)
website_ website_phishing
90.77
36.90
82.66
90.04
85.98
87.08
91
91.88
(7, 10)
bascball
93.66
78.73
92.54
93.28
93.66
pendigits
99.50
$9.75
72.40
98.18
92.81
93.04
99.41
99.45
(3, 4,)
Gendor_Gan Gap Spanish_ WP
60.84
33.68
$9.68
wine-qualiy-whaee
62.35
10.51
49.29
$5.10
$4.08
55.31
63.57
64.39
(8. 11. 12)
satimage
91.21
(8,
mffaatfouriie
90.00
55.50
46.75
85.75
77.75
82.25
89.50
89.50
(2,7, 12)
VulNoneVal
98.95
1.05
98.95
98.33
98.95
98.95
98.95
www-schoolladdmssio---ee
100.0
99.83
79.76
98.82
100.0
100.0
100.0
100.0
KDD
80.34
78.45
62.36
76.76
79.34
78.35
81.23
(L,8, 10)
33


Reviews:
[{"review_id": "VKkmCAjvQ5", "reviewer": "Reviewer_wWLo", "summary": "The paper provides an in-depth analysis on the recent tabular foundation model, TabPFNv2. The results show characteristics of TabPFNv2 that can aid strong performances, and be used as a feature extractor. Moreover, the paper addresses some limitations and approaches to overcome them.", "questions": "Please refer to weaknesses.", "limitations": "It could be interesting to elaborate more on some possible extensions for repurposing the extracted features through feature encoding of TabPFNv2, and some more analysis with regression tasks.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is generally well-written and easy to follow.\n- The paper provides insights on important factors for in-context learning tabular foundation models.\n- The paper provides some solid directions for overcoming the limitations of TabPFNv2.\n\nWeakness:\n- In regards to the point 'internalizes attribute token learning', it could be interesting to compare it with the TabICL, which might be doing something similar in a different way, and emphasize the importance of this mechanism for in-context tabular foundation models.\n- On the results on section 6.3, it would be interesting to see results on regression problems with simple linear models. \n- Also, in regards to the feature encoder, it could be useful to note that the work may be useful to integrate with other modules, such as domain adaptations methods.\n- Overall, how would or what are the (expected) results if TabPFNv2 is fine-tuned?\n- It could be interesting to compare with TabICL [1] on 'Multi-Class Problems with More Than 10 Classes' (Or mention the work, if it has similar strategies)\n- On a similar topic, it would be interesting to compare with TabICL for larger than 10K samples.\n\n[1] Qu, Jingang, et al. \"Tabicl: A tabular foundation model for in-context learning on large data.\" arXiv preprint arXiv:2502.05564 (2025).", "was_revised": true, "final_justification": "The authors have addressed many of my concerns and added empirical studies that could strengthen the quality of the paper.", "rebuttal": "We sincerely thank Reviewer wWLo for the thoughtful and constructive feedback. We address each point below:\n\n**Q1: Comparison with TabICL and internalized attribute token learning**\n\nThank you for the suggestion. We were aware of TabICL and have cited it [35], and we will extend the discussion. Both TabPFN and TabICL leverage in-context learning for tabular prediction, but they adopt distinct approaches to attribute tokenization. (Even TabPFN v1 and v2 adopt quite different tokenization mechanisms; see Lines 90 - 93 & sect. 3.) TabPFN v2 uses randomized attribute tokens (RTs), re-sampled per inference, which internalize feature relationships without requiring semantic alignment (Lines 207 - 214). In contrast, TabICL employs set transformers to encode distributional properties of an attribute, combined with RoPE for positional differentiation. While TabICL requires specifically learning distribution-aware tokens, TabPFN v2’s RTs enable plug-and-play generalization across heterogeneous tasks. We will expand this comparison in the final version to highlight the conceptual and practical implications of each approach.\n\n**Q2: Regression results and extensions for feature reuse**\n\nWe have extended our feature extraction strategy to regression tasks. The embeddings generated via our leave-one-fold-out method show simplified and structured distributions, similar to the classification case. (Figures will be included in the final version.) To assess their utility, we trained linear regression (LR) on raw features, vanilla embeddings (c.f. section 6), and our leave-one-fold-out extracted embeddings. The average rank across regression datasets is summarized below:\n\n| Embedding           | Regressor | Avg. Rank |\n|---------------------|-----------|-----------|\n| Raw                 | LR        | 2.42      |\n| Vanilla    | LR        | 2.97      |\n| Ours                | LR        | 1.58      |\n\nOur embeddings lead to **better regression generalization than raw features and vanilla TabPFN features**. These results confirm that the extracted embeddings can serve as high-quality, reusable features.\n\n**Q3: Integration with domain adaptation methods**\n\nWe agree this is a promising direction. Our feature extraction method offers a way to generate robust embeddings without retraining, which could complement domain adaptation techniques and enable cross-domain generalization. We will highlight this in future discussions.\n\n**Q4: Fine-tuning TabPFNv2**\n\nThank you for the question. We note that a recent concurrent study ([Rubachev et al., 2025]) explores fine-tuning TabPFN v2 and demonstrates improvements using simple full finetuning. In our analysis (Table 2), we observe that combining embeddings from multiple layers enables strong linear probing performance—outperforming the original TabPFN output. This suggests that probing can serve as a lightweight, interpretable alternative to fine-tuning.\n\nHowever, fine-tuning TabPFN v2 remains challenging for high-dimensional or many-class tasks due to (a) the scarcity of high-dimensional datasets and (b) the architectural constraint that TabPFN v2 is designed as a 10-class output format. To adapt it to many-class problems, one must jointly fine-tune both the label encoding scheme and the output head—adding substantial complexity and reducing generality. In contrast, our post-hoc divide-and-conquer strategies handle these challenges effectively without retraining, aligning with recent trends in test-time scaling in LLMs.\n\nFor large-scale datasets, fine-tuning is an effective choice given enough training time. Due to rebuttal time constraints, we fine-tuned TabPFN v2 on two large-scale datasets and compared it with our post-hoc divide-and-conquer strategies.\n\n| Dataset            | Method        | Accuracy (%) | Training Time (s) |\n|--------------------|---------------|---------------|---------------------|\n| **nomao** (34K × 118) | PFN v2        | 96.08         | 9.2                 |\n|                    | PFN v2*-DF (ous)  | 96.75         | 8.0                 |\n|                    | Fine-tuned    | 97.01     | 1771                |\n|                    | CatBoost      | 97.03         | –                   |\n| **jannis** (83K × 54) | PFN v2        | 70.24         | 37.9                |\n|                    | PFN v2*-DF (ous)   | 72.57         | 17.7                |\n|                    | Fine-tuned    | 73.77    | 12201               |\n|                    | CatBoost      | 72.26         | –                   |\n\nThese results show that while fine-tuning can yield marginal performance gains, it comes with a significant computational cost. In contrast, our PFN v2*-DF approach achieves competitive performance **with substantially lower overhead** by taking advantage of its \"divide-and-conquer\" strategy, offering a practical alternative for scaling TabPFN v2.\n\n**Q5: Comparison with TabICL on many-class classification**\n\nWe followed the setup in Figure 5 (middle) to evaluate TabICL on multi-class datasets (>10 classes). The average accuracy of TabICL is 0.6901, while our method achieves 0.7093. These results suggest our strategy scales more effectively to complex label spaces. We will extend this discussion in the final version. \n\n**Q6: Comparison with TabICL on large-scale datasets (>10K samples)**\n\nUsing the configuration in Figure 5 (right), we report the average ranks on the large-scale classification datasets:\n- PFNv2*-DF: 4.50 (with 4 ensemble models)\n- TabICL: 6.25 (with 4 ensemble models)\n\nOur method demonstrates stronger generalization in large-scale settings, highlighting its scalability advantages. We will extend this discussion in the final version. \n\nIn light of our further clarification, we would appreciate it if the reviewer would consider increasing the rating. We would be happy to address additional questions you may have in the author-reviewer discussion phase."}, {"review_id": "OuojATzjMi", "reviewer": "Reviewer_pvYD", "summary": "The paper presents an in-depth empirical and analytical study of TabPFNv2. Rather than proposing a new model architecture, the authors aim to dissect the mechanisms behind TabPFNv2's success and address its known limitations (e.g., performance on large-scale, high-dimensional, and multi-class datasets). The paper makes three main contributions: (1) it uncovers how TabPFNv2 internalizes attribute token learning despite randomized inputs; (2) it repurposes TabPFNv2 as a feature encoder via a novel leave-one-fold-out embedding extraction strategy; and (3) it introduces effective post-hoc, test-time divide-and-conquer extensions to improve scalability.", "questions": "- The introduction states 'TabPFN v2 internalizes attribute token learning to handle data heterogeneity.’: I could not understand what the new insight from the paper is. That TabPFN does prediction in a single forward pass was already a major selling point for the first TabPFN publication and the paragraph reads as if this is just reformulated. The explanations and illustrations in Section 5 are interesting, but don't seem like they present particularly new insights. That TabPFN(v2) can reliably infer meaningful attribute relationships through in-context learning was already presented in the original TabPFN paper. Could you briefly summarize the core insights that are novel?\n- Line 60: 'Our findings reveal that TabPFN v2 effectively maps tabular instances into a nearly linearly separable embedding space' - Yes, TabPFNv2 maps instances into a linearly separable space. So does every other deep learning model since the last layer is always a linear layer with target-specific activation. Therefore, it is obvious that the embeddings are as close to being linearly separable as possible and that training a linear model on top yields strong performance. I did not understand what the new insights are, besides that out-of-fold embeddings need to be used? \n- The random forest method proposed in 7.1 seems to be the same as the one used in the TabPFNv2 paper for handling large sample sizes, but in this case for features? Also the method introduced in 7.2 seems to be entirely taken from the TabDPT paper? Also the method to handle large datasets seems to be a variation of an existing method. It seems like no conceptually new proposal has been made?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Understanding capabilities of prior fitted networks is an important and timely research topic, given their recent popularity\n- The paper is well organized and clearly written.\n- Extensive evaluation with several baselines and clearly described evaluation protocol.\n- Careful considerations on dataset selection excluding datasets that were used for checkpoint selection by the TabPFN authors.\n- The authors repeatedly emphasize that analytical studies are ‘more impactful than proposing yet another architecture.’ – I agree with that. \n- The proposed extensions are practical, easy to apply, and follow successful practices in related applications (i.e. the use of out-of-fold data is used in state-of-the-art ensembling approaches for tabular data). \n\n### Weaknesses\n- The major weakness of the paper is that many of the insights seem obvious and are limited in novelty. Please refer to the questions section for details.\n- A single validation and test set is not enough to establish statistical significance on dataset level, hence, conclusions based on dataset level results are not reliable.", "was_revised": true, "final_justification": "The rebuttal has resolved some of my concerns, therefore, I update my score.", "rebuttal": "We sincerely thank Reviewer pvYD for the thoughtful review and for recognizing the importance and timeliness of our work, the clarity of our paper, and the rigor of our evaluation methodology. \n\nWe respond to the specific questions and comments below:\n\n**Q1: Statistical significance and use of a single validation/test split**\n\nThanks for the comment. We followed standard evaluation protocols from prior work (e.g., [Gorishniy et al., 2021]) to ensure comparability, using a 64/16/20 training/validation/test split. We agree that multiple train/validation/test splits would provide stronger statistical guarantees; however, this would incur a prohibitive computational cost due to the need for full hyperparameter tuning across all baselines and 273 datasets.\n\nTo assess robustness, we performed an additional comparison using **5-fold cross-validation** on 10 classification datasets from the *Talent-tiny benchmark*. We observed consistent relative rankings across both hold-out and cross-validation settings:\n\n| Method     | Avg. Rank (hold-out) | Avg. Rank (5-fold CV) |\n|------------|-----------------------|------------------------|\n| TabPFN v2  | **1.86**                  | **1.64**                   |\n| ModernNCA  | 1.95                  | 2.64                   |\n| TabM       | 3.05                  | 2.45                   |\n| FTT        | 3.14                  | 3.27                   |\n\nThese results confirm that the superiority of TabPFN v2 reported in our paper is robust to the evaluation protocol. We will include this consistency check in the final version.\n\n**Q2: Novelty of insights on randomized tokens and attribute token learning**\n\nWe appreciate your comment. We want to first reiterate that our novel insight lies in understanding how “randomized tokens” work and enable TabPFN v2 to handle data heterogeneity (Lines 48 - 50). We certainly are aware that both TabPFN and TabPFN v2 adopt in-context learning and a single forward pass for inference (Line 89). However, their mechanism to handle data heterogeneity are drastically different (section 3). \n\nThe original TabPFN v1 relied on padding features to a fixed dimension (e.g., 100) to preserve their original semantics, whereas **TabPFN v2 introduces randomized tokens**, sampled at inference time, for each attribute. This design eliminates the need to define attribute- or dataset-specific tokens, syntactically enabling direct application of the pre-trained model across heterogeneous datasets (section 5.1).\n\nOur novel insight is that, although randomized tokenization appears to discard attribute semantics, TabPFN v2 can *still* consistently infer meaningful attribute relationships through in-context learning. Specifically, we show that randomized tokens act like high-dimensional “continuous one-hot encodings” that reliably differentiate attributes across instances (Line 245). Pre-training on diverse tasks enables TabPFN v2 to infer co-occurrence patterns and relative attribute importance purely from the statistical structure of each dataset, without pre-defined attribute semantics.\n\n**Q3: Linearly separable embeddings and leave-one-fold-out strategy**\n\nWe appreciate your comment. Before diving into the claim in line 60, we kindly invite the reviewer to revisit our preamble in lines 54 - 56. Just like what the reviewer argued, after seeing the superior performance of TabPFN v2, we expect to see a linear separable feature space produced by TabPFN v2, similar to other deep learning models. However, verifying, visualizing, and extracting it is non-trivial and not obvious (Lines 56 - 58). \n\nThis is because **TabPFN v2 is a non-standard in-context learner**, and naive extraction of embeddings from its output tokens leads to surprisingly discrepant features between training and test instances—stemming from their different label embedding types (true vs. dummy). As demonstrated in Figure 4(b), embeddings extracted in this way create divergent clusters for training (darker crosses) and test (lighter circles) points, leading to poor generalization when used in a linear classifier.\n\nOur novelty lies in a carefully designed approach to extract features from TabPFN v2 (section 6.2). We propose a **leave-one-fold-out embedding extraction strategy** to resolve this issue: training instances are iteratively placed in the query set with dummy labels, while the remaining folds form the support set with true labels. This aligns the roles of training and test embeddings, resulting in a comparable embedding space. We empirically show that this strategy significantly improves linear probing performance (Table 2) and yields embeddings that match TabPFN v2’s in-context performance.\n\nIn sum, our contribution lies not merely in observing separability but in **identifying and addressing a fundamental inconsistency in naive embedding extraction** that was not discussed in prior work.\n\n**Q4: Extensions may overlap with existing methods (e.g., Random Forest, TabDPT)**\n\nWe appreciate your comment, and we emphasize the differences and our contributions as follows. First of all, the purpose of section 7 is to demonstrate that the limitations of TabPFN v2 (Lines 13 - 15 and 65 - 71) can be effectively addressed without re-training the model, reminiscent of test-time scaling. In our humble opinion, our key novelty and contribution lie in pointing out and demonstrating this concept, not the particular methods we applied. Indeed, we deliberately choose not to propose a completely new mechanism but borrow and adapt existing mechanisms originally proposed in different contexts to TabPFN v2.  \n\n- **Large-scale datasets**:\n\n The decision tree integration in TabPFN v2’s original paper is used for *small-and medium-scale datasets suffering from within-dataset heterogeneity*. We adapt this approach for a **different purpose**—handling large-scale datasets by partitioning them into smaller subsets where TabPFN v2 can operate effectively.\n\n- **Many-class classification**:\n\nWe have provided the detailed implementation and the difference between ours and TabDPT (denoted as PFN-DPT in our paper) in Appendix C lines 92-99.\nWhile decimal encoding (TabDPT) is an existing idea, our TabPFN v2∗ improves robustness by *randomly permuting class-to-digit mappings √C times* and ensembling predictions. This mitigates artificial correlations introduced by fixed decimal encodings. We ensure fair comparison by matching the number of inferences of TabPFN v2 when ensembling their results in different methods.\n\n| Method               | Mean Accuracy |\n|----------------------|----------------|\n| TabDPT (no ensemble) | 0.6047         |\n| TabDPT (ensemble)    | 0.6567         |\n| TabPFN v2∗ (ours)    | **0.7093**         |\nOur method is superior, and we explicitly state in the paper that TabPFN v2-DPT (Figure 5, middle) refers to the ensemble variant.\n\n\n- **High-dimensional datasets**:\n\nWe adapt Random Forest-style feature subsampling and bagging specifically for TabPFN v2 (Section 7.1), a direction unexplored in the original paper.\n\nIn light of our further clarification, we would appreciate it if the reviewer would reconsider the rating. Thank you."}, {"review_id": "5NURfE2u72", "reviewer": "Reviewer_vnsZ", "summary": "The authors propose a systematic study of TabPFN v2, a popular ICL tabular foundation model.\nMore specifically, they analyze how this model learns feature dependencies, show that it can be repurposed into an encoder and propose some test-time strategies to improve performance.", "questions": "See question in the text above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "As tabular ICL foundation model becomes more widespread understand how they work, can be used and improved becomes more important.\nFirst, it is appreciated to have an overview of some of the internal mechanics of TabPFN v2 publicly available as the nature paper is quite scarce on details.\n\n**Attribute relationships**: It is interesting to see that features tend to attend to target first then to other features. Did you have more insights or other methods of explanation on feature relationship? It can be hard to relate attention maps to explanations. For example does a pair of attributes with high attention weight really contributes more to model performance than one with lower one?\n\n**TabPFN as a feature encoder**: While I agree that naive feature encoding will fail and that you method is sound, I am wondering, what happens if you simply use context (X_train, y_train) and queries = (X_train, X_test) and use embeddings of the queries for train and test, i.e no partitioning. Is that a bad feature encoder? Intuitively the network probably doesn't \"replace\" right away the dummy label of a train example in Q by its true label in S so I would have actually assumed this to perform decently.\n\nAnother question, this discrepancy disappears through the layers as the test examples get closer to training ones (fig 4 in layer layers), so does the naive approach actually also work decently for later layers?\n\n**Improvements**: this is the most interesting section to me, I like that you did some good ablations comparing different methods.\nCouple questions.\nFor 5.b, you are doing $\\sqrt{C}$ ensemblings vs $\\log_{10) C$: did you control for the number of ensembles in any way? Or does PFNv2* use more compute (and maybe even more ensembles over features and so on?).\nFor 5a I am not too sure this can be called divide and conquer: this usually implies solving the subproblems recursively, this seems to be a simple Monte Carlo ensembling no?\nIs 5c also compute controlled? The decision forest/SQ seem quite more expensive than the rest.\n\nOverall the novelty of the paper is not very high, but the content is clear, exposes some important ideas, test them extensively and has some nice insights.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer vnsZ for the thoughtful and constructive feedback. We appreciate your recognition of the value in analyzing and understanding tabular foundation models, particularly in clarifying the internal mechanisms of TabPFN v2. We address your specific questions and comments below:\n\n**Q1: Attribute relationships and attention maps**\n\nThis is a very good question. In our humble opinion, understanding the inner workings of transformers—especially in the tabular context—remains an open and actively evolving research area, encompassing topics such as interpretability, explainability, and mechanistic understanding. In this work, we primarily rely on the most widely adopted approach: analyzing attention maps. While this provides some qualitative insights into how TabPFN v2 models feature interactions, we agree that attention alone may not fully capture the model’s reasoning process.\n\nWe plan to explore more advanced interpretability techniques in future work. Potential directions include probing-based analyses, attribution methods (e.g., SHAP adapted for in-context learning), and layer-wise relevance propagation. However, these approaches are still under development for transformer-based tabular models, and challenges remain in defining reliable ground truth for feature importance and interpreting high-dimensional interactions in models trained across diverse datasets. We appreciate the reviewer’s encouragement to pursue these avenues.\n\n**Q2: Using the same context and query without partitioning**\n\nIndeed, we experimented with this variant where the context contains all training data, and the query set includes both training and test points. This approach improves upon the vanilla feature extraction baseline but still underperforms compared to our proposed leave-one-fold-out method. We refer to this as the “non-partitioned” strategy, which suffers from role ambiguity—query points that appear in the support set (with dummy or ground-truth labels) are treated inconsistently. The network may not fully differentiate between training and test roles, degrading feature consistency. The performance ranking (lower is better) on classification tasks is as follows:\n\n| Method                     | Avg. Rank |\n|----------------------------|-----------|\n| Leave-one-fold-out (Ours)                       | **2.10**     |\n| TabPFN v2            | 2.72      |\n| Non-partitioned (12 layers)| 3.07      |\n| Non-partitioned (9 layers) | 4.29      |\n| Non-partitioned (6 layers) | 4.43      |\n| Vanilla (12 layers)        | 5.86      |\n| Vanilla (9 layers)         | 6.62      |\n| Vanilla (6 layers)         | 6.90      |\n\n**Q3: Does the naive strategy improve in later layers?**\n\nYes, embeddings from the vanilla extraction do become smoother and more clustered in deeper layers. However, their quality, as measured by downstream linear probing, still lags behind both TabPFN’s native outputs and our leave-one-fold-out embeddings. This suggests that deeper layers alone cannot fully mitigate the role ambiguity caused by naive embedding extraction.\n\n**Q4: Are the ensemble sizes in Figure 5b controlled?**\n\nYes. For a fair comparison in many-class classification (Figure 5b), all methods, including TabPFN v2*, TabPFN v2-DPT, and our ensemble extensions, use the same number of ensemble members. For instance, TabPFN v2* uses √C ensembles, where C is the number of classes.\n\n**Q5: Is “divide-and-conquer” a fair term for 5a?**\n\nThank you for pointing this out. While our method does not follow a recursive divide-and-conquer paradigm, we use the term more broadly to describe the process of decomposing complex problems (e.g., many-class classification) into simpler subproblems (e.g., binary digit classification), inspired by ideas in test-time strategies and Monte Carlo ensembling. We agree that clarification is needed and will revise the terminology in the final version.\n\n**Q6: Is compute controlled in 5c (TabPFN v2-DF)?**\n\nYes, we carefully controlled compute budgets to ensure a fair comparison across all methods. Specifically, we set the number of ensembles to four for each method in the large-scale classification setting, including our PFN*, TabICL, PFN-Bagging, matching the cost of four forward passes of TabPFN v2. We will clarify this further in the final version of the paper.\n\nBelow, we report a subset of average rank results across large-scale classification datasets, demonstrating that our proposed PFN\\* remains competitive under controlled compute:\n\n| Method                | Avg. Rank |\n|-----------------------|-----------|\n| CatBoost              | 2.83      |\n| RealMLP               | 2.92      |\n| **PFN\\*-DF** (ours)      | **4.50**  |\n| XGBoost               | 4.67      |\n| ModernNCA             | 4.73      |\n| TabICL             | 6.25      |\n| PFN\\*-SQ              | 8.42      |\n| PFN-v2-Bagging      | 10.25     |\n| RandomForest          | 11.08     |\n| PFN-v2                | 11.42     |\n\nThese results confirm that PFN\\* achieves strong performance even with strict compute constraints. We sincerely thank the reviewer for raising this point and helping improve the clarity and rigor of our experimental design.\n\nIn light of our further clarification, we would appreciate it if the reviewer would consider increasing the rating. We would be happy to address additional questions you may have in the author-reviewer discussion phase."}, {"review_id": "PNWUUEC5eu", "reviewer": "Reviewer_xoSe", "summary": "This paper presents a detailed empirical analysis of the recently proposed TabPFN v2. The authors investigate the model's mechanisms for handling data heterogeneity and its performance on standard benchmarks. They demonstrate that TabPFN v2 can effectively infer attribute relationships even with randomized token inputs and can be repurposed as a powerful feature extractor that maps data into a linearly separable space. Furthermore, the paper identifies the model's known limitations concerning high-dimensional, large-scale, and many-category datasets. To mitigate these issues, the authors propose and evaluate several post-hoc, \"divide-and-conquer\" strategies, such as feature subsampling, instance subsampling, and task decomposition, showing that these methods can improve TabPFN v2's performance in these challenging regimes without model retraining.", "questions": "See Weakness", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper's primary strength lies in its extensive and rigorous empirical evaluation. The authors conduct experiments on a very large suite of datasets (273 for the main analysis) and compare against a wide range of baselines. The analysis of the model's internal workings, through visualizations of embedding spaces and attention maps, is detailed and provides valuable phenomenological insights.\n\n2. The paper is exceptionally well-written, well-structured, and easy to follow. The research objectives are stated clearly in the introduction, and the narrative flows logically from analyzing TabPFN v2's strengths to identifying its weaknesses and then proposing solutions. The figures and tables are informative and support the main claims effectively.\n\n**Weaknesses:**\n\n1. The core contribution is not the proposal of a new method, but the analysis of an existing model and the application of well-established, standard machine learning techniques to mitigate its known weaknesses. For example, The method for high-dimensional data is essentially feature subsampling and ensembling (bagging), a core idea behind Random Forests.\n\n2. The work feels more like a technical report or an extended ablation study that would have been suitable for the evaluation track. The attempt to create a theoretical framing by linking the methods to \"Chain-of-Thought\" prompting is superficial and unsubstantiated, as the underlying mechanisms are entirely different.", "was_revised": true, "final_justification": "After reading the author's response and the comments of other reviewers, I am inclined to maintain the rating, as I have already given it a positive rating.", "rebuttal": "We sincerely thank Reviewer xoSe for the positive feedback highlighting our paper’s extensive empirical evaluation, in-depth model analysis, and clear presentation.\n\nWe respectfully address the two concerns as follows:\n\n**Q1: On the novelty of our contributions**:\n\nWhile we do not introduce a new model architecture, our central contribution lies in systematically evaluating, understanding, and extending TabPFN v2—a model that is rapidly becoming foundational in the tabular domain. As argued in the main text, principled analysis and adaptation of such pre-trained models is increasingly impactful and necessary, especially as foundation models become more opaque and widely adopted. Our extensions (e.g., TabPFN v2*) are not just reapplications of known techniques like bagging, but carefully tailored adaptations that address the scalability limitations of TabPFN v2 without retraining, enabling its use in previously unreachable domains like high-dimensional or many-class tasks.\n\n**Q2: On the connection to Chain-of-Thought (CoT)**:\n\n We acknowledge that the CoT analogy is conceptual rather than mechanistic. As clarified in Section 7, the idea is that sequential model reuse (e.g., using TabPFN v2 in a divide-and-conquer manner) mimics the layered reasoning process in CoT prompting. We agree this could be better framed and will revise the language to avoid overstating the parallel.\n\nWe hope this clarifies the intention and value of our work as a comprehensive, principled effort to advance the use and understanding of tabular foundation models."}]

Author rebuttal:

Meta-review: This paper provides an empirical and analytical examination of TabPFNv2. The work focuses on analyzing the factors underlying TabPFNv2’s performance and addressing limitations related to large-scale, high-dimensional, and multi-class datasets. The contributions are threefold: (1) demonstrating how TabPFNv2 learns attribute tokens even with randomized inputs; (2) adapting TabPFNv2 as a feature encoder through a leave-one-fold-out embedding; and (3) proposing test-time divide-and-conquer extensions to improve scalability.

Afer the discussion, most of reviewers' concerns are addressed and all recommended for acceptance.
Decision: Accept (poster)