Title: A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tabular Data

Tabular Foundation model
Abstract: Tabular datasets are inherently heterogeneous, presenting significant challenges for developing pre-trained foundation models. The recently introduced transformer-based Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented *in-context learning* performance across diverse downstream datasets, marking a pivotal advancement in tabular foundation models. In this paper, we take a closer look at TabPFN v2 to examine how it effectively handles heterogeneity and achieves high predictive accuracy, and to explore how its limitations in high-dimensional, many-category, and large-scale tasks can be mitigated. We find that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs, eliminating the need to explicitly learn dataset-specific attribute embeddings to address heterogeneity. We further show that TabPFN v2 can be transformed into a feature extractor, revealing its ability to construct a highly separable feature space for accurate predictions. Lastly, we demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-conquer strategy, enabling scalable inference without requiring re-training. By uncovering the mechanisms behind TabPFN v2's success and introducing strategies to extend its applicability, this study offers key insights into the design of future tabular foundation models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a detailed empirical analysis of the recently proposed TabPFN v2. The authors investigate the model's mechanisms for handling data heterogeneity and its performance on standard benchmarks. They demonstrate that TabPFN v2 can effectively infer attribute relationships even with randomized token inputs and can be repurposed as a powerful feature extractor that maps data into a linearly separable space. Furthermore, the paper identifies the model's known limitations concerning high-dimensional, large-scale, and many-category datasets. To mitigate these issues, the authors propose and evaluate several post-hoc, \"divide-and-conquer\" strategies, such as feature subsampling, instance subsampling, and task decomposition, showing that these methods can improve TabPFN v2's performance in these challenging regimes without model retraining.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper's primary strength lies in its extensive and rigorous empirical evaluation. The authors conduct experiments on a very large suite of datasets (273 for the main analysis) and compare against a wide range of baselines. The analysis of the model's internal workings, through visualizations of embedding spaces and attention maps, is detailed and provides valuable phenomenological insights.\n\n2. The paper is exceptionally well-written, well-structured, and easy to follow. The research objectives are stated clearly in the introduction, and the narrative flows logically from analyzing TabPFN v2's strengths to identifying its weaknesses and then proposing solutions. The figures and tables are informative and support the main claims effectively.\n\n**Weaknesses:**\n\n1. The core contribution is not the proposal of a new method, but the analysis of an existing model and the application of well-established, standard machine learning techniques to mitigate its known weaknesses. For example, The method for high-dimensional data is essentially feature subsampling and ensembling (bagging), a core idea behind Random Forests.\n\n2. The work feels more like a technical report or an extended ablation study that would have been suitable for the evaluation track. The attempt to create a theoretical framing by linking the methods to \"Chain-of-Thought\" prompting is superficial and unsubstantiated, as the underlying mechanisms are entirely different.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a systematic study of TabPFN v2, a popular ICL tabular foundation model.\nMore specifically, they analyze how this model learns feature dependencies, show that it can be repurposed into an encoder and propose some test-time strategies to improve performance.\n\nSTRENGTHS AND WEAKNESSES:\nAs tabular ICL foundation model becomes more widespread understand how they work, can be used and improved becomes more important.\nFirst, it is appreciated to have an overview of some of the internal mechanics of TabPFN v2 publicly available as the nature paper is quite scarce on details.\n\n**Attribute relationships**: It is interesting to see that features tend to attend to target first then to other features. Did you have more insights or other methods of explanation on feature relationship? It can be hard to relate attention maps to explanations. For example does a pair of attributes with high attention weight really contributes more to model performance than one with lower one?\n\n**TabPFN as a feature encoder**: While I agree that naive feature encoding will fail and that you method is sound, I am wondering, what happens if you simply use context (X_train, y_train) and queries = (X_train, X_test) and use embeddings of the queries for train and test, i.e no partitioning. Is that a bad feature encoder? Intuitively the network probably doesn't \"replace\" right away the dummy label of a train example in Q by its true label in S so I would have actually assumed this to perform decently.\n\nAnother question, this discrepancy disappears through the layers as the test examples get closer to training ones (fig 4 in layer layers), so does the naive approach actually also work decently for later layers?\n\n**Improvements**: this is the most interesting section to me, I like that you did some good ablations comparing different methods.\nCouple questions.\nFor 5.b, you are doing $\\sqrt{C}$ ensemblings vs $\\log_{10) C$: did you control for the number of ensembles in any way? Or does PFNv2* use more compute (and maybe even more ensembles over features and so on?).\nFor 5a I am not too sure this can be called divide and conquer: this usually implies solving the subproblems recursively, this seems to be a simple Monte Carlo ensembling no?\nIs 5c also compute controlled? The decision forest/SQ seem quite more expensive than the rest.\n\nOverall the novelty of the paper is not very high, but the content is clear, exposes some important ideas, test them extensively and has some nice insights.\n\nQUESTIONS:\nSee question in the text above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an in-depth empirical and analytical study of TabPFNv2. Rather than proposing a new model architecture, the authors aim to dissect the mechanisms behind TabPFNv2's success and address its known limitations (e.g., performance on large-scale, high-dimensional, and multi-class datasets). The paper makes three main contributions: (1) it uncovers how TabPFNv2 internalizes attribute token learning despite randomized inputs; (2) it repurposes TabPFNv2 as a feature encoder via a novel leave-one-fold-out embedding extraction strategy; and (3) it introduces effective post-hoc, test-time divide-and-conquer extensions to improve scalability.\n\nSTRENGTHS AND WEAKNESSES:\n- Understanding capabilities of prior fitted networks is an important and timely research topic, given their recent popularity\n- The paper is well organized and clearly written.\n- Extensive evaluation with several baselines and clearly described evaluation protocol.\n- Careful considerations on dataset selection excluding datasets that were used for checkpoint selection by the TabPFN authors.\n- The authors repeatedly emphasize that analytical studies are ‘more impactful than proposing yet another architecture.’ – I agree with that. \n- The proposed extensions are practical, easy to apply, and follow successful practices in related applications (i.e. the use of out-of-fold data is used in state-of-the-art ensembling approaches for tabular data). \n\n### Weaknesses\n- The major weakness of the paper is that many of the insights seem obvious and are limited in novelty. Please refer to the questions section for details.\n- A single validation and test set is not enough to establish statistical significance on dataset level, hence, conclusions based on dataset level results are not reliable.\n\nQUESTIONS:\n- The introduction states 'TabPFN v2 internalizes attribute token learning to handle data heterogeneity.’: I could not understand what the new insight from the paper is. That TabPFN does prediction in a single forward pass was already a major selling point for the first TabPFN publication and the paragraph reads as if this is just reformulated. The explanations and illustrations in Section 5 are interesting, but don't seem like they present particularly new insights. That TabPFN(v2) can reliably infer meaningful attribute relationships through in-context learning was already presented in the original TabPFN paper. Could you briefly summarize the core insights that are novel?\n- Line 60: 'Our findings reveal that TabPFN v2 effectively maps tabular instances into a nearly linearly separable embedding space' - Yes, TabPFNv2 maps instances into a linearly separable space. So does every other deep learning model since the last layer is always a linear layer with target-specific activation. Therefore, it is obvious that the embeddings are as close to being linearly separable as possible and that training a linear model on top yields strong performance. I did not understand what the new insights are, besides that out-of-fold embeddings need to be used? \n- The random forest method proposed in 7.1 seems to be the same as the one used in the TabPFNv2 paper for handling large sample sizes, but in this case for features? Also the method introduced in 7.2 seems to be entirely taken from the TabDPT paper? Also the method to handle large datasets seems to be a variation of an existing method. It seems like no conceptually new proposal has been made?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an in-depth analysis on the recent tabular foundation model, TabPFNv2. The results show characteristics of TabPFNv2 that can aid strong performances, and be used as a feature extractor. Moreover, the paper addresses some limitations and approaches to overcome them.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is generally well-written and easy to follow.\n- The paper provides insights on important factors for in-context learning tabular foundation models.\n- The paper provides some solid directions for overcoming the limitations of TabPFNv2.\n\nWeakness:\n- In regards to the point 'internalizes attribute token learning', it could be interesting to compare it with the TabICL, which might be doing something similar in a different way, and emphasize the importance of this mechanism for in-context tabular foundation models.\n- On the results on section 6.3, it would be interesting to see results on regression problems with simple linear models. \n- Also, in regards to the feature encoder, it could be useful to note that the work may be useful to integrate with other modules, such as domain adaptations methods.\n- Overall, how would or what are the (expected) results if TabPFNv2 is fine-tuned?\n- It could be interesting to compare with TabICL [1] on 'Multi-Class Problems with More Than 10 Classes' (Or mention the work, if it has similar strategies)\n- On a similar topic, it would be interesting to compare with TabICL for larger than 10K samples.\n\n[1] Qu, Jingang, et al. \"Tabicl: A tabular foundation model for in-context learning on large data.\" arXiv preprint arXiv:2502.05564 (2025).\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nIt could be interesting to elaborate more on some possible extensions for repurposing the extracted features through feature encoding of TabPFNv2, and some more analysis with regression tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}