Title: TabDPT: Scaling Tabular Foundation Models on Real Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tabular Foundation Models

In-Context Learning

Self Supervised Learning
Abstract: Tabular data is one of the most ubiquitous sources of information worldwide, spanning a wide variety of domains. This inherent heterogeneity has slowed the development of Tabular Foundation Models (TFMs) capable of fast generalization to unseen datasets. In-Context Learning (ICL) has recently emerged as a promising solution for TFMs, enabling dynamic adaptation to new tasks without additional tuning. While many studies have attempted to re-purpose large language models for tabular ICL, they have had limited success, so recent works have focused on developing tabular-specific foundation models. In this work, we propose an approach to combine ICL-based retrieval with self supervised learning to train tabular foundation models. We also investigate the utility of real vs. synthetic data for model pre-training, and show that real data can contain useful signal not easily captured in synthetic training. Specifically, we show that incorporating real data during the pre-training phase can lead to significantly faster training and better downstream generalization to unseen data. Our resulting model, **TabDPT**, achieves strong performance on both regression (CTR23) and classification (CC18) benchmarks. Importantly, we also demonstrate that with our pre-training procedure, scaling both model and data size leads to consistent performance improvements that follow power laws. This echoes scaling laws in LLMs and other foundation models, and suggests that large-scale TFMs can be achievable.
We open-source our full pipeline: inference code including trained model weights can be found [here](https://github.com/layer6ai-labs/TabDPT-inference), and the training code to reproduce experiments can be found [here](https://github.com/layer6ai-labs/TabDPT-training).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TabDPT, a tabular foundation model that has been pre-trained on (real) datasets from the internet. TabDPT leverages self-supervised learning and certain dataset-level augmentations to extract more information from available datasets, and the paper shows that its pre-training follows scaling laws for data size and model size. TabDPT also includes context retrieval allowing it to scale to larger datasets. The paper includes experiments on different benchmarks, where TabDPT is the best-performing method for classification and regression, and for few-shot learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**:\n- The paper is well-written.\n- The paper provides interesting experiments on a few aspects of pre-training.\n- The paper introduces some new and original ideas on pre-training with real-world data and other aspects.\n- Comprehensive code is provided (though the evaluation of baselines seems to be missing).\n\n**Weaknesses**:\n- Details on the benchmarking setup are missing, like: How are training+validation sets generated and used by different methods? What search spaces do these methods use and for how many steps are they tuned (specifically, when do you use the settings from TabZilla and when the ones from TabR, regarding search spaces, tuning method, number of tuning steps, etc.)? How are TabPFNv2 and TabDPT evaluated (does it get train+validation data or only train data)? How are large datasets (> 10K samples) handled for TabPFNv2?\n- From what I can tell, the comparison of TabDPT to baselines is not fair. Specifically, TabDPT can use the full training+validation set as context, while TabZilla uses holdout validation for baselines. However, the baselines could be much better by using cross-validation, which has even been explicitly demonstrated for TabZilla (https://arxiv.org/abs/2503.09159). I understand that many benchmarks use holdout validation because cross-validation is expensive, however, in this case it would be \"more fair\" in my opinion to only give the foundation models access to the training set and not the validation set, since it gives them a similar disadvantage to the one that other models have by not using cross-validation.\n- The paper seems to overclaim its results considering the weaknesses in the evaluation. Generally, the new TabArena benchmark (https://arxiv.org/abs/2506.16791) evaluates TabDPT and comes to quite different results than this paper. I know that the benchmark was uploaded after the NeurIPS deadline, so I am definitely not suggesting that the authors should have known about it when writing the paper. Nonetheless, I think the benchmark demonstrates that if the authors had used best practices that were known for a while (cross-validation, weighted ensembling, more tuning (?), more recent baselines like RealMLP/TabM/TabICL), they would have come to the conclusion that their method is not state of the art. I know that the paper has been on arXiv for a while and I don't mean to demand that all of the newest baselines be included, but given these circumstances I think the paper should be more modest about \"SOTA\" claims. Perhaps including the TabArena results would be the easiest way for the authors to address the concerns about the evaluation.\n- The retrieval of an individual context per sample does not allow for \"TabPFN-style batching\", making inference very slow (and this is not properly discussed / ablated in the paper - Section 4.7 is about training+inference, not just inference). (Adding to this comes the fact that model selection usually requires predictions for the validation set(s), which also take long to generate for methods with slow inference.)\n- The paper shows that TabDPT outperforms TabPFNv2, but it is unclear if this actually holds because of the retrieval procedure for TabDPT that presumably allows it to perform better for large datasets, or if TabDPT is also competitive with TabPFNv2 in the regime that TabPFNv2 is actually designed for.\n\n**Minor weaknesses**:\n- l. 73: You claim \"We show that applying our pre-training procedure to real data leads to faster convergence and better downstream accuracy than using purely synthetic data.\", but you only show it for one specific way of generating synthetic data (the TabPFNv1 way). Please don't overgeneralize the results.\n- The many-class classification approach in l. 207 is not ablated vs. some related approaches (e.g. TabPFNv2 extensions or TabICL) and it has the weakness that it can only model a certain factorized probability distribution over classes, instead of arbitrary probability distributions. Hence I would expect that this approach should not work well for probability-based metrics like log-loss or Brier loss. It might be good to ablate it or at least disscuss the potential advantages/disadvantages.\n\nQUESTIONS:\n- See the questions in the weaknesses part.\n- Does TabDPT use ensembling of different shuffles/settings like TabPFN?\n- What type of head and loss do you use for regression? Is it the same strategy as TabPFNv2?\n- In the few-shot evaluation, why did you only evaluate 7 of the 8 dataset of the \"STUNT\" paper? Is the \"income\" dataset included in the pre-training?\n\n**Minor comments**:\n- l. 96: \"another concurrent method requires a complex, three-stage procedure to learn from synthetic generators [47]\" - you make it sound like the three-stage procedure is required because of the synthetic generators, when in fact the three-stage procedure is used to efficiently scale to larger context windows, regardless of the type of data used for pre-training.\n- l. 159: what if you pick a categorical column but still want to do regression?\n- l. 329: \"suing\" -> \"using\"\n\n**Conclusion**:\nThe paper brings interesting contributions to the community, and if the concerns about evaluation fairness and accuracy of claims can be sufficiently resolved, I am willing to recommend acceptance.\n\nLIMITATIONS:\nSome limitations of the model are discussed at the end of the main paper, but limitations of the evaluation methodology are not discussed. I would argue that the latter should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an in-context-learning based tabular foundation model. The model is trained using a self-supervised learning pipeline on real data, which is in contrast to close relatives such as TabPFN that are trained on synthetic data. The authors claim that models trained on real data lead to significantly faster training and better performance. They also present tabular foundation model \"scaling laws\", which characterize how scaling model size and training set size leads to better performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper takes a step towards building fast and generalizable algorithms for predicting small tabular datasets. The paper is valuable as an exploration of the TFM space, and in particular, focuses on the value of training a TFM using real datasets. \n\nWeaknesses: In my opinion, there are two major weaknesses of this paper.\n\nFirst, there are various issues with how this paper is framed. For instance, the abstract can be interpreted as being somewhat misleading. The authors state that \"In this work, we propose an approach to combine ICL-based retrieval with self supervised learning to train tabular foundation models.\". There have been previous such approaches, some of which are cited by the authors in the main text, most notably TabPFN and its latest variant TabPFN v2. It would help the authors' case if they did *not* motivate their work using statements such as \"This direction is gaining popularity but is still in early stages with relatively few tabular-based TFMs developed\" (lines 50-51), and rather precisely state what their paper contributes relative to what is already out there. \n\nOther claims such as \"In this paper, we hypothesize that real tabular data contains much more information than heavily engineered synthetic tabular generators, thus allowing more straightforward improvements by scaling model and data size, which is supported by experiments in Section 4.5.\" lack empirical support. Moreover, I found critical statements such as \"However, they are completely reliant on synthetic data generators; ensuring that this mechanism captures the full diversity and nuances of real-world data is challenging, and making meaningful improvements to it is difficult.\" rather odd -- methods that perform close reliably while being trained only on synthetic data would in my opinion be considered advantageous as real data is hard to come by. \n\nSecond, I found quantitative comparisons between the different algorithms somewhat lacking. Averaging the rank (or the pairwise win-rate) over many datasets is a rather odd metric for comparison. The AUC is visible in Figure E, and it is clear that all models are quite close to each other. It is unclear if the authors have based their calculation of the average rank (in Table 1) based on a proper statistical comparison amongst the models. Can the authors please clarify? It would also be useful to plot the predictions from the model on some toy datasets to show that it generates reasonable predictions. \n\nA main claim of the paper is that using real data is more efficient. The data that supports this claim is presented in Figure 4b -- however, it is unclear how much and what synthetic data is used. This is important as the test loss for synthetic data has not yet saturated.  The real value of using synthetic data is that a potentially limitless amount of it can be generated. A useful comparison between real data and synthetic data would show that the synthetic data generators that are being used fundamentally do not capture statistical features of real datasets even with infinite data. Does your model trained on a small, real dataset saturate at a smaller test loss compared to when it is trained on a large amount of synthetic data?\n\nQUESTIONS:\nPlease see the Strengths and Weaknesses section for questions/suggestions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose TabDPT, an ICL tabular foundation model that is trained on real datasets compared to other ICL counterparts by using SSL as a form of data augmentation. The proposed method is compared against relevant baselines on classification and regression benchmarks, where the proposed method achieves the best results. The authors ablate the different decisions/components of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n- The paper is well written.\n- The proposed method is compared against relevant baselines on standard well-established benchmarks.\n- The implementation and weights are provided.\n- The authors provide ablations for the different design decisions.\n\n**Weaknesses**:\n\n- Figure 3 (b) reads as inference runtimes, however the authors include the train + hpo training time which is not fair for the non-meta learned baselines and confusing. I can also advocate that a non-meta learned model is only learned once per task and mostly then performs inference. I think it would be interesting to provide the inference times for all methods.\n- The authors use a retrieval based context for TabPFNv1 but not for TabPFNv2. I think it is fair to use the same procedure for TabPFNv2. It can be that the proposed method only surpasses TabPFNv2 only for large datasets.\n- The authors train the method on a large amount of real datasets and incorporate a considerable number of benchmarks, which makes it difficult to extensively evaluate the claims of the paper. \n\n**Typos:**\n\n- Line 329s suing -> using\n\nQUESTIONS:\n- Could the authors incorporate a retrieval based context for TabPFNv2 and analyze if the gains of TabDPT are over all datasets or only in larger ones. \n- Could the authors provide critical difference diagrams to observe whether the difference in results is statistically significant.\n- Moreover, the authors should correct Figure 3 b) in my perspective and provide the inference times for all methods, this would provide a greater insight for the domain practitioners.\n- Do the authors include the retrieval context preparation in the inference time for TabDPT?\n- It could be interesting to include TabM [1] in the results as a non-meta learned DL method, given that it claims state-of-the-art results and outperforms TabR.\n\n\n[1] Gorishniy, Y., Kotelnikov, A., & Babenko, A. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations.\n\nLIMITATIONS:\n- The authors have adequately addressed the limitations. Although I believe a sentence about the higher inference time would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TabDPT, an Open Tabular Foundation Model. By designing a Tabular Transformer Encoder, employing Self-Supervised Learning, and utilizing retrieval during both the Pre-Training and Inference stages, TabDPT achieves faster training, better generalization, and top performance on regression and classification benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths：\n1. The paper introduces the TabDPT, which demonstrates robust capabilities and has been validated across various downstream tasks.\n2. The paper conducted extensive experiments, including not only the main experiments but also comparisons between real and synthetic data, as well as experiments on scaling laws. These research findings contribute to the advancement of TFMs. \n3. The proposed Tabular Transformer Encoder, Self-Supervised Learning on Tabular Data, and Retrieval-Based Pre-Training exhibit high originality, with the rationale behind their design clearly explained. \n\n# Weakness:\n1. The paper's presentation is somewhat unclear. For instance, in the Introduction, it mentions, \"LLMs’ results also vary based on the specific prompt format and are sensitive to the order of the given examples, whereas tabular data is inherently unordered.\" However, it does not explain how their method addresses this issue, nor does it provide experimental validation. \n2. The comparison for the \"No retrieval\" baseline is incomplete, as it only contrasts with \"random subsampling\" and does not include a comparison against using all rows. \n3. The paper lacks details in some sections. For example, in line 180, it mentions \"leveraging retrieval during training batch construction\", but does not explain the method for encoding row data into the feature space. Similarly, in line 183, it states, \"These rows are partitioned into two groups\", but does not describe the partitioning method.\n\nQUESTIONS:\n1. When there is abundant row information, will treating \"each table row as a 'token'\" result in significant information loss? \n2. How is the value of K in Top-K determined in the paper? Will different K values have a significant impact on the experiments? Is it possible to use different K values during training and inference? \n3. If retrieval is not used during pre-training, and instead all rows are utilized, can the model learn to make more effective use of the rows? Consequently, when retrieval is applied during inference to replace all rows with more relevant rows, will this lead to better performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}