Title: Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: tabular foundation model

synthetic priors
Abstract: Since the seminal work of TabPFN, research on tabular foundation models (TFMs) based on in-context learning (ICL) has challenged long-standing paradigms in machine learning. Without seeing any real-world data, models pretrained on purely synthetic datasets generalize remarkably well across diverse datasets, often using only a moderate number of in-context examples. This shifts the focus in tabular machine learning from model architecture design to the design of synthetic datasets, or, more precisely, to the prior distributions that generate them. Yet the guiding principles for prior design remain poorly understood. This work marks the first attempt to address the gap. We systematically investigate and identify key properties of synthetic priors that allow pretrained TFMs to generalize well. Based on these insights, we introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world tabular data. Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks, with better sample efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Mitra, a mixture of priors to train tabular foundational models. The authors provide detailed analysis/ablations on the different components of the mixture, how to pick them and their impact. The proposed mixture of priors yields state-of-the-art results across three diverse benchmarks when used to train tabular foundation models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The paper is well written.\n- The authors consider diverse well-established benchmarks.\n- The proposed work achieves state-of-the-art results.\n\n**Weaknesses**:\n- The related work section can be improved, in particular the deep learning aspect of tabular methods. [1][2][3][4]\n- The code is not provided (with what I observed).\n- The novelty is limited given that prior works do use tree-based priors (TabForest, TabICL)\n- The authors provide a list of criteria on the different component selection of the mixture of priors, however, the results paint a slightly different picture (the RF prior fits the Diversity and Distinctive attributes before the ET prior, yet the performance does not improve substantially.)\n- The proposed work does not outperform the current state-of-the-art without fine-tuning. The main idea/motivation of TabFM is not training models on downstream tasks, while the authors do go back to fine-tuning. It would be great to point out the time it takes to fine-tune, although the comparison with AutoGluon and TabPFNv2 in Figure 3, does give an idea.\n\n**Typos:**\n\n- Line 227: all three datasets -> all three benchmarks\n\n**References**:\n\n[1] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T. (2021). Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.\n\n[2]  Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.\n\n[3] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34, 18932-18943.\n\n[4] Gorishniy, Y., Kotelnikov, A., & Babenko, A. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations.\n\nQUESTIONS:\n- Could the authors provide the code (I could not find it)\n- The authors mention that they sample 50% from the SCM prior and 50% from the TBP in Line 192. This is a bit vague, how is the 50% spread out for the TBP components?\n- The authors mention that a reason why Mitra benefits more from fine-tuning, is because it was trained with synthetic datasets with up to 16 features compared to TabPFNv2 which was trained with 100. I am curious on the reasoning behind the choice of training with 16 features and not 100?\n- Was HPO performed for the baselines? It would be great if TabM was included in the experimental protocol, it is a recent work which claims SOTA results in the tabular domain.\n\nLIMITATIONS:\nThe limitations are not part of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a tabular foundation model (TFM) named MITRA by attempting to carefully analyze the priors used for synthetic data generation to train the TFM. They show that carefully controlling the data distribution provides consistent gains in model performance on both tabular classification as well as regression tasks. The paper also dives into two different architectures, which attempt to highlight the model-agnostic nature of their priors.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The paper covers an important problem\n- The paper is well-written\n- Comprehensive list of experiments\n\n# Weaknesses\n- **Unclear reasons for performance gains with only specific tricks (finetuning and ensembling):** the model without finetuning and ensembling is significantly inferior to other approaches in most cases, making it hard to understand the reason for this performance gain with these tricks in place.\n- **Inconsistency in evaluation:** the '+e' and '+f' modes are selectively reported in many tables (e.g., Table 3), further masking the true results. I think it should be fine to highlight that this doesn't work in all cases, instead of trying to incorrectly convince the reader that it does.\n- **Missing training details:** the paper didn't discuss in any way the pretraining of the model including the dataset size, model parameters, training epochs, which are mentioned in passing towards the end (Fig 5). I would expect a better reading flow where the authors first specify how they pretrain the model and then talk about finetuning. It is just immediately assumed that we have a foundation model that we can evaluate, which is not ideal given that the paper is about the foundation model training.\n- **Hard to follow details in some areas:** the paper e.g., highlights that selection based on table 1 for the second best prior isn't optimal without mentioning anything else which makes it really confusing to the reader if they are missing something. They clarify this later in Table 6, but I would expect that the restructuring should make the flow of the paper much smoother to read in general.\n\nQUESTIONS:\n- Pretraining details in the main paper are missing, despite the main contribution being the development of a TFM. Can you please specify all the training details?\n- How is the prior importance color assigned in table 1?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith the emergence of tabular foundational models (TFMs) trained on vast collections of synthetic data, this work aims to systematically identify the best combination and type of synthetic priors for effective real-world data generalisation. The authors propose three qualities of an effective prior: standalone performance on real-world data, diversity (not overfitting on itself), and distinctiveness (not covered by other proposed priors). Guided by these criteria, they assemble curated sets of synthetic priors and train MITRA, a new TFM that performs comparably or outperforms state-of-the-art TFMs across a range of classification and regression benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. **Comprehensive, Systematic Evaluation** - The work involves a wide range of trained models on different combinations of priors to empirically sustain the claims regarding the effectiveness of each prior. The study is systematic and thoroughly conducted, with a wide range of ablations.\n2. **Robust Baseline Comparisons** - MITRA is benchmarked against a wide array of state-of-the-art baselines—including TabPFNv2—using multiple metrics, ensuring a fair and thorough evaluation.\n3. **Broad, Insightful Analysis** - Beyond addressing the choice of priors, the study examines sample efficiency, scaling laws (model and data size), and the effects of fine-tuning and ensembling. Taken together, these help understand what contributes to the success of the model the most.\n4. **Clear Motivation and Practical Impact** - The paper is well-written and directly addresses a relevant question in the light of the rise of TFMs: how to select synthetic data sources that genuinely improve real-world tabular model performance.\n\n**Weaknesses**\n1. **Incomplete model specification** - Although the experiments are very well explained and detailed, the model specification is not as thorough. Section B.3.1 gives some details about the transformer hyperparameters, but the authors do not give too much detail about:\n   - More detailed data pre-processing steps (e.g. are the categorical variables encoded using one-hot-encoding?),\n   - Test-set normalisation procedure, given that training uses dataset-specific min–max scaling on the support set.\n   - Exact form of the likelihood mentioned in L176 (distributional assumptions, factorisation across query points, etc.).\n2. **Reliance on Post-Training Techniques** - Performance of vanilla MITRA (no fine-tuning, no ensembling) underperforms many baselines, implying that selecting strong synthetic priors is not enough to reach SOTA. To get there, the choice of priors need to be combined with well-tuned post-training techniques. These in turn increase the average runtime of the model (which is at least 20x higher than for TabPFNv2 (+e)), as shown in Figure 14, while bringing only marginal performance improvements.\n3. **Marginal Gains vs. Compute Trade-Off** - While MITRA (+ef) is consistently the best, the 95\\% confidence intervals for Elo sometimes overlap (e.g. in Table 2), and the std of the metrics are quite high. Other considerations that are worth taking into account when developing TFMs are training costs, and runtime performance. Given that MITRA’s runtime is quite high (because of the need for fine-tuning as a consequence of maximum dimension of 16 in the training data), and given that the differences between models are sometimes small, some other models may offer better accuracy–efficiency trade-offs. \n4. **Minor figure formatting issues** - Figures 3, 4, and 5 are fairly small. Perhaps for Figure 5 at least the authors could include only the zoomed in versions?\n\nQUESTIONS:\n1. In L915 in A.2.1 you mention that you design the classification datasets in order to encourage a balanced distribution of labels across classes. Have you tested the model in cases where there is significant class imbalance too? Would that be out-of-distribution for the model given the design choice for the priors?\n2. Could you please give more details about the exact parameterisation of the training loss, as well as any unmentioned pre-processing techniques that might have been applied to the data?\n3. Could you please comment on how normalisation is performed on the test sets? What happens if some of the real-data shows some very large values that would be out-of-distribution if any min-max normalisation based on the training data were performed?\n4. It seems that the 2D attention is needed for SOTA performance. Up to how many nodes and rows do the authors think it is feasible to scale?\n\nLIMITATIONS:\nPerhaps one additional limitation not mentioned by the authors is the increased runtime of the best performing MITRA model (which requires fine-tuning on the training set of the target data). This might become limiting at high throughput rates.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach to select synthetic data priors for tabular foundation model pre-training. Specifically authors propose three criteria for prior selection - performance, diversity and distinctiveness - and design an empirical procedure to select priors that optimise these criteria. Empirical results show that when a foundation model is pre-trained with a proposed mixture better downstream generalisation results can be achieved.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-The paper is very well written an easy to follow. The proposed criteria for prior selection is innovative and conceptually sound.\n-The procedure to derive the mixture is novel although it does look computationally expensive.\n-Extensive empirical evaluation is provided on real-world datasets with detailed ablation study.\n\nWeaknesses:\n-I have a major concern with the empirical results. In the main result table TabPFNv2 base/e/ef would be the closest baseline to compare to for Mitra base/e/ef. However, looking at direct pairwise comparisons TabPFNv2 vs Mitra, TabPFNv2 (+e) vs Mitra (+e), and TabPFNv2 (+ef) vs Mitra (+ef), only the fine-tuned Mitra (+ef) performs better. This really calls into question the premise of the paper that the proposed procedure for prior selection leads to better downstream generalisation. The gap between non-finetuned TabPFNv2 and Mitra is consistent and quite large, particularly for the ensembled versions.\n\nQUESTIONS:\n-Do you have any additional explanation for why Mitra is not beating TabPFNv2 in non-finetuned settings? For current results it does look like TabPFNv2's synthetic data strategy is superior and leads to a better \"foundation\" model base.\n-The procedure to select prior mixture in Section 3.1 is quite expensive and requires training M models. I might have missed it in the appendix but do you have run time results for this procedure and stability across different restarts?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-05"}