Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models
Authors: Xiyuan Zhang
Danielle C. Maddix
Junming Yin
Nick Erickson
Abdul Fatir Ansari
Boran Han
Shuai Zhang
Leman Akoglu
Christos Faloutsos
Michael W. Mahoney
Cuixiong Hu
Huzefa Rangwala
George Karypis
Bernie Wang
Venue: NeurIPS 2025 poster ()
Keywords: tabular foundation model
synthetic priors
Primary area: deep_learning
TL;DR: We characterize key factors for good synthetic priors and build a SOTA tabular foundation model by pretraining on these priors.

Abstract: Since the seminal work of TabPFN, research on tabular foundation models (TFMs) based on in-context learning (ICL) has challenged long-standing paradigms in machine learning. Without seeing any real-world data, models pretrained on purely synthetic datasets generalize remarkably well across diverse datasets, often using only a moderate number of in-context examples. This shifts the focus in tabular machine learning from model architecture design to the design of synthetic datasets, or, more precisely, to the prior distributions that generate them. Yet the guiding principles for prior design remain poorly understood. This work marks the first attempt to address the gap. We systematically investigate and identify key properties of synthetic priors that allow pretrained TFMs to generalize well. Based on these insights, we introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world tabular data. Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks, with better sample efficiency.

MITRA: Mixed Synthetic Priors for Enhancing Tabular Foundation Models
Xiyuan Zhang Amazon
Danielle C. Maddix
Junming Yin Amazon
Nick Erickson Amazon
Amazon
Abdul Fatir Ansari
Boran Han
Shuai Zhang
Leman Akoglu Amazon and CMU
Amazon
Amazon
Amazon
Christos Faloutsos Amazon and CMU
Michael W. Mahoney
Cuixiong Hu Amazon
Huzefa Rangwala Amazon
Amazon
George Karypis
Bernie Wang
Amazon
Amazon
Abstract
Since the seminal work of TabPFN (16. research tabular foundation models (TFMs) based on in-context learning (ICL.) has challenged long-standing paradigms in machine learning. Without seeing any real-world data, models pretrained purely synthetic datasets generalize remarkably well across diverse datasets, often using only moderate number of in-context examples. This shifts the focus in tabular machine learning from model architecture design to the design of synthetic datasets, or, more precisely, the prior distributions that generate them. Yet the guiding principles for prior design remain poorly understood. This work marks the first attempt to address the gap. We systematically investigate and identify key properties of synthetic priors that allow pretrained TFMs to generalize well. Based on these insights, we introduce MITRA TFM trained on curated mixture of synthetic priors selected for their diversity. distinctiveness, and performance on real-world tabular data. MITRA consistently outperforms state-of-the -fethh--rt TFMs, such as TabPFNy2 [17] and TabICL (29], across both classification and regression benchmarks, with better sample efficiency.
Introduction
Tabular data lie at the core of many real-world applications, including healthcare, finance, e-commerce, and the sciences (32]. Predictive modeling tabular data is central to statistical data analysis and underpins decision-making systems across these diverse domains
14]. Tree-based models, such
random forests, gradient boosting, and ensemble methods 3 7. have long dominated tabular predictions, due to their strong empirical performance and ease of use. However, these methods are typically tailored individual datasets, and they exhibit limited ability transfer across different distributions. Despite advances in transfer learning, truly general-purpor approach to tabular prediction has remained elusive, until the introduction of TabPFN [16).
Inspired by the success of large language models (LLMs), TabPEN and its follow-up works have introduced the notion of tabular foundation models (TFMs) based on in-context learning (ICL) [16] 17 29 5]. These models are pretrained on synthetic tabular tasks, and they make predictions on
'We released both classifier (autogluon/mitra classsse and regressor (autogllonm/iirrerrrrssss model weights HuggingFace.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

How to Identify Good Priors?
Testing
Mixture of Synthetic Priors
Generulizahiltyy Matrix
Trees
Real
Performance Vector
SCM X,
Gradient Boosting
Forest Trees
Standalone Performance on Real Duta
X, y
X2
X y
X
SCM
Training Data
Diversity: Do Not Overfit on Itseif
Onl-
Trees
Viler
Valee
Distinctiv
Mixture Improves
Not Cavered Other Priers
Mix
Single Priors!
Fit
Fit
Fit
Figure 1: Mixture of priors in MITRA, Generalizability Matrix G and Performance Vector P. We consider three factors for good priors: (1) the performance of a pretrained TFM using data generated solely from that prior on real tabular data (Point and P); (2) its diversity and (3) distinctiveness when included in a mixture (Points 2 and 3, captured by the diagonal and off-diagonal entries of G). This leads to our mixture comprising SCM, gradient boosting, random forest and decision tree priors.
real downstream tasks by conditioning on labeled training samples from the downstream tasks as in-context examples. Synthetic data, generated on-the-fly during pretraining, provides broad task coverage and enables adaptation, without the need for large amounts of real-world downstream data.
While most previous TFM efforts focus on architectural innovations 24
we advocate that greater attention should instead be on the design of the data priors- the distributions used to generate the synthetic datasets. Existing work typically relies on fixed or heuristic data priors [29 16 17 3).
29
leaving fundamental questions open. For example: What makes a synthetic data prior effective? How should one construct a mixture of priors for better generalization?
In this paper, we investigate these questions, with the goal of identifying key properties of synthetic priors used for pretraining TFMs. Our findings sharpen vague rule of thumb that "diversity of the prior is important." " We show that the effectiveness of a synthetic prior depends on: (1) the performance of a TEM pretrained solely on data generated from that prior, when evaluated on real tabular data; (2) its diversiry, i.e., how difficult it is for a TFM pretrained on this prior to overfit on its own distribution; and (3) distinctiveness within a mixture of priors, i.e., how hard it is for data generated from this prior to be predicted by TFMs pretrained on other priors. See Figure I for simplified illustration.
Based on our insights, we construct a mixture of synthetic priors that consists of structural causal models (SCM) [I6] and tree-based priors (TBP), including gradient boosting, random forest, decision tree, and extra tree models. We choose SCMs because we show that they are diverse and achieve the best standalone performance on real tabular datasets. We choose TBPs because we identify that TFMs pretrained with data from SCMs do not always generalize well to all types of data generated from TBPs, showing the distinctiveness property of TBPs
Our mixture of priors enables effective coverage the diverse distributions in real-world tabular data. Notably, our priors are model-agnostic, and they consistently demonstrate performance improvement for both row-based 1D) attention [66] 2 and more advanced clement-based 2D attention architec- tures 17 (5 Building
the 2D attention architecture, we propose MITRA, a TFM pretrained with our mixture of priors that obtains state-of-the-art (SOTA) results. MITRA outperforms existing TFMs, e.g., TabPFNv2 [7], TabICL [29), and other strong baselines, on both classification and regression tasks. Additionally, MITRA models pretrained with our prior mixture demonstrate better sample efficiency, i.e., they consistently have stronger performance with fewer in-context examples. This highlights the benefits of our principled prior mixture analysis.
We summarize major contributions follows:
Characterize Key Factors for Good Priors. We conduct the first principled analysis of how to combine various synthetic data priors for TFM pretraining, and we identify three key factors for good priors: performance, diversity and distinctiveness.
Construct Effective Mixture of Priors. Following our analysis, we construct novel and diverse mixture of synthetic priors that is effective and model-agnostic.
Build SOTA TFM with Prior Mixture. We propose MITRA, TFM pretrained using our mixture
of priors. MITRA sets a new SOTA on both classification and regression tasks. MITRA outperforms strong baselines across three major benchmarks -enbbearks-TTaRRee [30), TabZilla (25) and AMLB [II] and it demonstrates better sample efficiency, given fewer in-context examples.
2

---

2 Related Work
Traditional and Deep Learning-Basee Baaeingg-BBedddd Tabular Models. Historically, the tabular domain has been mainly dominated by traditional statistical methods, most notably gradient boosting (GB) decision trees, e.g., XGBoost []. LightGBM (20), and CatBoost (28). These methods are widely adopted in practice due their strong performance, robustness, and interpretability. To further improve generalization and automation, ensemble-based systems (e.g.. AutoGluon 77] combine multiple base learners and automatically optimize hyperparameters and stacking strategies. More recently, deep bearning-based approaches have been proposed to model complex and rich interactions in tabular For example, RealMLP [8] introduces optimized multilayer perceptron (MLP) architecture, tuned over a broad set of mneta-benchmrr datasets. While both traditional statistical methods and neural networks remain competitive on many real-world benchmarks, they require retraining from scratch for each new dataset, and they struggle to generalize across different distributions. This need for repeated per-data retraining poses scalability challenges and limits model reuse in real-world applications.
data [31 19 13 (2).
TFMs: Semantically- Rich Models. Recent work has explored adapting LLMs to structured data by serializing tables into text. TabLLM [15], GTL (34), and Tabula [TO] enable zero-shot or few-shot inference by formatting rows and tasks into promptable inputs. TP-BERTa (6) propose a pretrained language model tailored for tabular prediction tasks, with relative magnitude tokenization and intra- feature attention mechanism. Beyond text serialization, several approaches focus on non-text non-textual pretraining by leveraging cross-table techniques. These include independent featurizers [39], modular encodings with multi-task masked reconstruction [37] and prompt masked table modeling (38). CARTE [2] leverages pretraining on knowledge graphs and column-level metadata facilitate downstream tabular tasks. These methods often rely heavily on semantic metadata (e.g.. column names or textual descriptions). curated schemas, or large-scale real corpora. This dependence limits their applicability in settings where this auxiliary information is unavailable or unreliable. and it may require expensive inference costs in practical deployments due to their reliance on L.LMs.
TFMs: ICL-Based Models. A complementary line of work frames tabular prediction an ICL task, where models are pretrained on synthetic or real datasets and downstream datasets are used as in-context examples. These efforts primarily focus on two directions: (1) designing better data priors; and (2) improving model architectures. In the first direction, TabPFN [6) introduces this paradigm by pretraining a Transformer 33 SCM-generated synthetic data simulate ICI. for tabular
classification problems. TabDPT 23 extends this approach by incorporating real datasets during pretraining. Several subsequent
TabForestPFN (2), Attic [5], and TabICL (29 combine tree-based priors with SCMs for synthetic data generation. Specifically, TabForestPFN and Attic mix SCMs with decision trees, and TabICL combines SCMs with XGBoost. These approaches adopt tree-based priors in a heuristic manner, without explaining why incorporating these priors is beneficial. In the second direction, Attic (5] introduces an dlement-based attention mechanism, which treats each cell (rather than every row or column) in a table as a separate token in the Transformer. TabPFNv2 17 adopts a similar element-wise Transformer architecture, and further improves scalability and generalization by refining the synthetic prior distributions. More recently, TabICI. [29] proposes two-stage architecture to first build xeedddimensional embeddings of rows, followed by an efficient attention mechanism for ICL. Recent efforts have also explored how to scale ICL through
compressing prompts, selecting informative contexts 2 35 24] and hypernetworks 226 1
3 MITRA: TFM Pretrained on Mixture of Priors
In this section, we describe how we characterize effective priors with the following three criteria: (1) strong performance on real datasets; (2) diversity; and (3) distinctiveness within the mixture. These characteristics together lead the development of new SOTA TFM, MITRA.
3.1 Data-Generating Priors
To pretrain a TFM on purely synthetic data in the supervised tabular learning setting, a data-generating prior G; takes as input uniformly randomly generated hyper-parameters, e.g., feature size, number of samples, class count (for classification tasks), and categorical feature count; and it outputs a dataset
D {(xli) yn (i) > This dataset consists of N, feature- pairs, where x!!) E Rd denotes
d-dimensional feature vector with continuous or categorical attributes; and yn E ) C Z denotes the (+) corresponding target label for a classification task, yn e y C IR the target value for regression
3

---

Table 1: Three factors of prior importance. Each entry Gúj in the Generalizability Matrix represents the AUC of TFM pretained on data generated from G and evaluated on data from Gj. Each element P. in the Performance Vector corresponds to the AUC of a TFM pretained on data from 9j and evaluated on a real-world dataset. We use color gradient to visually indicate the relative quality of each prior. The best/worst- perfornnng priors are shown in the darkest green/red.
Train
Test DT
SCM
ET
GB
RF
DSRF
TabRepo
Prior Importinn inne
SCM
0.830 (0.201)
go.6)))))))
0.892
0.876
0.708pu140
0,960uess
0.857
High
ET
0.805
0.871 (0.115)
0.938
0,948
0.76610130
0.975/004)
0.847 (0.140)
GB
0.805
(.80
0 (0004)
0,916m.080)
0.745
0.978
0.843 (0.147)
DT
0.866
0.940)anss)
0.951 (un5s))
m.768
0.974
0.839 (0.148)
Low
0.804)
0.8 (o.14)
0.91 aa.ssm
g.905
0.761 (0.129)
(.965
0.831 (0.147)
DSRF
0.8 41.000
0.944g
0.937 (0
(o
0.984 (0.034)
0.849 (0.144)
Generalizabillty Matrix G
task. The entire dataset can be represented as a woodimmeniional matrix/table D( E (Nx((d))) (See Appendix A.1 for details problem settings and preliminaries.)
In MITRA, we propose to use a mixture of M data-generating priors (g.) Concretely, we include the following types of priors: structural causal models (SCMs); and tree-based priors (TBPs).
SCM. The data-generating prior, GSCM, which was originally introduced in the pretraining of TabPFN [16], is capable of capturing causal relationships among columns observed in tabular data. Moreover, SCMs model both feature dependencies and the conditional distribution p(y|x). To sample a dataset D from GsCM, directed acyclic graph (DAG) is first randomly constructed, after which the features and target y are generated in a sequential manner, following the conditional dependencies defined by the DAG structure.
TBP. The data-generating tree-based priors, GTBP, consist of trees and ensembles of trees, which are known for their strong predictive performance on tabular tasks and which are commonly used to model decision boundaries in tabular data. TBPs primarily focus on modeling p(y|x) x) using complex threshold-based splits, with ensemble trees helping to smooth the resulting axis-aligned decision boundaries. In this work, we consider GDT. GEr. GGB, and GRF. where DT, ET, GB, RF refer to decision tree, extra tree, gradient boosting, and random forest, respectively. To use these models as data generators, they are first fit on a synthetically generated training dataset, after which features x are sampled from a simple distribution (e.g.. multivariate standard normal) and targets y are drawn according to the learned conditional distribution p(y x). (See Appendix A.2. for additional details on the data generation process.) We group these priors together, and we reter to them as "indirectly sampled" priors. addition, we introduce custom "directly sampled" RF prior, GDSRF, that does not require model fitting. Instead, it directly constructs random conditional distributions p(v x) by sampling random split indices and thresholds. (See Appendix(AA2. for details on each of the priors.)
3.2 Prior Mixture Promoting Diversity and Distinctiveness
Here, we provide an in-depth study on characteristies of effective mixture g' = (C)) of M' data-generating priors from G {G. } where M' < M. We first pretrain M models on data from each candidate prior, and we evaluate them across priors to construct generalizability matrix G E MMM, where the rows of G denote the model pretrained on draws from 9,, and the columns denote the test data generated on draws from Gj. with Gi denoting the metric value. We also evaluate these M models on real-wort wooldd datasets to form a performance vector P E RM, where P. denotes the performance of model pretrained on data from G, and evaluated on these real datasets. We find three key factors when characterizing good priors: (1) Performance, quantified by higher value of P.; (2) Diversity, quantified by a lower diagonal value G. which indicates greater difficulty in overfitting to the same prior; (3) Distinctiveness, quantified by a lower off-diagonal value G.y. which shows how well a model pretrained on G. performs on data from G,. More specifically, given a current mixture G', the maximum of the off-diagonal G.j in the j column for such that G E G' is measure of the distinctiveness of Gj. Then, adding the prior G with the smallest maximum, i.e., minj S maxişis M, EG Gıj. increases the coverage of the mixture. Ultimately, prior importance balances performance and diversity.

---

Pre-training: Mixed Synthetic Priors
Inference: Real Tabular Data
Split
Support
X
Into
X₂
P
Query
Model Agnostic
Mitra (2D)
Mitra (ID)
across
across Cols
MLP
Attn
MLP
Entry
Row
Mask
Figure 2: MITRA Pipeline: Model agnostic visualized with either 1D 2D attention.
Table provides illustrative example using the AUC metric of how these three factors help explain the prior importance findings in our ablation study in Table (below; see Appendix C for similar findings in other aggregated metrics). We see that GScM is of high quality due to both diversity (low diagonal value G,;) and strong performance on real datasets P. Interestingly, while GDSRF ranks second in performance
P. the third best performing prior GET shows higher quality based its ability to increase the diversity and distinctiveness, as validated in Table 6 For diversity, the diagonal of GDSRE is significantly higher than the diagonal of GET (0.984 0.871). For distinctiveness, the off oif-diagnnals in the SCM row G measure how much unique information is added from the next prior. We see that the model pretrained with GSCM predicts test data drawn from GDSRF significantly better than that from GEr (0.960 vs. 0.751), showing that GEr has higher distinctiveness.
3.3 Pretraining TEM on Prior Mixture
We use the insights from the prior subsection to assign weights to each prior in the mixture g' ( subject to Σi=1 Wi; = 1. During pretraining, we sample generators Gj proportional to wi. and we generate synthetic datasets for pretraining. Given a sampled table D from a generator g in the mixture G', randomly sub-sample entries the support set in-context examples and g entries as the query samples. We then optimize the likelihood over the masked query labels:
[ = E [log Po (Very, :ry, XSsup,sup suup, [sup,,ssn Xury, :pry, }]- Algorithm in Appendix A.3 summarizes synthetic prior generation procedure. We present more pretraining details in Appendix [B..22 Figure lllustratss the overall MITRA model pipeline with two popular architectures: one-dimensional row-wise attention [16 2) and two-dimensional element- wise attention [7]. Our priors model- agnostic, and we demonstrate their effectiveness across both architectures in the next section.
Empirical Results
In this section, we show that MITRA achieves SOTA performance on both classification and regression tasks (Section 4.2). We demonstrate that MITRA model agnostic and consistently improves the performance with both ID attention (MITRA ID) and 2D attention architectures (Section 4.3) Furthermore, we highlight MITRA'S better sample efficiency (Section 4.4). strong performance when combined with advanced ensembling techniques (Section 4.5), and strong fine-tuning performance (Section[4.6) We also conduct an ablation study to quantify the importance of each prior (Section 4.7) which supports our findings in Section 3.2) Finally, analyze the scaling law with respect to both model size and synthetic dataset size (Section 4.8). See Appendix B for additional experimental setting details and Appendix C for additional experimental results.
4.1 Experimental Settings
Datasets. For the classification task, we compare MITRA on three established 10-fold benchmarks:
TabRepo [30]: Tabzilla 255 and AutoML benchmarks (.]. We additionally evaluate on concurrent
8 in Appendix C.4 For the regression task, we compare on the 10-fold TabRepo 33) benchmark. We evaluate both MITRA and its variant MITRA ID that is trained on ID attention model with the same mixture of priors. To compare with ID models, e.g., TabPFN that support features up to 100, and 2D models, e.g., TabPFNv2 that support features up to 500, we evaluate on both small-feature and larturfeature benchmarks. For the small-feature benchmark, we use TabRepo classification datasets, TabZilla classification datasets, and 10 TabRepo
benchmark TabArena
regression datasets, that have up 3,000 rows and 100 features following TabPFN [16). To evaluate
5

---

Table 2: MITRA wins all three classification benchmarks (We show the overall merged results
and detail the separate benchmark results in Appendix [.4]] i.e., Table 13
(TabRepo), Table [4] +e means ensembling
(TabZilla), Table Table (AMLB) due to space limits). Winner/runner-up in in ICL, and +f means fine-tuning. The 95% confidence interval is shown in parentheses for the Elo.
Aggregated metrics show mean and std (shown in parentheses) of the corresponding metric.
Ranking Metries
Aggregated Metrics
Model
Avg. Rank
Elo
Winrate
RAce
CA
AUC
ACC
CE
MITRA (+ef)
(14-4)
0.69
0.905 (.124)
0.858 (0.143)
0.328 jo.
Attic (+ef)
1128 (1444)
0.81
21.7
0.903 (0.125)
(L113)
(0.31))
TabPPNv2 (+c)
(4074
0.8
23.3
(a13)
0.856%.444
g.338
TabPFNv2 (+ef)
8.6
1085
0.62
0.76
25.3
0.897
0.846
0.363
TabiCL (+e)
9.5
0.58
0.75
30.9
0.889
0,836
0.367
MITRA (+e)
9.7
1935
0.57
0.73
31.2
0.896
0. 0,847 1485
0.36v0.25)
TabPFNv2
9.8
1043
0.56
0.73
29.2
0.891
0.846 147)
0.3522g.114
Attic (tc)
(+43)
0.55
0.73
31.8
(.89
0,848 148)
0.364.o.322
TabICI
10.6
101 (14444
0.52
0.7
33.5
0. 1.884..14 141:
0.8 8822..1
0.374
MITRA
10.6
1015 (4464)
0.52
0.69
32,9
0. 19:
0 (.841
0. 0.368
MITRA (+f
11.2
-92.+41-1
0.49
0.68
34.5
0.842
0.367
Attic
11.2
988 +3-4)
0.68
35.2
0. 0.884
0.8340.550
0.376
CatBoost
11.3
0.48
0.67
35.7
0.888
0.837
0.375
TahForestPPN (+f)
35.3
0. 0.8866 196)
0.84g 148)
0.377
RealML.P P
12.2
95R
0.44
0.62
35.4
0.878
0.827
0.4122g.354)
XGBoost
13.1
926,444)
0.58
39,5
0. 0,883 1330
0.388
LightGBM
13.4
917,+1-D
0.38
0.56
39.8
0.876 141;
O 8299..155
0,3933
Random Forest
13.7
003
0.36
0.54
44.1
0.874
0.822..525
0.471 (o.423)
MLP
13.7
903
0.36
0.51
40.1
0,869 14y;
0.82
0.414,a.
MITRA
14.0
894
0.35
0.55
42.5
0.869 14:
0.815
0.414
TahForesiPFN
14.3
883
0.34
0.52
42.6
0.8644
0.8 14..1167
0.414
on feattre benchmarks, we use 29 classification datasets from AMLB benchmark with up to 10,000 rows and 500 features. This is consistent with the evaluation protocol of TabPPFN22 11 We provide the dataset IDs in Appendix B.T
Baselines. We compare MITRA with both SOTA TFMs (TabPFNv2 17.. TabICL 29 Attic [5].
TabPFN [16), TabForestPFN
(2), and competitive classical and neural baselines requiring dataset-
specific tuning (RealMLP [8]. AutoGluon 77. LightGBM 200, XGBoost [3, CatBoost (28).
MLP 7). For the latter, we use their bagged version implemented in AutoGluon 1.3,
Metrics. For classification tasks, we report aggregated metrics including AUC-ROC (AUC), accuracy (ACC) and cross-entropy (CE). For regression tasks, we report R², root mean squared error (RMSE), and mean absolute error (MAE). Aggregated metries can be disproportionately influenced by a small number of datasets with extreme performance, as noted in prior work (30. To address this, we complement these metrics with more robust and comprehensive rank-based metrics that better capture relative performance across datascts: average rank, Elo (6), winrate, rescaled accuracy (RAcc), and champion delta (CA). We provide the definitions of these metrics in Appendix B.2]
Evaluation Protocols. For TFMs including MITRA, we evaluate the following three settings: (I) in-context learning (ICL) performance, (2) ICL with ensembling techniques of feature shuffling, class order shuffling and random feature transformations [17 (2), denoted as "te" in the following sections; and (3), fine-tuning that continues training the model on the training set of the target data, denoted as "+f" in the following sections. We use "bagging" to describe fine-tuning with bagging ensemble, and we show its advanced ensemble performance in Section 4.5
4.2 MITRA achieves SOTA classification and regression performance
Classification. We merge the three classification benchmarks and keep unique set of 137 datasets report an overall ranking and aggregated performance Table Detailed method configurations and individual benchmark results on TabRepo (Table 13). TabZilla (Table 14), and AMLB (Table 15) are provided in Appendix CC4) MITRA wins in the overall results and across all three benchmarks with varying feature dimensionality or sample size, and it consistently achieves the best performance with fine-tuning and ensembling, across both ranking-
and aggregated metrics. Notably, the ICL performance of MITRA closely matches that of TabPFNv2, despite being pretrained on a maximum of 16 features (one-tenth of the maximum pretraining features TabPFNv2), which showcases the strong generalizability of our priors.

---

Table 3: MITRA also demonstrates better performance with 1D attention model, showing that priors are model-agnostic. Winner/runner-up in shades of green / The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics mean and std (shown in parentheses) of the corresponding metric.
Ranking Metrics
Aggregated Metrics
Model
Avg. Rank
Elo
Winrate
RAce 1
C
4
AUC:
ACC
CE J
MITRA LD (+0
3.0
1057
0.6
0.68
16.5
0.886
0.835 (0.15))
0.38
TabForestPFN (+f)
3.2
1038 (17-4)
0.56
0.64
18.8
0:878 (0142))
0.832 (0.154)
0.391 (0.337)
TabPFN (+c)
3.4
1012
0.52
0.6
24.4
0. 0.8620.155)
0.809
0.418
MITRA ID
972
0.45
0.53
27.2
(.865
0.417
TapPEN
3.8
(70,
0.45
0.53
26.3
0. 0,86
0.808
0.426
TabForestPEN
3,9
0.42
0.49
27.7
0.859
0.81
0.419)a.349
Table 4: MITRA wins on TabRepo 10-fold regression benchmark. Winner/runner-up in shades of +e means adding ensemble in ICI., and +f means adding fine-tuning.
green
Ranking Metries
Aggregated Metries
Model
Avg. Rank I
Elo
Winrate
CA
1
MAE J
MITRA (+ef)
2020020
(a 506
1401.274,200 (756000
15 sm
TabPFNv2 (te)
1000 (-150 1e)
0.63
0.22
12.7
0.61.5pmmm
2374.55
(7195-1))
1304,54 pa00
RealMPP
0.56
16
0.627
2424.34
(424.34
57,00 aty
CatBoost
5,8
0.56
0.69
15.7
0.629 (1300)
2465.00
(115009
1444. 57,e
(teFF)
(88-188)
15.5
(.600
(7513.58)
1295. 36,00 25)
MITIA (+0)
6.4
0,51
0.63
19.5
g.604)33111
2469. 1 1222022000
1372 43 (3,40
TabPFNv2
6.4
10030
0.51
0.59
15.9
0.501
24366 73:666667700.8
1337 64 (406373)
19-15)
18.2
0.625
2572. p025
1573.52, 5244522400000 09)
LightGBM
6.8
984
0.47
0.61
20.3
(.629
2665.
90
157 68.272220
MITRA
7.1
963, 100
0..4
0.59
20.6
a.177
2465
02 70y
1387. 42244442
MI.P
8.1
904
200
0.36
0.5
22.3
0.595
2778.
1557.7 Lurz0
Random Forest
9.4
224-22/111
0.23
0.32
27.9
p.585
2797. 1797.35//033
1705.43,310..0
Regression. We evaluate on TabRepo regression datasets (Table MITRA again demonstrates the best performance across the various metrics, showing that our mixture of prior is task-agnostic.
4.3 MITRA priors model agnostic
As shown in Table 3) when pretrained with the same mixture of priors, MITRA ID also outperforms other ID attention basedd counterparts, e.g.. TabPEN and TabForestPFN, which rely on less diverse priors. This highlights that mixture of priors is model-agnostic and can consistently enhance performance across different architectures. Note that TabForestPFN does not offer native ensemble logic, and TabPFN does not offer native fine-tuning logic. We report results under the capabilities available in the respective baselines to ensure a fair comparison.
4.4 MITRA is more sample efficient
We compare the sample efficiency of MITRA against leading TFMs, i.e., TabPFNy2 and TabICL, in Table 5) We down-sample the number of ICL examples of TabRepo classification benchmark to 10%, 25%, 50%, and 75% of the original size, and MITRA consistently achieves better performance with ensemble and fine-tuning. We demonstrate in Appendix C.2 that such improvement can be attributed to the increased diversity of priors in the mixture, which enhances the model's 's ability to generalize from limited data.
4.5 MITRA shows the best performance with advanced ensembling techniques
To further boost performance, we implement a bagging-basee ensemble for MITRA, denoted as MITRA (bagging). Specifically, we fine-tune separate MITRA instance for each fold of an 8-fold (stratified) cross-validation ensemble (22). and we aggregate their predictions via uniform averaging test time. This allows MITRA benefit from both data-level diversity and model-level robustness. While cross-validation ensembles are widely used for achieving top performance with classical tabular models 77, to the best of our knowledge, our work the first to demonstrate cross-validation ensembles for fine-tuned TFMs. We compare MITRA A (bagging) against the strongest ensemble methods reported in previous works on TabRepo: the Post-Hoc Ensemble (PHE) of TabPFNv2, and the AutoGluon 1.3 best quality preset, which combines diverse set of classical and neural tabular
7

---

Table 5: MITRA shows better sample efficiency. Winner in shades of green
Ranking Metries
Aggregated Metrics
Model
Avg. Rank
Elo
Winrate
RAce
C.A.
AUC 1
ACC
CE
MITRA (ik=1)
4.2
1234 (48--)
0.77
0.88
13.7
0.882 in 12125)
0.84 10
0.36 (0.361)
TahPFNy2 (ds=1)
4.5
(217))744)
0.75
0.87
16.3
0.879)
0.838
0.372 360)
TabiCL (ds=1)
5.6
1144
0.67
0.82
25.4
0.858
0.816
0.406
MITRA (ds=0,75)
5.3
1163 (4807))
0.69
0.84
20.3
0.876 a
0.835 (oues))
0.371 (0.06))
TabPFNw2 (ds=0.75)
5.5
(155
0.68
0.83
21.9
0.872
0u832
0. 0.384...0 365)
TabiCL (ds=0.75)
6.7
108 (4200)
0.59
0.77
29.6
0.852 146)
0.419
MITRA (ds=0.5)
6.7
(-207)
0.59
28.3
0 (1.922)
TabPPNNv (ds=0..)
6.9
1072
0,58
0.77
29.4
0.86
0.401m
TabICI.(dss:.....
7.9
1012
0.72
34,9
0.844m. 1463
g.799
0 0.4
MITRA (ds=0.25)
9.4
(1777)
0.4
41.1
(0.14))
(0.174)
0.43 (ounTy
TabPFNv2 (ds=0.25)
9.6
9124-9-71
0.39
0.62
42.8
0.843
O.
0 0.44
(55
10.5
855 (82-3)
0.56
46.3
1.819
0.776
0.487
MITRA (ds=0.1)
12.1
740 (-009))
0.21
0.36
54.1
0.808 1813
0.771 18)
0,515 (0,77)
TabPFNv2 (ds=0.1)
12.4
119
0.19
0.33
55.1
0.764
0.519 0.519
TabICL IabIC((b=:0.)
12.7
689
0.16
0.24
56.8
0.777
0.742
0.561, 1994)
TaXPFNY2PHE
Bagging
Champ à
AUC
60
Figure 3: Mitra (bagging) shows the best performance compared with TabPFNv2 PHE, AutoGluon best-quality and Mitra (+ef) across 300- to 3600-second budgets on 1-fold TabRepo benchmark.
models. As shown in Figure across different training budgets from 300, 600, 900 and 3600 seconds per dataset, MITRA (bagging) consistently outperforms both TabPFNy2 PHE and the AutoGluon ensemble, demonstrating the best performance with advanced ensembling techniques in the most competitive settings. Table reporrts more details on ranking and aggregated performance over the unified classification benchmark.
4.6 MITRA shows the best fine-tuning performance
We compare the combined fine-tuning and ensemble performance of MITRA with TabPFNv2, TabICL, and Attic on TabRepo as function of the number estimators in the ensemble. As shown in Figure MITRA consistently shows better fine-tuning performance across various ensemble sizes. Fine-tuning and ensembling of TabPFNv2 barely improves their ensemble-alone performance A likely reason for the strong gains from fine-tuning in MITRA is that it is pretrained with maximum of 16 input features, so that adapting to downstream datasets with larger feature spaces provides substantial benefits. We also hypothesize that more diverse priors in MITRA contribute to its fine-tuning effectiveness, as they enable the model to generalize from a broader set of inductive biases, making the model more generalizable and adaptable to task-specific fine-tuning.
Winrata
RACC
Champ
AUC
0.6
22.5
0.55
0.69
D/425
0.600
850
0.5
17.
087
0.19
0.375
04
2
2
Figure 4: MITRA shows significantly better fine-tuning performance across ensemble sizes.
4.7 Prior Importance Ablation Study
We systematically study the importance of each prior in the MITRA prior mixture by iteratively adding the best-performing prior at each step. (See Appendix C.3 for details.) Table 6 presents several key findings that support and extend the analysis in Table I from Section 3.2 Both diversity and performance on real datasets are important. Among all priors, data generated from
We verified the conrectness of the TabPFNv2 fine-tur tue-ttning procedure via communication with its authors.
8

---

Table 6: TabRepo 10-fold classification ablation study the effect of each prior in the mixture consisting of 50% SCM and 50% TBPs with equal weighting. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric. Note that SCM + DT is a variant of Attic [S
Model
Ranking Metrics
Aggreguted Metries
Arg. Rank
Elo f
Winrate
RAec
CA
A
1CC
CE
RF
15.9
815
0.25
0.38
42.8
0.83
0.782
0 0.52 lqueso
840
42.0
0.839uu4:
0.791a
0u52Nuss
GB
15.0
855
0.3
0.52
38.3
0.8431m
0,796 177
0. a.
ET
14.3
883 803
0.36
0.58
36.2
0.8770
0.803g sen
0 0.500 toe
DSRF
13.9
899
0.36
0.58
34.5
0.84%
0.799
0 0.4730
SCM
1000
0.68
25.5
0.85%
0.81
0.4 16ymmo
SCM RF
1005
0.858pa43
0.816
SCM DT
10.3
0.53
0.23
24.5
0.86
0.81 sen
SCM - GB
10.2
1028 en-33)
0.54
0.75
22.3
0.899
0.817
0.4 loume
SCM DSRF
9.4
1058
0.58
0.75
21.3
0.862m144
0.81%
0.40aas
SCM + ET
9.3
1063 (11)))
0.59
0.75
20.7
1386
0,823pp 194)
0 3700 170
SCM - ET + DT
10.2
1028
0.54
0.74
22.4
0.869
0,824 1851
0 0.00or
SCM - ET RF
9.9
1012
0.56
0.75
22.4
0.870 (2120
0.825,
0 m.10m
SCM ET DSRF
9.6
(050
0.57
0.74
22.1
0.8600
0.815 172)
0,40mmmm
SCM + ET + GB
8,9
1076 (1:-))
0.6
0.77
20.2
0 ja
0.827 ma14)
0.396 (2:300
ET + DSRF
22.2
0.8600ue4
0.817
SCM ET + GB
9.6
1050
0.57
0.75
22.2
0.868
0. 0.823 187)
0 au
SCM GB
9.4
0.58
0:76
21.1
0.870 sa (a.
0 0.826 auset
0. 1.997 0.399
SCM ET + GB DT + DSRP
9.5
0.57
0.75
22.9
0.859pu440
0.81 1721
SCM +
DT
(063))
0.59
0.76
21.5
0.860
0,822 1871
SCM ET DT RF DSRF (MITRA)
(062
0.59
21.4
0.868
0.822 193)
0.403
GSCM shows the highest importance, aligning with its high entry in the performance vector and low diagonal in Table In contrast, despite being the second-best best stand-alon aand- prior, 9DSRE contributes the least when added to the mixture, due to its high diagonal (0.984) and strong overlap with other priors (off-diagonals > 0.96). On the other hand, while GRF exhibits low overlap (lowest diagonal of 0.761 and off-diagonals in [0.708, 0.768)), it also shows the worst performance as indicated by the performance vector, thus explaining why it is not as beneficial in the mixture. (2) Our mixture of priors promotes complementary strengths and improves generalization. We observe that combining GSCM with every free-based prior improves over either alone, which emphasizes the importance of a prior mixture. In particular, we see that combining GET with GSCM significantly boosts performance, yielding an Elo improvement of 63. This supports the findings in Table that adding GET into the mixture is effective since is both diverse, as measured by its lower diagonal (0.871), and distinctive, as measured by its low off- off-diagonal in the SCM row (0.751) Similarly, we see that adding GGB to the mixture further improves the performance. Adding the remaining priors GDT. GDSRF. GRF leads to a few configurations with similarly good performance on real datasets. This aligns with our findings in Table I that they are less important due to either lower performance on real datasets or higher diagonal or off-diagona, values. We choose to include these priors in the final mixture to represent a more complete family of tree priors. Moreover, including them in the full mixture improves sample efficiency (See Appendix C.2 that shows MITRA is more sample efficient than (GSCM, GET, GGR} (MITRA-Mix2) and Attic 5
real datasets,
4.8 Scaling Behavior for TFMs
Beyond prior construction, two critical factors influencing pretraining are the model size and the amount of training data. We investigate their respective scaling behaviors in Figure 5 by evaluating the performance on TabRepo given different model sizes and varying amount of pretraining data. Specifically, we vary the model depth across 6 configurations (4, 8, 12, 16, 20, 24 layers), all pretrained with the same on-the-fly generated mixture priors. For the model size scaling law, we observe that larger models achieve better performance in early training stages and also converge to higher final accuracy. However, performance gains begin to saturate beyond 12 layers, indicating a trade-off between model capacity and computational efficiency when selecting the appropriate model size. Regarding the sample size scaling law, we find that performance improves rapidly in the early stages and then gradually plateaus, with diminishing returns after approximately 18K steps. In our setting each step involves 2,048 new synthetic datasets, so this suggests the model saturates after encountering around 37 million unique datasets.
9

---

ALC
ALC (Siep > 20009
ACC
ACC (Siep -app-aa 2000)
CB
(Stup > 2008)
138
170
Step
Step
Step
Step
Seep
Step
Figure 5: Model size scaling law. We vary the model size across six configurations (4, 8, 12, 16, 20, and layers), each pretrained on 45 million unique samples. The plot zooms in on pretraining steps beyond 2,000 to provide clearer view of performance trends. A larger version focusing only this zoomed-in region is provided Figure 25 in Appendix.
Conclusion and Discussion
Conclusion. We provide the first systematic investigation into the role synthetic priors in pre- training TFMs, demonstrating how prior effectiveness depends both its standalone performance on real tabular datasets as well as its diversity and distinctiveness within a mixture. Based on our analysis, we construct diverse, high-performing. and model-agnostic mixture of synthetic priors. Leveraging this mixture, we develop MITRA, a SOTA TFM that consistently outperforms existing TFMs and other strong tabular baselines, across both classification and regression tasks. Limitations are discussed in Appendix [] Broader Impact. Our work advances the understanding and design of synthetic data for pretraining foundation models in structured domains, reducing the need for costly real labeled data and reducing privacy risks associated with training on sensitive real-world records.
10

---

References
[1] David Bonet, Daniel Mas Montserrat, Xavier Giró-i Nieto, and Alexander G loannidis. Hyper- fast: Instant classification for tabular data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11114-11122 123, 2024.
[2] Felix den Breejen, Sangmin Bae, Stephen Cha, and Se- Young Yun. Fine-tuned in-context learning transformers are excellent tabular data classifiers. arXiv preprint arXiv:2405. 13396 2024.
[3] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[4] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[5] Felix den Breejen and Se- Young Yun. Attic: A new architecture for tabular in-context learning transformers,
[6] Arpad E Elo. The proposed usef rating system, its development, theory, and applications. Chess life, 2288)2242-447 1967.
[7] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.066505 2020.
[8] Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller,, Prateek Mutalik Desai, David Salinas, and Frank Hutter. Tabarena: A living benchmark for machine learning on tabular data. arXiv preprint 1XXXv22506.16911 2025.
[9] Benjamin Feuer, Robin Sehirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. Advances Neural Information Processing Systems, 37:83430- 83464, 2024.
[10] Josh Gardner, Juan Perdomo, and Ludwig Schmidt, Large scale transfer learning for tabular data via language modeling. arXiv preprint arXiv:2406. 12031, 2024.
[11] Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LcDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. Amlb: automl benchmark. Journal of Machine Learning Research, 25(101):1-65, 2024.
[12] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter- efficceet ensembling. arXiv preprint arXiv:2410.24210, 2024.
[13] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:18932- 18943, 2021.
[14] Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009.
[15] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR, 2023.
[16] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. International Conference Learning Representations, 2023.

---

[17] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister, and Frank Hutter. Accurate predictions on small data with tabular foundation model. Nature, 637(6045):31--326, 2025.
[18) David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mips and boosted trees tabular data. Advances in Neural Information Processing Systems, 37:26577-26658. 2024.
[19] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 01-23328--3944 2021.
[20] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
[21] Myung Jun Kim, Léo Grinsztajn, and Gaäl Varoquaux. Carte: pretraining and transfer for tabular learning. arXiv preprini 1rXiv:2402.16785, 2024.
[22] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994.
[23] Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony I. Caterini. Tabdpt: Scaling tabular foundation models. arXiv preprint arXiv:2410./8110 2024.
[24] Junwei Ma, Valentin Thomas, Guangwei Yu, and Anthony Caterini. In-context data distillation with tabpfn. arXiv preprint (tl::2402.0697//, 2024.
[25] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances Neural Information Processing Systems, 36:76336-76369, 2023.
[26] Andreas Müller, Carlo Curino, and Raghu Ramakrishnan. Mothernet: A foundational hypemnt-- work for tabular classification. arXiv preprint arXiv:2312.08550 2023.
[27] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning python. Journal of Machine Learning Research, 12(0ct):2825-2830, 2011.
[28] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances neural information processing systems, 31, 2018.
[29] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: tabular foundation model for in-context learning large data. arXiv preprint (xXix:2502.05566 2025.
[30] David Salinas and Nick Erickson. Tabrepo: A large scale repository of tabular model evaluations and its automl applications. arXiv preprint arXiv:2311.02977 2023.
[31] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2206.011 2021.
[32] Boris Van Breugel and Mihaela Van Der Schaar. Why tabular foundation models should be a research priority. arXiv preprint arXiv:2405.01 147, 2024.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
12

---

[34] Xumeng Wen, Han Zhang, Shun Zheng, Wei Xu, and Jiang Bian. From supervised to generative: novel paradigm for tabular deep learning with large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3323-3333, 2024.
[35] Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. Mixture of in-context prompters for tabular pfns. arXiv preprint arXiv:2405. 16156, 2024.
[36] Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Z Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. arXiv preprint arXiv:2403.01841 2024.
[37] Yazheng Yang. Yugi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: universal pretraining protocol for tabular foundation model in data science. arXiv preprint arXiv:2307.09224, 2023.
[38] Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross-table masked pretraining for web data mining. Proceedings the ACM Web Conference 2024, pages 4449-4459, 2024.
[39] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305. 06090, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our paper proposes mixture of synthetic priors that leads to state-of-the-art tabular foundation model, which is reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not involve new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experimental settings in Section B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

reproducible, they Answer: [No]
Justification: We only use public datasets which are all properly cited. Our code release needs go through legal review process from our institute, and will release the code after the legal review is complete.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
public/guides htt////odSuumiiiionnnee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccppbllc/guuudeee Sttp:e///doooeSeeeseeeieiiie
ssioiiooiiiiioiociicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper has specified all the training and test details in Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have properly provided the variances of our results in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have detailed the compute resources in Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://www......o..........aaaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact of this paper in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose high risk for misusing data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the code and data used in this paper, and respect their license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Contents
Introduction
2
Related Work
3
3
MITRA: TEM Pretrained on a Mixture of Priors
3.1
Data-Generating Priors
3.2
Prior Mixture Promoting Diversity and Distinctiveness
3.3
Pretraiming TFM on Prior Mixture
4
Empirical Results
Experimental Settings
4.2
MITRA achieves SOTA classification and regression performance
MITRA priors are model agnostic
4.4
MITRA is more sample efficient
4.5
MITRA shows the best performance with advanced ensembling techniques
4.6
MITRA shows the best fine-tuning performance
4.7
Prior Importance Ablation Study
8
4.8
Scaling Behavior for TFMs
5
Conclusion and Discussion
10
MITRA
26
TFM Preliminaries
26
A
2
Data Generation
26
A.2.1 Feature and Target Generation
26
A.2.2 Synthetic Generatng Priors
26
A.3
Algorithm
28
E Experimental Settings
B. I
Benchmarking Datasets
31
B.2
2
Metries
32
B.2.1
Ranking-Baut Based Metries
................................
32
B.2.2
Aggregated Classification Metrics
33
B.2.3
Aggregated Regression Metrics
33
B.3
Training and
................................
33
B.3.1
Transtormer Architecture
33
B.3.2
MITRA Pretraining
34
B.3.3
Ensembling and Finetuning Parameters for TFMs
34
B.3.4
Statistical Model Hyperparameters)
34
Additional Empirical Results
35
21

---

Generalizability Matrix and Performance Vector Metrics
35
Sample Efficiency
35
Ablations
36
Classification
37
C.S
Regression
38
Critical Differences
38
Timing Efficiency
40
C.8
Decision Boundary Visualizations
40
Limitations and Future Work
42
22

---

List of Tables
Three factors of prior importance. Each entry Gú in the Generalizability Matrix represents the AUC of a TFM pretained on data generated from Gj and evaluated on data from Gj. Each element P; in the Performance Vector corresponds to the
AUC of a TFM pretained on data from Gi and evaluated on a real-world dataset.
We use a color gradient to visually indicate the relative quality of each prior. The best//oon priors are shown in the darkest green/red.
2
MITRA wins on all three classification benchmarks (We show the overall merged
results and detail the separate benchmark results in Appendix [C.4] i.c., Ta- (TabReeo), Table [14 (TabZilla), Table (AMLB) due to space limits). Winner/runner-up up in / +e means ensembling in ICL, and +f means fine-tuning The 95% conhdence intevaa is shown in parentheses for the Elo. Aggregated metrics show mean and std (shown in parentheses) of the corresponding metric.
ble | [13
3 MITRA also demonstrates better performance with 1D) attention model, showing that our priors are model-agnostic. Winner/runner-up in shades of green / The 95% confidence interval is shown in parentheses for the Elo. The columns in the
aggregated metries are mean and std (shown in parentheses) of the corresponding metric.
4
MITRA wins on TabRepo 10-fold regression benchmark. Winner/runner-up up in te means adding ensemble in ICL, and +I means adding
shades of green Iine-tuning,
5
MITRA shows better sample efficiency. Winner in shades of green
6
TabRepo 10-fold classification ablation study on the effect of each prior in the mixture consisting of 50% SCM and 50% TBPs with equal weighting. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric. Note that SCM
+ +TT is variant of Attie [5]
7
Benchmarking Datasets Description and Statistics.
32
8
Number of Estimators for the Ensemble TFMs.
34
9
Diversity (diagonal) and distinctiveness (off- diagonals) of each prior distribution. entry Gij in the Generalizabiliity Matrix represents the ACC of a TFM pretained
on data generated from 9i and evaluated on data from Gj Each element P, in the Perlormance Vector corresponds to the ACC of a TFM pretained on data from G, and evaluated on woaldworld datasets.
35
10
Diversity (diagonal) and distinctiveness (off-diagonal, of each prior distribution.
Each entry G.y in the Generalizability Matrix represents the CE of TFM pretained
on data generated from G. and evaluated on data from Gy. Each element P, in the
Performance Vector corresponds to the CE of a TFM pretained on data from G. and
evaluated on real-world datasets.]
35
More diverse priors (MITRA) show better sample efficiency. Winner under each down- hown-ssmpling ratio ds E (1.0, 0.75, 0.5, 0.25, 0.1} in bold.
36
12
TabRepo 10-fold classification ablation study on the effect of the percentage p of
SCM in the mixture of priors and the total amount of tree-based priors (TBP) is 1 p.
36
13
MITRA wins on TabRepo 10-fold classification benchmark. Winner/runner-up in
/ +e means ensembling in ICL, and +f means fine-tuning. The 95% confidence
interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric.
37
14 MITRA wins on TabZilla 10-fold classification benchmark. Winner/runner-up
+e means adding ensemble in ICL, and +I means adding fine-tuning.
37
23

---

MITRA wins on A.LB 10-fold classification benchmark. Wimner/runner-up in 15
+e means ensembling in ICL. and means fine-tuning. The 95% confdence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown parentheses) of the corresponding metric.
38
16 Adding MITRA (bagging) and Attic (bagging) in the aggregated classification bench- mark results in Table of the main text. +e means adding ensemble in ICL, and +I means adding fine-tuning. The 95% confidence interval is shown in parentheses for the Elo. The columns the aggregated metries are mean and std (shown in parentheses) of the corresponding metric.
40
17 Additional results TabArena. "Single" represents single model results without HPO. "HPO" represents results after 200-iteration hyperparameter optimization.
"HPO ensemble" represents the ensemble results of hyperparameter optimization.
MITRA is the strongest single model, outperforming all methods even when they
perform hyperparameter tuning. "Train Time" and "Infer Time" represent the median
training and inference time in seconds per IK rows. MITRA achieves Pareto efficiency both in training time and inference time.
18
AMLB 10 fold regression benchmark. +e means adding ensemble in ICL, and +f means adding fine-tuning. The 95% confidence interval shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric.
42
24

---

List of Figures
Mixture of prioss in MITRA, Generalizability Matrix G and Performance Vector P. We consider three factors for good priors: (1) the performance of a pretrained TFM
using data generated solely from that prior on real tabular data (Point and P); (2) its
diversity and (3) distinctiveness when included in a mixture (Points 2 and 3, captured by the diagonal and off -ii-diiaon entries of G). This leads to our mixture comprising
SCM, gradient boosting, random forest and decision tree priors.
2
MITRA Pipeline: Model agnostic visualized with either ID or 2D attention.
3
Mitra (bagging) shows the best performance compared with TabPFNv2 PHE, Auto-
Gluon best-quality and Mitra (+el) across 300- to 3600-second budgets on 1-fold
TabRepo benchmark.
4
MITRA shows significantly better fine-tuning performance across ensemble sizes.
8
5
Model size scaling law. We vary the model size across six configurations (4, 8, 12,
16, 20, and 24 layers), each pretrained on 45 million unique samples. The plot zooms
in pretraining steps beyond 2,000 to provide a clearer view of pertormance trends.
A larger version focusing only on this zoomed- in region is provided in Figure (25 in
Appendix.
10
6 Randomly generated 2D Dataset from mixture of priors in MITRA. The features
& drawn from our mixture of SCM and TBP geaeeeeenertinnggg priors in MITRA on classification tasks. The classification labels y are shown in color.
27
t-SNE visualization of random nigh-dimensional dataset from mixture of priors
in MITRA. The features x [e drawn from our mixture of SCM and TBP data-
generating priors in MITRA. The classification labels y are shown in color.
28
8
Comparison of training accuracy with respect pretraining steps, with data generated from the mixture of priors in MITRA, vs. data from its prior individually.
34
9
Critical difference plot on TabRepo classification benchmark.
39
10
Critical difference plot on TabZilla classification benchmark,
39
11
Critical difference plot on AMLB classification benchmark.
39
12
Critical difference plot on TabRepo regression benchmark.
39
13
Critical difference plot on AMLB + OPenML-CTR23 regression benchmark.
40
14
Average running time vs. performance metrics (Elo, winrate, and AUC) on TabRepo
42
15
Decision boundaries of MITRA and baselines on 2D checkerboard data. MITRA
shows more regular and less fragmented decision boundaries than TabPHNv2.
43
16
Decision boundaries of MITRA and baselines on 2D Gaussian Process data.
43
17
Decision boundaries of MITRA and baselines on 2D Julia set data.
43
18
Decision boundaries of MITRA and baselines 2D linearly separable data.
44
19
Decision boundaries of MITRA and baselines on 2D nonlinear Gaussian mixture data.
44
20
Decision boundaries of MITRA and baselines on 2D sine wave data.
44
21
Decision boundaries of MITRA and baselines 2D sinusoidal checkerboard data.
MITRA shows more regular and less fragmented decision boundaries than TabP-Nv2]
45
22
Decision boundaries of MITRA and baselines on 2D spiral data.
45
23
Decision boundaries of MITRA and baselines 2D star data.
45
24
Decision boundaries of MITRA and baselines on 2D Swiss roll data (from 27
46
25
Large view of Figure showing only the zoomed-in in region for greater clarity.
46
25

---

A MITRA
In this section, we provide details on the in-context learning (ICL) preliminaries, our data generation methods and priors, and the overall MITRA algorithm.
A.1 TFM Preliminaries
To pretrain a TFM on purely synthetic data, each dataset D = {(xn: }n) sampled from a prior distribution G that generates datasets with varying numbers of features, samples, classes (for classification tasks), and categorical attributes. Once a TFM fo has been pretrained, it can be used to perform ICL as follows. The model is given a support set Dsup consisting of Nuup labeled rows (Xamp,, Msap, ) along with = Nary unlabeled query rows {Xpr,, 19 I==1. where + g = N. then predicts the corresponding query labels (Jyry, In=1 in a single forward pass:
Japy
Bayy = fo([(x Mop, .. (Xup,, Vsap, Xqrrs" Xay,
without the need to update its parameter 0.
A.2 Data Generation
Here, we discuss the specific parameters and modeling choices in the data generation for feature- pairs (x, y) E "D and the details on the priors that we use in our data-generating mixture of priors in MITRA. For simplicity, Figure [ shows visualization of 2D dataset generated from mixture of priors in MITRA. In addition, Figure 7 shows a t-SNE visualization of nigh-dimensional data samples from our mixture used during pretraining. We see both continuous and categorical features represented. The classification labels y are depicted in color.
A.2.1 Feature and Target Generation
Feature x. We design x to include Noont continuous and Neat categorical components, such that Nooat + Neat = d. The number of categorical components is determined by Neat [Peat (d + 1)J. where Pear is categorical percentage uniformly sampled. We then uniformly sample the Neat categorical feature indices Iem in I = (1..... .d without replacement. For each continuous feature index j E Trum
Teas, we take x, to be i.i.d. Gaussian noise. For each categorical feature X we generate its number of classes from Geometric distribution. Lastly, we model each X₃ via multinomial distribution over number of classes No
Target y/ For each target y V, we handle its generation differently depending whether the generating prior uses "direct" or "indirect" sampling. For direct sampling, we directly simulate y from random conditional distribution p(y | x). For indirect sampling, a classifier or regressor on a synthetically generated training dataset (x, y) for the corresponding task. In classification tasks, the label y y c Z is generated similarly to the aforementioned label generating process for the categorical features. In regression tasks, we take to be normal. The final label y2 generated as the output from the fitted estimator new feature vector X2.
first require fitting
A.2.2 Synthetic Data-Generating Priors
We include mixture of SCMs with TBPs, with both indirectly (ET, GB, DT, RF) and directly sampled priors (SCM, DSRF).
Indirectly Sampled Priors. Algorithm 2 shows the mataa-eenratiig procedure for "indirectly" sampled priors. We refer to these priors as indirectly sampled methods since they first require training data D = X Y] E Bx to fit the estimator, i.c., classifier or regressor for the corresponding task, where B denotes the base size. Then, we generate features X₂ E R. Nxd according to the feature generation procedure described in Subsection 1.2. The final Y, E R. is output from the fitted estimator by predicting on this X₂ input. We note that TFbForestPFN 2 and Attic (5) use an indirectly sampled DT prior, and TabICI (29 uses an indirectly sampled XGBoost prior. Our data generation similar to that in TabForestPFN (2 and Attic (5] with the differences that we directly fit Classifier for classification tasks rather than using a Regressor, and use our direct multinomial label
26

---

25
35
-3 00
50:00
85
on
Figure Randomly generated 2D Dataset from mixture of priors in MITRA. The features x E R drawn from our mixture of SCM and TBP data- aata-generating priors in MITRA on classification tasks. The classification labels y are shown in color.
generation procedure (see subsection A.2.II for both the target labels and categorical feature labels. Hence, we eliminate the need for using the quantile transform to bucketize the continuous values to labels. For these indirectly sampled TBPs, we use the classifiers and regressors from scikit-learn 27
Directly Sampled Priors. Algorithm 5 shows the lata-generating procedure for our directly
sampled random forest (DRSF) TBP prior. We refer to these priors directly sampled because they first sample a function o E from function space F and then generate the targets yn = f(x,) for data x,. Hence, directly sampled methods only need to form X E N once and then the fane-ainn priors directly output Y2 E R.
For DSRF, we generate the features X₂ using the same feature generation process from Subsec- tion A.2. For DSRF, we must first construct random trees from F (see Algorithm 4. To do so, we sample the following: (1) random split indices (0....
1}. where denotes the feature dimension; (2) random split thresholds in a specified range thres-int. Each node in the tree stores its corresponding split index and split threshold. We store split intervals dictionary to track sub-intervals corresponding to the feature split index to sample from future splits as the algorithm progresses. We follow the convention from DTs, where the tree is split on feature index i and value and all datapoints such that x[i] < split to the left side of the tree and those such that
27

---

-2
Figure 7: t-SNE visualization of random high-dimensional dataset from mixture of priors in MITRA. The features x Rdd drawn from mixture of SCM and TBP data-generating priors in MITRA. The classification labels are shown in color.
x[i] >> are on the right side of the tree. We also control the number of nodes with children. We construct the leaf node labels differently depending on the task type. For classification, we uniformly randomly sample a starting index in {O...., N₂ - 1), where N₂ denotes the number of classes, and we increment it for each subsequent leaf node added modulo the number of classes. For regression, we sample the leaf nodes from a Gaussian distribution. We sample number of estimators (random trees) Ne. and we traverse each tree until a leaf node is reached to get target value for that tree (see Algorithm 3). Lastly, we compute the final label y using majority voting over these N₂ values for classification tasks and using the mean for regression tasks.
For details on the SCM prior, see Appendix C.I of the TabPFN paper [66]
A.3 Algorithm
Algorithm provvdes overview of our MITRA method.
The population version of the likelihood in Section 3.3 takes form an expectation over table D, with each table sampled from the prior mixture. Given the sampled table, the model assumes query rows are conditionally independent given the in-context examples, so that the ligelikeldoood within the expectation is decomposed into sum of q individual terms, one per query row. When
28

---

the query label corresponds to a classification task, its (conditional) distribution is assumed to be categorical, making the training objective equivalent to minimizing the cross-entropy loss. When the query label corresponds to regression task, its (conditional) distribution is assumed to be Gaussian and the training objective corresponds minimizing the Mean Squared Error (MSE) loss.
Algorithm 1 MITRA: Mixture of Synthetic Priors for pretraining TFMs
Require: Set of data generators (9i h with mixture weights {w;} M such that E Wi = 1, Require: Support set size s, query set size q, number of pretraining steps T.
1: for t= to 7 do
2: Sample generator index ut, for I,..., M. (6) (6)
3: Sample synthetic dataset ( {(xn yn )
+ Gz.
4: Randomly partition D(i) into support set Duip and query set Day with (Daapl and
[D4)] (5)
9.
5:
Construct input sequence: [(xsup; Ysup, )..... (Xsup, 100 ) (()
xgry.
Xgr
6: Predict query labels via TFM: Marring = fo(.)-
7: Compute loss: L = log pe Merry
D Xapy
(8)
8: Update model parameters using gradient of L.
9: end for
Algorithm 2 Indirectly Sampled TBPs.
Require:
Task generator TBP, base size B, feature dimension d, number of samples N, number of classes Ne, task type E (CLASSIFICATION, REGRESSION).
Generate training data
X, YJ E Bx where Xii... XB
E
R Bxd and
Y
[y/.....,
MB
R according the procedure in Subsection A.2.I
Instantiate model TBPClassifier, if CLASSIFICATION;
2:
TBPRegressor, otherwise.
3: Fit model: 2X Y).
4: Sample testing features X; E Rxd
5: Generate predictions: Y, <- x.predict(X.) ERN.
6: return X2, Y2) E (x++()))
Algorithm 3 Target Generation From DT Traversal
function DT-TRAVERSAL(X, tree)
2:
(ind, thres) +- tree. value
3:
if tree. target is not None then < tree. target else x x < thres then
Leaf node reached
4:
5:
6:
< TRAVERSALL,, tree. left)
Traverse left subtree
7:
else
8:
< DT-TRAVVESSALX tree. right
Traverse right subtree
9:
end
10: return y
11:
end function
29

---

Algorithm 4 Construct Random Decision Tree (DT)
Require:
Feature dimension d, number of classes No.
task type T
E
{CLASSIFICATION, REGRESSION), minimum and maximum depths dmin > dmax split threshold interval thres-int, probability of no children Pae. Gaussian parameters for regression leaf nodes (), global leaf counter Meat 0.
class NoDF(depth, dmiin dmaxx thres-int)
2: value f- None
Split rule: (feature index, threshold)
3: left, right <- None
Child nodes Node depth
4: d f depth
5:
target + None
Leaf prediction value
6: (
Dict. of (feature index, split intervals) > Min and max depths, threshold interval
7: Store duins dmaxx and thres-int
8: end class
9: function RANDDT((1b, ub]=thres-int, tree=None, depth=0, (), ind None)
10: tree +- NoDF(depth, dmin dmax. thres-int)
if' depth > 0 then
12:
tree.rs + rs; tree. rs [ind] < [1b, ub] else if depth 0 and T CLASSIFICATION then target ~ LI, No - 1}]
14:
end
16:
ind ~ u((0....,d d-1})
[1b, ub) < tree. rs [ind] if ind tree. rs, else tree. thres-int
18:
thres ~ [,b, ub])
free.valuse < (ind, thres)
20:
if (tree. d < dmax and p ~ U(0, 1]) 2 Pnc) tree. d < dmin then
Add children
21:
tree. tree.left + RANDDT((1b, thres], tree. tree.left, cree.d+ + I, tree.re. rs, ind)
22:
tree.right +- RANDDT((thres, ub), tree.right tree. .d + 1, tree rs, ind) b Assign target value leaf node
23: else
24:
if T CLASSIFICATION then tree. target + Nieat mod Ne Niea t Niaa +1
25:
26:
27:
else
28:
tree. target (1,,))
29:
end
end if
31: return tree
end function
30

---

Table provides the description and statistics of the benchmarking datasets with various number Algorithm 5 Directly Sampled Random Forest (DSRF)
Require:
Feature dimension d, number of samples N, number of classes No. task type T E {CLASSIFICATION, REGRESSION), number of estimators Ne, minimum depth dmin > 0, maxi- mum depth dmax, split threshold interval thres-int, probability of terminal node Pae. Gaussian parameters for regression leaves (p,
1: Generate feature matrix X2 N described in Subsection A.2.1
2: for = 1 to N do
3: for 1 N₂ do
4: tree f- RANDDT()
Sample a random DT from function space F (Alg. Evaluate tree on input (Alg.
5: target[i] + DT-TRAVERSAL tree)
6: end for
7: if CLASSIFICATION then
8:
12(n] < MAJORITY Vorettaree)))))))
9: else
10:
[2 < MEAN(target)
end if
12:
end for
13:
return X2, Y2] E (x(+4)
B Experimental Settings
In this section, we discuss the details on the benchmarking datasets, metrics, and implementation.
B.1 Benchmarking Datasets
of rows and features. As discussed in Section 4.1 to compare with ID models (e.g., TabPFN that support features up to 100) and 2D models (e.g.. TabPFNy2) that support features up 500, evaluate on both small-feature and farge-fcature benchmarks. For the small-feature benchmark, we use 66 TabRepo classification datasets, 75 TabZilla classification datasets, and 10 TabRepo regression datasets, that have up to 3,000 rows and 100 features following TabPFN [16]. To evaluate on large- feature benchmarks, we use 29 classification datasets from AMLB benchmark with up to 10,000 rows and 500 features. We additionally evaluate on a farge-feature regression benchmark of 28 datasets from AMLB and OpenML-CTR23 benchmarks, with up to 10,000 rows and 500 features. This is consistent with the evaluation protocol of TabPFNv2 17
The dataset task IDs provided follows:
TabRepo: 2, 11, 37, 2073, 2077, 3512, 3549, 3560, 3581, 3583, 3606, 3608, 3616, 3623, 3664, 3667, 3690, 3702, 3704, 3747, 3749, 3766, 3783, 3793, 3799, 3800. 3812. 3903, 3913, 3918, 9904, 9905, 9906. 9909, 9915, 9924, 9925, 9926, 9970, 9971, 9979, 14954, 125920, 125921, 146800, 146818,
146819, 168757, 168784, 190137, 190146, 359954, 359955, 359956, 359958, 359959, 359960,
359962, 359963, 361333, 361335, 361336, 361339, 361340, 361341, 361345
AMLB: 2073, 146818, 146820, 168350. 168757, 168784, 168911, 190137, 190146, 190392, 190410,
190411, 359954, 359955, 359956, 359958, 359959, 359960, 359961, 359962, 359963, 359964, 359965, 359968, 359969, 359970, 359972, 359974, 359975
TabZilla: 9, 10, 11, 14, 15, 16, 18,
25, 27, 29, 31, 35, 37, 39, 40, 42, 47, 59, 2079, 2867, 3512, 3540, 3543, 3549, 3560, 3561, 3602, 3620, 3647, 3731, 3739, 3748, 3779, 3797, 3902, 3903, 3913, 3917, 3918, 9946, 9957, 9971, 9978, 9979, 9984, 10089, 10093, 10101, 14954, 14967, 125920, 125921, 145793, 145799, 145847, 145977, 145984, 146024, 146063, 146065,
47,4,, 48, 50, 53, 54,
146192, 146210, 146800, 146817, 146818, 146819, 146821, 146822
RabRepoReg: 167210, 359930, 359931, 359932, 359933, 359935, 359942, 359944, 359950, 359951
AMLB OpenMI--CTR2: [167210, 233215, 359930, 359931, 359932, 359933, 359934, 359939, 359940, 359942, 359944, 359945, 359948, 359950, 359951, 360945, 361235, 361236, 361237,
361243, 361251, 361256, 361258, 361259, 361617, 361619, 361621, 361622
31

---

Table 7: Benchmarking Datasets Description and Statistics.
Dataset
Task
Num. Tables
Max Num. Rows
Max Num. Features
TabRepo
Classification
66
3,000
100
AMLB
Classification
29
10,000
500
Tabzilla
Classification
75
3,000
TabRepoReg
Regression
10
3,000
100
AMLB+OpenMLLCTR23
Regression
28
10,000
500
B.2
Metrics
To ensure comprehensive evaluation of model performance across datasets and tasks, we employ diverse set of ranking and aggregated metrics for both classification and regression tasks. Below, we provide their formal definitions used in this work. When computing the rank-based metrics, each fold of dataset is considered as separate evaluation unit.
B.2.1 Ranking-Based Metrics
To assess model performance across suite of datasets, adopt the following ranking-
ranking-based based metries:
Average Rank. Let M denote the models and D the of datasets. For a model m M and dataset 5 E D, let ranks(m) denote the model's rank (I is best) on dataset ó according to performance metric (e.g., accuracy). The average rank is defined as:
Avg. Rank(m) =
rank
D
RED
Elo Rating. Elo rating generalizes pairwise win/loss outcomes into a competitive rating system. Each model is treated as a player, and its rating is updated based on pairwise performance comparisons across datasets. The final Elo rating reflects the model's relative strength across all pairwise matchups. We implement the metric based existing work (6]
Winrate. Winrate captures the proportion of datasets on which model outperforms other models. Formally, for a model m:
Winrate(m)
Σ Σ
(I[E)(m) < Es(m')) + 1 [[B((m) = Es(m')))
1
DIIM
BEDm'EM zém
where error E = 1 AUC for classification task, and E = 1 R² for regression task, and [ denotes indicator function. Tie contributes half win.
Rescaled Accuracy. To address the effect of dataset difficulty on raw scores, we scale by the best-performing model within each dataset:
Rescaled Errors(m) Rescaled Error; (m) = E(m) - E(m*)
RAccs(m
E(m') E(m*)'
where m = arg min,. E(m) and m/ = arg max E(m) respectively denote the model with the smallest and largest errors. Error - AUC for classification task, and E R² for regression task. The overall Rescaled Accuracy is the average over datasets:
RAce(m)
RAccs(m).
[D]
RED
Champion Delta. Let m² arg min, E(m) denote the champion model. The Champion Delta for model m is defined as:
CA(m)= (1 E(m²) x 100.
E(m)
This reflects the percentage performance margin between the current model and the best-performing model. Error E = 1 AUC for classification task, and E - R² for regression task.
32

---

B.2.2 Aggregated Classification Metrics
For aggregated classification metrics, we report standard metrics averaged across datasets.
AUC (Area Under ROC Curve). For a binary classifier, AUC measures the under the Receiver Operating Characteristic curve, which plots the true positive rate (TPR) against the false positive rate (FPR). Formally:
AUC / (P)( dFPR((),
where threshold on predicted probability scores. For multiclass classification, adopt the one-vs-one (OvO) strategy and compute the average AUC over all pairwise class comparisons.
Accuracy (ACC). Accuracy is the fraction of correctly classified instances:
ACC
N Σ
1/ 0t)
where yı is the ground-truth label, and is the predicted label, and [[:] denotes the indicator function.
Cross-Entropy (CE). Let p; be the predicted class probability vector for instance i, and ys the one-hot encoded ground-truth vector. The cross-entropy loss is:
CE
Vi.c log Pi.c.
N
1=1
where the number of classes.
B.2.3 Aggregated Regression Metrics
For aggregated regression metrics, we report standard metrics averaged across regression datasets.
R² (Coefficient of Determination). The R² score quantifies the proportion of variance explained by the model:
R = E N == (y: 0.)2 E1 (U: 7)2
where ÿ the mean of the ground-truth values.
Root Mean Squared Error (RMSE). RMSE penalizes large prediction errors more heavily:
RMSE
0:)3
N
1=1
Mean Absolute Error (MAE). MAE measures the average magnitude of prediction errors:
N
MAE
1ys
Vzl
N
B.3 Training and Inference
Our implementation is based on PYTORCH. We discuss the specific pretraining details and model hyperparameters below.
B.3.1 Transformer Architecture
MITRA is built on Transformer architecture 133 tion heads. Each Transformer layer includes both row-wise attention and column-wise attention implemented using FlashAttentin (4). The resulting model contains 72M parameters. MITRA 1D is built on Transformer architecture, and each layer contains row-wise attention. The resulting model contains 37M parameters.
with 12 layers, 512 embedding size and 4 atten-
33

---

B.3.2 MITRA Pretraining
Training ACC
Mitra
0.8
DSRF
0.6
ET
GB
DT
RF
5000
10000
15000
20000
Pretraining Steps
Figure 8: Comparison training accuracy with respect pretraining steps, with data generated from the mixture of priors in MITRA, vs. data from its prior individually.
For pretraining MITRA, we eight 40GB A A100 GPUs. MITRA is trained 45 million synthetically generated datasets. This training takes approximately 60 hours on 8 GPUs (Nvidia A100s). To normalize features, we apply uniform quantile transform based on the support set, followed by standard normalization based on the mean and standard deviation from the support set. For regression tasks. additionally apply min-max normalization for the target column using the minimum and maximum values of the support set for each table. Figure 8 illustrates that training curves tend to converge quickly around 2000 steps and oscillatory. Interestingly, Figure g hhighlights the diversity findings in the diagonal of the ACC generalizability matrix in Table 9 We see that the training accuracy converges to lower value for models generated with data from more diverse priors. The mixture priors of MITRA lie in between the less diverse TBPs (DSRF, GB) and the more diverse TBPs (RF, ET, DT), and slightly below SCM, which indicates that the TBPs add diversity our experiments, we observe that the validation performance on real-world datasets improves as the pre-training continues.
mixture. In
B.3.3 Ensembling and Finetuning Parameters for TFMs
For models incorporating ensembling, use the default number of estimators for each model, i.e.. 4 for TabPFNv2 on classification tasks, for TabPFNv2 on regression tasks, 32 for TabICL, 3 for TabPFN. We find that on TabZilla and AMLB classification benchmarks, TabPFNv2 with 8 estimators performs better than using 4 estimators and the performance saturates after that. Accordingly, we increase the number of estimators to 8 for TabPFNv2 on these two benchmarks ensure competitive performance. All models with +f are fine-tuned for 50 epochs, which is setting that typically triggers early stopping on most datasets. Note that TabPFN only supports up to 100 features so is not evaluated on the AMLB classification dataset. In addition, TabPFN and TabICI do not support regression. Attic has been observed to have training stability issues on regression tasks, and its reported performance is inferior to that of XGBoost, which we include baseline for regression. Table Summmaizes the number estimators for each dataset.
Table Number of Estimators for the Ensemble TFMs.
Dataset
MITRA (tel)
Attic (tef)
'haPFNv2 (+el)
TabPHNv2 (+e)
MITRA (+e)
Attic (+e)
TabICL (+e)
TaPPEN (te)
TabRepo
4
4
32
32
32
3
AMLB
8
8
8
32
32
Tahzilla
8
8
32
32
32
Reg.
8
8
32
B.3.4 Statistical Model Hyperparameters
For RealMLPP LightGBM, XGBoost, CatBoost, MI.P, we use their default hyperparameters in AutoGluon ..
34

---

C Additional Empirical Results
In this section, we report additional empirical results including the generalizability matrix G and performance vector P computed in other aggregated metrics, additional sample efficiency results and 2D decision boundary visualizations, ablations, classification and regression per dataset results, and the further improved performance of MITRA on the aggregated metries using the advanced ensembling bagging method.
C.1 Generalizability Matrix and Performance Vector Metrics
Here, we show the generalizability matrix G and performance vector P on TabRepo on metrics, i.e. accuracy (ACC) (Table 9 and cross-entropy (CE) (Table (T)) in addition to the AUC table reported in T The rows of denote the model pretrained with data generated from each prior. We then test those pretrained models on synthetic data generated from each prior distribution. We generate 100 tables, each with N 1000 samples (rows), where the number of samples in the support = 800 and the number of samples in the query q 200.
We see that similar findings hold across the various aggregated metries in these tables. In particular, Table 9 further emphasizes the distinctiveness of the TBP, ET, where the off- off-diagonal G.y corre- sponding the model pretrained with data drawn from GEr is only 0.577. For the ranking- metrics of each individual prior, see Table
Table 9: Diversity (diagonal) and distinetiveness (off-diagonals) of each prior distribution. Each entry Gy in the Generalizability Matrix represents the ACC of TFM pretained on data generated from G. and evaluated on data from Gy. Each element P, in the Performance Vector corresponds the ACC of a TFM pretained on data from Gj and evaluated on real-world datasets.
Train
Test DT
SCM
GB
RF
DSRF
TahRepo
SCM
0.841 (0.155)
0.577
0.902
0.747
0.562
0.919)
0.812 (0.176)
ET
0.808 (.199
0.715 (0.210)
0.928
(.854
0.613
0.94 (0006))
0.803 (0.169)
(.84)
0,625
0.942 (0.060)
0.814
0.588
0.944(0092)
0.796 (0.177)
DT
0. 0.799
0.712
0.929
0.861 (0.134)
0.61 (02238)
p.939go..em
0.791 (0.177)
(.807
0.629
0.904
(.785
0.602 (0.230)
(o.11)
0.782 (0.177)
DSRF
0.806
0.672
0.927
0.840
0.607
0.952 (0.081)
0.799 (0.175)
Table 10: Diversity (diagonal) and distinctiveness (off-diagonals) of each prior distribution. Each
entry Gty in the Generalizability Matrix represents the CE of a TFM pretained on data generated from G. and evaluated on data from Gj. Each element P; in the Performance Vector corresponds to the CE of TFM pretained on data from G; and evaluated on real-world datasets.
Train
Test DT
SCM
ET
GB
RF
DSRF
TabRepo
SCM
0,366 (a 389)
(.054
0.309
0.690
1. 1.128
0.273
0.416 (0.378)
(.465)
0.764 (o.565)
0.220
(.399
1.001
0. 0.173
0.503
(0.579)
GB
0.469
1.022
0.162 (0.165)
(.535
1.107
0,1
0.501
(0.464)
(.405)
(o.77))
0.216
0.383 (0.308)
1.007 (0.04)
0.177
0.523 (0.532)
RF
0.457
0.967
(.273
(.579
1.028
(0,025)
0.248
0.521 (0.408)
DSRF
0.449).402)
0,86 p..68)
0.219
0.445
1.021
0.4 (0.232)
0.473
(0.410)
C.2
Sample Efficiency
Table 11 illustrates the improved sample efficiency of MITRA compared to its ablations: MITRA- Mix2 (SCM + ET + GB) and Attic (a variant of SCM DT). particular, the differences in the Elo
become larger as the downsampling ratio, ds, is further decreased with the largest gains occuring when ds 0.1. This result emphasizes the generalizability our mixture priors comparison to mixtures of less priors in data-scarce scenarios.
35

---

Table 11: More diverse priors (MITRA) show better sample efficiency. Winner under each down- sampling ratio ds E {1.0, 0.75, 0.25, 0.1} in bold.
Ranking Metries
Aggregated Metrics
Model
Rank
Elo
Winrate
RAcc
CAA 4
AUC
ACC T
CE
MIXRA-Mix2 (ds=1.0)
4,4
1234 (45140
0,76
0.88
12.1
0.8817
0.8407 at Hem)
MITRA (ds=1.0)
4.5
1227
0.75
0,88
11.7
0,8818
(.8396
0.36019
(dc=1.0)
4.6
1221,
0.75
0.87
12.8
0.881
m.839
0.3 0.3682,0307
MITRA (ds=0.75)
5,6
1151 (42140
0.67
0.83
18.5
0,8758 17890
0.8344 at 16i7)
0.3716 joowes
Attic (ds=00.75)
5.6
11500
0.67
0.83
19.0
p.8743
0.8324 Me)
0.3795,m
MITRA-Mix2 (ds=D.75)
1671
m.8755
0,833801541
0.3727
MITRA (ds=0..)
7.1
1060 (x207)
0.56
0.76
26.6
0.8684 (113))
0.8267 20 1609
0 1ont
MITRA-Mix2 (ds=0.5)
7.2
1056
0.56
0,76
26.3
0 8674441185)
0.825 (pus00))
0.3899
(ds=D.5)
10477
(11291)
0.824pu
0.3965 swaas
MITNA (ds=0.25)
888 (17118)
0.36
39.7
0.8492 (0.427)
0,8076 10.17144
0.4317 (0.550))
MITRA-Mix2 (ds=0.25)
10.0
0.36
0.59
40.1
0.8 0.8488
0. 0.807
0.4327
Altic (ds=0.25)
0.58
pp
0,4416 1337)
MITKA (ds=0.1)
12.5
700 cooso
0.18
0.25
53.3
0.8078 sa.ste)
0,7702 so.rmW
0.5158 (0.m72)
MITRA-Mix2 (ds=D.1)
12.7
68 68
0.17
0:23
$3.8
0.8065
0.76622pr mew0)
0.5 182 17901
Altic (ds=0.1)
12.7
0.16
0.22
53.7
0.7995
0.7648pu174)
0.5237
C.3 Ablations
In T we perform an ablation study to analyze the importance of each prior in our mixture. We begin with ranking the performance of the model pretrained on data drawn from each prior individually. For the next step, we select the prior with the best performance, which in this case is SCM. We then add each remaining prior one-by-one with that selected prior to determine the next best pairing. We continue this procedure until we have added all the priors in MITRA A in a forward process. We see that the ranking of the priors in decreasing order is (SCM, ET, GB, DT, RF, DSRF), which aligns with the performance, diversity and distinctiveness findings from the generalizability matrices G and performance vectors P in the various metries Tables and Tables
10
we study the effect of the percentage p between SCM and the tree-based priors (TBP) in mixture of priors in MITRA the TabRepo dataset. In our experimental results, set p 0.5 for an equal weighting of SCM and TBP. We see that on TabRepo there are several values of p that show improved performance of the mixture over SCM alone (p- 1.0) and TBP alone (p- 0). The average rankings for p = 0.5, 0.4, 0.6 are tied at 3.2 with slightly better Elo for = 0.5 at 1040 vs. 1030 for the other variants. In addition, SCM alone has better performance over the TBPs alone with an average ranking of 3.7 vs. 4.4. The mixture of TBPs without SCM has a lack of distinctiveness since the models trained on data drawn from TBP can predict on data drawn from these other TBPs better than models trained on SCM can, as measured by the off-diagonals of the generalizability matrix G. Removing SCM from the mixture also removes the top performing prior measured by the performance vector P. Note that priors works have combined SCM with 1 TBP but not multiple TBPs. In addition, these works have not studied the effect of the percentage of SCM and the corresponding TBP, e.g., TabICL [29 combines 0.7 for SCM with p = 0.3 XGBoost SCM, and TabForestPFN 2 and Attic combine p = 0.5 SCM with p = 0.5 DT.
In Table 12
Table 12: TabRepo 10-fold classification ablation study on the effect of the percentage p of SCM in the mixture of priors and the total amount of bree- priors (TBP) is p.
Ranking Metries
Aggregated Metries
Avg. Rank
1
Winrate
RAce
CA $
AUC 1
ACC
:
0.5 (MITRA)
3.2
1040
(49-10)
0.57
0.69
14.3
0.868 (0.135)
0,822 (0100)
0.403 (0:30))
0.4
3.2
1030
(1910)
0.55
0.68
15.1
0.864m..13)
0.817
0.406 369)
0.6
3.2
1030 (1909))
0.55
0.67
14.6
0.865 1355
0.822 (0.109)
0.404 (130
0.7
3.3
(00-00
0.55
0.66
15.3
0.863
0.818 (0.171)
0.406
1.0 (SCM)
3.7
983
0.47
0.57
19.1
0.81 220.1799
0.
0.0 (TBP)
4.4
888
0.32
0.35
29.5
0.8566m.135)
01.088
0.451
36

---

Table 13: MITRA wins TabRepo 10-fold classification benchmark. Winner/runner-up in
means ensembling in ICL, and +f means fine tineituning. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric.
Ranking Metrics
Aggregated Metrics
Model
Avg. Rank
Elo f
Winrute
RAcc
CA
AUC
ACC
CE
MITRA (+ef)
7.8
1141 (+St-))
0.882 (0.124)
0.837 (0164)
0.370 (0.66))
Attic (+ef)
7.9
1135 (156.0)
0.69
0.79
19.2
0.880 (0.125)
0.835 (016)
0.377 (0 (0.300)
TahPFNv2 (+c)
8.4
5120
0.67
0.78
20.4
0.879
0.835
0.382.
TabPFNv2 (4ef)
(4:00
0.64
0.76
21.6
0.878 1255
0.831 167)
(.396
MITRA (+e)
9.6
1076
0.61
0.74
25.8
0.877
0.831
0.394
Attic (+e)
10.0
u5:5
0.74
26.7
0.877 (0.12))
0. 0.830 164)
0 (o.37)
TabICL (+c)
10.2
1057
0.58
0.73
28.6
0.859 1-4)
0.813
0,415
TabPFNv2
10.3
1055
0.58
0.72
24.8
0. 0.866 199)
0.823 167)
0,396
MITRA
10.7
(5:55
0.56
0.71
28.1
0.868
0.821 1636
0.40S
TahICL
11.4
(6:55
0.53
0.69
30.7
0. (.84))
0. 0.807 (716))
0 0.4222g.755)
Attic
11.7
(009
0.51
0.67
30.7
0. (.818
0.812
CatBoost
11.9
1003
0.50
0.68
32.3
0.865
0,816
0.416
MITRA (+f)
12.4
987
0.65
31.7
0 868
O (o.05
TabForesaPNN (+f)
12.6
980..1-9
0.47
0.64
33.2
0.861 (5
0.817 (19)
0.417
RealMLP
13.4
0.44
0.6
33.3
0.85 go 100)
0. 0.802 183)
0.453
TahPFNvI (+c)
14.2
0.40
0.55
36.8
0.832
0.787 (0
0 0.468
LightGBM
14.3
926.,515)
0.40
0.58
36.1
0. (.85
0.812
O 0.427 m.327333
XGBoost
14.3
924
0.39
0.57
37.5
0. (.859
0.813
0 0.429
MITRA ID
15.0
902,11/0
0.36
0.54
38.3
0.842
0. g2
0.448
Forest
15.0
901,111)
40.9
0.844 1360
m.797
O. m.540
TabPFNvI
15.0
901..999
0.36
0.51
38.4
(.829
0.785 184
0.469 199)
MLP
15.3
850
0.35
0.47
37.7
0.839
0. 0.794 1909
0 (.4
TahForestPPN
15.6
(80
0.34
0.5
40.2
0.834
0.791 0.791 (o.s))
0. 0.447 po.30)
Table 14: MITRA wins on TabZilla 10-fold classification benchmark. Winner/runner-up in +e means adding ensemble in ICL, and +f means adding fine-tuning.
Ranking Metries
Aggregated Metrics
Model
Rank
Elo 1
Winrate
RAcc
CAA
AUC
ACC
CE
MITRA (+ef)
8.6
1110 (+56-))
0.66
0.82
20.7
0,913 (1187)
0.862 (014))
0,304 (100))
Attic (+ef)
8.8
1102 (4565)
0.65
0.82
21.8
0.912 (0.33))
0.867 (0.42)
0.305 (0,02))
TabPFNv2 (+c)
(079
0.61
0.8
25.1
0.909 nu 142)
0. 0,863 144)
0.315
TahPFNv2 (+ef)
9.9
1067
0.6
0.77
25.7
0.903
0.851 (0.16)
0. go.445).342)
TabICL (+c)
10.3
1055
0.58
0.77
30.0
0.907 0.907 (0.4))
0 850 145)
0.333
MITRA (+c)
10.3
1055
0.58
0.77
30.2
0.907 (0
0 1.860 sen)
(.
Attic (+e)
10.6
(0:5
0.56
0.77
31.4
0. (.
0. 1.862 1413
(.
MITRA
11.1
1028
0.54
0.74
30.8
0.905
0.858 1409
0.329
TabPFNv2
11.2
(026
0.54
0.74
30.9
0.901 (0110)
0. 0.856
0.327 (.27m..00
TabICL
11.2
1025
0.54
0.74
32.3
(.
0.8 (4:1))
0,338 s03)
Allic
11.6
1014
0.52
0.73
33.8
0.901 gu341)
0.85 1435)
p.3400
MITRA ID (+f)
12.3
990...-9)
0.48
0.69
34,5
0.901 (0.190
0.8 0.850 146)
go
CatBoost
12.7
479..4-5
0.47
0.68
35.8
0.898
0.848 165)
0.352
RealMLP
12.7
(794))14
0.47
0.66
33.2
0. 0
0. 0.844
0. g.
TabPFNvI (4c)
12.7
0.47
0.67
35.1
0. (..92
0.837
0. 0.367,a.33)
TabForestPEN (+f)
12.8
976..3-5)
0.47
0.66
35.4
0. pp.45)
0.849
m.359
TabPFNvI
13.3
960
0.44
0.63
36.4
0. 1s
0.8 0,836mu15)
0.374,m
MLP
13.8
942,,515
0.42
0.59
37.7
0.892
0.365
XGBoost
(29,
0.59
39.4
0.892-a.14))
0.841am ma
0.367
MITRA ID
14.4
924..85)
0.39
0.61
39.8
0. 0.886
p5834
Random Forest
14.7
915,,515)
0.38
0.57
43.4
0. 158
0.836m151
q.458
TabForestPEN
14.7
913,,314)
0.38
0.59
39.8
0 0.884
0.834ma19
0,384 3489
LightGBM
14.8
911:4515)
0.37
0.55
40.4
0.879
0
m
C.4 Classification
We report detailed results for individual benchmarks in Table 13 (TabRepo), Table 14 (TabZilla), and Table 15 (AMLB), with aggregated results across all three benchmarks in Table 2 of the main text. These tables show that MITRA (+ef) has the best performance across all of these benchmark datasets, which is consistent with the aggregated results.
To further enhance performance, we apply the advanced bagging strategy introduced in Section 4.5 of the main text to both MITRA and the top-performing baseline Attic. Table 16 shows the results
37

---

Table 15: MITRA wins AMLB 10-fold classification benchmark. Winner/runner-up in / +e means ensembling in ICL, and +f means fine-tuning. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric.
Ranking Metrics
Aggregated Metrics
Model
Avg. Rank
Elo 1
Winrate t
RAce 1
CA
AUC 1
ACC 1
CE
MITRA (+ef)
5.8
1202 (+HAAT))
0.76
0.84
17.6
0.926 (0/06)
0.858 (0.124)
0.341
Attic (tef)
6.2
1186 (+10-10)
0.74
0.83
19.4
0.926 (0000))
0.857 (0.14)
0.344
TahPFNv2 (+c)
6.9
10-10
0.71
0.81
19.7
0.927 uunse
0.858 (0.124)
0.345
(+e)
7.5
(129)
0.67
0.78
23.7
(9.190000)
0.847
0. 0.360
TabPFNv2 (4ef)
7,8
-117
0.66
0.76
24.2
929 (0.07))
0.848
0.368,0.200
TahPFNy2
9.4
1059
0.58
0.74
27.1
0.99 0.9223..mm
0.85220.20
0.359)
TabICL
9.4
1058..999
0.58
29.1
0.91 470.87)
(0.120
0.372o.298)
MITRA ID (+f)
10.6
1014.-899
0,52
0,66
30.9
0.850
0.360
CalBoost
10.8
1006
0.51
0.64
33.1
0.9 0.916
0.844..126
0,376 300
TahForestPFN (+f)
10.8
(005
0.51
0.65
30.1
a.920
0.848p.120
0.364,0.255
MITRA
11.2
993
0.49
0.62
36.1
0.91 (0.055)
0.831 (o.17)
0 0.406..3500
Attic (+c)
11.3
986,
0.48
0.63
34,9
0.910
0.832
0.401
MITRA
12.6
(42
0.42
0.54
39.0
0. 909 (u.07)
0.826
0.415
RealMLP
12.6
-40-0--10
0.42
0.58
35.0
0.91 (q.022)
0.834
0,406,0.300
Attic
12.7
935
0.41
0.58
38.1
0. (.908)..09)
(.829
0.405 399
XGBood
13.1
924
0.4
0.54
38.2
0.912
0.839
0,388 306)
LightGBM
13.4
0.38
0.51
38.0
0.910
0.838p.122
0. 0.384
Random Forest
13.6
905 905
0.37
0.5
42.3
0.908
0,833
0 0.440a
MLP
14.5
869
0.33
0.46
39.9
0.8 (..7))
0.820
0.424m
MITRA ID
15.3
834 834
0.28
0.44
45.1
0.899
0.8 0.818p.1 17.147
0.423
TahForestPFN
15.5
(28
0.28
0.899
0.817,0140
0.423
of MITRA (bagging) and Attic (bagging) evaluated on the unified set of the three classification benchmark datasets. Notably, MITRA (bagging) further improves MITRA (+ef) and shows even larger gains compared to other baselines.
We additionally evaluate MITRA and baselines the TabArena benchmark { in Table 17 MITRA remains the state-of-the-art tabular models on TabArena. Specifically, MITRA achieves Pareto efficiency both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower. MITRA is the strongest single model, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. MITRA is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We leave constructing a search space for MITRA for HPO and HPO + ensemble results as future work.
C.5 Regression
In Table 18 report additional regression results larger benchmark that combines AMLB and OpenML-CTR22 with features up to 500 and rows up to 10k. This benchmark contains more datasets with larger number of rows than the other benchmark datasets. Results show that MITRA (+ef) and TabPFNV2 (+ef) have similar performances, with TabPFNv2 (+e) having the top performance. We choose to separate the two regression benchmarks (Table 4 and Table 18) to show differences in performance on these small-scale and large-scale regression datasets. These results show that MITRA performs better small-scale datasets. Its performance limitation can be explained by the fact that in pretraining it only sees up to 16 features and up to 640 rows. Notably, it can outperform TabPFNy2 some benchmarks with up to 500 features and 10k rows, despite being pretrained on one-tenth the maximum pretraining features and one-third of the maximum pretraining rows in TybPFNv2. These findings suggest that MITRA generalizes well beyond its pretraining regime. Future work includes increasing the maximum number of rows and features during the pretraining process.
this larger-scale dataset
benchmarks with up to 100 features and 3k rows, and
C.6 Critical Differences
We visualize the critical differences between MITRA and baselines across three classification bench- marks (TabRepo Figure 9 TabZilla in Figure and AMLB in Figure TT) and two regression benchmarks (TabRepo in Figure 12 and AMLB OpenML-CTR23 in Figure
38

---

10
14 1971111119992221
Mitra (+ef)
TabForestPFN
Tab (+e)
Random Forest
TaDPFNv2 (+e])
MitralD
Mitra (+e)
XGBoost
ATti (+6)
LightGBM
TaD2FNv2
RealMLP
TabiCL (+e)
TaDForestPEN (+f)
MitralD (+f)
TabiCL
Attic
Figure 9: Critical difference plot TabRepo classification benchmark.
CD
3
5
10
11
14 161111111111110
Mitra (+e0)
LightGBM
Attic (+:0
TabForestPFN Random Forest
Tab NV2 (+0)
TabPFNv2 (+e))
MitralD
Mitra (+e)
XGBoost
TabForestPFN
Mitra
CatBoost
TabPFNV2
RealMLP MitralD (+f)
TabiCL
Attic
Figure 10: Critical difference plot on TabZilla classification benchmark.
CD
7
10 11 12
15 16 18
Mitra (+ef)
TabForostPFN MitralD
Attic ef
TabPFNV2
MLP
Forest
BbPFNV2 (+ef
LightGBM
18OPFNy2
XGBoost
TabiC
Attic
RealMLP
CatBoost
Milra
TabForestPFN (+f)
Attic (++)
Mitra (+e)
Figure 11: Critical difference plot on AMLB classification benchmark.
12
TabPFNV2 (+ef)
Random Forest
Rea|MLP
Mitra
CatBoost
LightGBM
TabPFNv2 (+ef)
XGBoost
Mitra (+e)
TabPFNv2
Figure 12: Critical difference plot on TabRepo regression benchmark.
39

---

Table 16: Adding MITRA (bagging) and Attic (bagging) in the aggregated classification benchmark results of the main text. +e means adding ensemble in ICL, and +f means adding fine-tuning. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metrics are mean and std (shown in parentheses) of the corresponding metric.
Ranking Metries
Aggregated Metrics
Model
Avg. Rank
Elo f
Winrate
RAce
CA
AUC
ACC
CE
MITRA (bagging)
7.9
1135 (+3-4)
0.68
0.82
20.8
0.905 (0.128)
0.86 (0.149)
0.325 (6.18))
MITRA (+ef)
8.2
1124 (14:4)
0.67
0.82
20.7
0.904 (0.12))
0.858 (0,14))
0.327 (0.318)
Attic (bagging)
8.4
1119
0.66
0.81
22.9
0.9 0.91.131)
0.855
0.333
Attic (+ef)
8.5
11 1115
0.66
0.81
22.3
0.903 go
0.858
1.33
TahPFNv2 (+e)
9.1
(-:3)
0.63
0.79
23.8
0.901
0.856.1140
0.337
(+ef)
(1:3
0.76
25.8
(0.129
0.846
0.362
TabICL (+c)
10.7
104] 104]
0.56
0.74
31.2
0,888
0.836
0.366,a329
MITRA (+C)
11.0
-032
0.55
0.73
31.7
g.
0.847 148)
0.359
TahPFNv2
11.1
(0:3
0.54
0.73
29.6
0.89 1
0.846
0.352
Attic (+e)
11.3
(
0.53
0.73
32.2
0. 0.8
g.848
0.363
TabICL
11.9
4003
0.51
0.7
33.9
0 p4
0.833
0.373
MITRA
12.0
1001
0.5
0.68
33.5
0.891 (0844)
0.841:a115)
0.368
MITRA (+f)
12.6
(31-4
0.67
34,9
0.892
0.843
0.365
Attic
12.7
979
0.47
0.67
35.7
0. 8.88
0.834aa116
0.375
CatBoost
12.8
476-444
0.47
0.67
36.0
0. 0.888
0.837
0.374
TahForestPFN (+f)
13.0
(414
0.46
0.65
35.6
0. 0,886
0.84
0.375
RealMLP
13.6
(313))3)
0.43
0.62
35.7
0. 1..
0.827
0.41 1g,390
XGBoost
14.6
914..263)
0.38
0.58
39.8
0. 1s0
0.833
0.388
LightGBM
14.9
905
0.37
0.56
40.1
0.876
0.829 (.829
0.392
MLP
493)
40.4
1.869 145)
0.82
0.413
Random Forest
15.3
892.414)
0.35
0.54
44.5
0.874ga.4
0.822
0.47mam0
MITRA
15.6
882
0.34
0.54
42.8
0. 0.869
0.815
4g414
TahForestPFN
15.8
473)444)
0.33
0.52
42.8
0. 0.864
0.814..167
0.414
CD
3
6
9
10
11
12
TabPFNv2 (+e)
Random Forest
RealMLP
Mitra
Mitra (+ef)
MLP
TaB (+ef))
Mitra (+e)
CatBoost
XGBoost
LightGaM
Figure 13: Critical difference plot AMLB + OpenML-CTR23 regression benchmark.
C.7 Timing Efficiency
We compute the running time of MITRA and baselines on eight 40GB A100 machines. In Figure 14 we present the performance metrics (Elo, winrate, and AUC) alongside the average running time on the TabRepo benchmark. MITRA (+ef) achieves a gain of 138 Elo over CatBoost while being approximately 3.5x faster. We have additionally measured single-GPU fine-tuning performance. MITRA (+ef) on single GPU takes similar time (89 seconds) as CatBoost (83 seconds) while achieving a gain of 138 Elo. We present more details on training and inference times on the TabArena benchmark Table
C.8 Decision Boundary Visualizations
We visualize the decision boundaries of MITRA and baseline methods on a set of representative simulated datasets. Each dataset consists of 1,000 samples drawn from known ground-truth distribution, with 10% used as support samples and the remaining 90% query samples. For MITRA and TabPFNy2, we adopt the ICI. setting without ensembling. For classical models, we their default hyperparameters. Overall, MITRA demonstrates effective few-shot generalization
40

---

Table 17: Additional results on TabArena. "Single" represents single-model results without HPO. "HPO" represents results after 200-iteration hyperparameter optimization. "HPO + ensemble" rep- resents the ensemble results of hyperparameter optimization. MITRA is the strongest single model, outperforming all methods even when they perform hyperparameter tuning. "Train Time" and "Infer Time" represent the median training and inference time in seconds per KK rows. MITRA achieves
Pareto efficiency both in training time and inference time.
Model
Avg. Rank
Elo
Winrate
CA
Train Time
Infer Time
TabPFNv2 (HPO + ensemble)
6.2
1745
0.89
0.05
3445.6
48,2
Mitra (single)
74
1699
457.2
TabM (HPO ensemble)
9.9
1620
0.8
0.1
2828.4 4
1.6
TabICL (single)
10.1
1617
RealMLP (HPO+ + ensemble)
10.9
1597
0.78
0.09
6796.3
12.4
TabPFNv2 (HPO)
10.9
1596
0.78
0.08
3445.6
AutoGluonI.3 (4h)
12.6
1551
0.74
0.1
2309.2
2.6
TabPFNy2 (single)
12.7
1,448
0.74
0.1
4.1
0.4
LightCBBM cHsembl)))
13.7
1.525
0.72
0.12
647.6
1.7
TabM (HPO)
14.2
1512
0.71
0.11
2828.4
0.2
LightGBM (HPO)
16.2
1470
0.66
0.12
647.6
0.3
CatBoost (HPO + ensemble)
16.5
1465
0.66
0.12
1465.9
0.7
CatBoost (HPO)
17.3
1444
0.64
0.12
1465.9
0.1
TabM (single)
17.8
1437
0.63
0.14
10.4
0.2
CatBoost (single)
17.8
1434
0.63
0.14
5.7
0.1
ModemNCA (HPO)
1428
0.62
0.12
5944.9
XGBoost (HPO ensemble)
18.5
1420
0.61
0.13
766.1
FBM (HPO + ensemble)
20
1390
0.58
0.16
1109.1
0.2
XGBoost (HPO)
20.1
1386
766.1
RealMLP (HPO)
20.2
1383
6796.3
ModernNCA (HPO
enPooensemblee)) ensemble)
20.4
5944.9
ModernNCA (single)
20.8
1368
TorchMIP (HPO ensemble))
20.9
1370
0.56
0.14
2862.1
2.2
FastaiMLP (HPO ensemble) eHPOben)eblle))
21.1
1366
0.55
1358.6
TabDPT (single)
22.7
1329
0.52
0.15
27.5
EBM (HPO)
23.1
1323
0.51
0.17
1109.1
EBM (single)
23.8
1307
0.49
0.18
5.3
0.1
RealMLP (single)
25.6
1270
0.45
0.16
22.5
FastaiMLP (HPO)
25.6
1273
0.45
0.17
1358.6
0.9
ExtraTrees (HPO ensemble)
26.1
1261
0.44
0.18
370,9
1.5
TorchMLP (HPO)
27
1241
0.42
0.16
2862.1
0.2
XGBoost (single)
28.2
1214
0.39
0.17
2.4
0.2
ExtraTrees (HPO)
28.6
1205
0.39
0.2
370.9
0.2
RandomForest (HPO ensemble)
30.5
1159
0.34
0.2
527.4
1.4
LightGBM (single)
30.7
11599
0.34
0.18
2.9
0.1
RandomForest (HPO)
33.1
1094
0.29
0.21
527.4
0.1
TorchMLP (single)
33.4
1087
0.28
0.22
10.4
0.2
FastaiMLP (single)
34.6
1056
0.25
0.23
4.7
0.6
Linear (HPO ensemble)
35.2
1032
0.24
0.29
88.6
0.3
Linear (HPO)
36.2
1005
0.22
0.29
88.6
0.1
RandomForest (single)
36.3
1000
0.21
0.26
0.4
0.1
Linear (single)
36.8
985
0.21
0.31
2.3
0.1
ExtraTrecs (single)
37.7
952
0.19
0.28
0.4
0.1
KNN (HPO + ensemble)
42,6
716
0.08
0.48
3
0.2
KNN (HPO)
43.7
623
0.05
0.5
KNN (single)
45,2
415
0.02
0.59
0.1
As illustrated in Figure and Figure 21 when the data distribution is axis-aligned, MITRA produces more regular and less fragmented decision boundaries than TabPFNv2. This result suggests that a lower functional complexity that appears to support better generalization. On other
capabilities
representative 2D datasets- GP data (Figure 16). linearly separable data (Figure 18 Gaussian
mixtures (Figure sine waves (Figure 20) and star-shaped distributions (Figure 23 MITRA has
MMTRa achieves comparable or superior generalization performance TabPFNv2 on most datasets. One notable exception arises on the spiral dataset (Figure (2 However, we observe MITRA demonstrates greater robustness to increasing noise levels in this simulated data. In contrast, TabPFNv2 exhibits a pronounced performance drop, resembling phase transition, noise level increases.
41

---

Table 18: AMLB 10 fold regression benchmark. +e means adding ensemble in ICL., and +f means adding fine-tuning. The 95% confidence interval is shown in parentheses for the Elo. The columns in the aggregated metries mean and std (shown in parentheses) of the corresponding metric.
Ranking Metries
Aguregated Metries
Model
Avg. Rank
Elo
Winrate
RAcc
C.A. |
R2-
RMSE !
MAE 4
TabPFNv2 (4c)
4,4
1137
0.69
0.82
14.8
0.683
1720.58
1002 0022.4 (3540
ReaIMLP
(+13013))
0.77
16.4
0.685 (0.17)
1743.95 (991930)
(3
MITRA (tef)
0.63
0.79
18.3
0.678 3071
1773. y6006
1064.6 p35
TabPFNy2 (=of)
0.63
0.77
16.7
0.672
1749. (2r2e0))))
1017. 3 my
CatBoost
5,3
1077
0.61
0.75
19.1
0.678
1773, 29
1083 56
TabPENy2
(0
0.56
0. (.66
1889. (o0ss77))
1140.7 caerr 30
LightGBM
6.6
995
0.499
0.65
22,3
0.676g2211
1827 (225013)))
1140 90garh
XGBoost
984
0.47
0.65
21.7
0.670
1804,7 (561677)
1121 99,3770.50 50
MITRA (tc)
7.6
0.4
0.59
28.8
0.650
1964,38,e0669
1263. 88,118
MLP
8.3
883
0.34
0.47
28.1
0.638
2153.05
1105.97 (1,919156)
(12
0.51
g.64
1992.53 45)
1294.52 52,233000
Random Forest
9.4
805
0.24
0.37
34.9
0.639
1960.06
1233.4 1233.41
+
Elo
Winrate Miar
AUC
1150
1100
Mirr
ATAPPENI (MT
A (-P0
Mira (ref)
0.65
(r
0.88
Mire (ea)
Atk
RabreNd (ru))
Mirra (14f)
0.87
2050
Antr
Malt
0.55
BARCL
LigheGaM
CalBooi $6
2000
Anfr
Calllont
Anti
ConBont
TAMPFN
bafomers'
(10
Reahtl
0.45
TighGGRM
(-4)
28 RealML
0.85
0.84
MMlerl 28
MLP
900
MirraD
(e MLP
Miralll ML
T.FFaroFFN
TRUPFN
(s)
(5)
Figure 14: Average running time vs. performance metrics (Elo, winrate, and AUC) on TabRepo.
decision boundaries that fall between those of tree-based classifiers and the TabPFNv2 model, which highlights the effect of pre-training on a mixture of synthetic priors. In the spiral (Figure (2) and Swiss roll (Figure 24) examples, the Gaussian Process (GP) classifier shows strong performance, which motivates our future work to incorporate GP-based priors into the pretraining mixture.
D Limitations and Future Work
While our current mixture of priors demonstrates strong performance, it can be further improved by employing hyperparameter optimization (HPO) to adapt the mixture weights for specific downstream tasks or domains. In addition, we plan to incorporate other continuous priors, e.g.. Gaussian Processes. which model smooth boundaries directly into the mixture, to better generalize to tasks outside of tabular domain, e-g., time series forecasting. Lastly, although MITRA A achieves competitive results overall, it does not consistently outperform TabPFNv2 on large-feature regression tasks, and we plan to scale pretraining to datasets with larger numbers of rows and features to yield further gains in generalization to real-world, high-dimensional settings.
42

---

5OC
453
03
1700 200
Figure 15: Decision boundaries of MITRA and baselines on 2D checkerboard data. MITRA shows more regular and less fragmented decision boundaries than TabPFNv2.
MO0% ADC Iol0
Ralec
Figure 16: Decision boundaries MITRA and baselines on 2D Gaussian Process data.
MAPLIC MAN
27.009
AL BAIN 18325
KULLE
Figure 17: Decision boundaries of MITRA and baselines on 2D Julia set data.
43

---

of
Praan Wow.ea MERR Shiee.
TORENGTH"
SL.AA RED
Figure 18: Decision boundaries of MITRA and baselines 2D linearly separable data.
MOOE MARRA SOLEPT
YORTH MASTE
-13
Figure 19: Decision boundaries of MITRA and baselines on 2D nonlinear Gaussian mixture data.
DAC
Figure 20: Decision boundaries of MITRA and baselines on 2D sine wave data.
44

---

KALIO 5OC
11.0N
30
Figure Decision boundaries MITRA and baselines on 2D sinusoidal checkerboard data, MITRA shows more regular and less fragmented decision boundaries than TabPFNv2.
3008
ALL TrIP AAC.AA0-111
11
Figure 22: Decision boundaries of MITRA and baselines on 2D spiral data.
10
13 25
Figure 23: Decision boundaries of MITRA and baselines on 2D star data.
45

---

ACC ADO
ACC BATPP
Figure 24: Decision boundaries of MITRA and baselines 2D Swiss roll data (from 27
AUC (Step 2000)
ACC (Step 2000)
CE (Step 2000)
5.46
0.82
0.81
8.80
0.79
5.40
3002
12000
15020
20200
5000
10000
15000
20000
5008
10000
15000
20000
Step
Step
Step
Figure 25: Large view of Figure 5 showing only the zoomed-in region for greater clarity.
46


Reviews:
[{"review_id": "46pwR263sG", "reviewer": "Reviewer_gtzp", "summary": "The paper proposes an approach to select synthetic data priors for tabular foundation model pre-training. Specifically authors propose three criteria for prior selection - performance, diversity and distinctiveness - and design an empirical procedure to select priors that optimise these criteria. Empirical results show that when a foundation model is pre-trained with a proposed mixture better downstream generalisation results can be achieved.", "questions": "-Do you have any additional explanation for why Mitra is not beating TabPFNv2 in non-finetuned settings? For current results it does look like TabPFNv2's synthetic data strategy is superior and leads to a better \"foundation\" model base.\n-The procedure to select prior mixture in Section 3.1 is quite expensive and requires training M models. I might have missed it in the appendix but do you have run time results for this procedure and stability across different restarts?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-The paper is very well written an easy to follow. The proposed criteria for prior selection is innovative and conceptually sound.\n-The procedure to derive the mixture is novel although it does look computationally expensive.\n-Extensive empirical evaluation is provided on real-world datasets with detailed ablation study.\n\nWeaknesses:\n-I have a major concern with the empirical results. In the main result table TabPFNv2 base/e/ef would be the closest baseline to compare to for Mitra base/e/ef. However, looking at direct pairwise comparisons TabPFNv2 vs Mitra, TabPFNv2 (+e) vs Mitra (+e), and TabPFNv2 (+ef) vs Mitra (+ef), only the fine-tuned Mitra (+ef) performs better. This really calls into question the premise of the paper that the proposed procedure for prior selection leads to better downstream generalisation. The gap between non-finetuned TabPFNv2 and Mitra is consistent and quite large, particularly for the ensembled versions.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) the proposed criteria for prior selection **innovative and conceptually sound**; (2) the procedure to derive the mixture **novel**; (3) the empirical evaluation **extensive with detailed** ablation study; (3) the paper **very well written and easy to follow**. We address the reviewer’s concerns as follows.\n\n> Mitra’s ICL performance relative to TabPFNv2 does not undermine the quality of our proposed prior mixture but reflects our design choices.\n\nWe clarify below why Mitra’s ICL performance relative to TabPFNv2 reflects our design choices rather than a limitation of the prior mixture. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL [1] (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the state-of-the-art performance.\n\n1. **Performance gap re: ICL vs ensemble + fine-tuning**: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by **~30-50 Elo** in Table 2 with std only +4/-4 and **~100** over the top ICL performance.\n2. **Performance gap due to more efficient feature space**: \n    1. Mitra was pre-trained with a maximum of 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with an order of magnitude more features (up to 160 features). Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with up to 100 features. This demonstrates the strong **generalization ability** and sample efficiency of our proposed prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces. \n    2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. **On these datasets, Mitra shows better ICL performance than TabPFNv2**. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [1] and TabFlex [2]. Our focus in this work is on developing an effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.\n    3. Mitra also has better **sample efficiency** as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra(+ef)|0.851±0.137|0.805±0.184|0.433±0.387|\n|TabPFNv2(+e)|0.849±0.139|0.802±0.183|0.447±0.384|\n|TabPFNv2(+ef)|0.851±0.137|0.804±0.181|0.450±0.381|\n|Mitra(+e)|0.847±0.138|0.800±0.185|0.453±0.388|\n|**Mitra**|0.837±0.145|0.791±0.187|0.460±0.390|\n|TabICL(+e)|0.828±0.163|0.784±0.188|0.470±0.385|\n|**TabPFNv2**|0.831±0.153|0.787±0.188|0.463±0.390|\n|TabICL|0.826±0.159|0.778±0.192|0.477±0.384|\n\n|Model|Avg. Rank|Elo|Winrate|RAcc|Champ Delta|\n|:-|-:|:-|-:|-:|-:|\n|Mitra(+ef)|3.9|1057(+13/-12)|0.59|0.67|13.5|\n|TabPFNv2(+e)|4.0|1047(+12/-12)|0.58|0.66|15.1|\n|TabPFNv2(+ef)|4.0|1041(+11/-11)|0.57|0.65|15.3|\n|Mitra(+e)|4.6|990(+12/-11)|0.48|0.55|20.1|\n|**Mitra**|4.7|982(+12/-12)|0.47|0.54|22.1|\n|TabICL(+e)|4.8|976(+12/-12)|0.46|0.53|23.1|\n|**TabPFNv2**|4.9|968(+13/-12)|0.45|0.52|19.8|\n|TabICL|5.2|938(+12/-12)|0.40|0.43|25.8|\n\n3. **Pretraining on our priors enables effective fine-tuning**: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our proposed mixture of priors.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra|0.868±0.133|0.822±0.168|0.403±0.369|\n|Mitra(+e)|0.877±0.126|0.831±0.164|0.394±0.367|\n|**Mitra(+ef)**|0.882±0.124|0.837±0.164|0.370±0.367|\n|**Mitra-no-pretrain(+ef)**|0.672±0.127|0.626±0.194|0.810±0.462|\n\n4. **Diverse priors create a better model foundation for efficient fine-tuning**: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN [3], Attic [4]), these are not perfect replicas of real-world datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also computationally efficient, running **~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14**, delivering state-of-the-art performance with minimal adaptation cost.\n5. **Additional results on TabArena also show Mitra’s SOTA performance**: We have run additional comparison on a recent benchmark TabArena [5] and show that\n    1. **Mitra achieves Pareto efficiency** both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.\n    2. **Mitra is the strongest single model**, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time. \n    3. **Mitra performs favorably to all non-foundational methods**, including TabM and RealMLP.\n\n|Method|Avg. Rank|Elo|Winrate|Champ Delta|\n|:-|-:|-:|-:|-:|\n|TabPFNv2 (HPO+ensemble)|6.2|1745|0.89|0.05|\n|**Mitra (single)**|7.4|1699|0.86|0.07|\n|TabM (HPO+ensemble)|9.9|1620|0.8|0.1|\n|TabICL (single)|10.1|1617|0.8|0.07|\n|RealMLP (HPO+ensemble)|10.9|1597|0.78|0.09|\n|TabPFNv2 (HPO)|10.9|1596|0.78|0.08|\n|AutoGluon1.3 (4h)|12.6|1551|0.74|0.1|\n|TabPFNv2 (single)|12.7|1548|0.74|0.1|\n|LightGBM (HPO+ensemble)|13.7|1525|0.72|0.12|\n|TabM (HPO)|14.2|1512|0.71|0.11|\n|LightGBM (HPO)|16.2|1470|0.66|0.12|\n|CatBoost (HPO+ensemble)|16.5|1465|0.66|0.12|\n|CatBoost (HPO)|17.3|1444|0.64|0.12|\n|TabM (single)|17.8|1437|0.63|0.14|\n|CatBoost (single)|17.8|1434|0.63|0.14|\n|ModernNCA (HPO)|18|1428|0.62|0.12|\n|XGBoost (HPO+ensemble)|18.5|1420|0.61|0.13|\n|EBM (HPO+ensemble)|20|1390|0.58|0.16|\n|XGBoost (HPO)|20.1|1386|0.58|0.14|\n|RealMLP (HPO)|20.2|1383|0.57|0.13|\n|ModernNCA (HPO+ensemble)|20.4|1381|0.57|0.13|\n|ModernNCA (single)|20.8|1368|0.56|0.15|\n|TorchMLP (HPO+ensemble)|20.9|1370|0.56|0.14|\n|FastaiMLP (HPO+ensemble)|21.1|1366|0.55|0.16|\n|TabDPT (single)|22.7|1329|0.52|0.15|\n|EBM (HPO)|23.1|1323|0.51|0.17|\n|EBM (single)|23.8|1307|0.49|0.18|\n|RealMLP (single)|25.6|1270|0.45|0.16|\n|FastaiMLP (HPO)|25.6|1273|0.45|0.17|\n|ExtraTrees (HPO+ensemble)|26.1|1261|0.44|0.18|\n|TorchMLP (HPO)|27|1241|0.42|0.16|\n|XGBoost (single)|28.2|1214|0.39|0.17|\n|ExtraTrees (HPO)|28.6|1205|0.39|0.2|\n|RandomForest (HPO+ensemble)|30.5|1159|0.34|0.2|\n|LightGBM (single)|30.7|1159|0.34|0.18|\n|RandomForest (HPO)|33.1|1094|0.29|0.21|\n|TorchMLP (single)|33.4|1087|0.28|0.22|\n|FastaiMLP (single)|34.6|1056|0.25|0.23|\n|Linear (HPO+ensemble)|35.2|1032|0.24|0.29|\n|Linear (HPO)|36.2|1005|0.22|0.29|\n|RandomForest (single)|36.3|1000|0.21|0.26|\n|Linear (single)|36.8|985|0.21|0.31|\n|ExtraTrees (single)|37.7|952|0.19|0.28|\n|KNN (HPO+ensemble)|42.6|716|0.08|0.48|\n|KNN (HPO)|43.7|623|0.05|0.5|\n|KNN (single)|45.2|415|0.02|0.59|\n\n> Computational complexity and stability for prior analysis.\n\n1. **Computational complexity**: Mitra’s pretraining cost is comparable to TabPFNv2, taking around 60 hours on 8 Nvidia A100 GPUs. Our prior analysis (Table 1) and ablation study (Table 6) are used for analyzing the prior importance in our mixture of priors in Mitra and demonstrating the robustness of our principled method, and they are not part of the actual pretraining process itself. Compared with prior works such as TabICL [1], we are the first to perform such rigorous study on prior importance.\n2. **Stability**: Mitra and its ablations are stable across runs. For example, the average AUC differences is within 0.003 across different restarts. \n\n[1] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data\n\n[2] TabFlex: Scaling Tabular Learning to Millions with Linear Attention\n\n[3] Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers\n\n[4] Attic: A New Architecture for Tabular In-Context Learning Transformers\n\n[5] TabArena: TabArena: A Living Benchmark for Machine Learning on Tabular Data"}, {"review_id": "w2MYY7cCy1", "reviewer": "Reviewer_Uza4", "summary": "With the emergence of tabular foundational models (TFMs) trained on vast collections of synthetic data, this work aims to systematically identify the best combination and type of synthetic priors for effective real-world data generalisation. The authors propose three qualities of an effective prior: standalone performance on real-world data, diversity (not overfitting on itself), and distinctiveness (not covered by other proposed priors). Guided by these criteria, they assemble curated sets of synthetic priors and train MITRA, a new TFM that performs comparably or outperforms state-of-the-art TFMs across a range of classification and regression benchmarks.", "questions": "1. In L915 in A.2.1 you mention that you design the classification datasets in order to encourage a balanced distribution of labels across classes. Have you tested the model in cases where there is significant class imbalance too? Would that be out-of-distribution for the model given the design choice for the priors?\n2. Could you please give more details about the exact parameterisation of the training loss, as well as any unmentioned pre-processing techniques that might have been applied to the data?\n3. Could you please comment on how normalisation is performed on the test sets? What happens if some of the real-data shows some very large values that would be out-of-distribution if any min-max normalisation based on the training data were performed?\n4. It seems that the 2D attention is needed for SOTA performance. Up to how many nodes and rows do the authors think it is feasible to scale?", "limitations": "Perhaps one additional limitation not mentioned by the authors is the increased runtime of the best performing MITRA model (which requires fine-tuning on the training set of the target data). This might become limiting at high throughput rates.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. **Comprehensive, Systematic Evaluation** - The work involves a wide range of trained models on different combinations of priors to empirically sustain the claims regarding the effectiveness of each prior. The study is systematic and thoroughly conducted, with a wide range of ablations.\n2. **Robust Baseline Comparisons** - MITRA is benchmarked against a wide array of state-of-the-art baselines—including TabPFNv2—using multiple metrics, ensuring a fair and thorough evaluation.\n3. **Broad, Insightful Analysis** - Beyond addressing the choice of priors, the study examines sample efficiency, scaling laws (model and data size), and the effects of fine-tuning and ensembling. Taken together, these help understand what contributes to the success of the model the most.\n4. **Clear Motivation and Practical Impact** - The paper is well-written and directly addresses a relevant question in the light of the rise of TFMs: how to select synthetic data sources that genuinely improve real-world tabular model performance.\n\n**Weaknesses**\n1. **Incomplete model specification** - Although the experiments are very well explained and detailed, the model specification is not as thorough. Section B.3.1 gives some details about the transformer hyperparameters, but the authors do not give too much detail about:\n   - More detailed data pre-processing steps (e.g. are the categorical variables encoded using one-hot-encoding?),\n   - Test-set normalisation procedure, given that training uses dataset-specific min–max scaling on the support set.\n   - Exact form of the likelihood mentioned in L176 (distributional assumptions, factorisation across query points, etc.).\n2. **Reliance on Post-Training Techniques** - Performance of vanilla MITRA (no fine-tuning, no ensembling) underperforms many baselines, implying that selecting strong synthetic priors is not enough to reach SOTA. To get there, the choice of priors need to be combined with well-tuned post-training techniques. These in turn increase the average runtime of the model (which is at least 20x higher than for TabPFNv2 (+e)), as shown in Figure 14, while bringing only marginal performance improvements.\n3. **Marginal Gains vs. Compute Trade-Off** - While MITRA (+ef) is consistently the best, the 95\\% confidence intervals for Elo sometimes overlap (e.g. in Table 2), and the std of the metrics are quite high. Other considerations that are worth taking into account when developing TFMs are training costs, and runtime performance. Given that MITRA’s runtime is quite high (because of the need for fine-tuning as a consequence of maximum dimension of 16 in the training data), and given that the differences between models are sometimes small, some other models may offer better accuracy–efficiency trade-offs. \n4. **Minor figure formatting issues** - Figures 3, 4, and 5 are fairly small. Perhaps for Figure 5 at least the authors could include only the zoomed in versions?", "was_revised": true, "final_justification": "I maintain my view that this paper is of interest to the community, supported by substantial empirical evidence and thorough ablations, and therefore I retain my rating of 5 (Accept). While further work is needed to convincingly demonstrate that pre-training on lower-dimensional datasets and fine-tuning at test time is more advantageous than directly amortising over higher-dimensional datasets, I believe this paper has the potential to open an important discussion on this topic.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds the paper (1) **comprehensive and systematic** in its evaluation; (2) **robust** in baseline comparisons; (3) **broad and insightful** in its analysis; and (4) **clear in motivation** and **practical impact** and **well-written**. We address the reviewer’s concerns as follows.\n\n> Running time of post-training techniques.\n\nWhile the running time of Mitra with ensemble and fine-tuning is higher than TabPFNv2 and Mitra in ICL mode, it is still reasonable and practical for tabular prediction models. For instance, it is **~3.5 times faster than CatBoost and also faster than RealMLP** as shown in **Figure 14**. Mitra is targeted especially for smaller tabular datasets which makes the practical running time fast. In addition, Mitra (bagging) is faster than TabPFNv2 with post-hoc ensemble, while achieving the best performance with advanced ensemble techniques (**Figure 3**). \n\n\n> On performance improvement.\n\nWe added a new experiment showing that Mitra offers substantial rather than marginal performance gains on a recent tabular benchmark TabArena [1]. We show that\n\n1. **Mitra achieves Pareto efficiency** both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.\n2. **Mitra is the strongest single model**, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time. \n3. **Mitra performs favorably to all non-foundational methods**, including TabM and RealMLP.\n\n|Method|Avg. Rank|Elo|Winrate|Champ Delta|\n|:-|-:|:-|-:|-:|\n|TabPFNv2 (HPO+ensemble)|6.2|1745(+41/-39)|0.89|0.05|\n|**Mitra (single)**|7.4|1699(+46/-38)|0.86|0.07|\n|TabM (HPO+ensemble)|9.9|1620(+42/-41)|0.8|0.1|\n|TabICL (single)|10.1|1617(+46/-43)|0.8|0.07|\n|RealMLP (HPO+ensemble)|10.9|1597(+39/-40)|0.78|0.09|\n|TabPFNv2 (HPO)|10.9|1596(+38/-46)|0.78|0.08|\n|AutoGluon1.3 (4h)|12.6|1551(+45/-39)|0.74|0.1|\n|TabPFNv2 (single)|12.7|1548(+41/-43)|0.74|0.1|\n|LightGBM (HPO+ensemble)|13.7|1525(+33/-46)|0.72|0.12|\n|TabM (HPO)|14.2|1512(+38/-35)|0.71|0.11|\n|LightGBM (HPO)|16.2|1470(+40/-44)|0.66|0.12|\n|CatBoost (HPO+ensemble)|16.5|1465(+36/-38)|0.66|0.12|\n|CatBoost (HPO)|17.3|1444(+37/-38)|0.64|0.12|\n|TabM (single)|17.8|1437(+37/-40)|0.63|0.14|\n|CatBoost (single)|17.8|1434(+30/-37)|0.63|0.14|\n|ModernNCA (HPO)|18|1428(+41/-33)|0.62|0.12|\n|XGBoost (HPO+ensemble)|18.5|1420(+32/-36)|0.61|0.13|\n|EBM (HPO+ensemble)|20|1390(+32/-34)|0.58|0.16|\n|XGBoost (HPO)|20.1|1386(+34/-32)|0.58|0.14|\n|RealMLP (HPO)|20.2|1383(+33/-33)|0.57|0.13|\n|ModernNCA (HPO+ensemble)|20.4|1381(+35/-32)|0.57|0.13|\n|ModernNCA (single)|20.8|1368(+31/-32)|0.56|0.15|\n|TorchMLP (HPO+ensemble)|20.9|1370(+33/-38)|0.56|0.14|\n|FastaiMLP (HPO+ensemble)|21.1|1366(+36/-42)|0.55|0.16|\n|TabDPT (single)|22.7|1329(+38/-36)|0.52|0.15|\n|EBM (HPO)|23.1|1323(+41/-36)|0.51|0.17|\n|EBM (single)|23.8|1307(+33/-30)|0.49|0.18|\n|RealMLP (single)|25.6|1270(+36/-37)|0.45|0.16|\n|FastaiMLP (HPO)|25.6|1273(+31/-41)|0.45|0.17|\n|ExtraTrees (HPO+ensemble)|26.1|1261(+31/-31)|0.44|0.18|\n|TorchMLP (HPO)|27|1241(+33/-34)|0.42|0.16|\n|XGBoost (single)|28.2|1214(+35/-31)|0.39|0.17|\n|ExtraTrees (HPO)|28.6|1205(+39/-39)|0.39|0.2|\n|RandomForest (HPO+ensemble)|30.5|1159(+39/-38)|0.34|0.2|\n|LightGBM (single)|30.7|1159(+34/-40)|0.34|0.18|\n|RandomForest (HPO)|33.1|1094(+47/-36)|0.29|0.21|\n|TorchMLP (single)|33.4|1087(+43/-32)|0.28|0.22|\n|FastaiMLP (single)|34.6|1056(+32/-32)|0.25|0.23|\n|Linear (HPO+ensemble)|35.2|1032(+43/-35)|0.24|0.29|\n|Linear (HPO)|36.2|1005(+42/-50)|0.22|0.29|\n|RandomForest (single)|36.3|1000(+0/-0)|0.21|0.26|\n|Linear (single)|36.8|985(+49/-32)|0.21|0.31|\n|ExtraTrees (single)|37.7|952(+42/-37)|0.19|0.28|\n|KNN (HPO+ensemble)|42.6|716(+42/-52)|0.08|0.48|\n|KNN (HPO)|43.7|623(+49/-69)|0.05|0.5|\n|KNN (single)|45.2|415(+77/-96)|0.02|0.59|\n\n> Evaluate under class imbalance settings.\n\nTo evaluate robustness to class imbalance, we computed the imbalance ratio (max class : min class) across TabRepo datasets and selected 14 datasets with a ratio > 10 as imbalanced datasets. Mitra **remains state-of-the-art performance on the class-imbalanced datasets**, as shown below, for both aggregate and ranking metrics.\n\n| Model                         | AUC          | ACC         | CE         |\n|-------------------------------|------------------|------------------|------------------|\n| **Mitra (+ef)**                   | 0.875 ± 0.128  | 0.871 ± 0.132  | 0.320 ± 0.306  |\n| TabPFNv2 (+e)                 | 0.872 ± 0.132  | 0.870 ± 0.134  | 0.328 ± 0.311  |\n| TabPFNv2 (+ef)                | 0.869 ± 0.130  | 0.863 ± 0.132  | 0.349 ± 0.306  |\n| TabICL (+e)                   | 0.864 ± 0.136  | 0.859 ± 0.139  | 0.346 ± 0.314  |\n| Mitra (+e)                    | 0.868 ± 0.132  | 0.863 ± 0.137  | 0.343 ± 0.319  |\n| TabPFNv2                      | 0.865 ± 0.136  | 0.864 ± 0.138  | 0.338 ± 0.317  |\n| Mitra                         | 0.867 ± 0.130  | 0.861 ± 0.139  | 0.347 ± 0.320  |\n| TabICL                        | 0.859 ± 0.136  | 0.856 ± 0.140  | 0.353 ± 0.316  |\n\n| Model             | Avg. Rank | Elo       | Winrate | RAcc | Champ Delta |\n|:------------------|--------:|:------------------|--------:|-------------:|--------------:|\n| **Mitra (+ef)**       |     3.6 | 1081 (+13/-13)    |    0.63 |         0.74 |          11.5 |\n| TabPFNv2 (+e)     |     4.1 | 1039 (+14/-13)    |    0.56 |         0.68 |          14.4 |\n| TabPFNv2 (+ef)    |     4.3 | 1022 (+12/-13)    |    0.54 |         0.64 |          16.6 |\n| TabICL (+e)       |     4.4 | 1008 (+13/-13)    |    0.51 |         0.62 |          20.5 |\n| Mitra (+e)        |     4.7 |  979 (+12/-14)    |    0.46 |         0.58 |          19.6 |\n| TabPFNv2          |     4.9 |  964 (+13/-12)    |    0.44 |         0.55 |          18.0 |\n| Mitra             |     5.0 |  956 (+14/-13)    |    0.43 |         0.55 |          21.6 |\n| TabICL            |     5.0 |  951 (+13/-14)    |    0.42 |         0.50 |          23.6 |\n\n> Model specification.\n\n1. **Categorical feature encoding**: We use ordinal encoding for categorical features.\n2. **Normalization**: In Appendix B.3.2, we mentioned min-max normalization. We would like to clarify that this applies only to regression targets and not to input features. We will make this clear in the final version. To normalize features, we apply uniform quantile transform based on the support set, followed by standard normalization based on the mean and standard deviation from the support set. To normalize targets for regression task, we use min-max normalization based on the support set. The normalization is consistently applied for both pretraining datasets and inference datasets.\n3. **Likelihood**: We will clarify the likelihood formulation in the final version: the population version of the likelihood takes a form of an expectation over the table $D$, with each table sampled from the prior mixture. Given the sampled table, the model assumes $q$ query rows are conditionally independent given the in-context examples, so that the log-likelihood within the expectation is decomposed into a sum of $q$ individual terms, one per query row. When the query label corresponds to a classification task, its (conditional) distribution is assumed to be categorical, making the training objective equivalent to minimizing the cross-entropy loss. When the query label corresponds to a regression task, its (conditional) distribution is assumed to be Gaussian and the training objective corresponds to minimizing the Mean Squared Error (MSE) loss. \n\n> Large values in test data.\n\nAs described in model specification part, to normalize features, we apply uniform quantile transform based on the support set, followed by standard normalization based on the mean and standard deviation from the support set. To normalize targets for regression task, we use min-max normalization based on the support set. The normalization is consistently applied for both pretraining datasets and inference datasets. \n\nWe observed that the model is able to extrapolate and generate predictions beyond [0, 1] for regression predictions. For example, on Tabrepo regression 167210 across 10 folds, 6 entries are out of distribution. Mitra successfully extrapolates 4 of these 6 entries, achieving a 67% success rate, with a mean absolute error of 0.03 and mean squared error of 0.001 in the normalized space. We leave other preprocessing techniques for improved extrapolation as future work.\n\n> Up to how many rows and features to scale.\n\nThere are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [2] and TabFlex [3]. Our focus in this work is on design of effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.\n\n> Revise figure format.\n\nThank the reviewers for the comment. We will reformat Figure 3, 4, 5 to make them larger and include only the zoomed in versions in the final paper.\n\n[1] TabArena: A Living Benchmark for Machine Learning on Tabular Data\n\n[2] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data\n\n[3] TabFlex: Scaling Tabular Learning to Millions with Linear Attention"}, {"review_id": "i0k5PKqfBx", "reviewer": "Reviewer_fccq", "summary": "The paper introduces a tabular foundation model (TFM) named MITRA by attempting to carefully analyze the priors used for synthetic data generation to train the TFM. They show that carefully controlling the data distribution provides consistent gains in model performance on both tabular classification as well as regression tasks. The paper also dives into two different architectures, which attempt to highlight the model-agnostic nature of their priors.", "questions": "- Pretraining details in the main paper are missing, despite the main contribution being the development of a TFM. Can you please specify all the training details?\n- How is the prior importance color assigned in table 1?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The paper covers an important problem\n- The paper is well-written\n- Comprehensive list of experiments\n\n# Weaknesses\n- **Unclear reasons for performance gains with only specific tricks (finetuning and ensembling):** the model without finetuning and ensembling is significantly inferior to other approaches in most cases, making it hard to understand the reason for this performance gain with these tricks in place.\n- **Inconsistency in evaluation:** the '+e' and '+f' modes are selectively reported in many tables (e.g., Table 3), further masking the true results. I think it should be fine to highlight that this doesn't work in all cases, instead of trying to incorrectly convince the reader that it does.\n- **Missing training details:** the paper didn't discuss in any way the pretraining of the model including the dataset size, model parameters, training epochs, which are mentioned in passing towards the end (Fig 5). I would expect a better reading flow where the authors first specify how they pretrain the model and then talk about finetuning. It is just immediately assumed that we have a foundation model that we can evaluate, which is not ideal given that the paper is about the foundation model training.\n- **Hard to follow details in some areas:** the paper e.g., highlights that selection based on table 1 for the second best prior isn't optimal without mentioning anything else which makes it really confusing to the reader if they are missing something. They clarify this later in Table 6, but I would expect that the restructuring should make the flow of the paper much smoother to read in general.", "was_revised": true, "final_justification": "I think there is a natural dissatisfaction among reviewers due to gains only being apparent after using a bag of tricks (e.g., ensembling and finetuning). However, I still think this is a useful contribution, and therefore, after also skimming through comments from other reviewers, would recommend accepting this paper. I hope that the authors would release the model and other artifacts as the reviewer EUuF suggested, facilitating further research in this important area.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) the paper covers an **important** problem, (2) the experiments are **comprehensive**, and (3) the paper is **well-written**. We address the reviewer’s concerns as follows.\n\n> Reasons of performance gains after ensemble and fine-tuning.\n\nWe explain below why Mitra shows performance gains after ensemble and fine-tuning, which is a direct consequence of our design choices. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL [1] (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the state-of-the-art performance.\n\n1. **Performance gap re: ICL vs ensemble + fine-tuning**: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by **~30-50 Elo** in Table 2 with std only +4/-4 and **~100** over the top ICL performance.\n2. **Performance gap due to more efficient feature space**: \n    1. Mitra was pre-trained with a maximum of 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with an order of magnitude more features (up to 160 features). Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with up to 100 features. This demonstrates the strong **generalization ability** and sample efficiency of our proposed prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces. \n    2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. **On these datasets, Mitra shows better ICL performance than TabPFNv2**. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL [1] and TabFlex [2]. Our focus in this work is on developing an effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.\n    3. Mitra also has better **sample efficiency** as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra(+ef)|0.851±0.137|0.805±0.184|0.433±0.387|\n|TabPFNv2(+e)|0.849±0.139|0.802±0.183|0.447±0.384|\n|TabPFNv2(+ef)|0.851±0.137|0.804±0.181|0.450±0.381|\n|Mitra(+e)|0.847±0.138|0.800±0.185|0.453±0.388|\n|**Mitra**|0.837±0.145|0.791±0.187|0.460±0.390|\n|TabICL(+e)|0.828±0.163|0.784±0.188|0.470±0.385|\n|**TabPFNv2**|0.831±0.153|0.787±0.188|0.463±0.390|\n|TabICL|0.826±0.159|0.778±0.192|0.477±0.384|\n\n|Model|Avg. Rank|Elo|Winrate|RAcc|Champ Delta|\n|:-|-:|:-|-:|-:|-:|\n|Mitra(+ef)|3.9|1057(+13/-12)|0.59|0.67|13.5|\n|TabPFNv2(+e)|4.0|1047(+12/-12)|0.58|0.66|15.1|\n|TabPFNv2(+ef)|4.0|1041(+11/-11)|0.57|0.65|15.3|\n|Mitra(+e)|4.6|990(+12/-11)|0.48|0.55|20.1|\n|**Mitra**|4.7|982(+12/-12)|0.47|0.54|22.1|\n|TabICL(+e)|4.8|976(+12/-12)|0.46|0.53|23.1|\n|**TabPFNv2**|4.9|968(+13/-12)|0.45|0.52|19.8|\n|TabICL|5.2|938(+12/-12)|0.40|0.43|25.8|\n\n3. **Pretraining on our priors enables effective fine-tuning**: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our proposed mixture of priors.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra|0.868±0.133|0.822±0.168|0.403±0.369|\n|Mitra(+e)|0.877±0.126|0.831±0.164|0.394±0.367|\n|**Mitra(+ef)**|0.882±0.124|0.837±0.164|0.370±0.367|\n|**Mitra-no-pretrain(+ef)**|0.672±0.127|0.626±0.194|0.810±0.462|\n\n4. **Diverse priors create a better model foundation for efficient fine-tuning**: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN [3], Attic [4]), these are not perfect replicas of real-world datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also computationally efficient, running **~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14**, delivering state-of-the-art performance with minimal adaptation cost.\n5. **Additional results on TabArena also show Mitra’s SOTA performance**: We have run additional comparison on a recent benchmark TabArena [5]. **Due to character limit, we only show the top methods. The full table can be referred to in Response to Reviewer Uza4**. We show that\n    1. **Mitra achieves Pareto efficiency both in training time and inference time**, with TabPFNv2 (HPO + ensemble) being significantly slower.\n    2. **Mitra is the strongest single model**, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time. \n    3. **Mitra performs favorably to all non-foundational methods**, including TabM and RealMLP.\n\n|Method|Avg. Rank|Elo|Winrate|Champ Delta|\n|:-|-:|-:|-:|-:|\n|TabPFNv2 (HPO+ensemble)|6.2|1745|0.89|0.05|\n|**Mitra (single)**|7.4|1699|0.86|0.07|\n|TabM (HPO+ensemble)|9.9|1620|0.8|0.1|\n|TabICL (single)|10.1|1617|0.8|0.07|\n|RealMLP (HPO+ensemble)|10.9|1597|0.78|0.09|\n|TabPFNv2 (HPO)|10.9|1596|0.78|0.08|\n|AutoGluon1.3 (4h)|12.6|1551|0.74|0.1|\n|TabPFNv2 (single)|12.7|1548|0.74|0.1|\n|LightGBM (HPO+ensemble)|13.7|1525|0.72|0.12|\n|TabM (HPO)|14.2|1512|0.71|0.11|\n|LightGBM (HPO)|16.2|1470|0.66|0.12|\n|CatBoost (HPO+ensemble)|16.5|1465|0.66|0.12|\n|CatBoost (HPO)|17.3|1444|0.64|0.12|\n|TabM (single)|17.8|1437|0.63|0.14|\n|CatBoost (single)|17.8|1434|0.63|0.14|\n|ModernNCA (HPO)|18|1428|0.62|0.12|\n|XGBoost (HPO+ensemble)|18.5|1420|0.61|0.13|\n...\n|KNN (single)|45.2|415|0.02|0.59|\n\n> Evaluation consistency.\n\nTable 3 evaluates 1D attention model. The TabForestPFN baseline does not offer native ensemble logic, and the TabPFNv1 baseline does not offer native fine-tuning logic. For these reasons, we report results under the capabilities available in the respective baselines to ensure a fair comparison. We present comparisons that are meaningful and applicable to each method, and are not hiding any results. We will make this distinction on the 1D vs. 2D models clearer in the final version.\n\n> Training details.\n\nWe discussed the dataset size, model size, epochs in Figure 5 and more details in Appendix B.3. We pretrain Mitra for 22,000 steps on 45 million synthetically generated datasets. Mitra is built on a Transformer with 12 layers, 512 embedding size and 4 attention heads. The resulting model contains 72M parameters. We will adjust the paper flow to introduce these pretraining details earlier before discussing fine-tuning and evaluation.\n\n> On the second-best prior.\n\nWe have explained and compared the second and the third best prior in Line 164 - Line 169. The major reason is that the third-best prior ET shows better diversity and distinctiveness compared to the second-best prior DSRF. We also revisited this in the ablation study section in Line 277 - Line 279. We will make this more explicit in the final version.\n\n> Prior importance color in Table 1.\n\nWe use a color gradient in Table 1 to visually indicate the relative quality of each prior. The best-performing priors are shown in the darkest green, while the worst-performing priors are shown in the darkest red. Specifically, for diversity score on the diagonal of the generalizability matrix, lower values are better, with the lowest being the darkest green and the highest being the darkest red. For performance vector scores, higher values are better, with the highest being the darkest green and the lowest being the darkest red. We will make this clearer in the final version.\n\n[1] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data\n\n[2] TabFlex: Scaling Tabular Learning to Millions with Linear Attention\n\n[3] Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifiers\n\n[4] Attic: A New Architecture for Tabular In-Context Learning Transformers\n\n[5] TabArena: TabArena: A Living Benchmark for Machine Learning on Tabular Data"}, {"review_id": "raOEhxHanq", "reviewer": "Reviewer_EUuF", "summary": "The authors propose Mitra, a mixture of priors to train tabular foundational models. The authors provide detailed analysis/ablations on the different components of the mixture, how to pick them and their impact. The proposed mixture of priors yields state-of-the-art results across three diverse benchmarks when used to train tabular foundation models.", "questions": "- Could the authors provide the code (I could not find it)\n- The authors mention that they sample 50% from the SCM prior and 50% from the TBP in Line 192. This is a bit vague, how is the 50% spread out for the TBP components?\n- The authors mention that a reason why Mitra benefits more from fine-tuning, is because it was trained with synthetic datasets with up to 16 features compared to TabPFNv2 which was trained with 100. I am curious on the reasoning behind the choice of training with 16 features and not 100?\n- Was HPO performed for the baselines? It would be great if TabM was included in the experimental protocol, it is a recent work which claims SOTA results in the tabular domain.", "limitations": "The limitations are not part of the main paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- The paper is well written.\n- The authors consider diverse well-established benchmarks.\n- The proposed work achieves state-of-the-art results.\n\n**Weaknesses**:\n- The related work section can be improved, in particular the deep learning aspect of tabular methods. [1][2][3][4]\n- The code is not provided (with what I observed).\n- The novelty is limited given that prior works do use tree-based priors (TabForest, TabICL)\n- The authors provide a list of criteria on the different component selection of the mixture of priors, however, the results paint a slightly different picture (the RF prior fits the Diversity and Distinctive attributes before the ET prior, yet the performance does not improve substantially.)\n- The proposed work does not outperform the current state-of-the-art without fine-tuning. The main idea/motivation of TabFM is not training models on downstream tasks, while the authors do go back to fine-tuning. It would be great to point out the time it takes to fine-tune, although the comparison with AutoGluon and TabPFNv2 in Figure 3, does give an idea.\n\n**Typos:**\n\n- Line 227: all three datasets -> all three benchmarks\n\n**References**:\n\n[1] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T. (2021). Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.\n\n[2]  Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.\n\n[3] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34, 18932-18943.\n\n[4] Gorishniy, Y., Kotelnikov, A., & Babenko, A. TabM: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations.", "was_revised": true, "final_justification": "The authors addressed the majority of my concerns during the rebuttal.\n\n- The authors proposed to amend the incorrect information regarding the runtime of the method compared to the baselines. Additionally, the authors provided training and inference times for the methods considered.\n- Lastly, the authors provided information regarding fine-tuning on a larger number of features proving that it does not yield higher performance compared to prior works.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer finds (1) our benchmarks **diverse**, (2) our results achieve **state-of-the-art** performance, and (3) our paper **well written**. We address the reviewer’s concerns as follows.\n\n> Novelty.\n\nWe acknowledge that prior works such as TabForest and TabICL have used tree-based priors. However, these works adopt tree-based priors in a heuristic manner, without explaining why incorporating these priors are beneficial. We are the first to systematically study what makes a mixture of priors effective and identify three key factors (performance, diversity, and distinctiveness) for better generalization. In Table 1 and Table 6, we show that following our principled analysis, our proposed mixture of priors performs better than the priors used in TabForestPFN (SCM + DT) and TabICL (SCM + GB), which only contain 1 tree-based method. \n\n> On fine-tuning.\n\nWe thank the reviewer for the thoughtful feedback. We acknowledge that one motivation for TabFMs is to introduce a powerful ICL model, but it is not the only motivation or benefit. While ICL performance is crucial, a truly foundation model for practical use must also be an excellent starting point for efficient adaption. Our central goal is to build a foundation optimized for maximum adaptability, providing the most practical path to state-of-the-art performance with efficient adaptation. Mitra’s ICL performance is comparable to TabICL (concurrent work) as shown in Table 2, and by fine-tuning from a strong and efficiently adaptable foundation, Mitra ultimately achieves the best performance.\n\n1. **Performance gap re: ICL vs ensemble + fine-tuning**: We acknowledge the performance gap between ICL and ensemble. We attribute this to (a) Mitra’s pretraining uses fewer features, and (b) TabPFNv2’s use of comprehensive feature preprocessing such as quantile transform and SVD that diversifies its ensemble members. Preliminary attempts to naively increase Mitra’s pretraining features or adopt TabPFNv2’s preprocessing did not yield gains out-of-the-box, highlighting a complex interplay between prior mixture, architecture, and training dynamics. We leave a systematic exploration of this as future work. Note that our primary focus is getting the best performance with minimal adaptation, to that end, Mitra sets a new state-of-the-art with fast fine-tuning, significantly outperforming TabPFNv2 (+e/+ef) by **~30-50 Elo** in Table 2 with std only +4/-4 and **~100** over the top ICL performance.\n2. **Mitra challenges SOTA in ICL mode with fewer pretrained features**: \n    1. Mitra was pre-trained with <= 16 features for pretraining computational efficiency, whereas TabPFNv2 was pre-trained with <= 160 features. Despite this discrepancy, Mitra achieves ICL performance comparable to TabPFNv2 on downstream datasets with <= 100 features. This demonstrates the strong **generalizability** and sample efficiency of our prior mixture, allowing the model to learn more robust patterns that generalize better to unseen, larger feature spaces. \n    2. To further validate that the performance gap is not an effect of our prior mixture, we evaluate Mitra and TabPFNv2 ICL on small datasets with <= 16 features and <= 2k rows. **On these datasets, Mitra shows better ICL performance than TabPFNv2**. \n    3. Mitra also has better **sample efficiency** as demonstrated in Section 4.4 and Table 5, making it highly effective in real-world scenarios with limited labeled data and an overall best choice across both low- and high-label regimes.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra(+ef)|0.851±0.137|0.805±0.184|0.433±0.387|\n|TabPFNv2(+e)|0.849±0.139|0.802±0.183|0.447±0.384|\n|TabPFNv2(+ef)|0.851±0.137|0.804±0.181|0.450±0.381|\n|Mitra(+e)|0.847±0.138|0.800±0.185|0.453±0.388|\n|**Mitra**|0.837±0.145|0.791±0.187|0.460±0.390|\n|TabICL(+e)|0.828±0.163|0.784±0.188|0.470±0.385|\n|**TabPFNv2**|0.831±0.153|0.787±0.188|0.463±0.390|\n|TabICL|0.826±0.159|0.778±0.192|0.477±0.384|\n\n|Model|Avg. Rank|Elo|Winrate|RAcc|Champ Delta|\n|:-|-:|:-|-:|-:|-:|\n|Mitra(+ef)|3.9|1057(+13/-12)|0.59|0.67|13.5|\n|TabPFNv2(+e)|4.0|1047(+12/-12)|0.58|0.66|15.1|\n|TabPFNv2(+ef)|4.0|1041(+11/-11)|0.57|0.65|15.3|\n|Mitra(+e)|4.6|990(+12/-11)|0.48|0.55|20.1|\n|**Mitra**|4.7|982(+12/-12)|0.47|0.54|22.1|\n|TabICL(+e)|4.8|976(+12/-12)|0.46|0.53|23.1|\n|**TabPFNv2**|4.9|968(+13/-12)|0.45|0.52|19.8|\n|TabICL|5.2|938(+12/-12)|0.40|0.43|25.8| \n\n3. **Diverse priors create a better model foundation for efficient fine-tuning**: The synthetic priors, particularly tree-based ones, are designed to capture decision boundaries. As suggested by prior works (e.g., TabForestPFN, Attic), these are not perfect replicas of real datasets but are ideal for creating a foundation that can be effectively adapted. The diversity of our mixture gives rise to a better model foundation than TabPFNv2, and the gap could not be closed by TabPFNv2 even after extensive fine-tuning, as noted by its fine-tuning performance in Table 2 and confirmed with its authors. More importantly, this aligns with what users care about: achieving the best predictive performance on their datasets efficiently. Mitra fine-tuning is not only highly effective but also efficient, running **~3.5× faster than CatBoost and faster than RealMLP as shown in Figure 14**, delivering state-of-the-art performance with minimal adaptation cost.\n4. **Pretraining on our priors enables effective fine-tuning**: Our mixture of priors provides a critical foundation that enables effective fine-tuning. To assess the impact of pretraining, we compare Mitra (+ef) with an identical model trained directly on the downstream dataset without any pretraining (denoted as Mitra-no-pretrain (+ef)). For Mitra-no-pretrain (+ef), we conducted a hyper-parameter search over training epochs {50, 100, 200} and maximum learning rates {0.01, 0.001, 0.0001}, reporting its best performance as follows. Training from scratch performs significantly worse, highlighting the importance of pretraining on our mixture of priors.\n\n|Model|AUC|ACC|CE|\n|-|-|-|-|\n|Mitra|0.868±0.133|0.822±0.168|0.403±0.369|\n|Mitra(+e)|0.877±0.126|0.831±0.164|0.394±0.367|\n|**Mitra(+ef)**|0.882±0.124|0.837±0.164|0.370±0.367|\n|**Mitra-no-pretrain(+ef)**|0.672±0.127|0.626±0.194|0.810±0.462|\n\n> HPO for baselines and add TabM.\n\nWe have added results on a recent benchmark TabArena [5], where all baselines are performed with HPO, including TabM. We show that Mitra remains the state-of-the-art tabular models on TabArena. **Due to character limit, we only show the top methods. The full table can be referred to in Response to Reviewer Uza4**. Specifically,\n\n1. **Mitra achieves Pareto efficiency** both in training time and inference time, with TabPFNv2 (HPO + ensemble) being significantly slower.\n2. **Mitra is the strongest single model**, outperforming all methods even when they perform hyperparameter tuning for 200 iterations. Mitra is only outperformed once the hyperparameter configurations of TabPFNv2 are ensembled together. We are still working on constructing a search space for Mitra, and thus do not have a HPO and HPO + ensemble result at this time. \n3. **Mitra performs favorably to all non-foundational methods**, including TabM and RealMLP.\n\n|Method|Avg. Rank|Elo|Winrate|Champ Delta|\n|:-|-:|-:|-:|-:|\n|TabPFNv2 (HPO+ensemble)|6.2|1745|0.89|0.05|\n|**Mitra (single)**|7.4|1699|0.86|0.07|\n|TabM (HPO+ensemble)|9.9|1620|0.8|0.1|\n|TabICL (single)|10.1|1617|0.8|0.07|\n|RealMLP (HPO+ensemble)|10.9|1597|0.78|0.09|\n|TabPFNv2 (HPO)|10.9|1596|0.78|0.08|\n|AutoGluon1.3 (4h)|12.6|1551|0.74|0.1|\n|TabPFNv2 (single)|12.7|1548|0.74|0.1|\n|LightGBM (HPO+ensemble)|13.7|1525|0.72|0.12|\n|TabM (HPO)|14.2|1512|0.71|0.11|\n|LightGBM (HPO)|16.2|1470|0.66|0.12|\n|CatBoost (HPO+ensemble)|16.5|1465|0.66|0.12|\n|CatBoost (HPO)|17.3|1444|0.64|0.12|\n|TabM (single)|17.8|1437|0.63|0.14|\n|CatBoost (single)|17.8|1434|0.63|0.14|\n|ModernNCA (HPO)|18|1428|0.62|0.12|\n|XGBoost (HPO+ensemble)|18.5|1420|0.61|0.13|\n...\n|KNN (single)|45.2|415|0.02|0.59|\n\n> About RF prior.\n\nAs we explained in Line 279 to Line 282, while RF shows good diversity and distinctiveness, it also shows the worst performance on real datasets, so it is not as beneficial in the mixture. We will explain this more clearly in the final version.\n\n> Code availability.\n\nThe fine-tuning and inference code is already publicly available (at the time of submission the code was going through our institutional legal review as we mentioned in the checklist). However, given the rebuttal policy we are not allowed to provide an anonymous external link. We will ensure the code repository is included and accessible in the final version. Our pretraining code is currently under institutional legal review process, and we plan to release it once this review is complete.\n\n> The reason of training with 16 features instead of 100 features.\n\nMitra was pre-trained to maximize fine-tuning performance with larger model capacity compared to TabPFNv2. Given computational constraints scaling up to larger features during pretraining requires smaller batch size and eventually leads to worse training efficiency and performance. There are concurrent papers on exploring architectural advances to scale up rows and features such as TabICL and TabFlex. Our focus in this work is on design of effective mixture of priors, and we plan to integrate these scalable architectures as future work to further improve scalability.\n\n> Related work.\n\nWe will revise the related work section to better cover deep learning methods and add references [1][2][3][4] in the final version.\n\n> Prior ratio for TBP.\n\nAs stated in Line 193, we evenly split the ratios among tree-based priors, each taking 10%. We will explain this more explicitly in the final version.\n\n> Typo.\n\nWe will revise “all three datasets” to “all three benchmarks” in the final version.\n\n[5] TabArena: A Living Benchmark for Machine Learning on Tabular Data"}]

Author rebuttal:

Meta-review: Claims and findings: The paper studies what makes synthetic priors effective for tabular foundation models trained via in-context learning. It proposes a few criteria, such as a prior’s standalone performance on real data, diversity, and distinctiveness. These criteria are used to curate a mixture of priors. This mixture is used to train a tabular foundation models that perform quite well. 

Strengths: It’s nice to have a systematic analysis of synthetic prior design for this tabular setting. The paper’s main problem is timely; tabular foundation models are increasingly popular. The results are generally pretty good, and the analysis is extensive. 

Weaknesses: Probably the main weakness is that the basic formulation doesn’t always perform better than a strong baseline, and it becomes necessary to fine-tune for these gains. 

Decision: Overall this is a nice combination of analysis, systematic  experimentation, and solid empirical results. It also sheds some light in an important area. Based on this, the paper is worth accepting. 

Rebuttal discussion: The discussion was quite extensive. A bunch of points were clarified, which will improve the final version of the paper.
Decision: Accept (poster)