Mixture of In-Context Prompters for Tabular PFNs
Authors: Derek Qiang Xu
F Olcay Cirit
Reza Asadi
Yizhou Sun
Wei Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Prior-Fitted Networks
Tabular Learning
Sparse Mixture of Experts.
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a mixture of prompters technique for tabular in-context learning.

Abstract: Recent benchmarks find In-Context Learning (ICL) outperforms both deep learning and tree-based algorithms on small tabular datasets. However, on larger datasets, ICL for tabular learning suffers in both efficiency and effectiveness. In terms of efficiency, transformers incur linear space and quadratic time complexity w.r.t. context size. In terms of effectiveness, contexts at inference encounter distribution shift compared to contexts from pretraining. We propose MixturePFN, which extends Sparse Mixture of Experts to the state-of-the-art ICL for tabular learning model. Specifically, MixturePFN finetunes a specialized ICL expert on each cluster of tabular data and routes new test samples to appropriate experts at inference. MixturePFN supports constant-size contexts by splitting large training datasets into more manageable clusters. MixturePFN addresses distribution shift by finetuning an expert on each training dataset cluster via bootstrapping. Extensive experimental results shows MixturePFN outperforms 19 baselines both in mean rank and as the Condorcet winner across 36 diverse tabular datasets under both accuracy and F1 score with statistical significance.

Published as a conference paper at ICLR 2025
MIXTURE OF IN-CONTEXT PROMPTERS FOR TABULAR PFNs
Derek Xu
Olcay Cirit
Reza Asadi
University of California, Los Angeles derekqxu@ucla edu
Uber Technologies Inc
Uber Technologies Inc
Yizhou Sun
Wei Wang
University of California, Los Angeles
University of California, Los Angeles
ABSTRACT
Recent benchmarks find In- Context Learning (ICL) outperforms both deep learn- ing and tree-based algorithms on small tabular datasets. However, on larger datasets, ICL for tabular learning suffers in both efficiency and effectiveness. terms of efficiency, transformers incur linear space and quadratic time complexity w.r.t. context size. terms of effectiveness, contexts at inference encounter distri- bution shift compared to contexts from pretraining. We propose MIXTUREPFN, which extends Sparse Mixture Experts to the state-of-the-an ICL for tabular learning model. Specifically, MIXTUREPEN finetunes specialized ICL expert on each cluster of tabular data and routes new test samples to appropriate ex- perts inference. MIXTUREPEN supports constant-size contexts by splitting large training datasets into more manageable clusters. MIXTUREPEN addresses distribution shift by finetuning an expert on each training dataset cluster via boot- strapping. Extensive experimental results shows MIXTUREPFN outperforms 19 baselines both in mean rank and as the Condorcet winner across 36 diverse tabular datasets.
INTRODUCTION
Tabular data is popular data format across various domains, consisting of column-wise features and row-wise data samples. Each feature can be either continuous, categorical, or ordinal. Thanks to the prevalence of relational databases, which ensure data integrity, consistency, and low redun- dancy, tabular data is widely used across various domains such as medicine, finance, and advertising. Hence, improving learning algorithms on tabular data is of interest to many researchers.
General tabular datasets remain unconquered by most deep learning algorithms (Popov et al. 2019 Gorishniy et al. [202]] Somepalli et al. 2021 Arik & Prister 2021 Yamada al.) [2200] Yoon al. 2020 Chen et al.) 2022). Instead, gradient- boosted decision trees (GBDTs) (Chen & Guestrin, 2016 Prokhorenkova et 20018), achieve better overall performance on tabular benchmarks (McEl- fresh et al.) 2023] ShwartzzZZv & Armon (022) considering wide range of number of samples, numbers of Teatures, feature types, and Teature distributions. Recently, transformer-based prior- fitted networks, PFNs (Hollmann et al.) 2022). have garnered interest, for their surprisingly strong and state-of-the-eee performance on small tabular datasets with < 000 samples (Hollmann al., 2022 McElfresh et al. 2023).
Unlike SGD-based deep learning, which trains model on each downstream dataset, PFNs learn the underlying patterns behind how downstream training datasets are generated. Specifically, PFNS are first pretrained on datasets generated from handcrafted dataset prior (Müller et al., 2021 Von Os- wald et al. 2023). Then, during inference, PFNS predicts test samples by modeling the underlying generative process of the observed training dataset. In this sense, PFNS are a form of In-Context Learning (ICL) (Brown et al. 2020), where the context consists of downstream training samples. In practice, PFNS are modeled as a transformer, where each dataset sample is converted into a token and each dataset converted into prompt. By pretraining multiple datasets generated

---

Published as a conference paper at ICLR 2025
by the handerafted prior, PFN learns better inductive bias than both conventional deep learning algorithms and gradient-boosted decision trees.
One reason PFN performance drops on larger datasets is due to how it handles efficiency bottle- necks. First, PFN inference is computationally expensive. Because the entire dataset is fed to the transformer "prompt". the inference time and space complexity for Narain training samples
(NNraa))) (Dao et al. 2022 Dao 2023) and O(N respectively. This is in stark contrast to GBDTs and traditional deep learning approaches where the inference time and space complexity is (1). To keep inference time low, existing works (Hollmann et al. 2022 McElfresh et al. 2023) randomly sample from the training dataset form the "prompt." However, randomly sampling the training dataset could throw away useful training samples, leading to information loss in the context.
Another reason PFN performance drops could be distribution shift between the inference-time con- text dataset and the pretraining dataset generating prior. Because the prior is handcrafted, large real-world datasets could follow different underlying dataset generating process than the one seen during pretraining. If this is the case, additional finetuning computation is required to align the pre- training and inference-time dataset generating priors. We justify such computation by comparing against baselines that also perform finetuning on downstream data.
In this work, we analyze recent claims (McElfresh et al. 2023) on PFN's effectiveness, finding PFN performance does not scale w.r.. dataset size. We improve PFN's performance w.r.t. dataset size by proposing Sparse "Mixture of In-Context Prompters" (MICP), which creates scalable "prompts" prompter specialized context relevant said test sample. We
by routing new test samples solve PFN's distribution shift limitations with "Context-Awar PFN" (CAPFN), which finetunes PFNS for downstream datasets via bootstrapping. We call our combined model MIXTUREPEN. To summarize:
To improve scalability, we are the first to propose Sparse Mixture of In-Context Prompters (MICP) which routes new test samples to pool of scalable prompters for In-Context Learning. MICP efficiently routes informative context to downstream testing points with O(1) memory and (logg(Nerain) time complexity, w.r.t training dataset size, Ntrain
To improve performance, we finetune Context-Aware Prior-Fitted Network (CAPFN), which aligns pretrained PFNs with inference-time data using novel bootstrapping pol- icy.
MIXTUREPFN scales transformer PFNs to tabular datasets with much larger number of samples, encountering minimal performance deterioration w.r.t. dataset size.
MIXTUREPFN both achieves the highest mean rank with statistical significance and is the Condorcet winner across 36 diverse tabular datasets against 19 strong deep learning and tree-based baselines. We will release our code on Github.
2 PRELIMINARIES
We consider tabular classification problems, where the inputs numerical, ordinal, categorical columns encoded as a d-dimensional feature vector, x E Rd the output is the corresponding label,
y E [l...., CJ. and the dataset consists of labelled input output pairs, = (x)(()) ()())) ) Given
the training dataset, Divain = {(=(1) (t) ).N and test samples, X [- lestlio0
(3) 1 Newst our
goal is to correctly predict the corresponding test labels, Yeest [Brest
(3) Niest
MIXTUREPEN is
inspired by Prior Fitted Networks, which we first introduce.
2.1 PRIOR FITTED NETWORKS
Prior Fitted Network (PFN) Müller et al., 2021) is parameterized model, qe. with weights, 8, that learns to approximate Bayesian inference given the dataset prior, p(D), via In-Context Learn- ing (ICL) (Brown et al. 2020). Specifically, PFN inference approximates the posterior predictive
We acknowledge PFN also encounter scalability challenges in terms feature count and label size, hut we choose to tackle dataset size first, as it is the most apparent scalability bottleneck.
We provide table with all math notations in the Appendix.
2

---

Published as a conference paper at ICLR 2025
distribution (PPD), Poly|æ, D) J P(yla, S o) ()p(()))p(()de) where & is the hypothesis mechanism behind how the tabular data is generated. For example, á can be structural causal model. Refer to Bayesian inference transformers (Müller et al. 2021) for further details.
2.1.1 PRETRAINING
To approximate the PPD, PFNS pretrained to minimize KL-Divergence between the param- eterized model, To (ylaz, a, D), and the PPD, p(y|æ, D), over the dataset prior, p(D), which was proven equivalent to optimizing the prior data negative log likelihood, LPEN- As shown in Equa- tion (Müller et al. 202T). this loss iteratively samples new datasets from a handcrafted dataset prior, p(D), via Monte-Carlo.
LPEN
[olog (vlor, D))]
s. D...D-
TABPEN (Hollmann et al.) 2022 is the tfae-offthhe-rrt pretrained PFN transformer for tabular data.
It treats the hypotheses, as randomly sampled structural causal models (SCM) (Pearl [209 PPtters)
et al. 2017) mixed with the original Bayesian Neural Network prior (Müller et al. 2021). )). Training
dataset samples are generated by first sampling a SCM graph, Φ p()), followed by sampling the
SCM, x. D pD)()))
Transformer- based Vaswani et al.
2017) PFNs tokenize the sampled dataset, (ar, D) as input to the parameterized model, qo, as shown in Figure discussed in Section 2.1.2 Note, PFN inputs are
analogous to "prompts" from In-Context Learning (ICL) (Brown et al. 2020 Dong al. 2022 Xu
et al. 2024), hence they are called "prompts" in this work.
2.1.2 INFERENCE
During inference, transformer-based PFNS tokenize the downstream dataset, (Xtest: Dtrain), into batched "prompts", consisting of N Naziin encoder tokens and Nbatch decoder tokens, where each data sample corresponds with one token Because tabular columns are permutation invariant, TABPFN shuffles feature orderings and scalings, running qo on each permutation of the "prompt", then re- turning ensembled prediction. We provide illustrations of this process in the Appendix. TABPFN does not perform finetuning, only inference, on downstream datasets.
2.1.3 EFFICIENCY LIMITS EFFECTIVENESS
Full-prompt TABPFN cannot scale datasets with large numbers of training samples (McElfresh et al. 2023). Because the prompt size is directly correlated with the training dataset size, Ntrains encoding the entire training dataset into TABPFN's prompt incurs significant compute costs. Hence, when the training dataset is too large, existing works randomly sample out of Ntraán train sam- ples to form the context, leading to information loss. We call this prompting approach TABPFN Most existing works (Hollmann et al. 2022 McElfresh et al. 2023 set = 3000.
TABPFN has important distinction in its batching protocol. Natively, TABPFN forward pass is less efficient than Large Language Models (LLMs) because each prompt is duplicated across N Nensemile feature shufflings and scalings ensure table column invariance. To counteract this, TABPFN forms more efficient prompts, fitting Nbatch test samples and the context in the same prompt. In contrast, In-Context Learning (ICL) with LLMs (Liu et al., (202)) assigns unique prompt and context to each test sample. Hence, TABPFN balances its N Nensemble x slower inference by storing Nhateh x more efficient prompts. As consequence, PFN prompting strategies must support PyN-style batching, where multiple test points share the same context, to perform efficient inference.
Finally, existing works (Hollmann al. 2022 McElfresh et al. 2023) utilize TABPFN for zero- shot inference. While this brings obvious efficiency benefits by removing reliance on downstream training data, TABPEN assumes minimal distribution shift between prefraining and inference. Furthermore, because TABPEN trained on datasets sampled from a dataset prior, not training samples sampled from a pretraining dataset, finetuning TARPFN is nontrivial.
Our dataset is split into train/dev/test sets. During hyperparameter tuning, decoder tokens are taken from the dev set instead.
3

---

Published as a conference paper at ICLR 2025
Thar
Age
regit
Page
D2
97p
122
15
19
31
85
PPp
44
22
62g
26
Eupa
Prompara
Frampler
Pramppor'
13
377
179
123M
ACENI
CAPN
11.
113
2831
9gg
CSPM
30382
05
552
9g
30000
294
Figure 1: Illustration of MIXTUREPFN, MICP (Left): New test samples are passed to a router that picks 1 out of K prompters to form scalable "prompt" with B training samples for the downstream PFN model. CAPEN (Right): TABPFN is frozen, fitted with adapters, then finetuned using data prior negative loss likelihood, Equation simulates the MICP inference mechanism. The finetuned model is called CAPEN.
|
our bootstrapped data prior, p Derain). This prior
3 METHOD
To improve both the efficiency and effectiveness of TABPFN (Hollmann et al. 2022). we propose method to combine Sparse Mixture of Experts (Shazeer et al. 2017 Lewis al. Prior-Fitted Networks (Müller et al. 2021): MIXTUREPFN.
2021) with
MIXTUREPEN is composed of 2 parts: Mixture of In-Context Prompts (MICP), which define how to seperate large training datasets into manageable clusters, and Context-Aware Finetuning (CAPFN), which finetunes an expert PFN model on each specialized cluster. In terms of efficiency, MICP is an efficient inference-time prompting technique. Thus, MICP, in isolation, performs zero- shot inference like TABPFN (Hollmann et 2022).
Under MICP, we hypothesize distribution shift exists between pretraining and inference. To address said distribution shift, we use bootstrapping to finetune each MICP cluster's transformer model. We find this additional finetuning cost is necessary to close the gap between TABPFN (Hollmann et al., 2022) and other finetuning algorithms, like XGBooST (Chen & Guestrin 2016).
3.1 MIXTURE OF IN-CONTEXT PROMPTERS (MICP)
The goal of MICP is to construct "prompts" that retains more information than TABPFN* random sampling policy. Inspired by Sparse Mixture of Experts (Shazeer et al. 2017 Lewis et al.. I where each test sample is routed to an specialized expert trained on a subset of the training dataset, Sparse Mixture of In-Context Prompters, MICP, routes each test sample to one of K "In-Context Prompters" (ICP), (T& k= specializing
cluster of the training dataset, using routing module, R Rdd + (0. (O.....A K 1}. Each ICP then constructs relevant "prompt' for incoming test samples, which are sent to the downstream PFN model batch. Specifically, ICP concatenates incoming test samples, {=test R(x (i)
k.). with its training cluster context, D Dprompt (k) C Dirain to ( k))) (fat()) R( = k)), D Drrompe (k))).
form the scalable "prompt": Te( R(a
(i)
Because larger clusters improve PFN performance (Hollmann et al.) 2022, McElfrrsss et al. 2023) the cost of efficiency (as we show in Section (53), existing works McElffres et al.) (023) adopt fixed prompt size at B = 3000. For fair comparison, we also follow this setting, where each prompter's context collects a training cluster context of fixed size, [k)) B.
Compared to TABPFN* which construct random global context for each test point, MICPconstructs prompts with relevant local context for each test point. Hence, MICP retains more useful information than TABPFN* To construct local clusters, we run K-Means on the training dataset: (D) K (2 cluster
k= (Foenter)
KMEANs(DDraa)) To construct contexts of
4

---

Published as a conference paper at ICLR 2025
Condorcet Statistics
Algo.
Top-10 Algo.
Method
#Votest
#Winst
#Ties
#Losses
Mcan = Std Rank/
MixturePFN
524
19
2.350 + 1.824
2.273 + 1.7106
XGBoost
500
18
5.500 + 4.621
4.000 = 2.663
CatBoost
474
17
2
4.900 4.158
3.955 = 2.688
SAINT
408
16
3
8,300 + 5.367
4.045 = 1.965
TabPFN*
381
4.550 = 2.747
4.040 1.311
LightGBM
373
14
4
9.150 = 4.351
6.409 = 2.839
DANet
312
9.050 3.369
7.045 1.988
FTTransformer
294
12
7
8.600 + 3.541
6.773 + 2.235
ResNet
286
8.400 + 3.262
6.864 1.961
SVM
285
9
10
11.300 + 4.766
7.500 = 2.482
STG
284
10
11.900 4.549
Fonso
247
7
12
11.600 = 4.443
NODE
243
12
13.350 3.410
MLP-rtdl
227
14
10,800 5.046
TabNet
210
5
14
13.550 5.296
LinearModel
202
15
12.400 4.652
MLP
191
5
13
13.700 3.621
VIME
134
17
15.350 3.851
DecisionTree
114
18
16.800 + 3.881
KNN
18.450 1.936
Table I: MIXTUREPEN is the Condorcet winner across 36 datasets against 19 baseline algorithms. MIXTUREPFN achieves the top mean rank across 20 datasets where all algorithms successfully and across 22 datasets where all Top-10 10 algorithms successfully run. To break ties, rank algorithms based on their mean ligliieessoods following TABZILLA (McHlfresh et al. (023). We report the Condorcet matrix, dataset breakdowns, and accuracy-metrre results in the Appendix.
desired size, expand K-Means clusters with less than B samples, KNN(x. (k)
(F I Dtrains B),
and subsample K-Means clusters with more than B samples, SAMPLE(D cluster: B) to form the context, D prompp()). Our router assigns test points to the nearest cluster center: R(x)
(Z)( (altaaaae )= MICP supports PFN-style batching by grouping test points that be- R(=(()) batch) R(x(() batthh Vii into a batched test "prompts":
long to the same ICP cluster:
(Xbatch, Dphompe) where multiple test points, Xbatch S. Xtest share the same context.
In total, router and prompter initialization takes OtNNtrain K (Ntrain + K Bl)logNNtaain time and ONNraai + K KB) space complexity and is done once before inference. Routing takes (log time and (1) space complexity, using efficient nearest neighbor search with ball-tree for each test sample. PFN transformer inference takes O(B² + BN time and space complexity, as MICP prompts contain at most B training samples and Nbatch |tbathh testing samples. We provide time and space complexity details in the Appendix. We illustrate MICP in Figure
3.1.1 EFFICIENCY AND EFFECTIVENESS TRADE-OR OFF
The effectiveness of MICP prompts depend on the number of ICPs used, K. As the complex- ity and size of data increase, more ICPS are needed to capture the entropy of the labels. This is natural each router's support set, D Drrompe (R( )), should be representative of test samples routed to that cluster, (NN ( B). which we call the support set. If the true support set becomes more granular as the dataset size increases. more ICPS are required maximize overlap: prrompt (R()))) ) n I B)).
We theoretically characterize this relationship between K. B. and overlap by analyz- ing conditions required for nonzero overlap on the training data: |Drrompe (R(2))) (R( n (NN (atriaa Dtrain: B)| Vi (0,... Ntrain Specifically, encourage nonzero overlap
5

---

Published as a conference paper at ICLR 2025
0.4
Dft. Log Linethood
(Mictureft PERfNN68399
0.4
0.3
Liathood
0.2
25d Kurtosts
10"
MixturoPFN ThoPFN
-.2
101
SAINT
10-1
103
10a
-0.
10
10:
10a
10a
#Samples
ASamates
#FFoats
(a) Ntrain W.E.t TABPFN (b) Nevain w.r.t. baselines
(c) Kurtosis and #Fcat
Figure 2: (a): We plot the difference Log Likelihood between MIXTUREPFN and TABPEN for each dataset of size Ntrain. MIXTUREPFN substantially improves the performance and TABPFN and runs on datasets with > 3, 000 samples. (b): We plot the Log Likelihood of the top deep learning (DL) PFN, and tree baselines across all 36 datasets and the best-fit line between rank and dataset size, compared to the top baseline. Unlike TABPFN, MIXTUREPEN maintains its good performance as the size of the dataset increases. (c) : We plot the best among the top DL. PFN, and tree baselines on all 36 datasets across different dataset properties. MIXTUREPEN performs well across different dataset irregularities. We provide further breakdowns in the Appendix.
by scaling the number of "prompts", K, linearly with the size of each "prompt", B, and training dataset size, [Ntrain as stated in Theorem 3.1.1 K 2 [Nevain/
[Nonzero Overlap] If every K-Means cluster contains most B samples, ID claster S B vk
10..... K -1] and training points route to their assigned K-Means cluster R (z = k train E
(i)
D duster then nonzero overlap on the training data is guaranteed, |D prommpe (R (z (R"
KNN Ptaai Dtrain: B)1 1 Vi E (N.r.in 1] VDD
This insight allows MIXTUREPEN to trade-off efficiency and effectiveness with a single hyper- parameter, 7 which controls the number of ICPs as ratio of training and support set sizes: K = [yNeraah BB) Intuitively, larger y improves effectiveness at the cost of efficiency. Assum- ing fixed 7, Nbatch, and B, MIXTUREPFN routing takes (log(Neraasn time and ((1) space complexity, and PFN inference takes (11) time and space complexity.
3.2 CONTEXT-AWARE FINETUNING (CAPFN)
The goal of CAPFN counteract distribution shift by specializing each prompter on its assigned context via parameter efficient finetuning. PFNs are pretrained on the ICL task over a synthetic dataset prior, p( D) p( ()))(())))))))))))))))))))))(() (Müller et al. 2021 works which aligns Large Language Models on ICL 2022 Wei et al. 2021 Gu et al. 2 we argue the pretraining data prior, p(D)
Hollmann al., 2022). Inspired by recent "prompts" via finetuning (Thoppiaan et al.) (), different than the true data generating mechanism during inference, p propoop Derain). which was described in Section 3. This leads to distribution shift. To better align the parameterized model, qo, with the inference-time dataset, D) Pprampt- CAPFN uses bootstrapping on the downstream dataset, Dtrain, to simulate ICI. "prompts": (X subtest, Y Ysubtest: Dsubtrrain) ~ p( (D] Derain). where Y train- and D Duubtrain C Dirain- Bootstrapped samples
X subtest C X Xtrains Yeubtest to tune adapters (Houlsby et al., 2019) via prior data negative log likelihood loss, as shown in Equation] [ except the dataset prior is now the bootstrap mechanism: p(D) ((Daai))))..
used
"Thse conditions can be satisfied via constrained K- Means Bradley et al. 2000). which ensures each cluster has at most B entries, and router that sends train points to their assigned clusters. In practice, we find the relationship with the tunable parameter 7 also holds for MIXTUREPEN's regular K-Means and Nearest- Neighhor Search router.

---

Published as a conference paper at ICLR 2025
10
9
8
6
5
3
2
SVM
MixturePFN
Frrransformer
CatBoost
DANct
XGBoost
TabPFN
SAINT
LightGBM
Figure 3: Wilcoxon-Signed Rank Test shows MIXTUREPFN significantly outperforms the Top- 10 baselines the 22 shared datasets. To break ties, rank algorithms based on their mean log-likelihoods following TABZILLA (McElfresh et al. 2023). We compute the rank across all 10 vross-validation splits. We report additional critical difference diagrams in the Appendix.
3.2. BOOTSTRAPPING LARGE MICP DATASETS
The bootstrap procedure mimics MICP on large Mtrain 3000 datasets: p(D)Dtrain) s(ppupprrs xx) (aaozDDrreee Dtrain). Specifically, we sample random training point from the training dataset, x ~ p(a] Dtrain), then run K-Nearest Neighbors from the sampled point, p P(D support [2) KNN Dtrain:
(trriin B), as defined in Section 2.1.3 to obtain a bootstrap dataset, Dhontstrop. We ran- domly split the bootstrapped dataset D Dbootstrap p(D|Drrain) into train/test splits to obtain the "labelled prompt", (Xeubtest, Yeubtest, D Deubtrain).
3.2.2 BOOTSTRAPPING SMALL DATASETS
MICP does not run smaller Ntroin I 3000 datasets. However, bootstrapping still be used to finetune the model on said datasets to match the downstream dataset distribution. In this case, we sample from p Draain) by randomly sampling 90% of training samples without replacement to obtain D subtrain and treating the remaining 10% of sample as X Xunttests Y subtest:
3.2.3 FINETUNING WITH ADAPTERS
To prevent overfitting and reduce parameter count for each new expert, we only train small set of new adapter (Houlsby et al., 2219;) Bapna et al.) 2019 Hu et al. 2021 Liu et al. 2022) parameters, vi p(D DI Dtrain), without modifying in the pretrained transformer's parameters, cally, we freeze a pretrained TABPFN transformer, go(ulzz pr. D). Next, for each downstream dataset,
Specifi-
Derain, we add linear adapter layers (Houlsby al. 2019). A (Derss)) with parameters v. to form (Derasn) (z)zz D. Qo. A (Derata)). During finetuning, only is optimized. Intuitively, go encodes the 90.0
handcrafted prior, p( p))()), and A (Dtrain) encodes the bootstrapped prior, p(D)Drraen). We
illustrate CAPEN in Figure
4 EXPERIMENT SETUP
the recently proposed TABZILLA benchmark (McElfresh et al. 2023 TABZILLA the largest tabular benchmark, with 36 hardest datasets out of 176 tabular classification datasets and 19 baseline algorithms, covering both deep learning and GBDTs. The benchmark covers diverse range of dataset properties, number of samples, number of features, and feature distributions. We choose this benchmark it considers larger range of dataset prop- erties than other existing benchmarks (Grinsztajn et al. 2022 Kadra et al. 2021 Gorishniy et al.) 2022). MIXTUREPEN'S goal is (1) improve TABPFN* * (McElfresh et al. 2023). which randomly samples B training pairs so that TABPFN (Hollmann et al. 2022) runs on larger datasets, and (2) outperform both GBDTs (Chen Guestrin [016]] |Prokhorenkova al.) 2018 Ke al., 2017)
We evaluate MIXTUREPFN
which were found state-of-the-art by TABZILLA, and recent deep learning models (Popov et al.
2019 Gorishniy et al. 2021 Arik Pfister 2021 Hollmann et 2022 Yamada et al. 2020 Yoon
et al. 2020 Somepalli et al. 2021
2 Chen et al. 2022).
Adapters also efficient because only small number of parameters are updated, p()) is not needed during finetuning, and different downstream datasets share common pretrained model.

---

Published as a conference paper at ICLR 2025
Mean
Std
Median
Min
Max
Method
Rank
Rank
Rank!
Rank
MIXTUREPEN
2.75 1.94
2
MIXTUREPEN (KNNv2)
4.00 =1.66
CatBoost
4.70 =3.16
MIXTUREPFN (CaPFN W. Full FT)
4.85 =2.03
4.5
2
9
XGBoost
4.95 =3.17
10
MIXTUREPEN (-CaPEN)
5.10 =1.12
5
3
8
MIXTUREPEN (-CaPFN-MICP) TABPPN T
5.30 1.59
MIXTUREPFN (KNNvI)
7.25 3.70
9
10
MLP-rtdI
7.25
2.79
MLP
8.85 0.99
7
10
Table 2: Ablation table results. MIXTUREPFN (KNNv1) and MIXTUREPEN (KNNv2) replace
MICP with scalable variant of KNN- Prompting. MIXTUREPEN (CaPFN w. Full FT) uses full finetuning instead of adapters. MIXTUREPFN (-CaPFN) and MIXTUREPFN (-CaPFN-MICP) re-
move each component iteratively, where MICP replaced by random sampling.
4.0.1 EVALUATION PROTOCOL
Since TABZILLA restricts the total runtime to 10 hours, not all algorithms run on the same datasets. To ensure fair comparisor we evaluate MIXTUREPEN and baselines using (1) Mean Rank, TABZILLA's 's original metric, (2) Satcoxon-Signed Rank Test to check for statistical significance, and (3) Condorcet voting (Gehrlein & Valognes| 2001 Wang al. 2012), which aggregates pair-
wise comparisons between each algorithm on each dataset.
We introduce the Condorcet voting deal with cases where not all algorithms TABZILLA 's Mean Rank metric is not ideal, because it considers either a subset of datasets or subset of algorithms. This throws away valuable information, since the relative ranking between algorithms on the excluded datasets are ignored. Incomplete ranked-choice voting is well-studied problem in political science (Gehrlein & Valognes 2001), where voters (datasets) rank candidates (algorithms) relative to each other. Condorcet voting ensures both that every voter's 's (dataset 's) rank- ings are accounted for and that no candidate (algorithm) voter (dataset) are artificially excluded, by decomposing full rankings into pair-wise rankings. An algorithm is the Condorcet winner if it wins all pairwise comparisons averaged across all datasets.
all datasets.
RESULTS
5.1 MIXTUREPEN: STATE-OF-THEEARYY PERFORMANCE
As shown in Table I MIXTUREPFN achieves state-of-the-arr performance on TABZILLA across 36 datasets and 19 baseline algorithms both using TabZilla's original metric, Mean Rank, and as the Condoreet winner, beating all other baselines in pairwise comparisons. MIXTUREPFN is followed by GBDTs (Chen & Guestrin 2016 Prokhorenkova et al. 2018). then TABPFN* (McEl- frrss al., 2023), then deep learning algorithms (Chen et al.) [2022] Gorishniy et al. 2021]] Somepalli et al.) 2021). These results corroborate recent findings (McElfresh et al., 2023) Grinsztajn et al.
2022) that most deep learning algorithms fail on general tabular datasets. MIXTUREPFN achieves its state-of-the-art results by scaling TABPFN*'s impressive performance to larger datasets. We provide additional metries in the Appendix. To understand what dataset regimes each algorithm performs best
evaluate MIXTUREPEN'S rankings w.r.t. dataset properties.
MIXTUREPFN substantially improves the scalability of TABPFN and TABPEN*, As show in Figure 2a unlike TABPFN which encounters memory bottlenecks on datasets with > 3000 sam- ples, MIXTUREPFN successfully runs on all said datasets. As show in Figure 2a MIXTUREPEN substantially improves the performance of TABPEN * by improving how samples chosen for the "prompt" and training on the downstream dataset.
We cannot follow the same experimental settings the October revision of TARZILA hecause they unfair, mentioned in recent Github issue ggit 2024 Further details in the Appendix.
8

---

Published as a conference paper at ICLR 2025
Defeult
Flash/llention2
R1+toochsssmple
00011
0.911
0.111
TataPy
1hohingg (1)
0920
0/210
0.110
6020
0.00
0.000
0037
A407
0.007
0026
0.00
20022
30000
30000
20000
Mixit
Naain
Runtime Costs for Inference
Delaul
RasiAAlinnionn
R124torch.com complee
MictorePhh
Mamary Unage
13
15
(b) Memory Costs for Inference
Figure 4: Computation Costs for attention compute with different hardware optimizations.
MIXTUREPFN encounters no performance deterioration w.r.t dataset size. As shown in Figure 2b unlike TABPEN whose performance deteriorates w.r.t dataset size, MIXTUREPFN's performance compared to the next best baseline is not correlated with dataset size. Hence, MIX- TUREPEN is necessary to scale TABPFN* impressive performance on to larger datasets.
MIXTUREPEN is robust to irregular datasets. We measure the irregularity of datasets using the standard deviation of the kurtosis of all features. Deep learning algorithms are especially susceptible irregular datasets with uninformative or heavy-tail features (Grinsztajn al. 2022). As shown in Figure 2c MIXTUREPFN is the Top-1 algorithms on datasets with both high and low kurtosis standard deviation. Because finetuned on downstream datasets, MIXTUREPFN robust to dataset irregularity.
MIXTUREPFN works better with fewer features. As shown in Figure 2c MIXTUREPFN loses against baselines on datasets with large number of features. PFN transformers are known to face scalability challenges with number of features (Hollmann et al.) (022), due to their handling of column order invariance. We believe better tokenization practices and feature selection can improve feature size scalability, and leave such exploration to future work.
state-of- -ft----rtttttttt--e performance is statistically significant. Specifically, we run the Wilcoxon Signed-Rand Rann test with p < 0.05 comparing the Top-10 Condorcet algorithms from Table across their 22 shared datasets and 10 volssssvliittionn.. splits. As shown in Figure MIXTUREPFN oate--of-theaan performance statistically significant.
MIXTUREPEN'S
5.2 ABLATION STUDY
Both MICP and CAPFN contribute to MIXTUREPEN state-of-the-art results. We perform ablation studies for MICP and CAPFN against common GBDTs and deep learning models across 10 shared datasets. As shown in Table 2 each component of the model, MICP and CAPFN, con- tributes to achieving state-of-the-art results. MICP helps by efficiently choosing an effective con- text for the "prompt". CAPFN helps by aligning the dataset prior through finetuning the PFN on MICP's prompting policy. Because overfitting is well-known issue for deep learning models tack- ling tabular data (Kadra et al. 2021 Grinsztajn et al.) (022), adapters are a key component to ensure CAPEN aligns the pretrained TABPHI transformer with the downstream data, without destroying its pretraining prior, p( (DD)))(()))
Under the same GPU resources, KNN-Prompting is much less effective than MICP on tabular datasets. As described in Section 2.1.3 KNN- ENN-Prompting does not support TABPFN -tylee batching. To empirically verify that MICP improves KNN-Prompting, replace MICP in MIXTUREPEN

---

Published as a conference paper at ICLR 2025
with 2 KNN--NONPnpi ANN-PPromting variants: MIXTUREPEN (KNNv1): Because each prompt contains most B + Nbatch tokens, we batch KNN-Prompts by considering B/ B/Nbotch nearest neighbors instead of B-nearest neighbors; MIXTUREPFN (KNNv2): Because LLM- LLM-batching fails due to TABPFN's ensembling overhead, we remove the ensembling procedure and run KNN-Prompting following ILMM-btchiig (Liu et al. 202T), described in Section 2.1.3] As shown in Table 2 both KNN- Prompting variants perform substantially worse than MIXTUREPFN because they compromise an essential component of PFN, either prompt size or ensembling, for performing efficient inference. The relative rankings suggest prompt size matters more than ensembling.
5.3 RUNTIME
Given the efficiency effectiveness tradeoff's listed in Section 2.1.3 we provide real runtime analysis be- tween TABPEN, TABPFN* and MIXTUREPEN. Specifically, we set batch size 32, ensemble size 32, feature count to 100, and time the transformer forward pass combined along with any preprocessing costs, namely MIXTUREPFN routing. We report median runtime and memory consumption across 8 runs to remove outliers. Note, recent hardware-level optimizations, specifically FlashAttention (Dao) 2023) and torch compile, greatly improve real runtimes of transformer models. For fair comparison, we improve TABPEN with FlashAttention (Dao 2023) and torch compile. MIXTUREPFN is more runtime and memory efficient than full-context TABPFN independent of hardware optimizations by only attend- ing to training samples in the local neighborhood grouped test samples. As seen in Figure 4 MIXTUREPEN incurs the same memory costs as TABPFN* and nearly the same runtime costs TABPFN due to highly optimized nearest-neighhor search algorithms (Douze et al. 2024).
6.60
8.1
2010
MESUIA #EBtestoreer
KLIIOO Caffoost
ciaatig 1i100
Ul
Figure 5: Train time costs for MIXTUREPFN.
We quantify the efficiency effectiveness tradeoff of CAPFN finetuning by computing the runtime of training. Note, Section 5.2 shows that finetuning costs do substantially boost performance. We compare MIXTUREPFN 's training times XGBooST, CatBoost, Saint, LightGBM, and FTTrans- former. We report training runtime for one hyperparameter setup of one cross cross-validation fold over datasets of varying sizes, and report the median runtime/memory consumption across runs to remove outliers. As shown in Figure 5 MIXTUREPEN takes roughly the same amount of train- ing time per expert as other deep learning algorithms, like FTTransformer. While decision tree algorithms still run faster than deep learning algorithms, MIXTUREPFN substantially improves the trade-offs between tree-based and deep learning algorithms through its impressive performance.
LIMITATIONS
As shown in Section 5.1 MIXTUREPFN successfully improves TABPFN's scalability w.r.t. dataset size to achieve saaie-of-the-aae results on the TABZILLA benchmark. However, we notice that TABPFN does not scale well with feature and label count, relying on ensembling to capture feature and label order invariance. Scaling TABPFN to datasets with large number of features and labels can further push ICL performance for tabular learning. While this work covers a large number of diverse datasets, we do not cover huge datasets with billions of samples (Zhu et al. 2023 Yang et al. 2023). We leave such investigation to future work.
CONCLUSION
In this work, we provide scalable framework for In-Context Learning (ICL) on tabular datasets.
To efficiently construct effective ICL "prompts", we propose routing test samples through Sparse Mixture of In-Context Prompters, MICP. To align the PFN with the inference-time datasets, we propose novel finetuning policy using bootstrapping, CAPFN. Our framework, MIXTUREPFN, achieves state-of-the-an performance against 19 deep learning and tree-based baselines across 36 general tabular datasets.
10

---

Published as a conference paper at ICLR 2025
8 ETHICS STATEMENT
This paper presents work whose goal is to advance the field of tabular and in-context learning. Our work reaches state-of-the-art tabular classification accuracy, which has broad positive impact for many industries using relational databases and tabular datasets. We hope our impressive results in- spire further research into PFNS and ICL for tabular learning. Our work is built on large transformer models, which are known to hallucinate in the natural language domain. While we observe no such behavior on our tabular datasets. we will open source our code, such that practitioners can plug in their own safe transformer models. We feel there are not any other noteworthy negative societal impacts.
ACKNOWLEDGEMENTS
This work done at internship Uher and was partially supported by NSF 2211557, 1937599, 21 19643, 2303037, 2312501, 2200274, 2106859, NIH U54HG012517, U24DK097771, NASA, SRC JUMP 2.0 Center, Amazon Research Awards, Snapchat Gifts, and Optum AI.
REFERENCES
http://gghhhbommmmmssssi/lillllr 2024a.
https//gghhubbcoommmsszlllllaalb 2024b.
Anonymous. Mixture-of-exxerts in prompt optimization, 2024. URL htttp::///ppeeeesssss
Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings
of the AAAI conference artificial intelligence, volume 35, pp. 6679-6687, 2021.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint 190900889 2019.
Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, IEEE Transactions Neural Net- works and Learning Systems, 2022.
Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz, Constrained k-means clustering. Microsoft Research, Redmond, 20(0)0, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling transformer to Im tokens and beyond with rmt. arXiv preprint 2rXiv:2304.1106 2023.
Jintai Chen, Kuanlun Liao, Yao Wan, Danny Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 3930 3938, 2022.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
11

---

Published as a conference paper at ICLR 2025
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297, 1995.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on infor- mation theory, 13(1):21-27, 1967.
David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statistical Methodology, 20(2)211-232, 1958.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint urXiv:2307.0869/ 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-1633 2022.
Chris Ding and Hanchuan Peng. Minimum redundancy feature selection from microarray gene expression data. Journal of bioinformatics and computational biology, 3(02):185-205, 2005.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey for in-context learning. arXiv preprint arXiv:2301.002.44 2022.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Benjamin Feuer, Chinmay Hegde, and Niv Cohen. Scaling tabpfn: Sketching and feature selection for tabular prior-data fitted networks. arXiv preprint arXiv:2311.10601 2023.
Benjamin Feuer, Robin Tibor Schirmeisterr, Valeriaa Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv preprint 202111740..11137 2024.
William V Gehrlein and Fabrice Valognes. Condorcet efficiency: A preference for indifference. Social Choice and Welfare, 18(1):193-205, 2001.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint arXiv: 2018.
Yury Gorishniy. Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances Neurul Information Processing Systems, 1933-993-1, 2021.
Yury Gorishniy, Ivan Rubachev. and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 25:24991-25005 2022.
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Unlocking the power of -augmetedd tabular deep learning. arXiv preprint arXiv:2307. /4338, 2023.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35: 507-520, 2022.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to lear in context. arXiv preprint arXiv:2305.00 037.7 2023.
12

---

Published as a conference paper at ICLR 2025
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training, In International conference on machine learning, pp. 3929-3938. PMLR. 2020.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713. 2022.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree
ensemble layer: Differentiability meets conditional computation. International Conference on Machine Learning. pp. 4138 4148. PMLR, 2020.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 1eXiv:2006.9900 106.09685 2021.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678 2020.
Manu Joseph and Harsh Raj. Gate: Gated additive tree ensemble for tabular classification and regression. arXiv preprint arXiv:2207.0885 200.008888 2022.
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928-23941, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Liran Katzir, Gal Elidan, and Ran EI- Yaniv. Net-dnf: Effective deep modeling of tabular data. In International conference on learning representations, 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 6274. PMLR, 2021.
Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. R news, 2(3): 18 28- 2002.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, Advances Neural Information Processing Systems, 35:1950-1965, 2022.
Jiachang Liu, Dinghan Shen. Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06800 2021.
Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Hamidreza Kamkari, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Maksims Volkovs, and Anthony I. Caterini. Tab- dpt: Scaling tabular foundation models. arXiv preprint arXiv:2410.1817 2024.
Dunean McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Gold- blum, Colin White, et al. When do neural nets outperform boosted trees on tabular data? arXiv preprint N0Xx:2305000997 2023.
13

---

Published as a conference paper at ICLR 2025
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Sergei Popov, Stamislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning tabular data. arXiv preprint wwXXi::909.03311 2019.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv: I0. 10340, 2022.
J. Ross Quinlan. Induction of decision trees. Machine learning, 1:81-106, 1986.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 14:17555-17566, 2021.
Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, and Sepp Hochreiter. Hopular: Modern hopfield networks for tabular data, arXiv preprint arXiv:2206.0066 006664 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 701606588 2017.
Ravid Shwartz- -iv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:8 81:84990, 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint 10X.:22006.01342, 2021.
Valentin Thomas, Junwei Ma, Rasa Hosseinzadch, Keyvan Golestan, Guangwei Yu, Maks Volkovs, and Anthony L Caterini. Retrieval & fine-tuning for in-context tabular models. Advances in Neural Information Processing Systems, 17:1000991186666 2025.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-T Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv: 008.39, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pp. 35151-35174, PMLR, 2023.
Sinong Wang, Belinda 7. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Tiance Wang. John Sturm, Paul Cuff, and Sanjeev Kulkarni. Condorcet voting methods avoid the paradoxes of voting theory. In 2012 50th Anmual Allerton Conference on Communication, Con- frol, and Computing (Allerton), pp. 201-203. IEEF, 2012.
14

---

Published as a conference paper at ICLR 2025
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 0tt1::2009006522, 2021.
Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. prompting: Beyond-context learning with calibration-free nearest neighbor inference. arXiv preprint arXiv:2303. 12.3303..324.. 13824, 2023.
Derek Xu, Shuyan Dong, Changhan Wang, Suyoun Kim, Zhaojiang Lin, Akshat Shrivastava, Shang- Wen Li, Liang-Hsuar Tseng, Alexei Baevski, Guan- Ting Lin, et al. Introducing semantics into speech encoders. arXiv preprint arXiv:2211.08402 2022.
Xin Xu, Yue Liu, Panupong Pasupat, Mehran Kazemi, et al. In-context learning with retrieved demonstrations for language models: survey, arXiv preprint arXiv:240.111624 2024.
Yutaro Yamada, Ofir Lindenbaum. Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pp. 10648-10659. PMLR, 2020.
Yazheng Yang, Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: Pretraining unified tabular encoder for heterogeneous tabular data. arXiv preprint arXiv:2307.0922 20233
Jinsung Yoon, Yao Zhang. James Jordon, and Mihaela van der Schaar. Vime: Extending the suc- cess of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 0030110333 2020.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 01772-18297777 2020.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint 2rXiv:2305.066000 2023.
A RELATED WORK
A.1 TABULAR LEARNING ALGORITHMS
Early tabular learning algorithms are based off decision trees, utilizing boosting, feature encoding,
and ensembling (Shwartz-Zii & Armon 2022 Borisov et al. [2022] Chen & Guestrin 2016). Early deep learning algorithms inspired by decision trees, making them end-to-end learnable (Popov)
jet al. 2019 Katzir et al. however, more thorough benchmarks find decison trees produce more reliable results. Hyperparam-
2020 Hazimeh et al. 2020 Somepalli et al. 2021 Arik & Phister 2021);
eter tuning (Kadra et al., 2021) and inductive bias (Grinsztajn et al. 2022) were identified as key weaknesses deep learning algorithms. Recent works focus optimizing transformer models for
specialized datasets (Huang et al. 2020 Gorishniy et al.) 2021 2022), and improving decision tree optimization (Joseph Raj [202)) or Bayesian learning Hoolmann al. 2022 Schäfl al.) 2022] Feuer et al. 2023) for small datasets. With the rise of LLMs, pretrained tabular learning models also achieve impressive performance the cost of billions of training points (Yang al. 2023) Zhu al.) 2023). Of these methods, Prior-Fitted Networks (Feuer et al.) 2023) were identified as promising direction in recent benchmarks among deep learning approaches for general tabular learning prob- lems (McElfresh et al.) 2023). Most recently, TuneTables (Feuer et al. 2024) appeared on arxiv using sketching to also scale TABPFN at inference; however, purely inference-time techniques will always have some performance gap due to distribution shift, which work addresses.
A.2 GRADIENT-BOOSTIE DECISION TREES
Gradient-boosse boosted decision trees (GBDTs) remain the preferred algorithm of tabular learning prac-
titioners (Chen & Guestrin 2016 Prokhorenkova et al. 2018). Deep learning algorithms (Popov) et al. 2019 Gorishniy 2021 Somepalli et al. 2021 Arik & Prister 2021 Yamada et al.
15

---

Published as a conference paper at ICLR 2025
Yoon al., 2020) fail on larger benchmarks considering different numbers of samples. num- bers of features, feature types, feature distributions, and numbers of labels (McElfresh et al. [023] Shwartz-Ziv & Armon 2022). Because GBDTs utilize boosted gradients and not rotationally invariant before training, GBDT learning algorithms have a better inductive bias than Stochas- tie Gradient Descent-based (SGD--based deep learning algorithms Grrnsztajn et al. (2022). Thus, GBDTs achieve state-of-the-ar performance on medium to large datasets with 3,000 to 1,000,000
2020
samples and competitive performance on smaller datasets McElffrsh al. (2023): Grinsztajn et al. 2022).
A.3 PRIOR FITTED NETWORKS
Prior Fitted Networks (PFN) (Müller et al. 2021) approximates Bayesian inference using a data prior, where a parameterized model is trained to minimize the KI.-Divergence between it and the posterior predictive distribution. The proof for PFNs is derived from meta-learning (Gordon et al.) 2018). PFNs fall under In-Context Learning (Brown et al., 2020). as the entire training dataset is fed to the model during inference. Hence, PFNs effectively learn the learning algorithm. This approach is particularly effective on tabular data (McElfresh et al. (023). where the data prior effectively regularizes model predictions. TABPFN (Hollmann et al. 2022) is a PFN model specific tabular data that achieves state-of-the-art performance on small datasets. Unlike preliminary works on scaling TABPFN (Feuer et al.] (203)), that benchmark KMeans and Coreset "promting". we propose sparse mixture of KNN prompters, capable of forming scalable batched "prompts" and a novel finetuning protocol for the mixture of prompters. Furthermore, our approach not only improves TARPFN results on large datasets, but achieves state-of-the-ar performance across tabular benchmarks (McElfresh et al. 2023).
A.4 MIXTURE OF EXPERTS
Sparsely gated MoE (Shazeer et al. 2017, Lewis et al. 202T) shows a significant improvement in model capacity, training time, and accuracy with gating mechanism. An expert is a sub-network, which better learns to predict similar data points. A gating mechanism, learnable or non- non-learnable method, decides to route each data point to the most suited experts (Shazeer al. 2017). Switch transformers (Fedus et al.) 2022) is a learning to route approach, where it assigns one data point to only one expert, instead of top-k, which reduces computation, while preserving accuracy. How- ever, learnable routing methods require auxiliary load balancing loss function, and further tuning. In (Roller et al. 202I), non-lcarnable routing method proposed, which uses hashing method to assign similar data points to similar experts. They show that this procedure can be better or be com- petitive with learnable routing MoE methods. However, hashing method is not necessarily flexible in assigning data points to suitable experts, as it can cause data skewness and choice of hashing function is sensitive to the downstream task. Inspired by these works, we proposed a non-learnable routing mechanism which assigns one data point to expert, and our routing method finds the most suitable expert based on similarity of data points with a K-Means clustering method. Also, used experts with shared weights opposed to general MoE in most of the previous works, as the prompts, not the model, contains training examples for In-Context Learning. We highlight tack- ling MoE in the context of prompting is newly emerging research interest (Anonymous) 2024). of which we are among the first
A.5 EFFICIENT TRANSFORMERS
Long sequence inputs have long been studied by the efficient transformer community. Several linear time and space complexity transformers have been proposed (Katharopoulos et 20,00 Wang et al.) 2020 Choromanski al., 2020 Qin et al. 2022), primarily be SVD decomposing the attention computation. While efficient transformers can help scale the base PFN model, linear complexity is too expensive for the scale of tabular data in industry. Furthermore, many approaches require finetuning on downstream data (Katharopoulos et al.) 2020 Wang et al. 2020) which is nontrivial for PFN models. Constant time transformers (Zahee et al. 2020 Bulatov et al.. 2023 Chowdhery et al. 2023) exploit the sequential nature of text data. These methods also do not apply to PFNS for tabular data, as tabular training data is not inherently sequential. Hence, technooogies outside of efficient transformers are needed to effectively scale PFNS for tabular learning.
16

---

Published as a conference paper at ICLR 2025
A.6 IN-CONTEXT LEARNING
In-Context Learning (ICL) prompts transformers with training examples prepended to the desired query (Brown al. 2020 Several works attempt to prompt engineer scalable in-context examples for better downstream performance (Hao et al. 2022), among which K-Nearest Neighbors emerge as a reliable choice (Liu al., 2021 Xu et al. 2023). However, ICL. for L.LMs consider queries one-at-a-time instead of in batch fashion, prompts are encoded as natural language, and in-context examples can come from a large corpus of natural lanaguage data (Brown al., 2020). These properties are not afforded to PFN-style ICL, where inference is directly run training set tokens. In addition to scaling "prompts". " LLMs can also be finetuned on ICL examples to reach better performance (Thoppilan al. 2022 Wei al. 2021). However, this is due to LLM pretraining objectives misaligning with the ICL task. Such misalignment is not as obvious in the PFN case, where the transformer is directly trained the ICL task (Mäller et al., 2021). Hence, propose Sparse Mixture of In-Context Prompters to support batching and our novel bootstrapping algorithm finetune PFN models. the spirit of multi-modal models (Radford al. [021] XXu al. (022), TABPFN (Hollmann et al. 2022) extends ICL techniques from natural language LLMs to tabular data.
B MATH NOTATIONS
We summarize our math notations below:
B: Number of training samples prompt
K: Number of experts
D: A generic dataset
Dirain: The training dataset
Xtest: The test samples
Xbateh: A batch of testing data
C: Number of classes
qo: PFN Model
A Divinn PFN Model Adapters trained on Drroin
0: PFN Model Parameters
v/. PFN Model Adapter Parameters
Φ: PFN hypothesis mechanism
R: Router mapping input test points to 1 of K Prompters
Tx: The k-th Prompter
(K))) D The k-th K-Means cluster of training samples
D The k-th Prompter's training samples context
(Xbatch, (k) p:ompp):: The k-th Prompter's "prompt"
NNSS(-)-): Nearest Neighbor Search Algorithm
KNN(.1): K-Nearest Neighbors Algorithm
K Keeans(F) K-Means Algorithm
Sample(-) )): Random Sampling
Ntroin: Full training dataset size
Nteat: Full testing dataset size
Nbatch: Batch size
7: Single hyperparameter trading off performance and efficiency.
17

---

Published as a conference paper at ICLR 2025
C NONZERO OVERLAP PROOF
We prove Theorem 3.1.1 here.
First we prove |D duster S
D (k)
S CKNN (Jcenterr Drrains B):
cluster
By KMeans definition,
d( train) < <d(reenter (train)
( E duster train E D(K)
(k)
dluster):
Means d( Ftrain) < "KMenn
(k)
(i) Ttrain
E D
cluster
By KNN definition,
KNN Derains B) { d( dl
center: (Pasn) < (x)NN())Drrain))
where (rain (K)
edeeeteerrs I train) < TKNN (Draan))
Given D
cluster! S B,
(k)
(k)
drustor I IKNN (I
center Derain B)]
KNN(x (Feenter | Derains B)
(k)
D ((k) eluster U a
(k) di (
TK Means
TKNN (x) D Dtrain))
(1) cluster
KNN
center Derains B)
(K)
Next, we prove Theorem 3.1.1
Given (Ddg) duster S B vk E [0,..., K 1] and R2 12"(((ran Zerain E (k) duster
G(k) =1
Etrain
KNN (1 train D B) KNN (NN(wee (6)
Test D Dirain B)
(i) Ptrain
D
(R' (mtrram)) claster
C KNN(2 (F
(R. (a
) Derains B) D promps (R" (t)
Etrain & Dprompt (R* (x (orrain)) )) NONN
test D Drrains B)
(8) prrropp (R" (R (1) n KNN (3) Dtrain: B) 2 (2 Vi E (0..., Ntrain - 1]
v Dirain.
D SIMILARITY AND DIFFERENCES COMPARED TO LARGE LANGUAGE MODELS
We illustrate the difference between ICL PFNs and LLMs in Figure
E SUPPORT SET AND KNN PROMPTING.
In Figure 15 we point out one key difference between KNN-Prompting with TABPFN (Hollmann ct al., 2022) and with ICL on LLMs (Liu et al. 2021): LLMS support batching multiple test pairs across multiple "prompts" where each "prompt" contains the nearest neighbors to test point. In contrast, TABPEN requires multiple test cases in "prompt", which necessitates techniques like MICP to route batch of test points to the same "prompt". As shown in Section 5.2 TABPEN more efficient prompts achieve better performance under same GPU constraints as ICL for LLM batched prompts. We leave application of MICP to ICL and LLMS as future work.
18

---

Published as a conference paper at ICLR 2025
Raview
Senmart
cerssage
nod
102
ago
12.9
wigei
11.
8:1
10e6
1.5
686
Zua Yea
'Sew sorise.
Tootwe
cep
112
379
00g
Daae Yoal
ahao
Wuldl
ldd
DecA Dir
Pascs
45
152
27
70
Cantrer
"LUP" STR
Usd
"Wooou Donikes Sundmont Pealive Pe reoe Bown let will beig bes
Loga
Ldb
Figure 6: We highlight the differences between In- In-Context Learning (ICL) on Prior Fitted Networks (ex. TABPFN), left, and Large Language Models (LLMs), right. TABPFN treats training data as tokens (where each token convert training data into natural language prompts. TABPEN uses an attention pattern (blue and arrows) supporting batch inference, whereas LLMS use generic encoder-decoder or decoder-only setups. TABPFN are pretrained Equation [ whereas LLMs are pretrained separately.
concatenation of feature and label), whereas LLMs use templates to
F TIME AND SPACE COMPLEXITY DETAILS
Router and prompter initialization takes Ot K + (Ntrain + Blog time and O(Nrrain K B) space complexity and is done once before inference. For initialization, K- Means with t-iterations takes (tNNerain K) time and O(K) space. To perform efficient nearest neighbor queries,
the ball-tree algorithm over the training dataset and cluster centers which takes O (NNerainlogl (orrainlog(Nerainn) Ntraan)) time and (NNeraén) space. Using ball-tree KNN queries, constructing D(Blog(Nerasn)) (Nerasn)) time and (BB) space.
each ICP support set takes O
Routing takes (K))((K)) time and (1) space complexity, using efficient nearest neighbor search with ball-tree for each test sample. Router overhead is practically overcome via highly- optimized NNS implementations (Douze et al. 2024), which scale K to the billions.
PFN transformer inference takes O(B² + BNbatcc) time and space complexity, as MICP prompts contain at most B training samples and Nhatch [Xtutchl testing samples. Note, unlike purely KNN-based prompting, MICP supports batched computation further amortize PFN inference cost.
G CHOOSING BASELINE DATASETS AND ALGORIHTMM
We chose our datasets from the TABZILLA benchmark, which curates 36 the hardest 176 consid- ered datasets across 19 algorithms. As noted in Section I.0.I not all algorithms run on all datasets. We note which datasets are shared by algorithms, Top- algorithms, and Top-5 algorithms in Table 3, We provide the number of datasets each algorithm successfully runs on in Table II We provide dataset names and statistics in Table 19
The 2 datasets MIXTUREPFN and TABPFN* fails on contain 100 classes, which is not currently supported by the pretrained TABPFN. However, as seen in Figure 16 described in Section G. highlight that the 34 datasets MIXTUREPFN and TABPFN* successfully run cover the full range of dataset properties: number of features, number samples, and dataset irregularity, Specifically, our 34 datasets include the ones with the least and most number of samples, the least and most number of features, and the least and most irregularities. Because the focus of this work is to scale TABPFN datasets with more number of samples, we leave extending TABPFN more number of classes as future work.
G.1 BASELINE DATASETS
We provide dataset statistics in Table 19 As shown, our considered datasets cover a wide range of dataset properties in number of features, number of samples, and std. kurtosis. As shown, MIX-
19

---

Published as a conference paper at ICLR 2025
Subset
Considered Algorithms
Considered Datasets
ALL
MixturePFN, CatBoost, TabPFN* XGBoost, ResNet, FTTransformer,
ada-agnostic, australian, balance-scue scale, colic, credit-approval, elevators, heart-h, jasmine, kc lymph, ffrat-fourier, mfeat-zernike, monks-prowe wanks-problems-2 phoneme, profb, gsar-biodeg, socmob, speeddating, splice, vehicle
LightGBM, SAINT,
NODE, MLP-rtdI, RandomForest, TabNet, MLP, DecisionTree, LinearModel, STG, VIME, KNN, DANet,
SVM
Top-10
MixturePFN, CatBoost, TabPFN', XGBoost, ResNet, LightGBM, SAINT, DANet, FTTransformer, SVM,
ada-agnostic, artificial-character, australian, balance-scale, colic, credit-approval, elevators,
gesurrplasseemenaaioonoosss heart-h, jasmine, kel, lymph, mfcat-fourict mfeat-zernike, monks-problems-2 phoneme, profb, qsar-biodeg, socmob, speeddating, splice, vehicle
Top-5
MixturePFN, CatBoost, TabPFN*, XGBoost, SAINT
ada-agnostic, artificial-chan characters, australian, balance-scale, colic, apedit-approval, electricity, elevators, albert, gsstreehhseeeemmnttiiopppessssse heart-h, higgs, jasmine, www.eeeee-ps----eedenneaaaeae lymph, mffat-fouricr mfeat-zernik,, monks wonks-problems-22 phoneme, profb, qsar-biodeg, socmob, speeddating, splice, vehicle
kel,
Table 3: Datasets and algorithms considered in each Top-K subset. For fair evaluation (git 2024a we only consider the shared set of datasets all algorithms run on in the Top-K subsets. By consider- ing different subsets, evaluate MIXTUREPFN against more less algorithms and datasets. 20 datasets are shared by all 20 algorithms. 22 datasets are shared by Top-10 algorithms. 25 datasets are shared by Top-5 algorithms, MIXTUREPEN achieves the best mean rank among all Top-K K sub- sets.
20

---

Published as a conference paper at ICLR 2025
Condoreet Statisties
Method
# #Votest
#Winst
#Ties
#ossess
MixturePFN
464
19
0
0
CatBoost
462
18
0
1
XGBoost
448
16
1
2
SAINT
402
15
0
4
ResNet
367
14
2
3
LightGBM
360
12
0
7
FTTransformer
345
13
5
T
330
11
8
O
NODE
304
8
DANet
300
10
8
RandomForest
299
12
6
MLP-rtdl
256
11
0
SVM
236
13
TabNet
231
4
0
15
0
MLP
229
12
STG
188
5
0
14
0
Decision Treee
155
17
LinearModel
144
0
16
KNN
112
0
0
19
VIME
93
0
18
Table 4: MIXTUREPEN is the Condorcet winner across 36 datasets against 19 baseline algorithms. We rank algorithms based on their accuracies.
All Algorithms (Log Likelihood)
Method
Mean + Std Rank
Median Rank
Min Rank
Max
Rank
MixturePFN
2.350 + 1.824
1.0
1.0
6.0
TabPFN*
4.550 + 2.747
4.5
2.0
13.0
CatBoost
4.900 + 4.158
3.0
1.0
14.0
XGBoost
5.500 + 4.621
4.0
1.0
16.0
SAINT
8.300 + 5.367
7.0
1.0
19.0
ResNet
8,400 + 3.262
8.5
3.0
15.0
FTTranfformer
8.600 3.541
8.5
3,0
15.0
DANet
9.050 + 3.369
8.5
3.0
17.0
LightGBM
9.150 0.150:4.33 4.351
10.5
2.0
16.0
MLP-rtdl
10.800 + 5.046
10.5
2.0
20.0
SVM
11.300 4.766
11.0
2.0
19.0
RandomForest
11,600 + 4.443
12.5
4.0
18.0
STG
11.900 4.549
12.5
4.0
19.0
LinearModel
12.400 4.655
12.5
5.0
20.0
NODE
13.350 3.410
13.5
7.0
18.0
TabNet
13.550 5.296
14.0
2.0
20.0
MLP
13.700 3.621
14.0
4.0
19.0
VIME
15.350 + 3.851
17.0
6.0
19.0
DecisionTree
16.800 3.881
18.5
7.0
20.0
KNN
18.450 + 1.936
19.0
14.0
20.0
Table 5: MIXTUREPEN achieves the top mean rank w.r.t. Log Likelihood across 20 datasets where all algorithms successfully run.
21

---

Published as a conference paper at ICLR 2025
All Algorithms (Accuracy)
Method
Mean + Std Rank
Median Rank]
Min Rank
Max Rank
MixturePFN
3,950 + 3.570
3.0
1.0
13.0
TabPFN*
5.500 + 4.056
5.5
1.0
15.0
CatBoost
6.350 + 5.313
4.0
1.0
17.0
XGBoost
7.100 5.234
5.5
1.0
18.0
ResNet
7,600 + 4.017
6.5
1.0
16.0
FTTransformer
7.900 4.158
7.5
1.0
17.0
SAINT
7.950 =
5.5
1.0
20.0
LightGBM
8.750 1.750:4.0 4.918
8.5
1.0
17.0
NODE
8.850 + 3.395
9.0
3.0
15.0
MI.P-rtdI
9.150 5.033
1.0
18.0
RandomForest
9.350
8.5
4.0
19,0
DANet
10.000 4.405
11.0
3.0
19.0
SVM
12.250 5.476
14.5
1.0
19.0
TabNet
13.050 + 4.780
13.5
2.0
20.0
MLP
13.100 + 4.253
14.5
5.0
18.0
LidearModel
14.100 + 3.846
14.0
8.0
20.0
DrcisionTree
14.250 4.426
15.0
3.0
20.0
STG
14.800 + 4.423
16.0
4.0
20.0
VIME
16.950 2.854
18.0
10.0
20.0
KNN
17.400 + 3.470
19.0
7.0
20.0
Table 6: MIXTUREPEN achieves the top mean rank w.r..
Accuracy across datasets where all
algorithms successfully run.
Top-10 10 Algorithms (Log-Lieelihood)
Method
Mean + Std Rankl
Median Rank
Min Rank
Max
Rank
MixturePFN
2.273 + 1.710
1.0
1.0
6.0
CatBoost
3,955 + 2.688
3.0
1.0
10.0
XGBoost
4.000 2.663
3.0
1.0
10.0
TabPFN
4.045 + 1.965
4.0
2.0
9.0
SAINT
6.136 = 2.473
6.5
1.0
10.0
LightGBM
6.409 2.839
7.5
2.0
10.0
FTTransformer
6.773 = 2:235
7.0
10.0
ResNet
6,864 1.961
7.0
3.0
9.0
DANet
7.045 = 1.988
7.0
3.0
10.0
SVM
7.500 = 2.482
8.0
2.0
10.0
Table 7: MIXTUREPFN achieves the top mean rank w.r.t Log Likelihood across 22 datasets where all Top-10 algorithms successfully run.
22

---

Published as a conference paper at ICLR 2025
Top-10 10 Algorithms (Accuracy)
Method
Mean + Std
Median
Min Rank
Max
Rank
Rank
Rank
MixturePFN
3.000 + 2.256
2.0
1.0
9.0
T
4.318 2.703
4.5
1.0
9.0
CatBoost
4.591 + 2.964
4.0
1.0
10.0
XGBoost
4.773 + 2.859
4.0
1.0
10.0
ResNet
5.591 2.103
5.5
1.0
9.0
LightGBM
5.727 2.847
6.5
1.0
10.0
SAINT
5.727 + 2.847
5.0
1.0
10.0
FTTransformer
5.864 2.282
6.0
9.0
DANet
6.955 = 2.184
7.5
3.0
10.0
SVM
7.773 + 2.907
9.0
1.0
10.0
Table 8: MIXTUREPFN achieves the top mean rank w.r.t.
Accuracy across 22 datasets where all
Top-10 10 algorithms successfully run.
Top-5 Algorithms (Log Likelihood)
Method
Mean + Std
Median
Min Rank
Max
Rank
Rank
Rank
MixturePFN
2.000 + 1.166
1.0
1.0
4.0
XGBoost
2.760 = 1.394
3.0
1.0
5.0
CatBoost
2.880 + 1.243
3.0
1.0
5.0
TabPFN*
3.320 + 1.085
3.0
2.0
5.0
SAINT
4.040 + 1.311
5.0
1.0
5.0
Table 9: MIXTUREPFN achieves the top mean rank w.r.t. Log Likelihood across 25 datasets where all Top-5 algorithms successfully run.
Top-5 Algorithms (Accuracy)
Method
Mean + Std Rank
Median
Min Rank
Max
Rank4.
Rank]
MixturePFN
2.360 + 1.292
2.0
1.0
5.0
XGBoost
2.880 + 1.395
3.0
1.0
5.0
CatBoost
2.920 + 1.354
3.0
1.0
5.0
TabPFN
3.000 + 1.386
3.0
1.0
5.0
SAINT
3.680 + 1.406
4.0
1.0
5.0
Table 10: MIXTUREPFN achieves the top mean rank w.r.t.
Accuracy across 25 datasets where all
Top-5 algorithms successfully
23

---

Published as a conference paper at ICLR 2025
35
MXturePFN
XGBoost
CatBoost
30
TabPFN
LightGBM
SAINT
FTTrans...
DANet
20
RandomF...
MLP-rtdl
SVM
15
STG
TabNet
MLP
10
LinearM...
NODE
VIME
Decisio...
KNN
MixturePFN
XGBoost
CatBoost
TabPFN*
ResNet
LightGBM
SAINT
FTTrans...
DANet
RandomF...
MLP-rtal
SVM
STG
TabNet
MLP
LinearM... ...
NODE
VIME
Decisio... KNN
Figure 7: Pairwise comparison matrix for Condorcet voting the log likelihood metric. Note, MIXTUREPEN is the Condorcet winner.
MixturePFN
XGBoost
CatBoost
ResNet
LightGBM
T
RandomF...
MLP-rtdl
FTTrans...
NODE
SAINT
DANet
15
TabNet
MLP
SVM
10
Decisio...
LinearM...
5
STG
KNN
VIME
MixturePFN
XGBoost
CatBoost
ResNet
LightGBM
TabPFN
RandomF...
P-rtdl
FTTrans...
NODE
SAINT
DANet
TabNet
SVM
Decisio...
LinearM...
STG
KNN
VIME
Figure 8: Pairwise comparison matrix for Condorcet voting the accuracy metric. Note, MIX-
TUREPEN is the Condorcet winner. Please refer to Section H for more discussion.
24

---

Published as a conference paper at ICLR 2025
0.25
0.2
10"
(Milcturen SeRRRN MeF--7e6e 2e7eereee
0.20
0.1
Off Accurssy
0.0
53d Rurross
10P
InOFFN XiBibosst
0.15
6.1
MisturoPFN
0.10
-02
0.05
-0.3
x
10:1
-04
103
10
10"
#Samples
#Samptes
#fwwets
(a) Ntrain W.L.L TABPFN
(b) Nevoin W.L.t. baselines
(c) Kurtosis and #Feat
Figure 9: We perform the same sensitivity analysis as Figure
in the main text on the accuracy
metric.
Der Leg Uantheod (=slihteee
Leg Uaethend (13300aas
Leg Ubethood (1-3000ass
bet tea Ubethood
Galooo DedANT swilD
10a
1:2
16"
10'
10"
1.P
101
10'
Mitrrpee
v1S. Karioes
winett
#labels
(a) Nerai w.r.t baselines (b) Kurtosis w.r.t. baselines (c) #Feat w.r.t baselines
(d) #Class w.r.t. baselines
Figure 10: We perform the same sensitivity analysis Figure 2 in the main text but across all dataset properties.
0.1
0.0
bit accurry (o000
0.0
bie Acc accurey to300
0.0
tie Accumey (o300
0.0
Die arc.
0.3
6.3
6,3
XwSANT swibbopp YwlllttrrFFN
6.3
10'
(a) Ntrain W.t.. baselines (b) Kurtosis w.r.t. baselines (c) #Feat w.r.t baselines
(d) #Class w.r.t. baselines
Figure 11: We perform the same sensitivity analysis as Figure in the main text but across all dataset properties and the accuracy metric.
10
9
8
5
5
2
MixturePFN
Df
XGRoost
SAINT
CatBoost
FTTransformer
TabPFN
HesNet
LighteBM
Figure 12: Wilcoxon- -ignned Rank Test shows MIXTUREPFN significantly outperforms the Top-10 10 baselines the 22 shared datasets, under the accuracy metrie. We compute the rank across all 10 cross-validation splits.
25

---

Published as a conference paper at ICLR 2025
20199 18 16 15
14
11
8
6
5
3
MixturePFN
VIME
CatBoost
DecisionTree
TabPFN*
NODE
XGBoost
LightGBM
MinearModel
ResNet
STG
DANet
TabNet
FTTransformer
MLP-rtdl
RandomForest
Figure 13: Wilcoxon-Signed Rank Test shows MIXTUREPFN significantly outperforms the all base- lines on the 20 shared datasets, under the log likelihood metric. We compute the rank across all 10 cross-validation splits.
20 19 18 16 1312
L
KNN
MixturePFN
VIME
TabPFN
UnearModel
XGBoost
Decision Tree
FTTransformer
MLP
ResNet
Tab Net
LightGBM
DANet
SAINT
RandomForest
Pirtl
Figure 14: Wilcoxon-Signed Rank Test shows MIXTUREPEN significantly outperforms the all base- lines on the 20 shared datasets, under the accuracy metric. We compute the rank across all 10 cross- validation splits.
TUREPFN achieves the best average performance across all datasets. Furthermore, we plot the full range of dataset properties covered, along with the held out datasets from TaBZILLa in Figure 16 showing that the datasets successfully run on are representative of the benchmark as a whole. Refer to Section G for more details.
We followed same experimental setup as TabZilla (MeElfresh al. 2023). which includes: imputing NaN features to its non-NaN mean and all other preprocessing handled by each spective baseline. MIXTUREPFN and TABPFN follow TabZilla's PFN preprocessing (McElfresh) al. )023); categorical features encoded ordinal features, outliers dropped, features normalized, results are ensembled across shuffling the feature ordering, and results are ensembled across power-law scaled and unscaled features.
G.2 BASELINE ALGORITHMS
G.2.1 PRIOR-FITTED NETWORK MODELS (PFN)
TABPFN* is the only PFN-based baseline, which uses pretrained 12-layer TABPFN transformer model. with embeddings size 512, hidden size 1024 in feed-forward layers, and 4-headed attention. TABPFN is pretrained on a handcrafted dataset prior consisting of randomly generated structural causal models (Hollmann al. 2022). During inference features and labels are randomly shuffled in batch size 32 then ensembled together, following the TABZILLA benchmark (McElfresh et al.) 2023 Recent work extend TabPFN through in-context learning. Most closely related, this MIX- TUREPEN was developed in parallel to LoCALPEN (Thomas et al. 2025) which also uses finetuning and retrieval to improve TabPFN's scalability. Unlike LoCALPEN, our work uses different experts for each test-time point, whereas LoCALPEN uses the expert. Recently, TabDPT (Ma al. 2024) was proposed to train models to perform TabPFN retrieval. Our work improves TABPEN'S scalability and efficacy along different dataset properties, particular number of training samples.
26

---

Published as a conference paper at ICLR 2025
Rays
Dyan
cor shape
age
weight
label ???
shape
weight
label
pointe
10.2
29.8
pointed
12.3
8.1
Cat
pointed
pointed
KMN
pointed
8.9
55.2
00g
drop
5.6
55.2
00g
PaspponPPeat)) !))
drop
00g
ear shape
Age
weight
label
pointed
pointd
12.2
11.0
OMt
pointed
0.5
3.2
poimed
60g
drop
2.1
15.2
60g
pointed
drop
5.6
55.2
dog
drop
Figure 15: We hypothesize only a subset of the training data, D Daupport (# ). is required for effec- tive in-context learning test sample, Piestt and this subset is the B nearest training samples in feature space: Daupport (J KNN (Frsst) Dirains B).
G.2.2 GRADIENT-BOOSTEE DECISION TREE MODELS (GBDT)
CatBoost (Prokhorenkova et al. 2018), XGBoost (Chen & Guestrin 2016), and LightGBM (Ke al. (017) are GBDT models. These models utilize boosting construct ensemble of small trees for evaluation. GBDTs are robust to uninformative or heavy tail features and achieve com- petitive performance over baselines across different dataset properties. Our work argues in-context learning is a potential competitor against GBDTs, as PFN transformers can potentially learn better dataset prior than GBDTs.
G.2.3
DEEP LEARNING ALGORITIMS
ResNet (Gorishniy et al. 2021). MLP-rtdll TabNet (Arik & Pfister 2021). STG (Yamada et al.
2020
VIME (Yoon al.] 2020), NODE (Popov et al. 2019). FTTransformrr (Gorishniy et al,
SVM (Cortes & Vapnik 1995), DANet (Chen et al. 2022) and SAINT (ooepalli et al. 2021 are deep learning-based algorithms. In particular, ResNet a Convolutional Neural Network designed for tabular learning. MLP-rtdl and MLP are 2 implementations of multilayer petcepttrr.......oooo... SVM is a support vector machine. TabNet and STG is a neural network architecture that aims to learn GBDT-like operations in fully differentiable manner. VIME is a gated neural network that first traing with self supervision. MIXTUREPFN outperforms deep learning algorithms by learning prior that better regularizes the learning procedure. NODE is a neural network architecture that aims to imitate GBDTs while being fully differentiable and end2end. DANet specialized deep learning architecture for tabular data. FTTransformer is feature encoding transformer model designed for tabular data. SAINT a self-supervised transformer designed for tabular data.
2021
G.2.4 SIMPLE ALGORITHMS
RandomForest, DecisionTree, LinearModel, and KNN are all standard machine learning algorithms. We highlight, although MIXTUREPEN is based on evaluating prompts only on KNN neighborhoods, it drastically outperforms KNN. This suggests that In-Context Learning- based models trained local neighborhood can outperform both complicated models trained on the entire dataset and simple models that return the average of local neighborhoods. Indeed combining KNN and Large Language
Models have been highly successful (Xu al. 2023 Liu et al. 2021
Guu et al. 2020).
H ADDITIONAL RESULTS
H.1 ACCURACY RESULTS AND STANDARD DEVIATIONS
We provide the same Condorcet experiments the main paper but with the accuracy metric in Table 4 We provide the Condorcet matrix in Figures and These results support the same conclusions found in the main paper.
27

---

Published as a conference paper at ICLR 2025
Method
Number of Datasets Completed On
CatBoost Prokhorenkova al.
2018
35
XGBoost (Chen & Guestrin
2016
36
MLP-rtdl (Goodfellow al.
2016
Gorishniy et
2021
36
MLP Goodrellow et al.
201
McBlfresh et al. 2023
36
ResNet (Gorishmyy et al.
2021
35
Randoml'ores (Liaw al.
2002)
35
Decision Tree QQinnln
1986)
35
MixturePFN
34
TabPFN (Hollmann et al.
2022
McElfresh et al. 2023
34
LinearModel (Cox 1958)
34
TabNet (Arik Phister 2021
33
KNN (Cover & Hart 1967)
33
LightGBM (Ke et al.
2017)
32
VIME Yoon et al. 2020)
32
STG Yamada et a..
2020)
31
NODE (Popov et al.
2019
30
FTTransformer (Gorishnry et al.
2021
29
SVM (Cortes Vapmik
1995)
29
SAINT Somepalli et al. 2021
27
DANet Chen et al. 2022)
27
The number of datasets each algorithm completed across the entire 36 dataset TABZILLA benchmark. Note, the 2 datasets that MIXTUREPEN and TABPFN (McElfresh et al., has too many labels, being unsupported by the pretrained TABPFN (Holl- mann et al. 2022). However these datasets are not outliers compared to the 34 datasets that are supported. Note, TABPFN achieves the same results as TABPFN* except only running on the 17 datasets with ,3,000 features, hence we compare against the more powerful TABPFN baseline in- stead.
Table 11:
2023) does not
28

---

Published as a conference paper at ICLR 2025
Condorcet Wins
Mean Rank
Median Rank
Min Rank
Max Rank
MIXTUREPFN
522
2.833
2
9
TABPFN*
501
4.333
4.5
2
7
TabR
324
17.333
18.5
8
22
FTTransformer
310
6.667
6.5
3
12
MLP-PLR
223
12.417
14
2
19
Table 12: Comparison additional recent deep learning algorithms.
We provide the detailed statistics of the ranking experiments with both log-likelihood and accuracy across all, Top- 10, and Top-5 subsets in Tables 6 conclusions found in the main paper.
10
These results support the same
We provide the Wilcoxon-Signed Rank Test with both the log-likelihood and accuracy metric across the all and Top-10 10 subsets in Figures 12 and 14 These results support the same conclusions found in the main paper.
We provide experiments with even lighter hyperparameter tuning, as discussed in Section ! we call this model, MIXTUREPFN-lite. MIXTUREPEN is tuned over 280% less configurations than baselines. MIXTUREPFN-lite is tuned only on 2 hyparparameter settings. We provide the main paper results and Condorcet matrices as presented in Table 18 and Figure 17
MIXTUREPEN is the Condorcet winner and achieves the top mean rank across all experimental settings, with statistical significance among all and Top-10 subsets. MIXTUREPEN's Log-likelihood results are slightly better, because many algorithms are tied in accuracy across the benchmark. When this occurs, MIXTUREPFN more confident than baselines when it is correct.
H.2 ADDITIONAL BASELINES
We implemented and evaluated TabR (Gorishnly et al. 2023) and MLP-PER (Gorishniy et al. 2022) across 5 hyperparameter settings on the TabZilla benchmark. We show the Condorcet wins and rank evaluated across all algorithms and datasets (now including TabR and MLP-PLR) Table 1. MIXTUREPFN outperforms recently proposed algorithms under the same hyperparameter tuning budgets. Note, both baselines require heavy hyperparameter tuning in their respective papers. We provide results Table T2
reported in
Compared LoCALPEN (Thomas al. 2025 which finetunes single TabPFN model for trieval, MIXTUREPEN finetunes a specialized expert on each subset of the training/context data, improving effectiveness. Hence, LoCALPFN and there is only one expert. As seen in Table 13 MIXTUREPFN outperforms both TabPFN and LoCALPEN by training specialized expert on each subset of the training dataset. The more experts that are trained, the better the performance.
specific instance of MIXTUREPFN: when y =
we use different types of categorical encodings (target and one-hot encoding), filtering each cate- gorical feature to have at most K dimensions using MRMR, and using different numbers of clusters (where number of experts proportional to 7) in Table 14 These additional ablations show per- formance is more sensitive to the number of experts than choice in categorical encoding algorithms. Hence, it is MIXTUREPEN 's MICP and CaPFN are the core reasons for its superior performance.
We find 3 very large tabular datasets to test our approach. We limit MixturePFN clusters to at most 64 and adopt only target encoding for categorical features. As seen in Table 7 even under these handicaps. MixturePFN consistently improves TabPFN*'s accuracy (dataset size in brackets).
H.3 NORMALIZED ACCURACY
We evaluate the normalized accuracy across XGBooST over shared datasets addition to provided metrics in Table 16, where find MIXTUREPEN is still the best performing model.
29

---

Published as a conference paper at ICLR 2025
Max Experts
Electricity Acc
Phoneme Acc.
Airplane Acc.
TABPFN*
no Finetune
no KNN
81.2%
88.3%
60.0%
LoCALPENi
1
00
85.4%
88. 75.7%
64.0%
MIXTUREPEN
85.6%
88.5%
64.3%
MIXTUREPEN
8
5 88.0%
89.1%
64.9%
MIXTUREPEN
1024
89.7%
90.2%
85.7%
Table 13: Comparison to recent retrieval models.
Encoding- K
Electricity Acc
Phoneme Acc.
T
target-1
no KNN
81.2%
88.3%
MIXTUREPEN
target-1 1
89.7%
90.2%
MIXTUREPEN
target-1
87.4%
87.4%
MIXTUREPEN
onehot-1
86.7%
86.7%
MIXTUREPEN
target-5
87.4%
87.8%
MIXTUREPEN
onehot-5
86.3%
87.9%
Table 14: Comparison with different categorical encodings.
Poker(1MM)
HiggsBig(940K)
BNG(IMM)
MIXTUREPEN
52.5%
66.7% 69.0%
87.4%
TABPEN
60.0%
89.0%
Table 15: Comparison on additional very large datasets.
Algorithm
Normalized Accuracy
XGBooST +0.0%
CATBOOST
-0.704%
TABPFN -3.381%
MIXTUREPEN
+0.561 %
Table 16: Normalized Accuracy Results
30

---

Published as a conference paper at ICLR 2025
Accuracy
K
Time
Method
Prompt
Mean
Mean
Mean
TABPFN
83,42%
1.00
1.24s
M (y = 1.0)
83.96%
2.25
0.90s
M (? 3.0)
84.23%
5.25
1.02s
M (7 = 5.0) 84.23%
8.54
0.83s
Table 17: Trade-off of 7. T T*MMICP is short for TABPFN* +MICP. Note 7y increases, the accu- racy and number of prompters increases, while TABPEN inference time remains constant. Routing are negligible with optimized nearest neighbor search (Douze et al. 2024).
X
10
103
std Kurtosis
#Feats
X
102
X
101
X
X
100
102
103
104 #Samples
105
106
Figure 16: Dataset properties of chosen algorithms from the TABZILLA benchmark. We plot dimensions the dataset properties of all 36 dataset from TABZILLA. The bold points represent the held-out datasets. As shown, the 34 chosen datasets covers wide-variety of dataset properties.
H.4 SENSITIVITY RESULTS ON MORE DATA
We provide the same sensitivity analysis conducted in the main paper but with the accuracy metric in Figure These figures support the same conclusions found the main paper.
We provide the same sensitivity analysis conducted in the main paper but across the number of features, number labels, and feature irregularity in Figures TO and 11 These figures support the same conclusions found in the main paper.
H.5 EFFICIENCY EFFECTIVENESS TRADE-OFF
? effectively trade-off's efficiency and effectiveness. As mentioned in Section 3.1.1 MIX- TUREPEN uses a single hyperparameter, to control the efficiency effectiveness tradeoff. we plot the average accuracy of TABPFN +MICP across y [1.0, 3.0, 5.0]. across the entire dataset. As shown in Table I7 as the hyperparameter increases, MICP's effectiveness is reliably trade-off
for efficiency.
H.6 DETAILED RESULTS
We provide MIXTUREPEN's and TABPEN* accuracies across the 10-folds on all datasets in Ta-
bles 20 and 21
31

---

Published as a conference paper at ICLR 2025
Condorcet Statisties
Method
#Votest
#Wins
#Ties
#Losses
MixturePFN-lice
503
19
0
XGBoost
502
18
0
CatBoost
479
17
0
2
SAINT
404
16
0
3
TabPFN
385
13
5
LightGBM
374
14
I
4
DANet
312
14
O
FTTransformer
295
12
0
7
ResNet
287
10
SVM
286
11
0
8
STG
286
9
0
10
RandomForest
248
7
0
12
NODE
244
0
12
MLP-rtdl
228
5
0
14
TabNet
210
5
0
14
LinearModel
202
3
15
MLP
193
5
13
VIME
134
2
0
17
DecisionTree
115
1
0
18
KNN
74
0
Table 18: MIXTUREPFN is the Condorcet winner across 36 datasets against 19 baseline algorithms. We rank algorithms based on their oog-liielihoods.
T IMPLEMENTATION
We implemented MICP by first preprocessing the train data into separate prompts via KNN, chunk- ing each prompt into batches, then called TabZilla APIs to the desired PFN model each batch. We implemented CAPFN by bootstrapping our training dataset then running maximum like- lihood loss the bootstrapped datasets. MIXTUREPFN's implementation is built off the official TaBZILLA codebase (git 2024b) and will be open-sourced on Github.
1.1 OPTIMIZING TABPFN'S IMPLEMENTATION
TABZILLA only obtained TABPFN results on 7 out of 34 benchmark datasets (cEFlfresh et al. 2023), due memory constraints. We identified an implementation inefficiency where "prompts" constructed with the entire test dataset, i.c. (Xtest: Dtrain), causing memory overflow. We optimized TABPFN* implementation by batching test samples, (train) X C Xtests with batch size 1024, and report results over all 26 datasets.
J HYPERPARAMETER SETUP
As TABPEN transformers can handle up to 3,000 training samples, we set B 3, 000. We em- pirically found the minimum number of iterations and batch-size required for loss convergence on the artificial- datasets. During inference, we use larger batch size, Nbatch 1024, as gradients no longer need to be stored. We finetuned the model using the Adam optimizer with a learning rate of 0.001. As TABPFN transformers can handle up to 100 features, for datasets with over 100 features and TABPFN-base models, we Maximum Relevance and Minimum Redundancy (mRMR) feature selection (Ding & Peng 2005) to reduce the number of features to 100. We follow the TABZILLA benchmark, setting Ne Nensemble 16, which shuffles features N Nensemble/ /2 times for both the orig- inal and applies power-law scaled features. MIXTUREPEN's router was implemented using the FAISS (Douee et al. 2024 library.
dataset to be 128 iterations and Noatch = 64, which we set for all other
32

---

Published as a conference paper at ICLR 2025
35
Mixture...
XGBoost
CatBoost
30
TabPFN
ResNet
LightGBM
25
SAINT
FTTrans...
MLP-rtdl
20
DANet
RandomF...
15
STG
TabNet
10
LinearM...
NODE
5
VIME
Decisio...
KNN
Mixture...
XGBoost
CatBoost
TabPFN*
ResNet
LightGBM
SAINT
FT
MLP-rtdl
DANet
RandomF...
SVM
STG
TabNet
MLP
LinearM...
NODE
VIME
Decisio... KNN
Pairwise comparison matrix for Condorcet voting over the log likelihood metric with lightly tuned MIXTUREPEN. Note, MIXTUREPFN-li the Condorcet winner.
Figure 17:
Due to the large variability in datasets in the TABZILLA benchmark, try 4 hyperparameter set- tings: (1) y = 5.0, (2) 7 = 1.0, (3) y = 1.0 but MRMR with 50 features instead of 100 features for feature count scalability, and (4) 7 1.0 but with Catboost instead of Ordinal encoding for categor- ical feature scalability (Hollmann et al., 2022). Hyperparameters are chosen by picking the setting which maximizes performance on the validation set. Models are evaluated on the test set, which not seen during hyperparameter tuning. In contrast to all other baselines, which are tuned across 30 hyperparameter settings, MIXTUREPEN performs much less hyperparmeter tuning than baselines. Baseline hyperparameter settings are the same as the TABZILLA (McElfresh et al.) 2023) bench- mark. Note, even if only the 7 parameter tuned (i.e, only settings (1) and (2)), MIXTUREPFN is still much better than TABPFN, 10-folds following TABZILLA (McElfresh et al., 2023) and OpenML. We tune the hyperparameters by splitting the train set of each fold into training and validation following TABZILLA (McElfresh et al., 2023). Ablation studies were performed modifying hyperparameter setting (1). Dataset pre- processing details be found in Appendix G.T
presented in Table [8 and Figure 17 All results were collected over
K HARDWARE
All experiments conducted Nvidia V100 GPU and AMD EPYC 7402 CPU. Each experiment is given a budget of 10 hours for a single dataset and algorithm.
33

---

Published as a conference paper at ICLR 2025
Dataset Properties
Top Algs.
Dataset
#Samples
#Feats
#lab.
Std. Kurt.
1st
2nd
lymph
148
18
17.04
XGBoost
TABPFN
audiology
226
69
24
None
XGBoost
heart-h
294
None
MLP-rtdl
MIXTUREPEN
colic
368
26
4.0
XGBoost
MIXTUREPEN
monks-prob...
601
None
MIXTUREPEN
MLP-rtdI
balance-scale
625
4
3
0.02
MIXTUREPEN
TABPFN
profb
672
0.95
MIXTUREPEN
MLP-rtdl
Australian
690
14
2.0
XGBoost
TABPFN
credit-approval
690
15
74.77
TABPEN
MIXTUREPEN
vehicle
846
18
4
15.16
MIXTUREPEN
TABPFN
credit-g
1000
20
1.92
MIXTUREPFN
TABPFN
qsar-biodeg
1055
41
93.24
MIXTUREPFN
TABPFN
cnae-9
1080
856
9
None
MLP-rtdl
MLP
socmob
1156
None
XGBoost
TABPEN
100plants
1599
64
100
17.66
XGBoost
mfeat-fourier
2000
10
0.64
MIXTUREPEN
TABPFN
mfeat-zemike
2000
47
10
1.42
MIXTUREPFN
TABPFN
kel
2109
21
1
28.34
MIXTUREPEN
TABPFN
jasmine
2984
144
47.6
MIXTUREPEN
XGBoost
splice
3190
60
3
None
MIXTUREPEN
XGBoost
Bioresponse
3751
1776
328.77
XGBoost
MIXTUREPEN
ada-agnostic
4562
48
None
XGBoost
MIXTUREPEN
phoneme
5404
1.23
MIXTUREPEN
XGBoost
SpeedDating
8378
120
36.43
MIXTUREPFN
XGBoost
GesturePhase.
9873
32
5
52.18
MIXTUREPFN
XGBoost
artificial-char...
10218
10
0.63
XGBoost
MIXTUREPFN
elevators
16599
2986.5
MIXTUREPFN
TABPFN
guillermo
20000
4296
None
XGBoost
TABPEN
nomao
34465
118
1100.34
XGBoost
MIXTUREPEN
jungle-chess...
44819
6
3
0.08
MIXTUREPFN
XGBoost
electricity
45312
2693.51
XGBoost
MIXTUREPEN
higgs
98050
28
15.53
XGBoost
MLP
MiniBooNE
130064
50
1686.9
MIXTUREPEN
XGBoost
alhert 425240
78
12162.65
MIXTUREPEN
XGBoost
airlines
539383
2.01
MIXTUREPEN
XGBoost
poker-hand 1025009
10
0.08
XGBoost
MIXTUREPEN
Table 19: Dataset statistics for valid TABZILLA benchmark datasets. Ranks are computed across algorithms that run on all datasets MIXTUREPFN runs on: MIXTUREPFN, TABPFN XGBooST, MIP, and MI.P-ridI. Note this list of datasets was originally curated from 197 datasets, to contain only those difficult for all models. We list the top 2 performing algorithms based on log likelihood, following TABZILLA, on each dataset. MIXTUREPFN achieves oftte-of-the-ar performance.
34

---

Published as a conference paper at ICLR 2025
Dataset
Model
Mean + Std Accuracy
TabPFN*
0.868=0.036
australian
MixturePFN
0.861=0.023
TabPFN*
0.791=0.018
bioresponse
MixturePFN
0.793=0.017
TabPFN*
0.569=0.014
gesturepha...
MixturePFN
0.704=0.011
T
0.927=0.003
miniboone
MixturePFN
0.046=0.003
T
0.856
0.006
speeddating
MixturePFN
0.887 0.011
TabPFN*
0.845 0.016
ada-agnostic
MixturePFN
0.842
T
0.600=0.003
airlines
MixturePFN
0.857=0.005
T
0.638=0.005
albert
MixturePFN
0.903=0.003
T
0.650=0.013
artificial...
MixturePFN
0.717=0.008
T
0.089-0.013
balance-scale
MixturePFN
0.997-0.010
TabPFN
0.896 0.029
cnae-9
MixturePFN
0.899 0.029
T
0.823 0.044
colic
MixturePFN
0.807 0.807-0.067 0.067
T
0.884-0.05
credit-app...
MixturePFN
0.872-0.053
0.729=0.028
credit-g
MixturePFN
0.740=0.021
T
0.812-0.005
electricity
MixturePFN
0,897-0.003
TabPFN*
0.900-0.006
elevators
MixturePFN
0.905=0.005
T
0.791 0.013
guillermo
MixturePFN
0.799 0.098 0.018
T
0.837 0.044
heart-h
MixturePFN
0.834-0.044
TabPFN*
0.665=0.007
higgs
MixturePFN
0.693=0.005
0.804-0.016
jasmine
MixturePFN
0.861=0.008
T
0.823=0.006
jungle-che...
MixturePFN
0.865=0.004
TabPFN*
0.862-0.011
kel
MixturePFN
0.866-0.013
T
0.096
lymph
MixturePFN
0.090
TabPFN* 0.828 0.025
mfeaa-fourier
MixturePFN
0.025
Table 20: Mean and Std. Accuracy of MIXTUREPEN and TABPEN* on all datasets across 10-folds (part 1).
35

---

Published as a conference paper at ICLR 2025
Dataset
Model
Mean + Std Accuracy 1
TabPFN*
0.828=0.015
mfeat-zernike
MixturePFN
0.846=0.024
T
1.00==0.000
monks-prob...
MixturePFN
1,000=0.000
T
0.953=0.003
nomao
MixturePFN
0.966=10002
TabPFN*
0.883=0.014
phoneme
MixturePFN
0.002
TabPHN
0.517=0.011
poker-hand
MixturePFN
0.002
TabPFN*
0.691=0.028
profb
MixturePFN
0.685=0.024
T
0.885=0.033
qsar-biodeg
MixturePFN
0.88==0.038
TabPFN*
0.933
socmob
MixturePFN
0.929-0.017
TabPFN
0.876
splice
MixturePFN
0.983=0.005
TabPFN* 0.84710.023
vehicle
MixturePFN
0.847+0.024
Table 21:
Mean and Std. Accuracy of MIXTUREPEN and TABPFN* all datasets across 10-folds
(part 2).
36


Reviews:
[{"review_id": "NKcIW6C5DI", "reviewer": "Reviewer_5Xhm", "summary": "The paper proposes a mixture of experts approach for in-context learning on tabular data. Each expert in the mixture is a K-means cluster and the model routes the input instance to the closest cluster. This addresses the problem of context size in large datasets and provides a better selection of prompt instance than random sampling. To adapt the model to this type of routing authors also propose fine tuning by selecting a cluster of each training instance and maximizing the likelihood.", "questions": "I could not find an ablation study on the number of clusters K vs model performance, have you done these experiments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is well written and proposes a justified solution to address the context length issue for in-context learning models such as TabPFN. Authors conduct extensive experiments on many real world dataset to demonstrate the effectiveness of the proposed approach and compare with leading tree-based and deep learning tabular methods.", "weaknesses": "There is a very related previous work \"Retrieval & Fine-Tuning for In-Context Tabular Models\" by Thomas et al, which proposes both nearest neighbor retrieval to improve the prompt and fine tuning with this approach to adapt the model to the target distribution. I think the authors have to compare with this work and highlight what is novel in MixturePFN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpFfU6KFWA", "reviewer": "Reviewer_DZw6", "summary": "The paper proposes the MixturePFN framework, which extends TabPFN for large tabular datasets by addressing the performance and scalability limitations of the number of table rows. The authors propose:\n1. Mixture of In-Context Prompters (MICP), which optimizes inference by using a sparse mixture of experts to route test samples to specific \"prompters\" that create context-specific prompts to separate large training datasets into manageable clusters. \n2. Context-Aware Finetuning (CAPFN), which addresses distributional shift issues by specializing each prompter on its assigned\ncontext via parameter efficient finetuning.", "questions": "See weaknesses.\n\nCan categorical features simply be encoded as ordinal features? Is that not implying false relationships between unordered elements?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The MICP strategy effectively reduces memory usage, allowing the model to handle larger datasets compared to existing TabPFN\n- CAPFN bootstrapping and finetuning approach appears to be an effective way to mitigate distribution shift ICL for tabular data\n- Extensive benchmarks against 19 strong baselines show good performance in both mean rank and Condorcet ranking across diverse datasets", "weaknesses": "- While MIXTUREPFN improves dataset scalability, it still struggles with feature-rich datasets, potentially limiting its applicability in domains with high-dimensional data, such as patient healthcare data. I realize the authors leave this to future work, but this is an area where simple XGBoost performs quite well, and I would be curious about their thoughts on tackling this issue.\n\n- MICP's reliance on K-Means clustering to segment data into meaningful clusters as the quality of clusters can vary significantly based on dataset properties / distance metric chosen. Poor clustering could lead to suboptimal routing and ineffective prompts for certain test samples. I'd be curious to see some ablations in this area.\n\n- The CAPFN bootstrapping method might introduce biases or overfitting if the sampled subsets are not representative of the entire dataset. Bootstrapping from small clusters may fail to capture enough diversity, especially in cases with imbalanced classes or rare features. I'd be also curious to see how this method works with highly imbalanced labels e.g. 1\\% positive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wfegIQtaJv", "reviewer": "Reviewer_X42j", "summary": "In this paper, the authors propose MixturePFN, an extension of Sparse Mixture of Experts to TabPFN to alleviate the context size limitations of the existing TabPFN. On the TabZilla benchmark, MixturePFN outperforms state-of-the-art tabular prediction models.", "questions": "1. Can you provide a comparison with LoCalPFN [1]? If not possible, I think the comparison should be done using k neighbor samples rather than random sampling, at least for TabPFN*.\n\n2. I see that the authors say in the limitations section that they didn't do it on a dataset with a million samples, but I'm somewhat curious about the effectiveness of MixturePFN on a dataset with a million samples, since the paper is aimed at the scale-up aspect.\n\n3. I'm also curious about the effectiveness of MixturePFN on datasets with hundreds or thousands of features, which is very practical in the real world.\n\n----\n[1] Thomas et al., Retrieval & Fine-Tuning for In-Context Tabular Models, NeurIPS 2024", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of Mixture of Experts blending into TabPFN seems novel.\n\n2. The effectiveness of MixturePFN is well evaluated in well-established benchmarks against a variety of baseline methods.\n\n3. Writing is easy to follow.", "weaknesses": "1. The biggest weakness I think is that the paper is missing a comparison with LoCalPFN [1]. Since LoCalPFN also tries to make TabPFN effective even on datasets with many-shots, I think it should be mentioned in the paper.\n\n----\n[1] Thomas et al., Retrieval & Fine-Tuning for In-Context Tabular Models, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)