TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Authors: Benjamin Feuer
Robin Tibor Schirrmeister
Valeriia Cherepanova
Chinmay Hegde
Frank Hutter
Micah Goldblum
Niv Cohen
Colin White
Venue: NeurIPS 2024 poster (2024)
Keywords: tabular data
prior-data fitted networks
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We introduce TuneTables, an algorithm that allows prior-data fitted networks to scale by orders of magnitude and achieve strong performance on large datasets.

Abstract: While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. However, current PFNs have limitations that prohibit their widespread adoption. Notably, TabPFN achieves very strong performance on small tabular datasets but is not designed to make predictions for datasets of size larger than 1000. In this work, we overcome these limitations and substantially improve the performance of PFNs via context optimization. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We conduct extensive experiments on nineteen algorithms over 98 datasets and find that TuneTables achieves the best performance on average, outperforming boosted trees such as CatBoost, while optimizing fewer than 5\% of TabPFN's parameters. Furthermore, we show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks
Benjamin Feuer', Robin Tibor Schirrmeister", Valeriia Cherepanova 4, Chinmay Hegde', Frank Hutter", Micah Goldblum', Niv Cohen" Colin White Micah
New York University. 2 University of Freiburg," University of Maryland, 4 Abacus.
Abstract
While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this ap- proach. Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single for- ward pass. However, current PFNs have limitations that prohibit their widespread adoption. Notably, TabPFN achieves very strong performance on small tabular datasets but not designed to make predictions for datasets of size larger than 1000. In this work, we overcome these limitations and substantially improve the performance of PFNs via context optimization. We introduce TuneTTbbes, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We conduct extensive experiments on nineteen algorithms over 98 datasets and find that TuneTables achieves the best perfor- mance on average, outperforming boosted trees such as CatBoost, while optimizing show that TuneTables
fewer than 5% of TabPFN's parameters. Furthermore, can be used as an interpretability tool and can even be used to mitigate biases by optimizing faimess objective. We open-source code and raw results bttpps//ee //gtthu/./tttt.g./// thub. com wttp:b/c//eeeeeeeeeee http///eeer/Ttnealllleee
Introduction
Tabular data, or data organized into rows and columns consisting of distinct features, are the oldest and one of the most ubiquitous types of data in machine learning in practice [10, 67]. Tabular data has numerous applications across medicine [41, 71], online advertising [33, 52, 64], finance [7, 18], and other areas [11, 12, 72].
Competitive classification algorithms for tabular data include gradient- gradient-boosted decision trees [16, 61] and deep neural networks [31, 42, 69]. Both approaches fit their respective models labeled dataset containing samples from distribution reflecting the task hand. recent breakthrough, prior-data fitted networks (PFNs) [35, 55], are are specific type of neural process which learn perform approximate Bayesian inference in a single forward pass using in-context learning [50]. PFNs do not require optimizing parameters training data into the context and conditioning on it. In particular, TabPFN achieved state-of-the-an classification small tabular datasets [35, 51].
fitting model on downstream training data, instead feeding
The in-context learning approach of PFNs parallels that of large language models (LLMs) [82]. Both approaches can be viewed as approximate Bayesian inference, whether implicitly [79] or explicitly [55]. While researchers have successfully used various context optimization strategies for enhancing LLM performance [49], prior work has studied context optimization strategies for PFNs. Furthermore, although TabPFN achieves very strong performance on small datasets, its limitations
I Equal advising. Correspondence to: bé9960nyu.edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

TabPFN
Tabee
Bias Mitigation
abjeccteeveeeee prompt tuning allows the user to milligate biases
Pretrain on synthetic data
1. Pretrain on synthetic data
Predict D..,, using D (size <1000), as context
2. Prompt tune on D Iran
3. Predict Ast
Interpretability
Tuned context helps in understanding the discriminative dataset features
F2
A2
eg
F4
44
66
FiG
Inference time
Remove training data from context for faster inference
Bid
02)
D
Fine-tuned
Dust
C4
content
[optional]
Figure 1: TuneTables: a novel prompt-tuning technique for prior-data fitted networks. Tunc bles performs prompt tuning pre-trained prior-fitted network (TabPFN) to distill real-world datasets into learned embeddings, allowing for stronger performance and faster inference time than TabPEN in many cases. Tune'Tables also expands the capabilities of pre-trale pre-trained PFNs; by way of example, we demonstrate its effectiveness for bias mitigation, and as an interpretability tool.
currently prohibit its widespread adoption: it only runs on datasets whose number of training samples, number of features, and number of classes most 1000, 100, and 10, respectively.
In this work, we perform the first investigation into context optimization strategies for PFNs, allowing us to substantially improve their performance when scaled to large datasets. Specifically, we introduce TuneTables, a novel parameter-efficien fine-tuning technique for PFNs that compresses large datasets into smaller learned context (Figure 1). We conduct an extensive empirical investigation into TaneTables' performance on the TabZilla Benchmark Suite, the largest benchmark considered by recent tabular data literature [51]. Over 98 datasets and 19 algorithms, we find that Tune Tuneeaables achieves the best average performance and the best-performing method on of them.
Because Tune effectively compresses the contents of large tabular datasets into the tuned prompt, no training data needed in the context during inference, significantly speeding up inference time (similar to works on neural processes; see Section 6). We also show that the learned prompt can be used as tool for interpretability. Finally, we show how use TuneTaables for multi-objective optimization, such optimizing both accuracy and fairness simultaneously. allowing users to mitigate biased predictions of pretrained PFNs with just a lightweight tuning procedure. We open-source our code and raw results at https: //githut.. com/ coompenfeet penfever// Taanffeeruuuueeee TTueeables..
Our contributions. We describe main contributions below.
We introduce Tanelables, parameter-efficient fine-tuning technique for PFNs. TuneTables achieves the highest number of wins (30) when compared to 19 algorithms over 98 datasets, while requiring less inference time than TabPFN.
We show how to use prompt tuning for multi-objective optimization, such as optimizing both accuracy and fairness simultaneously, allowing users to mitigate biased predictions of a pretrained PFNs with just lightweight tuning procedure.
We show that TuneTables' condensed representations can be used as an interpretability tool.
We conduct an extensive study on context optimization strategies for PFNs by performing an ablation study Tabela well studying sketching and feature selection techniques.
In order better manage the limitations of our method, we introduce Tune Tables-medium and TuneTables-B light, which achieve strong tradcoff's between precision and speed.

---

2 Background
PFNs:
review and limitations. this section, we give a background on PFNs and discuss their limitations. For complete description, see [35, 55, 58]. Assume that we have classification problem with features X' S R and labels y. Given a dataset D = Dusin U Duss, where Duain = (Zn) and Dest {Frest Vea)), our goal is to predict the conditional class
(121,11)..... yI....., (In, (Z.. probabilities p(- ). In the Bayesian framework for supervised learning, the mechanism for generating the data distribution is hypothesis yp, drawn from $, the space of all hypotheses. + encodes our prior beliefs on the system before observing data. In this framework, datasets D are generated by first drawing yp Φ and then drawing i.i.d. samples according y. The posterior predictive distribution (PPD) for a test sample Trest is the label distribution pl- TTest Duain) that follows from our prior. We can obtain the PPD by integrating over the space of hypotheses &:
ply x, D) & /. ply x, ()p((( p)p(p))
PFN is a transformer-bast architecture trained to approximate the PPD via synthetic prior-fitting. Given a prior, we first sample hypotheses ip p(y) and then synthetic datasets D p(D pp). We optimize the parameters of the PFN by predicting the class labels of Dress C D, conditioned on Dusin D Drea- We compute the loss by:
LPPN DD~D(D) |- log qo (Btest Trest Dirain]
for simplicity assuming all training and test sets are size and respectively. We then approximately solve this optimization problem, t arg ming Lpvx, allowing qo to approximate Equation (1):
To (yhest
Tiest
Dirain) Rs p(yeest
Trests Dirain).
Scaling challenges. While PFNs, specifically TabPFN, have shown remarkable success in classifica- tion by in-context learning, several important obstacles constrain their more widespread adoption:
1. PFNs only accept fixed number of features. The current design of PFNs fixes the quantity of features at the time of pretraining. This quantity cannot be changed without retraining the PFN.
2. PFNs scale poorly with the dataset size. While PFN accuracy can improve with more real-data samples at inference time [35, 58], the memory requirements scale with the context length, making extensions beyond certain number of samples impractical.
3. PFNs only select from a fixed number of classes. The MLP decoder head co-trained with the PFN fixes the number of classes that can be identified at test time.
A motivating example. In Figure 3 (left), we present a comparison between CatBoost and TabPFNs3000 from [51], version of TabPFN that, for datasets above 3000 data points, uses random subset of 3000 data points, for fitting; this version also evaluates 30 feature subsets based on mutual information (requiring 30x more time). We ablate our feature and sample subselection strategies for TabPFNs3000 Appendix Table 13. Although TabPFNs3000 performs very well on datasets with fewer than 1000 datapoints and 100 features, it significantly underperforms CatBoost beyond those constraints.
Approach. We propose sketching, feature selection, and fine-tuning as an attempt to remedy (1) and (2). Then in Section 3, we describe novel prompt-tuning compt-tunnng and class-extension strategies to create Tablea, robust classifier which remedies (1)-(3).
2.1 Classical sketching and feature selection
The number of samples that a PFN can handle is limited to around 3000 by conventional GPU sizes. However, in the real world, datasets are often much larger. Going forward, we refer the maximum allowable context size of the PFN as n, and to the size of the real-world dataset as N.
Sketching.
Given a training dataset Derin of size N >> n, one option is to select a representative subset of the dataset, compatt Durins to use the context. In general, researchers have studied variety of data
3

---

summarization techniques, often called sketching, for tabular data [56]. In the context of a pretrained PFN qo. the goal is to find sketching function R-N In such that
EDDP(D) [- log go (thest Trest; s sDirain))
E (D))(D) [- log qo
( (Jhest Frest Dusai))
Here, we consider three sketching methods for TabPFN: random, in which we select a random subset of n datapoints from the full training set; k-means, in which we compute the n-means clustering [3] of Drain and select the n centers; and CoreSet, in which we compute a core-set of size r [2].
Feature selection. In addition to the limitation on dataset size, PFNs, such as TabPFN, also impose a limitation on the number of features d. Similar to sketching, given a dataset with D >>> d features, we can perform feature selection or summarization in order to adhere to the constraint (formally, finding function that reduces the feature dimension and approximates an expression similar to Equation (4)). Feature selection is a critical part of tabular classification, and there are several popular feature selection methods [13, 17]. We investigate three different methods: random, in which we randomly select a set of d features; mutual information, in which we select d features with the highest mutual information of the target dataset [75]; and principul component analysis (PCA), in which we take the first principal components. Section 4, we find that all sketching and feature selection methods plateau in performance well before approaching parity with GBDTs, which motivates our investigation of new scaling techniques.
2.2 Fine-tuning
We conclude this section with a discussion of another potential approach for scaling PFNs: fine- tuning. In this approach, given a dataset of size N >>> we use gradient descent to continue training all parameters of the PFN. However, we show in Section that fine-tuning takes up considerable memory resources while still not achieving competitive accuracy on large datasets; we postulate that for most datasets, the synthetic prior of TabPFN is more robust to overfitting than the actual training data; fine-tuning overfits to the validation set. This is borne out by the observation that when fine-tuning TabPFN, validation accuracy continues to improve even as test accuracy declines. To remedy this, we present new parameter--efficiet solutions for scaling PFNs in the next section.
3 Tabee
Motivated by the strong performance of prompt tuning, we introduce Taneeables, tabular classi- fication algorithm. Using pretrained PFN (TabPFN, in our experiments) as a base model, TabeTe overcomes the limitations of PFNs, allowing them to be applied any tabular classification problem.
For the full details of the algorithm, please refer to Appendix D. We summarize here: (a) if a dataset is small enough to with the original zero-shot version of TabPFN, we include it in search, as the TabPFN is already highly optimized for such datasets; The reason why we do not only use TabPFN is that the exact size at which TaneTTables outperforms TabPFN is detaset (with an average transition of around 800 samples). (b) if there are more than 100 features, we perform grid search over set of feature subselection methods and select the which performs best on the validation set; (c) orthogonally, if there are too many labels, we fit a new decoder to a frozen TabPFN; (d) we optimize over a search space of tuned prompts, both with and without real-data context during training, fitted to the dataset; (e) we report the best-performing model according to accuracy.
Prompt tuning as a scalable context for PFNs. Motivated by the limitations of sketching for large contexts, explore soft prompt tuning an alternative. In soft prompt tuning [46], given a dataset D = Duain U Dreat parameterized matrix Doxe is prepended to the input embedding Daxe, where e is the transformer embedding dimension and p is a hyperparameter- the size of the tuned prompt.
The paper [46] demonstrates the effectiveness of soft prompt tuning for NLP tasks by prepending a small task-specific prompt (p & 5). These task-specific learned tokens prove effective at the extremely large model scales commonly encountered in NL.. Somewhat surprisingly, we show that prompt tuning is effective at similar scales of p even for the much smaller tabular data models (see Appendix D and Appendix F). However, prompt tuning increases in effectiveness when is larger.

---

15
12
11
VIMI
lune lables
KNNN
CatBoost
UneerMode]
XGBoost
MLP
LightGBM
STG
NODE
TeclsiorTree
ExcelFormer
TabNat
SAINT
rtll MLP
DANett
SVM
rtall Res/Vet
tul [TT mnsformrr
KandomFaress
Figure 2: Tabelables and state-of-the-art tabular models. A critical difference plot according to mean accuracy rank across the 98 datasets in Table I of [51]. Algorithms which are not significantly different (p > 0.05) are connected with a horizontal black bar. TuneTables achieves the highest mean rank of any algorithm.
Soft prompt tuning for tabular data. Unlike in NLP, transformers for tabular data, including PFNs, generally accept two input embeddings; continuous-valus valued embedding Diain and a categorically- valued Duinn which is passed through directly. We adapt the method of [46] by fitting the parameters of DRXe une x to Duain and randomly initializing D y with an equal number of labels from each class in Duain. These synthetic datapoints are optimized on the entire labeled set, and therefore give PFNs access to much larger training set not accessible by existing methods.
We further adjust the method of [46] to allow for the possibility that Dunn has learned, in essence, distilled version of Dunini at test time, we evaluate two settings, hereafter referred to C' ('context') and NC ('no context'). In the C setting, following [46], we have D prepended to the input embedding x In the NC' setting, we provide only DP at test time. Section 4, we empirically evaluate both approaches. We ablate specifics of our implementation choices in Appendix D. Unlike prompt tuning for NLP, we show that the NC setting is often competitive with, if not better than, the C' setting. We also ablate this choice during training, see Appendix D for implementation specifics.
Extending the number of predicted classes. TabPFN uses pretrained transformer, with two-layer MLP as a final classifier. The pretraining procedures limit the naïve use of TabPFN to classification tasks with at most 10 classes. Yet, in many cases, datasets of interest might contain larger number of classes, which would require pretraining new PFN from scratch.
Following the method of last-layer retraining [44, 45], for such datasets, we fit a PFN to new posteriors given by real-world classification datasets with more than 10 classes, freezing all weights except for those of the decoder ML. and the tuned prompt (see Appendix D for more implementation details). Even after this modification, our method remains highly parameter-effiiient, optimizing fewer than 5% of TabPFN's parameters.
Experiments
Algorithms and datasets used. We compare Tune to nineteen algorithms, including three
GBDTs: CatBoost [61], LightGBM [43], and XGBoost [16]: 11 neural networks: DANet [14],
FI-Transformer [31], two MLPs [31], NODE [60], ResNet [31], SAINT [69], STG [80], TabNet [6]. TabPEN [35], VIME [8]], and ExcelFormer [15]; and five baselines: Decision Tree [62]. KNN
[21], Logistic Regression [22], Random Forest [48], and SVM [20]. For all algorithms, we use their official implementation; see Appendix for more details. We also compare T3bPFNs3000. We run the algorithms on the TabZilla Benchmark Suite introduced in [51]. This suite consists of 98 classification datasets from OpenML [74] with a diversity of sizes and number of features [51]. See Table 4 in Appendix C for list of all datasets with their statistics.
Experimental setup. For all algorithms other than TuneTables, we perform light hyperparameter tuning by running one default setting and iterations of random search using Optuna [4]); see Appendix C for details. Following [51], all of the algorithms come with their default set of hyperpa- rameters used the official implementation, and we used all of these settings. For Tables,, we optimize via a grid search described in Appendix D.
5

---

Caltfona S0P1155000
Cartooss: Tunetacles
1000 turetables
fla Cefffred Rott
0015
E
Blue aasaaasoooo Sooio Red Tarchalks
10
12
-142
100
10
100
107
Figure 3: Tablea addresses TabPFN's limitations. (Left) Motivating example (using the subset of [on which both CatBoost and TabPFNs3000 report results): TabPFNs3000 is best on small datasets, but when scaled past 3000 datapoints and 100 features, TabPFNs3000 significantly underperforms. (Middle) CatBoost vs. Tabela on LARGESCALETABLES By contrast, TaneTa-- bles is competitive with CatBoost on all datasets, mitigating the limitations of TabPFN. (Right) TabPFNs3000 vs. Tabels on LARGESCALETABLE on datasets with a high number of datapoints or features. The colorbar on the y axis represents the comparative change in per-dataset accuracy between two algorithms (A: blue, B: red). Positive numbers represent the absolute gain in accuracy of B w.r.t. A, negative numbers represent the absolute gain in accuracy of A w.r.t. B.
TaneTTbbes outperforms TabPFNs3000
We fit for up to 100 epochs with early stopping- For all algorithms, we report the test performance of the hyperparameter set with the best performance on the validation set and cross-validate on three train/test folds from OpenML. We conduct experiments
NVIDIA L4 TPU with 24GB VRAM. We summarize our results across datasets by reporting mean accuracy as well as the mean normalized accuracy (after Z-score normalization) for each algorithm.
Algorithm comparison. We compute statistically significant performance differences among algorithms averaged across all datasets, as done in prior work [42]. We first use a Friedman test to check whether performance differences between all algorithms are significant [30]. We reject the null hypothesis for p < 0.05. Then we use a Wilcoxon signed-rank test to check which pairs of algorithms have significant performance differences [19]. We use a Holm-Bonferroni correction to account for multiple comparisons [36]. See Figure We find that TuneTables achieves the highest average rank, although the difference among the top three algorithms is not statistically significant. In Table 1, we present the accuracy, rank, and Zcooree of all algorithms, averaged across all datasets, and find that TuneTables performs very well on all metrics but runtime.
Large datasets. One limitation of the TabZilla Benchmark Suite is that even the largest dataset in the comparison contains only 45,211 samples. This scale is modest by the standards of mod- emn tabular problems. order to better understand the performance of Tables on extremely large datasets, we curate from [51] a novel benchmark, LARGESCALETARLLE
consisting of 29 datasets with up to 1 1900 000 samples, and 7200 features. We curate the datasets from OpenML, omitting image classification datasets. Since TabPFN generally outperforms boosted trees these smaller datasets, and Tabees extends TabPFN, we also heuristically select smaller datasets to LARGESCALETABLEES so not to favor either TabPFN or boosted trees. TuneTables achieves the highest average accuracy of any algorithm, and achieves the best performance on poker-hane a dataset of size 1 025 009. The complete results can found in Appendix Table 8 and Figure (right). In order to assess the performance of Tune'ables on datasets with many classes, we curate another subset of [51], presenting results on fifteen datasets with up to 100 classes. In appendix Table 7 we show that despite the large divergence from the PFN's pretraining, TuneTables achieves mean rank of 2.52, abead of CatBoost and a ResNet, second only to XGBoost, whose mean rank is 2.0.
Runtime comparison. We divide our consideration of runtime into inference and training. At inference time, Tables is around 9x faster than TabPFNs3000, on average; see Appendix Table 10 for the per-dataset details. However, the end-to-end runtime of Tabela is over 7x that of CatBoost and XGBoost, and also slower than TabPFNs3000, because of the increased training time.
In order to better understand the trade-off between accuracy and runtime, we introduce efficient variants of our method. TuneTTales-mediue utilizes a more efficient adaptive sequence size (i.e., the number of real data points received at train time) which scales with the size of the dataset, validates

---

Table 1: TaneTablle matches SOTA algorithms 98 datasets. In this table, compare algorithms over the 98 datasets in the TabZilla benchmark suite from [51]. For each algorithm, we compute its accuracy and mean rank in terms of accuracy. We also compute the mean Z-score, computed by normalizing the set of results on each dataset (by mean 0 std. 1), so that each dataset has the same weight, and averaging each algorithm's normalized performances. Std. 7-Score is computed with respect to random splits and averaged across datasets. Num. wins values are averaged over three
splits per dataset.
Model
Mean
Mean Rank
Mean Z-Scores
Std Z-Scores
Med Z-Scores
Num. Wins
TubeTables
0.860
7.435
0,494
0.624
0,490
30
CarBoost
0.856
7.514
0.496
0.669
0.566
13
XGBoost
0.854
7.991
0.411
0,783
0,533
16
ExcelFormer
0.847
9.349
0.212
0.863
0.384
LightGBBM
0.845
9.122
0.284
0.894
0.431
20
RundomfFurest
0.843
9.713
0.206
0.776
0.374
SAINT
0l840
9.619
0.132
0.985
0.337
10
DANet
0.840
9.646
0.209
0.763
0.345
ndl_ ResNet
0.839
9.691
0.175
0.798
0,356
ndl FTTrrasfoorrrrr
0.838
9.782
0.198
0.786
0.273
NODE
0,838
9.335
0.224
0.695
0.357
SVM
0.835
9.847
0.121
0.812
0.318
12
DecisionTree
0.823
12.192
00308
1.215
+0.038
MLPP
0.813
10.825
00.19
0,907
0.223
STG
0.810
12.196
-0.290
1.051
-0.057
TahNet
0.804
11.643
-0.216
1,104
0.075
MLP
0.802
12.220
40.232
0.845
40.105
LinearModel
0.793
13.069
+0,200
1,233
-0.345
KNN
0.781
14.101
-0.727
1.187
-0,608
VIME
0.756
14.711
-0.849
1.192
-0.694
Table 2: madies and TabelTables-iight substantially faster with only modest
decrease in accuracy. We compare the average accuracy and runtime (in seconds) of three versions of Tables and find that the medium and light versions of the algorithm are substantially faster large datasets; we also find that TanneTables-maan -eeles- sacrifices little accuracy.
Tone Tables
Tune madeem
Tune Tables-light
(LARGESCALLTABLEE size (0K)
0.831
0.830
0.810
Avg. runtime BARREEEAAAFAAAAL size <, 30K)
1325
1026
450
Avg. (LARGESCALETABLES. datascts)
0.830
0.828
0.787
Avg. runtime (LARGESCALFTABLES all datasets)
6975
1908
486
(TabZilla, all datasers)
0.861 $73
0.855
0.854
Avg. runtime (TahZilla, all datasets)
305
196
on subset of the available validation set when the validation set is large, employs lower patience for early stopping, omits a zero-shot TabPFN grid search over 30 random seeds which Tabela standard uses find more optimal feature selection subsets, and, most impactfully, omits ensembling for datasets larger than a cutoff hyperparameter (which we fix at 150 000 samples). TuneTables- light includes all of the efficiency modifications of Tune'Tables medee--mddimm, trains for just epoch, and uses TabPFN zero-shot to preselect the feature selection method rather than performing a grid search using TuneTTbles Tables itself. In Table 2, we compare these lighter methods to Tune Taale--tssaa benchmark.
the SAARGESSAALETARRA Taales--tandard, while the accuracy decreases by less than quarter of a percent. Furthermore, the runtime of Tune Tables-light is comparable to CatBoost; however, performance also degrades by 5% when going from TuneTables-medium TuneTables-lighh Still, TuneTables-lighh shows stronger performance than TabPFNs3000 (78.7% vs. 78.1 78.1% accuracy) while having lower inference time.
decreases runtime by 72% compared to
Ablations. In order to better understand the significance of the changes we introduce in our method, we separately ablate the tuned prompt size and the use of ensembles in Table 12 and Table 13, finding that smaller datasets pair well with smaller prompts and rarely benefit from ensembling.
We also compare Tune Tables with and without keeping the real data as additional context (referred to as C and NC, respectively, in Section 3); see Appendix Table 12. We find that smaller datasets cannot always fully learned by the tuned prompt, but for larger datasets, the tuned prompt alone suffices, and in some cases even outperforms the real data.

---

Table 3: Tune significantly improves accuracy and demographic parity. In these multi- objective optimization experiments, we consider prompt tuning for mitigating predictive bias, com- paring TabPFN Tabeea tuning for accuracy alone vs. accuracy and demographie parity. Tune Tables improves over TabPFN with respect to both objectives.
Adult
Speeddating
Compus
NLSY
Average
Acc 1
DP
Acc 7
DP
Acc
DP
Acc 7
DP
Ace 1
DP
TabPEN
0.832
0.174
0.86
0.012
0.688
0.22
0.986
0.326
0.842
0.183
Thne Tables (Ace)
0.865
0,006
0.200
0.843
Tine Tables (Ace DP)
0.837
0.034
0.863
0.003
0.693
0.121
0.965
0.277
0.840
0.109
Sketching and feature selection Finally, in Appendix Table 6 we give study on the three sketching and three feature selection techniques described Section 2. As described earlier, TabPFNs3000's performance when relying on feature selection plateaus well before approaching parity with CatBoost, a top perforrfonmin GBDT, on seven very large datasets.
Tabela Extensions
Mitigating bias with prompt tuning. Many real-world applications of machine learning involve a set of protected attributes (such as race or gender) that partition the dataset into groups, in which some have higher model performance than others. Removing the sensitive attributes does fix the algorithmic bias, because the sensitive attributes are often non-trivially correlated with other attributes in the dataset. Due to this issue, researchers have put in significant effort into mitigating the bias of ML models, with the majority of techniques devising new training strategies [9, 53].
Given TabPFN's pretrained nature and consider- able retraining cost, the only options for mit- igating biased predictions are to a post- processing routine on the output predictions, which generally do not perform well in-
Breast Cancer Tumar Type
Benign Malign
0.5
Feature
processing strategies [66]. We show how to use prompt tuning substantially reduce the bias of predictions while also improving accuracy.
-0.5
Clump Thickness
Cell, Size, Uniformity
Call Shape. Uniformity
Marginal Adhesion
Single, Epi Cell, Size
Nuclei
Bland Chromatin
Normal Nucieoll
Mitoses
We conduct experiments four datasets widely used for research fairness: the Adult Census Income database (with sex the sensitive at- tribute) [8], speed dating (with same race as the sensitive attribute) [83], COMPAS (with the sensitive attribute) [5]. and National Lon- gitudinal Survey (with gender the sensitive attriibute [73]. To quantify bias, we use de- mographic parity [25, 76], which measures the difference in probability of a positive outcome among the protected and unprotected groups. Formally, given protected group G,, unpro- tected group Go. and protected attribute F. it can be computed as
Figure Dataset with high accuracies from just two datapoints. Shown is a two-example prompt dataset for the breast cancer dataset [78]. Malign class example has higher values for all features than benign class.
Ple...DEG, (y: = (;,a)) - P(..,.)GG, (y: = x;aa)-
During prompt tuning, we employ a demographic parity regularizer that aims to minimize the difference positive outcome probabilities between the two groups:
Σ
P(y = Di.a)
Σ
P(y = Iiia)
(R..))EGo
We use the same experimental setup as in Section 4, except that we report one shift rather than the average of three, and T'uneTables is fitted to a single prompt rather than ensembled. We compare the default TabPFN to TuncTables, fine-tuning for accuracy alone accuracy and demographic
8

---

parity. The latter significantly improves demographic parity, compared to the default TabPFN and Tabela fine-tuned for accuracy, and enhances accuracy relative the default TabPFN across all datasets but one; see Table 3.
Summarizing and understanding datasets with prompt tuning. While we have demonstrated that Tables scales and improves the performance of PFNs, now we show that it can also help understand the discriminative features in a dataset. Often, besides a good predictive model for dataset, users want to gain further insights into the dataset. In prompt tuning, the tuned smaller dataset can be seen as a summary of the complete dataset that emphasizes discriminative features for the given task. As an example, in Figure 4 we show that on the Breast Cancer dataset [78], a prompt with just two synthetic examples is enough to reach high accuracies and at the same time allows an understanding of the predictive features. For example, in the Breast Cancer dataset, the malign example has higher values for all features compared to the benign example, suggesting high feature values indicate malignancy. We give further examples in Appendix F.
6 Related work
Neural Processes and Prior-Data Fitted Networks. Prior-data Fitted Networks (PFNs) [35. 55] are a prrentyypropose paradigm for machine learning, which show that fast approximate Bayesian inference is possible by training neural network mimic the posterior predictive distribution (PPD) in single forward pass using in-context learning [54, 55, 58]. PFNs were shown to yield state-of-thh-aa empirical performance on small tabular datasets [35, 51]. PFNs have been used in other applications, including Bayesian optimization [54] learning curve extrapolation [1], and as foundation models for hypernetworks [57].
PFNs are neural processes (NPs) [55]. Recent advances in NP are similar in nature to our work, especially recent works that aim to scale attention- methods to larger contexts. Feng al. [27] propose Latent Bottlenecked Attentive Neural Processes (LBANPs), a transformer-basee neural process which overcomes the quadratic complexity of transformers by encoding the context dataset into a constant number of latent vectors. Guo et al. [34] propose Versatile Neural Processes ( (VNP), which increases the capability of NPs to handle compex signals by using a new bottleneck encoder. [63] introduces semi aaii-paraettrc inducing point networks (SPIN), which can attend to a training set at inference time with linear complexity via inducing point methods. [28] introduce Constant Memory Attention Block (CMAB), attention block that is permutation-invariaa and has constant memory complexity when computing the output, as well as Constant Memory Attentive Neural Processes (CMANPs), NP that requires constant memory. For additional related work, see Appendix B.
7 Conclusions, limitations, and future work
In this work, we gave the first investigation into context optimization techniques for PFNs, allowing us to substantially improve their performance when scaled to large datasets. In particular, we introduced TuneTables, which uses novel prompt-tuning technique to achieve strong performance on large datasets. We demonstrate that Tune'Tables mitigates the constraints of TabPFN on the dataset size, number of features, and number of class labels. Additionally, we use prompt tuning to mitigate bias without retraining TabPFN and an interpretability tool. We open-source our code, results, and all other materials needed to reproduce our work. As PFN models improve, the context optimization techniques explored in our work will allow researchers to further optimize and scale. For example, next-gen ge-eeeeeeetnnn TabPFN might have a longer total context length, and prompt tuning will allow us to push the dataset size even further.
Limitations. No current TuneTables on par with GBDT's in terms of both accuracy and runtime simultaneously. We therefore emphasize that while we achieve strong performance metrics, we do not claim practical superiority of our method over gradient boosting, when taking into account training time. However, given the novelty of our method, we expect future research to further improve the accuracy-runtime runtime tradeoff of Tune'Table. also does not improve on TabPFN for small datasets (fewer than 1000 samples, 100 features and 10 classes); we postulate that this is result of overfitting.
Future work. Parameter-efficient fine-tuning can be used with PFNs to ensure high-quality, differ- entially private predictions, given that the pretraining is done on purely synthetic data, and prompt

---

tuning only updates a small number of parameters. Low-rank adaptation (LoRA) [37] and quantized low-rank adaptation (QLoRA) [23] have been used successfully for large language models and would be promising technique for parameter-effiientt fine-tuning of PFNs. Designing sparse mixture-of-experts PFN using router networks is another promising technique, due its success in the field of large language models [39].
References
[1] Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[2] Pankaj K Agarwal, Sariel Har-Peled Kasturi R Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry, 2005.
[3] Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: comprehensive survey and performance evaluation. Electronics, 2020.
[4] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2633 2019.
[5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. Ethics of data and analytics, pages 254-266- 264.264. Auerbach Publications, 2022.
[6] Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed- ings of the AAAI Conference Artificial Intelligence (AAAT), 2021.
[7] Kumar Arun, Garg Ishan, and Kaur Sanmeet. Loan approval prediction based on machine learning approach. IOSR J. Comput. Eng. 18(3):18-21 21, 2016.
[8] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
[9] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limita- tions and opportunities. MIT Press, 2023.
[10] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug. Martin Pawelczyk, and Gjergii Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01889) 889, 2021.
[11] Anna L Buczak and Erhan Guven. A survey data mining and machine learning methods for cyber security intrusion detection. IEEE Communications surveys & tutorials, (8(2):1153-111 2015.
[12] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1 2009.
[13] Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers & Electrical Engineering, 2014.
[14] Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. Danets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2022.
[15] Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Ziyi Chen, Jian Wu, and Jimeng Sun. Ex- celformer: A neural network surpassing gbdts on tabular data, 2024.
[16] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
10

---

[17] Valeriia Cherepanova, Roman Levin, Gowthami Somepalli, Jonas Geiping, C Bayan Bruss, Andrew Gordon Wilson, Tom Goldstein, and Micah Goldblum. A performance-driven bench- mark for feature selection in tabular deep learning. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[18] Jillian M Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint arXiv:2012.15330, 2020.
[19] William Jay Conover. Practical nonparametric statistics, volume 350. john wiley & sons, 1999.
[20] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning. 1995.
[21] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 1967.
[22] David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series (Methodological), 1958.
[23] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuming of quantized llms. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[24] Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Venkat Naidu, and Colin White. Forecastpfn: Synthetically-traine zero-shot forecasting. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[25] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, 2012.
[26] Y. Eldar, M. Lindenbaum, M. Porat, and Y.Y. Zeevi. The farthest point strategy for progressive image sampling. In Proceedings of the 12th IAPR International Conference on Pattern Recogni- tion, Vol. 2 Conference B: Computer Vision and Image Processing. (Cat. No. 9oC4444344-,5 pages 93-97 vol.3, 1994.
[27] Leo Feng, Hossein Hajimirsadeghi, Yoshua Bengio, and Mohamed Osama Ahmed. Latent bottlenecked attentive neural processes. In The Eleventh International Conference on Learning Representations, 2023.
[28] Leo Feng, Frederick Tung, Hossein Hajimirsadeghi, Yoshua Bengio, and Mohamed Osama Ahmed. Memory efficient neural processes via constant memory attention block. OpenReview, 2023.
[29] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[30] Milton Friedman. The use of ranks to avoid the assumption of normality implicit the analysis of variance. Journal of the american statistical association, 1937.
[31] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[32] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[33] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim: factorization-machine based neural network for etr prediction. In IJCAI, 2017.
[34] Zongyu Guo, Cuiling Lan, Zhizheng Zhang, Yan Lu, and Zhibo Chen. Versatile neural processes for learning implicit neural representations. In The Eleventh International Conference on Learning Representations, 2023.
11

---

[35] Noah Hollmann. Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In Proceedings of the International Conference on Learning Representations (ICI.R), 2023.
[36] Sture Holm, simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
[37] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICI.R) 2022.
[38] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
[39] Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint 240.02240.044088, 2024.
[40] Zhengbao Jiang, Frank F Xu, Jum Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 2020.
[41] Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[42] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 34, 2021.
[43] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. In Proceedings of the Anmual Conference on Neural Information Processing Systems (NeurIPS), 2017.
[44] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint 2rX1v 2022.
[45] Phuong Quynh Le, Jörg Schlötterer, and Christin Seifert. Is last layer re-training truly sufficient for robustness to spurious correlations? arXiv preprint 230x::3308.00473, 2023.
[46] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
[47] Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. ICLR, 2023.
[48] Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. news, 2(3):18-22 2002.
[49] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2023.
[50] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture opti- mization. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2018.
[51] Dunean McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Vishak Prasad, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
12

---

[52] H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner. Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: view from the trenches. In Proceedings of the Annual Conference on Knowledge Discovery and Data Mining (KDD), pages 1222-1230, 2013.
[53] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina I Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 2021.
[54] Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. Pfns4bo: In-context learning for bayesian optimization. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[55] Samuel Mülerr, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
[56] Alexander Munteanu and Chris Schwiegelshohn. Coresets-methods and history: A theoreticians design pattern for approximation and streaming algorithms. KI- KI-Künstliche Intelligenz, 2018.
[57] Andreas Müller, Carlo Curino, and Raghu Ramakrishnan. Mothernet: A foundational hypernet- work for tabular classification, 2023.
[58] Thomas Nagler. Statistical foundations of prior-data fitted networks. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[59] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-28300 2011.
[60] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[61] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2018.
[62] Ross Quinlan. Induction of decision trees. Machine learning, 1986.
[63] Richa Rastogi, Yair Schiff, Alon Hacohen, Zhaozhi Li, Ian Lee, Yuntian Deng, Mert R Sabuncu, and Volodymyr Kuleshov. Semi-parametric inducing point networks and neural processes. In The Eleventh International Conference on Learning Representations, 2023.
[64] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web, pages 521-530 2007.
[65] Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv preprint Xii::2207.03205 2022.
[66] Yash Savani, Colin White, and Naveen Sundar Govindarajulu. procesprocessig methods for debiasing neural networks. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
[67] Ravid Shwartz-Zi Ziv and Amitai Armon. Tabular data: Deep learning not all you need. Information Fusion, 81:84-90, 2022.
[68] Jack W Smith, James E Everhart, WC Dickson, William Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261. American Medical Informatics Association, 1988.
13

---

[69] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprini arXiv: /06.0:22 2021.
[70] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-2113 2021.
[71] Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health, pages 341-354. PMLR, 2020.
[72] Christopher J Urban and Kathleen M Gates, Deep learning: A primer for psychologists. Psychological Methods, 2021.
[73] US Bureau of Labor Statistics. National longitudinal surveys of youth data set, 2023.
[74] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 2014.
[75] Jorge R Vergara and Pablo A Estévez. A review of feature selection methods based on mutual information. Neural computing and applications, 24:175-186, 2014.
[76] Sahil Verma and Julia Rubin. Faimnsss definitions explained. In Proceedings of the international workshop on software fairness, pages 1-7, 2018.
[77] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nip. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
[78] William Wolberg, Olvi Mangasarian, Nick Street, and W. Street. Breast Cancer Wisconsin (Di- agnostic). UCI Machine Learning Repository, 1995. DOI: http://d...orr/o..444///////////
[79] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
[80] Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[81] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-1 11043, 2020.
[82] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[83] Yong Zheng, Tanaya Dave, Neha Mishra, and Harshit Kumar. Fairness in reciprocal recommen- dations: A speed-dating study. In Adjunct publication of the 26th conference on user modeling, adaptation and personalization, 2018.
[84] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, 2021.
[85] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lisiouelanguage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[86] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022.
14

---

A Broader Societal Impact Statement
The goal of our work is to investigate context optimization strategies for PFNs, such as prompt-tuning and sketching, which we use to scale and improve the performance of TabPEN. We do not see any negative broader societal impacts of our work that do not already exist in other classification methods. In fact, work may further facilitate the adoption of TabPFN, which has the benefit of being pretrained and therefore having a lower carbon footprint compared to most deep learning approaches that must be trained from scratch.
Furthermore, we demonstrate that our prompt-tuning strategy makes it possible to mitigate the bias of TabPFN while only fine-tuning a small set of embedding vectors; the authors of TabPFN mentioned that it is critical to study and improve TabPFN under the lens of algorithmic fairness and other dimensions of trustworthy AI [35]. This may allow practitioners to use TabPFN in sensitive settings for the first time, in which the original TabPFN would lead to biased preditions. Overall, our hope is that work will have positive impact for both practitioners and researchers, by facilitating the adoption of a model with a low carbon footprint, and by providing the tools to mitigate the bias of PFNs. Likewise, we hope that the possibility summarize datasets with prompt tuning will add to the toolbox of machine learning practitioners aiming to analyze and interpret their data better, and therefore may have positive impact the trustworthieness of machine learning.
B Additional Related Work
Tabular classification. Tabular datasets are the oldest and among the most widely used dataset types in machine learning [10, 67]. GBDTs [29] build an ensemble of decision trees, with each tree fitting the residual of the loss from the previous tree. XGBoost [16] and CatBoost [61] are two of the most widely-used and highest-performet based on neural nets [31, 38, 42].
GBDTs. Researchers have also explored many methods
There is an active debate in the community on which family of methods perform best on tabular data: neural nets [6, 42, 47, 60, 65] or GBDTs [10, 31, 32, 67], with the exception of small datasets, on which TabPFN performs the best [35, 51]. Finally, sketching and feature selection methods have been extensively studied in prior works [13, 56].
Neural Processes and Prior-Data Fitted Networks. Prior-data Fitted Networks (PFNs) [35, 55] are a recently-proposed paradigm for machine learning, which show that fast approximate Bayesian inference is possible by training neural network mimic the posterior predictive distribution (PPD) in a single forward pass using in-context learning [54, 55, 58). PFNs were shown to yield state-of-the-art empirical performance on small tabular datasets [35, 51]. PFNs have been used in other applications, including Bayesian optimization [54], forecasting [24], and learning curve extrapolation [1].
PFNs are neural processes (NPs) [55]. Recent advances in NP are similar in nature to our work, especially recent works that aim to scale ceanntionbbsed methods to larger contexts. Feng al. [27] propose Latent Bottlenecked Attentive Neural Processes (LBANPs), a transformer-basee neural process which overcomes the quadratic complexity of transformers by encoding the context dataset into a constant number of latent vectors. Guo et al. [34] propose Versatile Neural Processes ( (VNP), which increases the capability of NPs to handle compex signals by using new bottleneck encoder. [63] introduces semi- -armmiereeetric inducing point networks (SPIN), which can attend to a training set at inference time with linear complexity via inducing point methods. [28] introduce Constant Memory Attention Block (CMAB), an attention block that is permutation-invariant and has constant memory complexity when computing the output, well Constant Memory Attentive Neural Processes (CMANPs), NP that requires constant memory.
For additional related work. see Appendix B.
Prompt tuning. Researchers have extensively studied prompting techniques for large language models (LLMs) [49]. In such methods, one modifies the input into a prompt, and the LLM predicts the subsequent tokens, from which it derives the output predictions. Prompt tuning techniques typically start with a pretrained L.M and use the training data of downstream task to find the best prompt for this task. 'Hard' prompt tuning involves finding the best natural language prompts using discrete
15

---

search techniques [40, 77], while 'soft' prompt tuning optimizes a prompt directly in the embedding space of the model [46, 84). Soft prompt tuning has also been applied to models such as vision-language models [70, 85, 86].
C Additional experimental details
Hyperparameter settings for baselines. For the baselines, we the same hyperparameter search space as in prior work [51] (which is itself similar to other works [42]). We choose the best configuration among the default parameter sets and the additional 29 sets, using k-fold cross- validation with 10 folds. We follow the best configurations of [51] with exceptions; for CatBoost and XGBoost, increase the range of tree depth by factor of 10, since high depth often more appropriate for very large datasets. We find that this improves performance on large datasets, but sometimes harms performance small datasets, and increases the runtime- see Appendix G for the raw results. For each dataset, each algorithm is allowed a maximum runtime of 10 hours for the entire search process. For ExcelFormer, we decrease the number of dimensions and depth of the default transformer, as we find that without this modification, it times out on even medium-sized datasets.
Hyperparameter settings for TuneTables. Consistent with the experimental design in [51], we perform hyperparameter optimization of TuneTables for up to 30 variants; however, utilize grid search rather than Optuna. We use a batch size of 1 and no gradient aggregation. The space over which conduct grid search in Tune'Tables is conditioned on metadata; number of features, number of samples, number of classes. The feature and class splits are set by the limits of the particular TabPFN checkpoint we optimize. When we reach leaf node, TaneTablle--tandar and Tune Tables-mediun conduct a grid search over a fixed range of configurations. The size of our search space is always < 30, and usually 10. For feature-large datasets, YuneeTables-ligh conducts additional optimization in the feature space prior to grid search. Further details on our search a algorithm can be found in the repository associated with the paper. TuneTTables hyperparameter settings can be found in chart form in Table 5.
C.1 Memory Efficiency
In our experiments, we find that we exceed hardware limitations around 3000 samples with a batch size of 1 when using TabPFN GPU with 48GB of VRAM. While FlashAttention, a popular optimization, is memory-linear in sequence length, but it has not yet been implemented in the public release of TabPFN, which relies into a future release, in which case any benefits will carry over to our method. Even so, inference time continues to be quadratic, and there are other overheads on GPU memory. Overall, there remains a considerable need for new algorithmic methods which can be scaled up to the sequence lengths required by large tabular datasets.
older version of Py PoTch.. Flash attention may be integrated
C.2 Feature selection and sketching
We ablate strategies for both sketching (subsampling) and feature selection for scaling TabPFN. We consider three sketching methods (random, k-means, and CoreSet) and three feature selection methods (random, PCA, mutual information) described in Section 2. In addition to sketching, consider two different methods for sampling class labels: equal and proportional.
For efficient coreset selection, we use variant of Farthest Point Sampling [26]; after selecting an initial set of n=5 random points, we compute the distance of each point in the dataset to the set of already selected points. Next, add to the set of selected points the point whose distance to the selected points is maximal. Finally, we update the distances of all the points according to the updated selected set; and continue iteratively.
We limit our algorithmic comparison TabPFN CatBoost, which is the overall best-performing model in [51]. We compare all combinations of sketching and feature selection with both CatBoost and TabPFN; see Table Interestingly, we find that random sketching performs just as well as the more involved algorithms, k-means and CoreSet. On the other hand, PCA significantly outperforms mutual information and random feature selection methods, when the original number of features large.
16

---

Mean Rank
3
rtdl_ MLP
TuneTables CatBoost
ResNet
XGBoost
Figure 5: Tune'Table is competitive with state-of-the-art tabular models. A critical difference plot according to mean accuracy rank across all LARGESCALFTABLES datasets with fewer than 50 samples. Algorithms which are not significantly different (p > 0.05) are connected with horizontal black bar. TuneTables achieves the highest mean rank any algorithm. This plot is similar to Figure 2, but the search spaces for XGBoost and CatBoost are expanded to include more trees.
Mean Rank
rtdl MLP
XGBoost CatBoost
ResNet
TuneTables
Figure 6: We present critical difference plot according to mean accuracy rank on LARGESCALETA: BLES. Algorithms which are not significantly different (p 0.05) are connected with horizontal black bar. Across these datasets, TabeTables achieves performance that is not significantly different from CatBoost or XGBoost.
C.3 Additional results
First, we present the large class table from Section 4. See Table 7.
In Table 1, gave aggregate statistics for TaneTa and baselines across all datasets of size less than 50 000. Now, in Table 8, we present results for TuneTTables, CatBoost, and XGBoost on all datasets individually, including datasets of size nearly two million. Note that report the end-to-end runtime of all methods, including 30 iterations of search (for example, TabPFNs3000 takes about 30x more time than TabPFN reported in prior work [51]). In Figure 6, we present critical difference diagram similar to Figure 2 but over all 29 datasets. Across these 29 datasets, TuncTables achieves performance that is not statistically different from CatBoost XGBoost.
Neural net comparison. In Table 1 and Table 8, we compared TuneTables to GBDTs and two other neural nets. Now, we compare Tune'Tables to two additional neural nets: MLP and TabTransformer [38] (for four total neural net baselines). Since transformer-based methods have higher memory consumption, we obtained full results on 17 total datasets. See Table We find that TuneTables substantially outperforms all other neural nets on average across all datasets, achieving the lowest average rank at 1.93, and achieving the highest average Z-Score of 0.85, far above the second-place neural net's value of 0.18.
Inference Time
We give the full details for inference time; see Table 10.
Full results for TabeTTaaee--mdiine and Tablesibletl Recall the end of Section 4, introduced TuneTables mediiu and TuneTables-ligh showing that they can substantially decrease the runtime the standard TuneTables with just small reduction in accuracy. While we presented summary results in Table 2, now we present the full results in Table 1.. For descriptions of Tune variants, see Section
C.4 Ablation study
We report our results ablating the core components of our methods: the prompt length, its use during training and inference, the use of ensembles, and the prompt tuning procedure itself (against regular finetuning). Beginning with the prompt length, we see in Table 12 that while some dataset results are not sensitive to the prompt length, others vary significantly, and generally enjoy longer prompts. Ablating the importance of having real-data context during inference (same table) find that is
17

---

important for smaller datasets. On the larger datasets, the variant with no real data at all is better in specific cases. However, having no such data during inference, model would perform better not having it during training as well. Introducing real data first during inference is usually harmless, but significantly deteriorates the results on specific cases. Fine tuning the entire model is not only significantly more memory-intensive but also underperforms Tune'Tables in terms of accuracy.
In Table 13 we show the results of our method with or without ensembles, and with or without context (in training and inference). Our ensembling strategy is beneficial in both cases. When using ensembles, the results with or without context are generally similar, although one variant may outperform the other on specific datasets.
D TabeTables additional details
Tuned prompt implementation details. We implement our tuned prompts by prepending ran- domly initialized vectors of identical dimension the TabPFN encoder to the real data points fed to TabPFN as context during training.
In the CT ('context in training') setting, continue to provide real data points part of the training context; the quantity of real data provided for each batch is drawn with uniform probability from a random variable which ranges from zero fixed upper bound, which is passed the model as hyperparameter. Usually that upper bound is 1152, the default setting for TabPFN, unless the dataset is extremely small, in which
we select it according to the limitations of that dataset.
In the NCT (" ('no context in training") setting, no real data is provided as part of the training context; all data in the training set is used to fit the randomly initialized prompt. There are always fixed number of data points provided, and it is the same for every batch. Usually that number 128, unless the dataset is extremely small, in which case we select it according to the limitations of that dataset.
We note that it is also possible evaluate models trained in either setting either with context (C) or without (NC). In our experiments, we always evaluate all models on both, and report the setting with higher validation accuracy test time.
We find that on smaller datasets, the setting with context (C) sometimes outperforms the setting without context (NC), even after the prompt has been fitted, perhaps because the tuned prompt overfits on a small amount of available training data. However, when using ensembling, for datasets with more than 3000 samples in the training set, the NC setting is as good better than the C setting. See Table 13. We leave further investigation of this phenomenon to future work.
Loss function. Most TuneTables experiments optimized using the cross-entropy loss between labels assigned to the training data (P) and TuneTables's outputs (Q):
H(P.Q) Σ P(x) log (zx)
(5)
To reduce the effects of overfitting on datasets with fewer samples, we extend our grid search to include the choice of loss function. We find that many such datasets benefit from fitting the tuned prompt via the KL divergence loss between the PFN's outputs (P) and Tabela (Q).
((x) Q(x)
KL(PIIQ) Σ
P(z) log
(6)
Compute. We conduct all our prompt tuning experiments on Google Cloud Platform (GCP), using a single NVIDIA L4 TPU with 24GB VRAM for each experiment.
Size of tuned prompts. We ablate the size of tuned prompts in Table 12 and find that the larger tuned prompts generally perform better on datasets with more samples, while smaller tuned prompts excel on datasets with few samples. For this reason, the experimental results Section are reported on prompts of size 10 1000, conditioned on the number of samples in the input dataset.
18

---

Duration of training. Our experiments run for up to 100 epochs. We employ early stopping if our key metric (Accuracy) fails improve after fixed number of epochs.
Evaluation of tuned prompts. We validate our tuned prompts every epoch on a subset of the entire validation set the validation set is large. At fixed interval, we run the entire validation and test sets, After the experiment concludes, report results from the epoch with the best key metric score.
Fine-tuned setting. In the fine-tuned setting, our parameters are the same, except that we use lower fixed learning rate of le - 3.
Ensembling over tuned prompts. While individual tuned prompts are already substantial improvement on TabPFN for sample-large datasets, we find that these improvements sometimes compound when ensemble over multiple tuned prompts.
We draw the inspiration for our ensembling approach from the internal ensembling used in TabPFN, which averages predictions over permutations of feature indices and label indices[35]. For the results presented in Section 4, we ensemble over ten permutations of each dataset, averaging the top two ensemble member predictions (as measured by NC accuracy the validation set in the NCT setting, or C accuracy in the CT setting).
In TuneTablee ensemble, each ensemble member fits its own tuned prompt to the data. Variance in the ensemble members introduced by differences in the random initialization of the tuned prompt. as well as permuting the order of features and labels, a la TabPFN, one time before each tuned prompt fitted.
See Table 13 for ablation studies of the effectiveness of ensembling.
E Regression experiments
In this section, show that using prompt tuning, also able to extend PFNs perform well on regression datasets, further highlighting the flcxibility of our approach.
E.1 Adapting Tune Tables to regression problems.
Adapting Tabela to regression problems introduces new challenges, as TabPFN was not designed for use with such problems. In particular, we find that end-to-end fine-tuning outperforms prompt tuning in this setting, and that the most effective grid search for regression problems utilizes a space of PFN base models. Put another way, Tunc TanneTalle-reeee regessronrisiionn does not prompt tuming. and does not always use TabPFN as its foundation model. Rather, it searches a space of foundation models to find the best performer for particular regression problem. We search over three foundation models; TabPEN, the checkpoint released in [55] and new PFN we train from scratch for 10 epochs on a synthetic prior of regression-specife datasets.
E.2 Details on the datasets used for the experiments and baseline experimental design
Similar to [51], each algorithm is tuned for each dataset by maximizing the R R-squared (R2) metric. Each dataset corresponds to an OpenML task, and can be preprocessed exactly like the classifi- cation datasets used in other experiments. The 15 datasets used in these experiments are "Bank- Note-Auttenticationn UCI" (OpenML task 361002), "EgyptianSkulls" (5040), "Wine" (190420), "Wisconsin-breast cancer-cytology-feee fecturres" (361003), "bodyfat" (5514), "california" (361089), "chscase-food (5012), "cleveland" (2285), "colleges" " (359942), "cpu-small" (4883), "liver-disorders" (52948), "meta" (4729), "mv" (4774), "pbe" (4850), and "veteran" (4828). Unlike prior work, report non-none normaazade averages, as we believe it gives a more realistic indication of real-world performance glance.
We use 12 deep learning and boosted tree algorithms as baselines; VIME, TabTransformer, TabNet, DANet, STG, MLP, LightGBM, CatBoost, XGBoost, NODE, DeepFM.
19

---

Mean Rank
9
8
6
3
2
VIME STC TabNet DeepF MLP
LightGBM TuneTables
CatBoost
XGBoost
Figure 7: Tables is competitive with state-of-the-an tabular models on regression.
Input Data
Tuned Points
Decision Boundary
TabPFN Zero-Shot Decision Boundary
Figure 8: Decision boundaries 2D datasets. Shown prompt-tuned TabPFN decision bound- aries as well as regular zero-shot TabPFN decision boundaries. Larger dots middle row represent the tuned points for the two classes.
F.3 Results regression experiments
In Table 14 results and Figure 7, we show the results of our experiments. Overall, we see that TaneTables competitive with top-performing algorithms across a variety metries; mean win rank, average R² score, etc. Equally importantly, TuneTables is significantly stronger on several datasets than both boosted trees and deep baselines.
Summarization details and decision boundaries of prompt-tuned TabPFN on toy 2D Problems
For the experiments with prompts of only two examples, we prompt-tune one example per class for 500 epochs. The prompt is input directly into TabPFN without undergoing any preprocessing steps that the original data went through. Therefore, the tuned feature values do not correspond directly to the original values in the dataset. For the breast cancer dataset, it reaches 100% accuracy; for the diabetes dataset, it reaches 78.7% accuracy.
We also show decision boundaries of TabPEN and prompt-tuned TabPFN toy 2d classification problems using scikit-learn [59] in Fig. 8. The prompt contains four datapoints for each class. One see how the points tuned to recover the decision boundaries. Due the very low prompt length and the low dataset size, the prompt-tuned decision boundaries are slightly less accurate than TabPFN (consistent with results for small datasets in Table 12).
G Results for Deeper GBDTs
In the results in Section 4, we compare TuneTables to the GBDTs, XGBoost and CatBoost, using search space which the number of trees can range from to 1000. Now, we compare TuneTTbles to XGBoost and CatBoost when they have expanded search spaces, where the number of trees can
20

---

Diabetes Test
Feature Value
Negative Positive
0.5
#Pregnant
Plasma Glucose
Blood Pressure
Skin Thickness
Insulin
Body Mass Indes
gree
Age
Diabetes Ped
Figure 9: Diabetes dataset [68] with high accuracies from just two samples.
range from to 10000. We re-compute the critical difference plot and table results: see Figure 5, Table 15, and Table 16.
In Figure and Table 15, we see that Tane'Tbles still performs favorably compared to CatBoost and XGBoost on datasets of size up 50 000. Note that the larger search space may perform better larger datasets, it is also more challenging to search through 30 iterations. We present the full results on all datasets in Table 16.
21

---

Table List of all datasets used in Section and Appendix C. LARGESCALETABLES datasets are listed in part one. Datasets in part two were used for the class extension experiments in Table 7. Datasets in part three were used for fairness experiments part four was used for intepretability experiments in Section 5. Datasets in part 5 were used for the ablations on sketching and feature selection in Appendix C. For the list of 98 used in Table 8, see wttp://ainnatttttssssseeeee cev.
Section The additional dataset in
Sttpss//ee / / wwtppppaaaeeeeeeeteaaaaeeeeeeeee
Datasel
num. classes
num. Features
nam. sumples
prediction smnal
poker-hand
10
10
1025009
agrawall
BNG (labor)
2
16
1000000
airlines
2
539383 78
albert
2
425240
(votc)
2
131072
connect-4
2
28
98050
higgs
67557
volkert
180
58310
riccardo
2
4296
20000
elevators
2
18
16599
eeg-eyerstate
2
14980
har
6
561
10299
dilbert
5
2000
10000
robert
7200
10000
christine
2
1636
5418
hioresponse
2
1776
3151
kc-1
2
2109
1728
cme
3
1473
blood-transfusio
2
bulance-scale
3
625
climate
2
cylinder-bumn
2
37
540
dresses
colic
2
22
368
heart-e
2
13
303
breast cancer
openml
Idpa 9974
164860
openml
walking-activiity 9945
149332
openml
Devnagari-Seript...
1024
92000
openml
helena 168329
100
27
65196
openml
chess 3952
28056
openml
kropt 2076 letter
18
28056
openm]
20000
openml
isolet__3881
26
617
7797
openml texture
5500
openml oon-buundee--llatt-ttaaae
9956
100
64
1399
openml vowel 3022
openml soyhean 41
19
35
683
openml
3567
openml arrhythmia_5
13
279
452
openml pwwmary--umorr............ 146032
21
adult-census
2
15
32561
SpeedDating
2
121
8378
compas-two-years
2
4966
nisy (aational-oongiiddiin) bationa--oggiddnnlluuggghhaaa
2
17
4908
Diabetes
322
skin- segmentation
2
245057
FM (NIShT
784
70000
CIFAR-10
3072
60000
gdde cuas-diff-liffcrnt--ommeeeet
6
129
13910
pendigits
10992
infeat-factors
216
2000
mixeat-pixel
240
2000
semcion
256
1593
hill-alley
22

---

Table 5: TaneTables and TabPFNs3000 hyperparameter configurations based number of samples.
Parameter
TabPFN
Tune lables < 2000
Tune Tables >> 2000
Batch Size
Real Data Qiy
3000
[1122 0)
Ensemble Size
32
10
Tuned Prompt Dimension
10
1O00
Ensemble Method
Avg Top 2
Epochs
(7. 60)
I00
Warmup
10
10
Sequence Length Per Batch
Fixed
(Variable, Fixed)
Early Stopping
(2, 6}
6
Learning Rate
3e-2
le-3
Validation Frequency
2
2
Max Val Size During Training
2000
2000
AdamW, Detault Scttings,
AdamW, Default Settings,
Optimizer
Weight Decay 0 [Cross-Entropy.
Weight Decay 0
Loss
KL Divergence)
Tuned Prompt Labels
[Equal, Proportional)
Table Comparative performance of TabPFN and CatBoost with sketching, feature selection, and sampling methods. On a distinct subset of the datasets in [51] selected to emphasize datasets with many features or many samples, we compare CatBoost and TabPFNs3000. When both models are limited to 3000 samples, Tab0 performs better 12 of 17 datasets where significant differences exist. When CatBoost allowed access to the entire training data, the win rate identical. In most cases, random sample selection is sufficient for optimal performance. Both models benefit from PCA and mutual information dimension reduction when the feature space is large. The columns labeled SKT / FTS/ SMP list the best performing method for sketching. feature subsampling and label-aware sketching technique, respectively. Label-aware sketching refers to a strategy where either sample instances proportionate to their labels, or oversample minority classes with replacement to create a class-balanced distribution. While the choice of label-aware sketching strategy is often impactful (and
it in Tune'Tables) and the choice of feature subselection method can be important for some datasets, in all but one case, no sketching method we test outperforms random sampling. Bold indicates the best-performing model(s).
Full
Best CalBoos
Random CalBoost
Best
Random TabPFN
SKT/FTS / SMP
SKT / FTS / SMP TabPFN
CalBostt
TabPFN
CalBoust
airlines_ 189354
0.653
0.637
0.637
0.594
0.589
RND / RND/ /
RND/ RND / PR
alberi_189336
0.698
0.657
0.657
0.64
0.64
RND / RND/ PR
RND/ RND / PR
CIFAR_10 1AAAR_10_161222
0.434
0.37
0,342
0.373
0.372
RND/ PCA / PR
RND /ND/UO RND/ PR
connect-4_144 146195
0,749
0.716
0.716
0.66
0.659
RND/ / RND/ / PR
RND/ RND/ / PR
oeg-cyc-state 14951
0.832
0.808
0.806
0,932
0.932
RND/ / RND/ PR
RND/ RND/ / EQ
clevators_377 3711
0.855
0.838
0.838
0.899
RND/ / MUT /R
RND/ RND /
IMM_146825
0.843
0.787
0.787
0.835
0.812
RND RND/ PR
RND/ PCA/ PR
gdde_ 9987
0.97
0.976
0,955
0.994
0,993
RND/ / PCA / EQ
RND/ RND/ / PR
higgs_ 146606
0.71
0.684
0.684
0.665
0.661
RND/ / RND / PR
RND / RND / PR
hill-vallee_I4550
0.514
0,514
0.514
0.56
0.56
RND RND / PR
RND RND / PR
mfeat-factors_111
0.954
0.943
0.973
0.973
KMN RND EQ
RND RND/ /
mfcat-pixel_ 1446224
0.955
0,951
0,951
0.971
0.97
RND RND/ PR
RND RND/ /
pendigits_33 332
0.972
0.966
0.961
0.995
0.993
RND/ / RND / PR
RND RND / PR
poker-hand_ -890
0.664
0.572
0.561
0.519
0.515
RND/ / RND/ PR
RND / RND / PR
riccardo 168338
0,951
0.956
0.93
0.991
0.982
RND / PCA / BQQ
RND/ /ND/UM MUT EQ
Subert_168332
0.446
0.367
0.367
0,384
0.359
RND / RND/ PR
RND/ PCA/ EQ
semeion 9964
0.887
0.869
0.863
0.915
0.915
RND MUT EQ
RND/ RND/ / PR
.9.99665
0.994
0.989
0.987
0.999
0.999
RND / RND/ PR
RND/ RND/ / PR
volkert_16833)
0,608
0.56
0.56
0.557
0.555
RND/ / RND/ PR
RND RND/ PR
Average
0.773
0.746
0.740
0.761
0.757
RND / RND/ PR
RND/ RND / PR
23

---

Table 7: Comparison of algorithms datasets with large number of classes. TaneTables can effectively handle datasets with more classes than the ones used for pretraining, which was not possible with TabPFN. For each algorithm, we compute its mean test accuracy, and mean rank in terms of accuracy. We also compute the mean Z-score, computed by normalizing the set of results each dataset (by mean std. 1), so that each dataset has the same weight, and averaging each algorithm's normalized performances. We see that TuncTables performs the best across all performance-orient metrics. Fractional num. wins values averaged over three splits per dataset, and reflect the presence of multi-way way ties on certain splits.
Method
Mean Acc.
Mcan Rank
Mean Z-Score
Std. Z-Score
Med. Z-Score
Num. Wins
XGBoost
0.885
2,000
1,052
0.558
1.245
12.0
TuncTables
0.779
2.524
0.602
0.932
0.971
5.0
ResNet
0.776
0.826
2.0
KNN
0.789
3,976
0.121
0.262
0.103
0.0
CatBoost
0.781
4.524
0,960
RandomForest
0.751
5.214
-0.628
0.754
-0,767
0.0
MLP
Table 8: Comparison of the top-performing methods LARGESCALETABLE
For each algorithm, we show its test accuracy and end-to-end runtime in seconds. In this table, Table 11 and Table 16, report the summed runtime for all random seeds (30 in the case of all algorithms except for Tabela which generally requires fewer than 30). Since Tune'Talles uses TabPFN predictions on small datasets, we conservatively report its end to end runtime as the sum of TabPFNs3000 runtime and Tune grid search runtime. This prediction is also extremely conservative for TabPFNs3000, 30 random seeds unnecessary datasets with fewer than 100 features. We report runtime in this manner to mitigate concerns that we are advantaging PFNs over boosted trees. We also show the number of samples in each dataset. All datasets above the line included in Table 1.
Datasct
Size
TabPFN&3000
Tune'Tables
CatBoost
XGBoost Acc.
Acc.
Runtime
Acc.
Runtime
Ace.
Runtime
Runtime
breast-cancer
286
0.765
29
0.770
65
0.770
37
0.724
4
heart-c
303
0.848
0.903 40
66
0.903
21
0.839
ecoli
336
0.848
30
0.843
66
0.892
73
0.833
17
colic
368
0.856
39
0.892
66
0.874
218
0,883
3
dresses-sales
500
0.578
41
0,580
122
0.640
40 0.640
5
cylinder-bands
540
0.800
0.846 41
82
0.827
94
0.864
climate
540
0.959
59
0.951
97
0.963
25 0.926
4
balance-scale
625
0,990
29
0.995
0.899 55
53
0.894
145
blood-transfusion
748
0.801
25
0.782
56
0.756
24 0.760
58
emc
1473
0.554
91
0.556
109
0.561
78
0.561
13
ke-1
2109
0.862
168
0.856
187
0.856
49
0.859
bioresponse
3151
0.797
638
0.798
3012
0.788
113
0,800
102
christinc
5418
0.742
0.755
3920
0.736
331
0.743
rohert
10000
0.250
964
0.414
2397
0.464
599
0.503
2118
dilbert
10000
0.922
761
0.992
3749
0.949
809
1144
har
10299
0.936
370
0.981
2657
0.985
302
0.994
232
eeg-eye-state
14980
0.940
0.986
1929
0,946
elevators
16599
0.902
186
0.902
1297
0.891
176
0.896
234
riccardo
20000
0.922
1395
0.995
5247
0.997
692
0.998
735
volkert
58310
0.567
459
0.693
6331
0.666
230
0.703
604
higgs
67557
0.671
931
0.714
4084
0.724
80
0.725
46
connect-4
98050
0.668
696
0.817
5395
0.807
204
0.855
106
BNG (vote)
131072
0.968
1976
0.974
2493
0.975
42
0.975
16
albert
425240
0.642
2363
0.658
17518
1078
0.690
76
airlines
539383
0.600
2602
0.653
44434
0.664
155
0.673
184
BNG (labor)
1000000
0.937
5518
0.967
7717
226
179
agrawall
1000000
0.948
5158
0.950
45504
0.951
142
0.951
poker-hand
1025009
0.531
2423
1.000
10471
0.912
5331
0,940
439
piickpreeiitioo-smmall
1997410
0.833
10421
0.837
33148
0.842
85
0.843
213
Average
0.781
1320
0.830
6975
0.823
390
0.826
234
24

---

Table 9: Comparison of neural nets on LARGEŠCALETABLES We compare TaneTables to other prominent deep learning methods for tabular data the 17 datasets in LARGESCALETABLE which all algorithms reported results. For each algorithm, we compute its different metrics of accuracy and rank. We also compute the mean Z-score, computed by normalizing the set of results on each dataset (by mean 0 std. 1), so that each dataset has the same weight, and averaging each algorithm'ss normalized performances. We see that Tablls performs the best across all performance-orient metrics. Fractional num. wins values are averaged over three splits per dataset, and reflect the presence of multi-way way ties certain splits.
for
Method
Mean Acc.
Med Acc. Rank
Med. Rank
Mean Z-Score
Z-Score
Num. Wins
Tables
0.81
0.85
1.93
1.0
0.91
9.67
ndl ResNet
0.77
0.78
2.68
2.0
0.18
0.19
3.33
TabTransfommer
0.76
0.73
3.05
3.0
-0.10
0.05
3.00
ndl MLP
0.73
0.76
3.55
4.0
-0.34
-0.55
0.83
MI.P
0.72
0.74
3.79
4.0
-0.59
-0.55
0.17
Table 10: Comparison of inference times across datasets. In this table, we compare the inference time of TabPFNs3000 (which uses up 3000 points real data as context, when available) and four common configurations for TuneTables We report inference time for the entire test set, as well as per 1000 samples, estimated from the inference time for the entire test set. At inference time, Tables does not require ensembling achieve high accuracy, while TabPFNs3000 does. Therefore, TuneTables is considerably faster.
TahPENc3000
VuncTablesgtttcc
Tunn'TaMles
TuncTablessetHoo..
3000 pts. eas.
perlic
2418
perlkk
0pts
peelic
2010
porlk
0pts
porkk
Neczet-canczr beutie
0.415
0350
0,262
0.25
0.211
celi
358
1.345
2.54
01103
0.38
0.100
0.227
0.124
0.317
0.112
0.304
dhoso-ssios
12
500
1,024
2.345
01005
0.106
1100
0.15
01.05
0.212
0.02
0.144
clinne
540
1.007
1.57
01088
0.181
110
0.107
0.53
0.304
0.055
0.166
sylinde
746
0.505
0.705
01123
0.164
0.117
0.155
0.140
0.199
0.167
0.14
cere
1073
1.455
1935
11090
0.067
0.003
0003
01119
1.105
0005
0863
0.13
0.087
11
0053
01085
0.116
0055
boccspotae
1.785
dilher
3003
19030
9.365
0.99
0.34
0.014
0.304
0.00
1.136
0.113
0.312
0035
robert
10
7200
1000
9.309
0.718
0.02
0,302
0.02
1.143
0.114
0.208
0421
hat
361
1059
3.354
01913
01904
0.778
0.300
0121
1.27
0.123
0.236
0023
zzg-cyestalz
0.55
0.072
0005
0.112
0019
diccando
4286
8.904
1.309
0.111
0015
3.21
0.111
0.35%
001
wilket
10
180
38310
13.440
01299
1911
0.67
0.172
0013
6.336
0.109
0.857
0815
lighs
87357
21.991
0.506
0.440
0.055
1.356
0,02
10.404
0.154
1.577
0022
59050
0.204
0.366
0.009
albon
434360
77 946
11181
945
0.066
0014
47.844
0.13
6.17%
0015
diflnes
$38383
85.146
0.158
36,707
0.68
6:351
0013
61.477
0.114
8.123
0.15
9
1000000
103,083
0L193
70.65
0.011
12:12)
0012
112.533
0.113
14944
0.15
BNGilabee
1000000
19.50
0.500
0003
106.20 211
0015
podictoo
133.211
24.64
0002
0016
Aveiage
222079
41.555
1.109
15.213
0.134
2,590
0087
34.114
0.163
3.402
0.09
25

---

Table 11: Tane-aslss-mmdim and Janelables-light are substantially faster with only modest decrease in accuracy. We compare the average accuracy and runtime in seconds of three versions of Tune across all 19 datasets of size < 50K, as well as all 29 datasets. All datasets above the line were included in Table
Dataset
TuneTables-stanne
Tune/aables-melin Runtime
TuneTables-ligh
Acc. Runtime
Acc.
Acc. Runtime
breast-cancer
0.770
65
0.782
36
0.747
30
heart-c
0.903
66
0,892
27
0.892
23
ecoli
0.843
66
0.833
36
0.833
30
colic
0.892
66
0.892
27
0.892
23
dresses-sales
0,580
122
0.600
81
0.600
37
climate
0.951
0.963 97
38
0.963
30
cylinder-banee bands
0,846
82
0.870
41
0.870
34
balancc-scale
0.995
55
0.995
26
0.995
22
blood-transfiue
0.782
56
0.773
31
0.760
30
0.556
109
0.536
32
0.514
44
kc-1
0.856
187
0.843
68
0.842
50
bioresponse
0.798
3012
0.776
2374
0.766
653
christine
0.755
3920
0.740
3254
0.724
1366
dilbert
0.992
3749
0.990
2988
0.896
3107
robert
0.414
2397
0.420
1433
0.313
767
har
0.981
2657
0.985
2287
0.963
568
eeg-eye-state
0.986
1929
0.978
1752
0.938
213
elevators
0.900
1297
0.898
1110
0.897
211
riccardo
0.995
5247
0.995
3852
0.987
1309
volkert
0.693
6331
0.707
5872
0.562
1116
connect-4
0.817
5395
0.813
4699
0.682
471
higgs
0.714
4084
0.714
3153
0.648
719
BNG (vote)
0,974
2493
0.973
517
0.970
113
albert
0.658
17518
0.654
2056
0.642
350
airlines
0.653
44434
0.643
2230
0.601
352
BNG (labor)
0.967
7717
0.963
3942
0.956
641
agrawall
0,950
45504
0.950
2131
0.949
531
poker-hand
1,000
10471
0.998
8184
0.575
630
click-prediction-small
0.837
33148
0.835
3050
0.834
635
Average
0.830
6975
0.828
1907
0.787
486
26

---

Table 12:
Ablation on TuneTables variants. Comparison of the different variants of our method (TabPFN-PT) and the entire backbone fine-tuning (TabPFN-FT) to TuneTabbles itself on the LARGESCALETABLES benchmark. Tables is our full method described in Section 3. For each variant, we show its test accuracy. For all methods in this table except for Tune'Tables there is no hyperparameter optimization (HPO). For TuneTables, we use our standard grid search.
TapPEN-FT
TabPTN
TahPTN.PT
Tablls
Context in Training Context Inference
Varied
Varied
Tuned prompt length
N/A
1000
1000
1000
1000
100
10
Varied
Agrawall
0.946
0.948
0.950
0.950
0.946
0.950
0,949
0.950
BNG(labor)
0,961
0.965
0.965
0.966
0.965
0.960
0.967
BNG(v
0.964
0.971
0.974
0.973
0.973
0.974
0,971
0.974
Bioresponse
0.798
Click, Booo__prdictionneeee small
0.834
0,834
0.834
0.835
0.835
0.835
0.833
0.837
airlines
0.588
0.628
0.645
0.643
0.639
0.640
0.628
0.653
balance-scad scale
0.937
0.909
0.944
0.940
0.456
0.932
0.933
0.955
blood-transtusion
0.764
0.691
0.782
breast cuncer
0.730
0.743
0.713
0.741
0.661
D.724
0.718
0.770
christine
0.711
0.723
0.716
0.714
0.685
0.712
0,700
0.755
cme
0.542
0.525
0.559
0.540
0.436
0.532
0.548
0.556
colic
0.888
0,871
0,880
0.816
0.563
0.870
0.842
0.892
connect-4
0.678
0.787
0,810
0.795
0.804
0.767
0.679
0.817
cylinder-hande hands
0.842
0.791
0.759
0.870
0.556
0.820
0,856
0.846
dresses-sales
0.550
0.573
0.533
0.567
0.553
0.560
0.577
0.580
ecoli
0.814
0.765
0.775
0.823
0.314
0.833
0.823
0.843
eeg. eeg-eye-state
0.651
elevators
0.894
0.895
0.899
0.896
0.854
0.896
0.892
0.902
har
0.940
0,960
0.982
0.971
0.946
heam-c
0.863
0,839
0.860
0.806
0.718
0.838
0,855
0.903
biges
0.638
0,200
0.703
0.696
0,700
0.698
0.675
0.714
kel
poker-hand
0.601
0.479
0.993
0.982
0.985
0.690
0,572
1.000
riccando
0.918
0.990
0.993
0.990
0.990
0.936
0.995
robert
0,303
0,394
0.404
0.386
0.369
0.311
0.414
volkert
0.487
0.621
0.647
0.631
0.649
0.605
0,540
0.693
Average
0.774
0.797
0.794
0.689
0.830
27

---

Table 13: Ensembled models outperform models trained single tumed prompt; with ensembling, the training and testing without real-data context (NC) setting matches or exceeds the setting with context (C). Runs with prompt tuned only once are noted TabPFN-PT, and ensembles of such runs as TabPFN-PT- PT--ns. Although the improvements are often quite small, we find that ensembles generally outperform single tuned prompts in both C and NC settings. We also find that ensembles trained without additional real data context at train time or test time are as good or better than ensembles trained and tested with real data context on datasets larger than 3000 samples.
Datasct
TabPFN-PT-C TabPFN-PT-Ens-C TabPFN-PT-NC TabPFN-PT-Ens-NC
agrawall
0.949
0.95
0.95
0.949
airlines
0.645
0.649
0,645
0.646
albert
0,657
0.66
0.648
0.66
balance-scale
0,921
0,968
0.984
0,952
bioresponse
0.763
0,795
0.776
0.776
blood-transfusion
0,813
0.84
0.827
0.747
BNG (labor)
0.965
0,966
0.965
0.967
BNG (vote)
0,974
0,976
0.975
0.977
breast-cancer
0,793
0.793
0,759
0.69
car
0.96
0.971
0.977
0.965
christine
0.76
0.738
0.734
0.756
click- socckprediction- small
0.834
0.836
0.834
0.836
climate
0,926
0,944
0.963
0,981
eme
0,534
0.581
0.547
0,446
colic
0.811
0.892
0.865
0,865
connect-4
0.796
0.814
0.808
0.812
cylinder-bands
0.815
0,815
0.926
0.778
dilbert
0.87
0.951
0.948
0.968
dresses-sales
0.6
0.68
0.66
0.6
ceg-eye-state
0.74
0.977
0.666
0,983
elevators
0,903
0.908
0,902
0.902
har
0,94
0.984
0.976
0.987
heart-c
0,871
0.903
0.871
0.871
higgs
0.695
0.712
0,709
0.709
kc-1
0.867
0.872
0,867
0.872
poker-hand
0.992
1
riccardo
0
0.996
0.994
0.996
robert
0.391
0.415
0.421
0.444
volkert
0.633
0.665
0.658
0.672
Average
0.807
0.836
0.822
0.821
Table 14: TuneTTables is competitive with strong baselines range of regression datasets. In this table, we compare 7 algorithms on 15regression datasets from [51]. The terminology and methods of this table follow Table 1, except that report mean R score instead of mean accuracy as our primary statistic. TuneTables ties with XGBoost for the number of wins, has the highest average R² score of any algorithm, and the second-highest mean rank after LightGBM.
Model
Mean R2
Mean Rank
Mean Z-Scores
Std Z-Scores
Med Z-Scores
Number of Wins
Tablea
0.603
3,533
0.792
0.448
0.768
LightGBM
0.536
3.333
0.640
0.472
0.660
CatBoost
0.508
3.800
0.431
0,933
0,576
2.0
XGBoost
0,386
4.000
0.116
1.009
0.405
3,0
MLP
0.359
5.267
-0,009
0.673
0.109
1.0
STG
0.332
6.067
-0.369
0.864
-0.169
2.0
TabNet
0.239
5.867
-0.107
0.747
0.099
0.0
VIME
-0.007
7.733
-1.072
1.054
-1.159
0.0
DeepFM
-7.954
5.400
-0.422
1.248
0.027
2.0
28

---

Table 13: Tables matches SOTA algorithms on small and medium-sized datasets. this table, we compare algorithms over 19 datasets in LARGESCALETABLES with at most 50000 samples. For each algorithm, we compute its mean accuracy, mean runtime, and mean rank terms of accuracy. We also compute the mean Z-score, computed by normalizing the set of results each dataset (by mean std. 1), so that each dataset has the same weight, and averaging each algorithm's normalized performances. Std. Z-Score is computed with respect to random splits and averaged across datasets. Fractional num. wins values are averaged over three splits per dataset, and reflect the presence of multi-way ties certain splits. This table is similar Table 1, but the benchmark is LARGESCALETABLE and the search spaces for XGBoost and CatBoost are expanded to include
more trees.
Method
Mcan Acc.
Mcan Rank
Mean Z-Score
Std. Z-Score
Med. Z-Score
Num. Wins
Runtime
TaneTables
0.831
2.263
0.471
0.536
0.614
9.833
1325
CutBoost
0.827
2.579
0.320
0,644
0.558
4.500
915
XGBoost
0.825
2.684
0.267
0.625
0.720
3,500
304
rtdl ResNet
0.790
3.561
-0.341
0.461
-0.408
0.333
500
rtdl MLP
0.755
3.912
-0.717
0.678
-0.897
0.833
374
Table 16: Comparison of top-performing methods on the LARGESCALETAABLS benchmark. For each algorithm, show its test accuracy and runtime (seconds). We also show the number of samples each dataset. All datasets above the line were included Table 15. This table similar to Table 8, but the search spaces for XGBoost and CatBoost expanded to include more trees.
Dataset
Size
TabPEN
TuneTTables
CatBoost
XGBoost
Acc.
Runtime
Acc.
Runtime
Acc.
Runtime
Acc.
Runtime
breast-cancer
286
0.765
29
0.770
65
0.736
190
0.724
23
heart-c
303
0.848
0,903 40
66
0.839
164
0.849
27
ecoli
336
0.848
30
0.843
66
0.892
394
0.843
72
colic
368
0.856
39
0.892
66
0.874
458
0.883
27
dresses-sales
500
0.578
41
0.580
122
0.613
292
0.660
30
cylinder-bands
540
0.800
41
0.846
82
0.827
552
0.846
35
climate
540
0.959
0.951 59
97
0.951
282
0.938
29
balance-scale
625
0.990
29
0.995
55
0.931
241
0.894
112
-ooo-ttrnssuuiinn
748
0.801
25
0.782
56
0.756
92
0.760
27
cme
1473
0.554
91
0.556
109
0.547
224
0.552
58
ke-1
2109
0.862
0.856
187
0.855
138
0.853
40
bioresponse
3151
0.797
0.798
3012
618
131
christine
5418
0.742
0.755
3920
0.735
234
0.742
158
robert
10000
0.250
0.414
2397
0.534
3294
dilbert
10000
0.922
761
0.992
3749
779
har
10299
0.936
2657
0.993
3004
0.995
251
eeg-eye-state
14980
0.940
0.986
1929
0.957
2594
0.955
116
elevators
16599
0.902
186
0.902
1297
0.894
1316
0.896
103
riccardo
20000
0.922
1395
0.995
5247
0.996
473
0.996
468
volkert
58310
0.567
459
0.693
6331
0.715
2364
0.711
1550
higgs
67557
0.671
931
0.714
4084
0.729
222
0.726
128
connect-4
98050
0.668
696
0.817
5395
0.864
3492
0.860
593
BNG (vote)
131072
0.968
1976
0.974
2493
0.975
42
0.975
76
albert
425240
0.642
2363
0.658
17518
0.709
4001
0.694
529
airlines
539383
0.600
2602
0.653
44434
0.671
2023
0.673
305
BNG (labor)
1000000
0.937
5518
0,967
7717
0.971
226
0.970
184
agrawal1
1000000
0.948
5158
0,950
45504
0.951
142
0.951
98
poker-hand
1025009
0.531
2423
1.000
10471
1.000
6039
0.997
3469
dlick-preiiction-small
1997410
0.833
10421
0.837
33148
0.842
84
0.843
102
Average
0.781
1321
0.830
6975
0.832
1242
0.831
703
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction both state the main claims in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section and Appendix C lay out all information needed to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

reproducible, they Answer: [Yes]
Justification: We do release all code and materials needed to reproduce our results. We provided this link in the introduction: attps://anonymonn 4open. science/r/ TuneTables.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
public/guided SoaiddeeCooeeSumisssiiio.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwtpcccppppcccccccccccccccccccct htt//guisssss /CodeSubmemmm wwwminissooooooooo ssinnnolicyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section and Appendix C specify all experimental settings and details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our results in Section 4 do include standard deviations, and we also include critical difference plots in order to compute the statistical significance of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4 does provide information on the compute resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrp.p..pe ctt.iiss....puppe wttppppcccppppppppppppppppppcctt csGuieeleeee
Answer: [Yes]
Justification:
The paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As discussed in Appendix A, our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit the original owners of the resources we use in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release code which is documented with readme and comments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "eCQNM9IEJG", "reviewer": "Reviewer_YTTX", "summary": "The paper introduces a novel parameter-efficient fine-tuning strategy for prior-data fitted networks (PFNs) called TuneTables. PFNs, similar to large language models, utilize pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. However, existing PFNs like TabPFN are limited to small datasets (less than 1000 samples). TuneTables addresses these limitations by compressing large datasets into a smaller learned context, thus significantly improving the performance of PFNs. Besides, this paper gave an demonstration of TuneTables as an interpretability tool and for mitigating biases by optimizing a fairness objective.", "questions": "see above", "limitations": "The authors clearly summarized their limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- TuneTables enhances the performance of PFNs on large datasets, surpassing other state-of-the-art models like CatBoost.\n\n- Moreover, this paper provides comprehensive experiments and detailed analyses.\n\n- The presentation is good, and the methodology is sound.", "weaknesses": "- Since the dataset was collected by the authors themselves, there are some aspects I am unsure about. In previous benchmarks, FT-Transformer generally outperforms SAINT, but on the authors' benchmark, it does not. Given that tabular data is diverse, it is possible to collect datasets that are more friendly to sample-interaction-based architectures like SAINT, PFN, and TableTunes.\n\n- Excelformer, a well-known method that performs well on both small-scale and large-scale datasets, was not compared. I also suggest that the authors provide tests on Excelformer's large-scale datasets.\n\n- The authors mentioned, \"For all algorithms other than TuneTables, we perform light hyperparameter tuning by running one default setting and 29 iterations of random search using Optuna.\" This is clearly inconsistent with the original settings of most methods. I have concerns about the results.\n\n- This method improves upon TabPFN. However, TabPFN can only be applied to classification tasks, neglecting the regression tasks. Additionally, as stated in Lines 82-88, there are issues like a fixed number of features and a fixed number of classes. These problems persist, even though the authors employed techniques like sketching and feature selection (not to mention that sketching and feature selection can also be applied to other methods, as done in Kaggle competitions, potentially improving their performances). All these factors undermine the significance and value of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We appreciate that you found our paper to be presented well, and our methodology sound. We address each of your questions below:\n\n**W1: Chosen benchmarks**\n\nWe thank the reviewer for raising this clarifying point about our benchmarks. Our main results in Tab. 1 are reported on a standard benchmark from the literature, [tabzilla](https://arxiv.org/abs/2305.02997). In the tabzilla benchmark, SAINT outperforms FT-Transformer; this is also the case in the largest-scale comparison in the well-known [Grinsztajn](https://arxiv.org/pdf/2207.08815) benchmark (see Fig. 1), where SAINT outperforms FT-Transformer outright on regression tasks and on classification tasks with few iterations of random search (there is no statistically significant difference after up to 100 iterations of random search on classification tasks). The benchmark introduced in our paper, TuneTables-Hard, was curated in part to illustrate TabPFN’s scaling limitations – for further discussion on this point, please see our response to FY4B, Q3.\n\n**W2: ExcelFormer**\n\nWe thank you for raising this point about ExcelFormer, a recently published method. We have implemented this method for our rebuttal, and completed experiments on around half of the 98 datasets from the tabzilla benchmark suite; the initial results are intriguing, and we hope to extend them after the rebuttal period. Please see our global response -- tables B1 and B2; we hope this extension of results addresses your concern.\n\n**W3: Hyperparameter tuning**\n\nExperimental setup for hyperparameter tuning. This is a good question; thanks for allowing us to clarify! All of the algorithms come with their default set of hyperparameters used in the official implementation, and we used all of these settings. It is common in tabular papers to further improve performance for each dataset by conducting hyperparameter tuning (via k-fold cross validation) on a per-dataset basis. To be consistent with prior work such as Tabzilla, TabPFN, and [Grinsztajn et al](https://arxiv.org/pdf/2207.08815), we chose the best configuration among the default parameter sets and the additional 29 sets. We will clarify this in the paper, and we are happy to answer any further follow-up questions about our experimental setting.\n\n**W4: Regression tasks**\n\nWe thank you for raising this point about regression tasks. TabPFN does provide a [server](https://github.com/automl/tabpfn-client) to run regression tasks, but to the best of our knowledge, no regression results have yet been made public by the authors. We have completed extensive preliminary regression experiments on TuneTables for this rebuttal; please see our global response, Table D. We hope that this significant extension of results helps to address your concern.\n\nThank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "skTc0KXlvy", "reviewer": "Reviewer_FY4B", "summary": "This paper aims at removing 3 issues with the recently introduced TabPFN model (a transformer pretrained on synthetic datasets, showing great in-context performance on actual small tabular datasets), namely that it can only take as input few samples, few features, and few classes. The authors manage to solve these issues by a) for classes, training a new decoder with the rest of TabPFN frozen b) for features, searching over a few simple feature reduction methods (mutal info, PCA..) c) for samples, the authors use prompt tuning: some embeddings are trained to improve TabPFN performance while going through a dataset which can be bigger than TabPFN acceptable context length. Combining all these improvement creates a new method named TuneTables.\n\nExperiments on both a standard tabular benchmark and a new benchmark shows that TuneTables beats all their strong baselines for tabular data prediction, even on datasets with many samples, features or classes. While TuneTables is slower than Gradient Boosting Tree algorithms, the authors provides variants with different time-performance tradeoff, and suggest ways to speed up the model's inference time.\n\nFinally, the authors shows that TuneTables can also be used to do things which were not possible to do with TabPFN, for instance optimizing a different objective for fairness, or having access to \"summary points\" for a specific dataset.", "questions": "> While PFN accuracy can improve with more real-data samples at inference time [33, 55], the memory requirements scale quadratically with the context length\n\nIsn't it linear with FlashAttention (which is used by TabPFN if I remember correctly).\n\nWhich hyperparameter spaces are you using for each model?\n\nYou say several times that you do a grid search on TuneTables hyperparameters, using 30 steps. But it seems that the hyperparameters space is bigger than 30 possibilities. Are you doing a random search?\n\nHow were the datasets in tunestable-hard chosen? (also I wonder whether the name may be a bit misleading because it first got the impression that it contained only dataset with a lot of samples or features)\n\nIn Table 11, are the results compared for a given number of HPO steps?\n\nI'm a bit confused about how ensembling is working for TuneTables, with some sentences giving me the impression that you're taking a tuned prompt and permuting the order / labels (à la TabPFN), and other sentences giving me the impression that you're actually training several tuned prompts (\"a step constituting either a new tuned prompt fitted from scratch\n686 or a new member of a tuned prompt ensemble\")\n\nhow are coresets computed?", "limitations": "The authors have adequately addressed limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The contributions of the paper are important and novel: this can allow TabPFN to be used in a much bigger set of contexts.\n\nThe evaluation of the new TuneTables method is well-done: the method is compared to strong baselines (CatBoost, XGBoost, recent deep learning models, but also interesting variants like finetuning TabPFN) on a standard benchmark, with several aggregation metrics. Furthermore, the results of the method are impressive.\n\nThe paper is quite rich, with a lot of results and ablations.\n\nThe codebase seems quite easy to install and use (though I haven't actually run anything).\n\nI like that the paper starts off by considering simple solutions in Section 2.1 (some of which they keep) before moving to more complex solutions.\n\nThe paper is well written.\n\nThe authors are forthcoming when talking about the weaknesses of their method, for instance the slower runtime of their method compared to GBDTs.", "weaknesses": "Important details are missing, for instance the hyperparameter spaces used for the baselines and for TuneTables. This is an important part of the evaluation (which is not easy to find in the codebase), and I'm interested in how the space depends on the dataset.\n\nSome information is hard to find. I would recommend pointing directly to the relevant results (instead of a broad section like section 4). I would also recommend a table of content for the appendix. For instance I was quite interested in understanding \"In Section 4, we find that all sketching and feature selection methods plateau in performance well before approaching parity with GBDTs, which motivates our investigation of new scaling techniques.\". This lead me to follow: section 2.1 --> section 4 --> find paragraph on sketching --> appendix C1 --> table 5 --> not sure how to interpret.\n\nRelatedly, all ablations should be reported using aggregated scores, in addition to individual results. This would also make it easier to go through the many (which is good!) ablations.\n\nI also wonder why most ablations and variants are done on the new benchmark TuneTables-hard instead of the more standard TabZilla benchmark. I think it makes sense in some cases, but for instance for TuneTables-medium and light I would be interested in seeing the performance on the TabZilla benchmark.\n\nFor these two variants, I would also be interested in seeing a complete performance and runtime comparison with baselines like GBDTs in addition to what you say in the main text. This would help the reader to better understand the time-performance pareto frontier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We appreciate that you found our paper important and novel! We address each of your questions below:\n\n**W1: Hyperparameter spaces**\n\nWe thank the reviewer for raising this point. We add TuneTables hyperparameters in the PDF attached to the global comment. For all baselines, we use the same hyperparameter ranges as the Tabzilla paper with one exception: we increased the range of tree depth by a factor of 100, since a high depth is much more appropriate for very large datasets. The tabzilla authors point the reader to [this file](https://github.com/naszilla/tabzilla/tree/main/TabZilla/models) in order to see hyperparameter ranges. We are happy to add all of this information directly into our paper.\n\n**W2: Arrangement of the paper**\n\nWe thank you for drawing our attention to the indirection in Sec. 4. We will update this so that it refers to Table 5 directly. Regarding how to interpret Table 5, we clarify below:\n\nThe columns labeled SKT / FTS / SMP list the best performing method for sketching, feature subsampling and label-aware sketching technique, respectively. Label-aware sketching refers to a strategy where we either sample instances proportionate to their labels, or we oversample minority classes with replacement to create a class-balanced distribution. While the choice of label-aware sketching strategy is often impactful (and we use it in TuneTables), and the choice of feature subselection method can be important for some datasets, in all but one case, no sketching method we test outperforms random sampling.\n\nThe comment that “all sketching and feature selection methods plateau in performance well before approaching parity with GBDTs” is referring to the fact that in Table 5, TabPFN fails to match CatBoost’s performance on seven datasets, all of which are very large.\n\nIn the final version of our paper, also as you suggest, we will incorporate a table of contents for our appendix, and we will add the above discussion near Table 5.\n\n**W3: Ablation aggregate scores**\n\nThanks for this note! We will add relevant aggregate scores to all results tables in the main paper.\n\n**W4: TT-medium and TT-light**\n\nWe thank you for raising this point about reporting TuneTables-medium and TuneTables-light results for all of TabZilla, including runtimes, in order to better understand the time-performance pareto frontier of our method. While preparing this rebuttal, we have completed these experiments on the 98 datasets from the tabzilla benchmark suite; (please see our global response; section A). We hope this extension of results helps to address your concern.\n\n**Q1: Memory requirements**\n\nWe thank the reviewer for raising the topic of FlashAttention. For our reply, please refer to our comment on review 7Ntu, Q5.\n\n**Q2: Grid search**\n\nThe space over which we conduct grid search in TuneTables is conditioned on metadata; # features, # samples, # classes. The feature and class splits are set by the limits of the particular TabPFN checkpoint we optimize. When we reach a leaf node, TuneTables-standard and TuneTables-medium conduct a grid search over a fixed range of configurations. The size of our search space is always < 30, and usually < 10. For feature-large datasets, TuneTables-light conducts additional optimization in the feature space prior to grid search. See the PDF attached to our global response for further details. For the final version of our paper, we will add this clarifying material to the appendix.\n\n**Q3: TuneTables-Hard criteria**\n\nWe curated datasets with far more samples and features than in the standard datasets from tabzilla (max 45,000 samples) from OpenML to better understand the scaling challenges of PFNs, omitting image classification datasets. Finally, because we intended to include TabPFN in our analysis and every dataset we had curated was beyond the recommended limits of that model, we added several small datasets from tabzilla; since TabPFN generally outperformed boosted trees on these smaller datasets, and TuneTables extended TabPFN, we heuristically selected smaller datasets so as not to favor either TabPFN or boosted trees. For the final version of our paper, we will add these details to the appendix.\n\n**Q4: Table 11** \n\nWe thank the reviewer for this question. In Table 11, we compare different variants of our method and entire backbone fine-tuning (TabPFN-FT) to TuneTables itself. For all methods in this table except for TuneTables, there is no hyperparameter optimization (HPO). For TuneTables, we use our standard grid search. For the final version of our paper, we will clarify this.\n\n**Q5: Ensembling**\n\nWe thank the reviewer for this clarifying question about ensembling in TuneTables. In a TuneTables ensemble, each ensemble member fits its own tuned prompt to the data. Variance in the ensemble members is introduced by differences in the random initialization of the tuned prompt, as well as permuting the order of features and labels, a la TabPFN, one time before each tuned prompt is fitted. For the final version of our paper, we will clarify the sentence you cited in your analysis.\n\n**Q6: Coresets**\n\nFor efficient coreset selection, we use a variant of [Farthest Point Sampling](https://ieeexplore.ieee.org/document/577129); after selecting an initial set of n=5 random points, we compute the distance of each point in the dataset to the set of already selected points. Next, we add to the set of selected points the point whose distance to the selected points is maximal. Finally, we update the distances of all the points according to the updated selected set; and continue iteratively. We will add these details to the final version.\n\nThank you again for your excellent comments. We respectfully ask that if you now feel even more positively about our paper, to consider slightly increasing your score. We are happy to continue the discussion any time until the end of the discussion period. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tJmxW2DtX", "reviewer": "Reviewer_7Ntu", "summary": "The paper proposes TuneTables, a method for improving the performance of PFNs on large datasets with a variable number of features and classes. TuneTables uses prompt tuning (fine-tuning) to learn a small set of parameters (context / synthetic datapoints). Depending on the qualities of the dataset, a feature subselection method or a new decoder may be used. The experiments show that TuneTables outperforms TabPFN on larger datasets. In the analysis, the paper shows that bias (specifically demographic parity) can be mitigated with a regularizer.", "questions": "See Weaknesses. \n\n\n\"[TuneTables] also does not improve on TabPFN for small datasets (fewer than 1000 samples, 100 features and 10 classes); we postulate this is a result of overfitting.\" -- Could you clarify why you postulate it is due to overfitting? On page 4, it is mentioned that \"if a dataset is small enough to run with the original zero-shot version of TabPFN\". Wouldn't you just use the original zero-shot version of TabPFN in this case (without any tuning)?\n\n\"(b) if there are too many features\" -- Could you clarify how you measured what constitutes \"too many\"?\n\n\"limited to around 3000 by conventional GPU sizes.\" -- Is this assuming a batch size of \"1\" with \"3000\" context tokens? Or a larger batch size?\n\n\n\"memory requirements scale quadratically with the context length\" -- This is not necessarily the case. At inference time, attention can be computed recurrently (Rabe et al., 2021; Feng et al., 2024), resulting in transformers only needing linear memory with the context length (albeit the amount of computation in total is still quadratic).\n\n---\n\n\"Self-attention Does Not Need $O(n^2)$ Memory\", MN. Rabe, C. Staats, arXiv:2112.05682, 2021\n\n\"Attention as an RNN\", L. Feng, F. Tung, H. Hajimirsadeghi, MO. Ahmed, Y. Bengio, G. Mori, arXiv:2405.13956, 2024", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- Impressive number of experiments that show TuneTables outperforms TabPFN\n - Soft Prompt Tuning for TabPFN is an interesting idea", "weaknesses": "My main concern with this paper is its presentation.  \n\nMost notably, I found it hard to read Section 4. It is particularly dense. There are a lot of Table/Figure references (in total 8), all of which are not on the same page, so a reader would need to flip back and forth between pages to find the table and figures. \nFurthermore, many of the references in Section 4 point to various resources in the Appendix. \nFor example, many analyses are mentioned without the tables to back them up in the main paper. Instead, tables in the appendices are linked. The main paper should be complete. The reader should be able to draw conclusions from the provided experiments (tables and figures). In this case, the conclusions are mentioned but the Appendix is used as additional space for the figures and tables. \n\nHere are just some of the examples: \n - \"(see Appendix Table 7 for the results). [...] we find that TuneTables achieves [...]\"\n - \"In Appendix Table 6 we show that [...]\"\n - \"At inference time, TuneTables is significantly faster than TabPFN; see Table 9\" (Table 9 is in the Appendix). \n - \"ablate the tuned prompt size and the sue of ensembles in Table 11 and Table 12, finding that smaller\" (Tables 11 and 12 are in the Appendix). \n\nInterestingly, Section 5 and 6 are not dense and very easy to read which further makes the paper awkward to read (going from a dense Section 4 to a super not dense Section 5 and 6). One way to tackle this is to shorten Sections 5 and 6, making more space for Section 4, allowing for a small subset of Tables in the Appendix to be moved to the main paper which would make Section 4 easier to read. \n\nSome additional notes regarding presentation:\n - \"TuneTabes-hard\" is the name of a dataset. The dataset name is too close to that of the method: \"TuneTables\". The names should ideally be easily distinguishable\n - (Figure 3) \"subset of [48]\" -- Is there a dataset name for [48]? It would be easier to read if so. \"[48]\" is used to reference the paper but is also used as the \"name\" of the dataset. \n - (Figure 3) \"reduces or reverses TabPFN's limitations\" -> \"addresses TabPFN's limitations\"\n - \"One limitation of [48] Table 1\" -- I'm guessing this is referring to Table 1 from [48], but this is confusing as there is also a Table 1 in this paper. \n - \"GBDTs\" - this term is mentioned several times without clearly defining what it is an acronym for, i.e., Gradient Boosted Decision Trees.\n - \"Motivated by the limitations of sketching for large contexts\" -- To the best of my knowledge, no limitations of sketching were mentioned before this point, so it is unclear what it is being referred to\n - Section 7: \"PFNs are neural processes (NPs) [52]\" -- If I understand correctly, it appears that NPs and PFNs are being explored separately but are similar models if not the same. As mentioned in the paper, there have been several neural processes works that tackle similar issues with scaling transformer-based models, so it would be nice if the paper connected NPs and PFNs more. For example, clarify that PFNs are neural processes earlier. The focus of the paper on PFNs and the brief mention of \"(similar to works on neural processes; see Section 7)\" suggested to me that PFNs and NPs are considerably different for the majority of the paper until I read Section 7. \n\nMinor:\n - Typo: \"subset of [48]on\" -> \"subset of [48] on\" (missing a space)\n - Section 4 is titled \"Experiments\". Section 5 and Section 6 are experiment/analyses-focused, so it's more accurate for them to be subsections of Section 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We appreciate that you found our idea interesting, and that you appreciated the large number of experiments we provide. We address each of your questions below:\n\n**W1: Presentation**\n\nWe thank the reviewer for a comprehensive set of suggestions on how to improve the presentation of our work. We especially agree with you that the main paper should be complete by itself. We have thought through and made all of the changes to make this true and to address all of your comments. Because NeurIPS does not allow us to upload a modified version of our paper as part of the discussion, we cannot provide you with a highlighted manuscript with our changes; instead, we summarize the major points below.\n\n- We significantly compressed Sections 5 and 6 by having just one section titled “TuneTables extensions”, with mitigating bias and dataset understanding as two subsections, and substantially cutting down the text to have only the main points.\n- Even with the compression of Sections 5 and 6, we cannot add all of Tables 6-12 to the paper. In order to reduce the need for the reader to flip between pages while reading, we added one-sentence summaries of the contents of the main appendix tables referenced (6,7,9), where they are referenced in the Sec. 4. Here is an example of such a change:\n\n>In Appendix Table 6 we show that despite the large divergence from the PFN’s pretraining, TuneTables achieves a high rank, second only to XGBoost. -> In Appendix Table 6 we show that despite the large divergence from the PFN’s pretraining, TuneTables achieves a mean rank of 2.52, ahead of CatBoost and a ResNet, second only to XGBoost, whose mean rank is 2.0.\n\n- We move much of the ablation discussions in Sec 4 into the appendix next to tables 11 and 12, giving a high-level summary in Sec. 4\n- We now name the benchmark in our Table 1 as the “tabzilla benchmark suite” to be consistent with the language of recent work.\n- We also changed \"reduces or reverses TabPFN's limitations\" -> \"addresses TabPFN's limitations\", per the suggestion\n- Thanks for the suggestion; we renamed the “TuneTables-Hard” dataset to LargeScaleTables, to reduce confusion between the algorithm and the dataset\n- We have also made the smaller edits you suggest, including directly referencing neural processes earlier in the paper – we now write, “A recent breakthrough, prior-data fitted networks (PFNs) [33, 52] are a specific type of neural process (Garnelo et al., 2018) which learn to perform approximate Bayesian inference in a single forward pass using in-context learning.”\n\n**Q1: Overfitting**\n\nThanks for the question. In our experiments, we found that TuneTables would sometimes attain higher 3-fold cross-validation accuracy compared to zero-shot TabPFN, but lower test accuracy, particularly when val and test sets were small. TabPFN, which was pretrained on large quantities of synthetic data and did not optimize using the val set, exhibited low val/test divergence on all datasets. We will add this comment to the final version of our paper.\n\n**Q2: Zero-shot TabPFN** \n\nThanks for the question. Zero-shot TabPFN is indeed used as part of the TuneTables grid search on such datasets. The reason why we do not only use TabPFN is that the exact size at which TuneTables outperforms TabPFN is dataset-dependent (with an average transition of around 800 samples). We will add this comment to the final version of our paper.\n\n**Q3: Too many features** \n\nThanks for the question. By ‘too many features’, we mean more than the 100 features which TabPFN can utilize without feature selection methods. We will clarify this in our final draft as well.\n\n**Q4: Batch size 1** \n\nYes! The statement is assuming a batch size of 1. We conduct the relevant experiments on NVIDIA RTX8000 GPUs with 48GB of VRAM per GPU. We will clarify this in our final draft.\n\n**Q5: Memory requirements**\n\nWe thank the reviewer for raising the topic of FlashAttention. Indeed, FlashAttention is memory-linear in sequence length. However, please note that it has not yet been implemented in the public release of TabPFN -- it may be integrated into a future release, and any benefits will carry over to our method. Please also note that while Flash Attention is extremely useful, it still has other limitations: inference time continues to be quadratic, and there are other overheads on GPU memory. Overall, there remains a considerable need for new algorithmic methods which can be scaled up to the sequence lengths required by large tabular datasets. We will add a clarification in our paper regarding this point. \n\nThank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jjKufJyrO", "reviewer": "Reviewer_cmgc", "summary": "The paper proposes a method to compress large training sets into smaller contexts through soft prompt tuning for prior-data fitted networks. The proposed techniques relax the constraints of PFNs by allowing for an increased number of features, more in-context training examples, and a greater variety of classes. The authors show that the proposed methods achieve state-of-the-art performance across a wide range of datasets. In addition, the proposed method can also be used to optimize a fairness objective.", "questions": "1. For tabular data, do you assume the type of the attributes (numerical, categorical, etc.)?\n2. Do you have comparisons with previous works on data summarization?\n3. For bias mitigation, I suggest also including a baseline of the base TabPFN while debiasing the selected training data.\n4. Could you explain why PFNs only support a fixed number of classes?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow, with the problem well-motivated.\n2. The paper tackles an important limitation of PFNs, which can inspire follow-up studies and enhance the practical usability of PFNs.\n3. The authors conduct extensive experiments that study the advantages and limitations of the proposed method.", "weaknesses": "1. The proposed methods to deal with an increased number of features, training examples, and classes are disconnected, and some of the techniques are not new. The feature selection methods are classical methods that are widely used in machine learning and data sciences. Freezing the encoder of a network and fitting a new decoder is also very common in model finetuning.\n2. The proposed method shows close performance to GBDTs, while it is much more computationally expensive.\n3. The presentation of the results needs to be improved. In Figure 2, the lines are overlapping, making it hard to distinguish. Figure 3 is also confusing without a clear explanation of the y-axis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We appreciate that you found our paper well organized and easy to follow, and our problem well-motivated. We address each of your questions below:\n\n**W1: On novelty**\n\nThank you for raising this point. We note that our novelty lies in applying prompt tuning to PFNs, and that we fill a major gap for PFNs: the issue of scaling beyond their various capacity limits. We respectfully note that the NeurIPS reviewer guidelines includes novel combinations of well-known techniques to be included in originality, and overall there has been a trend towards emphasizing impact rather than novelty.\n\n**W2: Computational expense**\n\nThanks for bringing this up. We discuss this point as a limitation in Sec. 8 of our paper. However, we also note that, given the novelty of our method and the ongoing efforts to optimize the runtime of transformer-based architectures such as TuneTables, we can reasonably expect future research to further improve accuracy-runtime tradeoff of our method. Moreover, our method shows complementary strengths with GBDT, as each method outperforms on specific datasets. \n\n**W3: Presentation of results**\n\nWe thank the reviewer for pointing out the improvements that can be made to Figure 2. We have created a wider version of Figure 2 such that the vertical lines for the algorithms no longer overlap, which we added to the paper. We also added the following to the caption of Figure 3:\n\n> The colorbar on the y axis represents the comparative change in per-dataset accuracy between two algorithms (A: blue, B: red). Positive numbers represent the absolute gain in accuracy of B w.r.t. A, negative numbers represent the absolute gain in accuracy of A w.r.t. B.\n\n**Q1: Assuming attributes**\n\nWe do assume the basic type of each feature (numerical or categorical) on our tabular data. \n\n**Q2: Data summarization**\n\nWe thank the reviewer for this question on data summarization. While preparing this rebuttal, we have completed additional experiments on several different methods for feature summarization using our own benchmarks (please see our global response part C), as this provides a realistic estimate of how feature summarization methods perform in this particular setting. Our results for sketching summarization are discussed in Appendix Table 5; there, we observe that no sketching method outperforms random search for most dataset-algorithm pairs. We hope this extension of results addresses your concern. \n\n**Q3: Bias mitigation**\n\nWe thank the reviewer for making this suggestion. ​​In Sec. 5 of our paper, we cite fairness optimization as one particular example of a more general class of problems to which TuneTables is applicable, namely, multi-objective optimization problems. While in the particular case of fairness it might be possible to attain better results with TabPFN by debiasing the training data, this would not give us more insight into multi-objective optimization in TuneTables, as not all multi-objective optimization problems are amenable to such interventions. We nevertheless consider this an important and useful direction for future research into fairness with TuneTables.\n\n**Q4: Fixed number of classes**\n\nThanks for this question. The TabPFN authors provide three reasons in their paper why they choose to fix the number of classes at 10; see below.\n\n“We focus on small datasets because (1) small datasets are often encountered in real-world applications (Dua and Graff, 2017), (2) existing DL methods are most limited in this domain (Grinsztajn et al., 2022) and (3) the TabPFN would be significantly more expensive to train\nand evaluate for larger dataset.”\n\nThank you very much, once again, for your excellent comments. We respectfully ask that if you feel more positively about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for their valuable feedback. Our work introduces TuneTables, allowing PFNs to scale by orders of magnitude and achieve strong performance on large datasets. We appreciate that the reviewers find our techniques important, interesting and novel (FY4B, 7Ntu, cmgc), with the empirical results of TuneTables (YTTX, FY4B, 7Ntu, cmgc), and comprehensible, easily understandable writing (cmgc, FY4B) listed as strengths. Following your suggestions, we highlight further improvements:
- (A) We compare TuneTables-medium and light to our baselines on the 98 datasets curated in the [TabZilla Benchmark Suite](https://arxiv.org/abs/2305.02997). TuneTables outperforms all algorithms at less than 60% of CatBoost’s runtime; TuneTables-light performs nearly identically to CatBoost at less than 20% of its runtime (although it is still slower than a highly optimized XGBoost)
- (B) We present new results for the ExcelFormer algorithm on the TabZilla Benchmark Suite
- (C) We present a new ablation on feature selection
- (D) We add regression experiments with TuneTables, utilizing 10 datasets (also sourced from TabZilla)

We would be very happy to keep the discussion going, addressing any points that remain unclear, or any new suggestions. Thanks again for your suggestions!

**(A) TUNETABLES VS CATBOOST AND XGBOOST ON THE TABZILLA BENCHMARK SUITE**

These results are reported for the main 98 datasets in the TabZilla Benchmark Suite, with all algorithms reporting results for all datasets. We report the aggregate (mean) values, averaged over three splits. Accuracy is on the test split. TuneTables outperforms all algorithms, while requiring less than 60% of CatBoost’s runtime.

| Algorithm     	| Accuracy | Runtime |
|------------------|----------------|----------|
| TuneTables   	| 0.861      	| 573  |
| CatBoost     	| 0.857      	| 1061 |
| TuneTables-medium| 0.855      	| 305  |
| XGBoost      	| 0.855      	| 57   |
| TuneTables-light | 0.854      	| 196  |

**(B) EXCELFORMER RESULTS**

Per Reviewer YTTX’s request, we have run ExcelFormer on the TabZilla Benchmark Suite; given the limited time we had for the rebuttal, we report here on a 50 dataset subset of the TabZilla Benchmark Suite in which ExcelFormer, CatBoost, TuneTables and XGBoost all ran successfully (B1), as well as a 21 dataset subset in which all algorithms ran successfully (B2). ExcelFormer generally performs well, and is among the strongest deep learning algorithms in this limited subset. For the final version of this paper, we will extend ExcelFormer to more datasets for a comprehensive picture.

**TABLE B1**

| alg_name	| Accuracy | Runtime |
|-------------|----------------|----------|
| TuneTables  | 0.867      	| 681  |
| CatBoost	| 0.864      	| 972  | 
| XGBoost 	| 0.862      	| 68   |
| ExcelFormer | 0.852      	| 811  |

**TABLE B2**

| alg_name        	| Accuracy | Runtime |
|---------------------|----------------|----------|
| TuneTables   | 0.828      	| 13   |
| TabPFN.         	| 0.828      	| 69   |
| CatBoost        	| 0.819      	| 755  |
| NODE            	| 0.812      	| 2080 |
| XGBoost         	| 0.808      	| 47   |
| rtdl_ResNet     	| 0.808      	| 183  |
| SVM             	| 0.803      	| 1192 |
| ExcelFormer     	| 0.802      	| 371 |
| LightGBM        	| 0.801      	| 13  |
| RandomForest    	| 0.799      	| 9	|
| DANet           	| 0.798      	| 843 |
| SAINT           	| 0.797      	| 906  |
| rtdl_FTTransformer  | 0.795      	| 246 |
| TabNet          	| 0.786      	| 498  |
| DecisionTree    	| 0.774      	| 1	|
| rtdl_MLP        	| 0.770       	| 143  |
| LinearModel     	| 0.764      	| 1	|
| MLP             	| 0.749      	| 240  |
| STG             	| 0.752      	| 273  |
| KNN             	| 0.751      	| 3	|
| VIME            	| 0.726      	| 301  |

**(C) FEATURE SELECTION ABLATION**

| Model     | Metric | Random | Mutual Information | PCA  | PCA + Whitening | ICA  | Sparse Random Projection |
|-----------|--------|--------|--------------------|------|----------------|------|--------------------------|
| **TabPFN**| **Avg. Test Acc.**| 0.347  | 0.539              | 0.557| 0.557     | 0.565| 0.566                    |
| **CatBoost**| **Avg. Test Acc.**| 0.420  | 0.685              | 0.727| 0.727     | 0.726| 0.702                    |


We provide a preliminary ablation on our choice of feature selection methods in the paper, PCA and mutual information using three datasets from TabZilla. Similar to our findings in Appendix Table 5, we note that random feature selection is not a very strong baseline, particularly for TabPFN. We observe that PCA and mutual information are reasonable options, but in certain cases, other methods will perform better; in the interest of reducing our training time, we keep our search space small. We choose PCA and mutual information for TuneTables as they exhibit complementary strengths.

**(D) TUNETABLES REGRESSION RESULTS**

We report preliminary TuneTables regression results on 10 datasets from TabZilla. For these experiments, we train a new PFN from scratch on a new space of synthetic datasets designed for regression analysis, as we find this checkpoint performs better than TabPFN on average. We will add details of the model training to our final paper. We report non-normalized R2 scores on the test set, averaged over three shifts, as well as the average end-to-end runtime over three shifts. Unfortunately, the tabzilla authors have yet to release the raw results for their regression experiments, and we therefore were not able to compare these scores to any baseline models in time for the rebuttal – however, we will do so in the final draft of our paper. 

In general, TuneTables performs well, with an average R2 score of .808 across all datasets. See the attached PDF for the full results.

Meta-review: This paper represents a solid contribution to the field of tabular data analysis and thus deserves to be accepted to the main proceedings of NeurIPS. Reviewers in particular described the experimental results as extensive and mentioned the motivation was clear. There were some major concerns about writing raised by a reviewer that the authors have claimed to improved; I would like to stress that improvements in writing along those lines should be considered as a requirement for acceptance. The authors also provided another baseline at the request of a different reviewer and promised to flesh out the results fully for the main paper; I would expect this promise to be fulfilled as well.
Decision: Accept (poster)