Retrieval & Fine-Tuning for In-Context Tabular Models
Authors: Valentin Thomas
Junwei Ma
Rasa Hosseinzadeh
Keyvan Golestan
Guangwei Yu
Maksims Volkovs
Anthony L. Caterini
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
tabular data
retrieval
foundation models
transformers
Primary area: deep_learning_architectures
TL;DR: We use retrieval with fine-tuning to improve in-context learning on tabular data and show large improvement with better scaling

Abstract: Tabular data is a pervasive modality spanning a wide range of domains, and this inherent diversity poses a considerable challenge for deep learning. Recent advancements using transformer-based in-context learning have shown promise on smaller and less complex tabular datasets, but have struggled to scale to larger and more complex ones. To address this limitation, we propose a combination of retrieval and fine-tuning: we can adapt the transformer to a local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours in context. Using TabPFN as the base model -- currently the best tabular in-context learner -- and applying our retrieval and fine-tuning scheme on top results in what we call a locally-calibrated PFN, or LoCalPFN. We conduct extensive evaluation on 95 datasets curated by TabZilla from OpenML, upon which we establish a new state-of-the-art with LoCalPFN -- even with respect to tuned tree-based models. Notably, we show a significant boost in performance compared to the base in-context model, demonstrating the efficacy of our approach and advancing the frontier of deep learning in tabular data.

Retrieval & Fine-Tuning for In-Context Tabular Models
Valentin Thomas' valentin. valentin.t@layeree t@layer6.
Junwei Ma"
Rasa Hosseinzadeh rasa@layer6.. ai
jeremy@layer6.at ai
Keyvan Golestan keyvan@layer6.cm ai
Guangwei Yu guangelayer6. ai
Maksims Volkovs
Anthony Caterini anthony@layer6. ai
maks@layer6. ai
Abstract
Tabular data is a pervasive modality spanning wide range of domains, and this inherent diversity poses considerable challenge for deep learning. Recent advancements using mansformer-basee in-context learning have shown promise on smaller and less complex tabular datasets, but have struggled to scale to larger and more complex ones. To address this limitation, propose combination retrieval and fine-tuning: we can adapt the transformer to local subset of the data by collecting nearest neighbours, and then perform task-specific fine-tuning with this retrieved set of neighbours context. Using TabPFN as the base model currently the best tabular in-context learner and applying our retrieval and PFN, or
fine-tuning scheme on top results in what we call a locally-calibratee 95 datasets curated by TabZilla
LoCalPEN. We conduct extensive evaluation from OpenMI. upon which we establish a new state-of-the-art with LoCalPFN even with respect to tuned tree-based models. Notably, show significant boost in performance compared to the base in-context model, demonstrating the efficacy of our approach and advancing the frontier of deep learning in tabular data.
Introduction
Tabular data is the most pervasive modality for practical problems in data science, spanning across
a wide variety of domains including finance, healthcare, and science [3] 47 12 46 48]. The diversity
and heterogeneity of tabular data pose great challenges for deep learning approaches (21, unlike modalities such as text and images in which neural networks can be designed to specifically exploit inductive biases underlying the data ( such, obtaining performant neural network on particular tabular data task often results in expensive iterations of training and hyperparameter tuning. Meanwhile, tree-based methods such as XGBoost []] and CatBoost [40] have proven to more robust the inherent challenges of tabular data, and thus have remained the dominant approach for this setting (2 9.. Yet recently, there has been progress made with transformers and In- In-Context Learning (ICL): such example is TabPFN (26). which is trained using a prior-fittinn procedure 36 that exposes the network to millions of possible data-generating processes, thus taking step towards encapsulating the heterogeneity of tabular data. Such approaches differ from classical algorithms that they process entirely new datasets in a single forward pass and obviate the need for training and hyperparameter tuning.
Despite the promise of transformer-based ICL methods in the tabular setting particularly on smaller datasets - scaling remains issue: memory scales quadratically with the size of the context. This limits performance when the entire dataset cannot fit into memory, and contrasts with classical algorithms that tend to improve the amount of available data increases. In addition to this, and as depicted in Figure [ TabPFN in particular can struggle with underfitting as dataset complexity
Egqual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

AUC
0.8
0.7
TabPFN 100 Mi
TOEPPN 200 MN
1
Number of pairs of concentric circles
(a) Vanilla TabPFN, full context (b) TahPFN-ANN, A 100
(c) Performance vs. Complexity
Figure I: a) TabPFN sify patterns such as three pairs of concentric circles of two classes. Decision boundaries are in black and shaded areas show the predicted class. b) Applying its k nearest neighbours can easily solve this problem. c) We observe that this approach is robust to the numbers of neighbours used (k) even when the dataset complexity increases and always performs better than vanilla TabPFN using full context (N 1000). Each point is averaged over 25 seeds.
when using the entire training data as context underfits and cannot clas-
adaptive local context for each point using
increases, when the entire dataset fits into the context; observe this shortcoming in real datasets as well, and suspect this could apply to any ICL-based model for tabular data.
To improve the scaling of tabular ICL methods in both dataset size and complexity, we draw on two techniques that have been incredibly successful foundational large language models: retrieval [31] and fine-tuning ( On the retrieval side,
the k-Nearest Neighbours (kNN) of given query point as the context for classification; modifying the context this way empirically allows for both enhanced processing of larger datasets and more complex decision boundaries. We also fine-tune end-to-end for each task, using an approximate neighbour scheme to facilitate backpropagation, and demonstrate significant performance gains beyond just ANN. We named model Locally-Caliin Calibbatte PFN or LoCalPFN for short - to represent the addition of retrieval and fine-tuning on top of a base TabPFN model, although this idea should naturally transfer to potential future ICL-based tabular foundation models as well [49]. We demonstrate that LoCalPFN is state-of-ther thrtt when comparing against both neural approaches and well-tuned tree-based techniques across a 95-dataset benchmark from TabZilla (35] We summarize our contributions below:
1. Provide insights into TabPFN - the current state-of-the-ee tabular ICL transformer based framework and analyze how its performance scales across several axes in both synthetic and real datasets. We identify failures to scale in both dataset size and complexity.
Propose LoCalPFN to address the scaling failures mentioned above, using a combination of retrieval and fine-tuning to allow for more effective use of the context.
3. Show LoCalPFN compares favourably to strong baselines on a large variety of datasets through extensive experimentation, analysis, and ablation.
2 Improving Tabular In-Context Learning with Retrieval and Fine- Fine-Tuning
In this section, we describe ICL applied to tabular data in particular TabPFN and the limitations of such approach. Then, present our contributions where treat the in-context learner base model on top of which retrieval and fine-tuning are applied.
2.1 Preliminaries on In-Context Learning for Tabular Data and TabPEN
Our method generally applies in-context learners, specifically for classification tasks on tabular data. While, at the time of writing the only successful model of that type is TabPFN (26), we expect other such base models to be published the future. TabPFN is trained using a prior-fitting procedure (36) where large number of synthetic datasets are generated using randomly initialized neural networks. This approach trains an underlying transformer-basee network on various generative processes designed to simulate the diverse interrelations that exist among the features of realistic tabular datasets.

---

After the prior-fitting procedure, the learned TabPFN model ingests an entire training dataset Derain - { Virain) ) consisting of feature-label pairs s E R and Virinn E {i...., C} for E (1,...;u N), along with features of a query point Buy (potentially in batch), and outputs a distribution over labels Buy E (1.... C}. Specifically, denoting the TabPFN network (outputting logits) with parameters 0 as fo. the resulting posterior predictive distribution is modelled by:
exp(fo( Dunin) (By
pa Payyay
Tay, Dusin)
(1)
exp(fo (fo (Zay Donin [c])
where [-] denotes the vector indexing operation.
Contrary to classical machine learning methods which are trained on one dataset and then evaluated on the same distribution, TabPFN has been shown to be able perform classification on wide range of tasks without training, thanks to its diverse prior-fitting procedure. This makes it one of the rare foundation models for tabular data. Key to this is the ICL ability of TabPFN: by using various training exampley as context, analogous to how transformers on language use the preceding tokens as context, TabPFN can classify new query points in a single forward pass.
2.2 What Constitutes Good Context for Tabular Data?
The quadratic growth of memory usage with context length in transformers presents a challenge: the number of support examples we can use is limited. For instance, while TabPFN performs best on small and simple datasets, where the entire training set fits within the context, it is unclear how to best use TabPFN for large and complex datasets. Naively, we might consider random subsample of the training data as context 35 19 However, Ma et al. [3 show that this method does not scale either and observe drop in performance as the dataset size increases.
Given these limitations, it is natural to ask "What constitutes good context for tabular data?". This topic has been thoroughly researched in natural language processing, which resulted in various techniques for prompt engineering, The situation is more complicated in the tabular domain, as there no natural order to tabular data as opposed to the natural order of the words in language.
Specifically for TabPFN, some attempts have been made to use a summary of the dataset as context, through either k-means centroids [19] direct prompt optimization [19 34 Yet in either case the flexibility of the method is limited by the use of a single context for all query points. Instead, we propose different approach here, where we we wish to classify. For tabular data, we hypothesize that the most critical information to classify a query point Bay is contained in its vicinity. Extensive evaluations (35) support this fact by showing that a simple kNN classifier can rival modern deep architectures designed for tabular data, such as TabNet 2 and VIME [44] We thus believe that using nearby points context is good inductive bias for tabular data classification.
local context tailored to each individual point
2.3 Better Expressivity and Scaling Using Local Information
To do this, the first step is to replace the global context by local context, i.e., with k:N((aag))) the k-nearest neighbours of the query Tay in the training data Durain we replace equation T Tby
exp(fo(Fuy ANN( Mabl
Po(y Tay: Dimin)
(2)
Dc exp(Sf((ray) ANN (xay))))
Better Expressivity It is well known that in ANN regression and classification, the number of neighbours & controls the bias/variance trade-off and as such the expressivity of the model. More precisely, large & tends to "oversmooth" and suffer from high bias/underfitting, while small & enables more complex decision boundaries but can suffer from more variance/overfitingg 24]. We show that this phenomenon is still true for transformers, beyond the simple ŁNN classifier, in Figure [ We generate datasets of size N 1000 so that it can be used as context by TabPFN without subsampling. As we increase the complexity of the dataset, measured by the number of concentric circles in this case, TabPFN fails to accurately classify (e.g., for pairs of circles in (a) and more generally in (c)). Retrieving fewer samples (k- = 10, 30, 100, or 300) for each query point using its k-nearest neighbours from the training data leads to large improvements in AUC over TabPFN as the complexity of the data increases ((b) and (c)). Note that k 1000 would correspond to using all samples as
3

---

context, and thus is equivalent to vanilla TabPEN. As such there is a continuum between TabPFN using the full dataset as context and our local context method using İNN, which we call TabPFN-KNN.
While Figure D is on toy synthetic data, we believe this result remains surprising: a priori, we would expect 25-million-per parraammeeer model (TabPEN) to be able to learn few circles, even with just ICI., Meanwhile, we believe that using local contexts allows TabPFN to fit more complex patterns, such as the three circles of Figure [ in the same way that using local linear regression enables more expressive (and in that case non nonlinear) decision boundaries B 23
loss
Test loss
Test locs
10-1
2
212
2
2
2
24
2
2
26 25 24
24
2
Dataset
Dataset size
Dataset
(a) adult-census
(b) electricity
(c) eeg-eye-stan
vary the dataset size and the context length for three large datasets. TabPFN is in shades of green and TabPFN-KNN is in shades of blue. The opacity represents the context length used (also labelled each line). It corresponds to random training samples for TabPFN and nearest neighbours for TabPFN-KNN, TabPFN is limited by context size and cannot make efficient use of larger datasets. While for context length dataset size (k = N), TabPFN and TabPFN-KNN have the same performance, TabPFN-ANN can leverage larger datasets with &NN-based contexts and shows improvements, often even for lower context lengths, Each point on this plot is the average of 100 random resamplings of the data.
Figure 2: Example of the behaviour of TabPFN and TabPFN-ANN
Better Scaling Using a local context has another benefit: it allows our method's performance to scale with the training dataset size. In machine learning. it is generally expected that the performance of an algorithm improves as the training set size N increases, since the empirical risk converges to the expected risk [50]. However, ICL-based methods (such as TabPFN) that require subsampling when the maximum context length is smaller than N do not scale with N. TabPFN-ANN, on the other hand, can still benefit from larger training set sizes N even when the number of neighbours k is much smaller than N, as the search is performed over the whole training set. We demonstrate this fact in Figure for three real datasets. While the exact patterns in the loss curves differ, we observe similar trend across many datasets, where the benefits of using TabPFN-KNN grow as the dataset becomes larger. In Figure 9 we provide more detailed figures which include training loss.
2.4 Efficient End-to-End Fine- Fine-Tuning With Retrieved Samples
In addition to retrieval, we fine-tune the model end-to-end on each dataset to further improve performance, as is common in Retrieval- Aaggmetl-- Generation (RAG) BI However, naive fine-tuning is not computationally efficient. Transformer-base of shape (B, Lex + Lay. d) where B is the batch size, Lex and Lay are the context and query lengths, and d is the embedding dimension. TabPFN uses only one fixed context for all points, with B = 1, Lax the training dataset size (or maximum context length if too large), and Lay Nug the number of points to classify. Contrary to text, there is no auto-regressive attention mask: the context examples all attend to each other (blue arrows on Figure 3a) while the queries only attend to the context and not to each other (red arrows Figure Bal. Therefore, the predicted classes can be computed in parallel and at a reduced memory footprint.
in-context models work with inputs
By comparison, when using local context with exact neighbours, the context is no longer shared, and therefore the batch dimension must be used for queries: the input has shape B Nay. Lexx the number of neighbours and Lgy 1, since the queries use distinct contexts. This is significantly less efficient than the inference performed by TabPFN, which both requires much less memory, and also allows the queries to be processed in parallel. Therefore, our main limitation is in fact the forward and backward passes when using exact neighbours, unlike most applications where retrieval is the bottleneck. As such, most common approximate ANN methods cannot address this issue.

---

p( Many Drein)
1) Sample point
2) Compute ANN
TabPFN
3) Shuffle and split into context and queries
I:NN
Local context
Connest
O O
Queries
Duaie
Zay
(a) Overall architecture of LoCalPFN
(b) Efficient local context computation for fine-tuning
Figure 3: Details of the architecture and the efficient context used during fine-tuning, a) During inference, for each query Eny, we compute its kNNs and use them as context. b) During fine-tuning, we have a modified procedure allowing shared context between many queries. We first select random training point, then compute its &NNs. Finally we randomly split those into context and a query set, allowing us to have a shared (yet local) context for many queries, similarly to vanilla TabPFN. Colours correspond to classes, highlighting that different classes can (and should) appear in the same context.
Instead, to improve computational efficiency during the end-to-end fine-tuning, we opt for a simple neighbour approximation technique wherein many queries share the same context. An illustration of the method is provided in Figure 3 for single batch dimension. More generally, let us assume that we want to pass gradients on My examples at once, using a context length of Lexx We propose to only use B different contexts, which we will use to classify Nay/ B samples each: First, B training examples are sampled. Then, their individual &NN search is performed with k = Lexx + Lay for Lgy = Nay/ B. Finally, those batches of h samples are each shuffled and split into context vector of length Lexx and a query vector of length Lyy. constructing the input vector of size (B, Lexx + Lay d). This allows to efficiently trade-off accuracy of the neighbours versus computational complexity: with lower B we share contexts between many points but this comes at the cost of an approximation in the &NN search as the notion of neighbourhood is not transitive, i.e., the neighbour of your neighbour might not be your neighbour. However each sequence in each batch only contains examples which are the general vicinity of each other. In practice, we observe that this method does not lead to any significant degradation in performance while allowing much faster training.
3 Related work
Foundational Techniques to Improve Tabular Deep Learners Deep learning techniques have historically struggled tabular data (2I). where inductive biases much harder to capture architecturally [4] as compared to text or images. The comparative lack of progress on a large foundation model for tabular data [9] is yet more evidence of this. However, recent approaches have successfully begun to leverage foundational ideas to improve performance. For example, San-Parametric Transformers (00 and SAINT 45 both combine row-altentive
cunsformer-basee
backbones with some form of self supervised pre-training; however, the former is limited by context size (a common theme for naive ICL-based learners), whereas the latter is not based on ICL and thus does not as easily apply to novel datasets. Models such as RIM [4] and TabR [20] on the other hand demonstrate how to effectively design tabular deep learners incorporating retrieval modules, but still require costly and brittle rounds of hyperparameter tuning to adapt to any specific dataset. Our approach is meant to target some combination of all these methods: provide ICL-based generalization capabilities, but without limitations on the context size. The retrieval mechanism within TabR itself relies on ANN, which is one of the most straightforward and widely used retrieval- based machine learning methods (24). In fact, ANN is still being actively studied in the literature, e.g., in Differential Nearest Neighbours Regression (DNNR) [38] and follow-up work (53), which aims to make ANN
NNot that, as we sample training points. these original B points are part of their own ANN To avoid duplicates, exclude the original B points from the KNN search.
5

---

differentiable; this showcases the potential of simple methods like ANN in different forms, although DNNR tackles a separate scope from our method.
TabPFN and Extensions TabPEN 266 is a transformer-based in-context learner that has emerged as popular model for tabular data, demonstrating strong performance on some benchmarks [35) It uses prior-fitting process (36] allowing for rapid adaptation to new tasks. This strong ability to
quickly generalize makes TabPFN somewhat of a foundation model for tabular data [49], from which
techniques for generation (33 and dataset distillation
M for example can emerge - interpretability
is also being studied [43] TuneTaables [19] attempts to use tabular sketching [3 to summarize
the incoming dataset and more effectively scale TabPEN's context; however, much like Ma et al. [3], this approach is limited by the use of a single context for all datapoints, as opposed to an adaptive local context. den Breejen et al. [15] is able to show some limited improvements by fine-tuning TabPFN, which we extend here by more closely pairing the retrieval and fine-tuning aspects. Concurrently, Xu et al. [52) are able to improve TabPFN by first clustering the training data with K-Means and routing each testing point to a given cluster, which is then used as a prompt; our method does not require any clustering of the data.
Links with LLMs The idea of pre-training model on corpora of text prior to fine-tuning has been explored in the Natural Language Processing domain for both classification and generation Later iterations refined this idea train a model and use its in-context learning abilities for new tasks [10]. This elicited research into prompt engineering to determine what to actually put in a model's context (39 5T Similar to prompt engineering, to better utilize the model's context, one can search for similar examples from corpora and use them to facilitate the task; this
tasks [1] 27
known as Retrieval-Augmente of the idea include training jointly with retrieval [22 g and augmenting the output of the model
Generation (RAG) 31 in the generative context. Other variants
with ANN via interpolating [29]. These ideas are analogous to our approach of (i) fine-tuning and retrieving jointly, and (ii) disjoint ANN and fine-tuning in our ablations, respectively. LLMs have also been directly applied to tabular data [T6 25 [8] however, due to the pre-training of these foundation models on large text corpora, there is the possibility of data leakage, which causes concern with evaluations 77. Note that this is not the case with TabPFN as it has been trained on synthetic data.
4 Experiments
In this section, we showcase the performance of LoCalPFN against a wide range of alternatives. We release all code to reproduce our results at htttpp://giccubb..mmm////rrrrrr labs/LoCa1PP)
4.1 Experimental Setup
We evaluate our methods against competitive baselines using 95 out of the 176 datasets from TabZilla [35), originally sourced from OpenML [5]. These datasets originate from diverse sources, including academic research, competitions, government agencies, and corporations. The 95 datasets are filtered from TabZilla to meet TabPEN's architectural requirements by ensuring that each dataset has at most 100 features, at most 10 classes, does not contain NaN values, and has at least one instance per class for each split. The details of the datasets are described in Appendix A. We further split the datasets into two subsets: "small" datasets which contain less than 2,000 instances, and "medium/large"" which contain at least 2,000 instances (up to 130,064). For each dataset, we use the splits from TabZilla with train-vlliattion-tes ratio of 80: 10:10::01 10. Since TabPFN was trained with a maximum of 1,024 data points as context size, the small datasets are roughly considered in-distribution for TabPFN whereas the large datasets are considered -istif-distributtin..
We conduct our experiments using 10-fold cross-validation over all datasets for all methods. For all baselines, we apply 30 rounds of hyperparameter tuning as in McElfresh et al. (35) and choose the best hyperparameters for each fold according validation AUC. In addition, the TabPFN baseline is reported without further ensembling or transformations, unless otherwise noted. Our methods also build on top of this same TabPFN baseline without further processing. We also compare against TabPFN with transformations in Section 4.4 More details of the baseline models can be found in Ap- pendix I We use the faiss 22 17 library for efficient I:NN search in our methods; this enables us to harness parallel computation to accelerate the nearest neighbour search. We evaluate our methods TabPFN-ANN and I. LoCalPFN against other models the following sections. Notably, without further fine-tuning, LoCalPFN is identical to TabPFN-ANN, Details of our method are in Appendix A.2.2

---

For LoCalPEN we also conducted some small experiments on hyperparameter optimization, but saw no real difference in performance across hyperparameter choices, besides learning rate which we tuned by hand on global level (i.e., not on a per-dataset basis). Thus, we retained the default hyperparameters we had initially from the TabPFN repository (other than learning rate). We will see later in this section that LoCaIPFN is also insensitive to choices in embedding and retrieval metrie; combining this with the strong performance across hyperparameter choices shows that the approach is quite robust overall.
Note on evaluation and the computation of proper confidence intervals: While many works evaluate tabular data methods on a small set of datasets and report confidence intervals/standard deviations for those, we choose to evaluate on a large number of datasets in order to have more meaningful results. However, this makes it harder to compute meaningful uncertainty. Agarwal et al. I dealt with a related problem in reinforcement learning; we follow their lead by, for example, reporting the interquartile mean (IQM, i.e., the mean of the middle 50% of scores), and we use their library to compute 95% confidence intervals via stratified bootstrapping.
4.2 Main Experiments
As shown in Table [ averaged over 95 datasets, LoCalPFN outperforms all other baselines, with significant improvement over TabPFN itself. Among the 47 small datasets, we found that TabPFN is in fact quite competitive with other methods, similar to what had been reported by McElfresh et al. [B5] Nevertheless, LoCaIPEN further improves the performance even in this setting and positions itself as the best method. For the 48 medium/large datasets, TabPFN underperforms the tree-based methods by a wide margin. Simply applying kNN on top of TabPFN leads to drastic performance increase on top of TabPFN. Finally, LoCalPFN further improves on TabPFN-KNN, and either performs on par with, or outperforms, all other methods. We also measure the accuracy, Fl score, and relative AUC metrics such as average rank and z-score and see similar pattern; those details can be found in Tables 6 to g
Deep Learning Model Comparisons: Note that most deep learning baselines are significantly more expensive to train and tune larger datasets, and as such, most of them could not be run all datasets (55). Nevertheless, in Table 5 we compare TabPFN-KNN and LoCalPFN to other deep learning based methods on the datasets which the baselines have been able to run, and show even larger improvement in performance. The datasets we used for this comparison can be found in Table 4
Table 1: AUC scores and confidence intervals for all 95 datasets, 47 small datasets, and 48 medium/large datasets, respectively.
All
Small
Medium/Largg
Algorithm
IQM AUC
Mcan AUC
IQM AUC
Mcan AUC
IQM AUC
Mcan AUC
ANN
0.843 (188-116771
0.812
(1816-18101
0.807
[196612161
0.781
(LY1241701
0.882
[1301118811
0.848
(186041101)
TabPFN
0.917 (191-199991
0.867
(1866-01601)
0.898
0.849
0.927
(1925-11999
0.884 [188-1198]]
TabPEN 3k
0.924 (1922-0927)
0.873
570)
0.903
(1597-0000)
0.852
(1545-0854)
0.938
(1957-4.500)
0.893
[1892-3044]
LightGBM
0.940 (1ss:43421
0,885
0,884
(1376-229111
0.838
(1351-12651
0,966
(1966419671
0.931
(18604116111
RandomForest
0,936
0,886
po (0.655-0800)
0.895
(0555-0.0011
0.848
(0544108854)
0.955
[o (09554..555)
0,920
(1919..9211
CatBoost
0,942
0.891
(1.58-0.890 895]
0.907
(1501-0.914)
0.856
(0.44042602)
0,961
[1.6044
0.926
(1925-4.927)
XGBoost
0.943
po 9400 sud)
0.892
po 580-0 SM5]
0.907
0.861
10856-00007 907)
0.965
(a 966-0 (66)
0.931 I (1900-0 952)
TabPFN-ANN
0.943
(194144594)
0.891
0.922
M277
0,864
(0381-18711
0.955
(19554499666
0.916 (191541917]
LoCalPEN
0,958
(0056-00000)
0.908
(1.05-0.911]
0.937
(1531-0.942)
0,882
(0.575-0599)
0.968
(0.66-42.00)
0.934
(0953-0.555)
4.3 Analysis: Scaling with Dataset Size and Complexity
In this section, we further validate that LoCalPFN addresses the scaling problems of TabPFN. We see in Figures and that TabPFN scales badly with both size and complexity; here, verify this phenomenon in real datasets. While this may appear contradictory to Table of McElfresh et al. [5), which shows TabPFN excelling on a large benchmark suite, we note that the aforementioned study mostly contained small datasets and thus it did not show the same performance drop-off observed here.
Scaling with Size In Figure 4a Random Forest for different dataset sizes. We choose relative AUC for clarity as there is no clear correlation between the maximum AUC attainable dataset and its size. We see that, compared to the Random Forest baseline, TabPFN's performance drops drastically when the dataset size increases
we report the AUC of different algorithms relative to the AUC of
7

---

beyond 3,000, indicating poor scaling with N. On the other hand, the other methods we report scale more favourably with the dataset size. We also see that LoCalPFN scales favourably compared to the Random Forest baseline, and even outperforms XGBoost for large datasets. Error bars represent the 95% confidence interval.
Scaling with Complexity While in Figure[ we could easily control the complexity of the task, there no generally accepted measure of complexity for an arbitrary dataset. Here, we propose simple proxy for complexity: for a given dataset, we measure the difference between the best and worst AUCs of given set of algorithms, similarly McElfresh et al. [3]. The rationale is that AUC itself camnot capture complexity, as for instance learning to separate two Gaussians can be done optimally by linear classifier, but the еттог rate depends on their variances. In Figure [4b] we analyze performance across different levels of this complexity measure. We first calculate the difference in AUC for each dataset using all listed methods T then we divide the datasets into five quantiles on the r-axis, with increasing complexity as we move to the right; on the y-axis, we report the mean AUC relative Random Forest across 10 folds and across the datasets each bin. We see that TabPFN scales poorly with increasing complexity, and LoCalPEN still outperforms all other methods in the quantiles of higher complexity, demonstrating that its improvements are not just limited to "easy" datasets.
Mean AUC (relative RF)
Mean AUC (relative RF)
0.04
0.06
0.02
0.04
0.00
ANN
-0.04
XGBoost
0.00
TabPFN
-0.06
RandomForest
-0.08
LightGBM
0.04
LoCaIPEN
133
3-10
10-50
0.20%
20-40%
40-60% 60-80%
80 80-100%
Dataset size (x1000)
Complexity estimate (Easiest Hardest)
(a) AUC vs. Size
(b) AUC Complexity
Figure 4: Analysis of AUC as function of size and complexity. TabPFN fails to scale both in size and complexity while LoCalPEN is able to still outperform on the far end of the spectrum. See Figure & for version with absolute AUC. Note that each of the plots contain all datasets in the 95-dataset benchmark, and no subsampling is performed.
4.4 Ablation Studies
In this section, we provide ablation studies on different design choices for LoCalPEN.
Importance of Joint Retrieval and Fine-Tuning One could naively consider simply fine-tuning the in-context learner on randomly sampled context during training. In Figure (left) and Table 9 we see that this indeed improves performance over the original TabPFN baseline. However, applying TabPFN-ANN on top of a naively fine-tuned model does not improve performance further. Therefore, it is crucial to fine-tune the model end-to-end with the retrieval (LocalPFN).
(1eft)
Choice of Embedding We also try different embeddings. The simplest approach to use the raw standardized features for the nearest neighbour retrieval. In (centr)) (centre) and Table TO it is shown that this simple approach is actually very competitive. We compare it to two additional approaches: using one hot encodings (when the size of the resulting vector does not exceed 100 features), and the output of the encoder layer of TabPFN. For the latter, we recompute the search index every 30 gradient steps. The results show that the former, using one hot encodings, does lead to some improvement, however mostly for smaller datasets (see Figure [0]ad Figure
Why do simple embeddings work so well? While tabular data is complex in many regards [2]] features in tabular data are often semantically meaningful. For this reason, we expect metrics that decompose over individual features, i.e., d(x, _') = E, da(,;, z²). to be good inductive bias for tabular data, especially when it is normalized. This would not the case for most natural signals. We experimented with two different metrics here, including the Euclidean (L.2) distance and cosine similarity; in practice, these two choices are quite similar when applied to standardized features and stuck with the Euclidean distance.
8

---

importance of joint retrieval and fine-tuning
Impart of embedding choices
Impartance of using loral context
0.52
0.92
0.91
0.11
0.91
Mean AUC
0.59
0.49
5.59
0.88
0.69
0.57
0.47
5.57
1M0PP9
meeeuu
Pire tame keed
Dad 438s
Licreen
Locap Hntattn-nttattr enttater
UnCaeP mo
Lcalth fre
T2MPP9
TasPP 30
Tha
LoCatpn
Figure 5: Ablations for different design choices on all 95 datasets. Left: Fine-tuning jointly with retrieval yields better performance. Centre: The choice of embeddings for retrieval does not change the performance drastically but can lead to some improvements. Right: Methods using a context that does not depend on the current query do not match the performance of methods that use local context.
Importance of Using Local Context Up until now, have mostly compared to TabPFN with a random context of size 1,000. To prove our point that using a local context is inherently better than global context (same context for all queries), we attempt to find the best model using global context by first using an ensemble of 32 TabPFN models (with randomized feature and class ordering in Hollmann et al. (26). which we denote TabPFN-32ens, and then by increasing the context size of the ensembled TabPFN to 3,000 (TabPFN-3K-32ens). As depicted in Figure S (right) and detailed in Table TT while improving significantly upon TabPFN, these are still not competitive even with our TabPFN-KNN. As one can criticize the use of single context to classify queries, we further experimented with a "Bayesian" view of the probability by averaging it over contexts context obtained from the
Pe(Hay Tay. Puain) & fc pel (Bay Tqy. C)p(( Dusin)dC, where training data Darin. We experimented with splitting Pirain into chunks of size 3,000, and averaging the probabilities over those chunks. We call this method RabPFN-32ens-2 -n-int (for integral) and show that, while it does improve upon the single random context, it does not outperform TabPFN-KNN, Additionally, this method is very expensive as: (i) using 3,000 context examples is GPU memory intensive, and (ii) the integral over chunks makes the inference scale as O(N). The last method we compared to is "In-Context Distillation" (34) (TabPEN-ICD) where, similarly to Feuer et al. [19, authors directly optimize the context. While this last method leads to better performance (including on larger datasets, see Figure IT), since performs task-specific tuning is more comparable computationally to LoCalPEN, which remains superior.
Sensitivity Number of Neighbours We also ablate the choice of the number of neighbours used as context. This the only hyperparameter for TabPFN-ANT and also an important hyperparameter for LoCalPFN. In practice, for the number of neighbours, we use the minimum of (i)
0.935
0.830
Mean AUC
0.920
10 times the square root of the training set size, and (ii) a pre-defined maximum. For large datasets, the number of neighbours should roughly align with the pre-defined maximum. In 6 we vary this pre-defined maximum while observing the mean AUC on the 48 medium/large datasets. We found that TabPFN-ANN is not very sensitive to this choice long LoCalPEN is able to improve TabPFN-ANN on all context sizes. Surprisingly, we observe that LoCalPEN is able to outperform the random forest baseline using a maximum context size of only 50, and also outperform the XGBoost baseline with maximum context size of 500. The details of the ablation can be found in Table 12]
0.915
TobPF N-NN
0.905
Xcillost
Foree
0.900
20a
1000
Maximum number of neighbours
is at least 100. We also that
Figure 6: Ablating max # of neighbours
Quality of Approximate Local Context addition to the above, we also assess the quality of the approximate local context in real datasets in Appendix A.5.5 and Table I3

---

4.5 Runtime Study
We also conduct runtime analysis for LoCalPEN, TabPFN-KNN, and other tree-based models, show- ing mean test-set AUC function of runtime. In Figure 7a measure this runtime the total time taken for training and evaluation. We can see that the general trend for all our algorithms shows positive correlation between runtime and AUC. We also observe that TabPFN-KNN runs surprisingly fast while still achieving quite high AUC on the 95 datasets. The fast runtime together with very few hyperparameters suggests that TabPFN-RA NNN will perform very well practical machine learning en- gineering and research. LoCalPFN achieves significantly higher performance than all other techniques, and even though it obtains this performance which a much higher runtime, is also worth noting that the deep learning baselines shown in Table
even longer time for training and evaluation.
One of the drawbacks of using local context, though, is that TabPFN-ANN's and LoCaIPEN's inference time is slower by to 2 orders of magnitudes when compared to tree-based methods. This fact is be taken into account when extremely high throughput inference is needed.
Algorithm
Training (s)
Inference (s)
0,910
LoGaiPAN
CatBoost
6.86
0.01
0.905
LightdeH
1.000
LightGBM
2.94
0.02
RandomfFores: XGBloot
RandomForest
1.15
0.04
AUC
0.900
100
XGBoost
2.26
0.01
0.895
LoCalPEN-1000
43.47
0.72
LoCalPEN-200
22.45
0,55
0.890
200
LoCalPFN-
19.76
0.38
0.885
0
0.72
0.880
TabPFN-ANN-200 TabPFN-NNN-100
0
0.55
10P
10a
0
0.38
Time (s)
(b) Training and inference times. Note that LoCalPFN has the same inference time as TabPFN-ANN.
(a) Trainnenferencc time vs. AUC
Figure 7: a) AUC vs. Runtime for all 95 datasets. TabPFN-ANN has very low runtime and strong performance, while LoCalPFN is able to achieve the highest AUC overall. We use bold text to denote maximum number of neighbours used. b) Breakdown of the total time in training time and inference time for all algorithms. As local in-context methods are all significantly larger than tree-based methods, their raw inference time is slower.
5 Conclusion and Limitations
In this paper, we demonstrate how to use retrieval and fine-tuning improve performance on tabular data classification tasks, introducing LoCalPFN as a version of this framework that uses TabPFN as the base model. LoCalPFN breaks new ground for neural approaches on tabular data, even showing improvements over workhorse tree-based techniques. We also provide TabPFN-KNN as a variant without fine-tuning, demonstrating superiority over the base TabPFN model and practical utility.
However, despite its successes, our framework also has some limitations. The first is that we have only shown that retrieval and fine-tuning improve TabPFN, since is the only proven ICL-based tabular model. Thus, we cannot be certain that our ideas would directly transfer to new base models, although the success of these concepts in other domains provides some evidence. is also worth noting that the original RAG paper (31) only initially demonstrated success on BART. Next, the reliance on TabPFN as base model brings some limitations: besides the constraints on number of features and classes dis- cussed in [ection[4.] we are also unable to easily test our ideas in regression tasks since TabPFN is not designed for them. Although we expect these constraints to gradually be lifted as tabular foundation models improve and increase their scope, we also note that tree-based methods are not nearly as sus- ceptible these issues. Going further the comparison with based methods, while we note that LoCaIPFN performs better than them in our experimental study, we also point out in Section 4.5 that the runtime of LoCalPFN slower. However, is still faster than other deep learning approaches, and the cheaper TabPFN-KNN variant runs as fast as any tree-based method on datasets we studied, while still attaining respectable performance. Overall, we believe that the benefits of our framework far out- weigh the limitations, as LoCalPFN greatly expands the capabilities of deep learning tabular data.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
[2] Sercan Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI Conference Artificial Intelligence, pages 6679-6687, 2021.
[3] Omar Benjelloun. Shiyu Chen, and Natasha Noy. Google dataset search by the numbers. In The Semantic Web ISWC 2020, pages 667-682, 2020.
[4] Ege Beyazit, Jonathan Kozaczuk, Bo Li, Vanessa Wallace, and Bilal Fadlallah. An inductive bias for tabular deep learning. In Advances in Neural Information Processing Systems, 2023.
[5] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijshers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren. OpenMI. benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[7] Sebastian Bordt, Harsha Nori, and Rich Caruana. Elephants never forget: Testing language models for memorization of tabular data. arXiv preprint 203hv:2403.06644 2024.
[8] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- -antistt Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pages 2206 2240, 2022.
[9] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, 2020.
[11] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SigKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[12] Jillian M Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint arXiv:2012./5330 2020.
[13] William S Cleveland and Susan Devlin. Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association, 83(403): 596-610, 1988.
[14] Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, 2015.
[15] Felix den Breejen, Sangmin Bae, Stephen Cha, Tae- Youn Kim, Seoung Hyun Koh, and Se- Young Yun. Fine-tuning the retrieval mechanism for tabular deep learning. NewrIPS 2023 Second Table Representation Learning Workshop, 2023.

---

[16] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn. Dimitris Papailiopoulos, and Kangwook Lee. LIFT: Language-intertin fine-tumin lne-tuning for non-language machine learning tasks. In Advances in Neural Information Processing Systems, 2022.
[17] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. arXiv preprint 2401.08281, 2024.
[18) Xi Fang. Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models (LLLMs) on tabular data: Prediction, generation, and understanding A survey. Transactions on Machine Learning Research, 2024.
[19] Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. TuneTables: Context optimization for scalable prior-data fitted networks. In Advances in Neural Information Processing Systems, 2024.
[20] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. TabR: Tabular deep learning meets nearest neighbors. In International Conference on Learning Representations, 2024.
[21] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances Neural Information Processing Systems, 2022.
[22] Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training, In International Conference on Machine Learning, pages 9929-3938 2020.
[23] Trevor Hastie. Generalized additive models. Statistical models in S, pages 249-307. Rout- ledge, 2017.
[24] Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, 2009.
[25] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. TabLLM: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages
2023.
[26] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference Learning Representations, 2023.
[27] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifiça- tion. arXiv preprint 1801:06 2018.
[28] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):33--54,, 2019.
[29] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 1911:00/00007 2019.
[30] Jannik Kossen, Neil Band. Clare Lyle, Aidan Gomez, Tom Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. In Advances in Neural Information Processing Systems, 2021.
[31] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retreee generation for knneeeege-intensivee NL.P tasks. In Advances in Neural Information Processing Systems, 2020.
12

---

[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[33] Junwei Ma, Apoorv Dankar, George Stein, Guangwei Yu, and Anthony Caterini. TabPFGen Tabular data generation with TabPFN. In NeurlPS 2023 Second Table Representation Learning Workshop, 2023.
[34] Junwei Ma, Valentin Thomas, Guangwei Yu, and Anthony Caterini. In-context data distillation with TabPFN. arXiv preprint arXiv:2402.06997, 2024.
[35] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakr- ishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Advances in Neural Information Processing Systems, 2023.
[36] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian inference. In International Conference on Learning Representa- tions, 2022.
[37] Alexander Munteanu and Chris Schwiegelshonn. methhds and history: theoreticians design pattern for approximation and streaming algorithms. KI-Künntlliche Intelligenz, 32:37-53, 2018.
[38] Youssef Nader, Leon Sixt, and Tim Landgraf. DNNR: Differential nearest neighbors regression. In International Conference on Machine Learning, pages 16296-16317, 2022.
[39] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 14, 2021.
[40] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, 2018
[41] Jiarui Qin, Weinan Zhang, Rong Su, Zhirong Liu, Weiwen Liu, Ruiming Tang, Xiuqiang He, and Yong Yu. Retrieval & interaction machine for tabular data prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1379-1389, 2021.
[42] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.
[43] David Rundel, Julius Kobialka, Constantin von Crailsheim, Matthias Feurer, Thomas Nagler, and David Rügamer. Interpretable machine learning for TabPFN. arXiv preprint arXiv:2403. 10923, 2024.
[44] Ravid Shwartz-Ziv and Amitai Armon Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[45] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342. 2021.
[46] Qi Tang, Guoen Xia, Xianquan Zhang, and Feng Long. A customer churn prediction model based on XGBoost and MLP. In International Conference on Computer Engineering and Application (ICCEA), pages 608-612, 2020.
[47] Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable OOD detection on medical tabular data. In Machine Learning for Health, pages 341-354, 2020.
[48] Christopher J Urban and Kathleen Gates. Deep learning: A primer for psychologists. Psychological Methods, 26(6):743, 2021.
13

---

[49] Boris van Breugel and Mihaela van der Schaar. Why tabular foundation models should be research priority. International Conference on Machine Learning. 2024.
[50] Vladimir Vapnik. The Nature of Statistical Learning Theory. Springer Science Business Media, 2013.
[51] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
[52] Derek Xu, Olcay Cirit, Reza Asadi, Yizhou Sun, and Wei Wang. Mixture of in-context prompters for tabular PFNs. arXiv preprint arXiv:2405.16156, 2024.
[53] Han-Jia Ye, Huai-Hong Yin, and De-Chuan Zhan. Modern neighborhood components analysis: deep tabular baseline two decades later. arXiv preprint arXiv:2407.03257. 2024.
[54] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela Van der Schaar. VIME: Extending the success of self-and semi- suni-super learning to tabular domain. Advances in Neural Information Processing Systems, 2020.
14

---

A
Appendix
A.1
Datasets
Table 2: 47 Small Datasets
dataset
did
instances
# Eeat
# classes
# cat
imbalance ratio
Australian
146818
690
14
2
8
1.248
LEDDdisplay-dommin-- 7digit
125921
500
7
0
1.541
acute- inflammations
10089
120
6
5
1,400
balance-scale
11
625
4
0
5.878
banknote--nn aut-autttiiiiion
10093
blood transfusion service center
10101
748
4
0
3.202
hreast-cancer
2.365
car-evaluation
146192
1728
21
21
18 615
car
146821
1728
6
6
18.615
climate model simulation crashes
146819
540
0
10.739
cme
23
1473
9
7
1889
credit-g
diahetes
37
768
8
2
0
1,866
dresses-sales
Jertility
9984
100
9
0
7,333
huyes- roth
146063
160
O
2.097
hill vvalley
145847
1212
100
1,000
ipd
9971
583
10
2
1
2.491
ionosphere
1,786
iris
59
150
4
0
1,000
kc2
3913
522
21
0
3,879
monks-proolemss-2
146065
601
6
1,917
pel
3918
1109
0
13.403
pc3
3903
1563
8.769
ped
3902
1458
37
2
0
7,191
postoperative patient- data
profb
3561
672
9
4
2,000
qsar-biodeg
9957
0
1,963
socmob
3747
1156
4
3.516
somar
39
208
0
1.144
steel-plates--uul
146817
1941
0
12.236
tae
47
151
5
2
1.061
tic-tac-toe
transplant
3748
131
3
0
1,729
vehicle
53
846
0
1,095
wilbe
9946
569
1.684
yenst
145793
1269
8
0
2,704
Table 3: 48 Medium/Large Datasets
datasel
did
instances
Teaa
clusses
imbulance ratio
32
2.956
Japanese Vowels
3510
9961
14
9
2.064
MagicTelescope
3954
19020
10
2
1.844
MiniBooNE
168335
130064
50
2,563
Phishing Websites
14952
11055
30
2
30
1.257
Satellite
167211
adult census
3953
32561
14
2
3.153
artiiiiil-charactees
14964
10218
10
2.360
bank-marketing
14965
45211
16
2
7.548
cardiotocograpiyy
9979
2126
35
10.925
churn
167141
5000
20
2
6.072
146195
14951
14980
14
1.228
electricity
219
45312
1.355
clevators
3711
16599
18
2.236
9985
6118
51
5,255
jannis
168330
83733
54
22,835
kell
3917
2109
21
2
5.469
kr-vs-kp
magic
146206
19020
10
2
1.844
mffat-fouricr
2000
1.000
mffat-kaad karhunen
16
2000
64
10
1.000
mleat morphological
2000
1.000
Continued on next
15

---

Table 3: 48 Medium//.arge Datasets
dataset
did
instances
Teat
classes
# cat
imblance ratio
mfsat-zerikk
2000
47
1.000
mushrocm
24
8124
22
2
22
1,075
numerai28.6
nursery
9892
12958
4
13.171
optdigits
5620
64
1.032
orone-level-Shh
9978
2534
72
14,838
page-blocks
30
5473
10
S
175.464
peudigits
1.084
phoneme
9952
5404
2
2.407
pollen
satimage
2074
64.30
36
2,450
segment
146822
2310
16
1.000
shurtle
146212
58000
4558.600 600
spambase
43
4601
57
2
1,538
splice
sylvine
168912
5124
20
1,000
wall-nhhot-naviatiin
6.723
wilt
146820
4839
17,540
Table 4: 71 Datasets Selected for Benchmarking Deep Learning Models
dataset
did
instances
imblance
Australian
146818
690
14
2
1.248
LLDDdisplay-domaain
Satellite
167211
5100
36
2
67.000
intiimmmaaaiionss
balance-scale
11
625
5,878
hanknote authentication
10093
1372
1.249
blood-transfusion servico-center
10101
3,202
breast-canoor
145799
286
2.365
car-evaluation
146192
146821
1728
18.615
cardiotocography
2126
chum
167141
5000
20
6.072
146819
540
18
10.739
1473
1,889
credit-g
31
1000
20
13
2.333
diabetes
dresses-sales
125920
500
12
11
1.381
oog-cye-stare
14951
14
1,228
fertility
9984
100
7,333
first onder--teeeeeeeee wnw.rrreeeeeeemmrrnnnn
9985
6118
5,255
hayes-roth
146063
2,097
hill-valley
145847
1212
100
1,000
ilpd
2.491
ionosphere
145984
351
34
1.786
1.000
kel
3917
2109
5.469
kc2
3913
522
3.879
kr-vs-pp
3
mleat fourrer
14
2000
76
10
1.000
karhumen
mffat- morphologiaal
18
2000
10
1.000
nfeat-zerike
2000
47
1.000
monks problems-
146065
601
1.917
mushroom
8124
22
22
1.075
optdigits
orone level Shr
9978
2534
72
2
14,838
nage-blocks
3918
1109
21
13,403
pc3
3903
1563
37
8.769
3902
1458
37
7,191
phoneme
9952
5404
2.407
pollen
postoperative-pprien.cco
146210
88
2,667
profo
2,000
qsar-biodeg
9957
1055
41
1.963
satimage
2074
6430
36
2,450
segment
146822
2310
16
1,000
socmob
3797
1156
3,516
Continued next page
16

---

Table Datasets Selected for Benchmarking Deep Learning Models
dataset
did
# instances
feat
classes
cat
imblance ratio
spambasc
4601
57
1.538
splice
455
3190
60
3
60
2.158
steel. steel-plates- plates-fault
146817
the
47
151
1.06]
tic-tac-toe
958
1.886
transplant
3748
1,729
vehicle
53
846
18
1.095
wall-robot-mavvigation
6.723
wabbe
9946
569
30
2
1,684
yeast
145793
1269
2,704
A.2 Experiment Details
A.2.1 Baseline Details
We the experimental results from TabZilla [35 when they available; in particu- lar, we do not use TabZilla's results for the TabPFN variants because they are not always complete, and there is dataset for CatBoost which does not have any results in the TabZilla repository. These results include the tree-based models and the deep learning model baselines. These results be found in https: //gi thub com/ wttpa///////////////aa//l.. http://ww...a.aaaooooooooo////// different variations of TabPFN inference techniques, we conduct experiments directly using the TabPFN repository pettps://githun com/ autoom/Taabbbu
and For
A.2.2 LoCalPEN Details
For all TabPFN-KNN experiments, use fixed number of neighbours equal to the minimum of (i) 10 times the square root the dataset size, and (ii) 1000. We find this works well since it adapts to small and large datasets. We use batch size of 512 for inference using the faiss library for speedup.
For LoCalPFN experiments, we use the exact same setup as TabPFN-ANN during inference. There- fore, at step O, LoCalPFN and TabPFN-KNN are equivalent. For training LoCalPFN, we adopt the AdamW 332 optimizer with learning rate of 0.01 and weight decay of 0.01. We not have warmup or learning rate scheduler. For the approximate local context for training, we use the same number of neighbours as TabPFN-KNN. We use fixed number query points (1,000) sampled from the training set and a batch of 2. For our reported results, we also use one-hot encoding for neighbour retrieval and inference. In addition, we evaluate our model every 30 gradient steps and apply early stopping based on the validation set AUC for each fold respectively.
All experiments for our proposed methods can be run on machine with single NVIDIA RTX 6000 GPU Ada Generation, 995Gi RAM, and AMD Ryzen Threadripper PRO 5995WX 64-Cores CPU, Additional runtime analysis can be found Figure
A.3
Additional Experiments
A.3.1
Comparison to Deep Learning Models
In addition to tree-based models, we also compare LoCalPFN and TabPFN-ANN with deep learning based methods. We use the results directly from the TabZilla repository. However, due the fact that a lot of the deep learning baselines are very computationally expensive, many of them were not able
all datasets. Therefore, 71 datasets upon which all the deep learning methods could run. The details of the 71 dataset subset that LoCalPFN
propose subset of the 95 datasets which contains
can be found in Table 4 The complete results can be found in Table We still outperforms all other models.
A.3.2 Comparison with Other Metrics
Here we also compare the performance of LoCalPFN with other models using accuracy, F1 score. or relative AUC measures such as average rank and z-score as the metrie. We can observe a similar pattern here: LoCalPEN either matches outperforms other models either of these metrics well.
17

---

Table LoCalPPN outperforms deep learning baselines significantly.
All 71 Datasets
Algorithm
IQM AUC
Mean AUC
VIME
0.771 [0. 70.000007822
0.741 [0.732-0.750]
rtdl MLP
0.855 0.846-
0.812 [0.806-0.818]
TabNet
0.881 (0.87400.888)
0.825 (0.81800.82))
STG
0.877 80.872-0.883]
0.829 [0.823-0.834]
rtdl_ Re_RNe
0.917 [0.912 0.922]
0.862 [0.857-0. 867]
rtdl FTTranffomer
0,919 [0.913-0.924]
0.869 [0. (0.864-0.874)
TabPFN
0.929 (0.925-0.932]
0.875 [0.871-0.879]
Fine-Tune
0,936 (0.932-0.939]
0.881 [0.876-0.886
TabPEN-ENN
0.948 [0.944-0.951]
0.889 [0.884-0.894]
LoCalPFN-encode
0,956 [0.953-0.959]
0.892 [0.887-0.897]
LoCaIPFN-raw
0.957 [0.954-0.960]
0.893 [0.887-0 898]
Fine-Tume+KNN
0,951 [0.948-0.954]
0.893 [0.888 0.897
LoCalPEN
0.959 (0.956 9.962.)62
0.903 (0.899-0.907]
Table 6: Accuracy comparison for LoCalPFN and the baseline models.
All
Small
Medium/Larre Large
Algorithm
IQM Acc
Mean Ace
IQM Acc
Mcan Ace
IQM Acc
Mcan Acc
TabPFN
0.856
(1555-0299)
0.817 (0.15-0.200)
0,836 (L53000842)
0.806 (as01-as11)
0.871
[0.500-2872]
0.828 (1826-4.300
TabPFN 3k
0.862
0.823
po p.5200 sDs)
0.839
to 5355
0.808 (100-0113)
0.88
(0557000 5871
0.837
(o (0535-0
RandomForest
0,875
(1575-0878)
0.839
puss7.o sul]
0.834
(n (
0.807 (1802-08121
0,900
(a (0500-0.001)
0.866
nusccio 567]
LightGBM
0.878
(1875-28811
0.842
0.830
0.807
0,918
(1916-119199
0.886
(1885-1887 (877
CatBoost
0.883
0.847
0.844
0.815
0.908
0.876
XGBoost
0.889
(1006-330144
0.848
(1845-18811
0.840
pracs-r
0,811
(1066-81211
0,919
(1915-410a11
0.887
(1306642000)
TabPEN-ANN
0.877 (1577-0579)
0.843
(1544100845)
0.856
(0551-00 8x2)
0.825
(0520+0.20)
0.891
(092)
0.861
[0.5600..52]]
LoCalPFN
0,902
0.865
0.875
0.840
0,918
0.890
Table 7: Fl score comparison for LoCalPFN and the baseline models.
All
Small
Bodiuu/LLarge
Algorithm
JQM Fl
Mean Fl
IQM F1
Mean Fl
IQM F1
Mean Fl
TabPFN
0.843
0.796
0.818
(1312-1325)
0.783
0.861
[1899418611
0.809
(1800418111]
TabPFN 3k
0.850
(1547-0853)
0.801
0.821
(0514-0320)
0.784
(177044770)
0.872
[1870448554]
0.818
(0816602500)
Forest
0.875
(1812487771
0.837
[1a35-43991
0.831
(182--1331)
0,805
0,900
(1358410011
0.863
[A882-12941
LightGBM
0.877
(0577-0 5811
0.841
pu558 o sud]
0.829
(o (2525-0836)
0.806
poson 11101-5111]
0,917
(0996-0.9119
0.885
(n 0.858.0 580)
CatBoost
0.882
(1879430001
0.845
pu pusss- (9S]
0.842
pu B066
0.814
0,908
(a 900-04 MEM
0.875
(1876-1876)
XGRoost
0.888
585-0 2011
0.847
(n 8a6-n 260)
0.839
(a 835-0 Bi6)
0.810
808-08800
0.919
ja 915-1 9011
0.886
8888--0 887)
TabPFN-ANN
0.867
[106--03601]
0.829
puazz-a 1031
0.841
pu (1056-189777
0.804 ((301-1111
0.884
[u (0035113000
0.854
[I (1885-1885)
LoCalPEN
0.897
(0.50-0.900)
0.859
(0.8560 avl]
0.869
(n (1546-0854))
0.832
(0.827-0857)
0.915
(0.913-2.900)
0.885
(a (0885448800 (00)
Table Relative score comparison for LoCalPFN and the baseline models. For brevity, we exclude the split between small and mediun/large, but that split also tells much of the same story.
Algorithm
Mcan AUC Rank
Normalized AUC
AUC z-score
TabPFN
5.3 (65.55)
0.56 (.51..577
-0.28 (03).-025)
TabPFN 3k
4.3 (41.47)
0.62 069
-0,07 [0010-004)
RandomForest
4.2 [66,435
0.71 (000.229
0.18 juis, 021)
LightGBM
3.4 130
0.72 pos, emg
0.19 po1s, [22
CatBoost
3.1 (29.14]
0.76 poss em
0.33 (i30, (00)
XGBoost
3.3 (11.44)
0.73 pm
0.23 po
KNN
7.1 (65.7.1
0.23 pm 225
-1 1,38 (1142..133)
TabPEN-ANN
3.8 (3.400)
0.72 (000..79)
0.17 px15.
LoCaIPEN
1.7 13.13)
0.85 0.67)
0.62 (159, (000)
18

---

A.4
Additional Analyses
IDPFN-KNW
0.950
XGBoost
0.90
0.925
RbPAN
Mean AUC
0.900
RandumForest
LightOBR4
0.85
TabPFN-NNN
Mean
0.875
0.850
XG BoBoost
TabPFN
0.825
RandomForest
0.800
0.75
LightGBM
0.775
LoCalPEN
0-20%
20-40%
40-60%
60-80% 80-100%
0-1
1-3
3-10
10:50
50+
Complexity estimate (Easiest Hardest)
Dataset size (x1000)
(b) Absolute (i.e., non-relative) mean AUC vs. complexity
(a) Absolute (i.c., non-relative) mean AUC vs. dataset size
Figure 8: Analysis of AUC as function of size and complexity. TabPFN fails to scale both in size and complexity while LoCalPEN is able to still outperform on the far end of the spectrum.
19

---

0.55
0.50
0.6
0.5
Jgg.
1ast l055
(TL (mitm
0.4
0.35
7.4
0.30
10
1000
6000
12000
0.1
0.20
0.0
0.15
kmn
knn
0.0
0.4
0.6
0.7
0.20
0.25
0.35
0.45
0.55
loss
(a) Two pairs of concentric circles
(b) adult-censur
0.7
0.8
0.5
0.5
bXLCX mo6th
lest 1oss
0.4
0.4
0.3
4000
0.2
5000
4000
0.2
32000
6000
0.1
0.1
Cit
kmn
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Train loss
(c) electricity
(d) eeg-eye-stare
Figure 9: Test loss training loss for TabPFN-ANN (crosses), TabPFN (circles) for different dataset sizes and context/number of neighbours used on four datasets. We observe generally that for low number of neighbours (dark crosses) and especially for small datasets (small crosses) there is signifi- cant overfitting (higher test loss than train loss). TabPFN tends to overfit less, especially on larger datasets, which is expected. Overall, using TapPFN-AND results in better underfitiing/overitinng trade-offs where we obtain both lower test and train losses, however the gap between them increases.
20

---

A.5
Ablation Studies
Figure 10 and Figure 11 show summaries of ablations on only the small datasets, and only the medium/large datasets, respectively. In the remainder of this subsection we see tables that show even further detail on the results presented in the main text.
importance of joint retrieval and fine-tuning
Impact of embedding choices
Importance using local context
0.88
0.88
0.68
Mean AUC
0.87
0.87
0.67
0.85
0.5
0:85
0.55
TAUFH
Fine-tatie laes
TauFP A38
Lacelern
Locan enceter
Pioon Roe
taumn
naseet 3lens
zesera Kemiiinn
Lacalrn
Locamp
Tas
Figure 10: Ablations on Small Datasets
Importance of joint tine-buning
Impact embedding
Imp ortance using
0.34
0.34
0.54
0.93
0.83
0.91
Mean AUC
0.92
0.52
0.92
0.00
0.00
5.50
0.89
0.69
0.88
Reeeture
Locapea anceder
0.88
LoCaPpk
0.89
TRuPPa- 2amm Zamd
narri-i-- 2hen
NAPPAICO
Locuspp
cane
Locao
Figure 11: Ablations on Medium/Large Datasets
A.5.1 Importance of Joint Retrieval and Fine-tuning
Table Ablation for fine-tuning. Applying TabPFN-ANN on fine-tuned model degrades the overall performance. On the other hand, performing local calibration by jointly retrieving and fine-tuning improve performance drastically.
Small
Medium/ Cadium/Larrge
Algorithm
IQM AUC
Mcan AUC
IQM AUC
Mcan AUC
IQM AUC
Mcan AUC
TabPFN
0.917 [0140199
0.867
(88440200)
0.898
(1892200044
0.849
[o343-2850
0.927
(0925-0920)
0.884
(1353-0335]
Fine-Tune
0.934
0,885
0.905
0.854
pas4--20111
0.953
pusst-o054)
0.916
Fine-Tune + ANN
0.938 jonsast
0.887
0.928
0.870
pass-oa77
0.946
posts-sssss
0.903
TabPEN-ENN
0.943
900)
0.891
0.922
0.864
0.955
0.916 m5150497
LoCaPPEN
0.958
jossuus.sn 5600
(00000911)
(1931-9922)
(1a75-0.809
0.968
(use7.0000)
0.934
(33330003))
A.5.2
Choice of Feature Encoding
Table 10: Ablation for choices of embedding. Converting categorical variables to one-hot gives
relatively moderate gain over other configurations.
Small
Medium/Large
Algorithm
IQM AUC
Mean AUC
IQM AUC
Mcan AUC
IQM AUC
Mcan AUC
LoCal LoCalPEN eeenner
0.955 (19533-9997)
0.899 maseasmnn
0.926
etb-cor
0.864 m1ssiaa77))
0.969
907 (sse)
0.934
possasi
LoCaIPEN raw
0.956 (USSLI 9381
0. 0.900 m1896-200))
0.928
10 0033300944
0.866 0155-0873)
0.968
0.933
LoCaIPEN-on hot
0.958 p12ss-c WM0
0.908 (1525-0911]
0.937
(1901-20442
0.882 (1875-0889)
0.934
21

---

A.5.3
Other Inference Methods of TabPEN
Table 11 shows the detailed performance values for TabPFN with different inference methods.
Table 11: Ablation for different TabPFN inference methods.
All
Small
Mediun/Laarg
Algorithm
TQM AUC
Mcan AUC
1QM AUC
Mcan AUC
IQM AUC
Mcan AUC
TabPEN-Ik: Ik- Icns
0,917
(1314419:3)
0.867
[us0041370
0.898
(1912-3044)
0.849
(154543255)
0,927
(1829-1822))
0.884
pa5s2-335))
TabPFN- Ik-32ens
0.936 (asss
0.879
(187543871
0,923 (14170011))
0.863 50057-
0.943
0.894
TabPFN-33--35 3ln-enss
0.943 (181-4355))
0.885
(1551-2358)
0,924 p1s15-3300
0.864
(1357-03100)
0,954
(usss-2.55)
0,905
p0925-0.000)
RabPFN-3k-32emm
0.945 (34424547)
0.887
(185340.90)
0,924
(0915-999
0,864
0.956
(1955-8.57)
0,909
TabPFN-IC1)
0.946 91s
0.892
(995
0.924
0.864
in s5
0.958
(n (r o09)
0,919
TabPEN-ENN
0.943 (094)-0:546)
0.891
0.922 923
0.864 (0.57.0251))
0.955
0,916
(115-9997)
LoCaIPHN
0.958 (n-95--
0.908 (101
0.937 (991
0.882
(n (044
0.968
(1967-1 606)
0.934
m 900)
A.5.4
Ablation for Maximum Number of Neighbours
Table 12 shows the detailed performance values for varying size of maximum number of neighbours.
Table 12: Ablation for sensitivity of k. The number after indicates the maximum number of neighbours used.
All
Small
Medium/ (tedium/Large
Algorithm
IQM AUC
Mcan AUC
IQM AUC
Meun AUC
IQM AUC
Mcun AUC
TabPEN-ONN-c20
0.923 (213343225
0.874
(1811-277))
0.894
(1801449111
0.845 (1:35-0325))
0.937
0,903 p0o1-1104
LabPEN-ENN-c50
0.935 9993
0,886 (8346888
0.91
[00S0317]]
0.859 (0.52.0554))
0.949
(1945.000)
0.912 p0i00.93)
TahPFN- -NN--110N00000
0.943 (19s01911)
0.890
(18744001)
0.921
(196-1577
0.864 (057--5711)
0.954
ja pesn
0.916 pa1sa
TabPEN-ENN-c22
0.943 (1641-9945)
0.890
in (8710.094)
0.922
(99685771
0.864 m757-95711
0.955
yo (530005))
0,916 99150917)
BaPPEN-ENN-c500
0,943 pesst-as4ss
0.891
(1887-2201)
0.922
(196-44221
0.864 (1s57-0371)
0.955
0.917 po
TabPFN-GNN-c700
0.943 (0441-1943)
0,891
(1881-0.994
0.922
(J06643227]
0,864 (0557-0371)
0.955
(0:05-0050)
0.916 poss5-911
TaMPEN-GNN-c1000
0,943
0.891
(1887:C.09]
0.922 [096-15271
0,864 (u857-0371))
0.955
0.916 ponsamm
LoCuIPEN-c20
0.941
(1988-8944)
0.890
() (188.28.944
0.920 [01144322)
0,865
(0559-0872)
0.953
ya (0:6240033
0,916 p91s0119)
LoCaIPINN-c50
0.950
9531
0.898
0.932 (1955
0.873 (asss-5561)
0.960
0.923 pemasuu
LeCalPrN-c100
0,953
(1551-0955)
0,901
0.932 [600644333]
0,875 (0.568-0882)
0.963
0.9620064
0.926 90220
LoCaIPEN-c2DD
0.955
poess- one]
0.904
0,935 passe511]
0.879 (n 477-4 sAs)
0.965
32 961 (vea))
0,928 parmass
LoCaIPeN-c500
0.957
pusts 959)
0.905
in 901600000)
0.935 [ess08411]
0.877 (a 800 583)
0.968
10 9.967 nesa)
0.932 meni
LoCaIPFN-e700
0.958
0.906
0.935
paraH1911
0.879
0.968
0.933 (43124941)
LoCalPEN-c1000
0.958
(1236-4.960)
0,908
(a (.905-2911)
0.937
(088141422
0,882
(0.515-0289)
0.968
(8.6674099)
0.934 (0234-333)
A.5.5 Quality of Efficient Local Context
In order to show the efficacy of the efficient local context, we compare LoCalPFN with the exact version where we use the exact neighbours for the context during training. In Table [3 LoCalPFN- exact-b32 indicates the aforementioned configuration with a batch size of 32, which is capped because of the GPU memory constraint. We compare this with another variant of LoCalPPN where we use 32 queries for training, i.e., CoCalPFN-approx--ae These two variants turn out to have very similar AUCs, which indicates the efficacy of the efficient approximate neighbour search method.
Table 13: Exact nearest neighbour search vs. approximate nearest neighbour search.
Mediun/Large
Algorithm
IQM AUC
Mean AUC
LoCaiPFN-exact-322
0.967 [1.66-01968]
0.931 [0.90-01992]
CCCallPN--aprox-pe
0.968 [0.967 0.668)
0.931
LoCaIPFN
0.968 (0.06701069)
0.934
[0.093-0.935]
22

---

Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions are highlighted in abstract and introduction. These are supported experimentally in section 2 and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss related work thorough the paper and have related work section. Some limitations are discussed thorough the paper and others are addressed the limitation and conclusion section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
The paper does not contain proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We aim at providing all the information necessary to reproduce our figures. We explain how they are constructed and provide all our hyperparameters and the details of the tasks in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We are releasing code for all experiments and providing a simple-to-use library to encourage adoption of this method before the conference occurs.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppbbccc/uuudddeeoo Coaee//ddeeSebmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Yes, we provide all necessary details to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we provide 95% confidence intervals based on stratified bootstrapping following Agarwal al. [
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources needed to conduct all experiments in Appendix A.2.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics stctts://neuriin cttpriipssccppppaaa http://sww..ppeeecccccccccddddea
Answer: [Yes]
Justification: Our work does not involve humans proprictary data. We do not foresee direct risks from the method itself either.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do foresee societal impacts for this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use public models and public data. We do not foresee risks directly arising from the use of our method.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cite and reference all code repositories that we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
datasets.papersstt
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We use public models and data.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codee/modll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "bVQU8vIXrJ", "reviewer": "Reviewer_j47T", "summary": "The paper proposes LoCalPFN, a new method that improves the scaling of transformer-based in-context learning for tabular data. It uses retrieval and fine-tuning to adapt the transformer to local subsets of the data, and demonstrates state-of-the-art performance on a variety of datasets.\nThe paper makes the following contributions:\n\n* Provides a comprehensive analysis of TabPFN and identifies key limitations in TabPFN's ability to scale with dataset size and complexity.\n* Proposes LoCalPFN, a novel approach that combines retrieval and fine-tuning techniques. LoCalPFN leverages these methods to enable more effective utilization of context, thereby enhancing the scalability of in-context learning for tabular data.\n* Demonstrates the superiority of LoCalPFN through extensive evaluation and ablation studies. These evaluations reveal that LoCalPFN outperforms baselines on a wide range of datasets", "questions": "1) why the dataset size would affect the model quality? is it related to the task complexity and diversity of the samples?\n2) can you given an example of the query x_qy for tabular data? and the distance metrics being used in kNN?", "limitations": "The paper does not address the important practical considerations of the model's cost and latency at inference time, especially retrieval is used. This omission is a significant limitation, as it hinders the understanding of the model's real-world applicability.  As acknowledged in the paper, LoCalPFN has a slower runtime compared to tree-based methods, which are a popular choice for tabular data. While the paper mentions a faster variant (TabPFN-kNN), its performance is not as strong as LoCalPFN.\"\n\nThe proposed LoCalPFN method is heavily reliant on TabPFN as the base model. This raises concerns about the generalizability of the approach to other base models and task types (e.g., regression).", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "* Novel Combination of Retrieval and Fine-tuning: The paper introduces an new approach by combining retrieval and fine-tuning for tabular data. \n\n* Extensive Evaluation: The paper presents a thorough evaluation of the proposed method (LoCalPFN) across a wide range of datasets, comparing it with numerous baselines and conducting ablation studies.", "weaknesses": "The paper contains several inaccurate statements, For example, in abstract, \"Recent  advancements using transformer-based in-context learning have shown promise on  smaller and less complex datasets, but have struggled to scale to larger and more  complex ones.\" is inaccurate -- depending on the dataset and task definitions. There are many such examples in the paper, which make it hard to understand the scope of the paper. It raises concerns about the overall reliability of the paper's findings.\n\nThe paper fails to clearly define the complexity of tabular datasets, making it difficult to assess the relevance and significance of the proposed method (LoCalPFN) in addressing the challenges of complex datasets.\n\nThe paper does not address the important practical considerations of the model's cost and latency at inference time, especially retrieval is used. This omission is a significant limitation, as it hinders the understanding of the model's real-world applicability.  As acknowledged in the paper, LoCalPFN has a slower runtime compared to tree-based methods, which are a popular choice for tabular data. While the paper mentions a faster variant (TabPFN-kNN), its performance is not as strong as LoCalPFN.\"\n\nThe proposed LoCalPFN method is heavily reliant on TabPFN as the base model. This raises concerns about the generalizability of the approach to other base models and task types (e.g., regression).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time you took to review our paper. We hope to clarify some points of confusion below in our response.\n\n> \"Recent advancements [...] have struggled to scale to larger and more complex ones.\" is inaccurate -- depending on the dataset and task definitions.\n\nCan you please clarify this point? This sentence is meant to be understood in the context of tabular data, as the previous one begins with “Tabular data”. In any case, we will update it to “have shown promise on smaller and less complex **tabular** datasets“ to remove ambiguity. \n\nHowever, if this sentence is being challenged **in the context of tabular data**, several works have indeed shown TabPFN to be a competitive model [e.g., https://arxiv.org/abs/2305.02997, https://openreview.net/forum?id=XctSyEsBzx] however it doesn’t translate as well to large datasets because of the limited context size [https://arxiv.org/abs/2402.06971 Fig 3, or Fig 2 of our paper].\n\n> “There are many such examples in the paper, which make it hard to understand the scope of the paper. It raises concerns about the overall reliability of the paper's findings.”\n\nWe would be happy to have such examples pointed out so that we can better clarify the paper and convince you that our results are strong and reliable.\n\n> “The paper fails to clearly define the complexity of tabular datasets, making it difficult to assess the relevance and significance of the proposed method (LoCalPFN) in addressing the challenges of complex datasets.”\n\nWe would like to address the comment with several points.\n\nComplexity of a dataset is one of the reasons we invoked when explaining why retrieval can help, even when the dataset size is small. Note that the main motivation for our paper is still to be able to use such in-context models on large datasets. Neither our method nor our results (Table 1 and 6-7) are affected by how we measure complexity.\n\nThere isn’t one way to measure data complexity and it is still an open question. Many of the usual measures are based on a notion of “compressibility” dating back to Kolmogorov complexity. Many practical methods are based on a measure of discrepancy between the data distribution (usually $p(x)$ or $p(y|x)$) and a model of known capacity. For instance, let’s consider a linear model and a more complex non-linear model (e.g., polynomial). If both the linear and non-linear model provide a similar approximation to $p(y|x)$ (i.e., low $\\text{KL}(p(y|x)\\mid\\mid q(y|x))$ or equivalently cross-entropy), then we can argue that the classification problem is not complex as a simple linear model is enough to approximate it. On the other hand, if the discrepancy between the two models is large, then a non-linear model is necessary to explain the data and as such it is more complex. This is very much in line with our intuition and how the term is used in machine learning, where some tasks are considered “hard” (such as mathematical reasoning) as small models don’t perform well but more advanced ones (e.g., adding more parameters or some search capability) perform better. Thus we argue that measuring the discrepancy between the loss/performance of different algorithms is both intuitive and widely used. Indeed we see examples in the tabular domain [https://arxiv.org/abs/2207.08815, 3.1 “not easy” paragraph] where the gap is measured, or [ https://arxiv.org/pdf/2305.02997 ] where datasets are considered “hard” if a simple baseline does not achieve top performance. We are therefore very much in-line with the standards of the field, including the tabular sub-domain.\n\n> “The paper does not address the important practical considerations of the model's cost and latency at inference time, especially retrieval is used. This omission is a significant limitation, as it hinders the understanding of the model's real-world applicability.”\n\nWe believe you are referring to our mention of the runtime (L328) and Figure 11 in the Appendix. You can see there that the non-finetuned method (TabPFN-kNN) is faster than all tree based methods but does not significantly outperform XGBoost or CatBoost. As a reminder, TabPFN-kNN represents the raw cost of inference for our technique when only retrieval – not fine-tuning – is performed, which should hopefully assuage concerns about the runtime of the retrieval component. Meanwhile, LoCalPFN is slower (by a factor 10-25x on average) compared to a single XGBoost run, but achieves significantly better performance. Depending on the exact budget and the hardware available to a specific person, TabPFN-kNN, XGBoost, CatBoost, or LoCalPFN might be the best choice. Note that, as GPUs are improving and inference is made more efficient by the day (e.g., better flash attention, quantization, etc.), LoCalPFN’s inference speed should improve over time. Please see the general message for more details; we have included a runtime comparison against popular deep learning techniques there as well, showing that LoCalPFN leads on performance while remaining competitive on runtime.\n\n> The proposed LoCalPFN method is heavily reliant on TabPFN as the base model. This raises concerns about the generalizability of the approach to other base models and task types (e.g., regression).\n\nPlease refer to the general response.\n\n**Dataset Size and Model Quality**: We empirically notice that for the datasets & models we have, the task tends to be a bit harder as dataset size increases. \n\n**Query Example**: Could you clarify what kind of example you would like to see? For all queries, we first standardize its features using training statistics and then retrieve its neighbours in the training set using L2 distance. We then pass the (context, query) vectors to TabPFN.\n\n**Overall**: Given the strengths you pointed out (comprehensive analysis and extensive ablations, novel method), and that we have hopefully addressed all your concerns, **we ask you to re-consider your rating of 2**. This is generally reserved for papers that are technically wrong.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FY0IoMAYss", "reviewer": "Reviewer_iFbB", "summary": "The paper introduces Locally-Calibrated PFN (LoCalPFN), an advanced model for tabular data that enhances the transformer-based TabPFN by incorporating retrieval and fine-tuning techniques. By using k-Nearest Neighbours (kNN) to select a local context for each data point and fine-tuning the model on this retrieved set, LoCalPFN adapts more effectively to larger and more complex datasets. Extensive evaluations on 95 datasets from TabZilla demonstrate that LoCalPFN outperforms both neural and tree-based methods, setting a new state-of-the-art in tabular data classification. The key contributions include addressing TabPFN's scaling issues, proposing an improved context usage with retrieval and fine-tuning, and showcasing superior performance through extensive experimentation and ablation studies.", "questions": "- Have you tested the retrieval and fine-tuning techniques used in LoCalPFN on other in-context learning models? \n- Given the constraints on the number of features and classes due to TabPFN, how do you plan to extend LoCalPFN to handle datasets with more features and classes?", "limitations": "- The paper focuses solely on classification tasks and does not address regression tasks. This omission leaves a gap in evaluating the full potential of LoCalPFN.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces Locally-Calibrated PFN (LoCalPFN), which enhances transformer-based in-context learning for tabular data by combining retrieval and fine-tuning techniques.\n- The research is robust, with extensive evaluations on 95 datasets from TabZilla. The authors provide comprehensive experimentation and analysis, demonstrating the effectiveness of LoCalPFN compared to strong baselines, including neural and tree-based models.\n- The paper is well-organized and clearly written, making the complex concepts and methods accessible.", "weaknesses": "- Although LoCalPFN shows improved performance, the fine-tuning process increases computational complexity and runtime, especially with large datasets. \n- The paper's reliance on TabPFN as the base model restricts the generalizability of the proposed method. While LoCalPFN demonstrates significant improvements, it remains unclear whether these benefits would transfer to other in-context learning models for tabular data.\n- The current implementation of LoCalPFN is constrained by TabPFN’s limitations on the number of features and classes, as well as its incompatibility with regression tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback.\n\n> “Although LoCalPFN shows improved performance, the fine-tuning process increases computational complexity and runtime, especially with large datasets.”\n\nYes this is true, however note that the complexity added by the finetuning process is not specific to LoCalPFN but a cost we also must pay when fine-tuning with TabPFN (vanilla). Note that despite the additional runtime, our method has similar runtime to many other deep learning approaches while having very high performance.\n> “The paper's reliance on TabPFN as the base model restricts the generalizability of the proposed method. While LoCalPFN demonstrates significant improvements, it remains unclear whether these benefits would transfer to other in-context learning models for tabular data.”\n\nPlease refer to the general message. The headline is that we expect the techniques and concepts from this paper to generalize to future tabular foundation models.\n> “The current implementation of LoCalPFN is constrained by TabPFN’s limitations on the number of features and classes, as well as its incompatibility with regression tasks.”\n\nThis is correct. Our contribution is a post-training one so we don’t retrain a base model. However we agree with the reviewer concerning the limitations of TabPFN and are currently working towards removing those. Please refer to the general message for more discussion regarding the possibility of other models.\n\n> “Have you tested the retrieval and fine-tuning techniques used in LoCalPFN on other in-context learning models?”\n\nWe are not aware of tabular in-context learners except for TabPFN. Are you referring to general in-context learners (i.e. LLMs)? There are multiple issues arising:\n1. Since each row cannot be considered as a token but as a (potentially very long) sequence of tokens of varying length, comparing rows for retrieval is not straightforward.\n2. How to embed rows is actually not obvious in the first place; since LLM perform best on text, most successful methods ([https://arxiv.org/abs/2206.06565 ]) rely on describing each row as a sentence.\n3. Because each row would have many tokens, our context size would be much more limited: 1-2 orders of magnitude below what TabPFN can do [https://arxiv.org/pdf/2406.12031 ]. \n4. Despite all of this, LLMs for tabular data prediction still lag behind traditional methods and are harder to evaluate as they have memorized many well-known tabular datasets [https://arxiv.org/abs/2403.06644 ].\n\n> “Given the constraints on the number of features and classes due to TabPFN, how do you plan to extend LoCalPFN to handle datasets with more features and classes?”\n\nEven though the current model is limited (100 features, 10 classes, no regression), it is still possible to extend it: for instance we can easily perform feature selection/averaging as the forward pass is fast. Any multiclass problem problem can be turned into multiple binary classification problems. And as we showed in the main message we can even perform regression as classification.\nHowever we also wish to go beyond those limitations and are actively towards designing a new architecture and training a new model that does not suffer from those.\n\n> Regression\n\nPlease refer to the general message", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A31ry02JVy", "reviewer": "Reviewer_jqEN", "summary": "The authors extend the recently introduced TabPFN to larger and more complex datasets by fetching a relevant context for each test point using a KNN algorithm. The author evaluate two methods, TabPFN-knn, which consists of using the original TabPFN on the fetched context for each test point, and LocalPFN, which adds a finetuning step to adapt TabPFN to these new kind of local context. For this finetuning, an approximation is devised to use close points together in both the context and the query, enabling a shared context which is faster, but  with results close to the result of a KNN for each point.\n\nThe paper shows extensive evaluations of these two methods, demonstrating state of the art results against the original TabPFN, Gradient Boosting Trees like CatBoost, and (not pretrained) neural networks on a previously introduced benchmark.\n\nFurthermore, the authors provide several ablations, for instance showing the importance of jointly adding the knn and finetuning steps.", "questions": "For inference, you have to use a different context for each points to predict, right? How long / memory intensive is it? What is the runtime repartition between finetuning and inference?\n\nDo you think you can finetune with a smaller k than what you use for inference?\n\n> We use the experimental results from TabZilla [35] when they are available.\n\nI assume they are always available expect for TabPFN variants? I also think that the fact that your using the hyperparameter spaces from [35] should be said in the main text (if that's indeed the case).\n\nWhich distance are you using in faiss? I'm wondering if using the euclidean distance would improve the performance of the ordinal encoded features for the knn (compared to using one-hot-encoding).", "limitations": "The authors have adequately addressed the limitation of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Important and novel contribution, which allow to use TabPFN is many more settings.\n\nThe evaluation is well done. The authors benchmark their method on a previously introduced dataset, and the selection they use is principled, showing that no cherry-picking of dataset was used. The baselines are strong (modern GBDT like catboost, modern NN like ft transformer, and different tabpfn variants).\n\nThe ablations are interesting and numerous. For instance, I was wondering about TabPFN-3k-32ens-int and was happy to see it. The importance of doing the finetuning jointly with the knn is interesting. The approximation used are also ablated, like in table 12.\n\nThe paper is very well introduced and written, and make for a very pleasant read.", "weaknesses": "Some details are missing, for instance:\n - how are runtimes computed for figure 11? Which hardware? Which hyperparameters?\n- I'm not sure whether LocalPFN undergoes some HPO or not (I think not). If it's not the case, I think it should be made clearer, as it makes the comparison to tuned GBDT more impressive. If it does, the hyperparameter space should be provided.\n\nI think TabPFN-**1k**-32ens-int (maybe less than 32 ens if it's slow?) would be an interesting baselines as TabPFN is optimized for smaller datasets than 3K.\n\nI think more aggregation metrics should be provided (for instance mean rank, mean normalized score (z-score or other rescaled scores)..).\n\nfigure 4 lacks details: which datasets are used? Are the different datasets subsampled to reach smaller sample sizes? I also don't understand why absolute mean AUC is decreasing when the dataset size is increased after a certain point in Figure 7. Is it because the list of datasets is changing?\n\nThe code is not currently available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and helping us improve the paper.\n\n> How are runtimes computed for figure 11? Which hardware? Which hyperparameters?\n\nThe time reported is the average time to perform training+inference for a single run (averaged over parameters/datasets/seeds). As such it doesn’t indeed account for the HPO for baseline methods which used one.\nAs reported in Appendix A.2.2 the hardware is as follows:\nAll experiments for our proposed methods can be run on a machine with a single NVIDIA RTX 6000 GPU Ada Generation, 995Gi RAM, and AMD Ryzen Threadripper PRO 5995WX 64-Cores CPU.\n\n\n> I'm not sure whether LocalPFN undergoes some HPO or not (I think not). If it's not the case, I think it should be made clearer, as it makes the comparison to tuned GBDT more impressive. If it does, the hyperparameter space should be provided.\n\nWe conducted some minor experiments with HPO and saw no difference except for learning rate which we tuned by hand. Thus, we kept the default parameters we had initially (from TabPFN repo). We believe this, on top of other choices not having much effect, such as using embedding vs raw space or euclidean distance vs inner product, shows the approach is quite robust.\nIndeed we will make this clear in the main text, thanks!\n\n> I think TabPFN-1k-32ens-int (maybe less than 32 ens if it's slow?) would be an interesting baselines as TabPFN is optimized for smaller datasets than 3K.\n\nWe do not have the exact numbers but generally TabPFN generalizes well to larger context and on average using larger context helps. The performance of 1k-32ens-int was above 1k-32ens but below 3k-32ens-int.\n\n> I think more aggregation metrics should be provided (for instance mean rank, mean normalized score (z-score or other rescaled scores)..).\n\nWe provide the mean rank, normalized AUC, z-scores for the algorithms of Table 1 in Table 3 of the attached pdf. Please let us know if there are some metrics you’d particularly wish to see and we’ll try to include them during the discussion phase. We made the choice to avoid normalized measures as the main metrics in our paper as we believe it can hinder the reproducibility of the results as the scores are now dependent on an exact set of algorithms. However we agree that they are also interesting and will include them in the paper.\n\n\n> figure 4 lacks details: which datasets are used? Are the different datasets subsampled to reach smaller sample sizes? I also don't understand why absolute mean AUC is decreasing when the dataset size is increased after a certain point in Figure 7. Is it because the list of datasets is changing?\n\nGood question, we will clarify it in the main text. Your second guess is correct. In this figure we have not subsampled or tampered with the datasets in any way, but only binned them in the appropriate size range. This also means that the mean AUC of each bin is not directly comparable to another bin as the datasets within one bin are totally disjoint from the one in another bin. It may be that datasets in the range 1000-3000 for instance are on average (in Tabzilla) harder than those in the range 3000-10000.\nThis is the main reason why we use a standard algorithm as a baseline so that some phenomena (such as TabPFN’s performance decreasing) with dataset size become visible.\n\n> The code is not currently available.\n\nFollowing NeurIPS policy we are reaching out to the AC in order to be allowed to share an anonymous repo link with you. Note that we intended to release our code later (as we would like to see our method being used, we are particularly hopeful TabPFN-kNN could be used as a replacement for TabPFN), and it is currently not optimally refactored/documented.\n\n> For inference, you have to use a different context for each points to predict, right? How long / memory intensive is it? What is the runtime repartition between finetuning and inference?\n\nTabPFN-kNN is essentially LocalPFN without finetuning. By looking at Figure 11 it can be seen that LocalPFN takes an order of magnitude longer (or more) compared to TabPFN-KNN. This is due to LocalPFN having to do finetune. Thus finetuning takes an order of magnitude more time compared to inference.\n\n> Do you think you can finetune with a smaller k than what you use for inference?\n\nWe suspect this approach would work (refer to previous point about robustness). However, it would cause a disparity between finetune and inference and might have negative effects. In terms of smaller k helping with finetune time, most of time is spent for model parameter update not retrieval.\n\n> I assume they are always available expect for TabPFN variants? I also think that the fact that your using the hyperparameter spaces from [35] should be said in the main text (if that's indeed the case).\n\nWe thank the reviewer for the observation, we will update the paper to reflect these more clearly. Most of the values were indeed present but we needed to re-run TabPFN (missing many datasets) and perform HPO for catboost on one dataset it was missing as well. \n\n> Which distance are you using in faiss? I'm wondering if using the euclidean distance would improve the performance of the ordinal encoded features for the knn (compared to using one-hot-encoding).\n\nWe also had intuition on which encoding/distance would work best. We tried multiple approaches and they made little difference thus we ended up using the simplest variant. Please refer to the general message for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnuJGcLYoC", "reviewer": "Reviewer_ZbyM", "summary": "Tabular data is an important yet understudied modality in machine learning. Following recent success of Tab transformer and TabPFN, people have made significant progress on tabular tasks. This paper is an extension of TabPFN, a previous sota tabular learning model on small scale data, and the authors combine with retrieval and finetuning. The motivation is intuitive and the idea is natural. \n\nIn the retrieval part, the authors reuse the simple kNN trick and discuss in details the performance in TabPFN.\n\nIn the finetuning part, the authors construct the training data by shared contexts and finetune the TabPFN model.\n\nThe authors experiment on large scale datasets and proposes the state performance model LoCalPFN.", "questions": "1. L32, memory scales quadratically in the size of the dataset. Do the authors mean exactly the calculation of attention matrix? If so, it seems this memory limit s general and what the authors do is to leverage the limited context length better?\n2. In Table 1, is the gain from loCalPFN w.r.t TabPFN-kNN coming from only finetuning per dataset? \n3. Just make sure I understand correctly, do the authors fine tune on each dataset of 95 benchmarks?\n4. How are tree methods evaluated? Any hyperparameter optimization?\n5. What is the training time and inference time on each dataset and how this time scales with dataset size?\n6. What is the selection criteria of 95 datasets? How are they chosen, preprocessed, etc.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Tabular learning is an important topic in real life applications\n- The methodology of retrieval and finetuning is not dependent on ICL tabular model.\n- The performance gain by retriveal and finetuning is clear wrt TabPFN\n- Scaling experiment is provided which is important to large scale analysis", "weaknesses": "- There are many limitations on the datasets requirements, e.g. number of features, number of classes, tasks\n- No other deep learning or transformer based models as a baseline\n- Sec 2.4 is not quite clear in finetuning steps, though the general ideas are appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> There are many limitations on the datasets requirements, e.g. number of features, number of classes, tasks\n\nWe agree. There are ways to go beyond them while still using the same architecture though. For instance one can perform feature selection very efficiently as the forward pass of TabPFN is very fast. Furthermore, all multiclass problems can always be reduced to binary classification problems. Finally even regression can be cast as classification as we demonstrated in the general message\nBut we share your feelings and are currently investigating training a better base model which does not suffer from these limitations. This is however a different technical contribution on top on which the present method can be applied as well.\n\n> No other deep learning or transformer based models as a baseline\n\nWe address this point in the paragraph “Deep learning model comparisons” in Section 4.2 which points to Table 5 in appendix. Essentially deep learning models are usually slower to train and we could only obtain results on a subset of the datasets. We still report their performance and ours on those. We believe the contrast is very clear.\n\n> Sec 2.4 is not quite clear in finetuning steps, though the general ideas are appreciated.\n\nThis section is indeed the most technical part of the paper and we’d be happy to answer any specific question and update the main text. In the meantime to explain differently the main point:\nLet’s say we want to use a context of size N_ctx=1000  we wish to classify N_qy = 500. With vanilla TabPFN, we simply construct a sequence of size 1500 and tell the model which points are ctx and query through the attention mask.\nHowever when using a local context, now each of the 500 queries has its own 1000 context. We now have to fit $(1000+1) * 500$ points in the GPU! For doing inference this is fine, but when having to do backprop, and that over many steps, it just becomes too slow.\nWhat we do instead is find a way to select points which are all “local” to each other so that we can use a context within that neighborhood and share it for all points, even though it might not be the exact neighbors of each query. We find that this works very well and efficiently in practice.\n\n> L32, memory scales quadratically in the size of the dataset. Do the authors mean exactly the calculation of attention matrix? If so, it seems this memory limit s general and what the authors do is to leverage the limited context length better?\n\nYes, this is correct for the TabPFN-kNN method. Note that this poses some challenges on how to implement this efficiently but your understanding is correct.\n\n> In Table 1, is the gain from loCalPFN w.r.t TabPFN-kNN coming from only finetuning per dataset?\n\nYes this is correct. \n\n> Just make sure I understand correctly, do the authors fine tune on each dataset of 95 benchmarks?\n\nYes. (see L43, L132)\n\n> How are tree methods evaluated? Any hyperparameter optimization?\n\nWe use the results from TabZilla. All baselines used HPO indeed. Please refer to Appendix A.2.1 for more details.\n\n> What is the training time and inference time on each dataset and how this time scales with dataset size?\n\nWe provided some runtime results in the attached pdf. The main takeaway is that retrieval is not the bottleneck here, it is the cost of doing the finetuning.\n\n> What is the selection criteria of 95 datasets? How are they chosen, preprocessed, etc.\n\nWe tried to address this question in the first paragraph of the experiment section “The 95 datasets are filtered from TabZilla to meet TabPFN’s architectural requirements by ensuring that each dataset has at most 100 features, at most 10 classes, does not contain NaN values, and has at least one instance per class for each split”. These are essentially conditions so that the datasets can be used without modification by the base model, TabPFN. The list of datasets is available in Appendix A.1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General message
We would like to thank the reviewers for their assessment on our work. Overall, reviewers have appreciated our evaluations and experiments (mentioning points such as “extensive” [`j47T`], “robust and comprehensive” [`iFbB`], and “well done, .. principled, … no-cherry picking, … strong ablations” [`jqEN`]). Our paper was deemed “well-organized and clearly written” [`iFbB`] and “well written and pleasant to read” [`jqEN`, paraphrased] by two reviewers. The contribution was also deemed “novel” by multiple reviewers [`j47T`, `jqEN`].

However, there were some important and relevant points raised by multiple reviewers, and so we would like to address those here and not just in the individual responses.

## Dependence on TabPFN [`iFbB`, `j47T`, `ZbyM`]

Some reviewers (along with us in the paper) accurately noted that our results currently depend heavily on TabPFN. However we point out that our method is made for in-context architectures that consider each datapoint as a token. As far as we know, TabPFN is currently the only one based on this idea, but we expect many new models of this class to follow considering the success of TabPFN. We can draw an analogy to early techniques leveraging BERT as the base model when it was one of the very few performant language models; in particular, works promoting retrieval developed with a heavy reliance on BERT readily extended to future improvements, and many remain highly relevant today. We thus feel strongly that the ideas and concepts presented here will remain relevant for future tabular foundation models.

## Regression [`j47T`, `iFbB`]

A fair point that was raised is that we do not perform regression experiments, as TabPFN was not trained for regression. We would like to address this point with two arguments:

Our method, both retrieval and fine-tuning aspects, would apply exactly the same way to a pre-trained regression model that processes inputs in a similar manner.
Following the reviews, we also have tried to perform regression with our method by simply binning the regression targets into 10 classes, thus using a regression-as-classification approach that has shown success in other domains (e.g., https://arxiv.org/pdf/2402.13425, https://arxiv.org/abs/2403.03950). However, note that TabPFN was not trained to perform this directly. To improve performance when using local contexts, we can perform local binning and predict 10 local values for each point. On the other hand, using a global context would be akin to having a global binning as well which can restrict the precision of the output. 
We validate this idea on the well known california-housing dataset (about 20k samples, 8 features) and provide results in Table 2 of the joined pdf.

TabPFN using random samples as context and binning them is not a strong regressor: a kNN regressor can do much better. However, when using a local context and binning, we can predict finer-grained values, and thus our MSE and correlation is closer to XGBRegressor. Note here that we did not even fine-tune the method.

Thus, we have already observed a generalization of the ideas presented in this paper; as discussed above, we expect further generalizations of our ideas in the realm of tabular data to be discovered.

## Runtime Details [`j47T`, `jqEN`, `ZbyM`]

While we include runtime results in the Appendix in Figure 11, admittedly this only compares our technique against tree-based approaches. In Table 1 of the joined pdf, we include a more thorough comparison with deep learning baselines on the 71 datasets of Table 5. 
TabPFN-kNN and LoCalPFN outperform the deep learning baselines by a large amount. Furthermore LoCalPFN has similar runtime to other deep learning baselines while being significantly more performant.

## Distances and Details for the Retrieval [`j47T`, `jqEN`]

We will update the main text of the paper and appendices to include more details, which we also describe here. We always **standardized the features** of the training set to build the index. Next, for the retrieval, we experimented with different metrics, namely **Euclidean (L2) distance and cosine similarity**. The main takeaway from our experiments is that these two choices are quite  similar when applied to the standardized features and so we stuck with the Euclidean distance.

Yet we were initially convinced that using *embeddings* of the datapoints would be far better than just using the standardized features for retrieval. However, we found that techniques such as using the encoder outputs or some keys/values at different layers as embeddings – with either L2 or cosine – did not increase performance. Indeed, using embeddings affected performance negatively in many cases, while additionally forcing us to recompute the index during training as the embeddings of the training set were also changing, adding computational burden.

We also tried learning the distance function (e.g., Mahalanobis distance, weighted L2, etc.). Unfortunately though this is non-differentiable and thus requires less-standard optimization techniques such as zeroth-order optimization. While we were able to discard irrelevant features in toy examples, the noise introduced by this optimization did not lead to an increase in performance on real datasets.

Furthermore, we’d like to emphasize here a difference we have with Retrieval-Augmented Generation techniques (RAGs): while in RAGs only a few documents (1 to 5) are usually selected, here we select about 1000 neighbors. This adds robustness against selecting a few irrelevant retrieved examples. Furthermore, in the worst case scenario, if we retrieve examples based on irrelevant/noisy features, our context becomes random, which is what the TabPFN baseline uses. This fact is one of the reasons why retrieval helps so much: in the best case we have large performance improvements, and in the worst case we have something similar to TabPFN.

Meta-review: The paper provides a retrieval and finetuning method that stacks on top of the current best tabular data in-context learner: TabPFN. They show consistent improvements compared to a TabPFN + k-nn baseline. The reviewers agree the experimental design is solid and the paper is thorough. The authors also provided a solid rebuttal to all the comments provided by the reviewers. There was one very negative reviewer who did not respond and gave a low score -- aside from this reviewer the average was acrtually high (5,7,6). For this reasons + having read the paper myself and it seems experimentally solid, I vote to accept this paper.
Decision: Accept (poster)