Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data
Authors: Kai Helli
David Schnurr
Noah Hollmann
Samuel Müller
Frank Hutter
Venue: NeurIPS 2024 poster (2024)
Keywords: Temporal Distribution Shifts
In-Context Learning
Bayesian Inference
Prior-Data Fitted Networks
Temporal Domain Generalization
Structural Causal Models
TabPFN
Tabular Data Modeling
Out-Of-Distribution Generalization
Domain Generalization
Meta-Learning
Concept Drift
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We present Drift-Resilient TabPFN, an approach using In-Context Learning via a Prior-Data Fitted Network, to address distribution shifts in tabular data, outperforming existing methods in terms of performance and calibration.

Abstract: While most ML models expect independent and identically distributed data, this assumption is often violated in real-world scenarios due to distribution shifts, resulting in the degradation of machine learning model performance. Until now, no tabular method has consistently outperformed classical supervised learning, which ignores these shifts. To address temporal distribution shifts, we present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network that learns the learning algorithm itself: it accepts the entire training dataset as input and makes predictions on the test set in a single forward pass. Specifically, it learns to approximate Bayesian inference on synthetic datasets drawn from a prior that specifies the model's inductive bias. This prior is based on structural causal models (SCM), which gradually shift over time. To model shifts of these causal models, we use a secondary SCM, that specifies changes in the primary model parameters. The resulting Drift-Resilient TabPFN can be applied to unseen data, runs in seconds on small to moderately sized datasets and needs no hyperparameter tuning. Comprehensive evaluations across 18 synthetic and real-world datasets demonstrate large performance improvements over a wide range of baselines, such as XGB, CatBoost, TabPFN, and applicable methods featured in the Wild-Time benchmark. Compared to the strongest baselines, it improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration. This approach could serve as significant groundwork for further research on out-of-distribution prediction.

Drift- TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data
Kai Helli David Schnurr". Noah Hollmann 1.4 Samuel Müller!
Frank Hutter
University of Freiburg, Technical University of Munich, ETH Zurich, Charité University Medicine Berlin, 5 ELLIS Institute Tübingen, Equal contribution. Correspondence bei.hellle eellittummd.
Abstract
While most ML models expect independent and identically distributed data, this assumption is often violated in real-world scenarios due to distribution shifts, resulting the degradation of machine learning model performance. Until now, no tabular method has consistently outperformed classical supervised learning, which ignores these shifts. To address temporal distribution shifts, we present Drift-Resilient TaPPFN, fresh approach based on In-Context Learning with a Prior-Data Fitted Network that learns the learning algorithm itself: it accepts the entire training dataset as input and makes predictions the test set in single forward pass. Specifically, it learns to approximate Bayesian inference on synthetic datasets drawn from prior that specifies the model's 's inductive bias. This prior is based on structural causal models (SCM), which gradually shift over time. To model shifts of these causal models, we secondary SCM, that specifies changes in the primary model parameters. The resulting wrft-Reesilien TabPFN can be applied to unseen data, runs in seconds small to moderately sized datasets and needs no hyperparameter tuning. Comprehensive evaluations across 18 synthetic and real- world datasets demonstrate large performance improvements over wide range of baselines, such as XGB, CatBoost, TabPFN, and applicable methods featured in the Wild-Time benchmark. Compared to the strongest baselines, improves accuracy from 0.688 to 0. 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration. This approach could serve as significant groundwork for further research on out- -iittriibittttioonnnn prediction.
Introduction
In traditional machine learning the train and test data are assumed to be sampled from the same distribution [1]. However, this assumption of independent and identically distributed (i.i.d)) data commonly violated real-world scenarios due to distribution shifts, resulting in performance degradation of standard machine learning (ML) models
time [1, 2]. Research in the area of temporal domain generalization (Temporal DG) tries to address these shifts by developing methods that perform consistently across temporal domains and generalize beyond the training regimen, i.e. into the future. In fields such as healthcare. climate science, or finance, data is most often organized in tabular format [3, 4]. Here shifts driven by hidden variables such policy climate changes, equipment updates, seasonal changes, or activity cycles, limiting real-world model deployment [2].
Young and Stecle [5] describe declining mortality quantification in hospital system while Pasterkamp et al. [6] show deterioration of cardiovascular risk models over time, leading to increased mortality; Ganesan et al. [7] show the COVID-19 pandemic exacerbated this issue, ICU mortality prediction models failed to adapt to the unique characteristics COVID-19 patients; environmental models need continually adapt to climate changes [8]; fraud detection models need to continuously adapt
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HIGH-LEVEL OVERVIEW
(a) We generate synthetic datasets by sampling structural causal models whose edges shift over time.
A structural causal model (SCM) with sparse A 2nddrr SCM generates the strength of edge shifts indicated in red.
edge shifts depending on time domain.
RESK
(b) TabPFN accepts entire datasets as inputs. Given millions of datasets from (a), it learns make predictions on held-out test samples, learning the prediction algorithm itself with an inductive bias for addressing distribution shifts.
(c) Trained once, Drift-Resille TabPFN can be applied to novel real-world data and makes pre- dictions in single forward pass, automatically detecting and extrapolating distribution shifts.
Train Domains:
Tast Domain
Train Domains:
Tast Domain
Xirain
Gram Merwen
TabPEN dui Alceando and orlpest
Xise
Rosat
Yrest
CrossEbtzopy Stont)
Figure I: High-level overview of our method. We train a transformer that accepts entire datasets as input to learn the learning algorithm itself by training on millions of synthetic datasets once as part of algorithm development. The trained model can be applied to arbitrary real-worke datasets. In (b), X, and y refer to features, time domain, and label respectively. In (c), we show predictions on test domains 4 (left) and 5 (right), where we see a distribution shift. Drift-Resilient TabPFN accurately updates decision boundaries in this example.
as the strategies of fraudsters adapt to the models themselves [9]; these feedback loops often arise in practice, where model deployment inherently causes a system change [10].
Robustness to such distribution shifts stands as prominent challenge in current ML research [11]. So far multiple approaches have been proposed to address temporal distribution shifts using neural networks (NNs) [11-13]. However, modeling distribution shifts in tabular data presents a two-fold challenge: (i) NNs have struggled to model and extrapolate distribution shifts to date [11, 14] (ii) approaches for modeling distribution shifts have mostly employed NNs, while tree-based methods have consistently outperformed NNs handling tabular data [3. 15-17] leaving wide methodological gap in addressing this common real-workd scenario.
We provide a fresh perspective on predicting given distribution shifts by leveraging in-context-learning (ICL) to learn the prediction algorithm itself bypassing many challenges encountered in this setting, Our approach builds on the foundation of Prior- Data Fitted Networks (PFNs; 18) and TabPFN (19; see Section 3. 1). PFNs leverage targe-scale ML and ICL techniques to approximate Bayesian inference accurately for any prior that can be sampled from. They are trained on millions of synthetic datasets sampled from this prior. For each such dataset, supervised learning task is constructed, and the model is asked to predict on held out test samples. Then, the PFN is able to apply the principles learned on this synthetic data to real- world datasets, effectively having learned a prediction algorithm.
This paper introduces Drift-Resilient TabPFN, an adaptation of the TabPFN framework tailored for tabular datasets exhibiting temporal distribution shifts. Our idea is as follows: Data distribution shifts can be modeled gradual changes the structural causal model (SCM; 20, 21) underlying the data, By including the assumption that underlying models change over time into the approximated prior, our models learn to estimate, adapt to, and extrapolate these model changes. While likely no causal model exactly underlies real-world data, we find this approximation to be empirically and intuitively useful. Specifically, we suggest shifting edge weights between nodes that affect causal relationships while sampling instances in each dataset. To accurately model these edge shifts, we propose using 2nd order SCM, a secondary model that adjusts the primary SCM, that takes time indicator as input and outputs the weight shifts. To represent our temporal domain indicators, we use Time2Vec [22].

---

To validate the robustness and adaptability of our approach, we conduct a comprehensive evaluation of 18 tabular datasets. Among these, 8 are synthetic toy problems, that allow us to analyze model behavior in detail. The remaining 10 are real-world datasets, each representative of different temporal distribution shifts. Our model benchmarked against (i) state-of-the-an methods for handling temporal domain generalization (implemented in the Wild-Time benchmark [11]) as well as (ii) state- of-the-art methods for tabular data, that do not handle distribution shifts explicitly (the unmodified version of TabPFN, well-established classifiers such as XGBoost, Catboost, and LightGBM [23-25]). In addition, we qualitatively analyse the model's 's behavior and prediction characteristics.
Our model consistently outperforms all baselines on out- of--isiiiibuttionn data across both synthetic and real-world datasets while simultaneously exhibiting strong calibration. Qualitatively, we find that Drift- Pritt-Resilient TabPFN dynamically adjusts its decision boundary over time, suraassing existing models leveraging temporal domain information to extrapolate shifts far into the future and capture trends effectively.
Key contributions of this paper are:
1. We provide a novel perspective to learning under distribution shifts: Rather than specifying how exactly out-of-distribution data should be handled, we leverage in-context learning to learn an algorithm that handles such scenarios.
2. We propose a prior for temporal shifts, based on sparse, non-linear, and correlated mechanism shifts in structural causal models. This unifies shift types (covariate shift, prior probability shift, concept shift, and combinations thereof) and allows to extrapolate these changes.
3. We release distribution shifts, that automatically recognizes and adapts to shifts in the data and requires no hyperparameter tuning.
TabPFN, a tabular data model for predicting under temporal
4. We extensively evaluate our models and state-of-the-an baselines on 18 synthetic and real- world datasets demonstrating improved out-of-distribution performance while requiring, on average, only 10.9s for training and prediction combined.
2 Background
2.1 Distribution Shift Settings
Consider X, V/ and C, the sample spaces for features, labels, and domain indices, respectively. In these spaces, specific instances are represented by æ. y, and c. The corresponding random variables for these instances are denoted as X, Y, and C.
A dataset is then collection of tuples, denoted D = {(Z,, Vr, Gx)] Each tuple is thereby drawn from a conditional distribution IP(X, Y C c2) over the spaces X V. Here, C2 E serves domain index that conditions the distribution from which the respective sample is drawn. Given that the true underlying temporal domain is mostly unknown in real-world data, it often is approximated and represented é₂ within the dataset. To isolate samples from specific domain &x we introduce the sub-dataset De defined as Da, == {(I, y, é) E Diê= = ) This adapted notation allows for more nuanced analysis of datasets with domain shifts, building upon the frameworks presented by Wang et al. [26] and Sheth et al. [27].
Domain Generalization (DG). DG aims to train a model that generalizes across continuum of source domains Cirain to the target domains Crest without access to samples of the target domains. The objective is to learn mapping function f X x C + y that minimizes the expected loss when applicd to new, previously unseen target domains.
Temporal Domain Generalization (Temporal DG). In temporal DG, special case of DG, the domain index set C is one- me--imensional and follows total ordering, 01 S C2 S ... [12]. In this framework, the training set is limited to souree domains that precede target domains in this ordering, namely Ctrain {ci, C2, c,,,.....
).). ..). In this setting, the objective to learn a predictive model ʃ that performs well on these source domains while also robustly generalizing to future, unseen domains
Cleat (ce+1, C..2.....c0 ). The indices of all training domains Cuain and the index of the current
testing domain Cj. E Crest are provided to the model.
3

---

2.2
Related Work
While DG has drawn increasing attention in the research community [28-38], its temporal variant remains under-explored. We review existing DG benchmarks on tabular data, DG methods, the temporal DG benchmark Wild-Time [11], temporal DG methods, as well as relevant TabPFN studies.
Tabular Distribution Shift Benchmarks. Several benchmarks have been introduced to assess methods for tabular data under distribution shifts. (1) Shifts and Shifts 2.0 [39, 40] are uncertainty focused benchmarks. Shifts 2.0 includes five tasks, two of which involve tabular data subject to temporal and spatio-temporal shifts. (2) WhyShift [41], focusing on spatio-temporal shifts, offers five real-world tabular datasets, including the ACS dataset [42]. which we also use in subsampled form. Evaluating 22 methods like Gradient Boosted Decision Trees (GBDTs), MLPs, and robustness techniques, WhyShift finds that robustness methods do not consistently enhance outoof-distribution (OOD) performance. They also find that while GBDTs perform better, the gap between in in-distribution (ID) and OOD performance persists, likely due to GBDTs being better fitted to the ID distribution. (3) TableShift [14] includes 15 tabular binary classification tasks, with 10 relevant to DG. Their work evaluates 19 model types, including several DG methods, finding that methods tailored for distribution shifts do not consistently outperform GBDTs. While generalization gaps can be slightly reduced, each robustness method comes at the cost of ID performance. (4) Wild-Tab [43] focuses on domain generalization within tabular regression using three large datasets. Their study compares 10 generalization techniques against standard Empirical Risk Minimization (ERM) applied to MLPs. Similar to previous work [44], they find that ERM was not consistently outperformed by specialized DG methods. Notably, no advantage of GBDTs over ERM on MLPs was observed in their datasets.
Unlike these benchmarks, which focus on large-scale datasets, our work addresses the overlooked challenges of small-scale temporal distribution shift datasets. However, their insights on generaliza- tion and robustness methods align with our findings, providing valuable context for our approach.
DG Methods. Several techniques have been proposed to improve robustness to distribution shifts in DG [45]. Key approaches include domain-invariant learning methods, such as Deep CORAL [35], IRM [31], and DANN [36], which aim learn representations that generalize across domains. Data augmentation strategies, like Mixup [33] and LISA [34), contribute to generalization by generating synthetic data variations. Additionally, robust optimization techniques, including VRex [37]. GroupDRO [32], EQRM [38], and SWA [46], aim to improve performance under distributional shifts by optimizing for worst-case scenarios or incorporating model uncertainty.
In relation to our work, data augmentation strategies are conceptually similar, as we also teach the model DG rather than adding invariances to the architecture. In contrast augmentation, though, learn to generalize using completely artificial data on the meta level rather than through manipulation of the target dataset. The other methods focus on designing models specifically for DG, while our approach completely relies on the model to learn to handle distribution shifts. While similar to continual meta-learning (CML) approaches, our method uses domain identifiers to generalize to unseen domains, whereas CML continuously adapts to evolving contexts without identifiers [47].
Wild-Time Benchmark. Wild-Time [11] is a benchmark of five datasets designed to study the real-world effects of temporal distribution shifts - an area largely overlooked by previous benchmarks. While Wild-Time primarily uses non-tabular data, it evaluates wide array of techniques its tabular dataset, including classical supervised learning (ERM), fine-tuning, and several previously mentioned general DG methods adapted to temporal distribution shifts. Despite the diversity of methods evaluated, Wild-Time reveals a significant performance gap between ID and OOD data, with none of the 13 tested methods consistently outperforming the standard ERM approach.
In our evaluation, we employ their evaluation strategy Eval- Fix and also benchmark our approach against the methods they considered. An in-depth overview of these methods is given in Appendix A.6.
Temporal DG methods. Recent specialized temporal DG methods include: (1) DRAIN [12), which employs a Bayesian framework alongside a recurrent neural network for predicting the dynamics of the model parameters across temporal domains. (2) GI [13], which explicitly incorporates the temporal domain as feature, using specialized Gradient Interpolation loss function, time-sensitive activation, and enhanced domain reasoning via Time2Vec preprocessing [22]. However, both DRAIN and GI are limited in their ability to extrapolate beyond the near future, leading to their exclusion from our main evaluation. Notably, on the Rotated Two Moons Dataset- - where ARAIN and GI have demonstrated their capabilities our approach outperforms both. See Appendix A.4.3.4 for details.

---

Recent TabPFN Studies. To overcome the limitations of TabPFN in handling large samples and feature sets, typically found in DG benchmarks, several improvements have been proposed. Ma et al. [48] introduced a data distillation approach, where distilled dataset serves as the model's i context, optimized to maximize the training-data likelihood. Similarly, TuneTTables [49], uses prompt-tuning to compress large datasets into smaller context. Either of these methods could potentially be combined with Drift-Resilice TabPEN, as our modifications focus primarily on the pre-training phase, which remains unchanged in their approaches.
Another TabPFN variation, ForecastPFN [50], also introduces time dependence, but it does not consider any features. It only models simple time series data. Unlike our approach, which builds on TabPFN's SCM architecture for pre-training to handle large set of features, ForecastPFN models synthetic time series using a single handcrafted function with sampled hyperparameters, simplifying the architecture while trading off diversity of the synthetic datasets.
3 Methodology
Our approach is built on PFNs, which use ICL to learn the learning algorithm itself. This approach also has a theoretical foundation as described by Müller al. [18]: can be viewed as approximating Bayesian prediction for prior defined by the synthetic datasets. The trained PFN will approximate the posterior predictive distribution (PPD) and thus return Bayesian prediction for the specified distribution over artificial datasets used during PFN training.
3.1 Structural Causal Model Prior
Hollmann al. [19] introduce prior based on Structural Causal Models (SCMs; 20; 21) to model complex feature dependencies and potential causal mechanisms underlying tabular data. To sample one dataset, this prior samples an SCM which is then used to sample the examples in the dataset. In this approach, each causal representation of a sampled SCM is converted into a functional representation to enable forward computation and dataset sampling.
Consider a sampled SCM graph G = (Z, R), where Z= (21.... _n} are the nodes (mechanisms) and R {ri, .mm the edges (relationships). Each node is set using an individual assignment function ft. 2 = fi((z) j E PA:), es) with PA; being the parent nodes of z; and 6, random noise.
In the following, we will detail the underlying functional graph representation 9 of an SCM g sampled from the prior. We will later on apply distribution shifts on it. This representation is one level below the SCM and all nodes represent a single scalar value and all edges correspond to linear connections. The values of each node e in 9 are set just like neurons in a neural network as a simple weighted and activation function := hú (E (Ereps, Wi tz), where the scalar weights Wi.j and the activation function hy are randomly chosen at graph creation and the noise term e is sampled in each forward propagation to create a variety of samples within a dataset.
The functional graph representation & of an SCM G is defined as follows:
1. Node Expansion. Each node 2; is expanded into a set of subnodes Z; == {2) 2 each representing a scalar value.
2. Graph Expansion. The following steps are performed for each node 2j with a non-empty parent set PAy:
(a) A new set of subnodes F, = (f...... F.. is added.
(b) We add the edges E, == {Z, PA2} UF, x Zy.
Finally, the functional graph representation is defined as 9 = (UiEZ Z, UUJEZ SUUJEIFF,ULS U,,UIEI E.).
The features of our dataset are defined by a random subset X S Uiez Z, and the target is defined by a randomly chosen y E Ulez Z, X. Samples in a dataset are now generated by (i) sampling all noise terms Ej in G. (ii) propagating the values of all nodes through the graph, and finally (iii) retrieving values at the feature and target nodes. The resulting datasets, thus abide the computational flow of both G and G. For a more intuitive understanding of this construction, refer to the simplified example provided in Figure 2.
5

---

12
Tb
(a) Sampled SCM
(b) Functional representation
Figure 2: Illustrative transformation of SCM exemplary functional representation. Shaded nodes indicate that their activations cannot be sampled. Feature nodes are blue, the target node is green, input/noise nodes are purple, and all others gray. The figure also shows the mapping of shifted edges between a causal relationship and its functional form in red, ensuring that shifts specifically target the intended causal relationships without affecting others.
3.2 Inducing Temporal Robustness in SCMs
We extend TabPFN's prior to model distribution shifts, allowing the model to expand its posterior predictive distribution (PPD) calculations to incorporate temporal domain information. We propose modeling the dynamic of edge shifts using a 2"d-order SCM. This 2nd -odeer SCM is itself an SCM with feature nodes specifying the magnitude of edge shifts in the base SCM's functional graph.
Specifically, to extend the scope of the functional representation of the SCM G, we introduce version Go,, depending on the temporal domain Cj E C. Our objective is to construct a time- dependent dataset D which an aggregation of individual datasets De, Dey...... D.... where each dataset De. = {(rr Vo: Cg) ]== contains nee, samples. To do this, we sample the temporal domains C= C= c ... Cn and for each domain Ck. sample the number of samples nee it contains, An illustration exemplary domain distributions across datasets is shown Figure 14.
We then select sparse subset of relationships in the causal representation of the SCM G to undergo temporal shifts based on the evidence that sparse shifts allow for causal reasoning [1]. For these selected edges, the 2nd -order SCM H. is used to sample shift parameters that govern the corresponding edges in the functional representation G.g. See Algorithm for high-level overview.
Employing a 2nd 2nd-order SCM for Edge Shifting. The 2dd-order SCM H takes temporal domains as input and, through single forward pass on corresponding functional representation H. produces dynamic edge shifts for each edge weight Wi, in 9 that corresponds to an edge in g that should be shifted. Note that although we perform a forward pass, there is backward pass associated with it. Each 2nd-order SCM randomly generated and used solely for sampling the weight shifts. This design allows for Bayesian reasoning over the edge shifts and enables H to generate complex, often correlated shifts over time. While we emphasize this construction does not reflect the true underlying dynamics edge shifts in many real-world datasets, we use this heuristic prior construction for the following reasons: (a) Features in the SCM are correlated each other in varying degrees, mimicking real-world processes of correlated changes in the underlying generating mechanisms of weal-world data (b) An SCM with NN based causal mechanisms and nonlinear activation functions can extrapolate values outside of the data distribution, generalizing the underlying function future domains (c) As demonstrated by TabPFN, causal model prior is sufficiently general approximation to many real-world datasets. We have visualized this approach in Figure and selection of the functions generated by 2od-erder SCM in Figure 15.
Shifting SCM edges can model various Types of Distribution Shifts. A distribution shift characterized by changing distributions between contexts (P(X,) Y C c.) * PPX, C' cx)). This definition can be further broken down into the following scenarios:
Covariate Shift: Changes in the feature distribution (PP(X) while the conditional label distribution (P(Y)X)) remains constant.
Prior Probability Shift: Changes in the label distribution (P(Y)) while the feature distribution given the labels (IP (PP()Y)) remains constant.
Concept Shift: Changes in the relationship between features and labels (P(()X) or P(X(Y) while the marginal distributions of features or labels remain constant.

---

-arler SCM
ulau. E7)
Figure 3: Diagram illustrating the integration 2nd SCM for adaptive edge shifting across evolving temporal domains. On the right, the primary network 9 generates data samples over multiple time domains, with red arrows indicating shifted edges. On the left, the 2 2dd -odeer SCM an auxiliary network H. takes an input domain C and outputs parameters to adaptively shift each edge weight we in the base network.
X
x
X
(a) Shift
Covariate
(b) Prior Probabil-
(c) Concept Shift
(d) Concept Shift (Y X)
(e) Other Types of Shift
ity Shift
(X - Y)
Figure 4: Types of distribution shifts based on the definitions by Moreno-Toon -orres et al. [51] represented Bayesian networks defined by Kull and Flach [52]. Here X, Y, and C' denote the random variables of the features, label, and context, respectively. Note that all these types of shifts naturally arise in prior, since sample feature and target positions, well as the locations of shifted edges, randomly at various positions the synthetic datasets.
These shifts can also be viewed as Bayesian networks shown in Figure 4 all arising in our prior by sampling features and targets varying SCM positions. For further visualizations, refer to Figure 13.
Encoding the Temporal Domain. When encoded as inputs to our model, the cemporally-dependen datasets D Uerec Dr,, partitioned at particular instance into Durin and Diest
Temporal domains and features are normalized using only the training data, and so, to effectively encode temporal information and provide normalized inputs when projecting into the future, we use Time2Vec [22]. converts each temporal domain index into an m-dimensional vector, using lincar and sinusoidal functions characterized by learned parameters wi and po, Specifically, the Time2Vec Vec transformation for temporal index Cą E C is formulated as:
[wick Pps
if 0. if 1 S i m.
(2v(c))[
sin(wick + Pi),
4 Experiments
Evaluation Strategy. We evaluate analogous to the Eval-Fix setting outlined Wild-Time [11], measuring both, ID and OOD performance. Here, each dataset D is split into three subsets: Dirain D'D, and DOOD Splits are based on randomly sampled temporal domain C that serves as the boundary between the train and test (OOD) portion. We only use such splits, where Dirain comprises between 30% and 80% of the total domains and samples. To assess ID performance, we subsample 10% of the instances in each domain of Dirain as the ID test set and the remainder as the train set. An illustration the Sval-Fix strategy is provided in Figure 16.
Each class in the training set is required to be represented in both DJD and DOOD and vice versa. For all datasets, we generate three random splits and average metrics across these splits. We had limit the number of splits to three due the constrained number of available domains and the requirement for classes to present both the train and test splits. Each method trained three times, and we report the average and 95 95%-confidence intervals calculated across model initializations.

---

Metrics. We evaluate Accuracy, F1-Score (Harmonic mean of precision and recall, useful in imbalanced datasets). ROC AUC (Area under the receiver operating characteristic curve), and ECE (Expected Calibration Error; reflects the reliability of the model's 's probability outputs).
Datasets. Our benchmark comprises 18 test datasets, 8 synthetic and 10 real-world. In addition, 12 validation datasets, 4 synthetic and the remaining 8 real-world, were used to optimize the hyper- parameters of our approach via random search. While some of these datasets have been analyzed in previous work, there has been no comprehensive benchmark focusing on small tabular datasets undergoing distribution shifts. To address this gap, we carefully selected and generated diverse range of datasets that exhibit temporal distribution shifts. The selected datasets originate from open dataset platforms or previous work in DG. Ground truth domain indices cj E are known for synthetic datasets. For real-world datasets, we approximated domain indices Gg based on features that encode temporal information, which we transformed into discrete intervals. Also, some real-world datasets required subsampling due to their large size, which was beyond the current architecture of TabPFN. We provide full details, including descriptions for each dataset and pre-processing steps in Section A.7 of the Appendix.
Baseline Setup. Our baselines include state-of-the-art methods for tabular prediction. These include advanced GBDTs like CatBoost [24], XGBoost [23], and LightGBM [25], which have demonstrated superior performance to standard neural network approaches in handling tabular data [17]. We also include TabPFN in its unmodified form (TabPFNease 19). Methods from the Wild- Time benchmark are examined separately and detailed in Section A.4.4.3 of the Appendix. However, we have added the two rest-performing Wild-Time methods, ERM and Stochastic Weight Averaging (SWA), to the main results table. All baseline methods besides TabPFN are subject to a time budget of 1, 1,200 seconds on 8 CPUs and GPU. For each method except TabPFN, which does not require tuning, a random hyperparameter search with 3-fold time series gross-validatis was used. We chose the best-performing hyperparameters based on OOD ROC AUC within the allocated time.
Among our baselines, we considered three strategies:
1. Providing the full dataset Dirain along with the corresponding domain indices Curain as a feature, aiming allow for better reasoning of the shifts in the dataset (all dom. w. ind.).
2. Using the dataset without domain indices Dirain {<>!tain " y/ (all dom. wo. ind.).
3. Limiting the training set to samples from the last training domain q. In this setting, we also omit the corresponding domain indices, resulting in the set Dusin = {((xttiin 3 w } (last dom. wo. ind.). The rationale behind the last scenario provide only training data closest to the subsequent test distribution. This strategy is not used for distribution shift baselines.
TabPFN Setup. For the TabPFN variants, both the original and our modified method (TabPFNais) were pre-trained for 30 epochs across 8 GPUs. This results in a total of 30,720,00 synthetically generated datasets processed during pre-training While this pre- pre-training step is moderately expensive. it is done offline, in advance, and only once as part of our algorithm development. Furthermore, the preprocessing parameters of both methods were optimized once on the validation datasets by random search over 300 configurations. We chose the configurations that yielded the best OOD ROC AUC performance. The resulting model and hyperparameters are used for all datasets, resulting in, on average, 110 times faster training and prediction time on our benchmark.
Quantitative Evaluation. Our method demonstrates superior predictive performance across all metrics for OOD data in 18 test datasets, for synthetic datasets and real-world datasets, as detailed in Table 1. Compared to the strongest baseline, our method improves accuracy from 0.665 to 0.754 on synthetic datasets and from 0.712 to 0.736 on real-world datasets. It also enhances the F1 score from 0.588 to 0.697 on synthetic datasets and from 0.668 to 0.682 on real-world datasets. Additionally, increases the ROC AUC from 0.749 to 0.844 on synthetic datasets and from 0.82 to 0.822 on real- world datasets. Furthermore, our method shows much stronger calibration on OOD samples, reducing ECE from 0.164 to 0.126 on synthetic datasets and from 0.083 to 0.062 on real-world datasets. While baselines are often overconfident on OOD data, our method is able to predict uncertainty accurately. Compared to TabPFN and GBDTs, the NN-based methods in the Wild-Time Benchmark (Section A.4.4.3) show a substantial drop in performance, likely due to the limited training data. Since our method focuses on enhancing OOD robustness rather than optimizing ID) tasks, we find lower predictive performance on ID tasks. While performance gains are observed on both, real-world and synthetic data, observe stronger improvements on synthetic datasets. This can be partly attributed
8

---

to the, on average, stronger distribution shifts between ID and OOD data in our synthetic benchmark. Furthermore, real-world datasets often show multifaceted and complex shifts that are much more difficult to extrapolate into the future. Combined results across all datasets are provided in Table 5.
Table I: Comparison of Drift-Resilient TabPFN with various baselines and settings across the subsets of synthetic and real-world datasets. Metrics include accuracy, F1, ROC, and ECE for both in- distribution (ID) and dus-o--distriuutiin (OOD) data, averaged over three initializations and reported with 95% confidence intervals. The best mean of each metric within dataset subset is marked in bold. Metric arrows indicate optimization direction.
Model
Variant
Acc. 1
F1 1
ROC1
ECE
OOD
ID
OOD
0OD
OOD
ID
Tab
all dom. w. ind.
0.754
.033 0.959
0.697
0.935
.03
0.844
.03
0.987
.002
0.126
0.038
all dom. w. ind. 0.658 .1s 0.963
.006
0.567
.015
0.935
0.749
.017
0.986 .07
0.164
014
0.029 003
TabPFN
all dom. wo, ind. 0.571 .014 0.901
.oo8
0.467
.01e
0.848
.008
01631
0.955
.003
0.322
.018 0.053 .o06
last dom. wo, ind. 0.651
.002
.016 0.939
0.374 .022
0.918
.031
0.727
.006 0.975
.001
0.27
,011
0.066 .001
SYNTHETIC
all dom, w. ind. 0.605 .oos
0.958 .oo8
0.588
.008
0.94
.009
0.73
0.989
.002
0.297 .o08
0.037 .on6
CatBoost
all dom. wo. ind. 0.575
.005
0.885
.003
0.476
.008
0.831
.002
0.613
.018
0.942
.007
0.325
ooo
0.063
.014
last dom. ind. 0.639
.004
0.932
.017
0.564
.005
0.916
.021
0.684
.005
0.962
.012
0.301
018
0.065 .006
all dom. w. ind. 0.645 .188
0.936 .012
0.57
.019
0.931
.005
0.705
.013 0.968 .02
0.253 032
0.075 .013
XGBoost
all dom, wo. ind.
0,582 .z
0.872
0.48
.074
0.818
.038
0.621
0.926
.o35
0.245 oss
0.097 .034
lust dom, wo, ind. 0.645 .0o8
0.916
0.565
.009
0.88
0.688
.017
0.956
.012
0,256
.018
0.111
.on8
all dom, w. ind. 0.646
0.943
0.57
.015
0.927
.015
0.687
.013
0.982 .002
0.281 oos
0.056
LightGBM all dom, wo, ind.
0.581 .01
0.884 .0o6
0.482
.007
0.829
.006
0.617 .016
0.935
.001
0.273
0.069 .017
last dom. wa, ind. 0.629
.004
0.917 .003
0.553 .oos
0.892
.018
0.662
.005
0.958
.007
0.288
012
0.077 .007
Wild-Time
ERM
all dom. w. ind. 0.648 .046
all dom. wo. ind. 0.576 0z1
last dom. wo, ind. 0.632 .0os
0.945 .017
0.921 .017
0.885 .04
0.487 .05
0.566 .007
0.489
.0o2
0.837
0.908
0.9
.035
.048
.026
0,621 .03
0,688
0.65
.042
0.962
0.943
0.973
.005
.015
.o21
0,282
0,282
0.304
018 0.091 .016
oos 0.041 .on6
012
0.058 .024
Wild-Time
all dom. w. ind. 0.636 .os
0.923 .054
0.489
.088
0.877 .08
0.651
.0a5
0.958 .oa
0.313
.048
0.054 .015
SWA
all dom. wo. ind,
0.573 0.887
0.48
.042
0.837
.017
0.631
.048
0.913 .012
0.3
.865
0.059 .002
TabPFNas
all dom. w. ind. 0.736 .0o7
0.814 .014
0.682
.012 0.759
.016
0.822 .01
0.887
.006
0.062
.004 0.103 .026
all dom. w. ind. 0.712 .012
0.822
.013
0.661
.022
0.777
.018
0.816
.oo6
0.894
.013
0.083
001 0,097 .007
TabPEN
all dom. wo. ind.
0.701 .o1
0.813
.023
0.668
.014
0.76-1
.oa
0.82
.006
0.882
.011
0.106
019
0.095 .011
last dom. wo. ind. 0.685
0.809
0.637
.005
0.746
.036
0.787
.oo8
0.867
.oa6
0.109
oos
0.177
.012
REAL- WEL-WWRLD
all dom, w. ind. 0.687 .003
0.807 .012
0.646
.008
0.753
.017
0.796
.004
.010 0.862
0,161
o166
0.122 .010
CatBoost
all dom. wo. ind. 0.677
U.797
0.6-12
0.741
.015
0.794
.013
0.856
0.172
0.125
last dom, wo. ind. 0.671
.002
0.788
.023
0.627
.004
0.728
.os
0.752
.007
.022 0.863
0.221
.017
0.188 .023
all dom, w. ind. 0.68 0.797
0.612
0.715
.oo3
0.793
0.845
0.147
0.141
XGBoost
all dom, wo. ind.
0.674
.255
0.798
.019
0.639
.004
0.746
.026
0.795
.012
0.845
.024
0.153
.228
0.138 .049
last dom, ind. 0.679
.014
0.75
.043
0.626
.084
0.66
1100
0.769
.008
0.833
.032
0.154
0.212 .021
all dom, w. ind. 0.654
.015
0.761
.042
0.614 .o1
0.698
.0s1
0.778
.on8 0.848
.o18
0.133
.017
0.123 .02
LightGBM
all dom, wo. ind, 0.66
.029
0.79
.031
0.624
.028
0.728
.22
0.778 .003 0.849
.018
0.14
.008
0.12 .018
last dom. wo, ind. 0.629
0.701
0.533
.oos
0.582
.ose
0.706
0.768
0.172
.114
0.206
Wild-Time
ERM
all dom, ind. 0.61 .0or
all dom, wo. ind. 0.587 .oas
last dom. wo. ind. 0.551
.o1
0.721 .006
0.737
0.674 .045
0.3-15
0.353
0.498
.o2s
.046
.677
0.664
0.673
.03 0.612
.013
.058
0.714 .012
0.719
0.648
.017 0.8
.039
0.798
0.749
.024
.042
.057
0,233
om 0.355
0.23
.07
043
0.267 .034
0.161 .o35
0.14 .008
Wild-Time
all dom, w. ind. 0.62 .049
0.745 .oas
0.57
.055
0.697
.046
0.712
.039
0.805
.007
0.242 ese
0.155 .021
SWA
all dom, wo. ind. 0.6
.019
0.733 0017
0.57
.019
0.677
.001
0.721
.029
0.798
.008
0.232
021
0.15
.017
Qualitative Analysis. Next, we take an in-depth look at the predictions made by our method. Figure 5 illustrates the decision boundaries of our method and TabPFN on the synthetic Intersecting Blobs dataset. In this evaluation, we restrict the training domains to euain = {0. 1, 2, 3) and aim to predict samples in test domains Crest = {4, 5, 6} without adding additional data to the training set. This setup requires the model to extrapolate the temporal shifts into the future based solely on existing training data. In this setting, our model accurately extrapolates decision boundaries to future domains, while TabPFNhase base tends to retain its initial boundary. Our analysis reveals two key attributes of our model: (i) The model decreases prediction certainty over time, improving calibration. (ii) Our model adjusts the decision boundary dynamically. boosting accuracy. Further visualizations, including decision boundaries for the Rotated Two Moons dataset, are available in Figure 7. Likewise. plots illustrating the overall shifts in these datasets are provided in Figure 6.
9

---

Train Domains: 0-3
Test Domain
Train Domains: 0-3
Test Domain
Train Domains: 0-3
Test Domain 6
Tab
1.0
TabPFNoak
-22
XL
Figure 5: This figure displays the predictive behavior of TabPFNais in the top row and TabPFN in the bottom row on the Intersecting Blobs dataset. It illustrates how each model adapts to unseen test domains when trained on domains Cuain = {0, 1, 2. 3). The baseline is given the domain indices as feature in train and test. The coloring indicates the probability of the most likely class at each point. Incorrectly classified samples are highlighted in red.
Impact of Time2Vec Preprocessing on Model Performance. To examine Time2Vec's contribu- tions improved OOD performance, conduct ablation study in Appendix A.4.1. The ablation reveals that while Time2Ver Vec provides at most slight improvements, the substantial performance gains be attributed the prior construction used during the model's pre-training phase.
Conclusions & Limitations
In this work, we presented a Bayesian approach to address the issue of temporal domain general- ization in tabular data. Specifically, we focused on enhancing TabPFN to improve its robustness to temporal distribution shifts. Within this framework, we introduced novel approach that changes the causal relationships in the SCM prior over time, thereby enabling TabPFN to inherently adapt to these shifts. Our method outperforms all baselines on the evaluated datasets and demonstrates notable improvements both qualitatively and quantitatively, particularly on synthetic OOD datasets. Furthermore, it requires no hyperparameter tuning, is not limited to particular types of distribution shifts and takes only 10.9s for training and prediction combined.
Despite these advancements, our methodology inherits certain limitations from the underlying TabPEN model. (1) Due the quadratic scaling of the attention mechanism with respect to the number of samples, our method does not scale to large datasets. Here, our research will benefit from the continued improvements of TabPFN, ICL, and sequence- models general. (2) The TabPFN, like many mansformer-basee models, acts as a "black box", making it challenging to interpret the model's predictions and understand the recognized distribution shifts. (3) The underlying prior for structural causal models with sparse mechanism shifts may not accurately describe all real-world datasets. While we find to be empirically and intuitively useful, real-world shifts might have underlying complexities that our prior currently does not capture.
For future work, next addressing the existing limitations, we have identified in initial experiments promise in extending our model to (1) transductive and (2) online continual learning settings. Also, our prior could adapted to support (3) spatial or apatio-tempors distribution shifts. Employing prior that (4) models shifts in the underlying causal model could improve the robustness of the baseline TabPEN standard classification tasks where temporal shifts are often implicit.
We provide code, pre-trained models, and a Colab notebook at https: //gi i thub. com/ automl/ Drift-Rest Hei-RReiiieeee linnt, TTbPFN. We further describe reproducibility, the release of code and models, the broader impact of our models, and the computational resources used for method development in Appendix A. A.2 and A.3. We add discussion of baselines in A.6, in-detail discussion of the evaluated datasets in A.7, and additional quantitative and qualitative evaluations in A.4.
10

---

6 Acknowledgments
Frank Hutter acknowledges the financial support of the Hector Foundation. This research was funded by the Deutsche Fosschugsseemeinschfft (DFG, German Research Foundation) under grant number 417962828.
We acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Funded by the European Union
References
[1] Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Proceedings of the 36th International Con- ference Advances in Neural Information Processing Systems (NeurlPS'22) 22), 2022. URL https:/ http::://oppneevieee wew. post/forue www.orrummdddeeeeeeeeee kFFiiddbDRCCpp
[2] Daniel Vela, Andrew Sharp, Richard Zhang, Trang Nguyen, An Hoang, and Oleg S. Pianykh. Temporal quality degradation AI models. Scientific Reports, 12(1):11654, July 2022. ISSN 2045-2322.
doi: 10. -22---2-5-252-
[3] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[4] Boris van Breugel and Mihaela van der Schaar. Why tabular foundation models should be research priority. arXiv preprint arXiv:2405.01144 2024.
[5] Zachary Young and Robert Steele. Empirical evaluation of performance degradation of machine bearning-based predictive models-a case study in healthcare information systems. International Journal of Information Management Data Insights, 2(1):100070) 2022.
[6] Gerard Pasterkamp. Hester M Den Ruijter, and Peter Libby. Temporal shifts in clinical presen- tation and underlying mechanisms of atherosclerotic disease. Nature Reviews Cardiology, 14 (1):21-29, 2017.
[7] Rajarajan Ganesan, Varun Mahajan, Karan Singla, Sushant Konar, Tanvir Samra, Senthil K Sundaram, Vikas Suri, Mandeep Garg, Naveen Kalra, Goverdhan D Puri, et al. Mortality prediction of covid-19 patients at intensive care unit admission. Cureus, 13(1 1), 2021.
[8] Tom Beucler, Pierre Gentine, Janni Yuval, Ankitesh Gupta, Liran Peng, Jerry Lin, Sungduk Yu, Stephan Rasp, Fiaz Ahmed, Paul A O' O'Gorman, et al. Climate-invariant machine learning. Science Advances, :0(6)sadjj2250, 2024.
[9] Yvan Lucas, Pieure-Edouard Portier, Léa Laporte, Sylvie Calabretto, Liyun He-Guelton, Frederic Oble, and Michael Granitzer. Dataset shift quantification for credit card fraud detection. In 2019 IEEE second international conference artificial intelligence and knowledge engineering (AIKE), pages 97-100. IEEE, 2019.
[10] George Alexandru Adam, Chun-Hap Kingsley Chang, Benjamin Haibe-Kaing and Anna Goldenberg. Hidden risks machine learning applied healthcare: Unintended feedback loops between models and future data causing model degradation. In Finale Doshi- Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pages 710-731, PMLR, 07-08 Aug 2020. URL https: http::://pocceeeeeeee mlr, press/ hts///v2/////// 226/mmmm00..... html.
11

---

[11] Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurlPS 2022. URL attpss//opeereviile net. orum? ddd F-
[12] Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In Proceedings of the International Conference on Learning Rep- resentations (ICLR 23)), 2023. URL http::///oppppeeeeee...////rrrrp Published online: iclr
4nTin.
[13] Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss generalize along time. In M. Ranzato, A. Beygelzimer, K. Nguyen. P. Liang, J. Vaughan, and Y. Dauphin, editors. Proceedings of the 34th International Conference on Advances in Neural Information Processing Sys- tems (NeurIPS 221) Curran Associates, 2021. URL U7SBcmRf65.
[14] Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift tabular data with tableshift. Advances in Neural Information Processing Systems, 36, 2024.
[15] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 18932-18943, 2021.
[16] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
[17] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do brsed models still outperform deep learning on typical tabular data? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Proceedings of the 36th International Conference on Advances Neural Information Processing Systems (NeurIPS'22), 2022. URL https: phhpp.
//openreview.o
[18] S. Müller, N. Hollmann, S. Arango, J. Grabocka, and F. Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR'22) 2022. URL https:// wwtpeeviwwwwwwweeeeeeee foooorrnnnnuuuuuabb... iclr.c.. cc.
Published
online:
[19] N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. TabPFN: A transformer that solves small tabular classification problems in second. In Proceedings of the International Conference on Learning Representations (ICLR'S 23), 2023. URL https: htt////eeeeeeeeeeeeeeeeeeeeereee wwt./eeummmtattttemmm id= cp5PvcI6w8_. Published online: iclr.cc.
[20] Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
[21] Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[22] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time, 2020. URL https://opperrevee/ http:/eerrevieeeeee/frrrrrrn net/forum? id=rkIk1cVYvB.
[23] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, and R. Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (6),'16), pages 785-794. ACM Press, 2016.
[24] L. Prokhorenkova, Gusev, A. Vorobev, A. Dorogush, and A. Gulin. CatBoost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Garnett, editors, Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems (NeurIPS' 18). Curran Associates, 2018
12

---

[25] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma. Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach. R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings of the 30th International Conference on Advances in Neural Information Processing Systems (NeuriPS 17). Curran Associates, 2017.
[26] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. In H. Daume IIII and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning (ICML'20), volume 98. Proceedings of Machine Learning Research, 2020.
[27] Paras Sheth, Raha Moraffah, K. Selçuk Candan, Adrienne Raglin, and Huan Liu. Domain generalization - a causal perspective, 2022.
[28] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning (CML')3). Omnipress, 2013. URL html..
attps://proceediing mlr prress/v28/mmaneeee
[29] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularrizaton. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Proceedings of the 31 International Conference on Advances in Neural Information Processing Systems (NeurlPS' 18). Curran Associates, 2018.
[30] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 5715-5725, 2017.
[31] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[32] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In Proceedings of the International Conference on Learning Representations (I) 2020. Published online: iclr cc.
[33] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference Learning Representations (ICLR' 18), 2018. Published online: iclr.cc.
[34] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving ofi-t--dist robustness via selective augmentation, 2022.
[35] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou, editors, Computer Vision ECCV 2016 Workshops. pages 443-450, Cham, 2016. Springer International Publishing. 1oi:10.1007/978. 10010079978--3319444409449
[36] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural networks. Journal of Machine Learning Research, (7(1):09962033 jan 2016. ISSN 1532- 4435.
[37] David Krueger, Ethan Caballero, Joem--Herik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML'21), volume 139 of Proceedings of Machine Learning Research. PMLR, 2021. URL https: : /ttp////eoeeeees..//////// mlr
hat////rreeeerr11 html.
[38] Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgen, Hamed Hassani, George J. Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Proceedings of the 36th International Conference Advances in Neural Information Processing Systems (NeurlPS 22), 2022. URL https: //wpeedddigg....s..eooe...a neurips cc/ cctppperr..e...eee pappppappprraaap files/ paper/2022/ htte://222/111111fffffoocotooefd hatpr/e--PPppee--oofeeeeeeee .ddf.
Conf
13

---

[39]
Andrey Malinin, Neil Band, Alexander Ganshin, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vat- sal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panos Tigar, and Boris Yangel. Shifts: dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455. 2021.
[40] Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark J. F. Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po-Jui Lu, Nataliia Molchanova, Antonis Nikitakis, Vatsal Raina, Francesco La Rosa, Eli Sivena, Vasileios Tsarsitalidiss, Efi Tsompopoulou, and Elena Volf. Shifts 2.0: Extending the dataset of real distributional shifts, 2022. URL https: /ttp///xxvvv..o/.//e wttpo/gg//bbb//eeee abs/2206. 15407.
[41] Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations on tabular datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[42] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In M. Ranzato, A. Beygelzimer, K. Nguyen, Liang, J. Vaughan, and Y. Dauphin, editors, Proceedings of the 34th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 21). Curran Associates, 2021.
[43] Sergey Kolesnikov. Wild-tab: A benchmark for out-of-distribution generalization in tabular regression, 2023. URL //atpi://axiiv.oot org/ abs//a////22 0179.
[44] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Proceedings of the International Conference on Learning Representations (ICLR 21), 2021. URL https: // net/forrumridd--laeeeeelbtt Published online: iclr
[45] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 35(8):8052-800) 8072,
2023. doi: 10.
[46] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. A. Globerson and R. Silva, editors, Proceedings of The 34th Uncertainty in Artificial Intelligence Conference (UAI'18) AUAI Press, 2018.
[47] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei Alex Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. In 4th Lifelong Machine Learning Workshop at ICML. 2020, 2020. URL //tttpss///peenevieewe net/ htt://ormmmmdddeeee IzVdd
[48] Junwei Ma, Valentin Thomas, Guangwei Yu, and Anthony Caterini. In-context data distillation with tabpfn, 2024. URL httpp://arrxvv.rr//bbb//// 06971
[49] Benjamin Feuer, Robin Tibor Schirmeesster, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable paior-data fitted networks. arXiv preprint 2rXiv:2402.11137, 2024.
[50] Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha V Naidu, and Colin White. Forecastpfn: Synthetically srnthetically-trainedd trained zero-shot forecasting. In Advances in Neural Information Processing Systems, 2023.
[51] Jose G. Moreno-Toreee Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V. Chawla, and Fran- cisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45 (1::511-300) 2012. ISSN 0031-3203. / //tww//cwinnaeeeeeee www.sssciecneeee..... com/ http///c///aatttteaa/////ooo/ooa
10110..001//j 2011..6.019 URL https:
[52] Meelis Kull and Peter Flach. Patterns of dataset shift. In First international workshop on learning over multiple contexts (LMCE) at ECML-PKDD, volume 2014.
[53] Vanschoren, J. van Rijn, B. Bischl, and L. Torgo. OpenMI.: Networked science in machine learning. SIGKDD Explorations, 15(2):49-66 60, 2014.
14

---

[54] Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987. URL
.awaar.rg///taae htgg/ttable/30009636
[55] Sture Holm. simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
[56] Salvador Garcia and Francisco Herrera. An extension on" statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons. Journal of machine learning research, (12), 2008.
[57] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33(4):117-96) 2019.
[58] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, I LL4(13)3351-33526 2017.
et al.
[59] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In D. Precup and Y. Teh, editors, Proceedings of the 34th International Conference on Machine Learning (CCL'17), volume 70. Proceedings of Machine Learning Research, 2017.
[60] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In Proceedings of the International Conference on Learning Representations (ICLR' 2019. Published online: iclr.co cc.
[61] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In H. Daume III and A. Singh, editors, Proceedings of the 37th International Conference Machine Learning (ICML '20), volume 98. Proceedings of Machine Learning Research, 2020.
[62] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.-F, Balcan, and H. Lin, editors, Proceedings of the 33rd International Conference on Advances in Neural Information Processing Systems ( Curran Associates, 2020.
[63] Bendi Ramana and N. Venkateswarlu. ILPD (Indian Liver Patient Dataset). UCI Machine Learning Repository, 2012.
[64]
Oguz Akbilgic. Istanbul Stock Exchange. UCI Machine Learning Repository, 2013.
[65] Beata Strack, Jonathan DeShazo, Chris Gennings, Juan L. Olmo, Sebastian Ventura, Krzysztof J. Cios, and John N. Clore. Impact of HbA1c Measurement on Hospital Read- mission Rates: Analysis of 70,000 Clinical Database Patient Records. BioMed Research International, 2014:781670, April 2014. ISSN 2314-6133. doi: 10001155/0114781661 URL nttps://doee http://o....rr1111.5550///11/1// 115511111///11117777
Publisher: Hindawi Publishing Corporation.
[66]
Albert Bifet and Elena Ikonomovska. Data Expo competition. OpenML, 2009. URL https: org/
/ /
wtw.aatttaatotttuuuuuaeae d= 1 at..ssssstaussacciviee
[67] Jack Smith, J. Everhart, W. Dickson, W. Knowler, and Richard Johannes. Using the adap learning algorithm forcast the onset of diabetes mellitus. Proceedings Annual Symposium on Computer Applications in Medical Care, 10, 11 1988.
[68] M. M. Faniqui Islam, Rahatara Ferdousi, Sadikur Rahman, and Humayra Yasmin Bushra. Likelihood prediction of diabetes early stage using data mining techniques. Mousumi Gupta, Debanjan Konar, Siddhartha Bhattacharyya, and Sambhunath Biswas, editors, Computer Vision and Machine Intelligence in Medical Image Analysis, pages 113-125, Singapore, 2020. Springer Singapore. ISBN 978-981-13-87988 doi: 100710077778988
12. [69] Luis Candanedo. Occupancy Detection. UCI Machine Learning Repository, 2016.
15

---

[70] Ricardo Ferreira, Andrea Martiniano, and Renato Sassi. Behavior of the urban traffic of the city of Sao Paulo in Brazil. UCI Machine Learning Repository, 2018.
[71] Jacob Montiel, Jesse Read, Albert Bifet, and Talel Abdessalem. Scikit-multfllow: A multi- output streaming framework. Journal of Machine Leurning Research, 19(72):1-5, 2018. URL http://jmlr atg/papers/v19/11-oe 251 html.
[72] Angela Dispenzieri, Jerry Katzmann, Robert Kyle, Dirk Larson, Terry Therneau, Colin Colby, Raynell Clark, Graham Mead, Shaji Kumar, L Melton, and S Rajkumar. Use of nonclonal serum immunoglobulin free light chains to predict overall survival the general population. Mayo doi: 10i:10.1116 10.1016/j.mayccppp 201.00000..1.03.0009
Clinic proceedings. Mayo Climic, 87:517-23, 06 2012.
[73] Robert Kyle, Terry Therneau, S Rajkumar, Dirk Larson, Matthew Plevak, Janice Offord, Angela Dispenzieri, Jerry Katzmann, and L Melton. Prevalence of monoclonal gammopathy of undetermined significance. The New England journal of medicine, 354:1366 9, 04 2006. doi: 10. 10.11//////M0005494...
[74] Michael Harries. Splice-2 comparative evaluation: Electricity Pricing. University of New South Wales, School of Computer Science and Engineering [Sydney], 1999. URL http: / //nla.gors golla.gov.auu wwwnlaa.......aaa...a arc- arc-32869.
[75] João Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Ana L. C. Bazzan and Sofiane Labidi, editors, Advances in Artificial Intelligence - SBIA 2004, pages 286-295, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540- 28645-5, doi: 10. 00110079988---54022865555
[76] Andrea Martiniano and Ricardo Ferreira. Absenteeism at work. UCI Machine Learning Repository, 2018.
[77] Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1988.
[78] Daniel Stolfi. Parking Birmingham. UCI Machine Learning Repository, 2019.
[79] Dean De Cock. Ames, iowa: Alternative to the boston housing data
end of semester regression project. Journal of Statistics Education, 19, 11 2011. 10::10.1080/100 198.2011.1188899
[80] Indré Žliobaité. Combining similarity in time and space for training set formation under concept drift. Intelligent Data Analysis, 15(4):589661,, 2011.
[81] Kartik Ahuja and David Lopez-Paz. A closer look at in-context learning under distribution shifts. arXiv preprint arXiv:2305. /6704, 2023.
16

---

Table of Contents
A Appendix
17
A.1
Reproducibility
17
A.2
Broader Impacts
18
A.3
Computational Resources
18
A.4
Additional Experiments
19
A.4.1
Ablation Studies
A.4.2
Perturbation of Temporal Domain Indices within the Prior
A.4.3
Qualitative Analysis
20
A.4.3.1
Overview of the Shifts in the Datasets Analyzed
20
A.4.3.2
Decision Boundaries on Rotated Two Moons Dataset
20
A.4.3.3
Decision Boundaries by Type Shift
21
A.4.3.4
Comparison Against DRAIN and
21
A.4.4
Quantitative Analysis
22
A.4.4.1
Combined Results Across All Datasets
22
A.4.4.2
Critical Difference Diagrams
23
A.4.4.3
Comparison Against Wild-Time Methods
24
A.4.5
Investigating Performance Saturation Main Baseline Methods
25
A.4.6
Analyzing Relationship Between Out-of- Distribution Performance and Difficulty Across Datasets
25
A.4.7
Number of Shift Observations Required for Effective Extrapolation
27
A.5
Supplementary Illustrations and Methodological Overviews
28
A.5.1
Plots Illustrating Types of Distribution Shifts
28
A.5.2
Plots Illustrating Sampled Parameters of the Adjusted Prior
28
A.5.3
Algorithmic Overview of Our Approach
29
A.5.4
Illustration of Adopted Evaluation Strategy
29
A.6
Detailed Overview of Wild-Time Methods
A.6.1
Classical Supervised Learning
30
A.6.2
Continual Leaminng
30
A.6.3
Temporally Invariant Learning
30
A.6.4
Self- Learning
30
A.6.5
Bayesian Learning
................................
31
A.7
Datasets
................................
31
A.7.1
Validation Datasets
31
A.7.1.1
Synthetic Datasets
31
A.7.1.2
Real World Datasets
31
A.7.2
Test Datasets
32
A.7.2.1
Synthetic Datasets
32
A.7.2.2
Real World Datasets
33
A.8 Hyperparameter Search Spaces
35
B NeurIPS Paper Checklist
38
Appendix
A.1 Reproducibility
Code Availability. an effort to ensure reproducibility, we release code, version specification of baselines, pre-trained Drift-Resilient TabPFN and interactive Colab notebook, that lets you interact with our seikit-lear interface, at https: /ttps::/ge gigh.u...... com/ TabPFN.
Data Availability. All real-world datasets used our experiments are freely available at OpenML. org [53] via the original sources referenced in Section A.7, with downloading scripts or instructions
17

---

included in the submission code. Code to generate our synthetic datasets can be found in our code repository, see above.
Details of Hyperparameters for Drift-Resilient TabPFN and Baselines. An overview of the hyperparameters used for running Drift-Resilient TabPFN and our baselines can be found in Tables 9 and 10 respectively.
Evaluation Reproducibility. Consistent dataset splits were pre-determined to ensure comparability across all evaluations, reinforcing the reliability of our findings.
A.2 Broader Impacts
As general method for handling distribution shift in tabular data, Drift-Resilient TabPFN does not have immediate direct societal implications in the same way as AI systems designed automate specific tasks or replace human jobs. However, Drift-Resilient TabPFN could offer several potential positive societal impact:
1. Increased Model Longevity: Our approach extends the usable lifespan of deployed MI. models by adapting to distribution shifts, reducing the need for retraining and leading to cost savings and more stable performance. In healthcare, this can ensure diagnostic and prognostic models remain reliable as data shifts over time.
2. Improved Decision Making and Long-Term Predictions: Drift-Resilient TabPFN en- hances decision-makinn in critical fields like finance and climate science by enabling more robust, longer-term predictions. Our Bayesian approach for tackling distribution shift provides a new perspective that can spur further methodological innovations.
However, we also recognize potential negative impacts:
1. Potential Misuse: Like any ML advance, more robust models could be misused for harmful purposes if not developed and deployed responsibly.
2. Environmental Cost: While the trained model is efficiently applicable various datasets, the considerable computational resources used for initial training should be noted. However, we note that these costs are one-time, while the resulting model can be applied with minimal energy usage.
A.3 Computational Resources
In the course of our research on Drift-Resilien TabPFN, we employed substantial computational resources across various stages model development, training, and evaluation. The computation specifics are as follows:
1. Infrastructure: The experiments were conducted internal SLURM cluster equipped with RTX 2080 TI GPUs and CPUs of type AMD EPYC 7502, 32C/64T, @ 2.50-3.35GH..
2. Baseline Experiments: Each baseline experiment utilized 8 CPUs, 1 GPU, and 62.5 GB RAM, with a hyperparameter optimization (HPO) runtime of 1200 seconds per dataset per split, repeated three times to ensure reliability.
3. Pre-training for Drift-Resilienn TabPFN and TabPFN-base: These models were pre- trained three times each, requiring 64 CPUs, 8 GPUs, and 500 GB RAM, requiring approxi- mately 7 and 8 days respectively.
4. Hyperparameter Optimization for Drift-Resilient TabPEN and TabPFN-base We used 32 CPUs, 4 GPUs, and 250 GB RAM, running approximately 40 configurations and taking about one day per pre-training session for optimizing the hyperparameters of the novel prior-data generating mechanism of Drift-Resilient TabPFN. Both TabPFN-base and Drift-Resilient TabPFN underwent preprocessing optimization that utilized 8 CPUs, 1 GPU, and 62.5 GB RAM across 300 runs, each lasting between 0.5 to hour.
Additional computational resources were allocated for method development tests and other exper- imental setups not detailed in the final publication. Thus the full scope of the research required more computational resources than those detailed above due these preliminary and unreported experiments.
18

---

A.4
Additional Experiments
A.4.1 Ablation Studies
our model's performance mostly based on Teme2Vec preprocessing? To address this question, we conducted an ablation study where we trained model with temporal domain indices normalized but not subjected to Time2Vec preprocessing (No T2V).
Table 2 presents the performance metrics of Drift-Resilient TabPFN, TabPFN-base, and our ablation model. The results indicate that while Time2 preprocessing may slightly improve model perfor- mance, it is statistically insignificant. Rather, the substantial performance improvements are largely due to our prior construction, used during the pre-training phase of the model. The decision to keep Time2 Vec in the final model was guided by our HPO, which indicated positive impact on average performance.
Table 2: Comparison of Drift-Resilient TabPEN with respect to the stated ablations. Metries include (OOD) data.
ROC AUC and accuracy for both in-distribution (ID) and
Model
Variant
f
1
ROC
ECE
00D
ID
O0D
ID
ODD
ID
ODD
ID
TabP all dom, ind.
0.744
ms 0.879 012
00.689
0.837
022
00.832
0.932 .002
0..091
0.074
No T2V
all dom, w. ind.
0.742
0o4 0.877 .077
0.685
.p0 0,834
0.832
0.931
0.093
0.071
all dom. w. ind.
0,688
.n1 0.885 o1
0.62
.012
0.847
.017
0.786
007
0.935 .01
0.119
.006
0.067 .005
Tab all dom. ind.
0.645
011
0.852
016
0.579
0014
0.801
0.736
001
0.914 .077
0.202
011
0.076 007
last domn wa. ind.
0.67
.005
0.867
.004
0.609
.004
0.823
011
0.76
.008 0.915
.019
0.181
.003
0.128 .07
A.4.2 Perturbation of Temporal Domain Indices within the Prior
This additional experiment analyzes the impact of perturbing temporal domain indices on the perfor- mance of our information is often unknown and must be approximated, creating crucial difference between these datasets and those generated by the prior in our methodology. To assess this, we conducted an experiment during model development wherein we intentionally modified the ground truth domain indices C during the pre-training phase prior to feeding them into the transformer.
TabPFN model. In real-world datasets, ground truth temporal domain
In this context, we explored three primary techniques:
Shifting Domain Boundaries Probabilistically. We shifted the boundaries of domains and reported instances near those boundaries as helonging to adjacent domains.
Merging Domains. Multiple domains were probabilistically merged into single domain, thereby introducing ambiguity in domain information.
Noise Injection. Random noise was added to each domain indicator, further complicating reasoning about temporal distribution shifts based on these indicators.
Our experiments on our validation datasets, listed in Table 3, show that overall these perturbations adversely affect model performance. The findings suggest that the model requires ground-truth domain indices for effective training, emphasizing the importance of accurate domain information in real-world applications.
Table 3: Performance evaluation of information. Three techniques were examined: (1) Shifting domain boundaries probabilistically. reporting instances near those boundaries as belonging to the other domain; (2) Probabilistically merging multiple domains into one; (3) Adding noise to each domain indicator. Our results indicate that perturbed domain indices overall led to decline in model performance, emphasizing the model's s requirement for ground-truth domain indices during training.
TabPFN against models with perturbed domain
Model
Variant
Acc. 1
F1
ROC
ECE, L
OOD
ID
0OD
OOD
ID
OOD
ID
TabPFNas
no changes
0.74 0.867
0.692
0.832
0.837
0.908
0.085 0.076
alt dom. hound.
0.734
0.86
0.682
0.822
0.829
0.905
0.092
0.078
TabPYN
alt dom. bound. merge dom.
0.742
0.857
0.69
0.815
0.833
0.906
0.09
0.079
alt dom. hound. merge dom, / noise 0.704
0.835
0.64
0.78
0.792
0.888
0.101
0.8
19

---

A.4.3
Qualitative Analysis
Overview of the Shifts in the Datasets Analyzed This section offers plots of the Inter- secting Blobs and Rotated Two Moons datasets across specific temporal domains. The Intersecting Blobs dataset is visualized in Figure 6a for domains C {0, 4, 8, 13). The Rotated Two Moons dataset presented Figure for domains C {0, 3, 6, 9).
A.4.3.1
Damain
Domain
Domain
Coss
Casss
-4
-10
Class
Class
10
Domain
Domain 13
Domain 6
Domain
-10
-15
-20
-20
1.0
(a) Intersecting Blobs
(b) Rotated Two Moons
Figure 6: This figure shows the temporal shifts of two synthetic datasets across selected domains.
A.4.3.2 Decision Boundaries on Rotated Two Moons Dataset In addition to the qualitative analysis conducted for the Intersecting Blobs dataset in the main text of this work, this subsection provides additional illustrations the Rotated Two Moons dataset. The corresponding visualizations for our approach as well as the TabPEN baseline provided in Figure 7.
Train Domains: 0-5 Test Domain
Trinn Domains: 01-
Tast Domain
Train Damains: 0-5 Test Domain 8
Tab
TabP
02
Figure 7: This figure contrasts the predictive behavior of TabPFN dist and TabPFNoas on the Rotated Two Moons dataset. illustrates how each model adapts to different testing domains when trained on domains Cuain {0, 1, 2, 3, 4, 5). The color shading indicates the maximum class probability at each point, with decision boundaries shown when this probability exceeds 50%. Incorrectly classified samples are highlighted in red.
20

---

Decision Boundaries by Type of Shift To analyze our models behavior under distinct shift types, we separate decision boundaries by shift category provide insights into how the model adapts to each.
A.4.3.3
(a) Intersecting Blobs - Combined Covariate
(b) Moving Line Dataset Covariate Shift
and Concept Shift Ctimni 1,,1,20 3)
Chtaii {0, 10
beth Die Socaer 2011
Tuke Dhenee
asa Carer
(c) Sliding Circle Dataset Concept Shift Crrain (0,... (O.
(d) Binary Label Shift Dataset Probability Shift Caaain
(0.
Figure 8: Each figure displays the predictive behavior of TabPFN in the top row and TabPFN basc in the bottom row. It illustrates how each model adapts to unseen test domains when trained on domains Curin The baseline is given the domain indices as feature in train and test. The coloring indicates the probability of the most likely class at each point. Incorrectly classified samples are highlighted in red.
A.4.3.4 Comparison Against DRAIN and GI To show improved performance compared to the state-of-the-art methods DRAIN [12] and GI [13], we compare our method with the qualitative analysis performed by the authors of DRAIN on the Rotated Two Moons dataset. In this setting, all domains except the last are used for training, with the final domain reserved for testing. We illustrate method's s decision boundary compared to those provided by DRAIN in Figure 9. The accuracy results are listed in Table 4. As the contour levels are unknown, we display only the pure decision boundary for clearer comparison. Our analysis shows that model forecasts the rotation of the two moons more accurately compared to the baselines and adapts its decision boundary more precisely.
Train Domains: 0-8 Test Domain
class
class
class A
class
-2
-1
-2
(a) TabPFNaj
(b) DRAIN
(c) GI
Figure 9: Comparison our method against DRAIN [12] and GI [13] on the Rotated Two Moons
dataset. The models were trained on domains C {0, 1, 1.... 8) and tested on domain 9. While the authors of DRAIN present different, unknown levels of the decision boundary, we present the decision boundary with 50% probability. The plots for DRAIN and Gl were taken from Bai et al. [12].
21

---

Table Comparison of Drift-Resilien TabPFN against DRAIN and GI on the Rotated Two Moons dataset. The metric reported is the mean out out-of-distriutiin (OOD) accuracy along with the standard deviation. Results for DRAIN and GI are taken from Bai al. [12].
Model
OOD Acc.
Tab
0.98
DRAIN
.012
GI
0.965
A.4.4 Quantitative Analysis
A.4.4.1 Combined Results Across All Datasets
Table 5: Comparison of Drift-Resilient TabPFN with various baselines and settings across the combined real-world and synthetic datasets. Metrics include accuracy, F1, ROC, and ECE for both in-distribution (ID) and dittr--distribution (OOD) data, averaged over three initializations and reported with 95% confidence intervals. The best mean of each metric is marked in bold. Metric arrows indicate optimization direction.
Model
Variant
Acc.
F1
ROC
1
ECE +
0OD
ID
0OD
ID
OOD
ID
OOD
TabPFNaa
all dom. w. ind.
0.744
.018 0.879
.122
0.689
.028
0.837
0.832
.018
0.932 .002
000091 .006
0.074.or
dom, w.
0.688 .01 0.885
.01
0.62
.012 0.847
.017
0.786
.007
0.935 .01
0.119
.008
0.067
.005
TabPEN
all dom. wo, ind.
0.645
0.852
0.579
0.801
0.736
0.914
0.202
0.076
last dom. wa. ind.
0.67 .00
0.867 004
0.609
ons
0.823 011
0.76
.003
0.915
.019
0.181
.003
0.128
007
all dom. ind. 0.677 .oon
.007
0.62
.005
0.836
.01
0.766
.013 0.919
0.222
.007
0.081
009
CatBoost
all dom. wo. ind
0.632
0.836
U.568
0.781
0.714
0.894
0.24
.02
0.097
last dom, wo. ind.
0.657 .0o2 0.852
014
0.599
004 0.811 024
0.722
005
0.907
01
0.256
.005
0.133
.012
all dom, ind. 0.664 .co6 0.859
.004
0.61
.013
0.828
0.754
.006
0.9
.010
0.194
.018
0.111 .02
XGBoost
all dom, wo, ind.
0.633 .oss
0.831
.024
0.568
.053
0.778
0.718
0.881
0.194
0.12
last dem. wo. ind.
0.664 .01
023
018
0.758
654
0.733
.099
.016
0.199
.024
.011
all dom, w. ind. 0.65
0.842
0.394
0.8
0.738
0.908
0.198
0.093
LightGBM
all dom, wo. ind
0.625 .02
0.832
019
0.561
.018
0.773
018
0.706
.008
0.888 o01
0.199
.009 0.097 .005
last dom, ind.
0.629
0.797
0.542
0.72
0.686
0.852
0.224
0.149 01.6
all dom, w. ind. 0.627
0.821
0.525
0.771
0.688
0.877
0.263
0.108
Wild-Time
ERM
last dom. wo. ind. 0.587 0.784
wn. ind.
.229
0.803 o01
0.528
0.519
028
0.746
0.74
018
0.666
0.673
.02
0.862 .008
0.843
0.323
0.255
.019
0.19
.016
Wild-Time
all dom, w. ind.
0.627 .647
0.824
.003
0.534
860
0.777
.014
0.685
.03
0.873
.015
0.274
.033
0.11 .007
SWA
dom, wo. ind
022
.007
0.53
.020
.008
0.681
.03
.008
0.262
.038
0.109
22

---

A.4.4.2 Critical Difference Diagrams
Acc.
5
2
LightGBM
38446
1,811
TabPFN
XGBoost
2.844
2,2000
base
CatBoost
F1
LightGEM XGBoost
1,058
TabPFNast TabPFNaasa
3777B
277788
CatBoost
2777
ROC AUC
5
4
2
LightGEM
427m
1.8111
TabPFNast
XOBoos!
38111
TabPFN base
CatBoost
31667
ECE
TabP dist 3.11
CalBoost
TabPFN base
7:5000 XGBoost
LightGEM 25538
Figure 10: This figure presents critical difference diagrams of our evaluated metrics on OOD data, analyzed using the Wilcoxon-Holm method [54-57] across the best performing OOD models evaluated in this work. The diagrams indicate sigmificant differences of Drift-Resilient TabPFN against the tree-based methods. For the FI metric our method shows significant differences against all top performing baselines. Arrows indicate optimization direction.
23

---

A.4.4.3 Comparison Against Wild-Time Methods This section offers a quantitative evaluation of our model against methods found in the Wild-Time benchmark [11]. We focus exclusively on methods applicable to tabular data, omitting SimCLR and SwAV which are specifically designed for image datasets. Detailed explanations of these methods are available in Section A.6.
As a base model, we used Multilayer Perceptron (MLP) optimized through Hyperparameter Optimization (HPO).
The findings this quantitative comparison are presented in Table 6. Notably, none of the evaluated Wild-Time methods demonstrated performance equal with approach or the other baseline methods on OOD data. This discrepancy is likely due to the small number of instances in the datasets used in our evaluations, which affects the generalization of these deep learning techniques. Among the Wild-Time methods, SWA emerged as the most effective in handling OOD data.
Table 6: Comparison of Drift-Resilient TabPFN with the applicable baselines of the Wild-Time benchmark [1 I] across the combined real-world and synthetic datasets. Metries include accuracy, FI, ROC, and ECE for both in-distribution (ID) and cuttof-dditributionn (OOD) data, averaged over three initializations and reported with 95% confidence intervals. The best mean of each metric is marked in bold. Metric arrows indicate optimization direction.
Model
Variant
Acc. 1
F1
ROCT
ECE 1
OOD
ID
OOD
ID
OOD
ID
0OD
TabPFNas
all dom, w. 0.744 .18 0.879 .12
0.689
o0s 0.837
.023
0.832
.18
0.932 .023
0.0011
ond 0.074. 0014
ERM
all dom. w. ind. 0.627 .aa 0.821
0.525
0.771
0.688
0.877
0.263
.054
0.108
all dom, wo.
0.582
.22 0.803 .01
0.519
.28
0.746
o088
0.673 .02
0.862
.008
0.255
.019
0.104
.015
last dom. ind.
0.587
.22 0.784
0.528
0.71
.011
0.666
0.843 .02
0.323
.038
0.19
FT
all dom. w. ind.
0,51 .oas
0,6455
014
0.422
.048
0.559
oss
0.594
.o2r
0.693
.os9
0.357
.013 0.256 .007
all dom. ind.
0.544
0.677
031
0.481
0.604
.03
0.65
0.756
0.33
0.231 .611
EWC
all dom, w. ind.
0.498
o22
0.621
.044
0.408
.038
0.522
0.569
.027
0.668
.oo8
0.342
.008
0.239
.054
all dom.
o2s
045
0.456
.32
0,617
051
0.603
.012
.026
0.309
.048
0.195
nos
all dom. w. ind.
0.478
037 0.63
0005
0.383
0.526
U.566
0.673
023
0.37
.016
0.238
.012
all dom, wo. ind.
0.495
019
0.674
035
0.425
036
0.588
035
0.597
o22
0.744
.03
0.346
.00
0.214 040
A-GEM
all dom. w.
031 0.677
0,405
.o1
0.576
045
0.594
o661
0.741
0039
U.367
.024
0.223
.021
all dom, wo. ind.
0.486
oas
0.684
0.403
0.583
0.58
0.747
0.364
.073
0.221
CORAL-T
all dom. w. all dom, wo. ind.
0.579
051 0.777
.02
0.481
.055
0.714
041
0.637
.036
0.837
0000
0.252
.008
0.143
.025
0.569
0.771
0.495
0.702
0.643
0.837
0.232
.027
0.15
GroupDRO-T
all dom, w. ind.
0.58
.025 0.779
014
0.503
0.723
000
0.642
0.84
.007
0.285
.019
0.5 .01
all dom. all dom, w. ind.
ind.
0.576
025 0.775
023
0.514
.22
0.725
086
0.638
.008
0.843
.oo6
0.2611
.076
0.135 .04
IRM-T
0.577
.211 0.746
0.49
0.689
0.633
0.819
0.259
0.12
all dom, wo. ind.
0.559
018
0.742
023
0.498
.031
0.678
647
0.644
.oos
0.822
.013
0.252
.0s1
0.134 _011
Mixup
all dom, ind.
0.617
.22 0.8
.028
0.521
.016
0.702
0008
0.681
.020
0.859
.021
0.239
.014
0.14 .007
all dom, wo, ind.
0.574
.054 0.782
027
0.492
.25
0.688
0.68
0.85
0.212
.021 0.142
.017
LISA
all dom. w. ind.
0.621 041 0.822
638
0.517
.079
0.76
.063
0.679
0.881
.012
0.272
.046
0,116
.02
all dom. ind.
0.583
013 0,804
015
0.512
.009
0.739
0.672
0025
0,859
02a
0.258
.025 0.108
.22
SWA
all dom. w. ind.
0.627 847
0,824 oo0
0.534
.069
0.777
.014
0.685
.0a0
0.873
.015
0.274
.oss
0.11
.007
all dom. ind.
0.588
025
0.802
007
0.53
.020
0.748
.oo
0.681
.03
0.863
.oo9
0.262
.036
0.109.
.01
24

---

A.4.5 Investigating Performance Saturation of Main Baseline Methods
In this subsection, we assess the computational budget allocated for HPO on our main baseline methods. By comparing model performance at 1200 seconds versus 3600 seconds per method and dataset split, we show that the increased budget for HPO does not yield significant gains overall, indicating sufficient saturation within the initial budget.
Table 7: Comparison of GBDT baselines across combined real-world and synthetic datasets for two different hyperparameter optimization (HPO) time budgets: 1200s and 3600s. Metries include accuracy, FI, ROC, and ECE for both in-distribution (ID) and out-of-distributton (OOD) data, averaged over three initializations and reported with 95% confidence intervals. Metric arrows indicate optimization direction.
Model
Variant
Acc.
ROC
1
ECE
O0D
ID
00D
ID
0OD
ID
00D
ID
all dom, w. ind. 0.678
0.872
.006
00.621
0.833
.004
0.764
.007 0.914
011
0.205
0.084
CatBoostts
dom,
0.632
.005 0.838
.0o
0.568
0077
0.782 01
0.713
.004 0.893
026
0.226
002 0.099 .018
last dom. ind. U. 6,636
0.85
0.398
0.813
0.721
.0o6 0.903
0.264
0.136 0228
all dom. w. ind. 0.677 . 0.874
.o0r
0.62
.01 0.830
0.766
.0o3 0.919
.011 0.222
.oo7 0.084 .009
CatBoostgonii
all dom.
0.632 ooss 0.836
.013
0.568
.009
0.714
.012 0.894
.014
0.24
.22 0.097 .015
last dom. wa. ind. 0.657
0.852
0014
0.599
0.811
0.722
.oon 0.907
0.256
0.133
all dom. w. ind. 0.662 .013 0.864
.01a
0.61
.01
.017
0.754
.008 0.911
.012
0.187
017 0.101 .017
1GBoost1h
all dom, wa. ind. 0.639
0.833
.002
0.578
0.781
0.724 .0oc 0.886
0.181
0.124 .022
last dom. wo.
0.604
.013 0.833
.021
0.508
.222 0.705
.04
0.736
.012 0.895
.014
0.195
.010 0.164 .005
all dom. w. ind. 0.661 .000 0.859
.004
0.61
.13 0.828
.003
0.751
.006
.010
0.191
.188 0.111 .02
2Gmion22tmin
all dom. wo. ind.
0.633
,oas 0.831
.024
0.568
loss 0.778
.oa1
0.718
.osa 0.881
.o2s
0.194
.054 0.12 .042
dom. wa. 0.664
on 0.824
023
00559
ons 0.758
.04
0.733
.con 0.887
o016
0.199
024 0.167 .011
all dom, w. ind. 0.647 .012 0.846
.00s
0.593
.113 0.805
.o0s
0.736
.oos 0.909 .oos
0.211
.oos 0.096 .007
LightGBM It
all dom, wn. last dom. wo. ind.
0.618
.005 0.822
.008
0.555
.007 0.767
.006
0.708
.co7 0.888 .o15 0.2
.014 0.102 .008
0.624
0.795
0.546
0.724
0.687
.oos 0.845
0.231
0,148 .017
all dom, w. ind. 0.65 .008 0.842
.044
0.504
.oss 0.8
.024
0.738
.oos 0.908
oos 0.198
.099 0.093 .066
2ightGBM2omii all dom. ind. 0.625 .02 0.832
.019
0.561
.01s 0.773
.018
0.706
.009 0,888 011
0.199 .009 0.097 .005
last dom. wo. ind. 0.629 .02 0.797
0.542
031
0.72
0.686
.0o6 0.852
.018 0.224
0.149
A.4.6
Analyzing the Relationship Between Out-of-Distribution Performance and Difficulty Across Datasets
In this supplementary evaluation, we investigate the correlation between the inherent difficulty of dataset concerning distribution shifts and the performance of our approach in comparison to the baselines.
Defining Difficulty Metrics. Our first step involves assigning a difficulty score to individual dataset splits, using basic XGBoost model that doesn't it include domain indices as feature of the dataset. This score is determined using the formula ID OOD, where both ID and OOD performances are measured using ROC AUC. This difficulty score quantifics the decline in model performance when transitioning from ID to OOD data. After calculating the difficulty scores for each dataset split, we proceed to train each of the evaluated methods across all domains, including the domain indices. We then compute the performance gap, which is also calculated ID OOD, for each method.
These metrics are then visualized by plotting the dataset's difficulty score against its corresponding performance gap for each dataset split. We also fit linear regression lines to the scatter plots representing each method. The final plot illustrated in Figure 11.
Although the data points cannot be interpreted directly, the linear regression suggests that our method experiences the least decline in ROC AUC performance as dataset difficulty increases. affirming its robustness. The TabPFN bascline is the second-bes best performer, while the other models show more significant drops in performance.
25

---

Comparison OOD Performance and Difficulty Across Datasets and Splits
more
Perfomancc Gep per Methos (ID 00DD
0.4
0.2
Method
CatBoost
-0.2
LightGiMM
Tablf Newt
-0.2 -0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Gap
Figure 11: This figure illustrates comparative analysis of the resilience of the listed methods to out-of-distribution (OOD) difficulty across multiple datasets and splits. The z-axis captures the difficulty of each dataset split, while the y-axis measures the performance drop of method compared to in-distribution (ID) performance. Individual methods are represented by scatter points and their corresponding linear regression lines, with shaded regions indicating the 95% confidence intervals for TabPFN methods. Directional arrows signify increasing or decreasing dataset difficulty. Flatter regression slopes indicate models that are more resilient increases dataset difficulty due to distribution shifts.
26

---

A.4.7 Number of Shift Observations Required for Effective Extrapolation
For this analysis, we evaluated the performance of both our method and TabPFN-base by fixing the prediction the last domain of the Intersecting Blobs dataset and gradually increasing the number of domains available during training. Our results confirm observations made during method development: Our method requires significantly fewer training domains to accurately extrapolate shifts into the distant future, while the TabPFN-base only has acceptable decision boundaries when there is data whose distribution close the test domain. Below Figure 12 are the results for this experiment with the Intersecting Blobs dataset discussed in Section A.4.3.1. The figure clearly shows that our model greatly improves its predictions with the first domains. When provided with the domains Cuaiin {0, ..... 3), the shift is largely understood. with newer domains having little impact on performance. On the other hand, TabPFN-base requires much more training domains to achieve similar performance
method.
Acc. 1
1.0
0.6
-abPF Nast
0.2
"abPF Noase
0.0
F1
1.0
Performance nce on Test Domain Set = (9)
0.8
0.6
-abPF Nast
0.2
"aDFF "aDPFNo..
ROC AUC
1.0
0.8
0.6
0.4
"ab Nast
0.2
"abPF Moase
ECE
0.6
"abPF Nast "abPFNuuse
0.4
0.2
(0.1)
(G.....) 2)
(0, (o....)) 3)
(o. 4)
(o.....) 6)
(0. .o....... 7)
(0. [o.....8) 8)
Train Domain Set trainn
Figure 12: This figure shows the performance of Drift-Resilient TabPFN and the baseline TabPFN on
the Intersecting Blobs dataset. Thereby, we always test on domain Crest = {9} and gradually increase on the r-axis the number of training domains starting with Quain {0, 1} up to Qurana {0, 1,..,, 8). The results show that frftt-Resilient TabPFN achieves effective extrapolation with as few as four training domains, while TabPFN-base needs significantly more reach similar performance.
27

---

A.5 Supplementary Illustrations and Methodological Overviews
A.5.1 Plots Illustrating Types of Distribution Shifts
Covtriate Shift
Prior Probubility
Concre Shift
Figure 13: Illustration of initial and shifted data distributions alongside their optimal decision boundaries. The left panel depicts the initial classification dataset with two features and its true-data- optimal decision boundary. The right panel presents the dataset subjected the three primary types of distribution shifts observed during test time.
A.5.2 Plots Illustrating Sampled Parameters of the Adjusted Prior
Sampled Temporal Domain Distributions Across Selected Batches
1.0
ZiP
0.8
Temporal Domain
D
100
200
300
500
Instance
Figure 14: Share of temporal domains in exemplary datasets prior seen up any instance i. The figure illustrates the range and structure of the sampled temporal domains OR C across four representative datasets. It highlights variations in domain size and demonstrates the presence of arbitrary gaps, simulating irregularities in data sampling.
Temporal Functions Generated the 2rd-order SCM A
Function
Sampled Function (y.c)
helc.)
hytc))
hylc.)
042
0.
Temporal Domain
Figure 15: This figure presents three exemplary functions sampled from nodes within the network of 2nd--odee SCM H. In the plot, the r-axis represents the input temporal domain Cg E C, while the y-axis displays the corresponding node activation.
28

---

A.5.3 Algorithmic Overview of Our Approach
Algorithm 1 This algorithm provides a high-level overview for generating synthetic dataset in our prior. Although steps are depicted sequentially for clarity, many can be parallelized in actual implementation.
1: procedure SAMPLEDATASET
2:
g + SAMPLESCM)
D Sample data-generating SCM b Expand to functional representation
3:
g + G. E.XPAND))
4:
H < SAMPLESCM)
>> Sample 2n -rd-rrder SCM > Expand to functional representation
5:
H. < H.EXPAND()
6:
C C2,,02 ④t)
> Sample temporal domains Initialize dataset
7:
D 0
8:
for all Ci E C do
9:
Wey H.FORWARD(Ce) Gor G.UPDATE(We)
Sample edge shifts Update edge weights
10:
11:
De. <0
Initialize sub-dataset b Sample sub-dataset
12:
for all E {1, ne do
13:
(2); Mi, Ck) t- Gey FORWARD(s))
Dec + Dec (1...B..c)))
15: end for
16:
D DUDD Den
Extend dataset >
end for
return
Return dataset
19: end procedure
A.5.4 Illustration of Adopted Evaluation Strategy
In Figure 16, we provide an illustration of the Eval-Fix evaluation strategy, originally proposed by Yao et al. [11] in the context of the Wild- Time benchmark. It should be noted that the formalism has been adapted to align with the notation used in this paper.
Dey
D
De,
Dera 100%
D Press
90%
10%
90%
10%
90%
10%
100%
C1
02
C1
Cc+1
C2+-5
Domains
OOD Domains
Dieain
DDD
DOOD
Figure 16: Adapted from the Wild-Time benchmark [11]. this illustration portrays the Eval-Fix evaluation strategy employed in our study. The domain boundary is indicated by ct beyond which datasets are considered part of the out-of-distribute (ID) performance, we subsample 10% of the samples from each dataset prior to this boundary, forming the datasets Ditain and DD
(OOD) test set DOOD To evaluate in-distribution
29

---

A.6
Detailed Overview of Wild-Time Methods
A.6.1
Classical Supervised Learning
Empirical Risk Minimization (ERM). ERM fundamental approach in supervised learning focuses on minimizing the average loss over the training dataset. In Wild-Time ERM is defined as typical supervised learning without making use of any temporal information.
A.6.2 Continual Learning
Fine-Tuning (FT). FT extends the ERM approach by training on the data of each successive temporal domain separately, allowing the model adapt to new distributions but risking catastrophic forgetting of past tasks.
Elastic Weight Consolidation (EWC). EWC [58] counters catastrophic forgetting by adding a regularization term that constrains the changes to important model parameters, thus preserving knowledge from previous tasks.
Synaptic Intelligence (SI). SI [59] captures a synaptic strength metric over time for each model parameter. This metric is used as regularizer to limit changes to important parameters during learning.
Averaged Gradient Episodic Memory (A-GEM). A-GEM [60] maintains a small episodic memory and computes gradients not just for the current task but also the average of the gradients over several past tasks stored in the episodic memory.
A.6.3 Temporally Invariant Learning
Deep Correlation Alignment (Deep CORAL). Initially developed for domain adaptation, Deep CORAL [35] aims to align the second-orded statistics of features between the source and target domains to minimize distribution shift. In the Wild-Time benchmark, this original purpose modified to align features across different temporal domains within the training set, thus converting into a DG method. For handling temporal shifts, it is extended into CORAL-T, which employs sliding windows to segment the data stream into temporal substreams, treating each as a separate domain for alignment. [11]
Group Distributionally Robust Optimization (GroupDRO). Originally designed at optimizing on the worst- performerf group within the training data, GroupDRO [32] aims learn model that robust across varying group distributions. In the Wild-Time benchmark. this method is adapted to the temporal context as GroupDRO-T utilizes sliding window-based segmentation create temporal substreams, treating each as a separate group for distributionally robust optimization. [11]
Invariant Risk Minimization (IRM). IRM [31] aims to identify data representation that is consistently predictive across different domains. In the context of Wild-Time, the method is adapted to temporal shifts and named IRM-T. It employs sliding window-based segmentation create temporal substreams, which are then treated as individual domains for invariant risk minimization. [11]
Mixup. Mixup [33] is an interpolation-based data augmentation technique that creates new training examples by blending the features and labels of existing samples. This technique aims to enhance the model's is ability to generalize across domains by diversifying the training data. It replaces the original training samples with these newly generated interpolations for more robust training.
Learning with Selective Augmentation (LISA). LISA [34], motivated by Mixup, employs selective interpolation to neutralize domain- specific information the training data. comes in two variants: intra-label LISA, which interpolates examples from different domains but having the same label, and intra-domain LISA, which interpolates examples within the same domain but having different labels. In Wild-Time, only intra-label LISA is used [11].
A.6.4 Self-Supervised Learning
Simple Framework for Contrastive Learning of Visual Representations (SimCLR). SimCLR [61] employs contrastive learning to maximize the agreement between different augmentations of the same image, thereby enhancing the quality of learned visual representations. The approach benefits from learnable nonlinear transformations and optimized contrastive loss parameters.
30

---

Swapping Assignments between multiple Views of the same image (SwAV). SwAV [62] employs clustering approach within the contrastive learning framework. enforces consistency between cluster assignments across different augmentations of the same image. This obviates the need for pairwise feature comparisons, offering computational efficiency.
A.6.5 Bayesian Learning
Stochastic Weight Averaging (SWA). SWA [46] averages multiple parameter values along the stochastic gradient descent (SGD) trajectory to improve in-distribution generalization. It operates with minimal computational overhead and aims to approximate the posterior distribution over model parameters, reflected in the flatness of the learned optima. [11]
A.7 Datasets
A.7.1 Validation Datasets
A.7.1.1 Synthetic Datasets
Dataset 1 (Shiftting Sin Classification). The Shifting Sin Classification dataset is a synthetic, binary classification dataset of 1,500 instances evenly distributed across 10 domains. Each domain is differentiated by a unique shift in the offset of the sinusoid wave function, creating distinct decision boundaries for classification. The dataset contains two features corresponding to the æ and coordinates of each instance. Instances are labeled as 1 if they lie above the sine curve and O otherwise in their respective domains.
Dataset 2 (Rotated Five Blobs). The Rotated Five Blobs dataset a synthetically generated dataset consisting of five blobs rotated sequentially counterclockwise -20 around central point in each domain. It comprises two numerical features representing the and y coordinates of each data point. Each blob consists of 40 samples, resulting in 200 samples per domain, for total of 2,000 samples across the 10 domains represented.
Dataset (Moving Square). The Moving Square synthetically generated dataset designed for multi-class classification task. It encompasses two features and is divided into six domains, each containing 200 instances, thereby leading to total of 1, 1,200 samples. the construction of this dataset, each of the four clusters representing distinct classes is initially located on one corner of a square. As we transition through the six domains, each cluster progressively moves along the edge of the square to the next corner.
Dataset 4 (Moving Diagonal Line). The Moving Diagonal Line dataset is a synthetic dataset, generated using the sklearn blobs function. It comprises 1,200 instances, divided across domains, with each domain holding 200 instances. There are two features, corresponding to the æ and coordinate of each instance. In this dataset, there are two clusters, each representing a class, following diagonal line that moves with each domain. Thereby. both clusters move in opposite directions along parallel diagonal next to each other. Each domain in this context represents different stages of the diagonal movement.
A.7.1.2 Real World Datasets
Dataset 5 (Indian Liver Patient Dataset). The Indian Liver Patient Dataset (ILPD), referenced from Ramana and Venkateswarlu [63], is tailored for the binary classification task of identifying liver disease. It contains 583 records featuring 10 attributes, including age, gender, and diverse biochemical measurements. Originating from Andhra Pradesh, India, it comprises 416 liver patient records and 167 non-liver patient records. In our settings, every 5-year age interval is considered as an individual domain. The dataset, sourced from the UCI Machine Learning Repository, is geared towards supporting the diagnosis of liver disease.
Dataset 6 (Istanbul Stock Exchange Returns). The Istanbul Stock Exchange Returns dataset
sourced from the UCI Machine Learning Repository provided by Akbilgic [64] includes 536 instances of returns from the Istanbul Stock Exchange and seven international indices from June 2009 to February 2011. The eight attributes represent various market return indices. The dataset is thereby used to predict the changes in the Istanbul stock exchange given all the other indices. The target was thereby discretized into 9 categories. The data was processed by dropping the USD column of the ISE and converting dates into monthly domain feature, introducing time-based distribution shift.
31

---

Dataset 7 (Diabetes 130-US Hospitals). The Diabetes 130-US Hospitals dataset provided by Strack al. [65] encapsulates decade (1999-2008) of diabetes care across 130 US hospitals, detailing 50 features related to patient demographics and hospitalization details. Criteria for inclusion are inpatient and diabetic encounters, with stays ranging from 1 to 14 days, where both lab tests and medications were administered. Features include patient identifiers, race, gender, age, admission type, duration of stay, attending physician's 's specialty, lab test counts, HbAle Ic results, diagnoses, medication details, and counts of healthcare visits prior to admission. The target of the prediction is to determine whether and in what time frame a patient will be readmitted. is categorized into <30 days, >30 days, No for no readmission. The original dataset, with 101,766 instances and 50 features, has been subsampled to 964 instances.
Dataset 8 (Airlines Delay). The Airlines Delay dataset, sourced from OpenML and provided by Bifet and Ikonomovska [66], contains 539,383 instances, each with 7 features. The task is to predict flight delays based on the scheduled departure information. Features include Airline, Flight, AirportFrom, AirportTo, DayOfWeek, and Time of departure. Notably, departure time, discretized to an interval of full hours, will be our distribution shift domain. The dataset was subsampled, thereby we sampled most 60 samples per discrete time step. This resulted in 1,380 instances.
Dataset 9 (Pima Indians Diabetes). The Pima Indians Diabetes dataset from the National Institute of Diabetes and Digestive and Kidney Diseases, referenced by Smith et al. [67], contains 768 instances and 8 medical diagnostic features. These data represent female Pima Indian patients aged 21 or older. The task involves binary classification for predicting diabetes onset. We categorize each successive 2-year age interval separate domain. highlighting shifts in the dataset across age groups.
Dataset 10 (Diabetes Prediction through Questionaire). This dataset, collected from Sylhet Diabetes Hospital in Bangladesh and provided by Islam al. [68], aims to predict early-stage diabetes. It comprises 520 instances and 16 features, representing symptoms and demographic information of patients. The task is binary classification, predicting whether a patient has diabetes or not. Age groups of every successive 5-year interval are considered as different domains, providing 14 age-based domains.
Dataset 11 (Room Occupancy Detection). The dataset is sourced from the UCI Machine Learning Repository and provided by Candanedo [69]. The preprocessed dataset, reduced to 1800 instances from the original 20,560, is used for binary classification of room occupancy based on Temperature, Humidity, Light, and CO2 levels. After removing 'date' and 'Id' features, 'day" and 'hour' were added. Thereby the 'day' was used as the temporal domain.
Dataset 12 (Sao Paulo Urban Traffic Behavior). The Sao Paulo Urban Traffic Behavior dataset, sourced from the UCI Machine Learning Repository provided by Ferreira et al. [70], captures records of urban traffic behavior in Sao Paulo, Brazil, from December 14 to 18, 2009, and tries to predict the slowness in traffic. The dataset contains 135 instances each with 18 attributes.
Attributes are various traffic indicators such as Hour, Immobilized bus, Broken Truck, Vehicle excess. Accident victim, and more. We have discretized the target "Slowness in traffic (%)" into intervals of 7.5 percent. Each day represents a different domain, thus introducing a time-based shift in the dataset.
A.7.2 Test Datasets
A.7.2.1 Synthetic Datasets
Dataset 13 (Rotated Two Moons). The Rotated Two Moons dataset as stated by Nasery et al. [13] and Bai et al. [12], is a derivative of the 2-entangled moons dataset and includes 220 instances in each of the 10 domains. Each domain is differentiated by counter-clockwin rotations of 18", resulting in rotation of 18 1 in domain i. Also, the distribution of a subset of the instances varies across domains.
Dataset 14 (Intersecting Blobs). The Intersecting Blobs Dataset is synthetically created set tailored for complex, binary classification tasks. The dataset contains two features per sample and 120 samples per domain in a total of 14 domains. Each domain comprises three classes, each represented by 40 samples appearing as blobs. These blobs move and vary, getting quite close to one another, almost intersecting. This dynamic creates sudden shifts in decision boundaries, increasing the difficulty and complexity of the classification tasks. The continuous shifts in blobs' positions across domains are implemented by adjusting their centers and standard deviations.
Dataset 15 (Binary Label Shift). The Binary Label Shift Dataset is a synthetic dataset tailored for binary classification in an environment with prior probability shifts. The dataset consists of
32

---

10 unique domains, with each containing 200 samples and each sample consisting of two features. The key feature of this dataset the systematic manipulation of class probabilities across domains. This is realized by starting with high probability of 0.95 for one class in the first domain, which progressively diminishes to 0.05 in the final domain. Simultaneously, the probability for the other class increases from 0.05 to 0.95, following the opposite direction. This dynamic essentially portrays a "fade out and fade in" pattern of the classes across the domains, representing the prior probability shift.
Dataset 16 (Rotating Hyperplane). The Rotating Hyperplane binary classification dataset is ar- tificially generated based on the package scikit-multifll provided by Montiel et al. [71]. It consists of five features, of which three shift over time. The dataset is divided into 15 domains of 100 instances each, providing a total of 1500 samples. As the name implies, the key aspect of this dataset is about a rotating hyperplane. In other words, the decision boundary or hyperplane shifts navigate from one domain to the next, making the classification task increasingly difficult.
Dataset 17 (RandomRB1 Drift). The RandomRBF Drift binary classification dataset is synthetically mesttmmuliiiil provided by Montiel et al. [71], This dataset has been constructed by introducing drifts in the data using Radial Basis Functions. It is characterized by the motion of cluster centroids, which are responsible for creating data drift. This movement can be visualized as clusters that change their positions, altering the distribution of data over time. The dataset consists of 8 features, 15 domains with 100 samples in each domain, for total of 1,500 samples.
generated based on the package
Dataset 18 (Rotating Segments). The Rotating Segments dataset is a synthetically generated dataset tailored for binary classification task. The dataset visualizes circle partitioned into four segments, similar to the slices of a cake. The data points are thereby labeled alternately. As we traverse through the ten domains, these segments undergo a rotation. Each domain contains 150 samples, accumulating to 1500 samples in total.
Dataset 19 (Sliding Circle). The Sliding Circle dataset is a synthetically generated dataset and represents binary classification task. It comprises two features, the dataset is partitioned into ten domains, each possessing 200 samples, summing up to a total of 2,000 samples. The unique aspect of this dataset is its visual representation: a smaller circle slides around the inner perimeter of larger circle. Within the larger circle, points are classified based on whether they lie inside the smaller sliding circle or outside of it. As we traverse through the ten domains, the position of the smaller circle changes, causing a shift in the classification of the points.
Dataset 20 (Shifting Two Spirals). The Shifting Two Spirals dataset is designed for binary classifi- cation tasks. The dataset visually represents two intertwined spirals. As we move from one domain to the next, the classification boundary in the spirals evolves. Specifically, one spiral gradually transitions its labels from the center towards the outer end, while the other spiral does the exact opposite, transitioning its labels from the outer end towards the center. This dynamic showcases a fascinating interplay of domain adaptation across the ten domains. Each domain has 200 samples, with 100 samples from each spiral, summing up to a total of 2,000 samples.
A.7.2.2 Real World Datasets
Dataset 21 (Free Light Chain Mortality). The free light chain dataset comprises data from a study investigating the link between serum free light chain (FLCC and mortality. It includes 1125 stratified samples per domain and target from an original pool of 7,874, featuring residents of Olmsted County, Minnesota aged 50 or more.
The task involves predicting mortality based on 9 features, which include age, sex. year of blood sample, FLC portions (kappa and lambda), the FLC group, serum creatinine, MGUS diagnosis, and days from enrollment to death or last follow-up. The feature 'chapter' was omitted because it is direct information on whether someone died or not. We treat "sample.yr", the year in which the sample was taken, as a domain, resulting in a total number of 9 domains. We suspect that both the measurements themselves and the selection of participants have changed over time. The dataset is sourced from studies by Dispenzieri et al. [72] and Kyle et al. [73].
Dataset 22 (Electricity). This dataset as used by Harries [74] and Gama et al. [75] contains electricity demand in the Australian New South Wales Electricity Market. Since the prices are not fixed, they fluctuate depending on supply and demand. It has 45,312 instances and 5 features. For reproducibility, the dataset includes additional features such as the New South Wales electricity price, which was
33

---

used to form the target class according to the original paper, and the Victoria electricity price, which was not used in the original paper. The dataset features the demand of electricity in to provinces as well as the transfer between those for periods of 30 minutes. The task is binary classification, which requires predicting whether the price in the current period is higher or lower than the average of the last 24 hours. The dataset contains seasonal data due to varying demand for electricity. The effects of long-term price trends on the class label are removed by the 24-hour moving average. We consider one-week periods as a single domain. To comply with TabPEN sequence length limits, we keep only two hourly intervals for each day and subsample 15 weeks of the whole time period.
Dataset 23 (A (Abeenteeism at Work). The Absenteeism at Work dataset, sourced from the UCI Machine Learning Repository provided by Martiniano and Ferreira [76], comprises 740 instances across 21 features. captures various attributes of employees and their working conditions, such as the reason for absence, day of the week, seasons, distance to work, and more, with the target feature being the absenteeism time hours which was discredited into 4-quantiles.
The primary shift in this dataset is supposed to be seasonal. Thereby each consecutive season is treated as a different domain. Furthermore, no significant preprocessing or subsampling was required due to the manageable size of the dataset.
Dataset 24 (Heart Disease Dataset). The Heart Disease dataset, sourced from the UCI Machine Learning Repository and provided by Janosi et al. [77], targets the classification task of identifying the presence (values 1,2,3,4) or absence (value 0) of heart disease in patients. We treat the task as a binary classification, predicting only whether a patient has heart disease or not. It includes 303 instances, each with 13 health-related features such as age, sex, resting blood pressure, cholesterol levels, etc. In this context, each consecutive 4-year age interval is viewed as a single domain. Rows with missing values have been omitted.
Dataset 25 (Parking Birmingham). The Parking Birmingham dataset, provided by Stolfi [78] via the UCI Machine Learning Repository, initially comprises the capacity and occupancy rtareet of multiple car parks. We have processed it to only include the car park labeled 'Others-CCCPS 13', thereby reducing the number of instances to 1, 1,294 from 35.717. The original 'LastUpdatedd attribute has been transformed into 'day', 'week', and 'month' features, with the 'week' serving as the temporal domain. The 'Occupancy' target, originally an absolute figure, now presented as a percentage of the parking space utilization, discretized into 25% intervals.
Dataset 26 (Ames Housing Prices). The Ames Housing Dataset, curated by De Cock [79], consists of 1460 instances each with 79 features detailing various aspects of residential homes in Ames, Iowa. We use only the training portion of this dataset due to the absence of ground truth targets in the test data. We have discretized the house price, which was originally a continuous variable, into a categorical variable. This transformation was achieved by partitioning the price data into intervals. Specifically, the intervals are defined as: [0, 125k), (125k, 300k), (300k, oo). The task is to predict the price range of a home based on its fcatures. We treat the 'YcarBuilt' attribute, divided into 15 15-year periods, as our domain to capture changes in housing trends over time. It is to be expected that the data set shows temporal shifts, as the price distribution between older and more modern houses differs.
Dataset 27 (Folktables US Census). The folktables datasets, derived from the US Census Public Use Microdata Sample (PUMS) data and published by Ding et al. [42] consists of demographic and socioeconomic data between 2015 and 2021. Each year within this timeframe represents a distinct domain for series of tasks: ACSIncome, ACSPublicCoverage and ACSEmployment. We purposefully limited our focus the state Maryland, to limit the shifts to the temporal domain. The size of the datasets necessitated a stratified subsample for the target per year to reach a total of approximately 1300 instances per dataset and meet the TabPFN model requirements. This subsampling ensured a representative yet computationally manageable sample.
The tasks are as follows:
The task predicts whether an individual's 's income surpasses $50,000, narrow- ing the ACS PUMS data to individuals over 16 who reported at least one working hour per week in the past year and minimum income of $100. The task consists of 10 features accross the 7 domains.
ACInnemm::
The objective is to predict if an individual is covered by public health insurance. The datasct is filtered to include only individuals under 65 with income below
ACSPublicCoverage:
34

---

$30,000, focusing the prediction on low-income individuals ineligible for Medicare. The task consists 19 features accross the domains.
ACEEmployment: The task predict whether an individual is employed. The dataset is filtered to include only individuals between 16 and 90 years of age. The task consists of 16 features accross the 7 domains.
Dataset 28 (Chess). The Chess dataset published by Zliobaité [80] is derived from recorded chess games, aiming to predict game outcomes (draw, lost, won) through a multi-class classification task. consists of nine features which provide insights into the game details and player attributes, including move sequences, player side (white or black), current rating, opponent's rating, type of game, speed, and the date of the game (broken down into year, month, and day). The dataset segmented into 27 domains, where each domain represents 20 consecutive games. This segmentation helps capture the evolution of player's progress over time. The dataset, its entirety, holds 533 instances. has been constructed based on games played between 7 December 2007 and 26 March 2010.
A.8 Hyperparameter Search Spaces
Table 8: Hyperparameter search spaces used for baselines.
XGBoost
Parameter
Values
learning_
(4(-))0)
max depth
44112......
subsample
4(0.2,
colsample_ bytres
(((02,
cobsample_by bylevel
1(002, 1]
min senn_hhld__woo_ weight
g c4(-16,8 16,6)
alpha
(6,2)
lamhda
((-16,2)
gamma
((116)
n. estimators
((000, .01..... 4000)
LightGBM
Parameter
Values
num, leaves
M15, 6....,50 50)
dopth
14,,, 20} (M(-3,0)
learning_rate rate
n_ esstiaatoss
41(50,51,.. 20000
child weight
le-3, 1e-2, le-1. lel, 2/(02, 0.8)
subsample
colsample bytres
(4(0,2, 0.8)
alpha rey. lamhda
0, Je-1. 2, 100
0, le:1, 1, 10, 20, 50, 100
CatRoost
Parameter
Values
learning_rate rundom_ strength
((-5.00
2(1, 2...., 20) (0(0,2ng(1))
P__lea_ree reg
hagging temperature
2/(0.0, 1.0)
leaf estimation iterations
2.1,2..... 20)
iterations
44(10, 101, ,., 4000)
35

---

Table Hyperparameter search spaces we used for Wild-time baselines.
Underlying MLP (+ ERM, A-GEM, FT)
Parameter
Values
vran__pdate__ies iier
500, 1000, 2000, 3000, 4000, 5000, 6000, 7000 (4(-14444)
Ir
use scheduler
True, False
scheduler gamma
(((0.9110))
weight
0.0, le -2 False 0.1.0.15.0. 0.2
early stopping
_urly.stop__ uuddoot holdout
early_stop_peiee
(100, 11,...... 30)
LISA
Parameter
Values
alpha
(,005,4000
mix
True, Falke
Mixup
Parameter
Values
alpha
(((-5,0)
BWC
Parameter
Values
gamma
(1,,0,200)
ewc wwc_lambda
(400,6,2.0)
GroupDRO-T
Parameter
Values
group_size
(11 ......
non_ovee overooaalgpiing
True, False
group. loss, adjustments group_loss_ loss hill
None, 0.1, 0.5,
True, False
SWA
Parameter
swa_ portion swa_Ir factoo
U(0.5, 0.9)
(11, 2...,,6
IRM-T
Parameter
Values
group_size
2..1,2.....
non overlapping
True, False
Jambaa
(11 2...., 100) 0, 250, 500, 750, 100
irm pon.lpyttaannnaallllloo aancall ittrs
S1
Parameter
Values
si_e epsilon
44(0.05, 0.2)
(4(000055 0.002)
CORAL-T
Parameter
Values
group_ size non overlapping
(1(1, (. 6)
True, False
caral_Jambda
110011
36

---

Table 10: Preprocessing search spaces for Drift-Resilient TabPFN and TabPFN-bas
Parameter
Search Space
Ltpe
single
ensemblleeeeee wwwsemble_oonfuuueeeeeeeeeeeeeee configurations
preprocess, peeprocas_transfoneu transforms
See Table 1I
sottmax, Jemperature
1og(0.75), log(0.8), log(0.9), log(0.95) True, False
poly_ features
max max_poly. features
50 -1, 9.0, 12.0
remove outliers
add Jingerprint reatures
True, False 0.9, 0.99,
subsample samples
Table 11: Parameters and values for enumerate pruperateepreproceess transforms function.
Parameter
Values
["safepower"), ["quantile, uni coarse"). l'quantile coarse" I.. ["adnptive"], ["nom_and and_kdd') l'quan- tile_uni" I. I"none"] I. I'robust" ['kdi_uni"], ("kdi" ["kdi_alphaaa0333 0.3",_ alppa__333000 "safepower', "quantileunt" und" ["kdi", "quantile_un"" _un"", (' n'onn"". "power"]
categorical name
("numeric", "ondinal_ vondinal_very. pery_comoon, cwt.ooonnnntttgrrrieee..o shuffled", "onehot", "none")
aopend_originn
[Tric, False]
sabsample_festiie features
0.9, 0.95, 0.9]
global transformer
[None, "svd"]
37

---

B
NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We make the following main claims and give justification alongside:
We claim "Comprehensive evaluations across 18 synthetic and real-world datasets"
We describe evaluation strategies in detail in Section 4 and datasets in detail in Appendix Section A.7.
We claim ".. demonstrate large performance improvements over a wide range of buselines, such XGB, Catboost and TabPFN. Compared to the strongest baselines, improves accuracy from 0.688 744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration." We justify these specific number in evaluations in Section 4.
We claim qualitatively improved adaption to distribution shifts - We show this in Figure 5 and Appendix Sections A.4.3.1, A.4.3.2 and A.4.4.1.
We claim novel technique for combining In- In-Context--ee Learnxi- and Distribution Shifts On May, 22nd 2024 Google Scholar Search yields one paper combining these concepts: "In-Context-Learning and Distribution Shifts" by Ahuja and Lopez-Paz [81]. This paper, however, studies the effects of distribution shifts in-context-learning capabilities, while not exploring strategies to mitigate issues in ICL.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper acknowledges several limitations, including challenges with compu- tational demands when scaling to larger datasets, interpretability. and model assumptions. We discuss those limitations in detail in our "Conclusions and Limitations" Section 5.
Justification:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present new theoretical proofs but focuses on algorithmic development and empirical validation.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
The paper provides detailed descriptions of the experimental settings, datasets, and methodologies used, including links to the code and datasets. See our "Reproduibillit"" Section A.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the full evaluation code, datasets, and pre-trained models at https:/ //tpt//gttttt/tt
TabPFN, See our "Reproducibility"
Section A.I.
6. Experimental Setting/Details
38

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Comprehensive details regarding the experimental setups, including data splits and hyperparameter selection, are thoroughly described, allowing for clear understanding of how experiments were conducted and how results were derived. See our "Reproducibilit"" Section A.I.
Justification:
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The paper reports error bars and statistical significance for the experiments. We
report 95% Confidence Intervals in all our quantitative results and mark this appropriately in the paper.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Details about the computational resources used for the experiments, including the types of processors and the computational cost, are explicitly stated, enabling an accurate estimation of the resources required for replication. See Section A.3 for details.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::///eeuiipppeeeee http::///ccppppcceciciceeeeeeeee
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper thoroughly explores the broader impacts of Please see Section A.2 for details.
TabPFN.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper focuses on a methodological innovation for handling distribution shifts in tabular data, which does not inherently include high-risk data or models such as pretrained language models or image generators. The research does not involve scraped datasets or applications that directly impact individual privacy or security, thus specific high-risk safeguards are not applicable.
Justification:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper credits all external resources and datasets appropriately and adheres the licensing agreements of each. It explicitly mentions the use of datasets, with all necessary citations in Scction A.7.
39

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release documentation of the released models and code alongside the paper. We also provide experimental notebooks for easy usage.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The research did not involve crowdsourcing or direct research with human subjects. Thus, this section is not applicable to the current study.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: As the study did not involve human subjects or sensitive personal data that would require IRB approval or equivalent ethical review, this section is not applicable.
40


Reviews:
[{"review_id": "BwCmLJASyX", "reviewer": "Reviewer_aHHC", "summary": "This paper studies the setting of non-iid train / test distribution shift in the tabular machine learning. The authors extend the previous TabPFN sota tabular model to deal with domain shift cases. TabPFN uses SCM to model data prior, and in this paper, the core idea is to update the SCM prior graph by hypernetworks. The output of hypernetworks is the SCM update. As a result, TabPFN could deal with data shift. The whole idea is very simple and the authors demonstrate the superior effectiveness in various synthetic and real datasets", "questions": "- If I understand correctly, the core part of the method lies in training a hypernetworks to update SCM. All the rest follows TabPFN?  \n- It is vague how hypernetworks is trained and processed. It is suggested to provide details on this\n- The authors mentioned in Figure 4, various cases of distribution shifts, how are these shifts mitigated in the experiments? Any investigation on different cases?\n- How good has the hypernetworks been trained? Since there is many synthetic datasets used, it is great to study cases where hypernetworks updated the SCM correctly\n- How much data is required to train hypernetworks? \n- How is the hypernetworks training related to number of features, number of classes, etc.  since TabPFN has limitations on the dataset requirement.\n- In Table 1, it is recommended to separate real datasets and synthetic datasets. It is also necessary to briefly discuss the nature of the used datasets", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors study the data distribution shift in the tabular domain, which is not commonly studied and provides practical value\n- The authors present a simple method (hypernetwork) to update SCM prior \n- The resulted method has been verified in numerical experiments with synthetic and real datasets", "weaknesses": "The main weakness lies in the experimentation. See the questions below. Technically, the core part is the integration of hypernetworks, which seems very simple. However, in the experiments, there is little discussion of this module. Besides, since many synthetic datasets are used, it is nature to study the SCM from a more explicit way and especially how hypernetworks work well in the setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aHHC,\n\nThank you for your insightful review. We appreciate the time and effort you've invested. Your comments have given us insights to clarify and strengthen several aspects of our paper.\n\nWe are especially encouraged by your recognition of the strong practical value of our approach and that temporal distribution shifts in tabular data are understudied. Your acknowledgment reinforces our belief that this work contributes meaningfully to the field.\nUpon careful consideration of your feedback, we realize that there have been fundamental misunderstandings regarding our work. We believe that using the term \"hypernetwork\" was potentially have thus updated this term to “$2^{\\text{nd}}\\text{-order SCM}$”  throughout the work. In our revised manuscript, we have made a concerted effort to address these areas, providing additional explanations and context to ensure our methodology and findings are more accurately presented.\n\nIn the following responses, we address your questions and recommendations point by point. We have taken the liberty of reordering them for clarity. \n\n> The authors mentioned in Figure 4, various cases of distribution shifts, how are these shifts mitigated in the experiments? Any investigation on different cases?\n\nGreat question! One key advantage of our method is that each type of shift, as well as any combination of them, is handled by our model implicitly, as long as our prior generated those shifts during the pre-training phase. During model development, we extensively analyzed the generated datasets to ensure that each type of shift and their combinations would indeed appear.\n\nRegarding the experiments, while these types of shifts are theoretically interesting on their own, in real-world data, they rarely occur in isolation. Nevertheless, we were also quite interested in investigating how our model deals with the different kinds of shifts, which can be seen nicely by examining the decision boundaries, which can be explored through the demo code in our Colab notebook. To simplify this process, we categorized the 2D-synthetic datasets according to their respective types or combinations of shifts (based on their ID):\n\n- **Prior Probability Shift:** test: [1]\n- **Covariate Shift:** valid: [3]\n- **Concept Shift:** test: [5, 6, 7]\n- **Concept Shift + Covariate Shift:** valid: [1,2], test: [0, 2]\n- **All Types of Shifts:** valid: [0]\n\nIn addition, we now provide three exemplary decision boundaries for each type of shift individually in Figure 1 of the PDF in the global response to all reviewers. When investigating the decision boundaries, we did not notice a significant difference in performance between the different types of distribution shifts, as our model adjusts well to each of them. For combinations of covariate and concept shifts, we observed that our model handles linear shifts particularly well, while (high-dimensional) rotations in synthetic data remain a challenging task. Nevertheless, we perform significantly better than our baselines in these scenarios.\n\n> In Table 1, it is recommended to separate real datasets and synthetic datasets. It is also necessary to briefly discuss the nature of the used datasets\n\nWe agree with your recommendation. In the camera-ready version, we will separate synthetic and real-world datasets in the main text. The initial combination of both types into one table in the main text, with separate tables in the appendix, was primarily due to space restrictions. Now that we have an additional page for the rebuttal, we are including both tables separately in the main text and have added additional discussion of these results to be consistent with the added tables. Additionally, we will provide a brief discussion on the nature of the datasets by adding the following paragraph to L251 in Section 4 - Datasets:\n\n“[...] While some of these datasets have been analyzed in previous work, there has been no comprehensive benchmark focusing on small tabular datasets undergoing distribution shifts. To address this gap, we carefully selected or generated a diverse range of datasets that exhibit temporal distribution shifts. The non-generated datasets were selected from open dataset platforms or previous work in DG. [...]”\n\nPlease note, however, that we had provided an in-depth discussion of each dataset used in our evaluation in Section A.11 of the appendix.\n\n*Due to the extensive rebuttal and feedback we wish to provide, we have split our response into two parts.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJhMIJzaeM", "reviewer": "Reviewer_LELq", "summary": "The paper presents a modification of TabPFN that incorporates a novel prior to incorporate temporal shifts. In particular, the proposed method introduces an additional SCM that, through a temporal representation (Time2Vec), learns to modify the model parameters in response to shifts. The authors train a version of the proposed model and compare it to TabPFN, GBDT baselines, and standard ERM and SWA baselines.\n\nOverall, the paper's presentation is good (but could be improved), and this appears to be a logical addition to the TabPFN method. A method capable of zero-shot transfer to new distributions seems especially well-motivated, given that no labeled data from the target domain is available in many real-world scenarios. However, some revisions and additions to the paper are badly needed. For example, the literature review is inadequate and needs a complete rewrite; there are no domain generalization or robustness methods included in the experimental results (besides the proposed method), and the ablation study is too limited (in fact, the current results suggest that there is no advantage to the Time2Vec addition, which raises questions about the overall approach). I think that with extensiv revisions this paper could be brought up to acceptance level and am open to raising my score, but the scope of additions and revisions feels on the borderline of too much for a simple camera-ready phase.", "questions": "See above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Full review here:\n\n# Major comments\n\n* The paper is framed in a way that I find somewhat misleading: the title, abstract, and main text repeatedly frame the current work as an approach to \"distribution shift\". However, the approach is limited to strictly *temporal* shifts, with no discussion or demonstration of its extensibility to non-temporal shifts. This introduces some unnecessary confusion into the paper and its potential applications -- please either clarify this in the title/abstract/text (preferred), or add a thorough discussion why this should be considered a general method for distribution shift (less preffered unless the authors can already demonstrate this capacity, in which case the current framing does indeed fit).\n\n* I like that the paper unifies all shifts (covariate, label, concept) under a single framework. This is a limitation of some (but not all) existing approaches to domain generalization, and it seems to be a clear advantage of this approach which the authors rightly highlight. However, it would also be useful to see a set of controlled experiments which separately vary these different components, to understand how the model's performance varies as different forms of shift are present; these impacts likely depend on the model's pretraining distribution in subtle ways).\n\n* The \"related work\" section neither surveys relevant related work at a level even close to comprehensive, nor does it actually discuss related works relevant to understanding the current paper. In my opinion, this section should be completely rewritten. For example:  (1) the paper does not discuss related benchmarking or empirical studies such as Shifts [1] and Shift 2.0, Wild-Tab [2], TableShift [3] (already cited but not mentioned in related work despite applicable findings), and WhyShift [4]. All of these should be discussed and related to the current work. (Concurrent work TabRed [8] seems also relevant, but of course was published after this submission.) (2) the paper does not mention at all the many previously-proposed methods for robust learning and domain generalization, including those covered in the previous work. The only methods mentioned, strangely, are two methods that the authors state are *excluded* from the current study. (3) Many works have recently proposed changes to or conducted  empirical studies of TabPFN e,g, [5, 6, 7]; it would be useful to position the current work in relation to these studies.\n\n* *Ablation study*: The ablation study should be expanded, and discussed in the main text. In particular: (1) the current results show no benefit from the Time2Vec addition, which seems to suggest that it shouldn't be included in the model; (2) the authors seem to have only conducted a single run, of the ablated model despite this evidence -- a number of trials equal to the other experiment seems in order. (3) please discuss the results in the main text, even if only in a sentence or two that state the conclusion from the experiment.\n\n* *Resources for baselines*: The authors restrict all baselines to 1200 seconds on 8CPU/1GPU, but the proposed method is trained for 30 epochs on 8 GPUs with a random search over 300 configurations. This setup seems to starve the baselines of resources. Please either provide evidence that the baselines have in fact saturated (i.e. they would not be improved from further tuning) or allocate similar resources to the baselines as to the main model -- otherwise it is impossible to distinguish whether the proposed method simply benefits from more compute + hyperparameter tuning vs. whether it is in fact an improvement.\n\n* *Lack of relevant baselines:* I was quite surprised to see that the authors do not include any domain generalization or robustness methods in their study. Open-source implementations of many relevant methods are available, including (I believe) through Wild-Time and TableShift. Please comment on why these were not included (less preferred) or add them to the experimental results (preferred), also conducting fair hyperparameter tuning as described above.\n\n# Minor comments\n\n* More details on the data should be in the main text. What are the datasets, where are they drawn from, why do the authors use 8 synthetic and 10 real-world? None of these are given, and so it is hard to contextualize the results or assess the reliability of the empirical study.\n\n* It is interesting that CatBoost performs quite strongly on the proposed methods, particularly ID on the synthetic data. This could perhaps be worth mentioning in the main text, as it is in line with the findings of other tabular studies (i.e. CatBoost tends to slightly outperform XGBoost and LightGBM).\n\n# Typos etc\n\n* Abstract: \"with even smaller gains for tabular data\" - this is unclear, please revise.\n\n* L48-49: \"Building on...\" this sentence is a fragment, please revise.\n\n* Table 1: I don't see the acronym \"SWA\" defined in the paper (ERM is not defined either, but this is more familiar to most readers).\n\n* L303: there are 2 lines hanging onto Page 9; the figure/table positioning should be changed to avoid this.\n\n# References\n\n[1] Andrey Malinin, Neil Band, Yarin Gal, Mark Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2022.\n\n[2] Kolesnikov, Sergey. \"Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression.\" arXiv preprint arXiv:2312.01792 (2023).\n\n[3] Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. \"Benchmarking distribution shift in tabular data with tableshift.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Liu, Jiashuo, et al. \"On the need for a language describing distribution shifts: Illustrations on tabular datasets.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Breejen, Felix den, et al. \"Why In-Context Learning Transformers are Tabular Data Classifiers.\" arXiv preprint arXiv:2405.13396 (2024).\n\n[6] Feuer, Benjamin, et al. \"TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks.\" arXiv preprint arXiv:2402.11137 (2024).\n\n[7] Ma, Junwei, et al. \"In-Context Data Distillation with TabPFN.\" arXiv preprint arXiv:2402.06971 (2024).\n\n[8] Rubachev, Ivan, et al. \"TabReD: A Benchmark of Tabular Machine Learning in-the-Wild.\" arXiv preprint arXiv:2406.19380 (2024).", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LELq,\n\nThank you for your thorough review and constructive feedback. We appreciate your recognition of our work's potential and your openness to raising your score. We have worked very hard during the rebuttal to answer your concerns and made major progress in rewriting the “related work” section. We have carefully considered your comments and have addressed them as follows:\n\n### 1. Framing of the Paper\n> The paper is framed in a way that I find somewhat misleading: the title, abstract, and main text repeatedly frame the current work as an approach to \"distribution shift\". However, the approach is limited to strictly temporal shifts, with no discussion or demonstration of its extensibility to non-temporal shifts. This introduces some unnecessary confusion into the paper and its potential applications -- please either clarify this in the title/abstract/text (preferred), or add a thorough discussion why this should be considered a general method for distribution shift (less preffered unless the authors can already demonstrate this capacity, in which case the current framing does indeed fit).\n\nWe fully agree that the focus on temporal distribution shifts should be clear from the start. Therefore, we have revised both the title and the abstract to explicitly highlight this focus. The new title is \"Drift-Resilient TabPFN: In-Context Learning *Temporal Distribution Shifts* on Tabular Data\".\n\n### 2. Controlled Experiments on Different Types of Shifts\n\n> I like that the paper unifies all shifts (covariate, label, concept) under a single framework. This is a limitation of some (but not all) existing approaches to domain generalization, and it seems to be a clear advantage of this approach which the authors rightly highlight. However, it would also be useful to see a set of controlled experiments which separately vary these different components [...]\n\nWe appreciate your positive feedback on the unified approach to handling various types of shifts. In fact, we were also quite interested in the decision boundaries of our models on the different types of shifts, which motivated some of the synthetically generated datasets. In the provided Colab notebook, we give readers easy access to view the decision boundaries on these datasets. For reference, we can categorize the synthetic datasets (based on their id) according to their respective types or combinations of shifts:\n\n- **Prior Probability Shift:** test: [1]\n- **Covariate Shift:** valid: [3]\n- **Concept Shift:** test: [5, 6, 7]\n- **Concept Shift + Covariate Shift:** valid: [1,2], test: [0, 2]\n- **All Types of Shifts:** valid: [0]\n\nIn addition, we now provide three exemplary decision boundaries for each type of shift individually in Figure 1 of the PDF in the global response to all reviewers. However, since we have only few datasets per shift type, quantitative results per individual shift would not allow for significant conclusions.\n\nRegarding the strength of the shifts, we conducted a detailed analysis in Section A.10.6 of the appendix. There, we assessed the impact of combined shifts across both synthetic and real-world datasets. Our findings indicate that as the strength of the shifts increases, Drift-Resilient TabPFN's performance remains more robust and keeps higher scores compared to the baselines.\n\n### 3. Related Work Section\n> The \"related work\" section neither surveys relevant related work at a level even close to comprehensive, nor does it actually discuss related works relevant to understanding the current paper. [...] We appreciate your feedback on the related work section as well as the pointers to more related work. \n\nWe only included a rather short section on related work as the space constraints for our initial submission were very tight. For the camera-ready version we have more space, though, and thus have already started rewriting the related work section from the ground up and will provide it here as a comment in the discussion period as soon as it is ready.\n\nIn our rewrite of the related work section we are integrating work on domain generalization outside of temporal domain shifts, as you suggest, as well as robustness methods, including the benchmarking studies and empirical works you mentioned (Shifts, Shift 2.0, Wild-Tab, TableShift, WhyShift, and the recent TabRed). In addition, we position our work in relation to recent empirical studies of TabPFN.\n\n*Due to the extensive rebuttal and feedback we wish to provide, we have split our response into two parts.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9MYJq4zn1", "reviewer": "Reviewer_84bT", "summary": "The authors propose an extension of the TabPFN framework such that the model can be resilient to distribution shifts during the inference phase by observing the shifted samples in-context. Akin to TapPFN, the pipeline involves pretraining based on an SCM, and the authors introduce a temporal aspect to the SCM construction such that the model will be aware that several common types of distribution shifts can be possible at inference time.", "questions": "Do the authors have any more considerations on the extent to which in-domain (ID) capabilities are affected when adopting the Distribution Shift TabPFN approach as opposed to the original TabPFN? This is hinted in line 287 but it seemed like an important point that gets at some quantitative and motivation-related questions. For instance, what is the false positive rate of the DSTabPFN guessing a distribution shift when there is none? What if there are some outliers in the inference phase despite the underlying distribution will continue longer term? Should users rely on the model to distinguish between one shift to another or only turn to DSTabPFN when a distribution shift is noticed? Some more intuition on this matter may convince readers further.\n\nWhat is the rationale for the compute budget imposed in line 262? Generally eg. XGB would be parallelized with dozens of cores and requires a long time to cross validate & fit, but perhaps this was not necessary for the scale of the datasets explored.\n\nIt would be great to understand how the approach scales with the amount of samples from the SCM. For instance, are x5 more samples (taking x5 times longer to train) needed for the model to be robust to 5 different types of distribution shifts? Or is a similar accuracy to TabPFN possible with an equal number of samples while also being robust to shifts? Is the reason why DSTabPFN does not match the accuracy of TabPFN because it crosses into an overfit regime with too many samples? \n\nA related analysis that could have been nice to have is to see how many “shots” in the shifted distribution are needed for DSTabPFN to adjust appropriately.", "limitations": "The work discusses limitations to a fair extent in the final section. I did not assess that the work could have a major negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper was well structured and well written. The motivation of the work was described in a compelling way. The objective of the work was clearly described. The description of the SCM was sufficiently clear and intuitively it makes sense that the described approach should lead to an improvement in handling cases where a distribution shift is noted. All the figures were helpful in understanding the details of the method, and I found the visualization of the decision boundaries especially convincing. It seems there could be a real world impact in areas where TabPFN-like methods are a good fit based on this work.", "weaknesses": "I did not feel the work had any major structural weaknesses within the scope of what the authors wished to achieve.\n\nA small suggestion that the title should imply that the work is about temporal domain shifts.\n\nThe “runs in seconds” clearly would not be true for all kinds of target scales, although it rightfully highlights the potential of the approach in some scenarios. I recommend adding a short qualifier on the scale where this is true.\n\nAdditional clarity on what exactly is the proposed input and output to the final model would have been appreciated in the methods section. For instance, does the model need to observe all previous intervals eg. [0 1 2] and the future intervals [3 4 5]? Only the previous intervals? Or just a trailing list of recent intervals? What is the justification that future intervals [4 5] will be available to the user if they are still in the present (domain 3)? \n\nDue to some of the ambiguity, it was also not clear at a high level if a user needs to directly indicate to the model that a domain shift is generally expected at certain domain indices, or the model makes such a decision for every domain index internally.\n\n“We approximated domain indices … based on features that encode temporal information, which we transformed into discrete intervals.” It seems that the accuracy attained by the models could vary a bit based on the way that the domains are approximated, unless the discretization was completely random uniform and guaranteed to be fair. Some additional assurance seems deserved on how this approximation was done.\n\nI think the authors motivate the problem enough such that readers will be convinced that existing methods would not do well under distribution shifts. Given this, I think a more competitive benchmark could be to see how well baselines trained on small parts of the OOD do well on the OOD domains, and how this compares with the various ways of applying DSTabPFN. I would be curious to know if this was considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 84bT,\n\nThank you very much for your thorough review and in-depth questions, which have sparked ideas for further analyses and follow-up work. We highly appreciate your positive feedback on the structure and clarity of our paper. We are glad that our approach was communicated comprehensively and that the visualizations were helpful for understanding. Additionally, we are pleased that we were able to compellingly motivate this important problem setting and demonstrate the potential real-world impact of our approach.\n\n> A small suggestion that the title should imply that the work is about temporal domain shifts.\n\nThis is a great suggestion. We have adjusted the title to explicitly highlight this focus. The new title is \"Drift-Resilient TabPFN: In-Context Learning *Temporal Distribution Shifts* on Tabular Data\".\n\n> The “runs in seconds” clearly would not be true for all kinds of target scales, although it rightfully highlights the potential of the approach in some scenarios. I recommend adding a short qualifier on the scale where this is true.\n\nWe completely agree that the \"runs in seconds\" claim is only applicable to certain dataset sizes. We have added a qualifier to clarify that these speeds are achieved within the dataset sizes evaluated in this work (e.g., approximately < 100 features, < 2500 instances).\n\n> Additional clarity on what exactly is the proposed input and output [...]\n\nThis is an excellent question, as other approaches are often more restricted in this respect. Our approach only requires that a temporal domain is reported for all training instances and that the temporal domain is known for the sample we want to predict. Beyond this, all kinds of combinations are possible, including gaps and variable time differences within our training and testing domains (e.g., train on [1, 4, 5, 8] and test on [5, 14, 15]). Overall, the temporal domain is just a special feature of each dataset instance.\n\nRegarding the concern about future domains being available during test time, there is a misunderstanding. Future domains are never provided, as this would violate the principles of Domain Generalization and result in data leakage. If this question arises from Figure 5, where the decision boundary is plotted over time, we clarify that the model is trained only on the domains [0, 1, 2, 3] and then tested individually on [4], [5], and [6]. When predicting domain 5, the model does not have access to data from domains 4 or 6.\n\n> Does the user need to indicate to the model that a domain shift is generally expected at certain domain indices?\n\nNo, the user does not need to indicate to the model that a domain shift is expected at certain indices. Our model internally reasons about the types of shifts present in the training data and determines when they occur, handling everything implicitly without needing external guidance. One insight here is that the weight shifts produced by the hypernetwork can remain constant across temporal domains, resulting in i.i.d. data generated by the data-generating SCM even as the temporal domain index in the dataset changes.\n\n> It seems that the accuracy attained by the models could vary a bit based on the way that the domains are approximated [...]\n\nThis is a valid point. While our model can theoretically handle continuous intervals, making discretization unnecessary, computational limitations require discretization. Pre-training for continuous intervals is currently challenging as each shifted SCM involves a separate set of shifted weight matrices, complicating parallelization of the matrix multiplications.\n\nThe discretized domain approximation should be approximately right as those can guide or misguide the model. However, our model does not fully rely on the domain indices. During pre-training, constant shifts can also be sampled, which helps the model learn to handle potential inaccuracies in domain indices during training. For synthetic datasets, no approximation was needed due to the availability of ground truth domain indices. For real-world datasets, we determined reasonable intervals based on the task at hand, without analyzing the data itself. This process is documented on a per-dataset basis in Section A.11 of the appendix. Thus, the approximation of domain indices can be seen as a downstream domain knowledge task specific to the data the model is applied to.\n\n> Test Baselines trained on small parts of the OOD domains\n\nWhile this is an interesting suggestion, it is currently beyond the scope of our study as it would violate the domain generalization setup and align more closely with a domain adaptation task.\n\n> Considerations on ID capabilities when adopting Drift-Resilient TabPFN compared to original TabPFN. Is DSTabPFN guessing a distribution shift when there is none? \n\nThis is a great question. One consideration is that our prior also generates datasets without any shifts during pre-training. Therefore, if no shifts are present in the training portion of a dataset, our model is unlikely to extrapolate a non-existent shift. This is evidenced by the strong in-distribution (ID) performance of our model, which is only slightly less than the TabPFN base.\n\nHowever, your question has motivated an interesting experiment that we have already started setting up. In this experiment, we evaluate the performance of our model on a strict i.i.d. dataset, where we (1) report all instances as belonging to the same domain, and (2) report different instances as belonging to different domains, even though they are i.i.d. We will share these results as soon as possible during the author discussion period.\n\n*Due to the extensive rebuttal and feedback we wish to provide, we have split our response into two parts.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8xeSCwUqP", "reviewer": "Reviewer_m2Lp", "summary": "This paper aims to address the temporal distribution shifts in Tabular data based on TabPFN. Concretely, the proposed method use two structural causal models (SCM) to model the prior for TabPFN, one for the gradual shift of inductive bias over time and the other is to model the shift of the first SCM. Empirical studies on various datasets illustrate the effectiveness of the proposed approach.", "questions": "1. Why do you need the functional representation graph $\\tilde{\\mathcal{G}}$? What are the benefits of the original sampled graph?\n2. Why do sparse shifts allow for causal reasoning (Line 195)? \n3. How are the edge weights decided?\n4. Why can SCM with NN-based causal mechanisms and nonlinear activations extrapolate values out of data distribution (Line 208)? Is this due to a causal mechanism or NN?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper aims to address temporal shifts in tabular data, which could be a useful setting in practice.\n2. The proposed method intuitively makes sense.\n3. Experiments on synthetic and real datasets have been tested. The code is released.", "weaknesses": "1. As far as I am concerned, CatBoost, XGboost, and LightGBM do not explicitly consider the temporal shift. The proposed method introduced additional sampled data from a temporal shift prior to PFN. I would doubt such a comparison is not that fair. Does any of the baselines in [1]  outperform the proposed method on the datasets used in this paper?\n2. The experimental setting could be more detailed in the main paper.\n3. It seems the proposed method is a simple modification to TabPFN that changes the SGM prior with additional temporal shift add-ons. So it's a little bit combinatorial to me, but not a huge problem.\n4. Modelling the temporal shifts of inductive bias shares similarity with some continual learning approaches using meta-learning techniques, which lack a brief discussion, such as [2, 3, 4].\n\n[1] Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems,36 (2024).\n\n[2] Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 33:16532–16545, 2020.\n\n[3] Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 32, 2019.\n\n[4] On the stability-plasticity dilemma in continual meta-learning: theory and algorithm. Advances in Neural Information Processing Systems, 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer m2Lp,\n\nThank you for dedicating your time to review our work and providing valuable feedback. We really appreciate your recognition that our intuitive approach could address a useful setting in practice.\n\n> As far as I am concerned, CatBoost, XGboost, and LightGBM do not explicitly consider the temporal shift. The proposed method introduced additional sampled data from a temporal shift prior to PFN. I would doubt such a comparison is not that fair. Does any of the baselines in [1] outperform the proposed method on the datasets used in this paper? \n\nWe did compare to all applicable baselines from the Wild-Time benchmark in Table 6 in Section A.10.7 of the appendix. The Wild-Time benchmark is similar to the TableShift benchmark [1], but focused on temporal domain generalization (DG). We will more prominently highlight these methods in our camera ready.\n\nWe compared to other DG methods in the appendix only, as traditional methods are the strongest baselines for out-of-distribution prediction on our Benchmark as can be seen in Table 6. This aligns with findings from both TableShift, you cite, [1] and Wild-Time benchmarks.\n\nOur comparison with DG methods in Table 6 also shows that the performance of DG methods significantly lags behind GBDTs and TabPFN. This discrepancy can be attributed to the nature of our studies, which involve small tabular datasets affected by temporal distribution shifts. Modeling distribution shifts on smaller datasets might be a very different problem from modeling on larger datasets. We'll add further justification for our baseline choices in the camera-ready version.\n\n> The experimental setting could be more detailed in the main paper.\n\nThank you for this feedback. We have further improved the clarity of our experiments. \n\n1. We have separated the quantitative results for synthetic and real-world datasets in the main paper, like previously done in the appendix.\n2. We have included a brief discussion of the nature of our datasets in Section 4 of the main paper.\n3. We have improved the plots in our qualitative analysis by coloring the probability of the most probable class at each point in the plot. These enhanced plots can already be seen in the provided demo code Colab notebook, and we have included one of these plots in the PDF in the global response to all reviewers.\n\nIf you have any specific suggestions to further improve the comprehensiveness of our paper, we would greatly appreciate your input.\n\n> It seems the proposed method is a simple modification to TabPFN that changes the SGM prior with additional temporal shift add-ons. So it's a little bit combinatorial to me, but not a huge problem.\n\nDue to our novel approach to tackling the task of temporal domain generalization being very intuitive, we understand that our work might initially appear simple. However, the problem setting we aim to address and the detailed implementation of our approach are certainly non-trivial, extending the scope of TabPFN in a substantial way and addressing the gap in studying temporal distribution shifts on tabular data. Furthermore, we would argue that having an intuitive approach that significantly outperforms the baselines is a strength rather than a weakness.\n\n> Modelling the temporal shifts of inductive bias shares similarity with some continual learning approaches using meta-learning techniques, which lack a brief discussion, such as [2, 3, 4].\n\nThank you for bringing this very relevant and interesting line of research to our attention. We will add a discussion of it to our camera-ready version.\nThe difference between this line of work, continual meta-learning, and ours is that they assume there is a distribution shift in the datasets they train on (at the meta-level). Whereas we train (at the meta level) on a static distribution across datasets, but deal with a distribution shift within datasets. That is, they assume that the distribution over tasks changes over time, while we assume that the distribution over data points within each task changes over time.\n\n> Why do you need the functional representation graph $\\tilde{\\mathcal{G}}$ What are the benefits of the original sampled graph?\n\nIn our paper, we introduce two distinct representations within structural causal models (SCMs) to fulfill specific functions. The traditional causal representation, denoted as $\\mathcal{G} = (Z, R)$, follows the framework established by Pearl [a], where $Z$ represents causal mechanisms, and $R$ the causal relationships. In this model, each node value $z_i$ is determined through a function $f_i$ that integrates the values of its parent nodes $PA_i$ and an independent noise component $\\epsilon_i$, mathematically expressed as $z_i = f_i({z_j\\ | j \\in PA_i}, \\epsilon_i)$.\n\nThe functional representation graph $\\tilde{\\mathcal{G}}$, however, then utilizes neural networks to model these assignment functions $f_i$ of our causal representation $\\mathcal{G}$. By randomly sampling elements such as linear layers, activation functions, and Gaussian noises, this approach allows the dynamic generation of diverse functional relationships. The functional representation then allows for the propagation of noise through the network, with certain nodes designated as features while others serve as targets in the generated dataset. A key advantage of this method over handcrafted functions is its ability to model a broader range of functional relationships. For a detailed mathematical definition and potential areas for clarification, please refer to Section 3.1 of our paper. We would welcome further feedback to refine our explanations.\n\n*Due to the extensive rebuttal and feedback we wish to provide, we have split our response into two parts.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their constructive feedback and insightful comments. We have addressed all the key criticisms raised in the reviews and made corresponding adjustments to our paper. We are delighted that the reviewers recognize our approach of modeling temporal distribution shifts in TabPFN via a secondary SCM (previously referred to as “hypernet”) as a “logical addition to the TabPFN method" (LELq) and that our “proposed method intuitively makes sense” (m2Lp). We value the acknowledgment that our paper is “excellently” (84bT) presented and “well structured and well written” (84bT). Furthermore, it is encouraging that the setting is recognized as “not commonly studied” (aHHc) and our method as “providing practical value” (aHHc) with a “real-world impact in areas where TabPFN-like methods are a good fit” (84bT). We also appreciate the recognition that the implicit, unified handling of shifts present in the data “seems to be a clear advantage of this approach” (LELq). We believe our contribution adds significant value to the NeurIPS community by demonstrating a novel approach for handling temporal distribution shifts in tabular data.

Regarding the concerns raised by reviewers LELq and 84bT about the computational budget of our baselines, we have conducted additional experiments to address these questions. The new results, included in Table 1 of the attached PDF, compare the performance reported in our work with 20 minutes of hyperparameter optimization (HPO) to the performance after 1 hour for each dataset split. These results demonstrate that, although the HPO had three times more computational budget, the out-of-distribution performance already saturated within the first 20 minutes.

In response to the feedback from all reviewers, we have made several improvements to our paper:
1. **Framing of Our Work:** We have renamed the paper to "Drift-Resilient TabPFN: In-Context Learning *Temporal Distribution Shifts* on Tabular Data" and have updated the abstract to better reflect the scope of our work.
2. **Terminology Update:** We have renamed “hypernet” to “$2^{\text{nd}}\text{-order SCM}$” to avoid confusion with the commonly used term "hypernetworks" in the literature, as our approach differs significantly.
3. **Dataset Discussion:** We have added a brief discussion about the nature of our datasets in the main paper, complementing the in-depth discussion already provided in the appendix.
4. **Quantitative Results:** We have further improved the clarity of our results by splitting the main table of quantitative results into two separate tables that were previously in the Appendix - one for synthetic datasets and one for real-world datasets. 
5. **Improved Visualizations:** We have enhanced the visualizations of the decision boundary to now show the probability of the most likely class at each point. The updated version of Figure 5 is included as Figure 1a in the attached PDF.

We thank the reviewers for their valuable input, which has significantly helped to refine our work. We have addressed all the key points raised during the review and made corresponding adjustments to our paper. We look forward to insightful discussions during the discussion period and are happy to answer any questions or address any misunderstandings that might arise from our responses. We look forward to the opportunity to contribute to NeurIPS and believe our work advances the application of neural networks in domains affected by temporal distribution shifts.

Meta-review: This paper extends Prior-Data Fitted Networks (PFNs) for tabular data (TabPFNs) to handle temporal distribution shifts by augmenting the base model with a structural causal model (SCM) as a prior.  The core idea is to use a hypernetwork (which the authors later call a 2nd-order SCM) to perform amortized computation to adjust the SCM to the drift.  The reviewers were unanimously in favor of acceptance albeit with varying degrees of enthusiasm (1 x borderline accept, 2 x weak accept, 1 x accept).  Reviewer aHHC had the most negative opinion of the work, with the primary criticisms being the lack of clarity in the description of the training process and the work being incremental in nature.  The former issue was cleared up the authors in the rebuttal.  The latter issue still remained, but in my judgement, the paper is a clear extension of TabPFNs, and while the novelty lies in including a standard technique from the SCM literature, the exact methodology does not yet exist and demonstrates empirical merits.  

Reviewer LELq took issue with the paper's framing and presentation of related work (which was echoed in other reviews).  In my own reading, indeed the current draft suffers from some over generalizations (e.g. scoping all shifts vs a focus on temporal, "Until now, no method has significantly outperformed classical supervised learning...") and a lack of organization in its description of related work.  The related work, while there is a dedicated section, is still dispersed throughout the introduction and even in Section 3.  The authors have promised to address these issues in the camera-ready version.  While the authors should certainly make these changes, I feel that these are mostly cosmetic issues that do not jeopardize the core contribution of the work.
Decision: Accept (poster)